Apparatus for detecting motion vector

Abstract: Apparatus for detecting a motion vector and encoding picture signals. Where a motion vector is obtained by block matching employing a luminance pattern obtained by eliminating dc components from a luminance signal of a picture, and thus containing only ac components, is compared to a motion vector obtained by block matching employing a luminance signal of a picture. An optimum motion vector obtained based upon the result of comparison is employed as a true motion vector for variable length encoding the picture signal. In this manner, a correct motion vector may be detected and employed for variable length coding even in cases where the average luminance of the picture on the screen is varied significantly.

Description: The present invention relates to a method and apparatus for detecting a motion vector and for encoding picture signals to transmit or record a moving picture signal from a transmitting side to a receiving side over a transmission channel and displaying the moving picture signal on a receiving side. In a system for transmitting moving picture signals to a remote place, such as a television system, video conferencing system or a television telephone system, it has been a conventional practice to encode picture signals by exploiting the intra-frame correlation or inter-frame correlation of picture signals for signal compaction in order to achieve efficient utilization of the transmission channel. Specifically, where intra-frame correlation is utilized, the picture signals can be compacted by, for example, discrete cosine transform (DCT). WO-A-93/17525 describes an image processing apparatus which produces motion vectors using a first technique unless a match error exceeds a certain level, in which case a second technique is used. On the other hand, should inter-frame correlation be utilized, the picture signals can be encoded for further signal compaction. For example, with reference to Fig. 5a, Fig.5b and Fig.8 supposing that a frame picture PC 1 as a first picture/frame (or search picture) and a frame picture PC 2 as a second picture/frame (or reference picture) are respectively generated at time t 1 and time t 2 . A difference between a picture signal of the frame picture PC 1 and a picture signal of the frame picture PC 2 is calculated, and a differential picture PC 12 is generated by the difference between the two picture signals. Since two frame pictures temporally neighboring each other are usually not changed significantly, the calculated difference between the frame picture PC 1 and the frame picture PC 2 is typically small. This differential may be encoded for compacting the data volume. In addition, the frame picture PC 2 is divided into a plurality of 16-dot by 16-dot blocks (16 vertical dots and 16 horizontal dots). Fig. 8 illustrates a frame picture wherein the blocks are 8-dot by 8-dot. For each of these blocks, picture portions of the frame picture PC 1 having strongest luminance signal pattern similarity thereto are searched by a block matching method. In other words, each block of frame picture PC 1 is sequentially compared to a corresponding block(search block) of pixels in frame picture PC 2 . This search block is of a predetermined size, or number of pixels, and is typically larger than the block of frame picture PC 1 which is under consideration. For example, the search block may constitute an area which is 32 dot by 32 dot (32 pixels x 32 pixels). The above mentioned difference becomes smaller by finding an offset (vector) of the position of the picture portion on the frame picture PC 1 thus detected by such block matching search with respect to the position of the corresponding block in the frame picture PC 2 . This vector is encoded by a variable length encoder 20 shown in Fig.5B and output as a bitstream, while the difference signal thus found is encoded by a variable length encoder 21 and output as a bitstream. The above-mentioned block matching method is further illustrated in Fig.6 and Fig. 8. In this block matching method, the portion of the frame picture PC 2 in a search range SR which has the strongest similarity in luminance pattern to that of the block under consideration BC from frame picture PC 1 is determined. In Fig.6, a block in the current memory M 2 as a reference picture is denoted as A and a block in a search memory M 1 , as a reference picture, whose candidate vector R is offset by ΔX, ΔY is denoted as A'. The sum of absolute values of the differences between the values of the total pixels belonging to the block A in the current memory M 2 and the values of the pixels on the block A' in the search memory M 1 associated with the respective pixel values in the block A, for example, is calculated. This sum value is termed the residue. The residue, herein termed D (ΔX, ΔY), is given by the following equation (1): D(ΔX,ΔY)=Σ |A(X,Y)- A'(X+ΔX,Y+ΔY)| In the search algorithm usually employed, such as full search or multi-search, a vector (ΔX, ΔY) which gives a minimum residue value is selected from among the plural residues D (ΔX, ΔY) obtained in the search area as explained above is simply employed as an optimum motion vector. Meanwhile, there are occasions wherein, if the vector which gives the smallest residue value is simply selected as an optimum motion vector as conventionally, the motion vector selected becomes vitally different from the actual vector which should be found. For example, if the average luminance of the picture on the display screen in its entirety is changed, as when the totality of pixels on the screen gradually becomes higher in luminance such that the average luminance of the picture on the display screen in its entirety becomes brighter (fade-in) or, conversely, the totality of pixels on the screen gradually becomes lower in luminance such that the average luminance of the picture on the display screen in its entirety becomes darker (fade-out), the picture portions having the strongest similarity in the luminance signal pattern are searched by the pattern matching method even though the object remains still on the screen. The resulting vector is significantly different from the real one. Specifically, on fade-in of a still picture shown in Fig.7a, blocks A, B and C in the picture shown in Fig.7b are detected by the block matching method as picture portions having strongest luminance signal pattern similarity to blocks A', B' and C' in Fig.7a. The pixel luminance on a line P'Q' of a pre-fade-in picture in Fig.7a is shown in Fig.7c, while the luminance on a line PQ of a post-fade-in picture in Fig.7b is shown in Fig.7d. Lines P'Q' and PQ each define a predetermined row or line of pixels on a display device. With the block matching method, a luminance portion B shown in Fig.7d, representing the luminance of a block B in the picture shown in Fig.7b, has been retrieved as a block having strongest similarity to a luminance portion B' of Fig.7c representing the luminance of a block B' in the picture shown in Fig.7a. In effect, the luminance of a block portion at the same position as the block B in the pre-fade-in picture of Fig.7a is represented by a luminance portion B'' shown in Fig.7c. Fig. 9 illustrates one example of an apparatus for performing the above described block matching function. It should be noted that reference numerals corresponding to like reference numerals in the illustration of the present invention shown in Fig. 1 represent like elements/components. In view of the foregoing, it is an object of the present invention to provide a method and apparatus for motion vector detecting method and picture signal encoding whereby a correct, or predetermined, vector may be detected although the average luminance on the screen is changed significantly. It is another object of the present invention to provide a recording medium on which picture signals are recorded thereon in accordance with the moving picture detection method and apparatus and the moving picture encoding method and apparatus. In one aspect, the present invention provides a method and apparatus for detecting motion vector wherein a motion vector obtained by block matching wherein a luminance pattern freed of dc components from a luminance signal of a picture and thus containing only ac components is compared to a motion vector obtained by block matching wherein a luminance signal of a picture and an optimum motion vector based upon the result of comparison is adopted as a true motion vector. According to the present invention there is provided a method for comparing data representing a first sequential picture frame and a second sequential picture frame to determine a motion vector comprising the steps of: generating a first motion vector based upon the difference in luminance values by block matching the luminance data between said first picture frame and said second picture frame and wherein said dc components of said luminance values have been removed; generating a second motion vector based upon the difference in luminance values between said first picture frame and said second picture frame; generating a first motion vector by block matching said data of said first sequential picture frame and said second sequential picture frame in which the dc components have been eliminating from the luminance signal so as to include only ac components; generating a second motion vector by block matching said data of said first sequential picture frame and said second sequential picture frame wherein the difference between said first sequential picture frame data and said second sequential picture frame data is calculated; comparing said first motion vector and said second motion vector to determine an optimum motion vector; and outputting either said first motion vector or said second motion vector as an optimum motion vector based upon said comparison. According to the present invention there is also provided an apparatus for detecting a motion vector between a present frame and a preceding frame of an input digital picture signal which includes DC components and AC components comprising: eliminating means for eliminating DC components from the input digital picture signal so as to provide a digital picture signal containing only AC components; first motion vector generating means for generating a first motion vector and first minimum value from the digital picture signal supplied from said eliminating means by utilizing a block matching algorithm; second motion vector generating means for generating a second motion vector and a second minimum value from the input digital picture signal by utilizing a block matching algorithm; evaluation means for evaluating said first and second motion vectors to determine which meets predetermined criteria; and selection means for selectively outputting said first motion vector or said second motion vector on the basis of said determination of said evaluation means. With the picture signal encoding method and apparatus according to the present invention, a true motion vector is obtained using the above motion vector detecting method. Encoding and quantization are performed on a prediction picture signal afforded by the true motion vector. The resulting quantized signal and the motion vector are encoded by variable-length encoding. According to the present invention, a motion vector and minimum value is selected from either a motion vector and minimum value obtained by a conventional block matching method or a motion vector and minimum value obtained by removing dc components from a luminance pattern so as to yield only ac components thereof. The selection between these two motion vectors and minimum values is made based upon the which of the two motion vectors has the minimum value. The motion vector having the minimum value is selected as an optimum motion vector obtained based upon the result of this comparison. This optimum motion vector may be employed for variable length encoding of the video signal even in cases wherein the average luminance of the picture on the screen is varied significantly. The present invention will be more clearly understood from the following description, given by way of example only, with reference to the accompanying drawings in which: Fig. 1 shows a schematic of one embodiment of the present invention. Fig. 2A is a graph showing picture luminance distribution with the method for detecting the motion vector according to the present invention. Fig. 2B is a graph showing picture luminance distribution with the method for detecting the motion vector according to the present invention. Fig.2C is a graph showing picture luminance distribution with the method for detecting the motion vector according to the present invention. Fig.3A is a graph showing picture luminance distribution with the conventional method for detecting the motion vector. Fig.3B is a graph showing picture luminance distribution with the conventional method for detecting the motion vector. Fig.3C is a graph showing picture luminance distribution with the conventional method for detecting the motion vector. Fig.4 is a graph showing a typical example of an evaluation function. Fig.5A illustrates picture signal encoding exploiting inter-frame correlation. Fig.5B illustrates picture signal encoding exploiting inter-frame correlation. Fig.6 illustrates a conventional method for detecting the motion vector. Fig.7a illustrates a motion vector on a picture during fade-in according to a conventional method for detecting the motion vector and the luminance distribution. Fig.7b illustrates a motion vector on a picture during fade-in according to a conventional method for detecting the motion vector and the luminance distribution. Fig.7c illustrates a motion vector on a picture during fade-in according to a conventional method for detecting the motion vector and the luminance distribution. Fig.7d illustrates a motion vector on a picture during fade-in according to a conventional method for detecting the motion vector and the luminance distribution. Fig.8 illustrates a current frame 1 and a reference frame 2 in relation to each other. Fig. 9 illustrates a conventional apparatus for motion vector detection. Block matching methods are described in U.S. patent # 4,897,720 for a Circuit Implementation of Block Matching Algorithm, to Lancelot Wu, et. al. Referring to the drawings, illustrative embodiments of the present invention will be explained in detail. Fig.1 schematically shows a construction of a device for carrying out the picture signal encoding method according to the present invention. Data, or pixel values B, of a first frame picture, or reference picture, are input via input terminal 21 to terminal 20a of input selector 20. Control circuit 22 controls input selector 20 to route the data (pixels value B) from input terminal 20a to either output terminal 20b or 20c. In the case of data, or pixel values B, of a first frame picture, control circuit 22 causes input selector 20 to route the data (pixel values B) from terminal 20a to output 20c where it is then stored in frame memory 1 and routed to average luminance calulating circuit 6. Data, or pixel values A, of a second frame picture, or search picture, is input and routed via input selector 20 to output terminal 20b where it is then stored in frame memory 2 and routed to average luminance calculating circuit 7. First, by way of calculation of residues, the total pixel values in a predetermined block of pixels under consideration, herein represented by a value of a sole pixel A, are read out from the frame memory 2. The pixel values in a block in a search range associated with the pixel A, herein represented by a value of a sole pixel B, are then read out from the frame memory 1. The pixel values A and B read from frame memory 2 and frame memory 1, respectively, in accordance with control circuit 22, are routed to an arithmetic-logical unit 3 where an absolute value of the difference between the pixel values A and B is calculated. This absolute value is routed to an arithmetic-logical unit 4 where the residue D (ΔX, ΔY), which is the total sum of the absolute values, is calculated. For carrying out calculations of the residue in accordance with the motion vector detecting method of the present invention simultaneously with the above-described calculations of the residue, the pixel values A and B are routed to average luminance calculating units 7 and 6, respectively. The average luminance calculating units 7 and 6 calculate average luminance values of the pixels A and B, respectively. In other words for a full evaluation of a picture frame block under consideration, the average value of all the A values and B values is calculated by the luminance calculating units 6 and 7. The average luminance values thus found are routed to an arithmetic-logical unit 8 where an absolute value of the difference between the average luminance value of the pixel A and that of the pixel B is found and routed to an arithmetic-logical unit 9. The arithmetic-logical unit 9 calculates a residue D AC (ΔX, ΔY) based on the absolute value calculated by arithmetic-logical unit 8. The calculations for finding the residue D AC (ΔX, ΔY) will now be explained in detail. The residue is usually employed by circuitry for retrieving picture portions having strongest luminance signal pattern similarity using the block matching method. With the motion vector detecting method according to the present invention, the block matching method is employed in such a manner as to subtract a dc component as a average luminance of a block under consideration from the pixel value in the block under consideration to give a difference, herein termed an ac component, which ac component is employed in calculating the residue. The residue D AC (ΔX, ΔY) is represented by the equation (2): D AC (ΔX, ΔY) =Σ | A(X,Y)- A -(A' (X+ΔX,Y+ΔY) - A')| This value of the residue D AC (ΔX, ΔY) is calculated with the dc component being removed from each pixel so that the average luminance of the block under calculation is equal to zero. Consequently, even if the average luminance is significantly changed between the reference picture (average luminance of search picture = A; average luminance of reference picture = A') and the search picture as in the case of fade-in or fade-out, the block matching may be performed without being affected by changes in the average luminance. Thus a motion vector may be calculated more accurately by comparing the residue D (ΔX, ΔY) and the residue D AC (ΔX, ΔY) to each other for selecting a more appropriate one of the residues. Fig.2a & Fig. 2b shows the luminance distribution in case the dc component has been removed from the pixel value when calculating the residue using the pictures shown in Figs.7a and 7b. Specifically, Figs.2a and 2b show the luminance distribution in case the dc component has been removed from the pixel values on the line P'Q' in Fig.7a and that in case the dc component has been removed from the pixel values on the line PQ in Fig.7b. Lines PQ and P'Q' represent a selected row of pixels across a display screen 700. Thus the difference between the luminance distribution in Fig.2a and that in Fig.2b is as shown shaded in Fig.2c. Fig.3a shows the luminance distribution of the pixel values on the line P'Q' of the picture shown in Fig.7a, as found by the conventional motion vector detecting method. Figs.3b and 3c show the luminance distribution of pixel values on the line PQ in Fig.7b and the difference between the luminance distribution of Fig.3a and that of Fig.3b. The luminance distribution shown in Figs.3a, 3b and 3c is associated with that shown in Figs.2a, 2b and 2c. The arithmetic-logical unit 4 of Fig.1 calculates the residue D (ΔX, ΔY) for the totality of blocks within the search range, while the arithmetic-logical unit 9 of Fig.1 calculates the residue D AC (ΔX, ΔY) for the totality of blocks within the search range. The residue D AC (ΔX, ΔY) as found by the arithmetic-logical unit 4 and the residue D AC (ΔX, ΔY) as found by the arithmetic-logical unit 9 are routed to minimum value calculating units 5, 10, respectively. The minimum value of the residue D (ΔX, ΔY) and the minimum value of the residue D AC (ΔX, ΔY) are selected by the minimum value calculating units 5, 10, respectively. The minimum value of the residue D (ΔX, ΔY) and the minimum value of the residue D AC (ΔX, ΔY), selected by the minimum value calculating units 5, 10, respectively, are routed to an evaluating unit 11 where the two minimum values of the residues are compared to each other in accordance with a pre-set evaluation standard. The resulting value of comparison is fed to signal switching units 12 and 13. The signal switching unit 12 is configured to effect switching between the minimum values of the residues as found by the above-described two motion vector detecting methods. The signal switching unit 12 has its input terminals a and b supplied with the minimum value of the residue D AC (ΔX, ΔY) from the minimum value calculating unit 10 and with the minimum value of the residue D (ΔX, ΔY) from the minimum value calculating unit 5, respectively. The signal switching unit 13 is configured to effect switching between the motion vectors obtained by the above-described two motion vector detecting methods. The signal switching unit 13 has its input terminals c and d with the motion vector from the minimum value calculating unit 10 and with the motion vector from the minimum value calculating unit 5, respectively. These signal switching units 12, 13 are changed over responsive to the results output from the evaluating unit 11 so that the minimum value of the residue and the motion vector as found by one of the two motion vector detection methods are routed to a motion picture encoder 14. Specifically, where the average luminance on the screen is changed significantly, the minimum value of the residue and the motion vector from the minimum value calculating unit 10 are routed via the signal switching units 12 and 13 to the motion picture encoder 14. Otherwise, the minimum value of the residue and the motion vector from the minimum value calculating unit 10 are routed via the signal switching units 12 and 13 to the motion picture encoder 14. The motion picture encoder 14 effectuates the encoding using predicted picture signals and processes the encoded signal in a pre-set manner. The motion picture encoder 14 also quantizes the processed signal and variable-length encodes the quantized signal and the motion vector. The evaluating unit 11 sets, or determines, an evaluation function based upon plural types of sequences and the function thus set is followed without simply selecting a smaller one of the evaluation functions. In other words, evaluating unit 11 outputs a signal based upon which of the minimum values signals D min or D acmin , from minimum value calculating units 5 and 10, respectively, meet the predetermined criteria. Fig.4 shows illustrative examples of the evaluation function(criteria). In this illustration the lines S1 and S2 are defined as follows: S 1 : D acmin. = α 1 D min. + β 1 S 2 : D acmin. = α 2 D min. + β 2 Assuming that, with a minimum value of the residue D (ΔX, ΔY) by the conventional method of detecting the motion vector being D min. and with a minimum value of the residue D AC (ΔX, ΔY) by the present method for detecting the motion vector being D acmin.. A pre-set value of the residue D (ΔX, ΔY) is e n1 which is defined by and corresponds to the point of intersection of lines S1 and S2 as shown in Fig. 4. The minimum value D ACmin of the residue D AC (ΔX, ΔY) is such that D acmin. > α 1 (D min. ) + β 1 , while the minimum value D min. of the residue D(ΔX, ΔY) is such that D acmin ≦ α 2 (D min. ) D min. + β 1 , for 0 ≦ < e n1 . Under the same assumption, the minimum value D acmin of the residue D AC (ΔX, ΔY) is such that D acmin > α 2 (D min ) + β 2 , while the minimum value D min. of the residue D(ΔX, ΔY) is such that D acmin. ≦ α 2 (D min. ) + β 2 , for e n1 ≦ β 2 . α 1 = tan  1 while α 2 = tan  2 . With reference to Fig. 1 and Fig. 4, it can be seen that where the values of D acmin and D min fall within the area "J" above the curve defined by lines S1 and S2, as illustrated for example by point V1 on Fig. 4, the evaluation unit 11 causes the switches 12 and 13 to output the signals from terminals B and D to motion picture encoder 14. Likewise, where the values of D acmin and D min fall within the area "K" below the curve defined by lines S1 and S2 (example: point V2), the evaluation unit 11 causes the switches 12 and 13 to output the signals from terminals A and C to motion picture encoder 14. Motion picture encoder 14 may be, for example, a MPEG or MPEG II type motion picture encoding device. In view of the above description of the present invention, it will be appreciated by those skilled in the art that many variations, modifications and changes can be made to the present invention without departing from the scope of the present invention as defined by the claims appended hereto. All such variations, modifications or changes are fully contemplated by the present invention.

Claims: A method for comparing data representing a first sequential picture frame and a second sequential picture frame to determine a motion vector comprising the steps of: generating a first motion vector by block matching said data of said first sequential picture frame and said second sequential picture frame wherein the dc components have been eliminated from the luminance signal of both sequential picture frames so as to include only ac components; generating a second motion vector by block matching said data of said first sequential picture frame and said second sequential picture frame wherein the difference between said first sequential picture frame data and said second sequential picture frame data is calculated; comparing said first motion vector and said second motion vector to determine an optimum motion vector; and outputting either said first motion vector or said second motion vector as an optimum motion vector based upon said comparison. A method for encoding a picture signal comprising the steps of: generating an optimum motion vector in accordance with the method of claim 1; effecting encoding and quantization on a prediction picture signal generated using said optimum motion vector; and variable-length encoding the resulting quantized signal and the motion vector. A method according to claim 1 or 2 in which: the step of generating a first motion vector includes generating a first minimum value; the step of generating a second motion vector includes generating a second minimum value; and the step of comparing determines the optimum motion vector on the basis of said first minimum value and said second minimum value. An apparatus for detecting a motion vector between a present frame and a preceding frame of an input digital picture signal which includes DC components and AC components comprising: eliminating means (8) for eliminating DC components from the input digital picture signal so as to provide a digital picture signal containing only AC components; first motion vector generating means (8, 9, 10) for generating a first motion vector and first minimum value from the digital picture signal supplied from said eliminating means by utilizing a block matching algorithm; second motion vector generating means (3, 4, 5) for generating a second motion vector and a second minimum value from the input digital picture signal by utilizing a block matching algorithm; evaluation means (11) for evaluating, based on said first and second minimum value, said first and second motion vectors to determine which meets predetermined criteria; and selection means (12, 13) for selectively outputting said first motion vector or said second motion vector on the basis of said determination of said evaluation means. An apparatus according to claim 4 wherein said first motion vector generating means further comprises calculation means (9) for calculating an integration value of the absolute values of differences in luminance values in each associated position between a reference block in a preceding frame and a block under consideration in the present frame. An apparatus according to claim 5 further comprising first means (10) for detecting the smallest integration value and generating the first motion vector based upon a position of the reference block having the smallest integration value wherein: said second motion vector generating means comprises calculation means (4) for calculating an integration value of the absolute values of differences in luminance value in each associated position between a reference block in a preceding frame and a block under consideration in the present frame and second means (5) for detecting the smallest integration value and for generating the second motion vector in accordance with the smallest integration value; and said evaluation means (11) evaluates which of said first or second motion vector is preferable on the basis of the smallest integration values supplied from said first detecting means and said second detecting means. An apparatus according to any one of claims 4-6, wherein: said first motion vector generating means includes first arithmetic logic unit for calculating a first difference signal between luminance values of said present frame and said preceding frame data in accordance with the expression: |A - B| where A represents a luminance value of said preceding frame and B represents a luminance value of said present frame wherein said first motion vector and first minimum value are calculated in accordance with said output from said first arithmetic logic unit; said eliminating means includes first average luminance calculating unit for calculating the average luminance value of said preceding frame data and second average luminance calculating unit for calculating the average luminance value of said present frame data; said second motion vector generating means includes a second arithmetic logic unit for generating a second difference signal in accordance with the expression: (A - A avg. ) - (B - B avg. ) where A avg. represents the total average luminance value of said preceding frame and B avg. represents the total average luminance value of said present frame; said evaluation circuitry compares the value of said first difference signal with the value of said second difference signal to determine which value meets predetermined criteria. An apparatus for encoding a video signal comprising an apparatus according to claim 7 and encoder circuitry for encoding said video signal in accordance with said optimum motion vector and value.

Inventors: ODA TSUYOSHI, UCHIDE KOUICHI, YASUDA KAZUNORI, ODA, TSUYOSHI, UCHIDE, KOUICHI, YASUDA, KAZUNORI
Applicants: SONY CORP, SONY CORPORATION
Application year: 1995
Main classification: H04N7
IPC codes: H04N5, H04N7