Motion vector processing device

Abstract: In motion compensated interpolation of TV-pictures it is usual to divide the picture into a plurality of blocks, a motion vector being determined for each block. This division into blocks has the disadvantage, that sometimes block boundaries become visible (dirty window effect). The invention generates a continuous vector field from the blocked vector field with the aid of a median filter which calculates for each subblock (H1) from a number of subblocks (H1...H4) into which each block (H) is divided, a motion vector based on the motion vectors of the original block (H) to which the subblock belongs, and of the original blocks (E, G) adjacent to the subblock (H1) concerned.

Description: The invention relates to motion vector processing (including storage or transmission), for processing motion vectors corresponding to pictures being subdivided into a number of blocks. For each of these blocks a motion vector is available for efficient storage, transmission or processing (e.g. scan rate conversion) etc. The availability of only one vector per block of pixels can give rise to the visibility of these blocks in a picture obtained by such motion compensated processing. As a possible solution to reduce the visibility of the blocks, the vector field could be post-filtered, but this has the disadvantage that new vectors are generated, the quality of which was never checked. US-A 4,703,350 describes such a post filter, which uses a raised cosine function or a linear interpolation filter to obtain motion vectors for each pixel. When the estimator uses a matching criterion, it is possible that the average of two vectors each yielding a good match, results in a vector that gives a very bad match. One could for instance think of a periodic structure on which several matches are possible, but on which the average vector may yield a bad match. EP-A-0,293,644 discloses a method for the determination of motion vector fields from digital image sequences. A motion vector field is derived from two successive image frames. The motion vector field relates a picture element of the other image frame to every picture element of the one image frame, whereby the relation is defined by a motion vector which reproduces the displacement of the picture elements one another, and whereby respectively all picture elements in a square or rectangular block of picture elements receive the same motion vector. The determination of the motion vectors is carried out by minimization of a composite objective function which, first, takes into consideration the difference in luminance values of the mutually allocated picture elements of the two established frames, and, then evaluates or weights the differences between adjacent or neighboring motion vectors, evaluating or weighting these with the assistance of a smoothing measure. The minimization of this objective function is carried out in such fashion that, first, the motion vectors minimizing the objective function are determined, given the restriction that the motion vectors in blocks larger than the blocks ultimately desired are constant, and that, subsequently, each of these blocks (16x16) is subdivided into smaller, preferably equalsized blocks until the desired block size (4x4) is achieved. In every stage of the block subdivision, every individual motion vector is optimized by a variation in an appertaining value range until a smaller value of the objective function can no longer be found for any motion vector in this way. It is (inter alia) an object of the present invention to provide a post operation on the vector field which - as does the linear post filter - yields a smooth vector field, but which does not generate unchecked motion vector components. For this purpose, a first aspect of the invention provides a motion vector processing device as defined in claim 1. A second aspect of the invention provides a motion vector processing method as defined in claim 4. According to these aspects of the invention, the output motion vector components which correspond to the subblocks into which each block has been divided, are selected from the input motion vector components which have been estimated for the original, larger blocks. Therefore, no new unchecked motion vector components are generated. Advantageous embodiments are defined in the dependent claims. The embodiments of claims 2 and 5 ensure not only that the output vector components correspond to selected input vector components, but also that the output vectors are not composed from components which do not belong to each other. A very smooth vector field and, consequently, very good motion compensated pictures appeared to be obtainable when using the embodiments of claims 3 and 6. If desired, the obtained subblocks may be further subdivided. If the subdividing operation is repeated often enough, motion vectors are obtained for each pixel. These and other (more detailed) aspects of the invention will be described and elucidated with reference to the drawings and examples. In the drawings: Fig. 1 gives an example of a vector field; Fig. 2 shows a vector field in which one block is divided into four subblocks; Fig. 3 shows a two-step block erosion; Fig. 4 shows a first embodiment of a one step block eroder; and Fig. 5 shows a second embodiment of a one-step block eroder. Fig. 1 gives an example of a vector field. For each block A..0 of pixels only one vector is available. As is shown for a given block H in Fig. 2, in an embodiment of the present invention, each block is divided into four subblocks. The x and y vector components for a subblock become the respective median values of: 1. the x and y vector components of the given block that contains the subblock; and 2. the x and y vector components of two neighboring blocks which are adjacent to the subblock. The x and y vector components are processed separately. Because of this separate operation, a new vector, which belongs neither to the given block nor to either of its two neighboring blocks can be created. That is why, according to a preferred embodiment of the invention, after the median operation it is checked whether the new vector is equal to one of the three input vectors. If not, the vector of the given block that contains the subblock is taken instead of the new vector; if yes, the new vector itself is used as the vector corresponding to the subblock for subsequent motion compensated processing. So for the subblocks H1, H2, H3, H4, the vector value is calculated as follows, using symbols derived from programming languages. H.x indicates the x component of the motion vector of block H. <> means "is not equal to". H1.x = median (H.x, E.x, G.x) H1.y = median (H.y, E.y, G,y) If (H1 <> H) and (H1 <> E) and (H1 <> G) then H1 = H H2.x = median (H.x, G.x, K.x) H2.y = median (H.y, G.y, K.y) If (H2 <> H) and (H2 <> G) and (H2 <> K) then H2 = H H3.x = median (H.x, I.x, K.x) H3.y = median (H.y, I.y, K.y) If (H3 <> H) and (H3 <> I) and (H3 <> K) then H3 = H H4.x = median (H.x, I.x, E.x) H4.y = median (H.y, I.y, E.y) If (H4 < > H) and (H4 < > I) and (H4 < > E) then H4 = H This process can - if necessary e.g. for large initial block sizes - be repeated for the subblocks. Each subblock is then subdivided into four parts. Subblock H1 for example, is divided into the parts H1a. H1d. The resulting vector becomes the median value of the vectors from the subblock and its neigboring blocks. Again, according to said preferred embodiment, it is checked that only existing vectors result from the process. Fig. 3 illustrates the process. The vector for part H1a now becomes: H1a.x = median (H1.x, E2.x, G4.x) H1a.y = median (H1.y, E2.y, G4.y) If (H1a < > H1) and (H1a < > E2) and (H1a < > G4) then H1a = H1 Fig. 4 shows a first embodiment of a one-step block eroder according to the invention. The example shown calculates 4 motion vectors for each one of the subblocks H1..H4 of block H. The implementation shown deals only with the x components of the vectors; the y components are processed similarly. As is apparent from Fig. 2, for subblock H1, blocks E, G and H are relevant; for subblock H2, blocks G, K and H are relevant; for subblock H3, blocks K, I and H are relevant; and for subblock H4, blocks I, E and H are relevant. Under the control of a subblock pointer sb, multiplexers 41x and 43x select the relevant motion vector components of the relevant neighboring blocks for application to the subsequent median filtering operation. Comparators 45x, 47x and 49x and look-up-table (LUT) 51x determine the median of the vector components 1 and 2 outputted by multiplexers 41x and 43x, respectively, and of the vector component 3 of block H itself. For a more detailed description of median determination, reference is made to US-A 4,740,842. Controlled by LUT 51x, multiplexer 53x selects the median of vector components 1, 2 and 3 for outputting as the x-component of the subblock concerned. Alternatively, multiplexer 53x could select from the input vector components E.x, G.x, K.x, I.x and H.x. In that case, the subblock pointer sb should be applied to LUT 51x as well. Fig. 5 shows a second embodiment of a one-step block eroder, in which both x and y components are dealt with by LUT 51' to ensure that the output x and y vector components originate from the same input vector. In the programming language statements mentioned above, this checking operation corresponds to each third statement. When the motion vector estimator itself already yields a rather smooth vector field, as e.g. the estimator described in Applicants' article "New algorithm for motion estimation" presented at the Third International Workshop on HDTV, Torino, 1989, this checking operation can be left out, since simulations showed that the checking operation changed the decision for only a small percentage of blocks, and leaving it out introduced no visible artifacts. The difference between Figs. 4 and 5 is mainly that separate LUTs 51x (shown) and 51y (not shown) are replaced by a single, more complex, LUT 51' which in addition has to perform the abovementioned checking operation. In Fig. 5 one dual 3-state multiplexer 53' is shown; however, two multiplexers 53x (as shown in Fig. 4) and 53y both controlled by LUT 51' are also possible. As already mentioned with reference to Fig. 4, if in an alternative embodiment the subblock pointer would be applied to LUT 51', one dual 5-state multiplexer or two separate 5-state multiplexers would also be possible.

Claims: Motion vector processing device for processing motion vectors corresponding to pictures being subdivided into a plurality of blocks, for each block a motion vector being available, comprising: means (E.x-H.x; E.y- H.y) for providing x and y motion vector components of a given block (H) and of blocks (E, G, I, K) adjacent to said given block (H); and characterized by: means (41-53) coupled to said providing means (E.x-H.x; E.y- H.y) for supplying, for each subblock (H1) of a number of subblocks (H1-H4) together forming said given block (H), x and y motion vector components respectively selected from said x and y motion vector components (H.x, H.y) of said given block (H) and from the x and y motion vector components (E.x, G.x, E.y, G.y) of two blocks (E, G) adjacent to said subblock (H1). Motion vector processing device as claimed in Claim 1, wherein said supplying means (41-53) comprise: means (51') for checking whether selected x and y motion vector components supplied for a same subblock (H1) correspond to a same block (H, E, G), to deliver said motion vector (H.x, H.y) corresponding to said given block (H) if said selected x and y components supplied for the same subblock (H1) do not correspond to the same block (H, E, G), and to deliver said selected x and y motion vector components otherwise. Motion vector processing device as claimed in Claim 1 or 2, wherein said supplying means (41-53) include means (45-53) for obtaining x and y median filtered vector components from the x and y motion vector components (H.x, H.y) corresponding to the given block (H) and from the two x and two y motion vector components (E.x, G.x, E.y, G.y) corresponding to the two adjacent blocks (E, G), to provide said selected x and y motion vector components. A method of processing motion vectors corresponding to pictures being subdivided into a plurality of blocks, for each block a motion vector being available, comprising the step of: providing (E.x-H.x; E.y- H.y) x and y motion vector components of a given block (H) and of blocks (E, G, I, K) adjacent to said given block (H); and characterized by the step of: supplying (41-53), for each subblock (H1) of a number of subblocks (H1-H4) together forming said given block (H), x and y motion vector components respectively selected from said x and y motion vector components (H.x, H.y) of said given block (H) and from the x and y motion vector components (E.x, G.x, E.y, G.y) of two blocks (E, G) adjacent to said subblock (H1). A motion vector processing method as claimed in Claim 4, wherein said supplying step (41-53) comprises the step of: checking (51') whether selected x and y motion vector components supplied for a same subblock (H1) correspond to a same block (H, E, G), to deliver said motion vector (H.x, H.y) corresponding to said given block (H) if said selected x and y components supplied for the same subblock (H1) do not correspond to the same block (H, E, G), and to deliver said selected x and y motion vector components otherwise. A motion vector processing method as claimed in Claim 4 or 5, wherein said supplying step (41-53) includes the step of obtaining (45-53) x and y median filtered vector components from the x and y motion vector components (H.x, H.y) corresponding to the given block (H) and from the two x and two y motion vector components (E.x, G.x, E.y, G.y) corresponding to the two adjacent blocks (E, G) to provide said selected x and y motion vector components.

Inventors: DE HAAN GERARD, HUIJGEN HENDRIK, DE HAAN, GERARD, HUIJGEN, HENDRIK
Applicants: PHILIPS ELECTRONICS NV, PHILIPS ELECTRONICS N.V.
Application year: 1990
Main classification: G06T7
IPC codes: G06T7, H04N7, H04N11, H04N5