A BLOCK-BASED DEPTH RECONSTRUCTION METHOD FOR A STATIC DISPLAY SCENE WHICH IS CONTAINED IN A SEQUENCE OF FRAMES THAT ARE DIVIDED INTO AN ARRAY OF BLOCKS

Abstract: A method for assigning block-wise depth-associated values for representing a static scene, to each of a sequence of frames that are dicided into a respective pattern of blocks.

Description: A block-based depth reconstruction method for a static display scene which is contained in a sequence of frames that are divided into an array of blocks. BACKGROUND OF THE INVENTION The invention relates to a method as recited in the preamble of Claim 1. Block- based reconstruction of images, as distinct from feature-based reconstruction, has a great potential for attaining high-speed solutions because many operations therefor may be executed in parallel. Furthermore, the matching between various images is systematically executed for all blocks in the image, and so, the solution is well applicable to situations where only few and/or only vague features might be identified in the image. Finally, the method of the present invention may serve as a preprocessing step for a feature-based method so that the combination of the two methods can be more economical with respect to the amount of processing required. By itself, feature-based reconstruction is part of the prior art. Usage of the present invention would inter alia be in the interpolating between successive frames. Previous block-based depth reconstruction algorithms have assigned a single depth value to each rectangular block of image pixels, often making the depthwise resolution of the depth map courser than the original image. Furthermore, the prior art method is essentially just a motion estimation algorithm, so that therein a set of depth data is assigned on the basis of only two successive images in the original sequence. The prior art will therefore produce a depth map with only a low resolution, and only a limited accuracy due to its inability to use time integration based on a longer sequence of images received. The present invention undertakes to mitigate these two problems. SUMMARY TO THE INVENTION In consequence, amongst other things, it is an object of the present invention to allow producing a depth map with improved resolution and accuracy, due inter alia to the assigning of a depth-associated value to each pixel of the block, to the allowing of a plurality of depth candidate depth values per block, and to systematic pruning of an excessively large set of candidate depth values. Now therefore, according to one of its aspects the invention is characterized according to the characterizing part of Claim 1. The invention also relates to an apparatus arranged for implementing the above recited method. Further advantageous aspects of the invention are recited in dependent Claims. BRIEF DESCRIPTION OF THE DRAWING These and further aspects and advantages of the invention will be discussed more in detail hereinafter with reference to the disclosure of preferred embodiments, and in particular with reference to the appended Figures that show: Figure 1, an image field as distributed into blocks and pixels; Figure 2, the relative spatial mapping of a block with varying instants in time; Figure 3, a first mapping of two blocks in two successive frames; Figure 4, a second mapping of two blocks in two successive frames; Figure 5, a mapping of a block versus a scene element; Figure 6, an apparatus arranged for implementing the inventive method. DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS Therefore, the principle of the present invention is as follows: a. For storing a frame, each pixel of a block uniformly gets the depth-associated value that has been assigned to the overall block in question. b. For assigning a depth-associated value to a block, usage is made of one or more candidate depth-associated values each deriving from earlier blocks in the sequence of frames mapping on that particular block; the actual assigning may however be postponed through retaining a plurality of candidates. c. The eventual assigning uses an error minimization procedure based on the actually available candidates for the block in question. In addition, in its preferred embodiment, the method proposed herein uses the following further measures, although various ones thereof, as recited supra, may either be left out, or be restricted to a less extensive version thereof instead: 1. Instead of the depth value itself, the preferred embodiment employs as depth- associated values rather reciprocal depth values, that are represented as uncertainty intervals (v, d), wherein v is an expected depth-associated value, and d is an estimated uncertainty therein, preferably the standard deviation in a Gaussian probability distribution. Instead of depth values proper, the preferred embodiment uses reciprocal depth values to reflect differences in accuracy between depth assignments to nearby parts of the scene and to remote parts thereof. It is expected that through using reciprocal depths, the achievable absolute accuracies will be roughly uniform across those nearby and remote parts. 2. Every pixel has such an individual depth uncertainty interval assigned thereto. 3. For the first two time steps, i.e. frames at instants t and t+1, a conventional block based matching procedure is used. This means that a single reciprocal depth value is assigned to every standard block, measuring for example 5x5 pixels, together with an estimated value for the associated uncertainty interval. 4. All pixels in that block receive this depth-associated value plus its associated uncertainty interval. 5. Next, the information between t+1 and t+2 is taken into account by interpreting each pixel in frame t as a single point plus an associated depth interval in 3-D space, and on the basis of an assumed camera motion, that is presented from an external world, transforming these points to the coordinate system associated to the image at t+1. 6. Then, the frame at t+1 is again partitioned into the standard-sized blocks. A block at time t+1 may then contain pixels that have originated from more than a single block of those associated to time t. As a result, more than a single candidate depth value may be associated to the later block or part thereof. 7. The block-based depth estimation procedure for the frames t+1, t+2, etcetera operates in the same manner as for the previous frames, except that the number of candidate depth values may grow still further. In particular, each block may contribute more than one depth value to the candidate set relating to the subsequent block of a later frame that has parts being mapped from the various earlier blocks. 8. In order to avoid too large candidate sets, first the number of values in one candidate set will be reduced. This may take place as follows: (a) a first reduction may be effected on the basis of confidence, wherein improbable depth associated values are deleted. For example, this may pertain to a candidate value deriving from only a very small part of an earlier block. (b) next, assume the candidates to be the ordered sequence v[0], v[l], ... v[n-l], wherein v[0] is the smallest one among the candidate values. We assume a tolerance value ε to be postulated and do not discriminate between depth values that are closer to each other than ε. (c) now, set of candidates w[0], w[l], .... is reduced as follows: w[0] = v[0]; i = l; j = l while (i < n) { if ((v[i] - w[j - 1] > epsilon ) or ( i = n-1 )){ w[j] = v[i]; j = j + 1;} i = i + 1; } Note that the number of candidates remains larger if the total range of their initial values was larger. 9. After completing the traditional block-based matching algorithm between frames t+1 and t+2 with the larger candidate sets, we have a set of depth estimates for the blocks in frame t+1. These values are used to update the values in the pixels in the blocks as follows: (a) consider a pixel with a reciprocal depth and associated uncertainty (v p , d p ), whereas the block to which the pixel belongs got ( b , dj, ) as a result of the block-based algorithm. (b) then the new value for v p , say v p ' , should be a convex combination of v p and V b , say v p ' = kv p + ( 1 - k ) vj,, which represents in fact a mixing procedure. A large value of k, 0 ≤ k < 1 expresses that the history of the depth assignments to that pixel brings about only a small contribution from time integration, whereas a small value expresses that the history, in casu the integration over the previous values, is important. Of course, other convex combinations are feasible as well. (c) the new value for d p , say d p ' should depend on the following items: • the previous d p , for example, proportionally • the value d b , for example, proportionally • the confidence level of the block to which the pixel belongs, such as inversely proportional • the agreement between v p and V b , that should be inversely proportional to |v p - v b |. 10. This process is iterated over the next time steps. Furthermore, pixels may occur that do not result from any earlier pixel being transformed to them. These "new" pixels can receive an initial depth assignment which could be taken as the average of their immediate neighbours; their uncertainty value should then be set to a large value. As to an embodiment of the present invention, Figure 1 shows a simplified image field or frame as distributed into pixels. The field 20 is made up of three rows and four columns of uniform blocks 22. In practice, the number of blocks is generally much higher. Block 22 is made up of four rows and four columns of pixels 24. In practice, the number of pixels of a block is often a square of an odd integer, such as 5X5 pixels. In contradistinction, the present invention may be practiced with a frame that is divided into non-uniform blocks, for example, in that the center of the frame would have relatively small blocks, whereas the perimeter of the frame would have relatively larger blocks. Furthermore, successive frames may be distributed into blocks in a non-uniform manner. For example, frames with relatively little spread in depth could get another pattern than frames having a greater spread in depth. Figure 2 shows an example of the relative spatial mapping of a block with varying instants in time. At a first instant, block 26 is situated according to the solid lines. At a next (or previous) instant, the corresponding block is situated according to the dashed lines. In this case, there is a uniform shift of two thirds of a pixel to the right, and one third of a pixel downward. Of course, the two shifts are relatively independent. Further feasible changes are rotation and change in magnification. All changes recited may occur in combination to produce about all possible movements of a physical object that is the source of the image. The method according to the present invention receives a motion quantity for the camera that images the scene from an outside entity. By itself, motion estimation is part of the prior art. For example, the motion value could derive from a known motion of a camera mounted on a robot arm. Given this motion, the present invention undertakes to assign a blockwise depth- associated value, as pertinent with respect to the plane of the image. The lateral motion will therefore give rise to a depth parameter with respect to the image plane, that may be individual to each respective pixel. The above scheme may be used for interpolating intermediately between successive images of a stream of images. The pixels come into various categories. First, the pixels in each of the two images fall completely within the block in question. Second, in either the first image, or in the second image, or in both, the pixel only falls partially inside the block in question. Third, in one of the two images the pixel in question falls completely outside the block in question. This feature may also occur in combination with the second case. Figure 3 shows a first mapping of two blocks bl(t) and b2(t) with depth- associated values zl and z2, respectively, in two successive frames t and t+1. Here, relative motion between the instants pertaining to frames t and t+1 is such that their block mappings recede from each other. This gives a relatively simple assigning of depth-associated values to the frame at t+1, because the set of candidates is only small. Figure 4 demonstrates a second mapping of two blocks b3(t) with depth- associated value z3 and b4(t) with depth-associated value z4 in two successive frames t and t+1. As shown, the relative motion between the instants pertaining to frames t and t+1 is such that their block mappings approach each other. This would make assigning a depth-associated value to the block b5(t+l) of the frame at t+1 relatively complex, because both parent blocks could produce a candidate depth-associated value. In practice, the set of candidates could be much larger than two. The invention prescribes that at some instant in time a choice must be effected, but that such choice may to some extent be postponed. Figure 5 demonstrates a mapping of a block 20 versus a scene element 22 in frame 24. The present invention recognizes that the assigning of a single depth-associated value to block 20 may raise the granularity of the block-based representation which in turn would decrease the representative quality. Through taking the history of all individual pixels belonging to the block in question into account, a distribution of differing depth-associated values over the block in question may still be achieved. Therefore, by delaying the actual assigning, the quality may be enhanced. Figure 6 illustrates an apparatus that has been arranged for implementing the inventive method. Storage block 30 receives the block-wise depth-associated values and for each thereof an uncertainty interval value (INI) and stores these two data separately for each pixel. Motion-implementing block 32 is fed by the various pixel data from storage block 30 and also receives the relevant motion information from an outer world. This allows to map all pixel data received on the various blocks of the next frame in the sequence. On the basis thereof, Assignment block 34 sets up a list of assignable candidates for the above two values to each block in the next frame. On the basis thereof, Reduce block 36 reduces the list of candidates for each block on the basis of a previously specified optimization criterium. Next, Select unit 38 from the available candidates for each block selects or mixes only a single one. This allows Display unit 40 to display each block with a single depth-associated value. Furthermore, this allows Storage block 30 to store for each pixel a single possible value for the depth association and its associated uncertainty interval.

Claims: CLAIMS: 1. A method for assigning block-wise depth-associated values for representing a static scene, to each of a sequence of frames that are each divided into a respective pattern of blocks, said method comprising the steps of: • for a first sub-sequence of frames (t, t+1) assigning to each block thereof a single depth-associated value and an estimation value for an associated depth uncertainty interval, and for each particular block uniformly assigning these two values to all pixels therein; • for a next-following frame (t+2), taking into account the information between the last frame of said sequence (t+1) and the next following frame, by interpreting each pixel in a non-last frame of said sequence (t) as a single point and an associated depth uncertainty interval in 3-D space, and on the basis of an estimated camera motion, transforming the various single points into a coordinate system associated to the final frame (t+1) of said sequence; • partitioning the final image of said sequence (t+1) into blocks, and allowing any such block to contain pixels that have originated from more than one block that is associated to said non-last frame (t), whilst then allowing to associate more than one depth value to any such block; • repeating the above block-based depth estimation procedure for subsequent frames t+2, t+3, etcetera in the same manner, whilst allowing a plurality of candidate depth- associated values per block, and in particular, allowing a particular earlier block to contribute more than one depth-associated value to the plurality of candidate depth associated values of a later block; • after completing a block-based matching between frames t+1 and t+2 with the actual plurality of candidates, using the set of depth-associated value estimates for the blocks in frame t+1, and updating the values for the pixels in still later blocks. 2. A method as claimed in Claim 1, wherein said depth-associated value is an expected reciprocal depth v subject to an estimated uncertainty interval d. 3. A method as claimed in Claim 2, wherein said uncertainty interval is a standard deviation in a Gaussian probability distribution. 4. A method as claimed in Claim 1, whilst under certain circumstances reducing an actual number of depth-associated values in a plurality of candidate values to a lower number: 5. A method as claimed in Claim 4, wherein said reducing is effected by joining candidate values that are closer to each other than a predetermined tolerance. 6. A method as claimed in Claim 3, wherein said reducing is effected by mixing two candidate depth associated values through forming a convex combination thereof. 7. A method as claimed in Claim 1, for use as a preprocessing step to a feature- based reconstruction procedure of images. 8. A method as claimed in Claim , for use as an interpolation method between frames. 9. An apparatus being arranged to implement a method as recited in Claim 1, for assigning block-wise depth-associated values for representing a static scene, to each of a sequence of frames that are each divided into a respective pattern of blocks, said apparatus comprising: • assign means for with respect to a first sub-sequence of frames (t, t+1) assigning to each block thereof a single depth-associated value and an estimation value for an associated depth uncertainty interval, and for each particular block uniformly assigning these two values to all pixels therein; • accounting means for with respect to a next-following frame (t+2), taking into account the information between the last frame of said sequence (t+1) and the next following frame, by interpreting each pixel in a non-last frame of said sequence (t) as a single point and an associated depth uncertainty interval in 3-D space, and on the basis of an estimated camera motion, transforming the various single points into a coordinate system associated to the final frame (t+1) of said sequence; • partitioning means for partitioning the final image of said sequence (t+1) into blocks, and allowing any such block to contain pixels that have originated from more than one block that is associated to said non-last frame (t), whilst then allowing to associate more than one depth value to any such block; • repeat means for repeating the above block-based depth estimation procedure for subsequent frames t+2, t+3, etcetera in the same manner, whilst allowing a plurality of candidate depth-associated values per block, and in particular, allowing a particular earlier block to contribute more than one depth-associated value to the plurality of candidate depth associated values of a later block; • update means for after completing a block-based matching between frames t+1 and t+2 with the actual plurality of candidates, using the set of depth-associated value estimates for the blocks in frame t+1, and updating the values for the pixels in still later blocks. 10. An apparatus as claimed in Claim 1, and comprising storage means (30)for storing depth-associated values pertaining to each block of a particular frame, motion implementing means (32) fed by said storage means (30) for under control of externally generated motion information (EN2) mapping all pixel data and appertaining depth value data on the various blocks of the next frame in the sequence, assignment means (34) fed by said motion implementing means (32) for setting up a list of assignable candidates for the depth- associated values and their respective uncertainty intervals to each block in the next frame, reduce means (36) fed by said assignment means for reducing the list of candidates values for each block on the basis of a previously specified optimization criterium, select means (38) fed by said reduce means (36) for from the available candidates values for each block selecting or mixing only a single one, for subsequent presentation to display means (40) and/or said storage means (30) for repeating said reconstructing for a next-following frame.

Inventors: VAN OVERVELD CORNELIS W A M, VAN OVERVELD, CORNELIS, W., A., M.
Applicants: KONINKL PHILIPS ELECTRONICS NV, KONINKLIJKE PHILIPS ELECTRONICS N.V.
Application year: 2000
Main classification: G06T
IPC codes: G06T7, H04N7