Patent ID: 12231608

DETAILED DESCRIPTION OF THE INVENTION

FIG.1ashows an embodiment for an apparatus10for reconstructing a multi-view signal12coded into a multi-view data stream14. The apparatus10comprises an input16for the multi-view data stream14, and two outputs18aand18bfor a reference view signal20and a dependent view signal22, respectively. Further, apparatus10comprises a reference-view reconstructor24connected between input16and output18a, and a dependent-view reconstructor26connected between input16and output18b. Optionally, a depth map estimator28of apparatus10may be connected between reference-view reconstructor24and dependent-view reconstructor26with a significance and the way of connection being set out in more detail below.

The apparatus or decoder10ofFIG.1is configured to reconstruct the multi-view signal12from the multi-view data stream14. In doing so, the apparatus may obey a predetermined coding/decoding order according to which the reference signal20is processed prior to dependent view22.

FIG.1billustrates possible components within multi-view signal12. As illustrated inFIG.1b, the multi-view signal12may not only represent a spatial sampling of one common scene from different view directions or view points associated with respective views20and22, but also a temporal sampling of this scene as it is illustrated inFIG.1bexemplarily by showing three consecutive time instants T−1, T and T+1 along a time axis30. For each time instant, each view20and22comprises a picture32t1and32t2, respectively, with a time instant to which these pictures belong, being indicated inFIG.1bin parenthesis. Each picture32t1,2represents a respective texture map, i.e. a spatial sampling of the color, luminance, intensity or brightness in this scene as seen from the respective view points, respectively.

It is noted thatFIG.1bshows both views20and22as having their pictures32ti,2temporally aligned. However, this is not necessarily the case. The pictures32t1and32t2may be interlaced, or the time resolution between view20and view22may even differ.

The decoder10is configured to process, or reconstruct, the multi-view signal12sequentially in time. To be more precise, decoder10is configured to reconstruct the pictures32t1,2of the views20and22of the multi-view signal12of a certain time instant, such as T−1, prior to continuing with processing the views20and22regarding the subsequent time instant T, i.e. prior to reconstructing pictures32t1,2(T). In this regard, it is noted that the temporal coding order among the time instants of the multi-view signal12may be equal to the presentation time order of the pictures32t1,2or may differ therefrom.

As also shown inFIG.1b, the multi-view signal12may additionally not only represent a spatial or spatio-temporal sampling of the texture of the common scene from the different view directions or view points associated with the respective views20and22, but also a spatial or spatio-temporal sampling of this scene with regard to depth. For example, as illustrated inFIG.1b, each picture32t1,2may comprise, or may have associated therewith, a respective depth map34dt1and34dt2. As known in the art, the additional information sent by these depth maps34dt1,2may be used by an intermediate-view synthesizer downstream to output18aand18b, respectively, in order to synthesize intermediate views between views20and22from the pictures32t1,2. Regarding the afore-mentioned coding order, the decoder10may be configured to use a decoding order according to which the texture of the reference view is reconstructed prior to the associated depth, with merely then stepping forward to reconstruct the texture, followed by its depth map, of the dependent view of the same time instant, wherein after the same information is reconstructed for the next time instant and so forth.

It is noted thatFIG.1assumes that each picture32t1,2comprises a respective depth map34dt1,2, but this does not need to be the case. The temporal resolution of the depth map34dt1,2may differ from the associated sequence of pictures32t1,2. Moreover, even the time resolution between views20and22may differ. Naturally, the same applies to the spatial resolution of the texture and depth maps. Texture map resolution and depth resolution may differ from each other in time and/or spatial dimensions.

Finally with regard toFIG.1b, it is noted that the term “picture” is used herein in two meanings, namely first, to denote the conglomeration of all spatially sampled information of a respective view for a certain time instant, such as texture and depth map together, and second, merely the texture at the current time instant. The context will reveal which meaning is respectively meant.

Thus, back toFIG.1a, the mode of operation of decoder10shall be explained. As already noted above, decoder10is configured to process the multi-view signal12, i.e. to reconstruct same, sequentially in time. To be more precise, decoder10may be configured to reconstruct the pictures32t1,2and the respective depth maps34dt1,2of a certain time instant, such as T−1, prior to continuing with processing the pictures and depth maps of another time instant T. In particular, the reference-view reconstructor24is configured to reconstruct the reference view20, while dependent-view reconstructor26is configured to reconstruct the dependent view22. Reference-view reconstructor24reconstructs the reference view20from a reference view portion36of the multi-view data stream14, while dependent-view reconstructor26reconstructs the dependent view22based on a dependent view portion38of the multi-view data stream14. In fact, reference-view reconstructor24and dependent-view reconstructor26may be configured to operate in a similar manner. For example, reference reconstructor24and dependent-view reconstructor26both use predictive coding in order to reconstruct the respective signal20and22, respectively. The dependent-view reconstructor is configured to reconstruct the dependent view22of the multi-view signal12using block-based predictive coding, and the reference-view reconstructor may likewise use block-based predictive coding in order to reconstruct the reference view20of the multi-view signal12. Both may, for example, be configured as a hybrid video decoder, respectively. The reference-view reconstructor24reconstructs, for example, the picture or texture map32t1of a current time instant T by assigning a respective one of available block coding modes to the blocks40into which this picture is subdivided. The subdivision of the picture32t1into blocks may be predefined by default, or may be signaled within the multi-view data stream14. The subdivision may subdivide picture32t1in a regular manner into blocks of the same size or blocks of different size. For example, the sub-division may first sub-divide the picture regularly into rows and columns of tree-root blocks which, in turn, are sub-divided using multi-tree subdivisioning with the information concerning the latter being, for example, signaled for each tree-root block in the data stream14. In other words, a multi-tree subdivisioning may be possible so that the block size of the blocks40may be locally adapted to the picture content.

The coding modes available may comprise one or more intra prediction modes according to which reference-view reconstructor24fills the respective block40by prediction from already reconstructed samples of already reconstructed blocks preceding the current block in a decoding order defined among the blocks of picture32t1, and/or one or more inter prediction block coding modes according to which reference-view reconstructor24reconstructs the respective block by motion compensated prediction using motion data such as motion vectors, reference picture indices and the like. For example, for illustration purposes two blocks are exemplarily shown to be reconstructed by inter prediction. The motion data42for these inter-predicted blocks may comprise motion vectors used by reference-view reconstructor24to copy respective portions of a reconstructed version of a reference picture32t1indexed by a reference index also comprised by the motion data42. The motion data42is comprised by the reference view portion36of multi-view data stream14.

Reference-view reconstructor24may, however, also be responsible for reconstructing the depth map34d1, if present, of the current picture32t1of the current time instance T of the reference view20from a reference view depth map portion of portion36. As illustrated inFIG.1a, reference-view reconstructor24may also use a block-wise processing in order to reconstruct the depth map34d1. The subdivision of depth map34d1may be spatially adopted from the subdivision of the corresponding picture32t1, or subdivision information may be contained in multi-view data stream14with regard to depth map34d1. The coding modes which reference-view reconstructor24supports for the blocks44of depth map34d1may or may not comprise the coding modes available for blocks40of picture32t1. Additionally, however, other coding modes may be available as well. For example, some blocks of depth map34d1may be predicted from already reconstructed portions40of picture32t1of the same time instant. It should be noted that even the coding modes assigned to blocks44may be adopted by reference-view reconstructor24from the coding modes of co-located blocks40of picture32t1. Otherwise, the coding modes for blocks44and40are conveyed within the multi-view data stream14. For illustration purposes, two blocks of blocks44of depth map34d1are exemplarily shown to be inter predicted using motion data46, such as by copying from corresponding portions—i.e. portions pointed to by a motion vector of the motion data from a position co-located to the respective block—of a referenced, already reconstructed depth map34d1indexed, for example, by a reference index which may also be comprised by the motion data46optionally.

Similarly to the coding modes, motion data46may be adopted for motion data42, or at least predicted therefrom. Together, motion data42and motion data46represent motion data48of the current picture or time instant T of view20.

The dependent-view reconstructor26may operate quite the same as reference-view reconstructor24with dependent-view reconstructor26, however, being configured to reconstruct the dependent view22from the dependent view portion38. Accordingly, in reconstructing a current picture32t2of current time instant T, dependent-view reconstructor26may also use a block-wise processing using a subdivision into blocks50which may be fixed or signaled within multi-view data stream14. Alternatively, depth map based inter-view prediction of the subdivision into blocks50as outlined in more detail below may be used by dependent-view reconstructor26so as to derive the subdivision into blocks50for view22from the subdivision into blocks40and/or blocks44of view20. Dependent-view reconstructor26may also use such a subdivisioning into blocks52for depth map34d2of view22. As far as the coding modes are concerned, dependent-view reconstructor26may support coding modes as they have been described with respect to the reference-view reconstructor24. Accordingly, illustratively, two blocks50and two blocks52are exemplarily shown to be subject to inter prediction using motion data54and56, respectively, so as to be appropriately copied from respective portions of a reconstructed version of previously reconstructed pictures32t2and the respective depth map34d2, respectively. Together, this motion data54and56represents the motion data for the current picture or current time instance of view22. In addition to these coding modes, however, dependent-view reconstructor26has the ability to support one or more inter-view prediction modes for using disparity-compensated prediction in order to copy respective blocks from portions of view20of the same time instant, which are spatially displaced from a co-located position by an amount and direction defined by associated disparity data. InFIG.1, one disparity predicted block in picture32t2and the depth map34d2are exemplarily shown along with the corresponding disparity data60and62, respectively. Disparity data may, for example, comprise a disparity vector or at least a disparity component along the view offset direction between views20and22, and optionally a view index indicating the reference view from which the respective block of the dependent view22depends, which index may be favorable in case of the coexistence of more than two views as exemplarily shown inFIG.1. Together, disparity data60and62form the disparity data for the current picture or current time instance T of view22.

That is, reference-view reconstructor24and dependent-view reconstructor26may operate in a manner so as to reduce the redundancies between a picture and its depth map, along the time axis30and in inter-view direction, between views20and22, as far as possible. This is also true, for example, for the prediction of the side information such as the motion data and disparity data as well as the coding modes and the subdivision information mentioned above. All of this information shows redundancies among each other in time direction, between the views and between a picture and its depth map.

Until now, the description ofFIG.1afocused on a possible base implementation for decoder10and reference-view reconstructor24and dependent-view reconstructor26, respectively. However, the base implementation may also be different. The above description is merely to be regarded as one possible implementation, but other coding concepts underlying reference-view reconstructor24and other predictive block-based coding concepts for dependent-view reconstructor26would also be feasible.

Proceeding with a most-interesting feature of the decoder ofFIG.1a, due to the predictive coding nature of the operation of reference-view reconstructor24, reference-view reconstructor24internally reconstructs a current picture of reference view20of the multi-view signal12via a prediction signal predicted from previously reconstructed portions of the multi-view signal12, such as a previously reconstructed portion of the reference view20, and a residual signal which reference-view reconstructor24derives from portion36of multi-view data stream14. In particular, owing to the predictive coding nature, reference-view reconstructor24refines the prediction signal with the residual signal such as by adding, although other possibilities would also exist. This residual signal internally obtained within reference-view reconstructor24may serve as a reference residual signal63for dependent-view reconstructor26. Accordingly,FIG.1ashows reference-view reconstructor24as being connected to a reference input of dependent-view reconstructor26. The dependent-view reconstructor26is configured to, in reconstructing the current picture of the dependent view22, predict a residual signal for block-based predictively coding of the current picture of the dependent view22from this reference residual signal63using block-granular disparity-compensated prediction. That is, in case of the dependent-view reconstructor26being implemented as a hybrid video decoder as outlined above, dependent-view reconstructor26may support a coding option for blocks50, or50and52, according to which the residual signal for refining the prediction signal as obtained via the aforementioned block coding modes, such as intra, inter and/or inter-view block coding mode, is refined by a residual signal which, in turn, is predicted from the reference residual signal63by use of block-individually defined disparity displacement such a displacement vectors. That is, dependent-view reconstructor26may predict the residual signal for a block50or52of the dependent view22via copying a respective portion of the reference residual signal63of the current picture of the reference view20, displaced from a location corresponding to this block of the current picture of the dependent view22according to the disparity displacement such as the afore-mentioned disparity vector. There are different possibilities for as to how the disparity displacement for the respective block may be obtained, with these possibilities outlined in more detail below. In any case, due to the prediction of the residual signal for the current picture of the dependent view22, dependent-view reconstructor26may not even have to refine the predicted residual signal of the current picture of the dependent view based on an second order prediction signal explicitly signaled within the multi-view data stream14and portion38thereof, respectively.

Thus, in the following description, different possibilities are described for as to how to obtain the disparity displacement for the currently reconstructed block of the current picture of the dependent view22. A further discussion will reveal different possibilities for realizing the block-based disparity-compensated prediction using the derived disparity displacement. However, before turning to the discussion of these possible implementation details, the above description of a possible base implementation is finalized with regard to further inter-view redundancy removal features which could be implemented in the decoder ofFIG.10in case of an implementation thereof in the form of an hybrid video decoder—inter-layer redundancy removal features also using the block-based disparity compensation using the derived disparity displacement for the blocks.

For example, in order to more efficiently exploit the redundancy between views20and22, the dependent-view reconstructor26may be configured to—with the preliminary assumption that the dependent-view reconstructor26has the block-individual disparity displacement at hand—predict the motion data54and/or56of inter-predicted blocks of the current picture (including its depth map34d2, if present) of the dependent view22based on the disparity displacement for these inter-predicted blocks within view22. Then, this predicted motion data is used in motion-compensated predicting these blocks. For example, the dependent-view reconstructor24may be configured to, in predicting the motion data54and/or56for a current block50/52, use the respective disparity displacement for the current block50/52to locate one or more corresponding positions at the current time instant of the reference view20and use the motion data for the one or more blocks of the reference view20at the located positions to serve as a predictor for the motion data54/56of the current block of the current picture of the dependent view22. Naturally, refinement data in order to signal a prediction residual for the motion data54/56may be comprised by the dependent view portion38, i.e. motion residual data, and used by dependent-view reconstructor26to refine the motion data54/56. Even the combination is possible: blocks50/52of the inter block coding mode, the motion data of which has been predicted from motion data42/46of blocks40/44located at positions determined by the respective disparity displacement for the blocks50/52, may be refined using a residual signal which, in turn, is likewise predicted by copying a respective disparity-displaced portion from the reference residual signal.

The following description is structured as follows. First, an embodiment for an encoder fitting to the decoder ofFIG.1ais described with respect toFIG.2. Then, a more detailed implementation possibility for the decoder and encoder ofFIGS.1aand2is described with respect toFIGS.3and4. Thereinafter, different possibilities for obtaining the disparity displacement for the individual blocks of the current picture of the dependent view are discussed. Later, specific implementation details concerning the disparity-compensated residual prediction are discussed. Then, specific possibilities for the mode of operation for the disparity estimator28are discussed in accordance with two specific ways of providing the disparity displacement data. Finally, further possible advantageous implementations details are described.

FIG.2shows an apparatus for encoding the multi-view signal12into the multi-view data stream14and comprises, to this end, a reference view encoder80and a dependent-view encoder82. Optionally, encoder90ofFIG.2may comprise a disparity estimator84connected between reference-view encoder80and dependent-view encoder82. Analogously to decoder10ofFIG.1a, reference view encoder80is configured to predictively encode the reference view20of the multi-view signal12while dependent-view encoder82is configured to encode the dependent view22of the multi-view signal12using block-based predictive coding. The dependent view encoder82is configured to, in encoding the current picture of the dependent view22, predict a residual signal of the current picture of the dependent view22from a reference residual signal63of the current picture of the reference view20using block-granular disparity-compensated prediction. As is known in the art, encoder90ofFIG.2, although acting very similar to the decoder ofFIG.1aas far as the finally chosen and finally coded prediction parameters and residual signals signaled within the multi-view data stream14are concerned, differs from the decoder in that the encoder may select the prediction parameters such as coding modes, the prediction parameters associated with the chosen coding modes, the accuracy of coding the residual signal and so forth, out of a magnitude of different possibilities and combinations by optimizing some cost function depending on, for example, compression rate and/or compression distortion.

As mentioned above,FIG.3shows a possible implementation of the decoder ofFIG.1ain more detail. According toFIG.3, the reference-view reconstructor24and the dependent-view reconstructor26are structured very similarly, so that the following description ofFIG.3starts with a description of the internal structure of reference-view reconstructor24and then proceeds with the description of dependent-view reconstructor26.

The reference-view reconstructor24comprises an input300for receiving the reference-view portion36, and an output302for outputting reference signal20. A further output304is provided for outputting the reference residual signal63. Internally, reference-view reconstructor24comprises a data stream extractor306, a main predictor308, a residual predictor310, and an inverse transformer312. The data stream extractor306is configured to extract prediction parameters314and residual data316from the reference view portion36. The extraction may be based on lossless entropy decoding such as VLC or arithmetic decoding. A prediction parameter output at which the prediction parameters314are output, is connected to a parameter input of main predictor308and, optionally, with a parameter input of residual predictor310. A residual output of data stream extractor306at which the residual data316is output, is connected to an input of inverse transformer312. Further, reference-view reconstructor24comprises a first combiner—here exemplarily depicted and embodied as an adder—318having a first input connected to an output of main predictor308, and an output connected to the input of main predictor308. Likewise, a second combiner—here exemplarily also embodied and depicted as an adder—320, is provided, which has its first input connected to an output of residual predictor310. An output of combiner320is connected to the second input of combiner318, and the second input of combiner320, in turn, is connected to an output of inverse transformer312so that, altogether, data stream extractor306, inverse transformer312and combiners320and318are serially connected—in the order of their mentioning—between input300and output302. As shown inFIG.3, the input of the residual predictor310may be connected to its output or to the output of adder320. Different options also exist for the connection to reference residual signal output304. The output of adder320, or the output of residual predictor310may be connected to output304.

The functionality of the reference-view reconstructor24ofFIG.3is as follows. The data stream extractor306extracts residual data316and prediction parameters314from the reference-view portion36of multi-view data stream14. As already outlined with respect toFIG.1a, the reference-view reconstructor24operates on a block basis so that the prediction parameters314, for example, assign a respective block coding mode to each block of the reference view20. For texture blocks40(seeFIG.1b) an intra prediction mode and an inter prediction mode may be available, with this possibility illustrated inFIG.3by showing the main predictor308as comprising an intra predictor308aand an inter predictor308b. The prediction parameters314may convey prediction parameters specific for the respective block coding mode for the respective blocks40. For example, the intra prediction mode, which intra predictor308ais responsible for, may be controlled via an intra prediction direction along which the current block is filled based on previously reconstructed samples of neighboring, already reconstructed blocks of the current picture. The inter predictor308bis responsible for the inter prediction block coding mode and controlled via motion data conveyed by the prediction parameters314so as to copy respective portions of previously reconstructed pictures of the reference view signal20with the location of the portions relative to the location of the inter-predicted block, and the reference picture being indicated by the motion data. Generally, intra predictor308aand inter predictor308bmay also be available, and may function similarly, with respect to depth blocks44if depth maps are transmitted within the reference view20. However, additional predictors not shown inFIG.3may be available or predicting the content of the depth map within the current block44using already reconstructed portions of the current texture map of view20. Altogether, the main predictor308provides at its output a prediction signal322, some blocks of which have been obtained by intra prediction while others have been obtained by inter prediction. Combining this prediction signal322at combiner318with a residual signal324reveals the reconstructed signal, namely the reconstructed reference view20.

Instead of explicitly transmitting this residual signal324in full by way of the residual data316for all blocks40/44, the reference-view reconstructor24may, optionally, have additionally the residual predictor310. It should be immediately noted that the residual predictor310forms merely an optional feature ofFIG.3and may, alternatively, be left away, with the output of inverse transformer312being in that case directly connected to the second input of combiner318so as to provide the residual signal324directly. However, if the residual predictor310is present, same may be configured to predict the residual signal324temporally by motion compensated prediction so as to obtain a residual prediction signal326which, when combined with the second order residual signal as output by inverse transformer312, namely328, then forms the residual signal324.

The intra predictor310aof residual predictor310may be configured to use, as a reference, either the reconstructed residual signal324of previous (previously reconstructed) pictures, or the residual prediction signal326of such previous pictures. That is, intra predictor310amay copy a portion out of this reference, located according to respective residual prediction motion data signaled within the prediction parameters314for those blocks for which the residual prediction that residual predictor310is responsible for, is activated.

The inverse transformer312uses the residual data316to form the second order residual signal328which is, for blocks for which the residual prediction is turned off, equal to the residual signal324. The inverse transformer312may perform a block-wise transform such as IDCT or the like, in order to obtain signal328from respective transform coefficient levels within residual data316. The transform blocks within which inverse transformer312performs the individual transformations may coincide with the blocks for which the block coding modes are signaled within parameters314or may form a subdivision thereof, with a subdivision possibly signaled within residual data316. Alternatively, transform blocks may cross block boundaries of blocks40/44.

Beyond the just provided description ofFIG.3, the other specific details mentioned above with respect toFIG.1aare also valid forFIG.3. For example, prediction parameters for a current block may be predicted from prediction parameters of previously reconstructed blocks of the same picture or a previous picture of the reference view22. This also applies to the coding mode of the current block. Further, all of the above mentioned possibilities with regard to the subdivision of the current picture into blocks40/44are also valid forFIG.3.

As already mentioned at the beginning of the description ofFIG.3, the internal structure of the dependent-view reconstructor26largely coincides with that of reference-view reconstructor24, and accordingly the same reference signs are used for the internal elements of dependent-view reconstructor26as far as these elements likewise occurring within the reference-view reconstructor24, are concerned. The only difference is an apostrophe used for elements within the dependent-view reconstructor26. Focusing on the difference in the internal structure of reconstructors24and26, residual predictor310′ is no longer an optional feature. Rather, the residual predictor310′ renders, at least for some blocks50/52of the dependent view22, an inter-view residual prediction mode available for which an inter-view predictor330of residual predictor310′ assumes responsibility. An inter residual predictor310b′ may optionally also by present so as to perform a alternative way of predicting the residual, namely vie respective residual prediction motion data as described above with respect to module310b. The inter-view residual predictor330is connected between a reference residual signal input332of dependent-view reconstructor26which, in turn, is connected to the reference residual signal output304of reference-view reconstructor24so as to receive the reference residual signal63, and, via the output of residual predictor310′, to the first input of combiner320′. A further difference to reference-view reconstructor24, is related to the additional presence of an interview prediction mode within main predictor308′, which an inter-view predictor308c′ is responsible for. The inter-view predictor308c′ copies respective portions of the reconstructed current picture of the reference signal20of the same time instant, located at a position determined by disparity data associated with the respective disparity-compensated block50/52. A respective connection connecting inter-view predictor308c′ with the output of combiner318of reference-view reconstructor24is not shown inFIG.3for the sake of focusing the present description to the residual prediction performed by inter-view predictor330.

In any case, the prediction signal322′ of the dependent-view reconstructor26is, in accordance with the specific example ofFIG.3, composed of blocks being obtained by intra prediction, blocks obtained by inter prediction and blocks obtained by inter-view prediction. For some of these blocks for which the residual prediction option is turned on by prediction parameters314′, inter-view predictor330predicts the residual signal324′ by copying a respective portion out of the residual signal324for the current picture of the reference view, or the residual prediction signal326for the current picture of the reference view20, i.e. the reference residual signal63. The position of the respective portion is determined by inter-view predictor330by use of a disparity displacement specific to this block. Different possibilities exist as to how inter-view predictor330determines the disparity displacement for the current block, with these different possibilities outlined in more detail below.

Before beginning to describe the different possibilities for obtaining the disparity displacement underlying the block-based disparity compensated residual prediction for the embodiment ofFIG.1aand the implementation ofFIG.3, it shall be noted that not only the residual predictor310of the reference-view reconstructor24is optional, but also the inverse transformers312and312′, which may be left away with the second order residual signal328being then signaled in the spatial domain within residual data316rather than in the transform domain as it was described above. Moreover, the inter predictor310b′ is optional as already noted above.

Further, for the sake of completeness,FIG.4shows an embodiment for an encoder fitting to the decoder ofFIG.3. Many of the elements within the encoder ofFIG.4are equal to those, or correspond to those, indicated inFIG.3, and accordingly, all of these elements are not described again. Rather, the elements differing from the decoder structure ofFIG.3are described, wherein it is additionally pointed out that, naturally, the encoder ofFIG.4first selects all the prediction parameters and the residual data finally transmitted within the data stream in accordance with some optimization routine as already outlined above.

In particular, the reference-view encoder80comprises an input400at which reference view signal20enters, and an output402at which the reference view portion36of multi-view data stream14is output. Internally, reference-view encoder80comprises two subtracters404and406, a transformer408and a data stream inserter410connected in series to each other between input400and output402. The inverting input of subtracter404is connected to the output of main predictor308so as to receive the prediction signal322. Subtracter406is arranged downstream to subtracter404so as to have its non-inverting input connected to the output of subtracter404. Further, the inverting input of subtracter406is connected to the output of residual predictor310. As a further difference to the structure shown inFIG.3, the main predictor308and the residual predictor310have a prediction parameter output rather than a prediction parameter input as their functionality also encompasses the finding of the optimal set of prediction parameters and sending these prediction parameters314finally selected to a respective prediction parameter input of data stream inserter410. Likewise, the transformer408outputs the residual data316to a residual input of data stream inserter410. The remaining elements of reference-view encoder80correspond to those indicated above with respect toFIG.3. Likewise, the dependent-view encoder82differs from the construction of the dependent-view reconstructor26in the way indicated inFIG.4. That is, the differences correspond to those already described with respect to the reference-view encoder80compared to the reference-view reconstructor24. For the sake of completeness, it is noted that the transformers408and408′, which perform a spectral decomposition such as an DCT, may be left away in case of the inverse transformers312and312′ being left away as well.

Thus, the functionality of the encoders80and82largely coincides with the functionality described above with respect to the decoder ofFIG.3. The subtracters404and406operate on the original versions of reference and dependent view20and22so as to obtain the actual first order residual signal412and the actual second order residual signal414(which may represent the first order residual signal in case of the residual prediction mode being switched off), wherein the transformer408lossy encodes the actual second order signal414so as to derive the residual data316, i.e. the residual data in the transform domain. The data stream inserter410may, corresponding to the data stream extractor306, perform a lossless entropy coding such as VLC or arithmetic coding so as to insert the residual data316and the prediction parameters314into the reference-view portion36and dependent-view portion38, respectively.

Now, after having described the embodiments ofFIGS.1aand2, and the corresponding possible detailed implementations with respect toFIGS.3and4, in the following, possible variants in order to obtain the disparity displacement underlying the block-based disparity compensated residual prediction are described with respect to these figures.

For example, in accordance with a first variant, the disparity displacement underlying the block-based disparity compensated residual prediction is obtained from an explicitly signaled depth-map34d1(T) of the current picture32t1of the reference view20. In particular, in accordance with this variant, the reference-view reconstructor24is configured to predictively reconstruct the current picture32t1of the reference view20of the multi-view signal12, including the depth map34d1of the current picture of the reference view20, so as to obtain a reconstructed version of the depth map34d1(T). Looking atFIG.3, this depth map34d1(T) may be coupled to disparity estimator28which, in accordance with this variant, has, to this end, a reference depth map input connected to the output302of the reference-view reconstructor24, while an output thereof is connected to a depth map estimate input of inter-view predictor330. In accordance with this variant, the disparity estimator28is configured to estimate the disparity displacement for a current block50/52of the current time instant of the dependent view22from the reconstructed version of the depth map34d1(T) of the current time instant T of the reference view20, and the dependent-view reconstructor26is configured to predict the residual signal324of the current block50/52of the current time instant of the dependent view22by copying a portion of the reference residual signal63, displaced from a location of the current block50/52of the current time instant of the dependent view22according to the disparity displacement thus estimated.

For example, and as illustrated inFIG.1c, the disparity estimator may warp66a reconstructed version of the depth/disparity map34d1(T) of the current time instant T of the reference view into the dependent22to obtain a depth/disparity map estimate64for the current time instant T of the dependent view22and obtain the disparity displacement for the current block50/52from the estimated depth/disparity map64. This means the following. Although the description so far suggested that the depth maps indicate the depth of the scene in terms of depth values, a disparity map may be used as well since depth and disparity are related to each other in a known manner. In case of the depth/disparity maps34d1,2actually being depth maps, the depth estimator28may be configured to, in warping66the depth map34d1of the reference view20into the dependent view22, derive disparity vectors of the current time instant T of the reference view20from the depth map34d1of the current time instant T of the reference view20and apply the derived disparity vectors onto the depth map34d1of the current time instant of the reference view20itself so as to obtain the depth map estimate64. The disparity estimator28may then obtain the disparity displacement such as the disparity vector, for the current block50/52subject to inter-view residual prediction, merely by converting the depth value of this estimated depth map64at the location of the current block50/52into a disparity vector, using this disparity vector in turn in order to copy the thus determined portion from the reference residual signal, as will be outlined in more detail below with respect toFIG.5.

In accordance with the just outlined variant, in order to derive the disparity displacement underlying the disparity compensated residual prediction, the disparity estimator28was present in order to provide the disparity displacement for the current block by depth-to-disparity conversion from a co-located portion of the depth map estimate64. Favorably, this disparity displacement is available prior to starting the reconstruction of the current time instant of the dependent view22in accordance with the coding order outlined above. However, there are also other variants which could be used in order to obtain this disparity displacement.

In accordance with a second variant, for example, the disparity estimator28is configured to estimate the disparity displacement of the current block50/52of the current time instant of the dependent view22—to be used for inter-view residual prediction—by spatial and/or temporal prediction from a disparity displacement associated with reference blocks within previously reconstructed portions of the dependent view, such as blocks50/52of the same time instant or blocks50/52of a previous time instant of the dependent view22. The disparity displacement associated with such reference blocks may stem from blocks50/52coded using the inter-view prediction mode using respective disparity data60and62, respectively. This disparity data60/62may serve as a basis for the estimation of the disparity displacement for the current block50/52of the current time instant of the dependent view22for which the inter-view residual prediction shall be performed. In accordance with this second variant, the disparity estimator28has a disparity data input connected to the prediction parameter output of data stream extractor306′ and a disparity displacement output connected to a disparity displacement input of inter-view predictor330. That is, in that case the, dependent-view reconstructor26or, to be more precise, the inter-view predictor330, may simply use the estimated disparity displacement for the current block50/52in order to locate an adequate portion of the reference residual signal63and use this portion for predicting the residual signal324′ within the current block50/52, namely by copying the located portion.

In accordance with an alternative variant, the disparity displacement for the current block subject to inter-view residual prediction is explicitly signaled within the multi-view data stream14and the dependent-view portion38, respectively. For example, the dependent-view reconstructor26may, in accordance with this variant, be configured to predict the residual signal324′ within the current block50/52by copying a portion of the reference residual signal63displaced from a location of the current block50/52according to this explicitly signaled disparity displacement. Imagine, for example, the current block50/52is, in accordance with the main prediction which the main predictor308′ is responsible for, associated with the inter-view block coding mode. In that case, this block50/52has, anyway, a disparity vector60,62associated therewith. This vector may be used as the just mentioned explicitly signaled disparity displacement, too.

In accordance with a further variant, the disparity estimator28is configured to estimate a disparity displacement of a current block50/52of the current time instant of the dependent view22by continuously updating a depth/disparity map of a scene of the multi-view signal12using disparity data60coded into the multi-view data stream12for previously reconstructed pictures32t2of the dependent view22and motion data42coded into the multi-view data stream14for previously reconstructed pictures as well as the current picture32t2of the reference view20and deriving the estimated disparity displacement of the current block50/52of the current time instant of the dependent view22from the continuously updated depth/disparity map. At random access points, the depth/disparity map estimate is initialized based on the disparity data of the picture32t2at the random access point. That is, in accordance with this variant, the multi-view signal12may not even comprise depth maps, neither for the reference view20nor for the dependent view22. Rather, in accordance with this variant, the disparity estimator28is configured to virtually provide such a depth/disparity map estimate by a continuous update using the information conveyed by the multi-view data stream in form of the motion data42and the disparity data60. In accordance with this variant, the disparity estimator28has a motion/disparity input connected to the prediction parameter outputs of data stream extractors306and306′, and a disparity displacement output connected to a disparity displacement input of inter-view predictor330.

With regard toFIG.1d, the just outlined variant is described in more detail. In particular, in accordance with this variant, the disparity estimator28ensures that each picture32t1,2has a depth/disparity map estimate64associated therewith, the estimate64being consecutively derived from each other in a chain of updates. That is, although the continuous update could pertain to a disparity map estimate as well, it is preliminarily assumed that the estimate concerns depth data. The disparity estimator28is configured to continuously update the depth map estimate64in a ping pong manner between views20and22primarily with the aim to provide each picture32t2of the dependent view22with such a depth map estimate64in order serve as a basis for the above outlined improved inter-view redundancy reduction including the inter-view disparity compensated block-based residual prediction.

Primarily, it is assumed that the disparity estimator28already has access to such a depth estimate for one or more previous pictures32t1of the reference view20such as time instance T−1. A way how disparity estimator28could have gained access to this depth map estimate74for the previously decoded picture32t1of the reference view20is described further below. It should be noted, however, that such depth map data could be intermittently signaled explicitly within the multi-view data stream14for first pictures32t1of the reference view20within so called random access units, i.e. groups of pictures32t1which are decodable without reference to any previous portions of signal12. In order to illustrate this possibility, a dashed line connects disparity estimator28with input16. In the following description, a possibility is presented where the extra transmission of such starting depth map is not necessitated. Rather, the disparity data within the data stream portion38for the first picture32t2of the dependent view22in coding order within the random access unit is exploited to construct the starting depth map of the first picture32t1of the reference view20in coding order within the random access unit.

In particular, the disparity estimator28is configured to generate the depth map64of the current picture32t2of the dependent view22by applying the motion data42for the current picture32t1of the reference view20at the current time instance T onto the depth map estimate74of any previous picture32t1of the reference view20at the time instant T−1, for example. As already noted above, the reference-view reconstructor24reconstructs the current picture32t1of the reference view20using motion compensated prediction based on the motion data42, which is signaled within the multi-view data stream14for the reference view20. The disparity estimator28has access to this motion data42and uses this motion data42for one of the mentioned updates of the chain of updates, namely the transition71from the depth map estimate74of the reference picture32t1at the previous time instant T−1 to the depth map estimate64of the current picture32t1at the current time instant T. A way how this may be performed will be outlined in more detail below. Preliminarily, it shall be sufficient to note that applying71the motion data42onto the depth map74for the previous time instance T−1 could mean that co-located blocks72, i.e. portions within depth map estimate64of the current picture32t1which are co-located to blocks40for which this motion data42has been signaled in the stream portion36, are updated with, i.e. copied from, content of the referenced depth map estimate, i.e. the depth map estimate74for the picture32t1of the previous time instance T−1 at portions within the referenced depth map estimate74pointed to by the motion data42′ equal to motion data42. Remaining holes may be filled by interpolation and/or extrapolation exploiting additional information offered by the intra-coded blocks among block40of the current picture32t1. As a result, the depth map estimate64has been updated (or generated by transitioning from T−1 to T).

Again, disparity estimator28performs this update/transition71merely in order to prosecute further the chain of updates described further below so as to serve as a basis for deriving the depth map estimate64of the current picture32t2of the dependent view22of the same time instants T. To finalize the derivation, disparity estimator28warps the updated depth map estimate64of the current picture32t1of the reference view20into the dependent view22so as to obtain the depth map estimate64of the current picture32t2of the dependent view22. That is, as the motion data42is defined merely at a block granularity, the update/transition71and the resulting depth map estimate64of view22as resulting from the warping78represent a quite coarse estimation of the depth, but as will be shown below such a coarse estimate is sufficient in order to significantly increase the efficiency in performing the inter-view redundancy reduction.

Although possible details regarding the warping76are also described further below, briefly spoken, the dependent-view reconstructor26may be configured to perform the warping78by deriving disparity vectors from the depth map estimate64of current picture32t1and applying the derived disparity vectors onto the depth map estimate64itself, so as to obtain the warped depth map estimate64of the current picture32t2of the dependent view22.

Thus, as soon as disparity estimator28has provided dependent-view reconstructor26with the result of the warping76, namely the depth map estimate64of the current time instant T for view22, dependent-view reconstructor26is able to use this depth map estimate64for performing the above-outlined inter-view redundancy reduction for which possible implementations are set out in more detail below.

However, disparity estimator28continues to update77this depth map estimate64so as to obtain an updated depth map estimate74for the current picture32t2of the reference view22and thereby maintaining the chain of updates leading to the estimate for the next time instance T+1. Accordingly, the dependent-view reconstructor26is configured to update77the depth map estimate64of the current picture32t2of the dependent view22of the current time instance T using the disparity and/or motion data54and60for the dependent view22in a manner similar, at least for the motion data54, as described above with respect to the update step71. That is, the dependent-view reconstructor26uses the disparity/motion data for the picture32t2for time instance T within stream portion38for reconstructing this picture32t2. As far as the disparity data60is concerned, disparity estimator28may easily convert the disparity vectors contained within the disparity data54into depth values and assign, based on these depth values, updated depth values to samples of the updated depth map estimate79bof the current picture32t2of the dependent view22which are co-located to the respective disparity-predicted block50in picture32t2. The motion data54could be used so as to copy content of the depth map estimate74of the picture32t2of a referenced previous time instance T−1 of the dependent view22, at portions thereof pointed to by motion data equal to motion data54, into portions within the updated depth map estimate74of the current picture32t2which are co-located to blocks50for which this motion data42has been signaled in the stream portion36. Remaining holes may be filled by interpolation and/or extrapolation exploiting additional information offered by the intra-coded blocks among block40of the current picture32t1. As a result, the updated depth map estimate74of the current picture32t2has been updated (or generated by transitioning from T−1 to T). A possibility for as to how the depth map estimate74of the picture32t2of a referenced previous time instance T−1 of the dependent view22may have been derived at the beginning of an random access unit, is described further below. However, the above mentioned possibly explicitly transmitted depth map for view20at the beginning of such random access unit may be warped to view22to obtain the depth map estimate74of the picture32t2of a referenced previous time instance T−1 of the dependent view22, alternatively.

In order to reduce blocking artifacts, the updates71and77could be performed by using weighting functions reducing the influence of the updates of the individual blocks at the block borders.

That is, on the basis of the depth map estimate64as obtained by warping76, the dependent-view reconstructor26reconstructs the current picture32t2of dependent view22using disparity and/or motion compensated prediction based on the disparity and/or motion data54and60for the dependent view22comprised by the dependent view portion38of the multi-view data stream14, and in doing so, the dependent-view reconstructor26provides the disparity estimator28with the disparity and/or motion data54,60, then used by disparity estimator28to perform update77.

After this update77, the disparity estimator28is able to warp-back78the updated depth map estimate74of the current picture32t2of the dependent view22into the reference view20so as to obtain the updated depth map estimate74of current picture32t1of the reference view20for a time instance T which may then serve as a basis/reference for the transition/update79to the next time instance T+1 and so forth.

From that time on, disparity estimator28merely repeats processes71,76,77and78iteratively (wherein step79corresponds to step71) so as to model the depth map estimate along the time axis30so as to continuously support the dependent-view reconstructor26with the depth map estimate64.

Further details regarding all these steps71,76,77,78, and79are described in further detail below. All of these further details shall be individually applicable to the description brought forward with regard toFIG.1.

The disparity estimator28may then, for a current block50/52subject to interview residual prediction, derive the disparity displacement, such as a disparity vector, merely by converting the depth value of this estimated depth map64, at the location of the current block50/52, into such a disparity vector, wherein the inter-view predictor330, in turn, uses this disparity vector in turn in order to copy the thus determined portion from the reference residual signal63, as will be outlined in more detail below with respect toFIG.5.

Thus, the embodiments described so far including the various variants with respect to the disparity displacement derivation for inter-view residual prediction, and possible modifications further various aspects, enable employing coded residual data63of already coded views20together with already coded disparity60/62and motion data42/46for coding a coded picture T of the current view22in multi-view video coding. By employing the already coded residual information63signaled via residual data316together with coded motion/disparity for predicting the residual324′ of the current/dependent view22, the rate associated with coding the residual for the current view22can be significantly be reduced since only the remaining residual328′ has to be performed anymore, which results in an overall bit rate savings for the coding of multi-view video sequences.

The embodiments outlined above are applicable to general block-based hybrid coding approaches without assuming any particular block partitioning, such as macro block or sub-macro block structure. The general structure of an existing block-based coding/decoding concept does not need to be modified in order to be inserted into the above outlined way of block-based disparity-compensated residual prediction. Only an additional residual prediction step or inter-view predictor330needs to be integrated, so that the above embodiments can be integrated in general block-based hybrid video coding schemes with a reasonable complexity increase.

The above embodiments further provide the possibility to adaptively decide for each block (or a subset of blocks)50/52whether a residual prediction330is applied or not. The disparity information (called disparity displacement) used for referring to a block/portion of the residual63of already coded pictures in the reference view22can be derived as explained above, including the variants according toFIG.1cand Fig. Id according to which the derivation is based on coded depth maps (if present) or based on coded disparity vectors60. In alternative embodiments, the disparity information/disparity displacement used for inter-view residual prediction can be explicitly coded as a part of the data stream14. Further, a correction of the disparity information/disparity displacement may be signaled within the data stream as has also been noted above.

One advantage of the embodiments outlined above is that they locate the portions to be copied out of the reference residual signal63of already coded views20via disparity displacement, such as disparity vectors, the derivation of which is completely based on already coded information, such as coded motion and disparity/depth information42and60or34d1, without assuming any particular structure of the actual disparity field of the scene of the multi-view signal12. In particular, it is not assumed that the disparity field can be well approximated by a constant displacement—constant over the whole filed of view—which assumption would not mirror realistic scenes. Instead, actually coded disparity information, such as the disparity data60/62or the depth maps34d1of the reference view, are used for accessing the residual63of an already coded view such as the reference view22.

Furthermore, the embodiments outlined above allow that inter-view residual prediction is used for two types of blocks, namely blocks for which motion data54/56is derived based on motion data42/46in an already coded view, such as the reference view20, and blocks for which motion data54/56has been explicitly transmitted (for example, using spatial or temporal motion vector prediction or without such prediction). That is, the reference-view reconstructor24and the dependent-view reconstructor28may be configured to use motion compensated prediction so as to reconstruct the current pictures of the reference and dependent view, and the dependent-view reconstructor26may be configured to predict motion data54/56of the current picture of the dependent view22, underlying the motion compensated prediction, by locating corresponding positions in the current picture of the reference view20using the derived disparity displacement, and using the motion data for the current picture of the reference view20at the corresponding positions to predict the motion data54/56of the current picture of the dependent view. The dependent-view reconstructor26may then be configured to predict the residual signal of the current block of the current picture of the dependent view, remaining after motion compensated prediction using the predicted motion data, by copying a portion of the reference residual signal63of the current picture of the reference view, displaced from a location of the current block of the current picture of the dependent view22according to the disparity displacement (which had also been used for this block for inter-view motion data prediction.

The disparity displacement, such as the disparity vectors, which are used for accessing the reference residual signal63, can be defined with a sub-sample accuracy, in which case the inter-view residual prediction includes an interpolation of the already coded residual, i.e. the reference residual signal63, at sub-sample locations. That is, the dependent-view reconstructor26(and inter-view predictor66, respectively) may be configured to perform the block-granular disparity-compensated prediction using disparity vectors defined at sub-sample resolution.

Moreover, advantageously, the inter-view residual prediction may be supported for more than one block size.

In some embodiments of the invention, a disparity correction vector can be additionally included in the data stream14for such inter-view residual predicted blocks in order to make the inter-view residual prediction more flexible. More generically speaking, the dependent-view reconstructor26(and inter-view predictor66, respectively) may be configured to refine a disparity displacement for a current block of the current picture of the dependent view20using an explicitly signaled disparity displacement residuum explicitly signaled within a multi-view data stream14(or38), and predict the residual signal of the current block of the current picture of the dependent view20by copying a portion of the reference residual signal63of the current picture of the reference view, displaced from a location of the current block of the current picture of the dependent view according to the refined disparity displacement.

Thus, the concept underlying the above embodiments and the possible variations outlined below can be decomposed into the following steps.Derivation of depth/disparity data for the current picture of the current view.Residual prediction using a derived disparity vector, andSignaling the usage of residual prediction inside the data stream.

Regarding the latter signalization, it should be noted that naturally the interview residual prediction could alternatively be applied to every block, or the activation of the interview residual prediction could be determined in the same manner at encoder and decoder sides based on previously coded/decoded portions of the data stream in which cases no signalization would have to be provided.

In the following, the just-mentioned steps including embodiments are described in more detail. All steps are described for block-based coding and translational disparity vectors. The embodiments—also the above outlined embodiments—are, however, also applicable to more general schemes in which a generalized set of samples such as a non-rectangular part of a block, or any other shape, is associated with a unique set of disparity parameters for defining the disparity displacement. These embodiments are also applicable for coding schemes in which the disparity compensation of the residual is carried out using higher order motion models such as affine motion models, or N-parameter motion models.

In the following, different possible implementation details regarding inter-view residual prediction are described, such as, for example, the way as to how the disparity displacement based on which the residual prediction, such as the residual prediction signal326′, for a current block, such as block50or block52, is actually preformed, may be derived from a depth map estimate64, is described in more detail. In particular, it is assumed that the disparity displacement underlying this inter-view block-based disparity-compensated residual prediction is derived in the form of a disparity vector. That is, a block of a current picture in a particular view, such as view22, that is not the backwards compatible base view, such as20, is inter-view residual predicted using a disparity vector and while different embodiments for deriving such a disparity vector have been described above, the embodiments which will be described in the following, concern those embodiments where the derivation of the disparity vector is performed via the depth map estimate64the estimation of which as been described in accordance with variants ofFIG.1candFIG.1d. Details concerning the actual locating procedure within the reference residual signal63are also described.

That is, in the following description, we preliminarily assume that an estimate of the depth map already exists. With respect toFIG.1candFIG.1dit has already been outlined as to how such estimate may be obtained. Later, we describe how this depth estimate can be derived in even more detail.

The depth data64for the current picture may either given by a pixel-wise or a block-wise depth map. If a pixel-wise depth map is given, the depth map specifies a depth value for each sample (or each luminance sample) of the associated picture. If a block-wise depth map is given, the depth map specifies a depth value for an M×N block of samples (or luminance samples) for the associated picture. For example, a depth value for each block of the smallest possible block size (e.g., 4×4 or 8×8 block) that can be used for motion compensation could be specified. Conceptually, a depth value d given by a sample of the depth map, specifies a function of the real-world depth z, which is the distance between the associated real-world object point (the projection of the real-world object point is the image sample at the given position) and the camera:
d=ƒdz(z)

The depth values are given with a particular precision (furthermore, depth values are often estimated, since the actual depths are usually not known). In most cases, depth values are given by integer numbers. Given the depth values and particular camera parameters (such as the focal length, distance between cameras, minimum and maximum depth values, or functions of these parameters), the depth value d can be converted into a disparity vector v=[vx,vy]T:
v(x)=ƒxd(d(x),x)
where ƒvdspecifies the function that maps a depth value d at sample location x=[x,y]Tto a disparity vector. In a particular important setup is the one-dimensional parallel camera configuration, which is characterized by the following properties:all cameras of the camera array are of the same type and have the same focal lengththe optical axes of all cameras are parallel and lie inside the same planethe scan lines of the image sensors are parallel to the plane that contains the optical axes

In this case, the vertical component of the disparity vector is zero, v=[v,0]T. Each real-world object point has the same vertical location in all views. Its horizontal location depends on the depth of the object point. The difference between the horizontal locations is given by the disparity
v=ƒvd(d).

In an important case, the relationship between the real-world depth z and the depth values d is given in a way that a linear relationship between the disparity v and the depth value d is obtained
v=mvd·d+nvd,
where mvdand nvdand are given by the camera parameters. The depth values d are usually given as integer values. And for internal calculations it is usually also advantageous, if the obtained disparity values are integer values. For example, the disparity can be expressed in the same units that is used for the motion/disparity vectors in motion/disparity-compensated prediction (e.g., half-, quarter, or eighth-sample accuracy). In this case, the integer values for the disparity can be obtained by the integer equation
v=[(m]*vd·d+n*vd)>>uvd,
where “>>” specifies a bit shift to the right (in two's complement arithmetic), and mvdand nvdare scaled (and rounded) versions of and, respectively.

Using the described basic relationships between the given depth values and the actual disparity, we describe embodiments for inter-view residual prediction using a disparity vector that is derived based on the given depth values (or estimate of the depth values) for the current picture of the current/dependent view22.

Derivation of the Residual Signal for a Coded Picture

In the following, an embodiment for deriving the above outlined reference residual63is described in more detail. In this section, it is assumed that the residual prediction in the form of block310is not used within the reference view coding path including reconstructor24and encoder80, although the details outlined below are easily transferable to this case. In other words, it is preliminarily assumed that the residual predictor310inFIG.4does not exist, in which case the output of inverse transformer312is directly connected to the second input of combiner318and the reference residual signal output304, respectively. Then, the subtracter406is also not present and the output of subtracter404is connected to transformer408directly.

In that case, the residual signal412for a block of the reference signal such as block40or44, is the difference between the original signal entering input400for the block, and the prediction signal322that is used for predicting this block. The residual signal412/414is coded using transform coding within transformer408, which includes quantization so that the reconstructed residual as obtained at the output of inverse transformer312is not equal to the difference between original and prediction signal as output by subtracter404. Thus, the reconstructed signal as output by combiner318such as obtained by adding the coded residual as output by inverse transformer312to the prediction signal322, is also not identical to the original signal input at input400. In order to obtain a residual picture for a given coded picture of the reference view20, the residual signals for the blocks40/44of the reference view are reconstructed by the inverse transformer such as by scaling and inverse transforming the transmitted transform coefficient levels316, and arranged to form a residual picture within which the inter-view predictor330locates the block/portion to be copied for an inter-view residual predicted block50/52of the dependent view using the disparity displacement derived for this block as outlined above and further below. This formed residual picture then forms the reference residual signal63passed on to the dependent-view reconstructor26.

In embodiments of the invention, the residual picture finally used for the reference residual signal63, is varied in that residuals as obtained by inverse transformer312for intra-coded blocks40/44are set equal to zero, since the residual for intra coded blocks has different characteristics and cannot be efficiently used for predicting the residual324′ of interview residual prediction coded blocks50/52. In some embodiments of the invention, the residuals for the disparity-compensated blocks, i.e. blocks that are predicted using already coded pictures of even different views as a reference picture (this would be possible if the reference view20itself forms a dependent view of an even more basic underlying reference view), are set equal to zero-additionally or alternatively. In other words, the reference-view reconstructor24may be configured to perform the reconstruction of the current picture of the reference view20of the multi-view signal12using block-based hybrid coding involving at least one intra or interview block coding mode and at least one temporal inter block coding mode, and the dependent-view reconstructor26may be configured to, in predicting the residual signal of the current picture of the dependent view22from the reference residual signal63of the current picture of the reference view, set the reference residual signal63to zero within blocks of the current picture of the reference view20of the intra or inter-view block coding mode. Out of this modified reference residual signal63, the residual signal prediction is then copied using the disparity displacement.

The generation of a particular residual block, i.e. the actual block copying, performed by inter-view predictor330, which residual block is then used for the inter-view residual prediction, can also be performed during the actual process of inter-view residual prediction, i.e. online on demand. That is, it is not necessitated that the complete residual picture, i.e. the residuums of all blocks of the current picture of the reference view in a format put together, as generated before another picture, such as the current picture of the dependent view, is coded/decoded.

Derivation of a Disparity Vector for a Given Block

In the following, it is assumed that a disparity vector is used to define the disparity displacement which specifies which portion out of the reference residual signal63of an already coded picture of another view is used for the inter-view residual prediction. In the following, different embodiments for obtaining such a disparity vector for a given block50/52are described.

In particular, firstly it is assumed that the inter-view predictor330has access to a depth map estimate in accordance with any of the variants ofFIGS.1cand1d. As described above, the inter-view predictor330may alternatively determine the disparity vector in another way such as described above with respect to the other variants.

In an embodiment, first a representing depth value d of the given block50/52subject to inter-view residual prediction, is obtained by inter-view predictor330based on the given sample-based or block-based depth map64. In one embodiment, a particular sample location x of the given block, which may be the top left sample, the bottom right sample, a middle sample, or any other particular sample, such as any other set of corner samples, may be considered. The depth value d=d(x) that is associated with the sample (as given by the given block-wise or sample-wise depth maps64) is used as representing depth value. In another embodiment, two or more sample locations xiof the given block (for example, the corner samples or all samples) are considered and based on the associated depth values di=d(xi), a representing depth values d is calculated as a function of the depth values di. The representing depth value can be obtained by any function of the set of depth values di. Possible functions are the average of the depth values di, the median of the depth values dithe minimum of the depth values di, the maximum of the depth values di, or any other function. After obtaining the representing depth value d of the given block, the depth value is subject to a depth-to-disparity conversion within disparity estimator28so as to convert this depth value into a disparity vector v=ƒvd(d), where the relationship between depth and disparity may be given by coded camera or conversion parameters which parameters may alternatively be set by default. In specific configurations, only horizontal displacements are possible and the depth can be estimated in a way that the depth value is equal to the horizontal displacement.

In another embodiment, a disparity correction vector is coded within the data stream such as within the prediction parameters314′ and the disparity vector that is used for inter-view residual prediction by inter-view predictor330is obtained in the inter-view predictor330before use thereof, by adding the explicitly signaled and coded disparity correction vector as obtained from the prediction parameter314′, to the derived disparity vector derived, for example, from the disparity estimator28or from other disparity information contained within the prediction parameters314′ as already outlined above. Namely, a disparity correction vector may, in accordance with a further embodiment, be coded and the prediction for the disparity vector for the current block to which the disparity correction vector is added, may be obtained by disparity estimator28by a spatial prediction using the disparity vectors of neighboring blocks50/52of the same time instant of the current/dependent view22and/or by a temporal prediction using the disparity vector of possibly co-located blocks in a temporal reference picture of view22. The block in a temporal reference picture may even be indicated by an explicitly coded motion vector so as to be used by inter-view predictor330to access the correct block within the temporal reference picture from which, in turn, the disparity vector is derived which, in turn, is used for accessing the appropriately positioned block/portion out of the reference residual signal63.

Residual Prediction for a Block Using a Derived Disparity Vector

Given the derived disparity vector for the current block50/52, the inter-view residual prediction, such as within inter-view predictor330, may be performed as follows. In particular, the location of the current block, i.e. the block50/52to be subject to inter-view residual prediction, is displaced by the determined disparity vector, and the residual block (with the same block size as the current block) at the displaced location in the reference view picture is used as residual prediction signal.

In order to explain the inter-view residual prediction derivation in more detail, reference is made toFIGS.5aand5b.FIG.5a,bshow the current picture34t2(T) of the dependent view, the reference residual signal, i.e. the reference residual picture,500for the current picture32t1(T) of the reference view and the estimated depth map642(T) for the current picture34t2(T) of the dependent view. As noted above, the residual picture500may be the residuum324which, when added to the prediction signal322, reveals the reconstruction of the current picture32t1.

The block within picture34t2(T) which is to be subject to inter-view residual prediction, is indicated at502. The content of the block502may have been predicted by any of the block coding modes in blocks308′ato308′c, or any combination thereof, with “combination” encompassing pixelwise adding as well as spatial subdivision. That is, although it is advantageous that the boundary of block502coincides with that of a block to which one coding mode is assigned, alternatively, different portions of block502may have been associated with different block coding modes. That is, the coding concept as described so far could also be modified to the extent that block502does not correspond to any of the blocks to which the individual coding modes are assigned, but to a block of a further subdivision which might be different from the block coding mode subdivision and is for deciding as to whether inter-view residual prediction is to be used or not.

In order to determine the portion/block504within the reference view22from which the residual signal for the current block502is to be predicted by inter-view disparity-compensated residual prediction, the location of the current block502is, frankly speaking, displaced by the determined disparity vector506, and thus block504represents a residual block out of the residual picture500for the current picture of the reference view20, having the same block size as the current block502and being displaced from a position co-located to the position of block502via disparity vector506. The location of block508is obtained as follows.

In the encoder, the derived residual prediction signal within block504is subtracted (such as by subtracter406′) from the original residual signal412′ of the current block502which, in turn, is the difference between the original signal of block502and the prediction signal322′ thereof as derived by, for example, motion/disparity-compensated prediction or intra prediction. The remaining signal, i.e.414′, is then coded, such as indicated above using transform coding including quantization.

At the decoder side, the coded residual signal316which is, as outlined above, exemplarily obtained by inverse scaling and transformation of the transmitted transform coefficient levels within data316, and the residual prediction signal326′, are added to the prediction signal322′ derived by, for example, motion/disparity-compensated prediction or intra prediction, in order to obtain the reconstructed signal for the block502as output by the output of adder318inFIG.3.

The exact way of deriving the location of residual block504may be as follows. One or more first sample positions x are used to identify depth values within depth map estimate642(T), from which a representative disparity vector is determined, namely disparity vector506. This might be done by firstly looking-up the one or more depth values in map642(T) and then forming a common depth value therefrom such as by averaging. Then, a depth-to-disparity conversion is performed to obtain vector506. However, it would be possible to switch averaging and conversion. In order to apply the disparity vector506, another determination of a certain position of the current block502is performed such as a vertex thereof or the like. InFIG.5ait's exemplarily the top-left sample. The position thereof in the reference view20, i.e. determined by the same coordinates, is used as a foot point for vector506which, thus, determines the corresponding position of block504, i.e. here the top-left corner thereof, thereby determining the position of block504.

In one embodiment of the invention, sample-accurate displacement vectors506are used so that a sample of the reference residual63,500is directly used for predicting a sample of the current residual326′.

In another embodiment of the invention, sub-sample accurate displacement vector506(for example, quarter-sample accurate displacement vectors) are used. In this case, the residual prediction includes an interpolation of the reference residual signal63,500. A residual sample at non-integer locations may be found by filtering the surrounding residual samples in63,500. In one embodiment, the interpolation is performed in a way that only samples of a single transform block (a block of samples that was represented by a particular transformation) are used for generating a particular sample at a sub-sample location, i.e., the interpolation filter is not applied across transform boundaries of modules312′ and408′. That is, as outlined above, the reference-view reconstructor24may be configured to, in reconstructing the current picture of the reference view20, use transform residual coding, wherein the dependent-view reconstructor26might be configured to, in predicting the residual signal of the current picture of the dependent view, apply an interpolation filter onto the reference residual signal63section-wise so as to not apply the interpolation filter across transform boundaries of the transform residual coding of the reference-view reconstructor.

In a further embodiment, the reference residual samples63,500are filtered before they are used for a prediction of the residual of a current block. As an example, such a filtering can be used for reducing the high-frequency (or low-frequency) components of the residual blocks. When a filtering is combined with the usage of sub-sample accurate displacement vectors506, the filtering can be performed before or after the interpolation (i.e., the original reference residual samples or the generated residual samples at non-integer locations can be filtered).

It is also possible that the residual prediction is not directly done in the spatial domain, but in the transform domain. i.e., the residual prediction signal63may be transformed and the resulting transform coefficients are added to the transform coefficients for the transmitted residual signal, then the final residual signal (which is added to the motion/disparity-compensated prediction signal) is obtained by an inverse transform of the accumulated transform coefficients. The transform of the residual prediction signal may include a quantization.

In that case, the inverse transformer312′ would be positioned between the output of combiner320′ and the input of combiner318instead of the position indicated inFIGS.3and4.

Signaling the Usage of Residual Prediction

In one embodiment of the invention, the usage of residual prediction can be adaptively chosen for a given block. Hence, the usage of residual prediction needs to be signaled to the decoder.

In one embodiment of the invention, a syntax element (for example, a flag) is included into the syntax for all motion/disparity-compensated blocks (but not for intra blocks) and indicates whether residual prediction is applied for the corresponding block. Intra-blocks are decoded without residual prediction. In another embodiment of the invention, the syntax element is only transmitted for motion-compensated blocks (but not for intra blocks and disparity-compensated blocks); whether a block is coded using motion or disparity compensation is signaled by the reference picture index or any other syntax element. If the syntax element indicating residual prediction is not transmitted, residual prediction is not used for the corresponding block.

In addition, the presence of the syntax element indicating residual prediction (and thus the possible usage of residual prediction) can be conditioned on any of the following (including all possible combinations):The syntax element is transmitted only for a subset of the supported block sizes, for example, only for block that are larger or equal to a minimum block sizes.The syntax element is transmitted only for particular coding mode. For example, the syntax element can only be transmitted for block that are not coded in a merge or skip mode.The syntax element is only transmitted if the residual prediction signal contains at least a particular number of non-zero samples (for example, at least one non-zero sample). Or the syntax element is only transmitted if a transform and quantization of the residual prediction signal results in at least a particular number of transform coefficients.

Furthermore, the presence of the syntax element indicating residual prediction may depend on the motion hypotheses that are used for predicting the current block and the reference block or blocks. The reference block (or blocks) is (are) coded blocks in the reference view that are located in the reference view and cover at least one sample of the residual block that is used for residual prediction. That means, the reference blocks can be derived by displacing the location of the current block by the determined disparity vector v. The corresponding conditions for transmitting the residual signal may include any of the following:The syntax element is only transmitted if a reference block is not intra coded.The syntax element is only transmitted if a reference block is coded using motion-compensated prediction (using a temporal reference picture), or only motion-compensated prediction.The syntax element is only transmitted if at least one of the employed reference pictures for the reference block has the same time instant as one of the reference pictures used for the current block. Instead of the time instant, similar measures such as the picture order count or the reference index (in combination with a reference list) can be used.The syntax element is only transmitted if reference pictures of the same time instant are used for both the reference block and the current block. It means, neither the current nor the reference block refer to an access unit that is not used by the other block. Instead of the time instant, similar measures such as the picture order count or the reference index (in combination with a reference list) can be used.The syntax element is only transmitted if the difference between the motion vectors that are used for a particular time instant (or picture order count, reference index) for the reference and the current block is less than or equal to a particular threshold. The motion vector difference can be measured as a component-wise difference or the absolute value of the difference vector or any similar measure. As an example, the syntax element is only transmitted if both motion vectors are the same or only if the difference for each motion vector component is less than or equal to a quarter-sample.

Besides the transmission of a syntax element that indicates the usage of residual prediction, a disparity correction can be transmitted (see above). The transmitted disparity correction can consist of two components (a horizontal and a vertical component), or it can consist only of the horizontal component while the vertical component is inferred to be equal to 0. The presence of the disparity correction can depend on the value of the syntax element that indicates the usage of residual prediction. Furthermore, the presence of the disparity correction can also depend on the block size (for example, it is transmitted only for blocks larger than or equal to a particular size) or on the employed coding mode (for example, it may only be present for blocks that are not coded in a merge or skip mode).

The derivation of the inter-view residual prediction as described so far necessitates, in accordance with the above outlined variations regarding the derivation of the underlying disparity displacement in accordance with, for example,FIGS.1cand1d, that an estimate64of the depth map for the current picture is available. As mentioned above, this depth map estimate64can specify a sample-wise depth map (a depth value is specified for each sample of the current picture) or a block-wise depth map (a depth value is specified for blocks of samples). The depth map estimate64may be derived based on already coded parameters, such as depth maps or disparity vectors and motion parameters. In principle, the possibilities for deriving a depth map estimate64for the current picture can be categorized into two classes. For one class, the depth map estimate is derived based on actually coded depth maps. The corresponding possibilities described below involve that the coded depth maps are present in the bitstream (before they are used). Concepts of the second class don't necessitate that depth maps are coded as part of the bitstream. Instead, the depth map estimate is derived based on coded disparity and motion vectors. The second class of procedures can be applied independently of whether depth maps are coded as part of a bitstream. Both cases discussed above with respect toFIGS.1cand1dfor which the following description in so far provides individually transferable details regarding individual aspects. It should also be noted that, when depth maps are coded, both classes of methods can be applied. It is also possible to select different methods for different frames.

In the following, the basic concept and embodiments for deriving depth maps estimates (with and without coded depth maps) are described.

Class 1: Derivation Based on Coded Depth Maps

If the depth map that is associated with the current picture32t2(T) would be coded before the current picture, the reconstructed depth map could directly be used as an estimate of the real depth map for the current picture. It is also possible to pre-process the coded depth map (e.g., by applying a filtering it) and use the result of the pre-filtering as the estimate of the depth map that is used for deriving the disparity vector used for residual prediction.

In most configurations, the depth map that is associated with a particular picture is coded after the picture32t2(T) (often directly after the associated picture). Such a configuration allows that coding parameters (such as motion parameters) that are transmitted for coding the conventional video pictures can be used for predicting the coding parameters that are used for coding the depth maps, which improves the overall coding efficiency. But in such a configuration, the depth map that is associated with a picture cannot be used as an estimate for the depth map in deriving the disparity vector used for residual prediction. However, the depth map for an already coded view (of the same access unit) such as20is usually available and can be used for deriving an estimate of the depth map of the current picture. At least, the depth map of the base view (independent view)20is available before coding any dependent view22. Since the depth map of any view represents the geometry of the projected video scene to some extent (in combination with camera parameters such as focal length and the distance between cameras) it can be mapped to another view. Consequently, if the depth map for the current picture32t2(T) is not available, the coded depth map for an already coded view of the same access unit20is mapped to the current view and the result of this mapping is used as depth map estimate.

In the following we describe a particular algorithm for realizing this mapping. As described above, each depth value d corresponds to a displacement vector v between two given views. Given transmitted camera or conversion parameters, a depth value d can be converted to a displacement vector my the mapping v=ƒvd(d). Hence, given a depth value d at a particular sample location xRin the reference depth map (already coded), the sample location xCof the same depth value in the current depth map is obtained by adding the disparity vector to XR, xC=xR+v. Hence, each depth value of the reference depth map can be mapped to a sample location of the current depth map in order to obtain a depth map estimate64for the current picture. However, since parts of objects that are visible in one view are not visible in another view, there are sample locations in the depth map for the current view22to which more than one depth values is assigned and there are sample locations in the depth map for the current view to which no depth values are assigned. These sample locations may be processed as follows:If more than one depth value is assigned to a particular sample location, it means that a foreground object is displaced in front of a background object. Consequently, the depth value d (of the potential depth values) that represents the smallest distance to the camera is assigned to such a sample location.If more no depth value is assigned to a particular sample location, it means that a foreground object has moved and the previously covered background is visible. The best that can be done for such regions is to assume that the disoccluded background has the same depth than the neighboring background samples. Hence, regions to which no depth value has been assigned are filled with the depth value of the surrounding samples that represents the largest distance to the camera.

This algorithm is specified in more detail in the following. For simplifying the following description, we assumed that larger depth values represent smaller distances to the camera than smaller depth values (but the algorithm can easily be modified for the opposite assumption):1. All samples of the depth map (estimate) for the current picture are set to an undefined depth value (e.g., −1).2. For each sample location xRof the reference depth map, the following applies:a. The depth value d at the sample location xRis converted to a disparity vector l? using the given camera or conversion parameters, the disparity vector v is rounded to sample accuracy (if applicable), and the sample location inside the current picture is derived by xC=xR+v=xR+[round(f]vd(d)).b. If the depth value at sample location xCin the current picture has an undefined value, the depth value at sample location is set equal to the depth value d.c. Otherwise, if the depth value at sample location xCin the current picture has a defined value dx, with dx<d, the depth value at sample location is modified and set equal to the depth value d.3. The regions in the current depth map that have undefined depth values are filled by a particular hole filling algorithm. For such a hole filling algorithm, the depth value of the background that is uncovered in the current view is derived based on the samples of the surrounding depth values. As an example, the smallest depth map value of the surrounding samples can be assigned. But more sophisticated hole filling algorithms are possible.

The algorithm for mapping a depth map of a given view to a different view is further illustrated inFIG.6on the basis of a very simple example.FIG.6illustrates a possible process for mapping a depth map such as32t1(T) given for one view20to another view22. At the left hand side, the given depth map for the reference view is shown, where the shaded area represents a background and the white area represents a foreground object; in the middle ofFIG.6, middle, the converted depth map obtained by displacing the samples of the given map with the disparity vectors that correspond to the depth values and keeping the foreground object for locations to which more than one sample is projected, is shown. The black area represents on disoccluded area to which no sample has been projected.FIG.6, right, shows the converted depth map after filling the disoccluded areas by the depth value for the background, i.e. by background filling.

In a particular embodiment of the invention, the hole filling can realized by a particularly simple algorithm which processes the lines of the converted depth map separately. For each line segment that consists of successive undefined depth values, the two surrounding values are considered, and all depth samples of the line segment are replaced with the smaller of these two depth values (background depth). If the line segment has only one surrounding depth value (because it is located at the image border), the depth samples of the line segment are replaced with this value. If complete lines have undefined values after this process, the same process is applied for the columns of the depth map.

Although the algorithm above has been described for sample-wise depth maps, it can also be applied to block-wise depth maps (resulting in a lower complexity) or the given sample-wise depth map for the reference view can first be converted into a block-wise depth maps (by downsampling) and then the algorithm can be applied for the block-wise depth map.

Class 2: Derivation Based on Coded Disparity and Motion Vectors

If no depth maps are coded as part of the bitstream, an estimate for the depth map can be generated by using the coded motion and disparity vectors. A basic idea of the following concept can be summarized as follows. The decoding of a (multi-view) video sequence generally starts with a random access unit. The picture for the base view in a random access unit is intra coded without referencing any other picture. The pictures for dependent views in the random access unit can be intra coded or predicted using disparity-compensated prediction. Typically, most blocks will be coded by disparity-compensated prediction, since it usually gives better prediction results than intra prediction. Since, the coded disparity vectors (which are used for disparity-compensated prediction) can be converted into depth values (using the inverse function ƒvd−i), the disparity vectors can be directly used for generating a block-based depth map that is associated with a dependent view in a random access unit (the depth for intra-coded blocks can be estimated based on the depth for surrounding disparity-compensated block). Then, this obtained depth map can be mapped to the base view. The next picture for the base view is typically coded using mostly motion-compensated prediction. Here, it can be assumed that the motion of the depth data is the same as the motion for the texture information (a depth and an associated texture sample belong to the same object point). Given this assumption, the estimated depth data for the first picture in the base view can be motion-compensated for obtaining an estimate for the depth map of the base view in the current access unit. And then, this (motion-compensated) depth map estimate for the base view can be mapped to a dependent view for obtaining a depth map estimate for the current picture (in the current view). If more than two views are coded, the creation of depth map estimates for the third view, fourth view, etc. can be simplified, since we also have a depth map estimate for the first two views of the access unit. One of these depth map estimates (advantageously the base view) can be mapped to the third, fourth, or any following view in order to generate a depth map estimate for this view.

The idea of generating a depth map estimate is further illustrated by some figures (showing the processing steps for multi-view coding with two views as they are performed by depth estimator28). The coding/decoding starts with a random access unit, for which the base view picture32t1(0) is intra-coded and the non-base-view pictures32t2(0) are coded using only intra and inter-view prediction (but no motion-compensated prediction). After coding the second view22in the random access unit “0”, a block-based depth map estimate for this second view22is generated120using the coded disparity vectors122for this view22, as illustrated inFIG.7. This depth map estimate642(0) for the second view22is then mapped124to the first view (base view)20and a depth map estimate641(0) for the first view20is obtained. It should be noted that for the second view22of a random access unit, no estimate of the depth map is available when the second view22of a random access unit is coded.

If a third view is coded, the depth map estimate of any of the first two views (advantageously the second view) can be mapped to the third view resulting in a depth map estimate for the third view, which can be used for deriving disparity vectors for the third view. And after coding the third view, a block-based depth map can be generated using the coded disparity vectors for the third view (which can than later be used for generating a depth map estimate for any following view). For any following view, basically the same process as for the third view can be used.

The pictures of the base view in non-random-access units are typically mainly coded by motion-compensated prediction, since motion-compensated prediction usually gives better coding efficiency than intra coding. After a picture of the base view is coded, an estimate of the depth map for this picture is generated140(cp.71inFIG.1) using the motion parameters42(1) for the picture32t1(1), as illustrated inFIG.8. Therefore, each block of the new depth map estimate641(1) is created140by motion-compensating the depth map estimate641(0) (cp.74inFIG.1) for the corresponding reference picture or pictures. The reference pictures and corresponding motion vectors42(1) that are used are the reference pictures and motion vectors that are coded in the data stream for the associated picture. The depth samples for intra-coded blocks can be obtained by spatial prediction. This depth map estimate for the base view is than mapped142(cp.76inFIG.1) into the coordinate system for the second view in order to obtain a depth map estimate642(1) for the second view which can be used for deriving disparity vectors, i.e. to perform inter-view redundancy reduction.

For any further coded view, a depth map estimate can be generated by mapping the depth map estimate for any already coded view (base view, second view, etc.) to the corresponding view.

After actually coding the picture of the second view (or any following view), the associated depth map estimate can be updated160(cp.77inFIG.1) using the actually coded motion and disparity vectors, as illustrated inFIG.9. For blocks that are coded using disparity compensation (or for which disparity vectors or disparity correction vectors are transmitted), the depth map samples can be obtained by converting162the coded disparity vectors60to depth values as described above. For blocks that are coded using a motion-compensated mode, the depth samples can be obtained by motion compensating the depth map estimate for the reference frame32t2(0). Or alternatively, a depth correction value, which is added to the current depth map estimate642(1), can be derived based on the coded motion parameters42(1) and54(1) for the current and for the reference view. The depth samples of intra coded blocks can be predicted using spatial prediction or using the motion parameters of neighboring blocks. After an updated depth map estimate74for the second view has been generated, this depth map estimate74is mapped164(cp.78inFIG.1) to the base view20for obtaining a depth map update64′ i(1) (cp.74inFIG.1) for the base view20.

If more than two views are coded, the depth map update process for these views is the same as for the second view. However, the base view depth map is only updated after the second view has been coded.

The motion compensation operations for depth maps can be performed using the coded sub-sample accurate motion vectors. It is, however, often advantageous (from a complexity as well as coding efficiency point of view), if the motion compensation operations for depth maps are performed with sample (or even block) accuracy. Therefore, the actually coded motion vectors are rounded to sample or block accuracy and these rounded vectors are used for performing the motion compensation. Furthermore, the described concept can be applied for sample-wise as well as block-wise depth map estimates. The advantage of using block-based depth maps is a lower complexity and memory requirement for all processing steps. With block-based depth maps, each depth sample represents the depth for a block of samples of the associated picture (e.g., 4×4 blocks or 8×8 blocks). All described operations can be performed for block-based depth maps in a straightforward way (i.e., by simply considering a lower resolution of the depth maps—one depth sample just represents multiple instead of one texture sample).

Besides the mapping of a given depth map from one view to another (which has been described above), the algorithm contains the following basic steps:

Creating a depth map based on disparity vectors for a picture of a random access unit.Temporal prediction of the base view depth map using the motion parameters of the associated picture.Update of a depth map estimate using actually coded motion and disparity vectors for the associated picture.
Particular embodiments for these algorithmic steps are described in the following.
Creation of a Depth Map for a Picture in a Random Access Unit

In a particular embodiment of the invention, the creation of a depth map for a picture of a dependent view in a random access unit proceeds as follows. In general, such a picture contains blocks that are coded using disparity-compensated prediction as well as blocks that are intra coded. First, all blocks that are coded using disparity-compensated prediction are considered. The disparity vectors are converted into depth values and these depth values are assigned to the corresponding samples of the depth map. If two or more motion hypotheses are used, either one hypothesis is selected or the final depth value is set equal to a function of the depth values for the individual motion hypotheses (for example, the average, the median, the maximum, or the minimum). After assigning the depth values for all disparity-compensated blocks, the depth values for intra coded blocks are obtained by spatial intra prediction. In one version, these samples can be obtained by using the same intra prediction modes that are used for the associated texture picture. In another version, the depth of an intra-block can be set equal to a depth values that is obtained by a weighted average of the surrounding samples (or blocks), where the weighting factors can be determined based on the used intra prediction modes. In a further version, the depth for an intra-block can be obtained by setting the depth samples equal to a value that is given by a particular function of the surrounding intra samples (e.g., the average, the median, the maximum, the minimum). Other spatial prediction algorithms are also possible. The depth assignment for intra-coded blocks can also be done inside a single loop over the blocks in an image. That means, the blocks are processed in a particular order (e.g., the coding order), and for both disparity-compensated and intra blocks, the depth values are generated in this order (i.e., the depth assignment for intra-coded blocks doesn't need to wait until all disparity-compensated blocks are processed).

Temporal Prediction of the Base View Depth Map

In general, pictures of the base view contain motion-compensated blocks and intra coded blocks. The depth values for motion-compensated blocks are derived by motion-compensated prediction of the depth map estimate for the corresponding reference picture. If a block of the texture picture is coded using a single motion hypothesis, the depth samples for this block can be obtained by displacing the depth samples of the depth map estimate for the reference picture (given by the signaled reference index) by the transmitted (or inferred) motion vector. This motion compensation operation can be performed with the accuracy of the transmitted motion vectors (which is usually a sub-sample accuracy) or with sample- or block-accurate motion vectors. If the motion compensation is performed with sub-sample accuracy, an interpolation filter is applied for generating the samples at sub-sample positions. If the motion compensation is performed with sample or block accuracy, the transmitted motion vectors are rounded before they are used. If the block of the associated picture is coded with more than two motion hypothesis, one of the hypotheses can be selected for motion compensation of the depth maps, or all motion hypotheses are used by generating the final depth map estimate for the block as a weighted sum of the depth prediction signals for the individual motion hypotheses. Furthermore, the depth samples for a block of a given sizes can be set equal to a representing depth. This representing depth can be obtained by selecting a particular location inside the block and deriving the depth value for this location using motion compensation, or it can be obtained by defining a function of the motion-compensated depth values for this block. Such a function can be the average of the depth samples, or the median of the depth samples, or the minimum or maximum of the depth samples, or the depth sample value that occurs most often in the block.

Update of a Depth Map Using the Coded Motion and Disparity Parameters

As mentioned above, the depth map estimate for a picture can be updated after coding the picture. In the following, we describe some embodiments of such an update algorithm. Blocks of the picture are processed in a particular order (for example, the coding order) and for each block, the following applies:If the block has been intra-coded (in the associated texture picture), the depth sample values for this block can be derived by spatial prediction using the samples of neighboring block. Some examples for such a spatial prediction technique have been described above. The depth sample values can also be obtained by motion compensation using the motion parameters of a neighboring block (see the description for motion-compensated blocks below). It is sometime advantageous, if the intra block are processed after all motion- and disparity-compensated blocks are processed, because then more neighboring are available and can be used for spatial prediction of depth samples or motion parameters.

Otherwise, if the block is coded using one or more disparity vectors (disparity hypotheses), the depth samples are derived by converting the disparity vectors to depth values. If only one disparity vector (disparity hypothesis) is used for the block, the depth value if given by the corresponding disparity vector. If two or more disparity vectors are used for the block, one of the disparity vectors can be selected for deriving the depth value, or for each disparity vector a depth value can be derived and the finally assigned depth value is obtained by applying a function of the individually derived depth values. Possible functions are, among others, the minimum or maximum of the individual depth values, the median of the depth values, the average of the depth values, or the depth values that occurs most often.

Note that a block that is coded using a disparity vector may additionally be associated with a temporal motion vector. In this case, the temporal motion vector can be ignored. Or the derived depth values can be combined with depth values that are derived for temporal motion hypotheses (see below) in any specific way (e.g., by averaging these two signals).

The disparity vector based depth estimation can also be used if a disparity vector or a disparity vector correction has been coded for updating/refining the derived disparity vector used for residual prediction.Otherwise, the block is coded using only temporal motion hypotheses and the temporal motion hypotheses are used for updating the depth samples for the block.

In a first embodiment of the invention, the depth map samples are derived by straightforward motion compensated prediction using the depth map estimates associated with the reference pictures for the current view. This motion compensation operation can be realized by any of the embodiments for temporal prediction of the base view depth map described above.

In a second embodiment of the invention, the depth map samples are not simply motion compensated, but instead a depth correction value is derived based on the motion vector coded for the current view and the motion vector coded for the corresponding block in the reference view and this depth correction value is added to the depth map estimated in order to obtain an updated depth map estimate. The advantage of such an approach is that depth changes between two instances can be considered.

Let dprdbe the current depth estimate for a sample or a block, let mcurrbe the motion vector that is actually used for motion-compensation of the current block, and mrebe the motion vector for the reference block in the reference view (the reference block is derived by using a disparity vector given by the depth estimate dprdas described above). Furthermore, let ‘vt-1be the disparity vector in the reference access unit and let vtbe the disparity vector in the current access unit. Given the basic equation for the interrelationship between the motion and disparity vectors that we derived in the beginning of the description (see illustration inFIG.10for s single sample),
mcurr+vt-1−mref−vt=0,
the current disparity vector can be set equal to
vt=vt-1+(mcurr−mref),
if we assume that the coded motion parameters and the disparity in the reference picture are correct. The disparity vector can be converted into a depth and vice versa. Hence, the current depth can be expressed as
d=dt-1+ƒvd−1(mcurr−mref),

The depth dt-1is the depth value given in the reference image, it can be accessed using the motion vector for the current block. It is also possible to replace the depth in the reference image with the current estimate for the depth dprd, since this depth has been derived using motion compensation from the depth in a reference image.

We showed the basic concept for obtaining an updated depth value using the motion vectors of the current and the reference block. This basic concept can be used in a variety of actual implementations. For example, if more than one motion vector is used for the current block and/or the reference block, a single motion vector can be selected or different depth values can be derived for the different motion vector and the final depth is obtained by using the average (or any other function) of the individually derived depth values. Furthermore, only those motion hypotheses in the current and the reference block should be considered that use the same reference access unit (or reference time instant, or reference picture order count, or reference index). If such motion hypotheses don't exist, the conventional motion compensation process described above can be used or the depth sample can be marked as not available and are later replaced by suitable post-processing steps. Furthermore, the new depth value can be derived based on a single sample for a block, or for all samples of a block, or for a selected subset of the samples of a block. When different depth values for different samples are derived, separate depth values can be assigned for different regions of the current block (e.g., for all 4×4 blocks inside the current block), or the different depth values are used for deriving a representing depth for the entire block, e.g., by using the average, the median, the minimum, the maximum value, or the depth values that occurs most often. The depth map update process can also be combined with some pre-processing steps. For example, not available depth values can be replaced by the depth values of neighboring samples or blocks.

The above embodiments thus enable reducing the bit rate associated with coding residuals in multiview video coding applications by employing the residual data coded in one view for an efficient coding of the residuals for other views. These embodiments are applicable to multiview video coding with two or more views and for multiview video coding with and without depth maps. According to these embodiments it is possible to re-use the coded residual in one view for efficiently coding the residual in another view. Since all views represent the same video scene, the changes of sample values from one frame to another are similar for different views and this effect can be exploited for an efficient coding of dependent view. The above embodiments describe a concept for efficiently employing the residual of already coded views for following views, by using estimated or coded depth data. In accordance with some embodiments, the motion data that are coded in one view, is additionally used for predicting the motion data of other views.

Favorably, the above embodiments enable to use the concepts of inter-view motion and residual prediction independently of each other. In rate-distortion sense, the coding efficiency can be usually improved when different tools can be independently selected. For blocks for which the derivation of motion parameters from a reference view works well, an additional residual prediction may decrease the coding efficiency (i.e., it may increase a cost measure D+λ·R, or in other words increases the distortion for a given rate, or increases the rate for a given distortion). Or for blocks for which the residual prediction from a reference view improves the coding efficiency, an additional derivation of motion parameters based on the reference view may decrease the coding efficiency.

Moreover, the above embodiments enable to use sub-sample accurate vectors. Same can potentially increase the coding efficiency (when it is combined with a better model for the disparity in a picture). Finally, the above embodiments are combinable with the usage of blocks of different block size. Recent investigations (see the development of HEVC) have shown that the provision of multiple block sizes can significantly increase the coding efficiency, and that's why it could be advantageous to provide residual prediction for different block sizes. In particular, the new video coding standardization project of the ITU-T and ISO/IEC JTC 1/WG 11, which is also referred to as HEVC, shows very promising improvements in conventional 2-d video coding technology. The current working draft of HEVC provides substantial coding gains compared to ITU-T Rec. H.264 I ISO/IEC 14496-10. For achieving these gains several concepts have been extended in comparison to ITU-T Rec. H.264 I ISO/IEC 14496-10, and the above described approach for using the residual data of a different view for predicting the residual of the current view cannot be straightforwardly applied to the HEVC design. On the other hand, these techniques have to compete with the improved residual coding in HEVC. The main improvements in the area of residual coding include:While the blocks sizes that are used for motion-compensated prediction in ITU-T Rec. H.264 I ISO/IEC 14496-10 range from 4×4 to 16×16 luma samples, a much larger variety of blocks sizes is supported in HEVC, which ranges from 4×4 to 64×64 luma samples. In addition, the basic coding units are not given by fixed macroblock and sub-macroblocks, but are adaptively chosen. The largest coding unit is typically a block of 64×64 luma samples, but the largest block size can actually be signaled inside the bitstream. The splitting of a block into subblock can establish a subdivision hierarchy of 4 or more levels. Furthermore, a block that is used for prediction can be further split into multiple transform blocks for the purpose of residual coding. Here, a hierarchy of multiple levels is supported.Different transform sizes ranging from 4×4 to 16×16 transforms are supported.The entropy coding of transform coefficients has been improved.

Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some one or more of the most important method steps may be executed by such an apparatus.

Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.

Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.

Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and/or non-transitionary.

A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet.

A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are performed by any hardware apparatus.

While this invention has been described in terms of several advantageous embodiments, there are alterations, permutations, and equivalents which fall within the scope of this invention. It should also be noted that there are many alternative ways of implementing the methods and compositions of the present invention. It is therefore intended that the following appended claims be interpreted as including all such alterations, permutations, and equivalents as fall within the true spirit and scope of the present invention.

REFERENCES

[1] ITU-T and ISO/IEC JTC 1, “Advanced video coding for generic audiovisual services,” ITU-T Recommendation H.264 and ISO/IEC 14496-10 (MPEG-4 AVC), 2010.[2] A. Vetro, T. Wiegand, G. J. Sullivan, “Overview of the Stereo and Multiview Video Coding Extension of the H.264/MPEG-4 AVC Standard”,Proceedings of IEEE, vol. 99, no. 4, pp. 626-642, April 2011.[3] H. Schwarz, D. Marpe. T. Wiegand, “Overview of the Scalable Video Coding Extension of the H.264/AVC Standard”,IEEE Transactions on Circuits and Systems for Video Technology, vol. 17, no. 9, pp. 1103-1120, September 2007.