Patent Description:
An example for a multi-view video codec supporting residual prediction is 3D-HEVC. In the current design of 3D-HEVC [<NUM>], the coding unit (CU) decoding process for residual prediction ([<NUM>], [<NUM>]) refers to three reference pictures. The pictures can be identified by a reference picture order count (POC), a reference view order index (VOI), the current POC, and the current VOI.

The reference POC is the POC of a picture which is a) included in the reference picture list RefPicList of the current picture; and b) included in the current layer. The reference POC is derived from syntax elements of the VPS and the slice header of the current picture and is constant for all CUs of the current picture ([<NUM>], [<NUM>]).

The reference VOI is the VOI of a picture which is a) included in the reference picture list RefPicList of the current picture; and b) included in the current access unit (AU). It is derived from syntax elements of the VPS, the slice header, and the current CU and can thus vary for CUs of the current picture.

The three reference pictures referred by a CU are included in:.

When a reference POC and a reference VOI can be derived and all three reference pictures are available, residual prediction is enabled for the current CU.

The availability of picture V and picture A is implicitly guaranteed since:.

Picture VA is available, when the decoded picture buffer (DPB) includes a picture which is marked as "used for reference" and has the reference POC and the reference VOI ([<NUM>]). Thus, whether the decoding process for residual prediction is invoked for a current CU depends on the state of the DPB.

Note that, although not explicitly tested in the decoding process, the picture VA is available when both of the following conditions are fulfilled:.

Note that, without loss of generality the description above and in the following assumes, that the current CU and the current slice only performs P prediction. For B prediction the process.

An Example <NUM> is shown in <FIG>, which depicts layers of a 3D-HEVC multilayer bitstream and picture dependencies of CUs of a current picture. The current picture is picture <NUM> of view <NUM>. The parameters sets VPS, SPS; pictures <NUM>, <NUM> and <NUM> of views <NUM>, <NUM>, and <NUM>; pictures <NUM> and <NUM> of view <NUM>; and the slice header of the current picture are decoded before decoding any coding units (CUs) of the current picture.

Moreover, before decoding the first CU of the current picture the reference picture list refPicList is derived from syntax elements of the VPS and the slice header of the current picture. The list refPicList includes pictures <NUM> and <NUM> of view <NUM> and pictures <NUM> of views <NUM> and <NUM>. The reference POC is selected among POCs of pictures included refPicList and view <NUM>, such that it is equal to <NUM>.

When CU <NUM> is decoded the reference VOI is derived from syntax elements of the VPS, the slice header of the current picture and CU <NUM>, such that it is equal to <NUM>. Hence, pictures V and A are picture <NUM> in view <NUM> and picture <NUM> in view <NUM>, respectively. To determine whether residual prediction is enabled for the CU <NUM>, it is furthermore tested if the picture <NUM> of view <NUM> (picture VA) is included in the DPB and "marked as used for reference". Since picture <NUM> of view <NUM> is present in the bitstream and included in RefPicSetStFoll of picture <NUM> of view <NUM> this is the case and residual prediction is enabled for CU <NUM>.

When CU <NUM> is decoded the reference VOI is derived from syntax elements of the VPS, the slice header of the current picture and CU <NUM>, such that it is equal to <NUM>. Hence, pictures V and A are picture <NUM> in view <NUM> and picture <NUM> in view <NUM>, respectively. To determine whether residual prediction is enabled for the CU <NUM>, it is furthermore tested if picture <NUM> of view <NUM> (picture VA) is included in the DPB and "marked as used for reference". Since picture <NUM> of view <NUM> is present in the bitstream and included in RefPicSetLtCurr of picture <NUM> of view <NUM> this is the case and residual prediction is enabled for CU <NUM>.

A problem of the current residual prediction decoding process is the dependency on the presence of a picture with the reference POC and VOI (picture VA) in the DPB. Such a dependency on the state of the DPB is in general avoided in HEVC specification by design choice. A drawback of the design is that a loss of picture VA is not necessarily detected when decoding the current picture.

Thus, when decoding a current CU and the picture VA is not present in the DPB, it is not clear whether it is lost or intentionally not present.

See Example <NUM> in <FIG> depicts the same setup as in example <NUM>, with the difference that picture <NUM> in view <NUM> has been lost. Thus, condition <NUM> (as provided above) is not fulfilled and picture VA for CU <NUM> is not in the DPB, such that residual prediction is disabled for CU <NUM>. However, the erroneous decoding cannot be detected when decoding the current picture, since picture <NUM> in view <NUM> is not required to be present. Note that, the presence of picture <NUM> of view <NUM> in RefPicSetStFoll or RefPicSetLtFoll of picture <NUM> in view <NUM> does not require its presence in the bitstream or in the DPB. Reason for this design choice in HEVC is e.g., in a scenario without residual prediction at all, to be able to discard picture <NUM> of view <NUM> without changing picture <NUM> of view <NUM>.

<CIT> discloses a method that includes determining a disparity reference block in a disparity reference picture indicated by a disparity vector associated with the first block, and determining whether a decoded picture buffer, or alternatively a reference picture list, of the disparity reference picture contains a temporal-disparity reference picture in the second view and having the picture order count value of the temporal reference picture. When the decoded picture buffer, or alternatively when the reference picture lists, of the disparity reference picture does not contain a temporal-disparity reference picture in the second view and having the picture order count value of the temporal reference picture, the method includes disabling an inter-view residual prediction process for predicting residual data of the first block.

Accordingly, there is a need for a multi-view video codec with an improved support of residual prediction involving a picture neither coinciding in timestamp nor in view with a currently coded/decoded picture, such as improved with respect to robustness against transmission lost and/or respect to processing costs.

The object of the present invention is to provide such multi-view video codec with an improved support of residual prediction involving a picture neither coinciding in timestamp nor in view with a currently coded/decoded picture, such as improved with respect to robustness against transmission lost and/or respect to processing costs.

This object is achieved by the subject matter of the independent claims.

In accordance with the present application, multi-view video coding/decoding supporting residual prediction involving a reference picture of different view and timestamp compared to a currently coded/decoded picture may be improved in terms of robustness and/or processing cost if checking any picture availability or non-availability in a decoded picture buffer is avoided. In accordance with a first aspect of the present invention, the enablement or disablement of the residual prediction for a predetermined motion- or disparity-compensatedly predicted coding unit is, at least with respect to a third reference picture different in view and timestamp compared to a current picture, performed dependent on an inspection of a parameter set in the data stream which relates to a second reference picture of the current picture which coincides in reference view with the third reference picture and coincides in timestamp with the current picture. From this parameter set, the set of all reference pictures is derived which are required, in terms of a bitstream constraint of HEVC, to be present in a decoded picture buffer when decoding the second reference picture and are of the reference view, i.e. the set of reference pictures that are available for intra-layer prediction of the second reference picture. By this measure, the decision on enablement/disablement of the residual prediction is no longer dependent on a presence or absence of the third reference picture in a decoded picture buffer, and accordingly a situation where the third reference picture is absent from the decoded picture buffer by accident due to transmission loss does not lead to an incorrect classification of coding units into ones to be subject to residual prediction and ones not to be subject to residual prediction. In accordance with another aspect of the present application and not covered by the claimed invention, the decision on enablement/disablement of residual prediction of a predetermined motion- or disparity-compensatedly predicted coding unit is, at least with respect to the third reference picture, performed by flags in a second parameter set within the data stream relating to the current picture. A first parameter set in the data stream which relates to the current picture is used to identify a first set of reference pictures included in a view different from the current view and potentially serving as the second reference picture for coding units of the current picture as well as a reference timestamp of a reference picture which potentially serves as the first reference picture for coding units of the current picture. The first set of reference pictures thus coincides in timestamp with a current picture. The second parameter set comprises, for each of a second set of reference pictures comprising one picture for each reference picture of the first set which coincides in view with the respective reference picture of the first set of reference pictures and is of the reference timestamp, a flag which indicates whether or not the respective picture of the second set of reference pictures is available as third reference picture. That is, in accordance with a second alternative, flags directly indicate the available or non-availability of the "third reference picture", denoted VA elsewhere in the description of the present application, and accordingly increases the error robustness and reduces the processing complexity involved in supporting residual prediction.

Advantageous implementations are the subject of dependent claims. Preferred embodiments of the present application are described below with respect to the figures, among which:.

Before describing various embodiments of the present application further below, the thoughts and advantages underlying these embodiments shall be explained by resuming the explanation of problems occurring in the current design of 3D-HEVC of the introductory portion of the specification of the present application and describing, on the basis of the examples described above and shown in <FIG> and <FIG>, solutions which may overcome the problems described.

An idea in accordance with a first aspect of the present application is to allow a direct detection of a loss of picture VA by enabling residual prediction only, when picture VA it is included in RefPicSetLtCurr, RefPicSetStCurrBefore, or RefPicSetStCurrAfter of picture V. It is a bitstream requirement, that pictures included in these sets are present in the DPB, when decoding picture V. Their presence is conventionally checked when decoding picture V. Thus, the loss of the picture VA would be detected already when decoding picture V, hence before decoding the current picture. Note that, between decoding of picture V and decoding the current picture, pictures included in RefPicSetLtCurr, RefPicSetStCurrBefore, or RefPicSetStCurrAfter of picture V are not removed from the DPB. Thus, the presence of picture VA when decoding picture V is sufficient to guarantee presence of picture VA in DPB when decoding the current picture.

See the Example <NUM> in <FIG> depicts the decoding process using the just proposed solution. Picture <NUM> of view <NUM> (picture VA of CU <NUM>) is included in RefPicSetStFoll of picture <NUM> of view <NUM> (picture V of CU <NUM>), but not in RefPicSetLtCurr, RefPicSetStCurrBefore, or RefPicSetStCurrAfter of picture V of CU <NUM>. Thus, residual prediction is disabled for CU <NUM>, since it cannot be inferred whether the presence of picture VA of CU <NUM> in the DPB is required.

Picture <NUM> of view <NUM> (picture VA of CU <NUM>) is included in RefPicSetStCurrBefore of picture <NUM> of view <NUM> (picture V of CU <NUM>). Thus, residual prediction is enabled for CU <NUM>, since it is guaranteed that picture VA of CU <NUM> is present in the DPB. When picture VA of CU <NUM> would not be present, this would have been detected when decoding picture <NUM> of view <NUM>, since the presence of picture <NUM> of view <NUM> is required there by a bitstream constraint.

Specification changes in the current specification of 3D-HEVC could be as follows.

Please note that insertions are indicated using underlining and canceled portions are shown struck through.

This process is invoked when the current slice is a P or B slice.

The variables RpRefIdxL0 and RpRefIdxL1 are set equal to -<NUM>, and the variables RpRefPicAvailFlagL0 and RpRefPicAvailFlagL1 are set equal to <NUM>.

The following applies for X in the range of <NUM> to <NUM>, inclusive:.

The variable RpRefPicAvailFlag is derived as specified in the following: <MAT>.

When RpRefPicAvailFlag is equal to <NUM>, the following applies for X in the range of <NUM> to <NUM>, inclusive:.

When RpRefPicAvailFlag is equal to <NUM> and RefRpRefAvailFlagL0[ refViewOrderIdx ] is equal to <NUM> for any refViewOrderldx in the range of <NUM> to MaxLayersMinus1, inclusive, it is a requirement of bitstream conformance that PicOrderCnt( RefPicList0[ RpRefIdxL0 ] ) shall be the same for all slices of a coded picture.

When RpRefPicAvailFlag is equal to <NUM> and RefRpRefAvailFlagL <NUM>[ refViewOrderIdx ] is equal to <NUM> for any refViewOrderldx in the range of <NUM> to MaxLayersMinus1, inclusive, it is a requirement of bitstream conformance that PicOrderCnt( RefPicList1[ RpRefIdxL1 ] ) shall be the same for all slices of a coded picture.

An alternative solution not covered by the claimed invention is exploited by embodiments in accordance with a second aspect of the present application described further below and is to:.

This way access to RefPicSetLtCurr, RefPicSetStCurrBefore, RefPicSetStCurrAfter of picV can be avoided. However, additional signaling costs are intruded by additional slice header signalization.

Here, specification changes following the alternative solution may be:.

ref rp ref avail flag l0[ i ] equal to specifies that the picture picA with PicOrderCnt( picA ) equal to PicOrderCnt( RefPicListX[ RpRefIdxL0 ] ), Viewldx( picA ) equal to ViewOrderIdx( RefPicLayerId[ i ] ), OtherDimsEqualFlaqf currPic, picA, DIM VIEW ) equal to <NUM> might not be present and is not used in the residual prediction decoding process of the current slice. ref rp ref avail flag l0[ i ] equal to <NUM> specifies that the picture picA with PicOrderCnt( picA ) equal to PicOrderCnt( RefPicListX[ RpRefIdxL0 ] ), Viewldx( picA ) equal to ViewOrderIdx( RefPicLayerId[ i ] ), OtherDimsEqualFlaqf currPic, picA, DIM VIEW ) equal to <NUM> is present and might be used for the residual prediction decoding process of the current slice,.

ref rp ref avail flag l1[ i ] equal to specifies that the picture picA with PicOrderCnt( picA ) equal to PicOrderCnt( RefPicListX[ RpRefIdxL1 ] ), ViewIdx( picA) equal to ViewOrderIdx( RefPicLayerId[ i ] ), OtherDimsEqualFlag( currPic, picA, DIM VIEW )equal to <NUM> might not be present and is not used in the residual prediction decoding process of the current slice. ref rp ref avail flag l1 [ i ] equal to <NUM> specifies that the picture picA with PicOrderCnt(picA ) equal to PicOrderCnt( RefPicListX[ RpRefIdxL0 ] ), ViewIdx( picA ) equal to ViewOrderIdx( RefPicLayerId[ i ] ), OtherDimsEqualFlag( currPic, pica, DIM VIEW ) equal to <NUM> is present and might be used for the residual prediction decoding process of the current slice.

When RpRefPicAvailFlag is equal to <NUM> and RefRpRefAvailFlagL1[ refViewOrderIdx ] is equal to <NUM> for any refViewOrderldx in the range of <NUM> to MaxLayersMinus1, inclusive, it is a requirement of bitstream conformance that PicOrderCnt( RefPicList1[ RpRefIdxL1 ] ) shall be the same for all slices of a coded picture.

After having exemplified and illustrated the thoughts and advantages of embodiments of the present application described further below, it is noted that the fact that the above illustration specifically referred to the current design of 3D-HEVC shall not be interpreted as a restriction of the subsequently described embodiments by any extent. That is, although the subsequently described embodiments emerged on the basis of thoughts explained above, they may be implemented in a manner differing from 3D-HEVC. In other words, although the embodiments described below may be implemented to result in a 3D-HEVC codec with the advantages set out above, the embodiments described below may alternatively be implemented differently.

<FIG> shows a multi-view video decoder <NUM> in accordance with an embodiment of the present invention. The multi-view video decoder <NUM> comprises a video decoding core <NUM> and a residual prediction switch <NUM>. As illustrated in <FIG>, the residual prediction switch <NUM> may, in fact, be included into the video decoding core <NUM>.

The video decoding core <NUM> is configured to decode a multi-view video <NUM> from a data stream <NUM> received by the multi-view video decoder. in <FIG>, the plurality of views comprised by the multi-view video <NUM> is four in order to ease the comparison of the embodiment of <FIG> with the description brought forward above with respect to <FIG>, but the number of views may alternatively be lower than four, such as two or any number greater than two. Each view V0. V3 is composed of a temporal sequence of pictures so that each picture <NUM> belongs to a certain timestamp Pi along the temporal axis <NUM>, wherein <FIG> shows, for illustration purposes, merely pictures belonging to two consecutive timestamps P1 and P2. The pictures of the various views show, for instance, a common scene at the different time instances or timestamps Pi from different view positions corresponding to the views Pj.

The video decoding core supports residual prediction of coding units of a current picture <NUM> of a current view. This means the following. The video decoding core <NUM> decodes pictures <NUM> of the views V1. V4 in units of coding units into which pictures <NUM> are subdivided. For instance, coding units may be leaves of a recursive multi-subdivision of tree root blocks into which the pictures <NUM> are, regularly in rows and columns, pre-subdivided. However, this is merely an example and the coding unit itself may form a regular subdivision of the pictures in rows and columns. The video decoding core obeys, in decoding data stream <NUM>, a certain decoding order of the coding units of pictures <NUM>. The video decoding core <NUM> may, for instance, decode the coding units sequentially along the decoding order. Alternatively, the video decoding core <NUM> may use parallel processing in order to decode coding units of the pictures in parallel with, however, obeying the decoding order defined among the coding units so that portions of pictures of the video <NUM> preceding the current coding unit in terms of the decoding order have already been processed/decoded. The decoding order may, for instance, traverse the coding units picture-wise, with traversing the pictures of one timestamp before proceeding to the coding units of pictures of another timestamp. The decoding order traverses, for example, the pictures of a certain timestamp sequentially along a view index i of views Vi.

Different coding modes may be associated with the coding units. The data stream <NUM> may signal these coding modes. For instance, some coding units may be motion-compensatedly predicted coding units which are predicted by the video decoding core <NUM> temporally by use of motion-compensated prediction on the basis of reference pictures which proceed in decoding order, but are of the same view. Other coding units may be disparity-compensatedly predicted coding units which the video decoding core <NUM> predicts using disparity-compensated prediction on the basis of pictures of, in terms of decoding order, preceding/lower layers belonging to the same time instant/timestamp.

Even further coding units may be intra-coded coding units for the decoding of which the video decoding core <NUM> neither uses temporal nor inter-view prediction. Spatial prediction, for instance, may be used for such intra-coded coding units.

The video decoding core <NUM> supports the residual prediction of a predetermined motion- or disparity-compensatedly predicted coding unit of a current picture <NUM> of a current view on the basis of a first reference picture of a current view, a second reference picture of a reference view, coinciding in timestamp at the current picture, and a third reference picture of the reference view coinciding in timestamp with a first reference picture. For illustration purposes, <FIG> illustrates a coding unit <NUM> in picture <NUM> of view V3 at timestamp P2. This coding unit <NUM> may be, for instance, the current coding unit.

As will be described below with respect to <FIG> and <FIG>, the video decoding core <NUM> may be configured to, in subjecting a current coding unit <NUM> to residual prediction, determine a subset of reference pictures out of the first, second and third reference pictures A, V and VA depending on the predetermined motion- or disparity-compensatedly predicted coding unit <NUM> being a motion-compensatedly predicted coding unit or a disparity-compensatedly coding unit performing the residual prediction using the determined subset of reference pictures. For instance, <FIG> shows current picture P, first reference picture A, second reference picture V and third reference picture VA of picture P and illustrates a current coding unit <NUM> which is assumed to be a motion-compensatedly predicted, i.e. temporally predicted, coding unit. That is, the data stream indicates that the coding unit <NUM> is to be temporally predicted and signals motion data such as one or more motion vectors <NUM> using which the coding unit <NUM>, i.e. the picture block covered by coding unit <NUM>, is to be predicted from the corresponding portion <NUM> pointed to by motion information <NUM> and lying within reference picture A. The details on whether or not residual prediction is applied to coding unit <NUM> are set out in more detail below. If, however, coding unit <NUM> is to be subject to residual prediction, video decoding core <NUM> may act as follows. Using disparity information <NUM> which the video decoding core <NUM> may derive from depth maps coded into the data stream <NUM> or by a respective additional signaling in the data stream or from coding units temporally or spatially adjacent to the current coding unit , and optionally by use of motion information such as motion information <NUM> or motion information conveyed for picture V, the video decoding core <NUM> locates corresponding portions <NUM> and <NUM> in reference pictures VA and V and forms a difference <NUM> of these portions <NUM> and <NUM> and uses a sum <NUM> of the first stage prediction signal <NUM> obtained by using of motion information <NUM> from portion <NUM> on the one hand and the difference signal <NUM> on the other hand as the final prediction <NUM> of coding unit <NUM>. Video decoding core <NUM> may add to this prediction signal <NUM> a residual signal <NUM> by deriving the residual signal <NUM> from the data stream using residual decoding <NUM>, which may, for instance, involve entropy decoding or transform coefficient levels, the quantization of the transform coefficient levels and spectral-to-space-transforming the dequantized transform coefficients. The result of adding the final prediction signal <NUM> and the residual signal <NUM> may be used to reconstruct the content of coding unit <NUM>.

<FIG> illustrates the case that the coding unit <NUM> is a disparity-compensated, i.e. inter-view predicted, coding unit for which the data stream <NUM> signals that this coding unit is inter-layer predicted along with motion information <NUM> which, however, now represents disparity information such as one or more vectors pointing to the corresponding portion <NUM> which, however, now lies within reference picture V. In this situation, the video decoding core <NUM> forms the difference signal <NUM> from corresponding portions <NUM> and <NUM> now lying, however, in pictures A and VA at positions located, for instance, by the video decoding core <NUM> by use of temporal information or motion vectors such as temporal information or motion vectors conveyed in the data stream for reference picture V and optionally disparity information such as disparity information <NUM> or disparity information otherwise derived from the data stream to account for the relative disparity shift between portions <NUM> and <NUM>. The sum of the first stage prediction signal <NUM> obtained by copying from portion <NUM> on the one hand and the difference signal <NUM> on the other hand is, again, used as the final prediction signal <NUM> which is then used to reconstruct the portion of picture P covered by coding unit <NUM> with or without adding the residual signal <NUM> to prediction signal <NUM>.

It should be noted that <FIG> concentrated on the case that residual prediction is used for a temporally predicted picture P. If not being used, however, the reference picture within the same view/layer the current picture may be different than the one depicted in <FIG>, i.e. different than the one serving as picture A in case of residual prediction being applied. A reference index may be signaled for coding unit <NUM>, which selects one out of a set of temporal reference pictures of the current picture P. Such set of reference pictures is mentioned later with respect to the layer/view of picture V with respect to reference sign <NUM>. Reference picture A, i.e. the one used in case of residual prediction being applied, may by the "temporally nearest" picture and may be selected inherently/automatically if residual prediction applies. Picture V may be chosen for picture P by explicit signaling such as by explicit appointment an associated reference picture V for each available temporal reference picture or inherently. In case of <FIG>, i.e. in case of an inter-layer predicted picture P, picture V may be signaled be a reference index and picture A may be again the "temporally nearest" out of the available set of reference pictures.

That is, <FIG> and <FIG> represent an example where the decoding core <NUM> used reference pictures VA and V in order to predict the prediction residual of the temporal first stage prediction <NUM> in case of the coding unit <NUM> being temporally predicted, and reference pictures A and VA in order to predict the prediction residual of the inter-view prediction signal <NUM> in case of the coding unit <NUM> being inter-view predicted. In both cases, all pictures A, V and VA are involved in the reconstruction process, but it should be noted that the embodiment of <FIG> and <FIG> is merely an example and modifications are feasible.

As explained in more detail below, not all of the motion- or disparity-compensatedly predicted coding units are subject to residual prediction by the video decoding core <NUM>. Rather, the decision whether or not a current coding unit of the motion- or disparity-compensatedly predicted type is subject to residual prediction depends, at least partially, on a result of a check performed by the residual prediction switch <NUM> which is described in more detail below. "Partially" means that the data stream <NUM> may, for instance, convey a signalization signaling for the current coding unit <NUM> whether or not same is to be subject to residual prediction. However, such signalization may be contained within the data stream <NUM> merely in case of the residual prediction switch enabling residual prediction for this coding unit. In accordance with this alternative, the video decoding core <NUM> would already be responsive to the residual prediction switch with respect to the parsing procedure, i.e. in parsing the data stream <NUM>. Only in case of the enablement of residual prediction for the current coding unit <NUM> would the video decoding core <NUM> read a signalization from the data stream signaling whether or not residual prediction is to be applied to the current coding unit. According to an alternative, such signalization for the coding unit <NUM> could be left off with performing residual prediction by decoding core <NUM> whenever residual prediction is enabled by switch <NUM>, or could be inevitably signaled in the data stream for the coding unit <NUM> with the switch <NUM>, however, modifying the residual prediction scheme so as to not rely on reference picture VA. For instance, instead of using the residual prediction scheme described above with respect to <FIG> and <FIG>, the decoding core <NUM> could, if disablement is signaled by a switch <NUM>, perform a residual prediction by way of, for instance, spatial prediction of the first stage prediction signal <NUM> or by some other means. In any case, the video decoding core <NUM> applies or does not apply, depending on the residual prediction switch <NUM>, the residual prediction, at least as far as the third reference picture is concerned, for the current coding unit.

In order to decide on the enablement or disablement of the residual prediction for current coding unit <NUM>, residual prediction switch <NUM> is configured to evaluate a certain parameter set contained in the data stream as described with respect to <FIG>.

In particular, as illustrated in <FIG>, the residual prediction switch <NUM> reads from the data stream a parameter set <NUM>. This parameter set <NUM> relates to reference picture V, which is assumed to have view index Vref. The residual prediction switch <NUM> derives <NUM> therefrom a set <NUM> of all pictures that must be available for prediction/reconstruction of reference picture V and are of the reference view Vref. That is, set <NUM> contains all pictures of view Vref which must be in the decoded picture buffer of the video decoding core <NUM> at the time of decoding picture V. <FIG> illustrates the decoded picture buffer at reference sign <NUM>. The video decoding core <NUM> empties, for example, and accordingly refills, the decoded picture buffer <NUM> on a timestamp-by-timestamp basis, i.e. each time having decoded pictures belonging to one timestamp. In other words, video decoding core <NUM> may maintain all pictures within the decoded picture buffer <NUM> until the finalization of decoding of the pictures belonging to the current timestamp so that the pictures belonging to the set <NUM> are required and guaranteed to be also present in the decoded picture buffer <NUM> at the time of decoding the current picture of view Vcurr. As a side note it should be mentioned that set <NUM> may be a super set relative to a set of reference pictures of reference view Vref which may actually referred to by corresponding indices in the CUs or prediction units of picture V. In HEVC notation, set <NUM> would be the union of RefPicSetLtCurr, RefPicSetStCurrBefore, RefPicSetStCurrAfter. A set of reference pictures of the reference picture V that are not of the reference view Vref would be, in HEVC notation, the union of RefPicSetlnterLayer1, and RefPicSetInterLayer0. The subset of reference pictures, thus the subset of pictures in a union of RefPicSetLtCurr, RefPicSetStCurrBefore, RefPicSetStCurrAfter, RefPicSetlnterLayer1, and RefPicSetlnterLayer0, actually referred to by the reference indices of motion-compensated and disparity-compensated prediction blocks of CUs within picture V would be RefPicList (per hypothesis if the current picture is a multi-hypothesis predicted picture) and is derived from the union using parameters of the slice header. The pictures in subset of reference pictures being RefPicList may actually be referred to by prediction units of the reference picture V, when predicting the reference picture V.

As described above, in connection with 3D-HEVC, the parameter set <NUM> inspected by residual prediction switch <NUM> may, for instance, be scattered in the data stream <NUM> across portions of the data stream <NUM> of different "scope", such as the video picture set VPS and the slice header of the current picture. Set <NUM> may be the union of RefPicSetStCurrBefore, RefPicSetStCurrAfter and RefPicSetLtCurr. These sets contain all reference pictures that may be used for inter-prediction of the current picture and one or more pictures that follow the current picture in decoding order. As explained above, the multi-view video decoder <NUM> may be configured to read from the data stream <NUM> another parameter set <NUM> which also relates to reference picture V in order to derive therefrom a further set <NUM> of reference pictures of the reference view Vref. The latter set <NUM>, however, is a facultative set of reference pictures. They are to be maintained in the decoded picture buffer <NUM> merely in case of the multi-view video decoder currently comprises any of these pictures, namely for use in prediction of the pictures of the reference view following reference picture V in decoding order. In other words, set <NUM> consists of all reference pictures that are not used for inter-prediction of picture V, but may be used in inter-prediction for one or more pictures of view Vref that follow picture V in decoding order. Accordingly, sets <NUM> and <NUM> are distinct, i.e. do not overlap or do not have any picture in common. The check <NUM> whether set <NUM> comprises picture VA, namely the picture of the reference view Vref coinciding in timestamp Pref with reference picture A as performed by residual prediction switch <NUM> so as to enable <NUM> or disable <NUM> the residual prediction is independent from set <NUM>. The check <NUM> is also independent from the third reference picture's VA actual presence in the decoded picture buffer <NUM>, thereby avoiding the need for residual prediction switch <NUM> to check this presence, which check would, as described above, suffer from the inability to distinguish between an absence of reference picture VA due to transmission loss and by intention of the video encoder having generated bitstream <NUM>.

As already described above, the multi-view video decoder could, in order to obtain Pref, i.e. could identify reference picture A by inspecting, i.e. reading and deriving, a corresponding parameter set in the data stream <NUM> for current picture P with determining that picture in the corresponding set of reference pictures of current picture P to be the reference picture A which is temporally nearest to the current picture P, i.e. the picture within the set of reference pictures that must be available for prediction/reconstruction of the current picture P and are of the same view as picture P with a timestamp nearest to the timestamp of picture P.

An example for a responsiveness of the video decoding core <NUM> to the residual prediction switch <NUM> is illustrated in <FIG> shows an example where, in addition to the dependencies outlined above, the decision whether residual prediction is applied to a certain coding unit additionally depends on a flag <NUM> in the data stream <NUM> which generally enables or disables, or switches on or off, the residual prediction. The scope of flag <NUM> may, for instance, be the whole video, an access unit, i.e. or pictures belonging to one timestamp, or the current picture. Flag <NUM> and the corresponding check <NUM> shown in <FIG> are optional. That is, multi-view video decoder <NUM> would check <NUM> whether residual prediction is enabled anyway on the basis of flag <NUM>. It should be noted that the above example using the for next loop "range of <NUM> to num_ref_idx_IX_active_minus1" rendered clear that the enablement/disablement may alternatively be signaled by syntax elements such as syntax elements signaling the non-availability of any temporal reference picture for picture the current picture P anyway.

Multi-view video decoder <NUM> would also check <NUM> whether residual prediction switch <NUM> enabled or disabled the residual prediction in steps <NUM>/<NUM> of <FIG>. Merely of both checks <NUM> and <NUM> confirm the enablement of the residual prediction, the multi-view video decoder <NUM>, or the video decoding core <NUM>, reads a signalization <NUM> for the current coding unit from the data stream <NUM> which specifically signalizes for the current coding unit whether or not residual prediction is to be applied for the current coding unit. Subsequent to the reading <NUM>, the video decoding core <NUM> checks the signalization <NUM> read in step <NUM>, namely in step <NUM>, so as decode the current coding unit using residual prediction in <NUM> in case of the signalization confirming the application of the residual prediction, and decode the current coding unit in steps <NUM> without residual prediction in case of the signalization read in steps <NUM> signaling that the residual prediction is not to be applied although being enabled. The decoding of the current coding unit using residual prediction in step <NUM> may be performed as described above with respect to <FIG> and <FIG>, wherein the decoding according to step <NUM>, i.e. without residual prediction, may coincide with this procedure except for the fact that the difference signal <NUM> is not determined so that the first stage prediction signal <NUM> becomes the final prediction signal <NUM>.

If any of checks <NUM>, <NUM> and <NUM> result in the residual prediction not being enabled or not to be applied for the current coding unit, then the current coding unit is coded without residual prediction. In case of no-enablement being the result of any of checks <NUM> and <NUM>, the signalization is not even read, i.e. the reading is skipped.

It should be noted that owing to the fact that set <NUM> collects all pictures which are obliged to be present in the decoded picture buffer <NUM> of decoder <NUM>, the multi-view video decoder <NUM> detects that reference picture VA is lost if picture VA is included in this set <NUM>, but absent in the decoded picture buffer <NUM>. If so detected, multi-view video decoder may, for instance, initiate or enter an error resilience mode. For example, the multi-view video decoder may resume decoding the bitstream from the next intra-coded picture onwards, or may use a substitute picture as a substitution of a reference picture VA.

Thus, the above description revealed, in other words, a multi-view video decoder comprising a video decoding core, e.g. 3D-HEVC core, <NUM> configured to decode, in units of coding units of pictures, a plurality of views Vi from a data stream <NUM>, the video decoding core <NUM> supporting motion- or disparity-compensated residual prediction of a predetermined motion- or disparity-compensatedly predicted coding unit of a current picture of a current view on the basis of a first reference picture A of the current view, a second reference picture V of a reference view, coinciding in timestamp with the current picture, and a third reference picture VA of the reference view, coinciding in timestamp with the first reference picture A. For example, a disparity compensated prediction of CU of current picture from second reference picture using disparity vector v is followed by predicting the residual of this prediction by applying a disparity vector corresponding to v onto a collocated (to CU) portion of the first reference picture so as to predict the same by disparity-compensated prediction from the third reference picture; merely the then remaining residual might by entropy coded for CU into data stream. Or motion compensated prediction of CU of current picture from first reference picture using motion vector v is followed by predicting the residual of this prediction by applying a motion vector corresponding to v onto a collocated (to CU) portion of the second reference picture so as to predict the same by motion-compensated prediction from the third reference picture; merely the then remaining residual might by entropy coded for CU into data stream.

Further, the multi-view video decoder may comprise a disparity-compensated residual prediction switch configured to read a parameter set, e.g. slice header, from the data stream, the parameter set relating to the second reference picture, and derive therefrom the set of all reference pictures that can be used for prediction of the second reference picture and are of the reference view, wherein this set is, e.g. the union of RefPicSetStCurrBefore, RefPicSetStCurrAfter, RefPicSetLtCurr in HEVC, i.e. union of pictures of the layer of the second picture that are required to be present when decoding the second reference picture. This set is, thus, required to be present (in the DPB) when decoding the second reference picture and is also present at the time of decoding the current picture as same belong to the same timestamp or AU and the DPB is emptied AU wise and owing to the coding/decoding order which traverses the views of one timestamp first before proceeding with the next timestamp in coding/decoding order. The disparity-compensated residual prediction switch may further be configured to check whether the third reference picture is included in the set of reference pictures; depending on the check, if the third reference picture is included in the set of reference pictures, enable the disparity-compensated residual prediction for the current coding unit; depending on the check, if the third reference picture is not included in the set of reference pictures, disable the disparity-compensated residual prediction for the current coding unit.

The video decoding core is configured to be responsive to the disparity-compensated residual prediction switch in order to apply or not apply disparity-compensated residual prediction based on the third reference picture for the current coding unit. Please note: in another list the CU could use a further alternative "reference picture", so that residual prediction may actually be used for the current CU. (Die CU könnte in einer anderen Liste noch ein alternatives "reference picture" verwenden, so dass doch residual prediciton für die aktuelle CU verwendet wird).

The residual is predicted by combination of sample values of the first or second picture with samples values of the third picture.

According to above, where residual prediction can be additionally disabled by other flags.

Both reference picture lists could be processed. Here, the timestamp of the first reference picture could differ for the first and the second list.

Note that the third reference pictures indicated to be available by flags are required to be present in the DPB.

The residual is predicted by combination of sample values of the first or second picture with samples values of the third picture, when enabled.

Only temporal or inter-layer prediction might be applied, when residual prediction is disabled.

If so, residual prediction can be additionally disabled by other flags or by conditions in the decoding process.

<FIG> and <FIG> show examples of how a possible extension of a CU syntax and SPS syntax of 3D-HEVC could look in order to form an intensification of the embodiments described above with respect to <FIG>, wherein reference signs and hints to reference signs used in <FIG> are added in <FIG> and <FIG> to show the concordance between syntax elements shown in <FIG> and <FIG> and the syntax elements and parameter sets denoted in <FIG>, and illustrates the possibility that the signalization <NUM> could be dependent on the enablement/disablement, but as already noted above, this dependency may also be left off for, for example, parsing robustness reasons so as to be, for example, merely dependently present in the data stream dependent on the also optional flag <NUM>.

For the sake of completeness, <FIG> shows a multi-view video encoder <NUM> which could fit to the multi-video decoder of <FIG> in that the video encoder <NUM> of <FIG> is able to generate a data stream <NUM> decodable by decoder <NUM> of <FIG>. The multi-view encoder <NUM> of <FIG> comprises a video encoding core <NUM> and a residual prediction switch <NUM> which may or may not be included within the video encoding core <NUM>. The functionality of the multi-view video encoder <NUM> and the elements <NUM> and <NUM> substantially mirrors the functionality of the multi-view video decoder <NUM> and its internal elements. That is, the video encoding core <NUM> is configured to encode the plurality of views Vi into the data stream <NUM> in a manner supporting residual prediction of motion- or disparity-compensatedly predicted coding units on the basis of reference pictures A, V and VA. The residual prediction switch <NUM> is configured to insert parameter set <NUM> into data stream <NUM> and depending on reference picture VA being included in set <NUM> enables or disables residual prediction for a current coding unit. The video encoding core applies or does not apply, depending on the residual prediction switch <NUM>, residual prediction based on reference picture VA, or at least with respect to reference picture VA, for the current coding unit.

The residual prediction switch may be configured to perform the enabling and disabling depending on the check in a manner independent from the third reference being present or absent in a decoded picture buffer of the multi-view video encoder.

The video encoding core may be configured to determine a subset of reference pictures out the first, second and third reference pictures depending on the predetermined motion- or disparity-compensatedly predicted coding unit being a motion-compensatedly predicted coding unit or a disparity-compensatedly predicted coding unit and perform, if enabled by the residual prediction switch, the residual prediction of the predetermined motion- or disparity-compensatedly predicted coding unit using the determined subset of reference pictures.

The video encoding core may be configured to perform, if enabled by the residual prediction switch, the residual prediction of the predetermined motion- or disparity-compensatedly predicted coding unit using the second and third reference pictures if the predetermined motion- or disparity-compensatedly predicted coding unit is a motion-compensatedly predicted coding unit, and the first and third reference pictures if the predetermined motion- or disparity-compensatedly predicted coding unit is a disparity-compensatedly predicted coding unit.

The video encoding core may further be configured to predict the predetermined motion- or disparity-compensatedly predicted coding unit using the first reference picture if the predetermined motion- or disparity-compensatedly predicted coding unit is a motion-compensatedly predicted coding unit, and the second reference picture if the predetermined motion- or disparity-compensatedly predicted coding unit is a disparity-compensatedly predicted coding unit, and apply, if enabled by the residual prediction switch, the residual prediction of the predetermined motion- or disparity-compensatedly predicted coding unit to a prediction residual of the prediction.

The video encoding core may be configured to if the residual prediction of the predetermined motion- or disparity-compensatedly predicted coding unit is enabled by the residual prediction switch, insert a signalization into the data stream signaling whether the residual prediction is to be applied to the predetermined motion- or disparity-compensatedly predicted coding unit or not, if the residual prediction of the predetermined motion- or disparity-compensatedly predicted coding unit is disabled by the residual prediction switch, skip inserting the signalization into the data stream, and apply the residual prediction to the predetermined motion- or disparity-compensatedly predicted coding unit if the residual prediction of the predetermined motion- or disparity-compensatedly predicted coding unit is enabled by the residual prediction switch, and the signalization signals that the residual prediction is to be applied to the predetermined motion- or disparity-compensatedly predicted coding unit.

The multi-view video encoder may be further configured to disable, irrespective of the residual prediction switch, the residual prediction of a predetermined motion- or disparity-compensatedly predicted coding unit of the current picture of the current view responsive to a flag in the data stream.

The residual prediction switch may be configured to enable and disable the residual prediction depending on the check with respect to the first, second and third reference pictures so that, depending on the check, a prediction residual of a first stage motion- or disparity-compensated prediction of the predetermined motion- or disparity-compensatedly predicted coding unit is coded into the data stream un-predictively or predictively.

The residual prediction switch configured to perform the inserting, checking, enabling and disabling per hypothesis if the current picture is multi-hypothesis predicted picture.

The description of <FIG> related to the first aspect mentioned above when illustrating the problems involved in the current design of 3D-HEVC and possible ways to overcome same. The following embodiments relate to the second aspect mentioned above, according to which additional flags are spent in order to signal to the decoder the availability or non-availability of potential "third reference pictures" for all potential "V pictures".

In accordance with the second aspect, the multi-view video decoder largely coincides with the description brought forward above with respect to <FIG>. The same applies to the multi-view video encoder. However, as shown in <FIG>, the enablement/disablement of residual prediction is, at least as far as the third reference picture VA is concerned, determined for a predetermined coding unit of a current picture differently. Here, the multi-view video encoder encodes into the data stream a first parameter set <NUM> which relates to the current picture P. VPS und slice header may be inspected to collect the first parameter set, for example. On the basis of this parameter set <NUM>, the multi-view video decoder or residual prediction switch <NUM> derives the set <NUM> of reference pictures included in views different from the current view and potentially serving as a second reference picture V for coding units of the current picture, i.e. the set <NUM> of pictures of reference layers relative to the layer of the current picture P. The first parameter set <NUM> also allows deriving therefrom the timestamp of the reference picture A of the current picture P. For all of these pictures in set <NUM> the multi-view video encoder inserts into the data stream <NUM>, and the multi-view video decoder reads therefrom, a flag <NUM> indicating whether the layer of the respective picture of the same timestamp as reference picture A which is available as third reference picture VA for residual prediction. The flags are comprised by, or together form, a second parameter set <NUM>. This parameter set might be included in the slice header of the current picture's slice(s). The "availability" of pictures of corresponding pictures <NUM> is thus determined at the decoding side by the residual prediction switch <NUM> on the basis of flags <NUM> only, thereby lowering the computational efforts necessary at the decoding side. The multi-view encoder or its residual prediction switch determines the availability of the pictures in set <NUM> as indicated by the flags <NUM> in data stream <NUM> in a manner such that availability indicated by a flag <NUM> for a picture of set <NUM> only or exactly then if this picture belongs to the set of potential temporal reference pictures of the corresponding picture of the same layer/view in set <NUM>, wherein the set had been denoted <NUM> with respect to one particular V reference picture of view/layer Vref. That is, in accordance with <FIG>, the availability is checked at the encoding side for each potential VA reference picture in set <NUM> temporally aligned to reference picture A and view-aligned to a corresponding picture of set <NUM> with the result being signaled in the data stream using flag <NUM>.

Thus, the residual prediction switch <NUM> of the multi-view video decoder simply needs to inspect the second parameter set <NUM> in the data stream <NUM> which includes flags <NUM> in order to determine for each coding unit of current picture P whether residual prediction is enabled or disabled for the respective coding unit. Having thus determined the enablement/disablement in check <NUM> the remaining functionality of the multi-view video decoder and its core <NUM> and switch <NUM> would be the same as shown in <FIG>. That is, the application/non-application signalization could be signaled within the data stream by the multi-view video decoder's switch and read by the switch <NUM> of the multi-view video decoder dependent on this enablement/disablement and a general enablement/disablement signalization <NUM> could be used as well.

Thus, with respect to <FIG>, there has been described, in other words, a multi-view video decoder <NUM> comprising a video decoding core, e.g. 3D-HEVC, <NUM> configured to decode, in units of coding units of pictures, a plurality of views from a data stream <NUM>, the video decoding core <NUM> supporting motion or disparity-compensated residual prediction of a predetermined motion or disparity-compensatedly predicted coding unit <NUM> of a current picture P of a current view on the basis of a first reference picture A of the current view, a second reference picture V of a reference view, coinciding in timestamp with the current picture, and a third reference picture VA of the reference view, coinciding in timestamp with the first reference picture. A disparity compensated prediction of CU of current picture from second reference picture using disparity vector v is followed by predicting the residual of this prediction by applying a disparity vector corresponding to v onto a - with taking motion compensation by use of a motion vector into account - collocated (to CU) portion of the first reference picture so as to predict the same by disparity-compensated prediction from the third reference picture; merely the then remaining residual might by entropy coded for CU into data stream.

Decoder <NUM> further comprises a disparity-compensated residual prediction switch <NUM> configured to read a parameter set <NUM>, contained e.g. at least partially in the slice header, from the data stream, the parameter set relating to the current picture, and derive therefrom a first set <NUM> of reference pictures included in a view different from the current view and potentially serving as the second reference picture for coding units of the current picture and a reference timestamp of reference pictures which potentially serve as the first reference picture for coding units of the current picture; for each of a second set <NUM> of pictures comprising one picture for each reference picture of the first set which coincides in view with the respective reference picture of the first set of reference pictures, and is of the reference timestamp, read a flag from a picture scope or finer scope parameter set <NUM>, e.g. in a slice header, within the data stream, the picture scope or finer scope parameter set relating to a portion, e.g. slice, of the current picture, the flag indicating whether the respective picture of the second set of pictures is available as third picture for motion- or disparity-compensated residual prediction. Note that not necessarily the whole set for which both coincidences are valid: merely those pictures of the greatest possible set which belong to the timestamp nearest to the timestamp of the current picture may be included in the second set. The flag could be ref_rp_ref_avail_flag.

The video decoding core is configured to be responsive to the flag related to a respective picture of the second set of pictures so as to apply or not apply motion or disparity-compensated residual prediction for a current coding unit comprised by the portion using the respective picture of the second set of pictures. Imagine the current CU uses disparity compensated prediction with respect to the second reference picture: if the third reference picture is indicated to be available, disparity-compensated residual prediction may be used for the residual prediction.

It should be noted that whenever prediction processes are described with respect to one of encoder or decoder, same processes are correspondingly likewise performed at the other side (decoder or encoder) so as to make sure that the predictions used are the same at encoder and decoder side, so that a description or claim concerning these processes with respect to one of decoder and encoder shall concurrently be interpreted as a corresponding basis for the other side.

The inventive encoded data stream or signal can be stored on a digital storage medium or can be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet. Where ever the insertion or encoding of some information into a data stream has been described, this description is concurrently to be understood as a disclosure that the resulting data stream comprises the respective information, syntax element of flag or so forth.

in some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein.

Claim 1:
Multi-view video decoder compliant to HEVC comprising
a video decoding core (<NUM>) configured to decode, in units of coding units, a plurality of views (V1 ...V3) from a data stream (<NUM>), the video decoding core supporting motion- or disparity-compensated residual prediction of a predetermined motion- or disparity-compensatedly predicted coding unit of a current picture (<NUM>) of a current view on the basis of a first reference picture (A) of the current view, a second reference picture (V) of a reference view, coinciding in timestamp with the current picture, and a third reference picture (VA) of the reference view, coinciding in timestamp with the first reference picture (A), and
a disparity-compensated residual prediction switch (<NUM>) configured to
read a parameter set (<NUM>) from the data stream, the parameter set (<NUM>) relating to the second reference picture, and derive therefrom the set (<NUM>) of all reference pictures of the reference view which are required, in terms of a bitstream constraint of HEVC, to be present in a decoded picture buffer of the video decoding core (<NUM>) when decoding the second reference picture (V);
check whether the third reference picture (VA) is included in the set (<NUM>) of reference pictures;
depending on the check, if the third reference picture (VA) is included in the set (<NUM>) of reference pictures, enable the disparity-compensated residual prediction for the predetermined motion- or disparity-compensatedly predicted coding unit with respect to third reference picture; and
depending on the check, if the third reference picture is not included in the set of reference pictures, disable the disparity-compensated residual prediction for the predetermined motion- or disparity-compensatedly predicted coding unit with respect to third reference picture,
wherein the video decoding core (<NUM>) is configured to, depending on the residual prediction switch, apply or not apply disparity-compensated residual prediction based on the third reference picture for the predetermined motion- or disparity-compensatedly predicted coding unit.