Patent Description:
A current project of the Joint Video Team (JVT) of the ISO/IEC Moving Pictures Experts Group (MPEG) and the ITU-T Video Coding Experts Group (VCEG) is the development of a scalable extension of the state-of-the-art video coding standard H. <NUM>/MPEG4-AVC defined in <NPL>and <NPL>, supports temporal, spatial and SNR scalable coding of video sequences or any combination thereof.

<NUM>/MPEG4-AVC as described in ITU-T Rec. & ISO/IEC <NUM>-<NUM> AVC, "Advanced Video Coding for Generic Audiovisual Services, " version <NUM>, <NUM>, specifies a hybrid video codec in which macroblock prediction signals are either generated in the temporal domain by motion-compensated prediction, or in the spatial domain by intra prediction, and both predictions are followed by residual coding. <NUM>/MPEG4-AVC coding without the scalability extension is referred to as single-layer H. <NUM>/MPEG4-AVC coding. Rate-distortion performance comparable to single-layer H. <NUM>/MPEG4-AVC means that the same visual reproduction quality is typically achieved at <NUM>% bit-rate. Given the above, scalability is considered as a functionality for removal of parts of the bit-stream while achieving an R-D performance at any supported spatial, temporal or SNR resolution that is comparable to single-layer H. <NUM>/MPEG4-AVC coding at that particular resolution.

The basic design of the scalable video coding (SVC) can be classified as a layered video codec. In each layer, the basic concepts of motion-compensated prediction and intra prediction are employed as in H. <NUM>/MPEG4-AVC. However, additional inter-layer prediction mechanisms have been integrated in order to exploit the redundancy between several spatial or SNR layers. SNR scalability is basically achieved by residual quantization, while for spatial scalability, a combination of motion-compensated prediction and oversampled pyramid decomposition is employed. The temporal scalability approach of H. <NUM>/MPEG4-AVC is maintained.

In general, the coder structure depends on the scalability space that is required by an application. For illustration, <FIG> shows a typical coder structure <NUM> with two spatial layers 902a, 902b. In each layer, an independent hierarchical motion-compensated prediction structure 904a,b with layer-specific motion parameters 906a, b is employed. The redundancy between consecutive layers 902a,b is exploited by inter-layer prediction concepts <NUM> that include prediction mechanisms for motion parameters 906a,b as well as texture data 910a,b. A base representation 912a,b of the input pictures 914a,b of each layer 902a,b is obtained by transform coding 916a,b similar to that of H. <NUM>/MPEG4-AVC, the corresponding NAL units (NAL - Network Abstraction Layer) contain motion information and texture data; the NAL units of the base representation of the lowest layer, i.e. 912a, are compatible with single-layer H. <NUM>/MPEG4-AVC. The reconstruction quality of the base representations can be improved by an additional coding 918a,b of so-called progressive refinement slices; the corresponding NAL units can be arbitrarily truncated in order to support fine granular quality scalability (FGS) or flexible bit-rate adaptation.

The resulting bit-streams output by the base layer coding 916a,b and the progressive SNR refinement texture coding 918a,b of the respective layers 902a,b, respectively, are multiplexed by a multiplexer <NUM> in order to result in the scalable bit-stream <NUM>. This bit-stream <NUM> is scalable in time, space and SNR quality.

Summarizing, in accordance with the above scalable extension of the Video Coding Standard H. <NUM>/MPEG4-AVC, the temporal scalability is provided by using a hierarchical prediction structure. For this hierarchical prediction structure, the one of single-layer H. <NUM>/MPEG4-AVC standards may be used without any changes. For spatial and SNR scalability, additional tools have to be added to the single-layer H. <NUM>/MPEG4. All three scalability types can be combined in order to generate a bit-stream that supports a large degree on combined scalability.

For SNR scalability, coarse-grain scalability (CGS) and fine-granular scalability (FGS) are distinguished. With CGS, only selected SNR scalability layers are supported and the coding efficiency is optimized for coarse rate graduations as factor <NUM>-<NUM> from one layer to the next. FGS enables the truncation of NAL units at any arbitrary and eventually byte-aligned point. NAL units represent bit packets, which are serially aligned in order to represent the scalable bit-stream <NUM> output by multiplexer <NUM>.

In order to support fine-granular SNR scalability, so-called progressive refinement (PR) slices have been introduced. Progressive refinement slices contain refinement information for refining the reconstruction quality available for that slice from the base layer bit-stream 912a,b, respectively. Even more precise, each NAL unit for a PR slice represents a refinement signal that corresponds to a bisection of a quantization step size (QP decrease of <NUM>). These signals are represented in a way that only a single inverse transform has to be performed for each transform block at the decoder side. In other words, the refinement signal represented by a PR NAL unit refines the transformation coefficients of transform blocks into which a current picture of the video has been separated. At the decoder side, this refinement signal may be used to refine the transformation coefficients within the base layer bit-stream before performing the inverse transform in order to reconstruct the texture of prediction residual used for reconstructing the actual picture by use of a spatial and/or temporal prediction, such as by means of motion compensation.

The progressive refinement NAL units can be truncated at any arbitrary point, so that the quality of the SNR base layer can be improved in a fine granular way. Therefore, the coding order of transform coefficient levels has been modified. Instead of scanning the transform coefficients macroblock-by-macroblock, as it is done in (normal) slices, the transform coefficient blocks are scanned in separate paths and in each path, only a few coding symbols for a transform coefficient block are coded. With the exception of the modified coding order, the CABAC entropy coding as specified in H. <NUM>/MPEG4-AVC is re-used.

The single-layer H. <NUM>/MPEG4-AVC coding standard has been developed for the use of a fixed sampling structure among possible chroma sampling structures, such as, for example, <NUM>:<NUM>:<NUM> and <NUM>:<NUM>:<NUM>, respectively. The different chroma sampling capabilities are included in different profiles of the standard. In this regard, reference is made to <NPL>. In <NUM>:<NUM>:<NUM>, for example, the chroma or coloring sampling content indicating the extent to which the color deviates from gray and being defined by two chroma components amounts to, regarding the sampling points, merely one fourth of the number of samples of the luma content representing brightness and being defined by one luma component. In other words, the number of luma component samples in both the horizontal and vertical dimensions is half the number of luma samples. The coding precision used per sample is fixed to be <NUM> bits or <NUM> bits, depending on the profile of the standard used. Again, reference is made to the just mentioned article. For sake of completeness, it is noted that the term luma, according to the standard, actually means a weighted sum of non-linear or gamma-corrected RGB contributions. However, according to another view, luma may be viewed as luminance which refers to the linear relationship of the RGB contributions. According to the present application, both views shall equally apply.

In general, the term chroma sampling format refers to the number and position of the chroma samples relative to the number and position of the corresponding luma samples. Three examples of possible sampling formats are described now. As has already been described, according to the <NUM>:<NUM>:<NUM> sampling, the chroma signal has half the horizontal and half the vertical resolution as compared to the luma signal. The format is illustrated in <FIG>, where the crosses indicate the locations of the luma samples, whereas the circles represent the locations of the chroma samples, where each chroma sample may consist of two chroma components, such as Cb and Cr. Another sampling format is <NUM>:<NUM>:<NUM>, where the chroma signal has half the horizontal and the same vertical resolution as the luma signal. This is shown in <FIG>. According to a <NUM>:<NUM>:<NUM> chroma sampling format, the chroma signal has the same horizontal and vertical resolution as the luma signal or content, respectively. This is illustrated in <FIG>.

<CIT> describes a method of coding a quality scalable video sequence, where an N-bit input frame is converted to an M-bit input frame, where M is an integer between <NUM> and N, the M-bit input frame would be encoded to produce a base-layer output bitstream, an M-bit output frame would be reconstructed from the base-layer output bitstream and converted to a N-bit output frame, and the N-bit output frame would be compared to the N-bit input frame to derive an N-bit image residual that could be encoded to produce an enhancement layer bitstream.

Problems arise when a color video source signal has a different dynamic range and/or a different chroma sampling format than required by the decoder or player, respectively. In the above current SVC working draft, the scalability tools are only specified for the case that both the base layer and enhancement layer represent a given video source with the same bit depth of the corresponding arrays of luma and chroma samples, and in addition with the assumption that the chroma sampling relative to the luma sampling, i.e., the chroma sampling format, is fixed for base and enhancement layer(s). Hence, considering different decoders and players, respectively, requiring different bit depths and chroma sampling formats, several coding streams dedicated for each of the bit depths and chroma sampling format requirements would have to be provided separately. However, in rate/distortion sense, this means an increased overhead and reduced efficiency, respectively.

Thus, it would be desirable to provide a coding scheme that overcomes this deficiency.

It is the object of the present invention to provide a coding scheme that enables a more efficient way of providing a coding of a picture or video being suitable for different bit depths or different bit depths and chroma sampling format requirements.

This object is achieved by an encoder according to claim <NUM>, a decoder according to claim <NUM>, and a method according to claim <NUM> or <NUM>.

The present invention is based on the finding that a more efficient way of addressing different bit-depths may be achieved when using a low bit-depth for providing a respective base layer data stream representing this low bit-depth as well as for providing a higher bit-depth so that a respective prediction residual may be encoded in order to obtain a higher bit-depth. By this measure, an encoder is enabled to store a base-quality representation of a picture or a video sequence, which can be decoded by any legacy decoder or video decoder, together with an enhancement signal for higher bit-depth, which may be ignored by legacy decoders or video decoders. For example, the base quality representation could contain an <NUM>-bit version of the picture or the video sequence, while the high quality enhancement signal contains a "refinement" to a <NUM>-bit version of the same, and the base quality representation could represent a <NUM>:<NUM>:<NUM> version of the picture or the video sequence, while the high quality enhancement signal contains a "refinement" to a <NUM>:<NUM>:<NUM> or <NUM>:<NUM>:<NUM> version of the same.

In the following, preferred embodiments of the present application are described with reference to the Figs. In particular, as it is shown in.

<FIG> shows an encoder <NUM> comprising a base encoding means <NUM>, a prediction means <NUM>, a residual encoding means <NUM> and a combining means <NUM> as well as an input <NUM> and an output <NUM>. The encoder <NUM> of <FIG> is a video encoder receiving a high quality video signal at input <NUM> and outputting a quality-scalable bit stream at output <NUM>. The base encoding means <NUM> encodes the data at input <NUM> into a base encoding data stream representing the content of this video signal at input <NUM> with a reduced picture sample bit-depth or a chroma-sampling format having same luma resolution, but reduced chroma resolution compared to the input signal at input <NUM>. The prediction means <NUM> is adapted to, based on the base encoding data stream output by base encoding means <NUM>, provide a prediction signal with full or increased picture sample bit-depth and/or full or increased chroma resolution for the video signal at input <NUM>. A subtractor <NUM> also comprised by the encoder <NUM> forms a prediction residual of the prediction signal provided by means <NUM> relative to the high quality input signal at input <NUM>, the residual signal being encoded by the residual encoding means <NUM> into a quality enhancement layer data stream. The combining means <NUM> combines the base encoding data stream from the base encoding means <NUM> and the quality enhancement layer data stream output by residual encoding means <NUM> to form a quality scalable data stream <NUM> at the output <NUM>. The quality-scalability means that the data stream at the output <NUM> is composed of a part that is self-contained in that it enables reconstruction of the video signal <NUM> with the reduced bit-depth and/or the chroma sampling format having the reduced chroma resolution without any further information and with neglecting the remainder of the data stream <NUM>, on the one hand and a further part which enables, in combination with the first part, a reconstruction of the video signal at input <NUM> in the original bit-depth and original chroma sampling format being higher than the bit depth and/or chroma resolution of the first part.

After having rather generally described the structure and the functionality of encoder <NUM>, its internal structure is described in more detail below. In particular, the base encoding means <NUM> comprises a down conversion module <NUM>, a subtractor <NUM>, a transform module <NUM> and a quantization module <NUM> serially connected, in the order mentioned, between the input <NUM>, and the combining means <NUM> and the prediction means <NUM>, respectively. The down conversion module <NUM> is for reducing the bit-depth of the picture samples of and/or the chroma resolution of the pictures of the video signal at input <NUM>, with maintaining the luma resolution of the video signal received from the input <NUM>. In other words, the down conversion module <NUM> irreversibly down-converts the high quality input video signal at input <NUM> to a base quality video signal. As will be described in more detail below, this down-conversion may include reducing the bit-depth of the signal samples, i.e. pixel values, in the video signal at input <NUM> using any tone-mapping scheme, such as rounding of the sample values, sub-sampling of the chroma components in case the video signal is given in the form of luma plus chroma components, filtering of the input signal at input <NUM>, such as by a RGB to YCbCr conversion, or any combination thereof. More details on possible prediction mechanisms are presented in the following. In particular, it is possible that the down-conversion module <NUM> uses different down-conversion schemes for each picture of the video signal or picture sequence input at input <NUM> or uses the same scheme for all pictures.

The subtractor <NUM>, the transform module <NUM> and the quantization module <NUM> co-operate to encode the base quality signal output by down-conversion module <NUM> by the use of, for example, a non-scalable video coding scheme, such as H. <NUM>/MPEG4-AVC. According to the example of <FIG>, the subtractor <NUM>, the transform module <NUM> and the quantization module <NUM> co-operate with an optional prediction loop filter <NUM>, a predictor module <NUM>, an inverse transform module <NUM>, and an adder <NUM> commonly comprised by the base encoding means <NUM> and the prediction means <NUM> to form the irrelevance reduction part of a hybrid encoder which encodes the base quality video signal output by down-conversion module <NUM> by motion-compensation based prediction and following compression of the prediction residual. In particular, the subtractor <NUM> subtracts from a current picture or macroblock of the base quality video signal a predicted picture or predicted macroblock portion reconstructed from previously encoded pictures of the base quality video signal by, for example, use of motion compensation. The transform module <NUM> applies a transform on the prediction residual, such as a DCT, FFT or wavelet transform. The transformed residual signal may represent a spectral representation and its transform coefficients are irreversibly quantized in the quantization module <NUM>. The resulting quantized residual signal represents the residual of the base-encoding data stream output by the base-encoding means <NUM>.

Apart from the optional prediction loop filter <NUM> and the predictor module <NUM>, the inverse transform module <NUM>, and the adder <NUM>, the prediction means <NUM> comprises an optional filter for reducing coding artifacts <NUM> and a prediction module <NUM>. The inverse transform module <NUM>, the adder <NUM>, the optional prediction loop filter <NUM> and the predictor module <NUM> co-operate to reconstruct the video signal with a reduced bit-depth and/or the chroma sampling format having reduced chroma resolution, as defined by the down-conversion module <NUM>. In other words, they create a low bit-depth and/or low chroma resolution video signal to the optional filter <NUM> which represents a low quality representation of the source signal at input <NUM> also being reconstructable at decoder side. In particular, the inverse transform module <NUM> and the adder <NUM> are serially connected between the quantization module <NUM> and the optional filter <NUM>, whereas the optional prediction loop filter <NUM> and the prediction module <NUM> are serially connected, in the order mentioned, between an output of the adder <NUM> as well as a further input of the adder <NUM>. The output of the predictor module <NUM> is also connected to an inverting input of the subtractor <NUM>. The optional filter <NUM> is connected between the output of adder <NUM> and the prediction module <NUM>, which, in turn, is connected between the output of optional filter <NUM> and the inverting input of subtractor <NUM>.

The inverse transform module <NUM> inversely transforms the base-encoded residual pictures output by base-encoding means <NUM> to achieve low bit-depth and/or low chroma resolution residual pictures. Accordingly, inverse transform module <NUM> performs an inverse transform being an inversion of the transformation and quantization performed by modules <NUM> and <NUM>. Alternatively, a de-quantization module may be separately provided at the input side of the inverse transform module <NUM>. The adder <NUM> adds a prediction to the reconstructed residual pictures, with the prediction being based on previously reconstructed pictures of the video signal. In particular, the adder <NUM> outputs a reconstructed video signal with a reduced bit-depth and/or reduced chroma resolution chroma-sampling format. These reconstructed pictures are filtered by the loop filer <NUM> for reducing artifacts, for example, and used thereafter by the predictor module <NUM> to predict the picture currently to be reconstructed by means of, for example, motion compensation, from previously reconstructed pictures. The base quality signal thus obtained at the output of adder <NUM> is used by the serial connection of the optional filter <NUM> and prediction module <NUM> to get a prediction of the high quality input signal at input <NUM>, the latter prediction to be used for forming the high quality enhancement signal at the output of the residual encoding means <NUM>. This is described in more detail below.

In particular, the low quality signal obtained from adder <NUM> is optionally filtered by optional filter <NUM> for reducing coding artifacts. Thereafter, the low quality video signal is used by prediction module <NUM> to form a prediction signal for the high quality video signal received at the non-inverting input of adder <NUM> being connected to the input <NUM>. This process of forming the high quality prediction may include multiplying the decoded base quality signal picture samples by a constant factor, i.e. linear scaling, using the respective value of the base quality signal samples for indexing a look-up table which contains the corresponding high quality sample values, using the value of the base quality signal sample for an interpolation process to obtain the corresponding high quality sample value, up-sampling of the chroma components, filtering of the base quality signal by use of, for example, YCbCr to RGB conversion, or any combination thereof. Other examples are described in the following.

In general, for example, the prediction module <NUM> may map the samples of the base quality video signal from a first dynamic range to a second dynamic range being higher than the first dynamic range and/or, by use of a special interpolation filter, interpolate the chroma samples of the base quality video signal to increase the chroma resolution to correspond with the chroma resolution of the video signal at the input <NUM>. In a way similar to the above description of the down-conversion module <NUM>, it is possible to use a different prediction process for different pictures of the base quality video signal sequence as well as using the same prediction process for all the pictures.

The subtractor <NUM> subtracts the high quality prediction received from the prediction module <NUM> from the high quality video signal received from input <NUM> to output a prediction residual signal of high quality, i.e. with the original bit-depth and/or chroma sampling format to the residual encoding means <NUM>. At the residual encoding means <NUM>, the difference between the original high quality input signal and the prediction derived from the decoded base quality signal is encoded exemplarily using a compression coding scheme such as, for example, specified in H. <NUM>/MPEG4-AVC. To this end, the residual encoding means <NUM> of <FIG> comprises exemplarily a transform module <NUM>, a quantization module <NUM> and an entropy coding module <NUM> connected in series between an output of the subtractor <NUM> and the combining means <NUM> in the mentioned order. The transform module <NUM> transforms the residual signal or the pictures thereof, respectively, into a transformation domain or spectral domain, respectively, where the spectral components are quantized by the quantization module <NUM> and with the quantized transform values being entropy coded by the entropy-coding module <NUM>. The result of the entropy coding represents the high quality enhancement layer data stream output by the residual encoding means <NUM>. If modules <NUM> to <NUM> implement an H. <NUM>/MPEG4-AVC coding, which supports transforms with a size of 4x4 or 8x8 samples for coding the luma content, the transform size for transforming the luma component of the residual signal from the subtractor <NUM> in the transform module <NUM> may arbitrarily be chosen for each macroblock and does not necessarily have to be the same as used for coding the base quality signal in the transform module <NUM>. For coding the chroma components, the H. <NUM>/MPEG4-AVC standard, as currently specified, provides no choice. When quantizing the transform coefficients in the quantization module <NUM>, the same quantization scheme as in the H. <NUM>/MPEG4-AVC may be used, which means that the quantizer step-size may be controlled by a quantization parameter QP, which can take values from -<NUM>*(bit depth of high quality video signal component- <NUM>) to <NUM>. The QP used for coding the base quality representation macroblock in the quantization module <NUM> and the QP used for coding the high quality enhancement macroblock in the quantization module <NUM> do not have to be the same.

Combining means <NUM> comprises an entropy coding module <NUM> and the multiplexer <NUM>. The entropy-coding module <NUM> is connected between an output of the quantization module <NUM> and a first input of the multiplexer <NUM>, whereas a second input of the multiplexer <NUM> is connected to an output of entropy coding module <NUM>. The output of the multiplexer <NUM> represents output <NUM> of encoder <NUM>.

The entropy encoding module <NUM> entropy encodes the quantized transform values output by quantization module <NUM> to form a base quality layer data stream from the base encoding data stream output by quantization module <NUM>. Therefore, as mentioned above, modules <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and <NUM> may be designed to co-operate in accordance with the H. <NUM>/MPEG4-AVC, and represent together a hybrid coder with the entropy coder <NUM> performing a lossless compression of the quantized prediction residual.

The multiplexer <NUM> receives both the base quality layer data stream and the high quality layer data stream and puts them together to form the quality-scalable data stream.

The way, in which the prediction module <NUM> forms the prediction signal from the decoded base representation, i.e. the prediction scheme, may be known to the decoder. However, it is also possible to signal prediction scheme information indicating the prediction used by the prediction module <NUM> as side information, i.e. to introduce such side information into the bit stream at output <NUM>. In other words, in order to allow using different schemes for prediction from the decoded base representation, the used prediction scheme may be signaled as side information, e.g., in an extension of the picture parameter set PPS of H. <NUM>/MPEG4-AVC. Further, there may be provisions in case there is no explicit prediction rule specified in the quality scalable bit stream. As described in an example not being part of the invention, for scalability in terms of bit-depth, the base quality samples at the input of the prediction module <NUM> are simply multiplied by <NUM>M-N, where M is the bit-depth of the high quality signal and N is the bit-depth of the base quality signal, which corresponds to a linear mapping. Alternatively, this may be viewed as a performance of a scaling and clipping of the base quality sample values at the input of the prediction module <NUM> according to min(<NUM>M-N x, <NUM>M -<NUM>).

Alternatively, in an example not being part of the invention, one of the following mapping mechanisms may be used for the prediction process. For example, piece-wise linear mapping may be used where an arbitrary number of interpolation points can be specified. For example, for a base quality sample with value x and two given interpolation points (xn,yn) and (xn+<NUM>, yn+<NUM>) the corresponding prediction sample y is obtained by the module <NUM> according to the following formula <MAT>.

This linear interpolation can be performed with little computational complexity by using only bit shift instead of division operations if xn+<NUM> - xn is restricted to be a power of two.

A further possible example mapping mechanism not being part of the invention represents a look-up table mapping in which, by means of the base quality sample values, a table look-up is performed in a look-up table in which for each possible base quality sample value the corresponding prediction sample value is specified. The look-up table may be provided to the decoder side as side information or may be known to the decoder side by default.

Further, scaling with a constant offset, in accordance with an embodiment of the invention, may used. According to this alternative, in order to achieve the corresponding high quality prediction sample y having higher bit-depth, module <NUM> multiplies the base quality samples x by a constant factor <NUM>M-N-K, and afterwards a constant offset <NUM>M-<NUM>-<NUM>M-<NUM>-K is added, according to, for example, one of the following formulae: <MAT> or <MAT> , respectively.

By this measure, the low quality dynamic range [<NUM>;<NUM>N-<NUM>] is mapped to the second dynamic range [<NUM>;<NUM>M-<NUM>] in a manner according to which the mapped values of x are distributed in a centralised manner with respect to the possible dynamic range [<NUM>;<NUM>M-<NUM>] of y within a extension which is determined by K. The value of K could be an integer value or real value, and could be transmitted as side information to the decoder within, for example, the quality-scalable data stream so that at the decoder some predicting means may act the same way as the prediction module <NUM> as will be described in the following. With respect to the definition of M and N reference is made to the above description. A round operation may be used to get integer valued y values.

Another example possibility not being part of the invention is scaling with variable offset: the base quality samples x are multiplied by a constant factor, and afterwards a variable offset is added, according to, for example, one of the following formulae: <MAT> or <MAT>.

By this measure, the low quality dynamic range is mapped to the second dynamic range in a manner according to which the mapped values of x are distributed within a portion of the possible dynamic range of y, the extension of which is determined by K, and the offset of which with respect to the lower boundary is determined by D. D may be integer or real. The result y represents a picture sample value of the high bit-depth prediction signal. The values of K and D could be transmitted as side information to the decoder within, for example, the quality-scalable data stream. Again, a round operation may be used to get integer valued y values, the latter being true also for the other examples given in the present application for the bit-depth mappings without explicitly stating it repeatedly.

An even further possibility not being part of the invention is scaling with superposition: the high bit depth prediction samples y are obtained from the respective base quality sample x according to, for example, one of the following formulae, where floor( a ) rounds a down to the nearest integer: <MAT> or <MAT>.

The just mentioned possibilities may be combined. For example, scaling with superposition and constant offset may be used: the high bit depth prediction samples y are obtained according to, for example, one of the following formulae, where floor( a ) rounds a down to the nearest integer: <MAT> <MAT>.

The value of K may be specified as side information to the decoder.

Similarly, scaling with superposition and variable offset may be used: the high bit depth prediction samples y are obtained according to the following formula, where floor( a ) rounds a down to the nearest integer: <MAT> <MAT>.

The values of D and K may be specified as side information to the decoder.

Further, it is possible to specify different mapping mechanisms for the luma and the chroma components of the base quality signal to take into account that the statistics, such as their probability density function, may be different. It is also possible to specify different mapping mechanisms for different regions of a picture, where a region not necessarily has to be contiguous. Furthermore, it is possible to specify that, after employing one of the above described mapping mechanisms, a pseudo-random noise signal ("dithering signal") is to be added to the high bit depth prediction signal within the prediction process preformed by prediction module at the encoder and at the decoder, respectively. Since this signal has to be exactly known at the decoder to be able to decode the high bit depth representation, certain parameters as initialization value of the pseudo-random generator, variance and shape (e.g., normal or uniform distribution) of the pseudo-random process may have to be transmitted as side information. In case no such side information is transmitted in the scalable bit stream, default values, as for example a uniform distribution of pseudo-random values in the range [<NUM>, <NUM>M - N -<NUM>] or [<NUM>, <NUM>M - N - K -<NUM>] respectively, depending on the selected mapping mechanism, could be used.

The above-mentioned possibilities of using prediction scheme side information or not also applies for the scalability in terms of chroma sampling format. For scalability in terms of chroma sampling format, the scheme of interpolation filtering used by the prediction module <NUM> for generating the upsampled chroma signal can be transmitted as side information, e.g., in the PPS of H. <NUM>/MPEG4-AVC. In the simplest case, if there is no upsampling rule specified, the decoder may assume that the upsampled signal is generated by linear interpolation between base quality chroma sample points for conversion from <NUM>:<NUM>:<NUM> to <NUM>:<NUM>:<NUM> and from <NUM>:<NUM>:<NUM> to <NUM>:<NUM>:<NUM> chroma subsampling and by bilinear interpolation for conversion from <NUM>:<NUM>:<NUM> to <NUM>:<NUM>:<NUM> chroma sub-sampling. With regard to the just-mentioned chroma-sampling format, reference is made to the above description with respect to <FIG>. Otherwise, it is possible that the prediction module <NUM> selects either, for example, an <NUM>-tap half-pel interpolation filter specified by MPEG4 Advanced Simple Profile in ISO/IEC <NUM>-<NUM>:<NUM>, "Information technology - Coding of audio-visual objects - Part <NUM>: Visual", or the <NUM>-tap half-pel interpolation filter specified by H. <NUM>/MPEG4-AVC for generating interpolated chroma sample values. However, the prediction scheme with respect to chroma sampling format scalability is not restricted to the aforementioned filters. It is also possible to specify a generic interpolation filter by transmitting the used filter taps or FIR filter coefficients, respectively, as side information in the quality-scalable bit stream, e.g., in the PPS of H. <NUM>/MPEG4-AVC.

Regarding the scalability in terms of chroma sampling format, it shall be expressively noted that the encoder <NUM> may support the following variants when considering the chroma sampling formats shown in <FIG>. Firstly, the down-conversion module <NUM> may down sample from <NUM>:<NUM>:<NUM> to <NUM>:<NUM>:<NUM>, in which case the prediction module <NUM> upsamples from <NUM>:<NUM>:<NUM> to <NUM>:<NUM>:<NUM> and the quality-scalable bit stream at the output <NUM> enables chroma sampling format scalability from <NUM>:<NUM>:<NUM> to <NUM>:<NUM>:<NUM>. Similarly, down-conversion module <NUM> may downsample from <NUM>:<NUM>:<NUM> to <NUM>:<NUM>:<NUM> with the prediction module <NUM> performing the upsampling in the reverse direction. Accordingly, the down-conversion module <NUM> may downsample from <NUM>:<NUM>:<NUM> to <NUM>:<NUM>:<NUM> and the prediction module <NUM> may, in turn, perform an upsampling from <NUM>:<NUM>:<NUM> to <NUM>:<NUM>:<NUM>.

Within the coded high quality enhancement signal output by entropy coding module <NUM>, the following information could be transmitted for each macroblock in case modules <NUM>, <NUM> and <NUM> implement an H. <NUM>/MPEG4-AVC conforming encoding. A coded block pattern (CBP) information could be included indicating as to which of the four 8x8 luma transformation blocks within the macroblock and which of the associated chroma transformation blocks of the macroblock may contain non-zero transform coefficients. If there are no non-zero transform coefficients, no further information is transmitted for the particular macroblock. Further information could relate to the transform size used for coding the luma component, i.e. the size of the transformation blocks in which the macroblock consisting of 16x16 luma samples is transformed in the transform module <NUM>, i.e. in 4x4 or 8x8 transform blocks. Further, the high quality enhancement layer data stream could include the quantization parameter QP used in the quantization module <NUM> for controlling the quantizer step-size. Further, the quantized transform coefficients, i.e. the transform coefficient levels, could be included for each macroblock in the high quality enhancement layer data stream output by entropy coding module <NUM>.

After having described an embodiment for an encoder, with respect to <FIG>, an embodiment of a decoder is described. The decoder of <FIG> is indicated by reference sign <NUM> and comprises a de-multiplexing means <NUM>, a base decoding means <NUM>, a prediction means <NUM>, a residual decoding means <NUM> and a reconstruction means <NUM> as well as an input <NUM>, a first output <NUM> and a second output <NUM>. The decoder <NUM> receives, at its input <NUM>, the quality-scalable data stream, which has, for example, been output by encoder <NUM> of <FIG>. As described above, the quality scalability may relate to the bit-depth and/or the chroma-sampling format. In other words, the data stream at the input <NUM> may have a self-contained part which is isolatedly usable to reconstruct the video signal with a reduced bit-depth and/or reduced chroma resolution and maintained luma resolution, as well as an additional part which, in combination with the first part, enables reconstructing the video signal with a higher bit-depth and/or higher chroma resolution. The lower quality reconstruction video signal is output at output <NUM>, whereas the higher quality reconstruction video signal is output at output <NUM>.

The demultiplexing means <NUM> divides up the incoming quality-scalable data stream at input <NUM> into the base encoding data stream and the high quality enhancement layer data stream, both of which have been mentioned with respect to <FIG>. The base decoding means <NUM> is for decoding the base encoding data stream into the base quality representation of the video signal, which is directly, as it is the case in the example of <FIG>, or indirectly via an artifact reduction filter (not shown), optionally outputable at output <NUM>. Based on the base quality representation video signal, the prediction means <NUM> forms a prediction signal having the increased picture sample bit depth and/or the increased chroma sampling resolution. The decoding means <NUM> decodes the enhancement layer data stream to obtain the prediction residual having the increased bit-depth and/or increased chroma resolution. The reconstruction means <NUM> obtains the high quality video signal from the prediction and the prediction residual and outputs same at output <NUM> via an optional artifact reducing filter.

Internally, the demultiplexing means <NUM> comprises a demultiplexer <NUM> and an entropy decoding module <NUM>. An input of the demultiplexer <NUM> is connected to input <NUM> and a first output of the demultiplexer <NUM> is connected to the residual decoding means <NUM>. The entropy-decoding module <NUM> is connected between another output of the demultiplexer <NUM> and the base decoding means <NUM>. The demultiplexer <NUM> divides the quality-scalable data stream into the base layer data stream and the enhancement layer data stream as having been separately input into the multiplexer <NUM>, as described above. The entropy decoding module <NUM> performs, for example, a Huffman decoding or arithmetic decoding algorithm in order to obtain the transform coefficient levels, motion vectors, transform size information and other syntax elements necessary in order to derive the base representation of the video signal therefrom. At the output of the entropy-decoding module <NUM>, the base encoding data stream results.

The base decoding means <NUM> comprises an inverse transform module <NUM>, an adder <NUM>, an optional loop filter <NUM> and a predictor module <NUM>. The modules <NUM> to <NUM> of the base decoding means <NUM> correspond, with respect to functionality and inter-connection, to the elements <NUM> to <NUM> of <FIG>. To be more precise, the inverse transform module <NUM> and the adder <NUM> are connected in series in the order mentioned between the demultiplexing means <NUM> on the one hand and the prediction means <NUM> and the base quality output, respectively, on the other hand, and the optional loop filter <NUM> and the predictor module <NUM> are connected in series in the order mentioned between the output of the adder <NUM> and another input of the adder <NUM>. By this measure, the adder <NUM> outputs the base representation video signal with the reduced bit-depth and/or the reduced chroma resolution which is receivable from the outside at output <NUM>.

The prediction means <NUM> comprises an optional artifact reduction filter <NUM> and a prediction information module <NUM>, both modules functioning in a synchronous manner relative to the elements <NUM> and <NUM> of <FIG>. In other words, the optional artifact reduction filter <NUM> optionally filters the base quality video signal in order to reduce artifacts therein and the prediction information module <NUM> retrieves predicted pictures with increased bit depths and/or increased chroma resolution in a manner already described above with respect to the prediction module <NUM>. That is, the prediction information module <NUM> may, by means of side information contained in the quality-scalable data stream or not, map the incoming picture samples to a higher dynamic range and/or apply a spatial interpolation filter to the chroma content of the pictures in order to increase the chroma resolution.

The residual decoding means <NUM> comprises an entropy decoding module <NUM> and an inverse transform module <NUM>, which are serially connected between the demultiplexer <NUM> and the reconstruction means <NUM> in the order just mentioned. The entropy decoding module <NUM> and the inverse transform module <NUM> cooperate to reverse the encoding performed by modules <NUM>, <NUM>, and <NUM> of <FIG>. In particular, the entropy-decoding module <NUM> performs, for example, a Huffman decoding or arithmetic decoding algorithms to obtain syntax elements comprising, among others, transform coefficient levels, which are, by the inverse transform module <NUM>, inversely transformed to obtain a prediction residual signal or a sequence of residual pictures.

The reconstruction means <NUM> comprises an adder <NUM> the inputs of which are connected to the output of the prediction information module <NUM>, and the output of the inverse transform module <NUM>, respectively. The adder <NUM> adds the prediction residual and the prediction signal in order to obtain the high quality video signal having the increased bit depth and/or increased chroma resolution which is fed via an optional artifact reducing filter <NUM> to output <NUM>.

Thus, as is derivable from <FIG>, a base quality decoder may reconstruct a base quality video signal from the quality-scalable data stream at the input <NUM> and may, in order to do so, not include elements <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>. On the other hand, a high quality decoder may not include the output <NUM>.

In other words, in the decoding process, the decoding of the base quality representation is straightforward. For the decoding of the high quality signal, first the base quality signal has to be decoded, which is performed by modules <NUM> to <NUM>. Thereafter, the prediction process described above with respect to module <NUM> and optional module <NUM> is employed using the decoded base representation. The quantized transform coefficients of the high quality enhancement signal are scaled and inversely transformed by the inverse transform module <NUM>, for example, as specified in H. <NUM>/MPEG4-AVC in order to obtain the residual or difference signal samples, which are added to the prediction derived from the decoded base representation samples by the prediction module <NUM>. As a final step in the decoding process of the high quality video signal to be output at output <NUM>, optional a filter can be employed in order to remove or reduce visually disturbing coding artifacts. It is to be noted that the motion-compensated prediction loop involving modules <NUM> and <NUM> is fully self-contained using only the base quality representation. Therefore, the decoding complexity is moderate and there is no need for an interpolation filter, which operates on the high bit depth or high chroma resolution image data in the motion-compensated prediction process of predictor module <NUM>.

Regarding the above embodiments, it should be mentioned that the artifact reduction filters <NUM> and <NUM> are optional and could be removed. The same applies for the loop filters <NUM> and <NUM>, respectively, and filter <NUM>. Further, the present invention is not restricted to video coding. Rather, the above description is also applicable to still image coding. Accordingly, the motion-compensated prediction loop involving elements <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> and the elements <NUM>, <NUM>, and <NUM>, respectively, may be removed also. Similarly, the entropy coding mentioned needs not necessarily to be performed.

Even more precise, in the above embodiments, the base layer encoding <NUM>-<NUM>, <NUM> was based on motion-compensated prediction based on a reconstruction of already lossy coded pictures. In this case, the reconstruction of the base encoding process may also be viewed as a part of the high-quality prediction forming process as has been done in the above description. However, in case of a lossless encoding of the base representation, a reconstruction would not be necessary and the down-converted signal could be directly forwarded to means <NUM>, <NUM>, respectively. In the case of no motion-compensation based prediction in a lossy base layer encoding, the reconstruction for reconstructing the base quality signal at the encoder side would be especially dedicated for the high-quality prediction formation in <NUM>. In other words, the above association of the elements <NUM>-<NUM> and <NUM> to means <NUM>, <NUM> and <NUM>, respectively, could be performed in another way. In particular, the entropy coding module <NUM> could be viewed as a part of base encoding means <NUM>, with the prediction means merely comprising modules <NUM> and <NUM> and the combining means <NUM> merely comprising the multiplexer <NUM>. This view correlates with the module/means association used in <FIG> in that the prediction means <NUM> does not comprise the motion compensation based prediction. Additionally, however, demultiplexing means <NUM> could be viewed as not including entropy module <NUM> so that base decoding means also comprises entropy decoding module <NUM>. However, both views lead to the same result in that the prediction in <NUM> is performed based on a representation of the source material with the reduced bit-depth and/or the reduced chroma resolution which is losslessly coded into and losslessly derivable from the quality-scalable bit stream and base layer data stream, respectively. According to the view underlying <FIG>, the prediction <NUM> is based on a reconstruction of the base encoding data stream, whereas in case of the alternative view, the reconstruction would start from an intermediate encoded version or halfway encoded version of the base quality signal which misses the lossless encoding according to module <NUM> for being completely coded into the base layer data stream. In this regard, it should be further noted that the down-conversion in module <NUM> does not have to be performed by the encoder <NUM>. Rather, encoder <NUM> may have two inputs, one for receiving the high-quality signal and the other for receiving the down-converted version, from the outside.

In the above-described embodiments, the quality-scalability did merely relate to the bit depth and/or the chroma resolution. However, the above embodiments may easily be extended to include temporal scalability, spatial scalability, and fine granular quality scalability. For example, at the encoder side, several of the encoders of <FIG> may be provided with inputting a spatially non-decimated and increasingly spatially decimated versions of an input signal into these spatial layer encoders. A redundancy among those layers could be exploited by using a reconstructed representation of a lower spatial resolution layer as a prediction for the next higher spatial resolution layer. The fine granular quality scalability could be implemented, for example, in the residual encoding means <NUM> and the residual decoding means <NUM>, respectively, by accordingly encoding the transform coefficient levels in layers of increasing quantization resolution, or layers corresponding to a decreasing quantization step size so that the transformation coefficient levels are derivable from a summation of the contributions of the individual FGS layers.

Thus, the above embodiments enable an extension of scalable video coding toward scalability in terms of sample bit depth and/or chroma sampling format, thereby enabling an encoder to store a base-quality representation of a video sequence, which can be decoded by any legacy video decoder, together with an enhancement signal for higher bit depth and/or reduced chroma subsampling, which is ignored by legacy video decoders. For example, the base quality representation could contain an <NUM>-bit version of the video sequence, while the high quality enhancement signal contains a "refinement" to a <NUM>-bit version of the same sequence.

Thus, in other words, the above embodiments describe a video coder (encoder/decoder) for coding (encoding/decoding) a layered representation of a video signal comprising, for example, a standardized video coding method for coding a base-quality layer, a prediction method for performing a prediction of the high-quality enhancement layer signal by using the reconstructed base-quality signal, and a residual coding method for coding of the prediction residual of the high-quality enhancement layer signal. In this video coder, the prediction may be performed by using a mapping function from the dynamic range associated with the base-quality layer to the dynamic range associated with the high-quality enhancement layer. In particular, in accordance with an example not being part of the invention, the prediction may be performed by scaling and clipping the sample values x of the base-quality layer according to the formula min(<NUM>M-Nx,<NUM>M-<NUM>), where the sample values x of the base-quality layer are represented with a bit depth of N and the sample values of the high-quality enhancement layer are represented with a bit depth of M with M > N. Other alternatives have been indicated above. Alternatively, the prediction may be performed by using a piece-wise linear mapping with a given number of interpolation points transmitted as a side information. Even alternatively, the prediction may be performed by using a value of the base-quality signal sample for indexing a look-up table, which contains the corresponding high-quality sample values and which may also be transmitted as side information. The residual coding may be performed along the H. <NUM>/MPEG4-AVC. The color-space representation of the video signal may be given in the form of one luma and two chroma components with a chroma sampling format of the base-quality layer and high-quality enhancement layer being <NUM>:<NUM>:<NUM> and <NUM>:<NUM>:<NUM>, respectively, and the prediction of the chroma sample values of the high-quality enhancement layer signal may be performed by using a spatial interpolation filter applied to the chroma sample values of the base quality layer. Similarly, the color-space representation of the video signal may be given in the form of one luma and two chroma components with a chroma sampling format of the base-quality layer and high-quality enhancement layer being <NUM>:<NUM>:<NUM> and <NUM>:<NUM>:<NUM>, respectively, and the prediction of the chroma sample values of the high-quality enhancement layer signal may be performed by using a spatial interpolation filter applied to the chroma sample values of the base-quality layer. Accordingly, the color-spatial representation of the video signal may be given in the form of one luma and two chroma components with a chroma sampling format of the base-quality layer and high-quality enhancement layer being <NUM>:<NUM>:<NUM> and <NUM>:<NUM>:<NUM>, respectively, and the prediction of the chroma sample values of the high-quality enhancement layer signal may be performed by using a spatial interpolation filter applied to the chroma sample values of the base-quality layer. In case of the base-quality layer and high-quality enhancement layer being <NUM>:<NUM>:<NUM> and <NUM>:<NUM>:<NUM>, respectively, the spatial interpolation filter used may be a linear interpolation filter vertically applied to the chroma sample values of the base-quality layer. Similarly, if the base-quality layer and the high-quality enhancement layer are <NUM>:<NUM>:<NUM> and <NUM>:<NUM>:<NUM>, respectively, a linear interpolation filter may horizontally applied to the chroma sample values of the base-quality layer, and, if the base-quality layer and the high-quality enhancement layer are <NUM>:<NUM>:<NUM> and <NUM>:<NUM>:<NUM>, respectively, the spatial interpolation filter may be a bilinear interpolation applied to the chroma sample values of the base-quality layer. Further, if the base-quality layer and the high-quality enhancement layer are <NUM>:<NUM>:<NUM> and <NUM>:<NUM>:<NUM>, respectively, the spatial interpolation filter may be a <NUM>-tap half-pel interpolation filter as specified by H. <NUM>/MPEG4-AVC vertically applied to the chroma sample values of the base-quality layer. Similarly, if the base-quality layer and the high-quality enhancement layer are <NUM>:<NUM>:<NUM> and <NUM>:<NUM>:<NUM>, respectively, the spatial interpolation filter may be the <NUM>-tap half-pel interpolation filter specified by H. <NUM>/MPEG4-AVC horizontally applied to the chroma sample values of the base-quality layer. If the base-quality layer and the high-quality enhancement layer are <NUM>:<NUM>:<NUM> and <NUM>:<NUM>:<NUM>, respectively, the spatial interpolation filter may be realized by a <NUM>-tap half-pel interpolation filter specified by H. <NUM>/MPEG4-AVC separably applied to the chroma sample values of the base-quality layer. Similarly, in case of base-quality layer and high-quality enhancement layer being <NUM>:<NUM>:<NUM> and <NUM>:<NUM>:<NUM>, respectively, an <NUM>-tap half-pel interpolation filter specified by MPEG4-Part2 visual, advanced simple profile, may be vertically applied to the chroma sample values of the base-quality layer in the prediction modules <NUM>, and <NUM>, respectively. Similarly, if the base-quality layer and the high-quality enhancement layer are <NUM>:<NUM>:<NUM> and <NUM>:<NUM>:<NUM>, respectively, the <NUM>-tap half-pel interpolation filter specified by MPEG4 Part2 visual, Advanced Simple Profile, may be horizontally applied to the chroma sample values of the base-quality layer. If the base-quality layer and the high-quality enhancement layer are <NUM>:<NUM>:<NUM> and <NUM>:<NUM>:<NUM>, respectively, an <NUM>-tap half-pel interpolation filter specified by MPEG4 Part2 visual, advanced simple profile, may be separably applied to the chroma sample values of the base-quality layer. A FIR interpolation filter as the spatial interpolation filter is also possible. For example, if the base-quality layer and the high-quality enhancement layer are <NUM>:<NUM>:<NUM> and <NUM>:<NUM>:<NUM>, respectively, a FIR interpolation filter may be vertically applied to the chroma sample values of the base-quality layer, where the filter taps of the FIR filter may be transmitted as side information. Similarly, if the base-quality layer and the high-quality enhancement layer are <NUM>:<NUM>:<NUM> and <NUM>:<NUM>:<NUM>, respectively, a FIR interpolation filter may be horizontally applied to the chroma sample values of the base-quality layer, where the filter taps of the FIR filter are transmitted as side information. If the base-quality layer and the high-quality enhancement layer are <NUM>:<NUM>:<NUM> and <NUM>:<NUM>:<NUM>, respectively, a FIR interpolation filter may be separably applied to the chroma sample values of the base-quality layer, where the filter taps of the FIR filter may be transmitted as side information. It is also possible that the color-space representation of the base-quality video signal is given in the form of one luma and two chroma components with a chroma sub-sampling format of <NUM>:<NUM>:<NUM> or <NUM>:<NUM>:<NUM> and wherein the high-quality video signal has a chroma sampling format of <NUM>:<NUM>:<NUM>, but a color-space representation that is different from that of the base-quality representation. In that case, the prediction of the chroma sample values of the high-quality enhancement layer signal within modules <NUM> and <NUM>, respectively, may be performed by first applying a spatial interpolation filter to the chroma sample values of the base-quality layer as described above, and subsequently color-transforming the resulting upsampled base-quality signal to the color-space of the high-quality enhancement layer.

Claim 1:
Encoder for encoding a picture into a quality-scalable data stream, comprising:
base encoding means (<NUM>) for encoding the picture into a base encoding data stream representing a representation of the picture with a first picture sample bit depth;
prediction means (<NUM>) for providing a prediction of the picture based on the representation of the picture with the first picture sample bit depth, the prediction of the picture having a second picture sample bit depth being higher than the first picture sample bit depth; and
residual encoding means (<NUM>) for encoding a prediction residual of the prediction into a bit-depth enhancement layer data stream; and
combining means (<NUM>) for forming the quality-scalable data stream based on the base encoding data stream and the bit-depth enhancement layer data stream,
wherein the prediction means further comprises mapping means (<NUM>) for performing, on the representation of the picture with the first picture sample bit depth, a mapping of samples from a first dynamic range corresponding to the first picture sample bit depth to a second dynamic range greater than the first dynamic range and corresponding to the second picture sample bit depth to obtain the prediction of the picture,
characterized in that the mapping means (<NUM>) is adapted to map the samples x from the first dynamic range to the second dynamic range by computing <NUM>M-N-K x + <NUM>M-<NUM> - <NUM>M-<NUM>-K, where N is the first picture sample bit depth, M is the second picture sample bit depth, and K is a mapping parameter, and wherein the mapping means is adapted to forward the mapping parameter K to the combining means (<NUM>) for formation of the quality-scalable data stream also based on the mapping parameter.