Drift reduction for quality scalable video coding

This invention is a method of scalable video encoding and decoding. The scalable video encoding codes both a base layer and an enhanced layer having greater resolution and/or refresh rate. Upon decoding some enhanced layer pictures may be dropped to reach a best resolution and refresh rate within a target data rate. Upon encoding a key picture in at least one group of pictures forming the video is a combined base layer/enhanced layer key picture. Such a combined base layer/enhanced layer key picture cannot be dropped on decoding. This technique reduces drift in the decoder.

CLAIM OF PRIORITY

TECHNICAL FIELD OF THE INVENTION

The technical field of this invention is quality scalable video coding, also known as signal-to-noise ratio (SNR) scalable video coding (SVC).

BACKGROUND OF THE INVENTION

Scalable Video Coding (SVC) is the name given to the extension Annex G of the H.264/MPEG4 AVC video compression standard. SVC is a technique that enables a video stream to be broken into multiple resolutions, quality levels and frame rates. In one embodiment, scalability refers to removal of parts of the video stream in order to adapt to various needs or preferences of end users as well as to varying terminal capabilities or network conditions. Applications of scalability include but are not limited to Internet Video, Video Telephony and wireless communication where bandwidth availability cannot be guaranteed. In video streaming application, a server usually serves a large number of users with different screen resolutions and network bandwidth. When the users screen is too small or the bandwidth between some users and the server is too narrow to support higher resolution sequences, spatial scalability coding provides different resolutions. This is turn helps the server accommodate different users with different bit rate or screen resolution capabilities.

SUMMARY OF THE INVENTION

This invention is a method of scalable video encoding and decoding. The scalable video encoding codes both a base layer and an enhanced layer having greater resolution and/or refresh rate. Upon decoding some enhanced layer pictures may be dropped to reach a best resolution and refresh rate within a target data rate. Upon encoding a key picture in at least one group of pictures forming the video is a combined base layer/enhanced layer key picture. Such a combined base layer/enhanced layer key picture cannot be dropped on decoding. This technique reduces drift in the decoder.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Scalable Video Coding (SVC) is the name given to the extension Annex G of the H.264/MPEG4 AVC video compression standard. SVC is a technique that enables a video stream to be broken into multiple resolutions, quality levels and frame rates. In one embodiment, scalability refers to removal of parts of the video stream in order to adapt to various needs or preferences of end users as well as to varying terminal capabilities or network conditions. Applications of scalability include but are not limited to Internet Video, Video Telephony and wireless communication where bandwidth availability cannot be guaranteed. In video streaming application, a server usually serves a large number of users with different screen resolutions and network bandwidth. When the users screen is too small or the bandwidth between some users and the server is too narrow to support higher resolution sequences, spatial scalability coding provides different resolutions. This is turn helps the server accommodate different users with different bit rate or screen resolution capabilities.

FIG. 1is a block diagram of a scalable video coding (SVC) encoder100. SVC encoder100includes base layer (Layer0) encoder110and enhancement layer (Layer1) encoder120. A full sized group of pictures (GPO)101supplies Layer1encoder120. GOP101are sized via special decimation102into group of pictures (GOP)103. GOP103are smaller in both spatial dimensions than GOP101. GOP103supplies Layer0encoder110.

Layer0encoder110includes motion-compensated and intra prediction unit111, base layer coding unit112and signal to noise ratio (SNR) coding unit113. Motion-compensated and intra prediction unit111forms motion-compensation and/or intra prediction on GOP103supplying texture and motion results to base layer coding unit112. Base layer coding unit112generates a H.264/AVE compatible base layer bitstream supplied to both SNR coding unit113and multiplexer131. Motion-compensated and intra prediction unit111also supplies inter-layer prediction of intra, motion and residual compensation for supply to both SNR coding unit113and motion-compensated and intra prediction unit121. SNR coding unit113directly receives GOP103and receives the H.264/AVE compatible base layer bitstream and the inter-layer prediction of intra, motion and residual compensation from motion-compensated and intra prediction unit111. SNR scalable coding unit112supplies corresponding SNR scalable coding data to multiplexer131.

Layer1encoder120includes motion-compensated and intra prediction unit121, base layer coding unit122and signal to noise ratio (SNR) coding unit123. Motion-compensated and intra prediction unit111121forms motion-compensation and/or intra prediction on GOP101including inter-layer prediction of intra, motion and residual compensation received from motion-compensation and intra prediction unit111. Motion-compensated and intra prediction unit121supplies texture and motion results to base layer coding unit122. Base layer coding unit122generates a H.264/AVE compatible base layer bitstream supplied to both SNR coding unit123and multiplexer131. SNR coding unit123directly receives GOP101and receives the H.264/AVE compatible base layer bitstream and the inter-layer prediction of intra, motion and residual compensation from motion-compensated and intra prediction unit121. SNR scalable coding unit122supplies corresponding SNR scalable coding data to multiplexer131.

Multiplexer131receives bitstream data and SNR scaling coding data from both Layer0110and Layer1120. Multiplexer131combines this data into scalable bitstream132.

In SVC only the base layer is coded independently. Other layers are coded dependently with each following layer coded with respect to previous layers. This generates only a single bit-stream. Parts of this single bit stream can be extracted so that the resulting sub-streams form another valid bit-stream for a target decoder.

Video streams may be scalable by different modes such as temporal, spatial and quality. In spatial scalability describe the scaled bit stream represents the source content with a reduced picture size. In temporal scalability the scaled bit stream represents the source content with a reduced frame rate. In quality scalability the scaled bit stream provides the same spatio-temporal resolution as the source bit stream but with varying degrees of fidelity. Quality scalability is also referred as signal-to-noise ratio (SNR) scalability.

Temporal scalability codes the video sequence into two layers at the same spatial resolution, but different frame rates. A base layer is coded at a lower frame rate. An enhancement layer provides the missing frames to form a video with a higher frame rate. Coding efficiency of temporal scalability is high and is very close to non-scalable coding. This scalability can be used in range of diverse video applications from telecommunication to HDTV. In many cases the lower temporal resolution video systems may be either the existing systems or less expensive early systems. More sophisticated systems may then be introduced gradually. Temporal scalable bit-streams can be generated using hierarchical prediction structures. The number of supported temporal scalability levels is dependent on the specified Group of pictures (GOP) size. A GOP consists of a key picture and hierarchically predicted B pictures that are located between the key picture of a current GOP and the key picture of a previous GOP.

Spatial scalability involves generating two or more layers with different spatial resolutions from a single video source.FIG. 1illustrates an example of spatial scalability. The base layer is coded providing a basic spatial resolution and the enhancement layer(s) employ the spatial interpolated base layer and carry higher or full spatial resolution of the video source. In each spatial layer, motion compensated prediction and intra prediction is employed as for single layer coding. But to improve coding efficiency in comparison to simulcast of different spatial resolutions, additional inter layer predictions are employed. This is explained in the next section. Spatial scalability supports interoperability between application using different video formats. For example the base layer can use QCIF (176×144) resolution at 4:2:0 while the enhancement layer can use CIF (356×288) resolution at 4:2:2.

In SNR (Quality) Scalability, the two layers are coded at the same rate and same spatial resolution but with different quantization accuracies. This is a special case of spatial scalability with identical picture size for the base layer and the enhancement layer. This is also known as Coarse Grain Scalability (CGS). The same interlayer prediction is employed as for spatial scalability. A variation of the CGS approach known as Medium Grain SNR Scalability (MGS) coding is also included in the SVC design. The difference of MGS from the CGS includes a modified high level signaling. This allows switching between different MGS layers in any access unit and key picture and allows adjustment of a suitable tradeoff between drift and enhancement layer coding efficiency for hierarchical prediction structures. A picture is called a key picture when all previously coded pictures precede this picture in display order. A key picture and all pictures temporally located between the key picture and a previous key picture form a group of pictures (GOP). The key pictures are either intra-coded or inter-coded using previous (key) pictures as reference for motion compensated prediction.

An SVC bit stream allows packets to be dropped to reduce spatial resolution and frame rate of quality. The MGS in SVC allows packets to be dropped arbitrarily to reduce bit rate. One or more enhancement layer packets can be dropped or kept. On the other hand, spatial scalability allows a layer to be dropped or kept.

Individual scalabilities can be combined to form mixed scalability for certain applications. With a properly configured scheme of combined scalability: the source content is encoded once for the highest requested resolution frame rate and bit rate; and at least one scalable bit stream is formed stream from which representations of lower resolution, lower frame rate and/or lower quality can be obtained by partial decoding. The main goal of Inter Layer prediction tool is use as much as lower layer information as possible for improving rate distortion efficiency of the enhancement layers.

In Inter Layer Intra Prediction, when a macroblock (MB) of the base layer (BL) is coded in Intra mode, the corresponding block of enhancement layer (EL) can use the reconstructed MB of the BL as the prediction signal. The reconstructed signal of the MB is up sampled via a one dimensional 4 tap finite impulse response (FIR) filters for the luminance component and via a bilinear filter for the chrominance component.

In Inter Layer Motion Prediction, since the motion vectors (MVs) of all the layers are highly correlated the MVs of the EL can be derived from those of the co-located blocks of the BL when both MBs of the BL and EL are coded in the inter mode. The associated MBs are scaled by a factor of 2 for the dyadic case. The MB portioning of the EL is obtained from the partitioning of the corresponding block of BL. For the obtained MB partitions the same reference indices of the corresponding MB of the BL is used.

In Inter Layer Residual Prediction, if the MV of the BL and EL are similar, coding efficiency can be improved when the residual of the MV of the BL is used to predict the MV of the EL. The residual prediction can be adaptively employed for MB prediction. The residual signal is up sampled using bilinear filter to predict the EL for dyadic case.

In Joint Layer Optimization all coding modes of every MB are examined and the resulting rate (R) and distortion (D) are calculated. Rate-Distortion optimization (RDO) techniques select a particular coding mode. The goal is to minimize the distortion D for a given rate Rcby appropriate selection of coding parameters: namely Min{D} Subject to R≦Rc.

FIG. 2is an exemplary illustration of a prediction structure used for MGS in the prior art. The stream is coded with two quality layers/levels: a base layer/base quality; and an enhancement layer/enhancement quality. A first set of pictures represented by A, B, B, B form a first group of pictures (GOP)201. A second set of pictures represented by A, B, B, B form a first group of pictures (GOP)202. The size of GOP201and GOP202in the exampleFIG. 2is 4.FIG. 2illustrates BL pictures210and EL pictures230.FIG. 2illustrates the flow of data in picture prediction. Picture231is predicted based upon picture211. Pictures212and232are predicted based upon picture231. Pictures213and233are predicted based upon picture232. Pictures214and234are predicted based upon picture233. Picture221is predicted based upon picture211. Picture241is predicted based upon pictures221and231. Pictures222and242are predicted based upon picture241. Pictures223and243are predicted based upon picture242. Pictures224and244are predicted based upon picture243. The quality of the A pictures (211,231,221and241) is extremely important for overall video quality of the sequence. These pictures are typically called as Key Pictures.

FIG. 3illustrates an example valid scaled version of the stream of pictures illustrated inFIG. 2(prior art). MGS allows enhancement layer packets to be dropped arbitrarily. Any picture which used to refer to the dropped packet would then refer to the quality layer below. However this creates a problem that is widely called as drift. Drift occurs when the decoder is not using the same reference picture as what the encoder used.FIG. 3illustrates EL layer pictures231and241as dropped. Pictures212and232are now predicted from picture211. Pictures222and242are now predicted from picture221.

This can cause video quality artifacts that can propagate until a closed-loop prediction is restored. Such a closed-loop prediction is restored in an Intra frame that doesn't refer to any previous frame. Drift may be acceptable in MGS as long as it doesn't propagate for long. The propagation and hence the resulting increase of the drift is a more severe problem than the drift itself.

Table 1 shows Bjøntegaard delta (BD) bit rate difference with respect to independent encoding of layers of a MGS SVC bit stream before and after dropping enhancement layer key pictures as illustrated inFIG. 3. This independent coding is known as SIMULCAST. The change in BD bit rate is due to the quality loss and error propagation from the dropping of packets.

TABLE 1MGS BD bit rateMGS BD bit ratechange after thechange with respectenhancement layer keyto simulcastpicture truncation−22%+3%
Table 1 shows that dropping key pictures causes severe drift. In this example, the drift in terms of BD bit rate is 25% (from −22% to +3%).

FIG. 4illustrates an alternate known in the prior art. The SVC uses a mechanism called as base quality prediction to contain drift. In this alternative a picture, typically a key picture, is predicted from a previous base quality picture.FIG. 4illustrates EL picture241predicted from BL picture211.FIG. 5illustrates the prediction structure when the enhancement layer key pictures231and241are dropped. Pictures212and232are now predicted from picture211. Pictures222and242are now predicted from picture221. This is similar to the prediction structure shown inFIG. 3.

FIG. 5shows that there is a drift in the second set of pictures (B) even in the alternative structure, but that drift is contained within the GOP and hence is not as severe as the case illustrated inFIG. 3. This MGS alternative provides a mechanisms for zero drift packet dropping if needed. This mechanism allows the second set of pictures (B) also to predict from base quality base picture. Using base quality key picture degrades the overall video quality since the highest quality of the key pictures are not used in the prediction.

Table 2 compares the video quality of with and without base quality predictions. The video quality is measured as a percentage bit rate change with respect to independent encoding of both the layers as separate H.264 streams, SIMULCAST. The GOP size used is 8 and the resolution of the video used is D1. A negative number shows bit rate reduction with respect to SIMULCAST, which is desirable. The numbers are highly sequence, resolution and GOP size dependent. These numbers are averaged over several sequences and bitrates.

TABLE 2MGS without baseMGS with base qualityquality predictionprediction−22%−17%
Table 2 shows that without base quality prediction MGS SNR scalability provides 22% bit rate savings. Notice however that there is a bit rate increase of about 5% (bit rate savings over SIMULCAST increases from −22% to −17%) while using base quality prediction.

There is a need to overcome drift propagation during packet drops in MGS and bit rate increase while using base quality prediction for MGS.

FIG. 6illustrates this invention employing combined key pictures for drift reduction. As shown in the prior art illustrated inFIG. 2the stream is coded with two quality layers/levels: a base layer/base quality; and an enhancement layer/enhancement quality. A first set of pictures represented by A, B, B, B form a first group of pictures (GOP)601. A second set of pictures represented by A, B, B, B form a first group of pictures (GOP)602. The size of GOP601and GOP602is 4.FIG. 6illustrates BL pictures610and EL pictures630.FIG. 6illustrates the flow of data in picture prediction. Pictures612and632are predicted based upon picture651. Pictures613and633are predicted based upon picture632. Pictures614and634are predicted based upon picture633. Picture661is predicted based upon picture651. Pictures622and642are predicted based upon picture661. Pictures623and643are predicted based upon picture642. Pictures624and644are predicted based upon picture643.FIG. 6differs from the prior art illustrated inFIG. 2by having key pictures combined across layers. Separate pictures211and231are replaced inFIG. 6with combined picture651. Separate pictures221and241are replaced inFIG. 6with combined picture661. This combines the key A pictures in each GOP across layers610and630.

This combination across layers has at least three advantages.FIG. 6includes combined key pictures, which cannot be dropped according to the SVC bit stream rules. This means there is no drift that propagates to the next GOP. The picture quality used for the prediction of key pictures is the enhancement layer quality.

The picture structure illustrated inFIG. 6has a disadvantage. The base layer610bit rate is now higher than the prior art ofFIG. 2because the full bit rate taken by the key pictures is applied to the base layer. This reduces the possible bit rate adaptation that may be achieved by packet dropping.FIG. 7illustrates a variant of the picture structure illustrated inFIG. 7. InFIG. 7key pictures are combined across layers once in several GOPs. Picture611is predicted based upon picture631. Pictures612and632are predicted based upon picture621. Pictures613and633are predicted based upon picture632. Pictures614and634are predicted based upon picture633. Picture661is predicted based upon picture611. Pictures622and642are predicted based upon picture661. Pictures623and643are predicted based upon picture642. Pictures624and644are predicted based upon picture643.FIG. 7differs fromFIG. 6in that not all key pictures are combined across layers.FIG. 7includes separate pictures611and631and combined picture661. This combines the key A pictures in some GOP across layers610and630. The SVC bit stream rules permit picture621to be dropped but not picture661. This provides more flexibility for bit rate adaptation than picture structure ofFIG. 6.FIG. 7illustrates one GOP with separate key pictures and one GOP with a combined key picture. Combining key pictures of every other GOP is not required. This invention could combine key pictures of every Nth GOPs where N is any integer 2 or greater.

Table 3 shows average result. The column “MGS with combined key picture” covers the every other GOP case illustrated inFIG. 7.

TABLE 3MGS withoutMGS withMGS withbase qualitybase qualitycombinedpredictionpredictionkey picture−22%−17%−28%
Table 3 shows that the inventive solution is better than the existing solution for drift reduction because the bit rate savings compared to SIMULCAST increases from 17% to 28%.

It is noted that when key pictures are used, it gives a bit rate increase of 6.07% compared to without using key pictures. Further, if temporal layer0of the base and enhancement layer1of the base and enhancement layer is combined it gives a bit rate saving of 13.47% as compared to using separate key picture. Not shown in Table 3, using combined key pictures as illustrated inFIG. 7reduces drift down to 0% from 12% using separate key pictures.