Unified scaling with differential coding for internal bit depth extension and reference frame compression

A disclosed system, method and computer readable storage medium is provided to receive an input video comprising a plurality of video frames, each of which has a plurality of input pixels. Each input pixel is of n-bit depth (e.g., 8-bit depth). The method optionally increases the bitdepth of each input pixel by a predetermined factor (e.g., 4-bit factor). The method further compresses and reconstructs each pixels of the input video. The method further compresses and reconstructs each pixel of intermediate frame buffers stored in a decoder frame buffer. The method calculates an adaptive offset for each block (e.g., of size of 4×4) of reconstructed pixel values of the intermediate frame buffers. Furthermore, the method can compute two-sub-block offset values and uses them to compute an optimized adaptive offset to be applied to each pixel within the corresponding sub-block.

BACKGROUND

1. Field of Art

The disclosure generally relates to video compression, more particularly, to improvements in video coding efficiency using internal bit depth extension and reference frame compression.

2. Description of the Related Art

Recent video compression development indicates increasing bit depth during encoding process improves video coding efficiency. According to existing video coding experimental results (e.g., TOSHIBA's adaptive scaling scheme), by using 12 bits of internal bit depth for an 8-bit video input, over 10% of bitrate saving for some 720p (“p” stands for “progressive”) and 1080p video sequences can be achieved compared to compression without increasing bit depth. The most coding gain comes from improvements in inter-frame prediction using an internal bit depth increase (IBDI) scheme.

IBDI technique refers to increasing the bit depth of pixels of a video sequence at the input to an encoder and adjusting the bit depth at the output of a decoder. IBDI can be used to improve standard elements of a block-based video codec (e.g., H.264 video coding) including arithmetic precision of prediction, transform and loop filter.

It is known that pixel bit depth has a direct impact on memory access bandwidth and capacity of the reference frame buffers used during the video encoding and decoding processes. One approach is to efficiently compress the IBDI reference frames so that the adverse impact on memory access and capacity of the frame buffers is minimized while maintaining most of the coding gain. Such an approach is referred to as reference frame compression (RFC). The existing IBDI codec faces a major challenge to balance the coding gain from bit depth increase against the adverse impact on implementation complexity. The challenges faced by the existing IBDI codec call for a system and method for improved adaptive bit depth scaling and efficient IBDI reference frame compression.

DETAILED DESCRIPTION

Configuration Overview—IBDI Compression

FIG. 2shows a block diagram of an example existing IBDI codec200. As illustrated inFIG. 2, the IBDI codec200comprises an IBDI encoder210and an IBDI decoder220. The IBDI encoder210comprises a scaling module212, a 12-bit compression module214, a 12-bit decompression module216and other standard video compression modules, such as a discrete cosine transformation (DCT) module211, a quantization module213, a prediction module215, an in-loop filter module217, an entropy encoder201and a decoded pictures buffer (DPB)219. Each input pixel of 8-bit depth of an input video sequence is processed by the scaling module212, which increases the bit-depth of the input pixels. One way of increasing the bit-depth of an 8-bit pixel is by left shifting the input pixel by 4 resulting in a corresponding pixel of 12-bit depth. The 12-bit compression module214encodes the pixels of 12-bit depth of the input video sequence. The 12-bit decompression module216decodes the compressed input video of 12-bit-depth into a corresponding decompressed video sequence of 12-bit depth.

The IBDI decoder220comprises a rounding module222, a 12-bit compression module224, a 12-bit decompression module226and other standard video decoding modules, such as an inverse discrete cosine (IDCT)/quantization module221, a prediction module225, an in-loop filter module227and a decoded pictures buffer (DPB)229. The IBDI decoder220performs similar steps as the IBDI encoder210in a reverse order. Specifically, the IBDI decoder220receives a compressed video from the IBDI encoder210and entropy decodes the compressed video sequence using an entropy decoder203. The IBDI decoder220further inverse transforms and motion compensates the decoded frames of the video. The rounding module222converts the pixels of the decompressed video sequence of 12-bit depth into pixels of 8-bit depth. One way of 12-bit-to-8-bit conversion is by right shifting the 12-bit pixels by 4. The resulting output video is a decompressed video sequence of pixels of 8-bit depth.

One existing implementation of the 12-bit compression module214operates upon 4×4 blocks, computes the minimum pixel value (M) within a block and keeps the 8 most significant bits (MSB) of the minimum value. The maximum pixel value within the block is also recorded. The difference between the maximum pixel value and the minimum pixel value M is used to determine the amount of scaling S applied to each pixel. S is chosen so that each pixel's residual (after subtracting M from the pixel value) can be presented in pulse-code modulation (PCM) using 7 bits. If the amount of scaling is 4 (i.e., S=4), a fixed-scaling mode is used where each pixel is stored using its 8 MSB after a rounding offset is added.

The following Table 1 contains example pseudo code of the existing implementation of the 12-bit compression module214described above:

The following Table 2 contains example pseudo code of the existing implementation of the 12-bit decompression module216:

The following Table 3 contains an example compression storage format of the existing implementation of the 12-bit compression module214described above:

As shown in the above Table 3, the existing implementation of adaptive scaling requires one bit (e.g., the “flag”) to signal the fixed-scaling mode. Thus, the worst case of the scheme requires 129 bits to represent each 4×4 block, which results in a compression that is slightly higher than 8 bits per pixel on average. Additionally, when adaptive scaling mode is used, there are 5 unused bits available that can be used for furthering coding efficiency.

One example embodiment of a disclosed system, method and computer readable storage medium includes receiving an input video comprising a plurality of video frames, each of which has a plurality of input pixels. Each input pixel is of n-bit depth (e.g., 8-bit depth). The method increases the bit-depth of each input pixel by a predetermined factor (e.g., 4-bit). The method further quantizes and de-quantizes each pixel of intermediate frame buffers stored in a decoder picture buffer (DPB). The method calculates an adaptive offset for each block (e.g., of size 4×4) of de-quantized pixel values of the intermediate frame buffers and applies the adaptive offset to each block of de-quantized pixel values of the intermediate frame buffers. Furthermore, the method can compute two sub-block offset values and uses the two sub-block offset values to compute an optimized adaptive offset to be applied to each pixel within the corresponding sub-block.

To improve the coding efficiency of compression with IBDI, one embodiment includes a method that efficiently codes residual pixel values of the input video. The method computes the residual pixel values as the scaled difference between the original pixel value of an input pixel and a quantized minimum value. The method scans the residual pixel values using a cyclic scanning pattern and applies delta coding to the residual pixel values based upon a selected scaling factor.

Computing Machine Architecture

The example computer system100includes a processor102(e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application specific integrated circuits (ASICs), one or more radio-frequency integrated circuits (RFICs), or any combination of these), a main memory104, and a static memory106, which are configured to communicate with each other via a bus108. The computer system100may further include graphics display unit110(e.g., a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)). The computer system100may also include alphanumeric input device112(e.g., a keyboard), a cursor control device114(e.g., a mouse, a trackball, a joystick, a motion sensor, or other pointing instrument), a storage unit116, a signal generation device118(e.g., a speaker), and a network interface device120, which also are configured to communicate via the bus108.

The storage unit116includes a machine-readable medium122(e.g., non-transitory computer-readable storage medium) on which is stored instructions124(e.g., software) embodying any one or more of the methodologies or functions described herein. The instructions124(e.g., software) may also reside, completely or at least partially, within the main memory104or within the processor102(e.g., within a processor's cache memory) during execution thereof by the computer system100, the main memory104and the processor102also constituting machine-readable media. The instructions124(e.g., software) may be transmitted or received over a network126via the network interface device120.

While machine-readable medium122is shown in an example embodiment to be a single medium, the term “machine-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) able to store instructions (e.g., instructions124). The term “machine-readable medium” shall also be taken to include any medium that is capable of storing instructions (e.g., instructions124) for execution by the machine and that cause the machine to perform any one or more of the methodologies disclosed herein. The term “machine-readable medium” includes, but not be limited to, data repositories in the form of solid-state memories, optical media, and magnetic media.

System Architectural Overview

FIG. 3illustrates a system architectural overview of an embodiment of an improved IBDI codec300. The IBDI codec300comprises an improved IBDI encoder310and an improved IBDI decoder320. The improved IBDI encoder310receives 8-bit pixels of an input video and converts the 8-bit pixels into 12-bit pixels. In the embodiment illustrated inFIG. 3, the improved IBDI encoder310left shifts each 8-bit pixel of the input video. Other embodiments may increase the bit depth using other schemes. The improved IBDI encoder310transforms and inverse transforms the input video. The improved IBDI encoder310further applies an in-loop filter to filter out noise from the transformation and inter frame prediction. The improved IBDI encoder310compresses the 12-bit pixels of the input video and stores the compressed video frames of the input video in a video frame buffer (DPB). The improved IBDI encoder310accesses the frame buffer and selects one or more reference frames for decompression. The improved IBDI encoder310uses the decompressed references frames for inter frame prediction. The improved IBDI encoder310entropy encodes the transformed and predicted video frames of the input video by an entropy encoder. The improved IBDI encoder310comprises an improved 12-bit compression module312for compressing 12-bit pixels of the input video and an improved 12-bit decompression module314for decompressing a compressed frame.

The improved IBDI decoder320performs similar steps as the improved IBDI encoder310in a reverse order. Specifically, the improved IBDI decoder320receives a compressed video from the improved IBDI encoder320and entropy decodes the compressed video sequence using an entropy decoder203. The decoder310further inverse transforms and motion compensates the decoded frames of the video. The improved decoder320comprises an improved 12-bit compression module322for compressing 12-bit pixels of the video, an improved 12-bit decompression module324for decompressing a compressed frame and other standard video decoding modules, such as an inverse discrete cosine (IDCT)/quantization module221, a prediction module225, an in-loop filter module227and a decoded pictures buffer (DPB)229. In one embodiment, the improved 12-bit compression module322is a separate entity of the improved decoder320, which performs the same functions as the improved 12-bit compression module312. Other embodiments of the improved decoder320may share the improved 12-bit compression module312with the improved IBDI encoder310. Similarly, the improved decoder320may have a separate entity of an improved 12-bit decompression module324or share the improved 12-bit decompression module314with the improved IBDI encoder310. The pixels of 12-bit depth of the decompressed video are converted to pixels of 8-bit depth by a rounding module222. To simplify one embodiment of the description of the invention, only the improved 12-bit compression module312and improved 12-bit decompression module314of the improved IBDI encoder310are described below.

FIG. 4shows a histogram400of block average of 4-LSB of the pixels at the input to the improved 12-bit compression module312for an example input video.FIG. 4illustrates that there is some variability at the block level that the improved 12-bit compression module312can exploit to improve its coding efficiency. In one embodiment, the improved 12-bit compression module312computes an adaptive offset per block (e.g., of size of 4×4) that is stored in the DPB and used by the improved 12-bit decompression module314to be added to the de-quantized pixels values in the reference frames compression (RFC) process. The improved 12-bit compression module312can compute the best adaptive offset per block given its knowledge of the fractional LSB of the 12-bit pixel values. It can be proved that the optimal value (in terms of least-square-error) for an adaptive offset to be added is the average of the fractional LSB values. The adaptive offset can be stored in the unused bits for the adaptive-scaling mode shown in Table 3, for example. The improved 12-bit compression module312applies the adaptive offset when the flag for fixed-scaling is not set (i.e., the flag value is zero). The following Table 4 contains the pseudo code for the improved 12-bit compression module312described above:

Variable M is the minimum pixel value within a block (e.g., a 4×4 block) computed with 8 leading bits of precision. Variable R represents the difference between the maximum pixel value within the block and M. S represents the amount of scaling applied to each pixel. P[i] is the scaled residual pixel value at i-th pixel in the block.

The following Table 5 contains an example compression format of the above embodiment using adaptive offset:

In one example embodiment, the improved 12-bit decompression module314decompresses a compressed pixel using the following pseudo code:

if (flag)D[i] = (P[i] << 4)elseD[i] = (P[i] << S) + M + offset
where D[i] is the pixel value of a decompressed pixel at i-th position in the block. The experimental results of the embodiment illustrated above shows decreased efficiency loss compared with an existing IBDI codec.

In another embodiment, the improved 12-bit compression module312uses a unified scaling scheme for compression. Comparing with the embodiment illustrated in Table 5, the embodiment illustrated in Table 6 eliminates fixed scaling, allows the scaling variable to vary from 0 to 5 (inclusive) and modifies the computation such that the improved 12-bit compression module312can compute and store the minimum pixel value in the block (i.e., variable M) using (12-S) bits and the offset with S bits. Therefore, both the minimum pixel value and the offset values can be stored using 12 bits.

In one example embodiment, the improved 12-bit decompression module314decompresses a compressed pixel using the following pseudo code:

D[i] = (P[i] << S) + M + offset, for all i
where D[i] is the pixel value of a decompressed pixel at i-th position in the block. The experimental results of the embodiment illustrated above show increased coding efficiency compared with an existing IBDI codec.

An example of storing the minimum pixel value and the offset values in 12 bits using the unified scaling scheme is illustrated in Table 7 below. The total number of bits used for a 4×4 block is 127.

The improved 12-bit compression module312can further reduce the number of bits in the compression format by noting that at least one of the compressed pixel (P variable) values is 0, which corresponds to the entry with the minimum pixel value. The improved 12-bit compression module312can signal the location of the minimum pixel value and only code the other 15 P values. The location can be coded using 4 bits, thus reducing the compression format to 124 bits. This is an implementation optimization that does not affect coding distortion. The extra 4 bits allows the improved 12-bit compression module312to consider further enhancements to the algorithm that can be made to use the extra 4 bits for side information.

Table 8 shows an example optimized compression format for the unified scaling method:

Shown in Table 8, the total number of bits used is (127-S). There are (S+1) bits left over, given a 128-bit budget. This is enough to code a second offset value. In one embodiment, the offset is computed for each block of 4×4 pixels. In another embodiment (“advanced unified scaling”), the improved 12-bit compression module312computes two offsets, one for each 4×2 sub-block. An alternative embodiment may use 2×4 sub-blocks. This can potentially reduce the distortion by better adapting the offsets to the local statistics. An example compression format for advanced unified scaling using a total 127 bits is shown below in Table 9. The compression and decompression algorithms are the same as with the unified scaling described above except that two offsets are computed for 4×2 sub-blocks.

The unified scaling scheme illustrated in Table 6 can be further improved with efficient coding of residual pixel values (i.e., P[i] values). In one embodiment, the improved 12-bit compression module312computes the residual pixel values P as the scaled difference between the original pixel value and the quantized minimum M: P[i]=(pixel_value[i]−M)>>S. The scale factor S is computed such that the residual pixel values P for the maximum pixel value R can be represented using 7 bits. For blocks where the residual changes gradually, the differences of the residual pixel values P between adjacent pixels can often be coded using fewer bits for a given scale factor using differential pulse-code modulation (DPCM). A smaller scale factor S can potentially be used with DPCM coding.

In one embodiment, the DPCM coding of the residual pixel values comprises scanning the residual pixel values and delta coding the residual pixels by the compression module312. For differential coding, the order in which the differences are computed needs to be considered. A scan pattern determines the order in which the differences are computed. Generally, a scan pattern should cover every sample location exactly once. Given that the index of the minimum sample is coded in the optimized compression format given in Table 8, the improved 12-bit compression module312starts the scanning at the location of the minimum sample. Given that the minimum sample can occur at any location, the scanning pattern is cyclic. In a traditional scan pattern such as the zig-zag scan for a 4×4 block of pixels, the pattern begins at the upper-left sample (0,0) and ends at the lower-right sample (3,3). Making the zig-zag scan into a cyclic scan may introduce a jump between (3,3) and (0,0). A better scanning pattern has the property that successive positions in the scan map to adjacent spatial positions.FIG. 5A-5Care three examples 510, 520 and 530 of cyclic scan that have this property of a 4×4 block of pixels. Alternative embodiments may consider rotations and flips of these basic patterns illustrated inFIG. 5A-5C. Other embodiments may use many other cyclic scan patterns.

With unified scaling illustrated in Table 6, the improved 12-bit compression module312codes the residual pixel value P using unsigned 7-bit integers because the residual is computed against the minimum sample. With DPCM coding, the delta values need to be represented as signed 7-bit integers. If using twos-complement representation, a 7-bit integer can represent the integers between −64 and 63, inclusive. However, if the previous sample is the minimum sample, for instance, then using twos-complement is inefficient because half of the range is not being used.

In one embodiment, the improved 12-bit compression module312uses excess-N representation, where N is adapted per coded sample to maximize the range of representable differences. The same technique can be applied when the previous sample is near the maximum sample. Since the unified scaling scheme does not signal the maximum sample value, the improved 12-bit compression module312needs to signal the maximum sample value to increase the range of representable differences when the previous sample is near the maximum value. Due to a limited bit budget, instead of coding the maximum, the improved 12-bit compression module312codes the difference between the maximum and minimum, which can be used as the range of representable differences. The range can be further quantized to reduce the number of bits needed to code it.

The following is an example pseudo code of delta coding of residual pixel values with an adaptive excess-N representation.

An example compression format for N-to-8 compression using unified scaling with DPCM coding is shown below in Table 10, where N is the internal bit-depth and can range from 9 to 14, inclusive.

Using unified scaling with DPCM coding, the improved 12-bit compression module312needs to determine both the scaling factor S and whether to use DPCM. It is possible to use a higher S when DPCM coding is used for the residual. Therefore, the improved 12-bit compression module312needs to signal per block when to use DPCM mode. Since the range is only needed in DPCM mode, in one embodiment, the improved 12-bit compression module312combines the coding of DPCM flag and range into one codeword.

From Table 10, the improved 12-bit compression module312can compute the number of bits used to code each block as follows:
bits(N)=k+┌ log2(N−6)┐+N+109,
where the values of k for each value of N, given a budget of 128 bits, are tabulated below in Table 11.

TABLE 11Number of Bits to Code S and dpcm_flag_rangeNbits for Sk = bits for dpcm_flag_range92810271135123413331432
The improved 12-bit compression module312codes the k most significant bits of range and reserves one codeword to signal dpcm_flag=0. In a preferred embodiment, the improved 12-bit compression module312reserves the codeword 0 to signal dpcm_flag=0. Any other codeword signals dpcm_flag=1.

In one embodiment, the improved 12-bit compression module312first computes S for the non-DPCM case as illustrated in Table 6. The improved 12-bit compression module312then determines whether to use a smaller value of S with DPCM coding. Table 12 below illustrates a pseudo code of an embodiment of unified scaling with DPCM compression and Table 13 is the corresponding pseudo code of an embodiment of unified scaling with DPCM decompression. Parameter k represents number of bits needed to code dpcm_flag_range defined in Table 11.

FIG. 6is a flow chart of encoding an input video sequence using improved IBDI encoding. Initially, the improved IBDI encoder310receives610an input video of multiple video frames, each of which has multiple pixels. Each pixel of the input video has an n-bit depth (e.g., 8-bit depth). The improved IBDI encoder310optionally increases620the bit-depth of each pixel by a predetermined factor, (e.g., increasing the 8-bit depth by a factor of 4). In another embodiment, the bit-depth of each pixel (e.g., 12-bit pixel) of the input video is not increased. Each pixel with the increased bit depth or with original bit depth is compressed and reconstucted630. The improved IBDI encoder310calculates640an adaptive offset for each reconstructed pixel of the input video, and applies650the calculated adaptive offset to each reconstructed pixel of the input video to generate an encoded input video. The improved IBDI encoder310optimizes660the encoding of the input video and outputs670the encoded input video to other video processing modules, e.g., the improved IBDI decoder320, for further processing.

There are various embodiments to calculate the adaptive offset for each de-quantized pixel of the input video. In one embodiment, the improved IBDI encoder310constrains the adaptive offset value to a predetermined range of values and calculates the adaptive offset based on two sub-offset values, each of which is determined using partial bits of the total bits available for encoding the pixels of the input video.

In another embodiment, the improved IBDI encoder310optimizes the IBDI encoding using the unified scaling method illustrated in Table 8. The improved IBDI encoder310computes residual pixel value for each pixel of the input video, where the residual pixel value is the difference between the original pixel value and a quantized minimum pixel value. The improved IBDI encoder310further scales the residual pixel value of each pixel of the input video based on a scaling factor such that the differences of the scaled residual pixel values are within a range of representable differences.

The disclosed embodiments beneficially allow for a system and methods for improving coding efficiency of compression with internal bit depth increase. Using adaptive offsets and unified scaling potentially reduces encoding distortion by better adapting scaling offsets to local statistics. Using DPCM coding with the unified scaling further improves the coding efficiency of compression with IBDI.

Additional Configuration Considerations

The various operations of example methods, e.g., described withFIG. 6, may be performed, at least partially, by one or more processors, e.g.,102, that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions. The modules referred to herein may, in some example embodiments, comprise processor-implemented modules.