Picture quality oriented rate control for low-latency streaming applications

A derived quantization parameter for a section of a currently encoding picture of a plurality of pictures is incremented to produce an updated quantization parameter when the derived quantization parameter is less than a minimum quantization parameter for the currently encoding picture. The section is then encoded using the updated quantization parameter. It is emphasized that this abstract is provided to comply with the rules requiring an abstract that will allow a searcher or other reader to quickly ascertain the subject matter of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims.

FIELD OF THE DISCLOSURE

Aspects of the present disclosure are related to encoding of digital data for streaming applications. In particular, the present disclosure is related to rate control for streaming applications.

BACKGROUND

Digital signal compression (sometimes referred to as video coding or video encoding) is widely used in many multimedia applications and devices. Digital signal compression using a coder/decoder (codec) allows streaming media, such as audio or video signals to be transmitted over the Internet or stored on compact discs. A number of different standards of digital video compression have emerged, including H.261, H.263; DV; MPEG-1, MPEG-2, MPEG-4, VC1; AVC (H.264), and HEVC (H.265). These standards, as well as other video compression technologies, seek to efficiently represent a video frame picture by eliminating the spatial and temporal redundancies in the picture and among successive pictures. Through the use of such compression standards, video contents can be carried in highly compressed video bit streams, and thus efficiently stored in disks or transmitted over networks.

It is within this context that aspects of the present disclosure arise.

DESCRIPTION OF THE DRAWINGS

Introduction

Aspects of the present disclosure are directed to picture quality oriented rate control for low latency streaming applications, such as online video gaming.

Before describing picture quality oriented rate control in accordance with aspects of the present disclosure, it is useful to understand how digital pictures, e.g., video pictures are encoded for streaming applications. Furthermore, it is useful to understand how encoded streaming digital picture data is decode, since the manner in which a picture or portion of a picture is encoded affects how it is to be decoded.

By way of example, and not by way of limitation, as shown inFIG. 1, a single picture100(e.g., a digital video frame) may be broken down into one or more sections. As used herein, the term “section” can refer to a group of one or more pixels within the picture100. A section can range from a single pixel within the picture, up to the whole picture. Non-limiting examples of sections include slices (e.g., macroblock rows)102, macroblocks104, sub-macroblocks106, blocks108and individual pixels110. Other examples include Coding Tree Blocks (CTB) and other section types that are specific to the HEVC (H.265) coding standard. As illustrated inFIG. 1, each slice102contains one or more rows of macroblocks104or portions of one or more such rows. The number of macroblocks in a row depends on the size of the macroblocks and the size and resolution of the picture100. For example, if each macroblock contains sixteen by sixteen pixels then the number of macroblocks in each row may be determined by dividing the width of the picture100(in pixels) by sixteen. Each macroblock104may be broken down into a number of sub-macroblocks106. Each sub-macroblock106may be broken down into a number of blocks108and each block may contain a number of pixels110. By way of example, and without limitation of the invention, in a common video coding scheme, each macroblock104may be broken down into four sub-macroblocks106. Each sub-macroblock may be broken down into four blocks108and each block may contain a four by four arrangement of sixteen pixels110.

It is noted that each picture may be either a frame or a field. A frame refers to a complete image. A field is a portion of an image used for to facilitate displaying the image on certain types of display devices. Generally, the pixels in an image are arranged in rows. To facilitate display an image may sometimes be split by putting alternate rows of pixels into two different fields. The rows of pixels in the two fields can then be interlaced to form the complete image. For some display devices, such as cathode ray tube (CRT) displays, the two fields may simply be displayed one after the other in rapid succession. The afterglow of the phosphors or other light emitting elements used to illuminate the pixels in the display combined with the persistence of vision results in the two fields being perceived as a continuous image. For certain display devices, such as liquid crystal displays, it may be necessary to interlace the two fields into a single picture before being displayed. Streaming data representing encoded images typically includes information indicating whether the image is a field or a frame. Such information may be included in a header to the image.

Modern video coder/decoders (codecs), such as MPEG2, MPEG4 and H.264 generally encode video frames as one of three basic types known as Intra-Frames, Predictive Frames and Bipredicitve Frames, which are typically referred to as I-frames, P-frames and B-frames respectively.

An I-frame is a picture coded without reference to any picture except itself. I-frames are used for random access and are used as references for the decoding of other P-frames or B-frames. I-frames may be generated by an encoder to create random access points (to allow a decoder to start decoding properly from scratch at a given picture location). I-frames may be generated when differentiating image details prohibit generation of effective P or B frames. Because an I-frame contains a complete picture, I-frames typically require more bits to encode than P-frames or B-frames. Video frames are often encoded as I-frames when a scene change is detected in the input video.

P-frames require the prior decoding of some other picture(s) in order to be decoded. P-frames typically require fewer bits for encoding than I-frames. A P-frame contains encoded information regarding differences relative to a previous I-frame in decoding order. A P-frame typically references the preceding I-frame in a Group of Pictures (GoP). P-frames may contain both image data and motion vector displacements and combinations of the two. In some standard codecs (such as MPEG-2), P-frames use only one previously-decoded picture as a reference during decoding, and require that picture to also precede the P-frame in display order. In H.264, P-frames can use multiple previously-decoded pictures as references during decoding, and can have any arbitrary display-order relationship relative to the picture(s) used for its prediction.

B-frames require the prior decoding of either an I-frame or a P-frame in order to be decoded. Like P-frames, B-frames may contain both image data and motion vector displacements and/or combinations of the two. B-frames may include some prediction modes that form a prediction of a motion region (e.g., a segment of a frame such as a macroblock or a smaller area) by averaging the predictions obtained using two different previously-decoded reference regions. In some codecs (such as MPEG-2), B-frames are never used as references for the prediction of other pictures. As a result, a lower quality encoding (resulting in the use of fewer bits than would otherwise be used) can be used for such B pictures because the loss of detail will not harm the prediction quality for subsequent pictures. In other codecs, such as H.264, B-frames may or may not be used as references for the decoding of other pictures (at the discretion of the encoder). In H.264, there are two types of B-frame, a reference B-frame and non-reference B-frame. A reference B-frame can be used as a reference frame for B-frame coding and a non-reference B-frame cannot. Some codecs (such as MPEG-2), use exactly two previously-decoded pictures as references during decoding, and require one of those pictures to precede the B-frame picture in display order and the other one to follow it. In other codecs, such as H.264, a B-frame can use one, two, or more than two previously-decoded pictures as references during decoding, and can have any arbitrary display-order relationship relative to the picture(s) used for its prediction. B-frames typically require fewer bits for encoding than either I-frames or P-frames.

As used herein, the terms I-frame, B-frame and P-frame may be applied to any streaming data units that have similar properties to I-frames, B-frames and P-frames, e.g., as described above with respect to the context of streaming video.

By way of example, and not by way of limitation, digital pictures may be encoded according to a generalized method200as illustrated inFIG. 2A. The encoder receives a plurality of digital images201and encodes each image. Encoding of the digital picture201may proceed on a section-by-section basis. The encoding process for each section may optionally involve padding202, image compression204and motion compensation206. To facilitate a common process flow for both intra-coded and inter-coded pictures, all un-decoded pixels within a currently processing picture201may be padded with temporary pixel values to produce a padded picture, as indicated at202. The padding may proceed, e.g., as described above in U.S. Pat. No. 8,711,933, which is incorporated herein by reference. The padded picture may be added to a list of reference pictures203stored in a buffer. Padding the picture at202facilitates the use of a currently-processing picture as a reference picture in subsequent processing during image compression204and motion compensation206. Such padding is described in detail in commonly-assigned U.S. Pat. No. 8,218,641, which is incorporated herein by reference.

As used herein, image compression refers to the application of data compression to digital images. The objective of the image compression204is to reduce redundancy of the image data for a give image201in order to be able to store or transmit the data for that image in an efficient form of compressed data. The image compression204may be lossy or lossless. Lossless compression is sometimes preferred for artificial images such as technical drawings, icons or comics. This is because lossy compression methods, especially when used at low bit rates, introduce compression artifacts. Lossless compression methods may also be preferred for high value content, such as medical imagery or image scans made for archival purposes. Lossy methods are especially suitable for natural images such as photos in applications where minor (sometimes imperceptible) loss of fidelity is acceptable to achieve a substantial reduction in bit rate.

Examples of methods for lossless image compression include, but are not limited to Run-length encoding—used as default method in PCX and as one of possible in BMP, TGA, TIFF, Entropy coding, adaptive dictionary algorithms such as LZW—used in GIF and TIFF and deflation—used in PNG, MNG and TIFF. Examples of methods for lossy compression include reducing the color space of a picture201to the most common colors in the image, Chroma subsampling, transform coding, and fractal compression.

In color space reduction, the selected colors may be specified in the color palette in the header of the compressed image. Each pixel just references the index of a color in the color palette. This method can be combined with dithering to avoid posterization. Chroma subsampling takes advantage of the fact that the eye perceives brightness more sharply than color, by dropping half or more of the chrominance information in the image. Transform coding is perhaps the most commonly used image compression method. Transform coding typically applies a Fourier-related transform such as a discrete cosine transform (DCT) or the wavelet transform, followed by quantization and entropy coding. Fractal compression relies on the fact that in certain images, parts of the image resemble other parts of the same image. Fractal algorithms convert these parts, or more precisely, geometric shapes into mathematical data called “fractal codes” which are used to recreate the encoded image.

The image compression204may include region of interest coding in which certain parts of the image201are encoded with higher quality than others. This can be combined with scalability, which involves encoding certain parts of an image first and others later. Compressed data can contain information about the image (sometimes referred to as meta information or metadata) which can be used to categorize, search or browse images. Such information can include color and texture statistics, small preview images and author/copyright information.

By way of example, and not by way of limitation, during image compression at204the encoder may search for the best way to compress a block of pixels. The encoder can search all of the reference pictures in the reference picture list203, including the currently padded picture, for a good match. If the current picture (or subsection) is coded as an intra picture, (or subsection) only the padded picture is available in the reference list. The image compression at204produces a motion vector MV and transform coefficients207that are subsequently used along with one or more of the reference pictures (including the padded picture) during motion compensation at206.

The image compression204generally includes a motion search MS for a best inter prediction match, an intra search IS for a best intra prediction match, an inter/intra comparison C to decide whether the current macroblock is inter-coded or intra-coded, a subtraction S of the original input pixels from the section being encoded with best match predicted pixels to calculate lossless residual pixels205. The residual pixels then undergo a transform and quantization XQ to produce transform coefficients207. The transform is typically based on a Fourier transform, such as a discrete cosine transform (DCT).

The transform outputs a set of coefficients, each of which is a weighting value for a standard basis pattern. When combined, the weighted basis patterns re-create the block of residual samples. The output of the transform, a block of transform coefficients, is quantized, i.e. each coefficient is divided by an integer value. Quantization reduces the precision of the transform coefficients according to a quantization parameter (QP). Typically, the result is a block in which most or all of the coefficients are zero, with a few non-zero coefficients. Setting QP to a high value means that more coefficients are set to zero, resulting in high compression at the expense of poor decoded image quality. For a low QP value, more non-zero coefficients remain after quantization, resulting in better decoded image quality but lower compression. Conversely, for a high QP value, fewer non-zero coefficients remain after quantization, resulting in higher image compression but lower image quality.

Since QP controls bit usage in encoding, many encoding programs utilize a rate controller that adjusts QP in order to achieve a desired bitrate. The use of such a rate controller may be understood by referring toFIG. 2BandFIG. 2C. As seen inFIG. 2B, a video encoding system220may include an encoder222and a rate controller224. The encoder receives uncompressed source data (e.g., an input video) and produces compressed output. The encoder222may be configured to implement the coding method200depicted inFIG. 2A. As noted above, the video coding method200uses a QP value that affects the bit usage for encoding a video section and therefore affects the bitrate. Generally, lower bit usage results in a higher bitrate. The rate controller224determines a QP value based on a demanded bitrate, which may be specified by an external application. The encoder222uses the QP value determined by the rate controller and determines the actual resulting bit usage and bit rate. The rate controller224can use the actual bit rate to adjust the QP value in a feedback loop.

A relationship between the bitrate and the value of the QP depends partly on the complexity of the image being encoded, as shown inFIG. 2C. The bitrate versus QP relationship can be expressed in terms of a set of curves with different curves for different levels of complexity. The heart of the algorithm implemented by the rate controller is a quantitative model describing a relationship between QP, actual bitrate and some measure of complexity, e.g., as depicted inFIG. 2C. The relevant bitrate and complexity are generally associated only with the differences between source pixels and predicted pixels (often referred to as residuals) because the quantization parameter QP can only influence the detail of information carried in the transformed residuals.

Complexity generally refers to amount of spatial variation within a picture or part of the picture. On a local level, e.g., block or macroblock level, the spatial variation may be measured by the variance of the pixel values within the relevant section. However, for a video sequence, complexity may also relate to the temporal variation of a scene of a sequence of images. For example, a video sequence consists of one object having substantial spatial variation that translates slowly across the field of view, may not require very many bits because temporal prediction can easily capture the motion using a single reference picture and a series of motion vectors. Although it is difficult to define an inclusive video complexity metric that is also easy to calculate, the Mean Average Difference (MAD) of the prediction error (difference between source pixel value and predicted pixel value) is often used for this purpose.

It is noted that the quantization parameter QP may be determined from multiple factors including, but not limited to the picture type of the source picture, a complexity of the source picture, an estimated target number of bits and an underlying rate distortion model. For example, QP may be determined on a section-by-section basis using a variation for a section of the currently encoding picture, e.g., a section (e.g., MB) variance. Alternatively, QP for a currently encoding section may be determined using an actual bit count for encoding a co-located section (e.g., MB) in a previous frame. Examples of such QP level calculations are described, e.g., in commonly assigned U.S. Patent Application Publication No. 2011/0051806, to Hung-Ju Lee, which is incorporated herein by reference.

Motion search and prediction depend on the type of picture being encoded. Referring again toFIG. 2A, if an intra picture is to be coded, the motion search MS and inter/intra comparison C are turned off. However, in embodiments of the present invention, since the padded picture is available as a reference, these functions are not turned off. Consequently, the image compression204is the same for intra-coded pictures and inter-coded pictures.

The motion search MS may generate a motion vector MV by searching the picture201for a best matching block or macroblock for motion compensation as is normally done for an inter-coded picture. If the current picture201is an intra-coded picture, by contrast, existing codecs typically do not allow prediction across pictures. Instead all motion compensation is normally turned off for an intra picture (e.g., I-frame) and the picture coded by generating transform coefficients and performing pixel prediction. In some implementations, however, an intra picture may be used to do inter prediction by matching a section in the current picture to another offset section within that same picture. The offset between the two sections may be coded as a motion vector MV′ that can be used that for motion compensation at206. By way of example, the encoder may attempt to match a block or macroblock in an intra picture with some other offset section in the same picture then code the offset between the two as a motion vector. The codec's ordinary motion vector compensation for an “inter” picture may then be used to do motion vector compensation on an “intra” picture. Certain existing codecs have functions that can convert an offset between two blocks or macroblocks into a motion vector, which can be followed to do motion compensation at206. However, these functions are conventionally turned off for encoding of intra pictures. In embodiments of the present invention, the codec may be instructed not to turn off such “inter” picture functions for encoding of intra pictures.

As used herein, motion compensation refers to a technique for describing a picture in terms of the transformation of a reference image to a currently processing image. In general, the motion compensation206acts as a local decoder within the encoder implementing the encoding process200. Specifically, the motion compensation206includes inter prediction IP1and (optionally) intra prediction IP2to get predicted pixels PP using the motion vector MV or MV′ from the image compression204and reference pixels from a picture in the reference list. Inverse quantization and inverse transformation IQX using the transform coefficients207from the image compression204produce lossy residual pixels205L which are added to the predicted pixels PP to generate decoded pixels209. The decoded pixels209are inserted into the reference picture and are available for use in image compression204and motion compensation206for a subsequent section of the currently-processing picture201. After the decoded pixels have been inserted, un-decoded pixels in the reference picture may undergo padding202.

In a conventional encoder, if the current picture is intra coded, the inter-prediction portions of motion compensation206are turned off because there are no other pictures that can be used for motion compensation. However, in embodiments of the present invention, by contrast, motion compensation may be performed on any picture201independent of whether a particular picture is to be inter-coded or intra-coded. In embodiments of the present invention, the encoder implementing the method200may be modified to add the padded picture to the reference picture list203and the inter-prediction portions of the motion compensation206are not turned off, even if the currently processing image is to be intra coded. As a result, the process flow for both inter coded sections and intra coded sections is the same during motion compensation206. The only major difference is the selection of the reference picture to be used for encoding.

By way of example, and not by way of limitation, in one type of motion compensation, known as block motion compensation (BMC), each image may be partitioned into blocks of pixels (e.g. macroblocks of 16×16 pixels). Each block is predicted from a block of equal size in the reference frame. The blocks are not transformed in any way apart from being shifted to the position of the predicted block. This shift is represented by a motion vector MV. To exploit the redundancy between neighboring block vectors, (e.g. for a single moving object covered by multiple blocks) it is common to encode only the difference between a current and previous motion vector in a bit-stream. The result of this differencing process is mathematically equivalent to a global motion compensation capable of panning. Further down the encoding pipeline, the method200may optionally use entropy coding208to take advantage of the resulting statistical distribution of the motion vectors around the zero vector to reduce the output size.

It is possible to shift a block by a non-integer number of pixels, which is called sub-pixel precision. The in-between pixels are generated by interpolating neighboring pixels. Commonly, half-pixel or quarter pixel precision is used. The computational expense of sub-pixel precision is much higher due to the extra processing required for interpolation and on the encoder side, a much greater number of potential source blocks to be evaluated.

Block motion compensation divides up a currently encoding image into non-overlapping blocks, and computes a motion compensation vector that indicates where those blocks come from in a reference image. The reference blocks typically overlap in the source frame. Some video compression algorithms assemble the current image out of pieces of several different reference images in the reference image list203.

The result of the image compression204and motion compensation206and (optionally) entropy coding208is a set of data211referred to for convenience as a coded picture. The motion vector MV, (and/or intra prediction mode motion vector MV′) and transform coefficients207may be included in the coded picture211. Once a digital picture or other form of streaming data has been encoded, the encoded data may be transmitted and then decoded.

FIG. 3illustrates an example of a possible process flow in a method300for decoding of streaming data301that may be used in conjunction with aspects of the present disclosure. This particular example shows the process flow for video decoding, e.g., using the AVC (H.264) standard. The coded streaming data301may initially be stored in a buffer. Where coded streaming data301(e.g., a video data bitstream) has been transferred over a network, e.g., the Internet, the data301may initially undergo a process referred to as network abstraction layer (NAL) decoding, indicated at302. The Network Abstraction Layer (NAL) is a part of streaming data standards, such as the H.264/AVC and HEVC video coding standards. The main goal of the NAL is the provision of a “network-friendly” representation of streaming data for “conversational” (e.g., video telephony) and “non-conversational” (storage, broadcast, or streaming) applications. NAL decoding may remove from the data301information added to assist in transmitting the data. Such information, referred to as a “network wrapper” may identify the data201as video data or indicate a beginning or end of a bitstream, bits for alignment of data, and/or metadata about the video data itself.

In addition, by way of example, the network wrapper may include information about the data301including, e.g., resolution, picture display format, color palette transform matrix for displaying the data, information on the number of bits in each picture, slice or macroblock, as well as information used in lower level decoding, e.g., data indicating the beginning or ending of a slice. This information may be used to determine the number of macroblocks to pass to each of the task groups in a single section. Due to its complexity, NAL decoding is typically done on a picture and slice level. The smallest NAL buffer used for NAL decoding is usually slice sized. The example illustrated inFIG. 3is described in terms of macroblocks and the AVC (H.265) standard. However, these are not limiting features of aspects of the present disclosure. For example, in the latest H265 (HEVC) standard, there is no macroblock concept. Instead, more flexible Coding Unit (CU), Prediction Unit, (PU), Transform Unit (TU) concepts are introduced. Aspects of the present disclosure may be operate in conjunction with such coding standards.

In some embodiments, after NAL decoding at302, the remaining decoding illustrated inFIG. 4may be implemented in three different thread groups or task groups referred to herein as video coded layer (VCL) decoding304, motion vector (MV) reconstruction310and picture reconstruction314. The picture reconstruction task group214may include pixel prediction and reconstruction316and post processing320. In some embodiments of the present invention, these tasks groups may be chosen based on data dependencies such that each task group may complete its processing of all the macroblocks in a picture (e.g., frame or field) or section before the macroblocks are sent to the next task group for subsequent processing.

Certain coding standards may use a form of data compression that involves transformation of the pixel information from a spatial domain to a frequency domain. One such transform, among others, is known as a discrete cosine transform (DCT). The decoding process for such compressed data involves the inverse transformation from the frequency domain back to the spatial domain. In the case of data compressed using DCT, the inverse process is known as inverse discrete cosine transformation (IDCT). The transformed data is sometimes quantized to reduce the number of bits used to represent numbers in the discrete transformed data. For example, numbers 1, 2, 3 may all be mapped to 2 and numbers 4, 5, 6 may all be mapped to 5. To decompress the data a process known as inverse quantization (IQ) is used before performing the inverse transform from the frequency domain to the spatial domain. The data dependencies for the VCL IQ/IDCT decoding process304are typically at the macroblock level for macroblocks within the same slice. Consequently results produced by the VCL decoding process304may be buffered at the macroblock level.

VCL decoding304often includes a process referred to as Entropy Decoding306, which is used to decode the VCL syntax. Many codecs, such as AVC(H.264), use a layer of encoding referred to as entropy encoding. Entropy encoding is a coding scheme that assigns codes to signals so as to match code lengths with the probabilities of the signals. Typically, entropy encoders are used to compress data by replacing symbols represented by equal-length codes with symbols represented by codes proportional to the negative logarithm of the probability. AVC(H.264) supports two entropy encoding schemes, Context Adaptive Variable Length Coding (CAVLC) and Context Adaptive Binary Arithmetic Coding (CABAC). Since CABAC tends to offer about 10% more compression than CAVLC, CABAC is favored by many video encoders in generating AVC(H.264) bitstreams. Decoding the entropy layer of AVC(H.264)-coded data streams can be computationally intensive and may present challenges for devices that decode AVC(H.264)-coded bitstreams using general purpose microprocessors. For this reason, many systems use a hardware decoder accelerator.

In addition to Entropy Decoding306, the VCL decoding process304may involve inverse quantization (IQ) and/or inverse discrete cosine transformation (IDCT) as indicated at308. These processes may decode the headers309and data from macroblocks. The decoded headers309may be used to assist in VCL decoding of neighboring macroblocks.

VCL decoding304may be implemented at a macroblock level data dependency frequency. Specifically, different macroblocks within the same slice may undergo VCL decoding in parallel and the results may be sent to the motion vector reconstruction task group210for further processing.

Subsequently, all macroblocks in the picture or section may undergo motion vector reconstruction310. The MV reconstruction process310may involve motion vector reconstruction312using headers from a given macroblock311and/or co-located macroblock headers313. A motion vector describes apparent motion within a picture. Such motion vectors allow reconstruction of a picture (or portion thereof) based on knowledge of the pixels of a prior picture and the relative motion of those pixels from picture to picture. Once the motion vector has been recovered pixels may be reconstructed at316using a process based on residual pixels from the VCL decoding process304and motion vectors from the MV reconstruction process310. The data dependency frequency (and level of parallelism) for the MV depends on whether the MV reconstruction process310involves co-located macroblocks from other pictures. For MV reconstruction not involving co-located MB headers from other pictures the MV reconstruction process310may be implemented in parallel at the slice level or picture level. For MV reconstruction involving co-located MB headers the data dependency frequency is at the picture level and the MV reconstruction process310may be implemented with parallelism at the slice level.

The results of motion vector reconstruction310are sent to the picture reconstruction task group314, which may be parallelized on a picture frequency level. Within the picture reconstruction task group314all macroblocks in the picture or section may undergo pixel prediction and reconstruction316in conjunction with de-blocking320. The pixel prediction and reconstruction task316and the de-blocking task320may be parallelized to enhance the efficiency of decoding. These tasks may be parallelized within the picture reconstruction task group314at a macroblock level based on data dependencies. For example, pixel prediction and reconstruction316may be performed on one macroblock and followed by de-blocking320. Reference pixels from the decoded picture obtained by de-blocking320may be used in pixel prediction and reconstruction316on subsequent macroblocks. Pixel prediction and reconstruction318produces decoded sections319(e.g. decoded blocks or macroblocks) that include neighbor pixels which may be used as inputs to the pixel prediction and reconstruction process318for a subsequent macroblock. The data dependencies for pixel prediction and reconstruction316allow for a certain degree of parallel processing at the macroblock level for macroblocks in the same slice.

The post processing task group320may include a de-blocking filter322that is applied to blocks in the decoded section319to improve visual quality and prediction performance by smoothing the sharp edges which can form between blocks when block coding techniques are used. The de-blocking filter322may be used to improve the appearance of the resulting de-blocked sections324.

The decoded section319or de-blocked sections324may provide neighboring pixels for use in de-blocking a neighboring macroblock. In addition, decoded sections319including sections from a currently decoding picture may provide reference pixels for pixel prediction and reconstruction318for subsequent macroblocks. It is during this stage that pixels from within the current picture may optionally be used for pixel prediction within that same current picture as described above, independent of whether the picture (or subsections thereof) is inter-coded or intra-coded. De-blocking320may be parallelized on a macroblock level for macroblocks in the same picture.

The decoded sections319produced before post processing320and the post-processed sections324may be stored in the same buffer, e.g., the output picture buffer depending on the particular codec involved. It is noted that de-blocking is a post processing filter in H.264. Because H.264 uses pre-de-blocking macroblock as reference for neighboring macroblocks intra prediction and post-de-blocking macroblocks for future picture macroblocks inter prediction. Because both pre- and post-de-blocking pixels are used for prediction, the decoder or encoder has to buffer both pre-de-blocking macroblocks and post-de-blocking macroblocks. For most low cost consumer applications, pre-de-blocked pictures and post-de-blocked pictures share the same buffer to reduce memory usage. For standards that pre-date H.264, such as MPEG2 or MPEG4 except MPEG4 part 10, (note: H.264 is also called MPEG4 part 10), only pre-post-processing macroblocks (e.g., pre-de-blocking macroblocks) are used as reference for other macroblock prediction. In such codecs, a pre-filtered picture may not share the same buffer with a post filtered picture.

Thus, for H.264, after pixel decoding, the decoded section319is saved in the output picture buffer. Later, the post processed sections324replace the decoded sections319in the output picture buffer. For non-H.264 cases, the decoder only saves decoded sections319in the output picture buffer. The post processing is done at display time and the post processing output may not share the same buffer as the decoder output picture buffer.

Picture Quality Oriented Rate Control Method

Aspects of the present disclosure address two issues found in traditional rate control used in low latency streaming applications in which the encoded frame size is capped by a limited or non VBV buffer to reduce network jitter. As is generally known to those skilled in the art VBV refers to Video Buffer Verifier, a theoretical buffer model used to ensure that an encoded video stream can be correctly buffered, and played back at the decoder device. A larger VBV buffer size usually improves quality in high action sequences, but tends to cause higher bitrate spikes.

The first issue is that a top priority of traditional rate control VBV buffering mechanism is meeting the target bitcount. This sometimes results in too many bits being used to encode static scenes or scenes with little motion, which results in use of a large number of bits to encode the scene without significant visual quality improvement. The second issue is that scenes with little or no motion tend to have a large number of intra-coded sections. Too many intra-coded sections in a frame causes a false detection of a scene change, which causes the entire frame to encoded as an I-frame. Aspects of the present disclosure address the above two issues by selectively limiting the quality associated with sections of a frame for a static or low-motion scene. Aspects of the present disclosure can achieve significant savings in the number of bits used to encode pictures while maintaining sufficiently good quality.

FIG. 4illustrates an algorithm400for implementing a method for picture quality oriented rate control in the context of streaming digital picture encoding. Certain abbreviations, notations and acronyms are used in the discussion ofFIG. 4below. As used herein, the following abbreviations and acronyms have the following meanings:

Coding unit: a portion of larger set of unencoded streaming data that is to be encoded. In the context of encoding of streaming picture data (e.g., streaming video) the coding units are sometimes referred to herein as sections.

Line: a row of sections (e.g., macroblocks, sub-macroblocks, blocks, pixels) is denoted a line

Line index 1 is the index to a line in a picture and is set to the current section number x divided by the picture width in sections.

SATD: Sum of Absolute Transformed Differences (SATD) is a widely used video quality metric used for block-matching in motion estimation for video compression. It works by taking a frequency transform, usually a Hadamard transform, of the differences between the pixels in the original block and the corresponding pixels in the block being used for comparison. The transform itself is often of a small block rather than the entire macroblock. For example, in the x264 coding standard, a series of 4×4 blocks are transformed rather than doing a more processor-intensive 16×16 transform. Alternatives to the Hadamard transform (also known as the Walsh function) include the Discrete-time Fourier transform (DTFT), the discrete Fourier transform (DFT), the discretized short time Fourier transform (STFT), the discrete sine transform (DST), the discrete cosine transform (DCT), regressive discrete Fourier series, in which the period is determined by the data rather than fixed in advance, discrete chebyshev transforms, generalized DFT (GDFT), Z-transform (a generalization of the DTFT), the Modified discrete cosine transform (MDCT), and the Discrete Hartley transform (DHT).

SATDav(l): average SATD calculated from the 1st line to the current line indexed by l in the previous predictive frame (e.g., P frame)

SATDav(x): average SATD calculated from the 1st section to the current section x in the current predictive frame.

SATD variation: either SATDav(l)/SATDav(x) or SATDav(x)/SATDav(l), depending on whether SATDav(x) is less than SATDav(l).

QP(x): quantization parameter for a section x of a currently encoding picture (e.g., a predictive frame).

QP′min(l): minimum QP from the 1 st line up to the current line indexed by l in the previous predictive frame.

QPmin(l): minimum QP up to the current line in the current predictive frame.

QPmin(x): the minimum QP up to the current section in the current predictive frame.

W: The width of a frame expressed as a number of sections.

TH1: a threshold to control SATD variation.

TH2: a threshold to control SATD.

The proposed algorithm400illustrated inFIG. 4can be used to determine if further refinement of QP(x) is needed. Note that the process is performed for P-frames only.

As shown inFIG. 4, encoding for a frame begins as indicated at402. The values of QPmin(l) and QPmin(x) may be initialized, as indicated at404. By way of example, and not by way of limitation, the value of QPmin(l) may be initialized to l and the value of QPmin(x) may be initialized to some number K that is greater than the maximum QP value permitted by the coding standard. For example, for the H.264 coding standard QP(x) ranges from 0 to 51 and K would therefore be greater than 51.

The implementation in the example illustrated inFIG. 4uses SATD as a video quality metric. However, aspects of the present disclosure are not limited to such implementations. One advantage of using SATD is that it adds almost no overhead since it is available in this context. In alternative implementations, other video quality variance metrics may be used. For example, section variance between the currently encoding picture and the previously encoded picture may be used. In practice, using section variance (e.g., macroblock variance) might produce a better result than using SATD, but at a cost of additional overhead for its intensive computations.

By way of example, and not by way of limitation, the section variance may be defined as the sum of the squares of the of the differences between each pixel value Pixel(i,j) in a section and the mean pixel value for the section divided by the number of pixels N in the section. For a section of (m+1)×(n+1)=N pixels, this may be expressed mathematically as:

VAR=∑i=0i=m⁢∑j=0j=n⁢(Pixel⁡(i,j))-MEAN)2/N,
where MEAN is given by:

By way of numerical example, if the section is a 16×16 pixel macroblock, m=n=15 and N=256.

The concept may be generalized to sections of arbitrary size.

Similar to the case with SATD as a video quality metric, the section variation may be a ratio of a first section variance to a second section variance. The first section variance may be an average variance calculated in a previous predicted picture from a first line to a current line containing the current section and the second average section variance may be an average variance calculated from the first section of the currently encoding picture to the current section of the currently encoding picture.

The quantization parameter QP(x) is then derived for the section x in the normal manner, as indicated at406, e.g., using a rate control algorithm suitable for use with the coding standard, as discussed above with respect toFIG. 2BandFIG. 2C. A QP refinement routine may then be called, as indicated at408. The QP refinement routine uses parameters409, such as TH1and TH2. In the QP refinement routine, if, at410, the current section x is at the beginning of a line, QP(x) is updated by setting it equal to the maximum of QP(x) and QP1,(l), as indicated at426. If the current section x is not at the beginning of a line and the SATD variation is less than TH1and SATDav(x) is less than or equal to TH2at422, QPmin(l) is set equal to QP′min(l)+1 as indicated at424before QP(x) is updated at426.

There are a number of ways to determine whether the variance (e.g., SATD variation) is less than TH1. By way of example, and not by way of limitation, inFIG. 4, if the current section x is not the first section in a line a flag (noMo) may be set equal to zero, as indicated at412. If, at414, SATDav(x) is less than SATDav(l), the ratio SATDav(x)/SATDav(l) is compared to TH1at420otherwise, the inverse ratio SATDav(l)/SATDav(x) is compared to TH1at416. If either ratio SATDav(x)/SATDav(l) or SATDav(l)/SATDav(x) is less than TH1the noMo flag is reset to 1 at418, which indicates at422that the SATD variation is less than TH1.

The values of TH1and TH2can be determined empirically. For example, if experimental results show that a variation (e.g., SATD variation) of 30% or less corresponds to known cases of no motion, then the algorithm400may use a value of 30% for TH1. By way of example, and not by way of limitation, TH2may be calculated from the target number of bits per frame. One possible formula for TH2has the form TH2=A·(NBFt/B)−C, where NBFtis the target number of bits per frame and A, B, and C are constants determined empirically. In some implementations, the formula may further restrict TH2to lie between upper and lower limits TH2minand TH2max. If the formula produces a value below TH1min, TH1=TH1Likewise, if the formula produces a value above TH1max, TH1=TH1max.

By way of non-limiting numerical example, TH2may be calculated using:
TH2=3·(NBf/100)−125 and limited to a range of [250,500]

Once QP(x) has been updated at426the current section x can be encoded using the value of QP(x). As discussed above, a larger QP value reduces the bit usage to encode a section. Therefore, incrementing QP(x) at426tends to reduce the bit usage for encoding section x at428. If, at430, QP(x) is less than QPmin(x), the value of QPmin(x) may be adjusted to the current value of QP(x), as indicated at432. Furthermore, the value of QP′min(l) may be set equal to the value of QPmin(x), as indicated at434. The value of QP′min(l) may be stored in a table and the value may be updated as the method400proceeds.

The process described above may be repeated for each section in the picture, as indicated at436and438until the last section at which encoding of the next picture may be triggered at440and the process may begin again with the first section of the next picture.

By way of example and not by way of limitation, each section may be a macroblock (MB) and the method depicted inFIG. 4may be implemented according to the pseudo-code below in which:

MBline is a row of macroblocks, also sometimes denoted as a macroblock line.

MBline_index is the index to a MBline in a frame, and is set to the current MB number divided by the frame width in macroblocks, e.g., 0≤MBline_index≤33 for 960×540p video;

avg_SATD_mblines[MBline_index] is the average SATD calculated from the 1st MBline to the current MBline indexed by MBline_index in the previous predictive frame (e.g., P frame);

SATD_per_mbline is the average SATD calculated from the 1st MB to the current MB in the current predictive frame;

min_qp_mblines [MBline_index] is the minimum QP from the 1st MBline up to the current MBline indexed by MBline_index in the previous predictive frame;

mb_minqp_mbline is the minimum QP up to the current MBline in the current predictive frame; and

mb_minqp is the minimum QP up to the current MB in the current predictive frame.

mquant is the QP for a macroblock x.

Assuming mquant is derived by a rate controller, e.g., as described above with respect toFIG. 2BandFIG. 2C, the proposed algorithm will determine if the further refinement of mquant is needed. Note that the process is performed for P frame only. The pseudo code of the algorithm is shown below and MB_QP_refinement is called for every MB.

Aspects of the present disclosure include systems configured to implement quantization parameter (QP) updating for picture quality oriented rate control in conjunction with encoding of digital pictures, as described above. By way of example, and not by way of limitation,FIG. 5illustrates a block diagram of a computer system500that may be used to implement aspects of the present disclosure. The system500generally may include a processor module501and a memory502. The processor module501may include one or more processor cores, e.g., in single core, dual core, quad core, processor-coprocessor, CPU-GPU, or Cell processor architectures.

The memory502may be in the form of an integrated circuit, e.g., RAM, DRAM, ROM, and the like. The memory may also be a main memory that is accessible by all of the processor cores in the processor module501. In some embodiments, the processor module501may have local memories associated with one or more processor cores or one or more co-processors. A codec program503may be stored in the main memory502in the form of processor readable instructions that can be executed on the processor module501. The codec503may be configured to encode digital pictures. By way of example, and not by way of limitation, the codec503may be configured to encode digital pictures or sections of digital pictures as discussed above with respect toFIG. 2A. The codec503may also be configured to decode encoded digital pictures, e.g., as described above with respect toFIG. 3. A rate control program503A may implement rate control by deriving QP, as described with respect toFIGS. 2B-2C. A QP updater503B may adjust QP to implement picture quality oriented rate control, as described above with respect toFIG. 4. The codec503and Rate Controller503A and QP updater503B may be written in any suitable processor readable language, e.g., C, C++, JAVA, Assembly, MATLAB, FORTRAN and a number of other languages.

Input or output data507may be stored in memory502. During execution of the codec503, rate controller503A, and/or QP updater503B, portions of program code, parameters505A,505B and/or data507may be loaded into the memory502or the local stores of processor cores for processing by the processor501. By way of example, and not by way of limitation, the input data507may include video pictures, or sections thereof, before encoding or decoding or at intermediate stages of encoding or decoding. In the case of encoding, the data507may include buffered portions of streaming data, e.g., unencoded video pictures or portions thereof. In the case of decoding, the data507may include input data in the form of un-decoded sections, sections that have been decoded, but not post-processed and sections that have been decoded and post-processed. Such input data may include data packets containing data representing one or more coded sections of one or more digital pictures. By way of example, and not by way of limitation, such data packets may include a set of transform coefficients and a partial set of prediction parameters. These various sections may be stored in one or more buffers. In particular, decoded and/or post processed sections may be stored in an output picture buffer implemented in the memory502. The parameters505A,505B include adjustable parameters505A, such as QPmin(x), QPmin(l), SATDav(x), and SATDav(l) that are re-calculated during the course of encoding, rate control, or QP updating. The parameters also include fixed parameters505B, such as TH1, TH2that remain fixed during the encoding of a picture or over the course of encoding multiple pictures.

The system500may also include well-known support functions510, such as input/output (I/O) elements511, power supplies (P/S)512, a clock (CLK)513and cache514. The apparatus500may optionally include a mass storage device515such as a disk drive, CD-ROM drive, tape drive, or the like to store programs and/or data. The device800may also optionally include a display unit516and user interface unit518to facilitate interaction between the apparatus500and a user. The display unit516may be in the form of a cathode ray tube (CRT) or flat panel screen that displays text, numerals, graphical symbols or images. The user interface518may include a keyboard, mouse, joystick, light pen, or other device that may be used in conjunction with a graphical user interface (GUI). The apparatus500may also include a network interface520to enable the device to communicate with other devices over a network522, such as the internet. These components may be implemented in hardware, software, or firmware, or some combination of two or more of these.

By way of example, and not by way of limitation, the system500may transmit encoded or unencoded streaming data to other devices connected to the network522or receive encoded or unencoded streaming date from such devices via the network interface520. In a particular implementation, encoded streaming data in the form of one or more encoded sections of a digital picture and/or one or more frames of encoded video may be transmitted from the system over the network522. To implement transmitting or receiving streaming data, the processor module may execute instructions implementing a network protocol stack.

By way of example, and not by way of limitation, digital pictures may be generated with a digital camera, which may be part of the user interface518or which may be a separate peripheral coupled to the system500, e.g., via the I/O elements511. According to some aspects, the digital pictures may be generated by a software application executed by the processor module501.

FIG. 6illustrates an example of a computing system600that is configured to generate, encode, and transmit digital pictures in accordance with aspects of the present disclosure. The system600may be configured to render graphics for an application665with in accordance with aspects described above. According to aspects of the present disclosure, the system600may be an embedded system, mobile phone, personal computer, tablet computer, portable game device, workstation, game console, and the like.

The system600may generally include a processor module and a memory configured to implemented aspects of the present disclosure, e.g., by generating digital pictures, encoding the digital pictures by performing a method having features in common with the method ofFIG. 4, and transmitting the encoded pictures over a network. In the illustrated example, the processor module may include a central processing unit (CPU)670, a graphics processing unit (GPU)671, and a memory672. The memory672may optionally include a main memory unit that is accessible to both the CPU and GPU, and portions of the main memory may optionally include portions of the graphics memory650. The CPU670and GPU671may each include one or more processor cores, e.g., a single core, two cores, four cores, eight cores, or more. The CPU670and GPU671may be configured to access one or more memory units using a data bus676, and, in some implementations, it may be useful for the system600to include two or more different buses.

The memory672may include one or more memory units in the form of integrated circuits that provides addressable memory, e.g., RAM, DRAM, and the like. The graphics memory650may temporarily store graphics resources, graphics buffers, and other graphics data for a graphics rendering pipeline. The graphics buffers may include, e.g., one or more vertex buffers for storing vertex parameter values and one or more index buffers for storing vertex indices. The graphics buffers may also include a one or more render targets693, which may include both color buffers694and depth buffers696holding pixel/sample values computed as a result of execution of instructions by the CPU670and GPU671. In certain implementations, the color buffers694and/or depth buffers696may be used to determine a final array of display pixel color values to be stored in a display buffer697, which may make up a final rendered image intended for presentation on a display. In certain implementations, the display buffer may include a front buffer and one or more back buffers, and the GPU671may be configured to scanout graphics frames from the front buffer of the display buffer697for presentation on a display686.

The CPU may be configured to execute CPU code, which may include an application665that utilizes rendered graphics (such as a video game) and a corresponding graphics API667for issuing draw commands or draw calls to programs implemented by the GPU671based on the state of the application665. The CPU code may also implement physics simulations and other functions.

To support the rendering of graphics, the GPU may execute shaders673, which may include vertex shaders and pixel shaders. The GPU may also execute other shader programs, such as, e.g., geometry shaders, tessellation shaders, compute shaders, and the like. In some implementations, the GPU may include a Video Coding Engine (VCE)674configured to implement video encoding and decoding tasks including, but not limited to, encoding digital pictures or sections of digital pictures as discussed above with respect toFIG. 2A, implementing rate control by deriving QP, as described with respect toFIGS. 2B-2C, and adjusting QP to implement picture quality oriented rate control, as described above with respect toFIG. 4. The GPU may also include specialized hardware modules678, which may include one or more texture mapping units and/or other hardware modules configured to implement operations at one or more stages of a graphics pipeline, which may be fixed function operations. The shaders673and hardware modules678may interface with data in the memory650and the buffers693at various stages in the pipeline before the final pixel values are output to a display. The GPU may include a rasterizer module675, which may be optionally embodied in a hardware module678of the GPU, a shader673, or a combination thereof. The rasterization module675may be configured take multiple samples of primitives for screen space pixels and invoke one or more pixel shaders according to the nature of the samples, in accordance with aspects of the present disclosure.

The system600may also include well-known support functions677, which may communicate with other components of the system, e.g., via the bus676. Such support functions may include, but are not limited to, input/output (I/O) elements679, power supplies (P/S)680, a clock (CLK)681, and a cache682. The apparatus600may optionally include a mass storage device684such as a disk drive, CD-ROM drive, flash memory, tape drive, Blu-ray drive, or the like to store programs and/or data. The device600may also include a display unit686to present rendered graphics687to a user and user interface unit688to facilitate interaction between the apparatus600and a user. The display unit686may be in the form of a flat panel display, cathode ray tube (CRT) screen, touch screen, head mounted display (HMD) or other device that can display text, numerals, graphical symbols, or images. The display686may display rendered graphics687processed in accordance with various techniques described herein. The user interface688may one or more peripherals, such as a keyboard, mouse, joystick, light pen, game controller, touch screen, and/or other device that may be used in conjunction with a graphical user interface (GUI). In certain implementations, the state of the application660and the underlying content of the graphics may be determined at least in part by user input through the user interface688, e.g., in video gaming implementations where the application665includes a video game.

The system600may also include a network interface690to enable the device to communicate with other devices over a network. The network may be, e.g., a local area network (LAN), a wide area network such as the internet, a personal area network, such as a Bluetooth network or other type of network. Various ones of the components shown and described may be implemented in hardware, software, or firmware, or some combination of two or more of these. In some implementations, the CPU code may optionally include a codec668configured to encode digital pictures generated by the GPU. By way of example, and not by way of limitation, the codec668may be configured to encode digital pictures or sections of digital pictures as discussed above with respect toFIG. 2A, implement rate control by deriving QP, as described with respect toFIGS. 2B-2C, and adjust QP to implement picture quality oriented rate control, as described above with respect toFIG. 4. The codec668or VCE676may also be configured to decode encoded digital pictures, e.g., as described above with respect toFIG. 3. The CPU code may also include a network protocol stack669configured to allow the system600to transmit the resulting encoded pictures or encoded sections over the network via the network interface690.

The memory672may store parameters605and/or picture data607or other data. During execution of programs, such as the application665, graphics API667, or codec668, portions of program code, parameters605and/or data607may be loaded into the memory672or cache682for processing by the CPU670and/or GPU671. By way of example, and not by way of limitation, the picture data607may include data corresponding video pictures, or sections thereof, before encoding or decoding or at intermediate stages of encoding or decoding. In the case of encoding, the picture data607may include buffered portions of streaming data, e.g., unencoded video pictures or portions thereof. In the case of decoding, the data607may include input data in the form of un-decoded sections, sections that have been decoded, but not post-processed and sections that have been decoded and post-processed. Such input data may include data packets containing data representing one or more coded sections of one or more digital pictures. By way of example, and not by way of limitation, such data packets may include a set of transform coefficients and a partial set of prediction parameters. These various sections may be stored in one or more buffers. In particular, decoded and/or post processed sections may be stored in an output buffer, which may be implemented in the memory672. The parameters605may include adjustable parameters and/or fixed parameters, as discussed above.

Programs implemented by the CPU and/or GPU (e.g., CPU code, GPU code, application665, graphics API667, codec668, protocol stack669, and shaders673) may be stored as executable or compilable instructions in a non-transitory computer readable medium, e.g., a volatile memory, (e.g., RAM) such as the memory672, the graphics memory650, or a non-volatile storage device (e.g., ROM, CD-ROM, disk drive, flash memory).

Aspects of the present disclosure provide for reduced bit usage and therefore better usage of available bandwidth in streaming data applications, such as streaming video. Aspects of the present disclosure may be incorporated into systems that produce digital pictures, encode them for transmission over a network, and transmit them over the network.