Techniques for high efficiency entropy coding of video data

Entropy coding/decoding techniques are disclosed in which data is coded alternately as a series of nonzero values and zero values until the transmitted data is consumed. Nonzero values may be coded first with transmission of data identifying a number of consecutive nonzero values that appear in scan order followed by transmission of the nonzero values themselves. Thereafter, if other data remains to be transmitted, data may be transmitted identifying a number of consecutive zero values that appear next in scan order followed by transmission of a next nonzero value encountered in scan order. By transmitting the nonzero values as a group, it is expected that the proposed entropy-coding process will achieve higher efficiency than competitive techniques.

BACKGROUND

Various encoding schemes are known for compressing video. Many such schemes are block transform based (e.g., DCT-based), and operate by organizing each frame of the video into two-dimensional blocks. DCT coefficients for each block are then placed in a one-dimensional array in a defined pattern, typically in a zig-zag order through the block. That is, each block is processed independently of each other block, and the DCT coefficients are grouped block-by-block. The coefficients are then encoded using standard run-length coding according to a predetermined scan direction; each encoded block is terminated by an end-of-block codeword. When decoding the video stream, the decoder uses the end-of-block codewords to identify when a new block is being decoded.

Other techniques entropy code video data by combining data from multiple coded blocks according to scan patterns that traverse like-kind coefficient positions of the multiple blocks consecutively, then advance to a new coefficient position and traverse the like-kind coefficient positions of the multiple blocks consecutively.

Conventional entropy coding techniques typically transmit data according to an iterative pattern that, first, identifies a number of zero-valued coefficients that are encountered in the scan direction (commonly called a “run”), followed by data identifying a value of a first nonzero coefficient thereafter. If several nonzero coefficients are encountered consecutively, the conventional entropy coding techniques require, first, a codeword to be transmitted indicating that no zero-valued coefficients exist before the first nonzero coefficient, followed by the value of the nonzero coefficient itself.

DETAILED DESCRIPTION

Aspects of the present disclosure provide entropy coding techniques in which data is coded alternately as a series of nonzero values and zero values until the transmitted data is consumed. Nonzero values may be coded first with transmission of data identifying a number of consecutive nonzero values that appear in scan order followed by transmission of the nonzero values themselves. Thereafter, if other data remains to be transmitted, data may be transmitted identifying a number of consecutive zero values that appear next in scan order followed by transmission of a next nonzero value encountered in scan order. By identifying counts of the number of values being transmitted for each sequence of nonzero values, the proposed techniques avoid inefficiencies of other techniques, which require transmission of data identifying the number of zero values that appear between consecutive nonzero values, even if there are none.

The inventors have determined that it is possible to have long sequences of nonzero coefficient values, which can lead to inefficiencies in the conventional entropy coding techniques described above. Under the entropy coding techniques proposed herein, when a sequence of consecutive nonzero values are encountered in a scan direction, an entropy coder may transmit a count value representing the number of nonzero values that are encountered, followed by the values themselves. This approach is expected to increase coding efficiency because it avoids transmission of codewords indicating the absence of nonzero coefficient values.

FIG. 1illustrates a video exchange system100according to an aspect of the present disclosure. The system100may include a pair of terminals110,120interconnected by a channel130. The terminals110,120may exchange coded video between them. For example, a first terminal110may generate a source video, code the source video by bandwidth compression and transmit the coded video to a second terminal120. The second terminal120may decompress the coded video by inverting coding operations applied by the first terminal110, which yields decoded video data. Thereafter, the second terminal120may consume the decoded video, for example, by displaying it or storing it locally at the terminal120.

Exchange of coded video may occur in a variety of applications. For example, a first terminal110may code video for on demand delivery to other terminals120according to a store-and-forward distribution model. In such applications, a first terminal110may code the video and store it locally until it is requested by another terminal120. In another application, a first terminal110may capture video for real-time delivery to terminal(s)120in a unicast or broadcast distribution model. Alternatively, terminals110,120may be engaged in bidirectional exchange of video as may occur, for example, in a video conference; for bidirectional video exchange, each terminal110,120would code video and deliver coded video to the other terminal110,120where it would be decoded. Thus, the principles of the present disclosure find application in a variety of use cases where exchange of video data is desired.

Although the terminals110,120are illustrated as servers and smartphones, respectively, the principles of the present disclosure find application with a variety of computing equipment. The principles of the present disclosure find application with various types of computers (desktop, laptop, and tablet computers), computer servers, media players, dedicated video conferencing equipment and/or dedicated video encoding equipment.

The channel130represents any of a number of different communication fabric for delivery of coded video data between the terminals110,120. In the example illustrated inFIG. 1, the channel130may be formed by a communication network140for example by wireline and/or wireless communication networks. The communication network140may exchange data in circuit-switched or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and/or the Internet. In other distribution models, the channel130can be formed by storage media, for example, an optical, electrical or magnetic computer-readable storage medium. Exemplary storage media include hard drive memory, flash memory, floppy disk memory, optically-encoded memory (e.g., a compact disk, DVD-ROM, DVD.+−.R, CD-ROM, CD.+−.R, holographic disk, high-definition storage media), thermomechanical memory, or any other type of computer-readable (machine-readable) storage medium. For the purposes of the present discussion, the architecture and topology of the channel130are immaterial to the operation of the present disclosure unless otherwise noted.

FIG. 2is a simplified block diagram of system200for encoding and decoding video according to an aspect of the present disclosure. The system200may include an encoder210and a decoder220provided in communication with each other by a channel230. As described, the encoder210may code a source image240and deliver a coded image to the channel230. The decoder220may receive the coded image from the channel, decode it, and generate a decoded image250therefrom.

FIG. 2illustrates components of an encoder210that are involved with entropy coding proposed by the present disclosure. They include a transform processor212, a quantization processor214, a slice scan system216, and an entropy coder218. The transform processor212may apply a selected transform to image data, such as a discrete cosine transform (commonly, “DCT”), which converts the image data from a pixel domain to a transform domain. The quantization processor214may apply quantization to the transform coefficients output by the transform processor214. Typically, each transform coefficient is divided by a respective quantization parameter, which can reduce the amount of data required to represent the transform coefficients and, in some cases, reduce the transform coefficients to zero. The slice scan system216may arrange the quantized coefficients into a predetermined order for processing by the entropy coder218. The entropy coder218may code the ordered coefficients as discussed herein. Coded video data output by the entropy coder218may be provided to the channel230.

FIG. 2also illustrates components of a decoder220that are involved with entropy coding proposed by the present disclosure. They include an inverse transform processor222, an inverse quantization processor224, an inverse slice scan system226and an entropy decoder228. As their names imply, these components222-228may invert coding operations performed by their counterparts212-218in the encoder210. Specifically, the entropy decoder228may decode coded video data as discussed herein. The entropy decoder228may output quantized coefficients, which are reorganized by the inverse slice scan system226. The inverse quantization processor224may perform inverse quantization by multiplying quantized coefficients by the same quantization parameters that were applied by the quantization processor214of the encoder210. In practice, quantization and inverse quantization is a lossy process; thus, the transform coefficients output by the inverse quantization processor224at the decoder220likely will resemble but not match the transform coefficients that were input to the quantization processor214of the encoder210. The inverse transform processor222may invert transform processes applied by the transform processor212of the encoder210. The inverse transform processor222may output image data in the pixel domain. A decoded image250may be generated from the pixel data output by the decoder220.

As discussed,FIG. 2is a simplified diagram of the components of an encoder210and a decoder220. In practice, encoders and decoder210,220often include other components for processing video data. For example, many modern video coding systems employ prediction to exploit spatial and/or temporal redundancy in video data. Rather than input source image data240directly to a transform processor212, source image data240may be coded differentially with respect to prediction data. In this case, a transform processor212may receive input data representing pixel-wise differences between the source image240and prediction data (not shown). Similarly, prediction may be used at the decoder220. Pixel data output by the inverse transform processor222may be added to prediction data (not shown) to generate a decoded image250. Such operations are not illustrated in the block diagram ofFIG. 2, nor are components that typically are provided to process other types of data such as audio data.

The operations illustrated inFIGS. 3A and 3Bfind application with many kinds of video coding systems. Conventionally, transform processors (FIG. 2) are applied to image data that has been parsed into pixel blocks of predetermined size. Thus, each pixel block represents a two-dimensional region of an input image and the transform processor212transforms pixel data (or pixel residuals) into a transform domain. The slice scan system216(FIG. 2) may reorganize quantized coefficients of individual blocks into a serial data steam as illustrated inFIGS. 3A and 3B. In this example, the slice scan process generates a data stream having x·y coefficients.

The principles of the present disclosure find application with larger arrays of image data than mere pixel blocks. For example, as shown inFIGS. 4A and 4B, image data from multiple blocks may be merged and formed into a serial data stream for processing by an entropy coder218(FIG. 2). In the example ofFIG. 4A, a plurality of pixel blocks are shown as arranged into a three-dimensional array in which like-kind coefficients are aligned (e.g., all coefficients at positions 0, 1, 2, . . . ,63are aligned with each other). InFIG. 4B, the same data is illustrated as a two-dimensional structure in which data of each block occupy a single row and columns maintain alignment of like-kind coefficient positions. In this example, the slice scan process generates a data stream having x·y·z coefficients.

In an aspect, a slice scan system216may traverse coefficients of multiple blocks in a single coding operation. The scan starts at a first coefficient position (say, position 0) and scans across all blocks (say, blocks0-3ofFIG. 4B) at that position. The scan advances to a next coefficient position (say, position 1) and scans across all blocks at that position. The scan incrementally advances to successor positions (say, positions 8, 9, 2, and 3 in order) and, at each scan position, the slice scan system may scan across all blocks in each of the positions before advancing to the next successor position.

During operation, because the array stores quantized transform coefficients, it is likely that the values at many of the coefficient positions will be zero. If there is significant redundancy in image content among the blocks in the array ofFIG. 4B, then the zero-valued coefficients are likely to be clustered among many columns of the array. Thus, the slice scan process coupled with run-length encoding of zero-valued coefficients may yield improved coding efficiency over a scan system that operates on blocks individually (FIGS. 3A, 3B) because the slice scan system will yield much longer runs of zero-valued coefficients.

As a specific example, the blocks illustrated inFIG. 4Bmay be the luma component blocks of a macroblock. The transform coefficients for each block at each component position 0-63 may be stored in a transform coefficient array in the order shown inFIG. 4B. An entropy encoder218(FIG. 2) then can encode the coefficients by processing coefficients at the same position in each block together. That is, the coefficients in the first column (the 0-position coefficients) may be processed first, followed by the coefficients in the second column (position 1), and so on. Generally, low-frequency coefficients may be processed first.

The principles of the present disclosure extend to other scan directions. Another scan protocol is illustrated inFIG. 3C.

The principles ofFIGS. 4A and 4Bfind ready application in coding systems that parse image data first into macroblocks (a 16 pixel by 16 pixel array) and then partition the macroblocks on a quadrature basis into subordinate blocks (often 8×8 each). Thus, each macroblock typically includes four subordinate blocks. The subordinate blocks may be organized and scanned as illustrated inFIGS. 4A and 4B.

The principles of the present discussion may be extended to larger groups of blocks. For example, as illustrated inFIG. 5, data from n macroblocks are shown as reorganized into a common data stream for processing by an entropy coder. In this example, the slice scan process may generate an array of x·y·z·n coefficients.

FIG. 6illustrates an entropy coding method600according to an aspect of the present disclosure. Entropy coders218,930(FIGS. 2, 9) may operate according to the method600ofFIG. 6. The method600may begin by transmitting data representing a number of coefficients being processed by a current instance of the method (box610). The operation of box610may be omitted in applications where the number of coefficients is known to a decoder through other means, for example, by being predetermined by a governing coding protocol. The method600then may engage an iterative process to transmit levels and counts working across the data array in a scan order established by the slice scan process. An iteration may begin by determining whether the end of a coefficient array has been reached or, on a first iteration, whether all remaining coefficients of the array are zero (box615). If either condition occurs, the method600may end. If not, however, then the method600may determine the number of nonzero coefficients that are next in scan order (box620) and transmit data identifying the determined number of nonzero coefficients (box625). The method600also may transmit data identifying each of the nonzero coefficients (box630).

After transmission of the last nonzero coefficient determined at box630, the method600may determine if the end of the coefficient array has been reached or if all remaining coefficients in the array are zero (box640). If so, the method600may end. If not, then the method600may determine the number of zero-valued coefficients that are next in scan order (box645) and it may transmit data identifying the number of zero-valued coefficients (box650). The method600may transmit identifying the next nonzero coefficient in scan order (box655) and return to the operation of box615.

Operation of the method600is expected to provide coding efficiencies as compared to other entropy coding processes. As discussed, modern entropy coding processes iteratively transmit data identifying a number of zero-valued coefficients that occur consecutively, then transmit the value of a nonzero coefficient that follows. When multiple nonzero coefficients occur consecutively, the prior process requires transmission of codewords that identify that no zero-valued coefficients occurred between each nonzero coefficient and its next consecutive nonzero coefficient. These coding processes induce inefficiencies because it is common to have large numbers of nonzero coefficients appear consecutively in scan order, especially in applications where the data rate is high. It is expected that the operation of the method600illustrated inFIG. 6will provide increased efficiencies by starting coding with identification of a count of consecutive nonzero coefficients that appear in scan order, followed by values of those nonzero coefficients (boxes625-630). Transmission of codewords indicating the absence of zero-valued coefficients can be avoided.

The method600illustrated inFIG. 6presents transmission of the count of nonzero valued coefficients and the nonzero coefficients' values (boxes625-630) as occurring before transmission of the count of zero-valued coefficients and the next nonzero coefficient (boxes650,655). In practice, many scan orders are expected to start at positions at which nonzero coefficient values are more likely to be present than zero-valued coefficients. In other implementations (not shown inFIG. 6), transmission of a count of zero-valued coefficients and the next nonzero coefficient might precede transmission of the count of nonzero valued coefficients and the nonzero coefficient's values. Such implementations may be appropriate in applications where scan orders begin with coefficient positions that are more likely to have zero-valued coefficients than nonzero valued coefficients.

The principles of the present disclosure accommodate variations of the method600illustrated inFIG. 6. As discussed above, the method600need not transmit data identifying the number of coefficients (box610) in applications where the number is determined by other means. For example, the number of coefficients may be predetermined by a governing coding protocol to a fixed number. Returning toFIG. 5, for example, in a case where n=8 and z=1, each scan operation may span eight 8×8 blocks of coefficient data for a total of 512 coefficients. The identification of the number of coefficients may be omitted if the number is fixed for every iteration of the method. In another embodiment, the number of coefficients may depend on other coding parameters provided in coded data, for example, sizes of coding units. In such applications, although the number of coefficients per scan operation may vary, the number may be derived from other coding parameters and, thus, an express transmission of the number of coefficients as shown in box610may be avoided.

In another aspect, a coder need not perform the operation of box655after transmission of a run of zero-valued coefficients in box650. In such aspects (shown in phantom inFIG. 6), the method600may advance from box650to box620, and the method600may determine the number of nonzero coefficients in the array that appear next in scan order.

FIG. 7illustrates an entropy decoding method700according to an aspect of the present disclosures. Entropy decoders228,1020(FIGS. 2, 10) may operate according to the method700ofFIG. 7. The method700may process a data stream of entropy-coded data that is received from a channel. As discussed, entropy decoding essentially inverts processes performed during entropy coding. The method700may begin by extracting from the channel data representing the number of coefficients in the entropy-coded data array and the data size (box710). As discussed in connection withFIG. 6, data representing the number of coefficients need not be extracted from channel data if the value is known to the decoder through other means. The method700then may engage an iterative process to recover zero-valued and nonzero coefficients working across the data array in a scan order established by the slice scan process. An iteration may begin by determining whether the end of the compressed data has been reached (box715). If so, the method700may end, as discussed below. If not, however, then the method700may identify, from the channel data, a number of nonzero coefficients that are next in scan order (box720). The method700also may extract the nonzero coefficients from the channel data (box725).

After extraction of the last nonzero coefficient, the method700may determine if the end of the compressed data has been reached (box730). If so, the method700may set remaining coefficients, if any, to zero (box735), and the method700may end. If not, then the method700may extract, from the channel data, data identifying the number of zero-valued coefficients that are next in scan order (box740). The method may generate a number of zero-valued coefficients corresponding to the number identified by the channel data (box745). The method700may extract, from the channel, data identifying the next nonzero coefficient in scan order (box750) and it may return to the operation of box715.

If, at boxes715or730, the method700determines that the end of the compressed data has been reached, then the method700may set zero values for all remaining coefficient positions in the data array (box735). Thereafter, the method700may end.

The method700illustrated inFIG. 7presents identification of the count of nonzero valued coefficients and extraction of the nonzero coefficient's values (boxes720-725) as occurring before identification of the run of zero-valued coefficients and extraction of the next nonzero coefficient (boxes740,750). As discussed, many scan orders are expected to start at positions at which nonzero coefficient values are more likely to be present than zero-valued coefficients. In other implementations (not shown inFIG. 7), however, identification of a count of zero-valued coefficients and extraction of a next nonzero coefficient might precede identification of the count of nonzero valued coefficients and extraction of the nonzero coefficient's values. Such implementations may be appropriate in applications where scan orders begin with coefficient positions that are more likely to have zero-valued coefficients than nonzero valued coefficients.

As with the method600ofFIG. 6, the principles of the present disclosure accommodate variations of the method700illustrated inFIG. 7. As discussed above, the method700need not extract data identifying the number of coefficients (box710) in applications where the number is determined by other means. Again, if the number of coefficients is set to a predetermined, fixed number or if the number of coefficients is to be derived from other coding parameters, the number of coefficients need not be extracted from channel data as shown in box710.

Also, a decoder need not perform the operation of box750after generation of zero-valued coefficients from an identified run in box745. In an alternative aspect (shown in phantom inFIG. 7), the method700may advance from box745to box720. This aspect corresponds with the variant of the method600(shown in phantom inFIG. 6) as described.

Many coding protocols represent data as variable length codes that are integrated into a serially-coded data stream. Thus, the extraction operations performed by the method700in boxes710,720,725,740, and750each may define a context for the data elements that follow the extraction operations. That is, a variable length code that identifies the number of nonzero coefficients in box720may define context for extraction of a nonzero coefficient that is performed in box725. Moreover, an extraction of a first nonzero coefficient in box725may define a context for identification and extraction of a next nonzero coefficient that also is performed in box725.

Table 1, for example, provides a syntax that may be used for entropy coding and decoding according to the embodiments ofFIGS. 6 and 7. In this example, entropy coding may be performed on a number of bits represented by the dataSize value.

Coding may proceed in a loop in which data representing the number of nonzero coefficients (level_count) and the number of zero coefficients (zero_run_length_minus_1) are transmitted in alternating fashion. Specifically, when transmitting nonzero coefficients, a level_count parameter may identify the number of nonzero coefficients, and it may be followed by data representing values of the nonzero coefficients themselves (abs_level_minus_1 and sign). Thereafter, the zero_run_length_minus_1 may identify the number of zero coefficients, and it may be followed by data representing the value of the next nonzero coefficient (again, abs_level_minus_1 and sign).

FIG. 8illustrates communication flow between terminal devices110,120that may occur according to the syntax defined in Table 1. As indicated, a transmitting terminal110may transmit a first DC coefficient (msg.810) and, thereafter, transmit DC coefficients of other blocks in a differential manner (msgs.820). The transmitting terminal110thereafter may transmit level run lengths and zero run lengths in an alternating manner until the end of the data array is reached. Specifically, the transmitting terminal110may transmit the level_count parameter (msg.830), which identifies the number of nonzero coefficients that follow, and the nonzero coefficients themselves (msgs.840). If the end of the data array has not been reached, the transmitting terminal110thereafter may transmit data identifying the zero_run_length (msg.850) and the next nonzero coefficient that follows (msg.860). If the end of the data array has not been reached, the transmitting terminal110may transmit a new level_count parameter and a new set of nonzero coefficients (msgs.830,840).

At some point, the transmitting terminal110will reach the end of the data array, at which point the transmitting terminal110may transmit data indicating the end of the array (msg.870).

As discussed, the principles of the present disclosure find application in predictive coding systems, where input data is coded differentially with respect to prediction data generated for the input data.FIGS. 9 and 10illustrate application of a slice scan system and an entropy coder to one such predictive coding system.

FIG. 9is a functional block diagram of a coding system900according to an aspect of the present disclosure. The system900may include a pixel block coder910, a slice scan system920, an entropy coder930, a pixel block decoder940, an in-loop filter system950, a reference picture store960, a predictor970, a controller980, and a syntax unit990. The pixel block coder and decoder910,940and the predictor970may operate iteratively on individual pixel blocks of an input frame. Typically, the pixel blocks will be generated by parsing frames into smaller units for coding. The predictor970may predict data for use during coding of a newly-presented pixel block. The pixel block coder910may code the new pixel block differentially with respect to prediction data from the predictor970. The slice scan system920may organize coded pixel block data into data arrays for coding by the entropy coder930. The entropy coder930may apply entropy coding to the data arrays and output coded block data to the syntax unit990, where it may be formatted for transmission to a channel (not shown).

The pixel block decoder940may decode the coded pixel block data from the pixel block coder910and decoded pixel block data therefrom. The in-loop filter950may perform various filtering operations on a decoded picture that is assembled from the decoded pixel blocks obtained by the pixel block decoder940. The filtered picture may be stored in the reference picture store960where it may be used as a source of prediction of a later-received pixel block.

The pixel block coder910may include a subtractor912, a transform processor914, and a quantization processor916. The pixel block coder910may accept pixel blocks of input data at the subtractor912. The subtractor912also may receive predicted pixel blocks from the predictor970and generate an array of pixel residuals therefrom representing differences between the input pixel block and the predicted pixel block at each pixel location. The transform unit914may apply a transform to the pixel residuals output from the subtractor912, to convert data from the pixel domain to a domain of transform coefficients.

The transform unit914may operate in a variety of transform modes as determined by the controller980. For example, the transform unit914may apply a discrete cosine transform (DCT), a discrete sine transform (DST), a Walsh-Hadamard transform, a Haar transform, a Daubechies wavelet transform, or the like. In an aspect, the controller980may select a coding mode M to be applied by the transform unit915, may configure the transform unit915accordingly and may signal the coding mode M in the coded video data, either expressly or impliedly.

The quantization processor916may perform quantization of transform coefficients output by the transform unit914. The quantization processor916may operate according to a quantization parameter QP that is supplied by the controller980. In an aspect, the quantization parameter QP may be applied to the transform coefficients as a multi-value quantization parameter, which may vary, for example, across different coefficient locations within a transform-domain pixel block. Thus, the quantization parameter QP may be provided as a quantization parameters array. In another aspect, however, the quantization parameter may be a uniform value that is applied to all transform coefficients. The quantization processor916may output quantized coefficients that have been rounded down to integer values. For some coefficients, the quantization may reduce the quantized coefficients to zero.

The slice scan system920may reorganize coefficients output from the pixel block coder910for processing by the entropy coding. In this regard, the slice scan system920may operate according to the principles ofFIGS. 3-5, discussed hereinabove.

The entropy coder930, as its name implies, may perform entropy coding of data output from the slice scan system920. It may operate according to the principles described inFIGS. 6-8and Table 1, described hereinabove.

The pixel block decoder940may invert coding operations of the pixel block coder910. For example, the pixel block decoder940may include a dequantization processor942, an inverse transform unit944, and an adder946. The pixel block decoder940may take its input data from an output of the quantization processor916. The dequantization processor942may invert operations of the quantization processor916of the pixel block coder910. The dequantization processor942may perform uniform or non-uniform de-quantization as specified by the quantization parameter QP. Similarly, the inverse transform unit944may invert operations of the transform unit914. The dequantization processor942and the inverse transform unit944may use the same quantization parameters QP and transform mode M as their counterparts in the pixel block coder910. Quantization operations likely will truncate data in various respects and, therefore, data recovered by the dequantization processor942likely will possess coding errors when compared to the data presented to the quantization processor916in the pixel block coder910.

The adder946may invert operations performed by the subtractor912. It may receive the same prediction pixel block from the predictor970that the subtractor912used in generating residual signals. The adder946may add the prediction pixel block to reconstructed residual values output by the inverse transform unit944and may output reconstructed pixel block data.

The in-loop filter950may perform various filtering operations on frame data that is constructed from recovered pixel block data. For example, the in-loop filter950may include a deblocking filter952and a sample adaptive offset (“SAO”) filter953. The deblocking filter952may filter data at seams between reconstructed pixel blocks to reduce discontinuities between the pixel blocks that arise due to coding. SAO filters may add offsets to pixel values according to an SAO “type,” for example, based on edge direction/shape and/or pixel/color component level.FIG. 9does not illustrate an exhaustive set of filters that may be used for in-loop filtering; in other aspects, the in-loop filter950may perform adaptive loop filtering (ALF), maximum likelihood (ML) based filtering schemes, deringing, debanding, sharpening, resolution scaling, and the like. The selection of filters to be applied by the in-loop filter950may be determined by parameters that are selected by the controller980.

The reference picture store960may store filtered frame data for use in later prediction of other pixel blocks. Different types of prediction data are made available to the predictor970for different prediction modes. For example, for an input pixel block, intra prediction takes a prediction reference from decoded data of the same picture in which the input pixel block is located. Thus, the reference picture store960may store decoded pixel block data of each picture as it is coded. For the same input pixel block, inter prediction may take a prediction reference from previously coded and decoded picture(s) that are designated as reference pictures. Thus, the reference picture store960may store these decoded reference pictures.

As discussed, the predictor970may supply prediction data to the pixel block coder910for use in generating residuals. The predictor970may include an inter predictor972, an intra predictor973and a mode decision unit974. The inter predictor972may receive pixel block data representing a new pixel block to be coded and may search reference picture data from store960for pixel block data from reference picture(s) for use in coding the input pixel block. The inter predictor972may support a plurality of prediction modes, such as P mode coding and B mode coding. The inter predictor972may select an inter prediction mode and an identification of candidate prediction reference data that provides a closest match to the input pixel block being coded. The inter predictor972may generate prediction reference metadata, such as motion vectors, to identify which portion(s) of which reference pictures were selected as source(s) of prediction for the input pixel block.

The intra predictor973may support Intra (I) mode coding. The intra predictor973may search from among pixel block data from the same picture as the pixel block being coded that provides a closest match to the input pixel block. The intra predictor973also may generate prediction reference indicators to identify which portion of the picture was selected as a source of prediction for the input pixel block.

The mode decision unit974may select a final coding mode to be applied to the input pixel block. Typically, as described above, the mode decision unit974selects the prediction mode that will achieve the lowest distortion when video is decoded given a target bitrate. Exceptions may arise when coding modes are selected to satisfy other policies to which the coding system900adheres, such as satisfying a particular channel behavior, or supporting random access or data refresh policies. When the mode decision selects the final coding mode, the mode decision unit974may output a selected reference block from the store960to the pixel block coder and decoder910,940and may supply to the controller980an identification of the selected prediction mode along with the prediction reference indicators corresponding to the selected mode.

The controller980may control overall operation of the coding system900. The controller980may select operational parameters for the pixel block coder910and the predictor970based on analyses of input pixel blocks and also external constraints, such as coding bitrate targets and other operational parameters. As is relevant to the present discussion, when it selects quantization parameters QP, the use of uniform or non-uniform quantization processors, and/or the transform mode M, it may provide those parameters to the syntax unit990, which may include data representing those parameters in the data stream of coded video data output by the system900. The controller980also may select between different modes of operation by which the system may generate reference images and may include metadata identifying the modes selected for each portion of coded data.

During operation, the controller980may revise operational parameters of the quantization processor916and the transform unit915at different granularities of image data, either on a per pixel block basis or on a larger granularity (for example, per picture, per slice, per largest coding unit (“LCU”) or another region). In an aspect, the quantization parameters may be revised on a per-pixel basis within a coded picture.

Additionally, as discussed, the controller980may control operation of the in-loop filter950and the prediction unit970. Such control may include, for the prediction unit970, mode selection (lambda, modes to be tested, search windows, distortion strategies, etc.), and, for the in loop filter950, selection of filter parameters, reordering parameters, weighted prediction, etc.

FIG. 10is a functional block diagram of a decoding system1000according to an aspect of the present disclosure. The decoding system1000may include a syntax unit1010, an entropy decoder1020, an inverse slice scan system1030, a pixel block decoder1040, an in-loop filter1050, a reference picture store1060, a predictor1070, a controller1080and a predictor1070. The syntax unit1010may receive a coded video data stream and may parse the coded data into its constituent parts. Data representing coding parameters may be furnished to the controller1080while the entropy-coded data may be output to the entropy decoder1020. The entropy decoder1020may apply entropy decoding, which generates recovered coefficients therefrom. The inverse slice scan system1030may reorganize the recovered coefficients as pixel blocks, which may be input to the pixel block decoder1040. The pixel block decoder1040may invert coding operations provided by the pixel block coder910(FIG. 9), generating recovered pixel data therefrom. The in-loop filter1050may filter frames that are assembled from the recovered pixel block data. The filtered frames may be output from the decoding system1000as recovered frame data.

Recovered pictures also may be stored in the prediction buffer1060for use in prediction operations. The predictor1070may supply prediction data to the pixel block decoder1040as determined by coding data received in the coded video data stream.

The pixel block decoder1040may include an inverse quantization processor1042, an inverse transform processor1044, and an adder1046. The inverse quantization processor1042may invert operations of the quantization processor916of the pixel block coder910(FIG. 9). Similarly, the inverse transform processor1044may invert operations of the transform processor914(FIG. 9). They may use the quantization parameters QP and transform modes M that are provided in the coded video data stream. Because quantization is likely to truncate data, the data recovered by the inverse quantization processor1042, likely will possess coding errors when compared to the input data presented to its counterpart quantization processor916in the pixel block coder910(FIG. 9).

The adder1046may invert operations performed by the subtractor912of the pixel block coder910(FIG. 9). It may receive a prediction pixel block from the predictor1070as determined by prediction references in the coded video data stream. The adder1046may add the prediction pixel block to reconstructed residual values output by the inverse transform processor1044and may output reconstructed pixel block data.

The in-loop filter1050may perform various filtering operations on reconstructed pixel block data. As illustrated, the in-loop filter1050may include a deblocking filter1052and an SAO filter1054. The deblocking filter1052may filter data at seams between reconstructed pixel blocks to reduce discontinuities between the pixel blocks that arise due to coding. SAO filters1054may add offset to pixel values according to an SAO type, for example, based on edge direction/shape and/or pixel level. Other types of in-loop filters may also be used in a similar manner. Operation of the deblocking filter1052and the SAO filter1054ideally would mimic operation of their counterparts in the coding system900(FIG. 9). Thus, in the absence of transmission errors or other abnormalities, the decoded picture obtained from the in-loop filter1050of the decoding system1000would be the same as the decoded picture obtained from the in-loop filter950of the coding system900(FIG. 9); in this manner, the coding system900and the decoding system1000should store a common set of reference pictures in their respective reference picture stores940,1060.

As withFIG. 9,FIG. 10does not illustrate an exhaustive set of filters that may be used for in-loop filtering; in other aspects, the in-loop filter1050may perform adaptive loop filtering (ALF), maximum likelihood (ML) based filtering schemes, deringing, debanding, sharpening, resolution scaling, and the like. The selection of filters to be applied by the in-loop filter1050may be determined by parameters that are provided in the coded video data.

The reference picture store1060may store filtered pixel data for use in later prediction of other pixel blocks. The reference picture store1060may store decoded pixel block data of each picture as it is coded for use in intra prediction. The reference picture store1060also may store decoded reference pictures.

As discussed, the predictor1070may supply the transformed reference block data to the pixel block decoder1040. The predictor1070may supply predicted pixel block data as determined by the prediction reference indicators supplied in the coded video data stream.

The controller1080may control overall operation of the coding system1000. The controller1080may set operational parameters for the pixel block decoder1040and the predictor1070based on parameters received in the coded video data stream. As is relevant to the present discussion, these operational parameters may include quantization parameters QP for the inverse quantization processor1042and transform modes M for the inverse transform unit1010. As discussed, the received parameters may be set at various granularities of image data, for example, on a per pixel block basis, a per picture basis, a per slice basis, a per LCU basis, or based on other types of regions defined for the input image.

Although the foregoing description has described the entropy coding techniques proposed herein operating within the context of a video coding system, the principles of the present disclosure are not so limited. Entropy coding processes typically are applied to many kinds of data, including still image data (e.g., JPEG) and audio data. Indeed, the principles of the present disclosure find application to code any kind of data set that is populated by zero-valued data items and nonzero valued data items for serial transmission and reduce transmission bandwidth of such data items.

The foregoing discussion has described operation of the aspects of the present disclosure in the context of video coders and decoders. Commonly, these components are provided as electronic devices. Video decoders and/or controllers can be embodied in integrated circuits, such as application specific integrated circuits, field programmable gate arrays and/or digital signal processors. Alternatively, they can be embodied in computer programs that execute on camera devices, personal computers, notebook computers, tablet computers, smartphones or computer servers. Such computer programs typically are stored in physical storage media such as electronic-, magnetic- and/or optically-based storage devices, where they are read to a processor and executed. Decoders commonly are packaged in consumer electronics devices, such as smartphones, tablet computers, gaming systems, DVD players, portable media players and the like; and they also can be packaged in consumer software applications such as video games, media players, media editors, and the like. And, of course, these components may be provided as hybrid systems that distribute functionality across dedicated hardware components and programmed general-purpose processors, as desired.

Video coders and decoders may exchange video through channels in a variety of ways. They may communicate with each other via communication and/or computer networks as illustrated inFIG. 1. In still other applications, video coders may output video data to storage devices, such as electrical, magnetic and/or optical storage media, which may be provided to decoders sometime later. In such applications, the decoders may retrieve the coded video data from the storage devices and decode it.

Several embodiments of the invention are specifically illustrated and/or described herein. However, it will be appreciated that modifications and variations of the invention are covered by the above teachings and within the purview of the appended claims without departing from the spirit and intended scope of the invention.