A spatially adaptive quantization-aware deblocking filter is used for encoding or decoding video or image frames. The deblocking filter receives a reconstructed frame produced based on dequantized and inverse transformed coefficients of a video frame or an image frame. The reconstructed frame is filtered according to adaptive quantization field data for the video or image frame. The adaptive quantization field data represents weights applied to quantization values used at different areas of the video or image frame. A number of blocking artifacts remaining within the resulting filtered frame is determined. The adaptive quantization field data is then adjusted based on that number of blocking artifacts. The filtered frame is then filtered according to the adjusted adaptive quantization field data. The resulting re-filtered frame is then output to an output source, such as for transmission, display, storage, or further processing.

BACKGROUND

Digital video streams may represent video using a sequence of frames or still images. Digital video can be used for various applications including, for example, video conferencing, high definition video entertainment, video advertisements, or sharing of user-generated videos. A digital video stream can contain a large amount of data and consume a significant amount of computing or communication resources of a computing device for processing, transmission, or storage of the video data. Various approaches have been proposed to reduce the amount of data in video streams, including encoding or decoding techniques.

The approaches for reducing the amount of data in video streams may also be used to reduce the amount of data in an image. Image content represents a significant amount of online content. A web page may include multiple images, and a large portion of the time and resources spent rendering the web page are dedicated to rendering those images for display. The amount of time and resources required to receive and render an image for display depends in part on the manner in which the image is compressed. As such, an image, and therefore a web page that includes the image, can be rendered faster by reducing the total data size of the image using encoding and decoding techniques.

SUMMARY

Disclosed herein are, inter alia, systems and techniques for video or image coding using a spatially adaptive quantization-aware deblocking filter.

A method for decoding an encoded frame from a bitstream according to an implementation of this disclosure comprises decoding syntax data associated with the encoded frame from the bitstream. The syntax data include quantized transform coefficients of encoded blocks of the encoded frame and adaptive quantization field data representing weights applied to quantization values used to encode the encoded blocks. The method further comprises dequantizing and inverse transforming the quantized transform coefficients of the encoded blocks to produce decoded blocks. The method further comprises reconstructing the decoded blocks into a reconstructed frame. The method further comprises applying a deblocking filter to the reconstructed frame according to the adaptive quantization field data to produce a first filtered frame. The method further comprises determining a number of blocking artifacts within the first filtered frame. The method further comprises adjusting the at least some of the adaptive quantization field data based on the number of blocking artifacts to produce adjusted adaptive quantization field data. The method further comprises applying the deblocking filter to the first filtered frame according to the adjusted adaptive quantization field data to produce a second filtered frame. The method further comprises outputting the second filtered frame for display.

An apparatus for decoding an encoded frame from a bitstream according to an implementation of this disclosure comprises a processor configured to execute instructions stored in a non-transitory storage medium. The instructions include instructions to decode encoded blocks of the encoded frame to produce decoded blocks. The instructions further include instructions to reconstruct the decoded blocks into a reconstructed frame. The instructions further include instructions to filter the reconstructed frame according to adaptive quantization field data associated with the encoded frame to produce a first filtered frame. The instructions further include instructions to subsequent to filtering the reconstructed frame, use a psychovisual model to adjust at least some of the adaptive quantization field data. The instructions further include instructions to subsequent to adjusting the at least some of the adaptive quantization field data, filter the first filtered frame according to the adaptive quantization field data to produce a second filtered frame. The instructions further include instructions to output the second filtered frame for display.

A method according to an implementation of this disclosure comprises applying first filter parameters to a reconstructed frame to produce a first filtered frame. The first filter parameters are defined based on adaptive quantization field data associated with the reconstructed frame. The method further comprises adjusting the first filter parameters to produce second filter parameters. The adjusting includes increasing at least one value of the adaptive quantization field data. The method further comprises applying the second filter parameters to the first filtered frame to produce a second filtered frame. The method further comprises outputting the second filtered frame for display.

These and other aspects of this disclosure are disclosed in the following detailed description of the implementations, the appended claims, and the accompanying figures.

DETAILED DESCRIPTION

Lossy encoding involves reducing the amount of data within an image or a video to be encoded, such as using quantization. In exchange for a decreased bit cost of the resulting encoded image or video, the image suffers certain quality loss. The extent of the quality loss depends largely upon the manner by which the image data or the video data was quantized during the encoding. In particular, the quantization of image data or video data can result in discontinuities along block boundaries, such as blocking artifacts. The quantization error resulting from lossy encoding is typically indicative of the amount of blocking artifacts resulting from the encoding. As such, the greater the quantization error, the greater the number of blocking artifacts, and, therefore, the greater the quality loss.

Blocking artifacts may be reduced by applying a filter, such as a deblocking filter, to the coefficients of a video block or of an image block. The deblocking filter may be applied to a reconstructed frame or a portion of a reconstructed frame at the end of a reconstructing phase in the encoding process or at the end of the decoding process. The deblocking filter removes blocking artifacts from a frame to reproduce that frame in its pre-encoded form. However, a typical deblocking filter does not have visibility into the amount of quantization that was used to encode a given frame or the specific blocks therein. As such, and particularly where different quantization levels are used for different blocks within a single frame, the deblocking filter may use a suboptimal or inappropriate filtering strength or filter radius size.

Implementations of this disclosure address problems such as these using a deblocking filter controlled based on adaptive quantization field data. The adaptive quantization field data is used to control the strength and spatial size of the deblocking filter. As such, the deblocking filter uses a greater strength and/or spatial size for blocks that the adaptive quantization field data indicates were encoded at a greater quantization level and a lesser strength and/or spatial size for blocks that the adaptive quantization field data indicates were encoded at a lesser quantization level. Fine details within a frame are preserved by controlling the strength and spatial size of the filtering based on the adaptive quantization field data. A psychovisual model is used in connection with the filtering to determine whether adjustments should be made to the adaptive quantization field data. For example, if a number of blocking artifacts remaining within an area of the frame to encode or decode is too large, the psychovisual model can indicate to increase vales of the adaptive quantization field data for that area.

Further details of techniques for video or image coding using a spatially adaptive quantization-aware deblocking filter are described herein with initial reference to a system in which they can be implemented.FIG. 1is a schematic of an example of an encoding and decoding system100. The encoding and decoding system100includes a transmitting station102, a receiving station104, and a network106.

The transmitting station102is a computing device that encodes and transmits a video or an image. Alternatively, the transmitting station102may include two or more distributed computing devices for encoding and transmitting a video or an image. The receiving station104is a computing device that receives and decodes an encoded video or an encoded image. Alternatively, the receiving station104may include two or more distributed computing devices for receiving and decoding an encoded video or an encoded image. An example of a computing device used to implement one or both of the transmitting station102or the receiving station104is described below with respect toFIG. 2.

The network106connects the transmitting station102and the receiving station104for the encoding, transmission, receipt, and decoding of a video or an image. The network106can be, for example, the Internet. The network106can also be a local area network, a wide area network, a virtual private network, a cellular telephone network, or another means of transferring the video or the image from the transmitting station102to the receiving station104.

Implementations of the encoding and decoding system100may differ from what is shown and described with respect toFIG. 1. In some implementations, the encoding and decoding system100can omit the network106. In some implementations, a video or an image can be encoded and then stored for transmission at a later time to the receiving station104or another device having memory. In some implementations, the receiving station104can receive (e.g., via the network106, a computer bus, and/or some communication pathway) the encoded video or encoded image and store the encoded video or encoded image for later decoding.

In some implementations, the functionality of the transmitting station102and of the receiving station104can change based on the particular operations performed. For example, during operations for encoding a video or an image, the transmitting station102can be a computing device used to upload the video or the image to be encoded to a server, and the receiving station104can be the server that receives the video or the image from the transmitting station102and encodes the video or the image for later use (e.g., in storing a bitstream, rendering a webpage, or the like).

In another example, during operations for decoding an encoded video or encoded image, the transmitting station102can be a server that decodes the encoded video or encoded image, and the receiving station104can be a computing device that receives the decoded video or decoded image from the transmitting station102and outputs or renders the decoded video or decoded image (e.g., to an output video stream, as part of a webpage, or the like).

In some implementations, the encoding and decoding system100may omit the network106. In some implementations, a transport protocol may be used to transport a video or an image, or an encoded video or encoded image, over the network106. For example, the transport protocol may be the real-time transport protocol, the hypertext transfer protocol, or another image or video streaming protocol.

In some implementations, each of the transmitting station102and the receiving station104may include functionality for both encoding and decoding a video or an image. For example, the encoding and decoding system100may be implemented using a video conferencing system. The receiving station104may be a computing device of a video conference participant. The receiving station104may receive an encoded video bitstream from a video conference server (e.g., the transmitting station102) to decode and view. The receiving station104may further encode and transmit another video bitstream to the video conference server, such as for decoding and viewing by computing devices of other video conference participants.

FIG. 2is a block diagram of an example of a computing device200that can implement a transmitting station or a receiving station of an encoding and decoding system, such as the encoding and decoding system100shown inFIG. 1. For example, the computing device200can implement one or both of the transmitting station102or the receiving station104shown inFIG. 1. The computing device200can be in the form of a computing system including multiple computing devices or in the form of one computing device. For example, the computing device200can be one of a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, a server computer, a game console, a wearable device, or the like.

A processor202in the computing device200can be a conventional central processing unit. Alternatively, the processor202can be another type of device, or multiple devices, now existing or hereafter developed, capable of manipulating or processing information. For example, although the disclosed implementations can be practiced with one processor as shown (e.g., the processor202), advantages in speed and efficiency can be achieved by using more than one processor.

A memory204in the computing device200can be a read-only memory device or a random-access memory device in an implementation. However, other suitable types of storage devices can be used as the memory204. The memory204can include code and data206that is accessed by the processor202using a bus212. The memory204can further include an operating system208and application programs210, the application programs210including at least one program that permits the processor202to perform the techniques described herein. For example, the application programs210can include applications1through N, which further include video or image coding software that performs some or all of the techniques described herein. The computing device200can also include a secondary storage214, which can, for example, be a memory card used with a mobile computing device. For example, an image can be stored in whole or in part in the secondary storage214and loaded into the memory204as needed for processing.

The computing device200can also include one or more output devices, such as a display218. The display218may be, in one example, a touch-sensitive display that combines a display with a touch-sensitive element that is operable to sense touch inputs. The display218can be coupled to the processor202via the bus212. Other output devices that permit a user to program or otherwise use the computing device200can be provided in addition to or as an alternative to the display218. When the output device is or includes a display, the display can be implemented in various ways, including as a liquid crystal display, a cathode-ray tube display, or a light emitting diode display, such as an organic light emitting diode display.

The computing device200can also include or be in communication with an image-sensing device220, for example, a camera, or another image-sensing device, now existing or hereafter developed, which can sense an image such as the image of a user operating the computing device200. The image-sensing device220can be positioned such that it is directed toward the user operating the computing device200. For example, the position and optical axis of the image-sensing device220can be configured such that the field of vision includes an area that is directly adjacent to the display218and from which the display218is visible.

The computing device200can also include or be in communication with a sound-sensing device222, for example, a microphone or another sound-sensing device, now existing or hereafter developed, which can sense sounds near the computing device200. The sound-sensing device222can be positioned such that it is directed toward the user operating the computing device200and can be configured to receive sounds, for example, speech or other utterances, made by the user while the user operates the computing device200.

Implementations of the computing device200may differ from what is shown and described with respect toFIG. 2. In some implementations, the operations of the processor202can be distributed across multiple machines (wherein individual machines can have one or more processors) that can be coupled directly or across a local area or other network. In some implementations, the memory204can be distributed across multiple machines, such as a network-based memory or memory in multiple machines performing the operations of the computing device200. In some implementations, the bus212of the computing device200can be composed of multiple buses. In some implementations, the secondary storage214can be directly coupled to the other components of the computing device200or can be accessed via a network and can comprise an integrated unit, such as a memory card, or multiple units, such as multiple memory cards.

FIG. 3Ais a diagram of an example of a video stream300to be encoded and subsequently decoded. The video stream300represents a typical video stream that can be encoded into a compressed bitstream, for example, using the transmitting station102shown inFIG. 1, and subsequently decoded into an output video stream, for example, using the receiving station104shown inFIG. 1.

The video stream300includes a video sequence302. At the next level, the video sequence302includes a number of adjacent frames304. While three frames are depicted as the adjacent frames304, the video sequence302can include another number of adjacent frames304. The adjacent frames304can then be further subdivided into individual frames, for example, a frame306. At the next level, the frame306can be divided into a series of planes or segments308. The segments308can be subsets of frames that permit parallel processing, for example. The segments308can also be subsets of frames that can separate the video data into separate colors. For example, a frame306of color video data can include a luminance plane and two chrominance planes. The segments308may be sampled at different resolutions.

Whether or not the frame306is divided into segments308, the frame306may be further subdivided into blocks310, which can contain data corresponding to, for example, 16×16 pixels in the frame306. The blocks310can also be arranged to include data from one or more segments308of pixel data. The blocks310can also be of any other suitable size such as 4×4 pixels, 8×8 pixels, 16×8 pixels, 8×16 pixels, 16×16 pixels, or larger. Unless otherwise noted, the terms block and macroblock are used interchangeably herein.

FIG. 3Bis a diagram of an example of an image frame312to be encoded and subsequently decoded. The image frame312represents a typical image that can be encoded into a compressed bitstream or storage, for example, using the transmitting station102shown in FIG.1, and subsequently decoded for rendering at a display, for example, using the receiving station104shown inFIG. 1. The image frame312may have the same form as the frame306shown inFIG. 3A. For example, the image frame312may be divided into the image segments314and/or the image blocks316for further processing during encoding or decoding.

FIG. 4is a block diagram of an encoder400. The encoder400can be implemented, as described above, in the transmitting station102shown inFIG. 1, such as by providing a computer software program stored in memory, for example, the memory204shown inFIG. 2. The computer software program can include machine instructions that, when executed by a processor such as the processor202shown inFIG. 2, cause the transmitting station102to encode video data in the manner described inFIG. 4. The encoder400can also be implemented as specialized hardware (e.g., an integrated circuit) included in, for example, the transmitting station102. In some implementations, the encoder400is a hardware encoder.

The encoder400has the following stages to perform the various functions in a forward path (shown by the solid connection lines) to produce an encoded or compressed bitstream420using the video stream300as input: an intra/inter prediction stage402, a transform stage404, a quantization stage406, and an entropy encoding stage408. The encoder400may also include a reconstruction path (shown by the dotted connection lines) to reconstruct a frame for encoding of future blocks. InFIG. 4, the encoder400has the following stages to perform the various functions in the reconstruction path: a dequantization stage410, an inverse transform stage412, a reconstruction stage414, and a deblocking filter stage416. Other structural variations of the encoder400can be used to encode the video stream300.

When the video stream300is presented for encoding, respective adjacent frames304, such as the frame306, can be processed in units of blocks. At the intra/inter prediction stage402, respective blocks can be encoded using intra-frame prediction (also called intra-prediction) or inter-frame prediction (also called inter-prediction). In any case, a prediction block can be formed. In the case of intra-prediction, a prediction block may be formed from samples in the current frame that have been previously encoded and reconstructed. In the case of inter-prediction, a prediction block may be formed from samples in one or more previously constructed reference frames.

Next, the prediction block can be subtracted from the current block at the intra/inter prediction stage402to produce a residual block (also called a residual or prediction residual). The transform stage404transforms the residual into transform coefficients in, for example, the frequency domain using block-based transforms. The quantization stage406converts the transform coefficients into discrete quantum values, which are referred to as quantized transform coefficients, using a quantizer value or a quantization level. For example, the transform coefficients may be divided by the quantizer value and truncated.

The quantized transform coefficients are then entropy encoded by the entropy encoding stage408. The entropy-encoded coefficients, together with other information used to decode the block (which may include, for example, syntax elements such as used to indicate the type of prediction used, transform type, motion vectors, a quantizer value, or the like), are then output to the compressed bitstream420. The compressed bitstream420can be formatted using various techniques, such as variable length coding (VLC) or arithmetic coding. The compressed bitstream420can also be referred to as an encoded video stream or encoded video bitstream, and the terms will be used interchangeably herein.

The reconstruction path (shown by the dotted connection lines) can be used to ensure that the encoder400and a decoder500(described below with respect toFIG. 5) use the same reference frames to decode the compressed bitstream420. The reconstruction path performs functions that are similar to functions that take place during the decoding process (described below with respect toFIG. 5), including dequantizing the quantized transform coefficients at the dequantization stage410and inverse transforming the dequantized transform coefficients at the inverse transform stage412to produce a derivative residual block (also called a derivative residual).

At the reconstruction stage414, the prediction block that was predicted at the intra/inter prediction stage402can be added to the derivative residual to create a reconstructed block. The deblocking filter stage416can be applied to the reconstructed block to reduce distortion such as blocking artifacts. Implementations and examples of a deblocking filter used at the deblocking filter stage416are described below with respect toFIG. 6.

Implementations of the encoder400may differ from what is shown and described with respect toFIG. 4. In particular, the encoder400as shown inFIG. 4is an example of an encoder for encoding video data, such as a video frame or a video block. However, in other implementations, the encoder400may be an example of an encoder for encoding image data, such as an image frame or an image block.

In such an implementation, the encoder400omits the intra/inter prediction stage402. For example, the input image data is first processed at the transform stage404and then the quantization stage406before it enters the reconstruction path of stages410,412,414, and416. The output of the deblocking filter stage416can be sent to the transform stage404for further processing. If the reconstruction path is not needed, such as because the error level for the image frame meets a threshold, the reconstruction path may not be followed and the quantized image data may instead proceed to the entropy encoding stage408and then output to the compressed bitstream420. In some implementations where the encoder400is used to encode image data, the encoder400may omit the entropy encoding stage408.

In some implementations, the encoder400may be a non-transform based encoder for image or video coding. In such an implementation, the encoder400can quantize the residual signal directly without the transform stage404for certain blocks or frames. In some implementations, the quantization stage406and the dequantization stage410may be combined into a common stage.

FIG. 5is a block diagram of a decoder500. The decoder500can be implemented, as described above, in the receiving station104shown inFIG. 1, such as by providing a computer software program stored in memory, for example, the memory204shown inFIG. 2. The computer software program can include machine instructions that, when executed by a processor such as the processor202shown inFIG. 2, cause the receiving station104to decode video data in the manner described inFIG. 5. The decoder500can also be implemented in specialized hardware (e.g., an integrated circuit) included in, for example, the receiving station104.

The decoder500, similar to the reconstruction path of the encoder400described above, includes in one example the following stages to perform various functions to produce an output video stream516from the compressed bitstream420: an entropy decoding stage502, a dequantization stage504, an inverse transform stage506, an intra/inter prediction stage508, a reconstruction stage510, a deblocking filter stage512, and an optional post-filtering stage514. Other structural variations of the decoder500can be used to decode the compressed bitstream420.

When the compressed bitstream420is presented for decoding, the data elements within the compressed bitstream420can be decoded by the entropy decoding stage502to produce a set of quantized transform coefficients. The dequantization stage504dequantizes the quantized transform coefficients (e.g., by multiplying the quantized transform coefficients by the quantizer value), and the inverse transform stage506inverse transforms the dequantized transform coefficients to produce a derivative residual that can be identical to that created by the inverse transform stage412in the encoder400. Using header information decoded from the compressed bitstream420, the decoder500can use the intra/inter prediction stage508to create the same prediction block as was created in the encoder400(e.g., at the intra/inter prediction stage402).

At the reconstruction stage510, the prediction block can be added to the derivative residual to create a reconstructed block. The deblocking filter stage512can be applied to the reconstructed block to reduce blocking artifacts (e.g., using deblocking filtering, sample adaptive offset filtering, other loop filter functionality, or the like, or a combination thereof). Implementations and examples of a deblocking filter used at the deblocking filter stage512are described below with respect toFIG. 6. Other filtering can be applied to the reconstructed block. In this example, the post-filtering stage514is applied to the reconstructed block to reduce blocking distortion, and the result is output as the output video stream516. The output video stream516can also be referred to as a decoded video stream, and the terms will be used interchangeably herein.

Implementations of the decoder500may differ from what is shown and described with respect toFIG. 5. In some implementations, the decoder500can omit or otherwise not process data using the post-filtering stage514. Further, while the decoder500as shown inFIG. 5is an example of a decoder for decoding encoded video data, such as an encoded video frame or an encoded video block, in other implementations, the decoder500may be an example of a decoder for decoding encoded image data, such as an encoded image frame or an encoded image block.

In such an implementation, the decoder500omits the intra/inter prediction stage508. For example, the output of the deblocking filter stage512can be sent to the dequantization stage504for further processing. In another example, where the error level for the image frame meets a threshold, the output of the deblocking filter stage512can instead be sent to the post-filtering stage514and/or as output to a display for rendering. In some implementations where the decoder500is used to decode encoded image data, the decoder500may omit the entropy decoding stage502.

FIG. 6is a block diagram of an example of a spatially adaptive quantization-aware deblocking filter600(hereafter referred to as the deblocking filter600) used for encoding or decoding a frame, such as a video frame or an image frame. The deblocking filter600may, for example, be the deblocking filter used in the deblocking filter stage416shown inFIG. 4, the deblocking filter using in the deblocking filter stage512shown inFIG. 5, or both.

The deblocking filter600receives reconstructed frame and adaptive quantization field data602and uses those to produce a filtered frame604. The reconstructed frame includes data produced by reconstructing some number of decoded blocks, which decoded blocks represent dequantized and inverse transformed coefficients. For example, the reconstructed frame can be produced using the reconstruction stage414shown inFIG. 4or the reconstruction stage510shown inFIG. 5. The adaptive quantization field data represents weights applied to quantization values used to encode those blocks. The adaptive quantization field data includes values indicative of the quantization levels used to encode different areas of the frame. The values of the adaptive quantization field data are determined by analyzing the entire frame, such as to determine the areas in which to apply a higher or lower weight to the quantization.

The deblocking filter600includes an artifact removal stage606, a psychovisual modeling stage608, and a parameter adjustment stage610. The artifact removal stage606receives the reconstructed frame and adaptive quantization field data602. The artifact removal stage606filters the reconstructed frame to remove some number of blocking artifacts from the reconstructed frame. The filtering of the reconstructed frame by the artifact removal stage606is controlled using the adaptive quantization field data.

The artifact removal stage606compares the reconstructed frame to the original (e.g., pre-encoded) frame to determine differences between those frames as a result of the encoding. In areas where those differences reflect that the reconstructed frame does not accurately represent the original frame, the artifact removal stage606applies a filter to reduce the number of blocking artifacts. The artifact removal stage606modules a transition from a small difference to a large difference based on the adaptive quantization field data. For example, the artifact removal stage606may change the neighborhood size of the deblocking filter600based on the adaptive quantization field data. In another example, the artifact removal stage606may use a greater weight for the values within the original frame when performing the filtering, such as to err on the side of preserving more of the original frame data.

The deblocking filter600may be a directional filter such that blocking artifacts removed by the artifact removal stage606may be filtered based on a filtering direction. In particular, the deblocking filter600(e.g., at the artifact removal stage606or another stage (not shown) preceding the artifact removal stage606) can determine a most invariant direction within a given area of the frame and filter along that most invariant direction. The most invariant direction refers to a directional line of pixels that has a lowest variation in color, light, or other intensity. The filtering can include calculating average values for the pixels on each side of the directional line and replacing blocking artifacts with those average values.

The artifact removal stage606filters along the determined filtering direction by using the adaptive quantization field data for the respective area or areas of the frame to modulate parameters of the deblocking filter600. The directional filter600may have a number of parameters it uses for filtering frame data, including, for example, a non-linearity selection parameter, a filter size parameter, or a directional sensitivity parameter. The non-linearity selection parameter reflects data or types of data to preserve within the frame (e.g., to not remove by the filtering). The filter size parameter reflects a number of pixels that the filter is applied against at a given operation. The directional sensitivity parameter reflects a sensitivity of the filtering direction selection.

The adaptive quantization field data is used to modulate some or all of the parameters of the deblocking filter600based on the quantization levels used to encode different areas of the frame. Thus, during the artifact removal stage606, the adaptive quantization field data indicates a quantization level used to encode a given area of the frame. The artifact removal stage606then uses that quantization level information to control the removal of blocking artifacts within that given area. For example, the artifact removal stage606can modulate the non-linearity selection parameter for filtering a given area of the frame upon determining a fine detail to preserve within the frame (e.g., a scratch on a painted surface).

Modulating the non-linearity selection parameter includes changing a threshold indicative of a difference between the original frame and the reconstructed frame. The threshold may, for example, reflect a maximum acceptable difference for a given area of the frame based on the values of the adaptive quantization field data that are associated with that given area. For example, a greater difference between the original frame and the reconstructed frame within a given area of the frame may be acceptable according to the non-linearity selection parameter where a greater quantization weight is applied in that given area.

The psychovisual modeling stage608uses software rules for processing the frame after filtering at the artifact removal stage606based on visually perceptible qualities of the frame. The software rules of the psychovisual modeling stage608focus on three properties of vision: first, that gamma correction should not be separately applied to every RGB channel; second, that high frequency changes in blue color data can be less precisely encoded; and third, that areas including larger amounts of visual noise within a frame can be less precisely encoded.

The first property of vision is driven by the overlap of sensitivity spectra of the cones of the human eye. For example, because there is some relationship between the amount of yellow light seen and sensitivity to blue light, changes in blue color data in the vicinity of yellow color data can be compressed less precisely. YUV color spaces are defined as linear transformations of gamma-compressed RGB and are therefore not powerful enough to model such phenomena. The second property of vision is driven by the color receptors of the retina of the human eye. In particular, the human eye has lower spatial resolution in blue than in red and green, and the retina has almost no blue receptors in the high-resolution area. The third property of vision is defined based on a relationship between visibility and proximal visual activity. That is, the visibility of fine structures in an area of an image may depend on the amount of visual activity in the vicinity of that area.

Although the software rules of the psychovisual modeling stage608are described with reference to three properties of vision, other numbers of properties of vision, other rules related to video or image encoding or perceptibility, or a combination thereof may be used to define or otherwise configure the software rules of the psychovisual modeling stage608.

The software rules of the psychovisual modeling stage608can thus be used to determine a number of blocking artifacts remaining within some or all of the areas of the frame. The psychovisual modeling stage608next compares that number of blocking artifacts to a psychovisual model threshold reflecting a maximum number of visually perceptible artifacts to include in the frame. The value of the psychovisual model threshold may be configured by default or set empirically, such as by iterating the deblocking filter600over N frames. If the psychovisual model threshold is exceeded such that the number of blocking artifacts remaining within the frame after the filtering at the artifact removal stage606is too large, the psychovisual modeling stage608sends the filtered frame and associated data to the parameter adjustment stage610. Otherwise, the filtered frame604is output.

The psychovisual model threshold may also or instead reflect a maximum acceptable difference between the original frame and the frame after processing at the artifact removal stage606. For example, the psychovisual modeling stage608can use the psychovisual model threshold as a basis for comparing coefficient or pixel values within a specific area of the original frame and the frame after processing at the artifact removal stage606. If that comparison indicates that the coefficient or pixel value differences between those frames is less than the psychovisual model threshold, the differences are preserved. However, if that comparison indicates that those differences are not less than the psychovisual model threshold, the psychovisual modeling stage608may cause the frame to undergo further filtering at the artifact removal stage606.

The parameter adjustment stage610adjusts some parameter used by the artifact removal stage606before returning the frame to the artifact removal stage606for further filtering. The parameter adjustment stage610may, for example, adjust one or more values of the adaptive quantization field data. For example, based on the results of the psychovisual modeling stage608, the parameter adjustment stage610can increase or decrease the adaptive quantization field data values for a given area of the frame, such as to correspondingly increase or decrease the amount of quantization within that given area.

Adjustments to the adaptive quantization field data may be limited by an error level definition representing a maximum quantization error for the frame. For example, where an adjustment to the adaptive quantization field data causes the total quantization error for the frame to exceed the error level definition, that adjustment is either discarded or offset by a corresponding adjustment to another area of the frame. For example, where a determination is made to adjust a first area of the frame by increasing the quantization weight for that area by X, and where that increase causes the total quantization error to exceed the error level definition, a corresponding determination is also made to adjust a second area of the frame by decreasing the quantization weight therefore by X.

Adjustments determined or otherwise made at the parameter adjustment stage610are looped back to the artifact removal stage606for further filtering. The deblocking filter600may iterate multiple times through some or all of the artifact removal stage606, the psychovisual modeling stage608, or the parameter adjustment stage610before the finally filtered frame604is output. Iterating through the filtering of the frame using the deblocking filter600can include producing a model for the deblocking filter600. For example, the deblocking filter600can include functionality for learning the types of filtering applied to and the types of adjustments that are made for a given frame. When another frame is received for filtering, the deblocking filter600may use the learned model to make multiple filtering applications or adjustments at a time, such as to improve processing speeds.

Implementations of the deblocking filter600may differ from what is shown and described with respect toFIG. 6. In some implementations, the psychovisual modeling stage608and the parameter adjustment stage610can be external to the deblocking filter600. For example, the deblocking filter600may only include functionality for performing the operations of the artifact removal stage606. The deblocking filter600may thus output the filtered frame604produced using the artifact removal stage606to the psychovisual modeling stage608, which may then process the filtered frame604to determine whether to re-filter the filtered frame604using the deblocking filter600.

In some such implementations, the psychovisual modeling stage608can process the reconstructed frame and adaptive quantization field data602before the artifact removal stage606. For example, the psychovisual modeling stage608can process the reconstructed frame and adaptive quantization field data602to determine a number of blocking artifacts to remove from the reconstructed frame and the locations of those blocking artifacts within the frame. The artifact removal stage606then removes those blocking artifacts and outputs the filtered frame604. The parameter adjustment stage610can then receive the filtered frame604output from the deblocking filter600and process the filtered frame604to determine whether to adjust values of the adaptive quantization field data and re-filter the filtered frame604using the adjusted adaptive quantization field data.

Where a determination is made to adjust the adaptive quantization field data and re-filter the filtered frame604, the parameter adjustment stage610accordingly adjusts the respective values of the adaptive quantization field data. The parameter adjustment stage610then sends the filtered frame and adjusted adaptive quantization field data back to the deblocking filter600for re-processing by the artifact removal stage606. Where a determination is made to not adjust the adaptive quantization field data and re-filter the filtered frame604, the parameter adjustment stage610causes the filtered frame604to be output, such as to a compressed bitstream, for storage, for further processing, for rendering at a display, or the like.

In some implementations, the performance of the psychovisual modeling stage608may be different depending on the type of data being processed by the deblocking filter600. For example, where the deblocking filter600is processing image data (e.g., such that the reconstructed frame is a reconstructed image), the psychovisual modeling stage608may use a larger threshold for determining visual perceptibility, as the filtered frame604will be affixed when displayed. However, where the deblocking filter600is processing video data (e.g., such that the reconstructed frame is a reconstructed video frame), the psychovisual modeling stage608may use a smaller threshold for determining the visual perceptibility, as the filtered frame604will only be displayed for a very short amount of time.

In some implementations, the deblocking filter600may filter the frame other than by determining and using a most invariant direction. For example, a machine learning algorithm may be used to determine an optimal filtering direction to use at the artifact removal stage606. In another example, data indicative of the filtering direction to use may be explicitly signaled to a decoder from an encoder, such as within a frame header.

FIG. 7is a block diagram representing portions of a frame700. The frame may be a video frame, for example, the video frame306shown inFIG. 3A, or an image frame, for example, the image frame312shown inFIG. 3B. As shown, the frame700includes four 64×64 blocks710, in two rows and two columns in a matrix or Cartesian plane. In some implementations, a 64×64 block may be a maximum coding unit, N=64. Each 64×64 block may include four 32×32 blocks720. Each 32×32 block may include four 16×16 blocks730. Each 16×16 block may include four 8×8 blocks740. Each 8×8 block740may include four 4×4 blocks750. Each 4×4 block750may include 16 pixels, which may be represented in four rows and four columns in each respective block in the Cartesian plane or matrix.

The pixels may include information representing an image captured in the frame700, such as luminance information, color information, and location information. In some implementations, a block, such as a 16×16 pixel block as shown, may include a luminance block760, which may include luminance pixels762; and two chrominance blocks770,780, such as a U or Cb chrominance block770, and a V or Cr chrominance block780. The chrominance blocks770,780may include chrominance pixels790. For example, the luminance block760may include 16×16 luminance pixels762and each chrominance block770,780may include 8×8 chrominance pixels790as shown. Although one arrangement of blocks is shown, any arrangement may be used. AlthoughFIG. 7shows N×N blocks, in some implementations, N×M blocks may be used. For example, 32×64 blocks, 64×32 blocks, 16×32 blocks, 32×16 blocks, or any other size blocks may be used. In some implementations, N×2N blocks, 2N×N blocks, or a combination thereof, may be used.

In some implementations, coding the frame700may include ordered block-level coding. Ordered block-level coding may include coding blocks of a frame in an order, such as raster-scan order, wherein blocks may be identified and processed starting with a block in the upper left corner of the frame, or portion of the frame, and proceeding along rows from left to right and from the top row to the bottom row, identifying each block in turn for processing. For example, the 64×64 block in the top row and left column of a frame may be the first block coded and the 64×64 block immediately to the right of the first block may be the second block coded. The second row from the top may be the second row coded, such that the 64×64 block in the left column of the second row may be coded after the 64×64 block in the rightmost column of the first row.

In some implementations, coding a block of the frame700may include using quad-tree coding, which may include coding smaller block units within a block in raster-scan order. For example, the 64×64 block shown in the bottom left corner of the portion of the frame700may be coded using quad-tree coding wherein the top left 32×32 block may be coded, then the top right 32×32 block may be coded, then the bottom left 32×32 block may be coded, and then the bottom right 32×32 block may be coded. Each 32×32 block may be coded using quad-tree coding wherein the top left 16×16 block may be coded, then the top right 16×16 block may be coded, then the bottom left 16×16 block may be coded, and then the bottom right 16×16 block may be coded.

Each 16×16 block may be coded using quad-tree coding wherein the top left 8×8 block may be coded, then the top right 8×8 block may be coded, then the bottom left 8×8 block may be coded, and then the bottom right 8×8 block may be coded. Each 8×8 block may be coded using quad-tree coding wherein the top left 4×4 block may be coded, then the top right 4×4 block may be coded, then the bottom left 4×4 block may be coded, and then the bottom right 4×4 block may be coded. In some implementations, 8×8 blocks may be omitted for a 16×16 block, and the 16×16 block may be coded using quad-tree coding wherein the top left 4×4 block may be coded, then the other 4×4 blocks in the 16×16 block may be coded in raster-scan order.

In some implementations, coding the frame700may include encoding the information included in an original, or input, frame by, for example, omitting some of the information in the original frame from a corresponding encoded frame. For example, the coding may include reducing spectral redundancy, reducing spatial redundancy, reducing temporal redundancy, or a combination thereof.

In some implementations, reducing spectral redundancy may include using a color model based on a luminance component (Y) and two chrominance components (U and V or Cb and Cr), which may be referred to as the YUV or YCbCr color model, or color space. Using the YUV color model may include using a relatively large amount of information to represent the luminance component of a portion of a frame, and using a relatively small amount of information to represent each corresponding chrominance component for the portion of the frame. For example, a portion of a frame may be represented by a high-resolution luminance component, which may include a 16×16 block of pixels, and by two lower resolution chrominance components, each of which represents the portion of the frame as an 8×8 block of pixels. A pixel may indicate a value, for example, a value in the range from 0 to 255, and may be stored or transmitted using, for example, eight bits. Although this disclosure is described in reference to the YUV color model, another color model may be used.

In some implementations, reducing spatial redundancy may include transforming a block into the frequency domain using, for example, a discrete cosine transform. For example, a unit of an encoder, such as the transform stage404shown inFIG. 4, may perform a discrete cosine transform using transform coefficient values based on spatial frequency.

In some implementations, reducing temporal redundancy may include using similarities between frames to encode a frame using a relatively small amount of data based on one or more reference frames, which may be previously encoded, decoded, and reconstructed frames of the video stream. For example, a block or pixel of a current frame may be similar to a spatially corresponding block or pixel of a reference frame. In some implementations, a block or pixel of a current frame may be similar to block or pixel of a reference frame at a different spatial location, and reducing temporal redundancy may include generating motion information indicating the spatial difference, or translation, between the location of the block or pixel in the current frame and corresponding location of the block or pixel in the reference frame.

In some implementations, reducing temporal redundancy may include identifying a portion of a reference frame that corresponds to a current block or pixel of a current frame. For example, a reference frame, or a portion of a reference frame, which may be stored in memory, may be searched to identify a portion for generating a predictor to use for encoding a current block or pixel of the current frame with maximal efficiency. For example, the search may identify a portion of the reference frame for which the difference in pixel values between the current block and a prediction block generated based on the portion of the reference frame is minimized, and may be referred to as motion searching. In some implementations, the portion of the reference frame searched may be limited. For example, the portion of the reference frame searched, which may be referred to as the search area, may include a limited number of rows of the reference frame. In an example, identifying the portion of the reference frame for generating a predictor may include calculating a cost function, such as a sum of absolute differences (SAD), between the pixels of portions of the search area and the pixels of the current block.

In some implementations, the spatial difference between the location of the portion of the reference frame for generating a predictor in the reference frame and the current block in the current frame may be represented as a motion vector. The difference in pixel values between the predictor block and the current block may be referred to as differential data, residual data, a prediction error, or as a residual block. In some implementations, generating motion vectors may be referred to as motion estimation, and a pixel of a current block may be indicated based on location using Cartesian coordinates as fx,y. Similarly, a pixel of the search area of the reference frame may be indicated based on location using Cartesian coordinates as rx,y. A motion vector (MV) for the current block may be determined based on, for example, a SAD between the pixels of the current frame and the corresponding pixels of the reference frame.

Although described herein with reference to matrix or Cartesian representation of a frame for clarity, a frame may be stored, transmitted, processed, or any combination thereof, in any data structure such that pixel values may be efficiently represented for a frame or image. For example, a frame may be stored, transmitted, processed, or any combination thereof, in a two-dimensional data structure such as a matrix as shown, or in a one-dimensional data structure, such as a vector array. In an implementation, a representation of the frame, such as a two-dimensional representation as shown, may correspond to a physical location in a rendering of the frame as an image. For example, a location in the top left corner of a block in the top left corner of the frame may correspond with a physical location in the top left corner of a rendering of the frame as an image.

As described above, the frame700may be a frame of a video sequence, or it may be an image. Regardless of whether the frame700represents image data or video data, the frame700includes a single picture to encode or decode. The picture included in the frame700may have different types of detail in different areas. An area of a frame as described herein refers to an M×N-sized region of the frame, where M and N may be the same or a different value. For example, an area of a frame may be a single block (e.g., an 8×8 block) within the frame. In another example, an area of a frame may be multiple blocks within the frame. In yet another example, different areas of a frame to encode may be different sized blocks (e.g., some 8×8, some 4×4, some 16×16, etc.).

For example, the frame700may include a picture of a forest with a sky over the forest. The forest may be shown in the lower two 64×64 blocks710, while the sky is shown in the upper two 64×64 blocks710. The data included in the lower two 64×64 blocks710may include more detail than the data included in the upper two 64×64 blocks710. For example, the depiction of the forest may include a number of small details for branches or leaves of trees along with other plant and/or animal life present in the forest. In contrast, the depiction of the sky may merely reflect gradient shades of blue. As such, the majority of the information in the frame700is located in the lower two 64×64 blocks710.

Encoding or decoding the frame700can include quantizing the different areas of the frame700(e.g., the different blocks710,720,730,740,750) by applying different weights to the quantization value for the frame700based on the data included in those areas. For example, a first area of an image that includes more information (e.g., greater detail) may be quantized using a smaller weight than a second area that includes less information (e.g., lesser detail). The information contained within the first area will be quantized less than the information contained within the second area. This results in a smaller loss of information within the first area than within the second area.

With the example in which the frame700shows a forest and a sky, the adaptive quantization field data for the frame700can be produced to reflect that greater quantization is used for the information within the blocks representing the sky and lesser quantization is used for the information within the blocks representing the forest. More particularly, values of the adaptive quantization field data for the blocks including the sky information reflect that those blocks are quantized using a larger weight than the blocks including the forest information. The values of the adaptive quantization field data for all four of the 64×64 blocks710may be controlled by an error level definition represents the maximum quantization error resulting from the encoding of the frame700.

Techniques for encoding or decoding video frames are now described with respect toFIGS. 8 and 9.FIG. 8is a flowchart diagram of an example of a technique800for encoding or decoding a video frame or an image frame using a spatially adaptive quantization-aware deblocking filter.FIG. 9is a flowchart diagram of an example of a technique900for iteratively filtering a video frame or an image frame according to adaptive quantization field data.

One or both of the technique800or the technique900can be implemented, for example, as a software program that may be executed by computing devices such as the transmitting station102or the receiving station104shown inFIG. 1, or otherwise by the computing device200shown inFIG. 2. For example, the software program can be or otherwise include an encoder, such as the encoder400shown inFIG. 4, or a decoder, such as the decoder500shown inFIG. 5.

The software program can include machine-readable instructions that may be stored in a memory such as the memory204or the secondary storage214shown inFIG. 2, and that, when executed by a processor, such as the processor202shown inFIG. 2, may cause the computing device to perform one or both of the technique800or the technique900.

The technique800and/or the technique900, or an encoder and/or decoder (e.g., the encoder400and/or the decoder500) used to perform the technique800and/or the technique900, can be implemented using specialized hardware or firmware (e.g., an integrated circuit). As explained above, some computing devices may have multiple memories or processors, and the operations described in the technique800and the technique900can be distributed using multiple processors, memories, or both.

Referring first toFIG. 8, the technique800for encoding or decoding a video frame or an image frame using a spatially adaptive quantization-aware deblocking filter is shown. The technique800can be performed for encoding a video frame or an image, such as to a bitstream. Alternatively, the technique800can be performed for decoding an encoded video frame or an encoded image, such as from a bitstream. For example, during encoding, the technique800can be performed by the reconstruction path (e.g., the stages410,412,414, and416) of the encoder400shown inFIG. 4. In another example, during decoding, the technique800can be performed by the stages504,506,510, and512of the decoder500shown inFIG. 5.

At802, quantized transform coefficients associated with a video frame or image (either hereafter referred to as a frame) to encode or decode are dequantized and inverse transformed. The quantized transform coefficients represent pixel values of the original video frame or original image after those pixel values are transformed and quantized during encoding. The quantized transform coefficients are coefficients of blocks of the video frame or image. Decoded blocks are produced by dequantizing and inverse transforming the quantized transform coefficients. The decoded blocks may, for example, be derivative residual blocks, such as where the frame is a video frame. At804, the decoded video blocks are reconstructed into a reconstructed frame.

At806, the reconstructed frame is filtered according to adaptive quantization field data associated with the frame. The adaptive quantization field data may be produced at a quantization stage of an encoder or received by a decoder, such as within a bitstream. Filtering the reconstructed frame according to the adaptive quantization field data includes applying a deblocking filter to the reconstructed frame to remove some number of blocking artifacts from the reconstructed frame. The number of blocking artifacts to be removed by the deblocking filter is controlled by the adaptive quantization field data. A first filtered frame is produced as a result of filtering the reconstructed frame.

The performance of the deblocking filter may controlled or otherwise configured using one or more parameters of the deblocking filter. As such, applying the deblocking filter to the reconstructed frame according to the adaptive quantization field data can include applying some or all of the filter parameters to the reconstructed frame, where those filter parameters are defined based on the adaptive quantization field data. The parameters may, for example, include a non-linearity selection parameter, a filter size parameter, a directional sensitivity parameter, or the like, or a combination thereof. Furthermore, given that the filter parameters are defined based on the adaptive quantization field data, the adaptive quantization field data can be used to modulate one or more parameters of the deblocking filter.

Using the adaptive quantization field data to module parameters of the deblocking filter can include modulating a non-linearity selection parameter of the deblocking filter according to the adaptive quantization field data to determine to preserve data within the reconstructed frame. For example, the non-linearity selection parameter can be modulated for determining which types of data to preserve within the reconstructed frame. As a result, the coefficients used to represent such types of data within the reconstructed frame are not processed by the deblocking filter.

For example, the non-linearity selection parameter can be modulated according to a psychovisual model. The psychovisual model can indicate that certain types of data may be more or less visually perceptible. Thus the non-linearity selection parameter can be modulated based on the psychovisual model to cause more visually perceptible data within the frame to be preserved and to cause less visually perceptible data within the frame to be subject to modification.

The adaptive quantization field data indicates to the deblocking filter the areas of the reconstructed frame that were processed using relatively higher or relatively lower quantization values. As described above, the adaptive quantization field data represents weights applied to quantization values used to encode the blocks of the frame. Thus, the quantization values applied at local areas of the reconstructed frame are used to guide the processing of the deblocking filter.

For example, the reconstructed frame may include a first area that was processed during encoding using a relatively high quantization value and a second area that was processed during encoding using a relatively low quantization value. The deblocking filter may thus use the adaptive quantization field data for the reconstructed frame to apply a relatively stronger filter to the first area of the reconstructed frame and to apply a relatively weaker filter to the second area. As a result, the deblocking filter removes a greater number of blocking artifacts from the first area than from the second area.

At808, a number of blocking artifacts remaining within the first filtered frame is determined. Determining the number of blocking artifacts remaining within the first filtered frame includes processing data of the first filtered frame using a psychovisual model. The psychovisual model may, for example, be a model configured to process video frame or image data based on human perceptibility. Implementations and examples for determining to adjust adaptive quantization field data are further described below with respect toFIG. 9.

At810, the adaptive quantization field data is adjusted. Adjusting the adaptive quantization field data includes changing (e.g., increasing or decreasing) the weight applied to the quantization values for at least some of the areas of the first filtered frame. For example, adjusting the adaptive quantization field data can include increasing a value of the adaptive quantization field data, where the value is associated with one or more of the decoded blocks used to produce the reconstructed frame. For example, the value can reflect a weight applied to a quantization value in a particular one of those decoded blocks, the value resulting from applying that weight to that quantization value, or another value reflected by the adaptive quantization field data.

The adjustments to the adaptive quantization field data can be limited by data associated with the frame being encoded or decoded. For example, increases to the adaptive quantization field data may be limited by an error level definition associated with the frame. The error level definition represents the maximum quantization error resulting from the encoding of the frame. As such, the adaptive quantization field data may not be increased in a way that causes the quantization error for the frame to exceed the error level definition. In another example, decreases to the adaptive quantization field data may be limited by a file size definition for the frame (e.g., during encoding operations). As such, the adaptive quantization field data may not be decreased in a way that causes the total file size of the resulting encoded frame to exceed the file size definition.

At812, the first filtered frame produced by the earlier filtering of the reconstructed frame is re-filtered according to adjusted adaptive quantization field data. Re-filtering the first filtered frame according to the adjusted adaptive quantization field data includes applying the deblocking filter to the first filtered frame to remove some number of blocking artifacts from the first filtered frame. The number of blocking artifacts to be removed by the deblocking filter is controlled by the adjusted adaptive quantization field data. A second filtered frame is produced as a result of filtering the first filtered frame. The re-filtering can be performed in the same or substantially the same way as the earlier filtering of the reconstructed frame.

In some implementations where the technique800is performed for decoding an encoded video frame or an encoded image, the technique800includes decoding syntax data associated with the encoded video frame or the encoded image, such as from a bitstream. The syntax data includes the quantized transform coefficients of the encoded blocks of the encoded video frame or of the encoded image. The syntax data also includes the adaptive quantization field data.

The syntax data may be some or all of the data stored in the bitstream or another data store and which is associated with the encoded video frame or the encoded image. For example, the syntax data may include the quantized transform coefficients for some or all of the encoded blocks of the encoded video frame or encoded image. In another example, the syntax data can include metadata, such as from a frame header for the encoded video frame or from an image header for the encoded image.

In some implementations, the technique800includes determining whether to adjust at the adaptive quantization field data prior to adjusting the adaptive quantization field data. For example, the determination can be made by comparing the number of blocking artifacts remaining within the first filtered frame to a threshold, such as a psychovisual model threshold.

In some implementations, the technique800includes outputting the second filtered frame produced by re-filtering the first filtered frame using the adjusted adaptive quantization field. For example, when the technique800is performed for encoding a video frame or an image, the second filtered frame can be output to a compressed bitstream, such as the compressed bitstream420shown inFIG. 4, or to storage for later transmission, such as to the receiving station104. In another example, when the technique800is performed for decoding an encoded video frame or an encoded image, the second filtered frame can be output to an output video stream, such as the output video stream516shown inFIG. 5, or for rendering, such as at the display218shown inFIG. 2.

In some implementations, the technique800includes further adjusting the adaptive quantization field data. For example, responsive to re-filtering the reconstructed frame (e.g., to produce second filtered frame data), the adaptive quantization field data may be further adjusted, such as based on a number of blocking artifacts within the second filtered video frame. For example, a determination can be made that the first filtered frame data produced by the first filtering includes too many blocking artifacts (e.g., based on a threshold value associated with a psychovisual model). The further adjustment to the adaptive quantization field data can be based on that determination.

Subsequent to the further adjustment of the adaptive quantization field data, the deblocking filter can be used to further re-filter the reconstructed frame data, such as to produce a third filtered frame. The third filtered frame may then be output, such as to a compressed bitstream, to a compressed image storage, or for further processing (e.g., during encoding), or to an output video stream or for image rendering (e.g., during decoding). In some implementations, the third filtered frame can be output along with the second filtered frame. In some implementations, the third filtered frame can be output instead of the second filtered frame.

Referring next toFIG. 9, the technique900for iteratively filtering a video frame or an image frame according to adaptive quantization field data is shown. The technique900can be performed during encoding operations for encoding a video frame or an image, such as to a bitstream. Alternatively, the technique900can be performed during decoding operations for decoding an encoded video frame or an encoded image, such as from a bitstream. For example, during encoding, the technique900can be performed using the deblocking filter600shown inFIG. 6(when included in an encoder) or the deblocking filter of the deblocking filter stage416shown inFIG. 4. In another example, during decoding, the technique900can be performed using the deblocking filter600(when included in a decoder) or the deblocking filter of the deblocking filter512shown inFIG. 5.

At902, frame data and adaptive quantization field data are received. The frame data may, for example, be a reconstructed frame, such as which may be produced based on dequantized and inverse transformed video or image coefficients. The adaptive quantization field data represents weights applied to quantization values used to quantize those video or image coefficients. At904, the frame data is filtered according to the adaptive quantization field data. Implementations and examples for filtering frame data according to adaptive quantization field data are described above with respect toFIG. 8. At906, a number of blocking artifacts remaining within the filtered frame data is identified.

At908, a determination is made as to whether the number of blocking artifacts remaining within the filtered frame data exceeds a psychovisual model threshold. The psychovisual model threshold reflects a maximum number of visually perceptible blocking artifacts that can be present within a frame without the frame incurring a loss in visual quality. As such, determining whether the number of blocking artifacts remaining within the filtered frame data exceeds the psychovisual model threshold includes using the psychovisual model to determine that the number of blocking artifacts remaining within the filtered frame data exceeds a maximum number of visually perceptible blocking artifacts allowed within the filtered frame.

At910, responsive to a determination that the number of blocking artifacts remaining within the filtered frame data exceeds the psychovisual model threshold, one or more values of the adaptive quantization field data are increased, such as to cause more quantization to occur within the one or more corresponding areas of the frame. The technique900then returns to904, where the filtered frame data is further filtered according to the increase value or increased values of the adaptive quantization field data.

At912, responsive to a determination that the number of blocking artifacts within the filtered frame data does not exceed the psychovisual model threshold, the filtered frame data are output. For example, where the technique900is performed to encode the frame, the filtered frame data is output from a reconstruction path of the encoder (e.g., the stages410,412,414shown inFIG. 4). In another example, where the technique900is performed to decode the frame, the filtered frame data is output to an output video stream (e.g., the output video stream516shown inFIG. 5) to a post-filtering stage (e.g., the stage514) before the output video stream.

In some implementations, after identifying a number of blocking artifacts remaining within the filtered frame data, the technique900includes determining whether a file size for the filtered frame data including that number of remaining blocking artifacts exceeds a file size threshold. The file size threshold reflects a file size definition for the frame data, such as a maximum total file size for an encoded form of the frame data. Determining the file size for the filtered frame data can include encoding the filtered frame data as a temporary encoded frame and identifying the total storage requirements for that temporary encoded frame.

Responsive to determining that the file size for the filtered frame data exceeds the file size threshold, one or more values of the adaptive quantization field data are decreased, such as to cause less quantization to occur within the one or more corresponding areas of the frame. In some implementations, the technique900may then return to904for further filtering of the filtered frame data.

FIG. 10is an illustration of examples of reproductions of an original video frame or image1000A using different filtering or filter-less techniques. The original video frame or image1000A represents a video frame or image before it is encoded and/or decoded, such as using the encoding and decoding system100shown inFIG. 1. A first reproduction1000B represents the original video frame or image1000A after encoding and/or decoding without use of a deblocking filter. A second reproduction1000C represents the original video frame or image1000A after encoding and/or decoding with the use of a constant deblocking filter. A third reproduction1000D represents the original video frame or image1000A after encoding and/or decoding with the use of a spatially adaptive quantization-aware deblocking filter, for example, the deblocking filter600shown inFIG. 6.

The aspects of encoding and decoding described above illustrate some examples of encoding and decoding techniques and hardware components configured to perform all or a portion of those examples of encoding and/or decoding techniques. However, it is to be understood that encoding and decoding, as those terms are used in the claims, could mean compression, decompression, transformation, or other processing or changing of data.

The word “example” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “example” is not necessarily to be construed as being preferred or advantageous over other aspects or designs. Rather, use of the word “example” is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise or clearly indicated otherwise by the context, the statement “X includes A or B” is intended to mean any of the natural inclusive permutations thereof. That is, if X includes A; X includes B; or X includes both A and B, then “X includes A or B” is satisfied under any of the foregoing instances.

In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more,” unless specified otherwise or clearly indicated by the context to be directed to a singular form. Moreover, use of the term “an implementation” or the term “one implementation” throughout this disclosure is not intended to mean the same implementation unless described as such.

Implementations of the transmitting station102and/or the receiving station104(and the algorithms, methods, instructions, etc., stored thereon and/or executed thereby, including by the encoder400and/or the decoder500and including using the technique800and/or the technique900) can be realized in hardware, software, or any combination thereof. The hardware can include, for example, computers, intellectual property cores, application-specific integrated circuits, programmable logic arrays, optical processors, programmable logic controllers, microcode, microcontrollers, servers, microprocessors, digital signal processors, or any other suitable circuit. In the claims, the term “processor” should be understood as encompassing any of the foregoing hardware, either singly or in combination. The terms “signal” and “data” are used interchangeably. Further, portions of the transmitting station102and the receiving station104do not necessarily have to be implemented in the same manner.

The transmitting station102or the receiving station104may be implemented using a general purpose computer or general purpose processor with a computer program that, when executed, carries out any of the respective methods, algorithms, and/or instructions described herein. In addition, or alternatively, for example, a special purpose computer/processor may be utilized, which can include other hardware for carrying out any of the methods, algorithms, or instructions described herein.

The above-described embodiments, implementations, and aspects have been described in order to facilitate easy understanding of this disclosure and do not limit this disclosure. On the contrary, this disclosure is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope is to be accorded the broadest interpretation as is permitted under the law so as to encompass all such modifications and equivalent arrangements.