Bilateral filters in video coding with reduced complexity

An example method of filtering a reconstructed block of video data includes obtaining, by one or more processors, reconstructed samples of a current block of the video data; and selectively bilaterally filtering, by the one or more processors, the reconstructed samples of the current block to generate a filtered current block. In this example, selectively bilaterally filtering the reconstructed samples of the current block comprises refraining from bilaterally filtering at least one reconstructed sample of the current block such that the filtered current block includes at least one non-bilaterally filtered sample.

TECHNICAL FIELD

This disclosure relates to video coding.

BACKGROUND

Video compression techniques perform spatial (intra-picture) prediction and/or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based video coding, a video slice (i.e., a video frame or a portion of a video frame) may be partitioned into video blocks, which may also be referred to as treeblocks, coding units (CUs) and/or coding nodes. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction with respect to reference samples in neighboring blocks in the same picture. Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction with respect to reference samples in neighboring blocks in the same picture or temporal prediction with respect to reference samples in other reference pictures. Spatial or temporal prediction results in a predictive block for a block to be coded. Residual data represents pixel differences between the original block to be coded and the predictive block. An inter-coded block is encoded according to a motion vector that points to a block of reference samples forming the predictive block, and the residual data indicating the difference between the coded block and the predictive block. An intra-coded block is encoded according to an intra-coding mode and the residual data. For further compression, the residual data may be transformed from the pixel domain to a transform domain, resulting in residual transform coefficients, which then may be quantized.

SUMMARY

In general, this disclosure describes filtering techniques that may be used in a post-processing stage, as part of in-loop coding, or in a prediction stage of video coding. The filtering techniques of this disclosure may be applied to existing video codecs, such as High Efficiency Video Coding (HEVC), or be an efficient coding tool in any future video coding standards.

In one example, a method of filtering a reconstructed block of video data includes obtaining, by one or more processors, reconstructed samples of a current block of the video data; and selectively filtering, by the one or more processors, the reconstructed samples of the current block to generate a filtered current block. In this example, selectively filtering the reconstructed samples of the current block comprises refraining from filtering at least one reconstructed sample of the current block such that the filtered current block includes at least one non-filtered sample and at least one filtered sample.

In another example, an apparatus for filtering a reconstructed block of video data includes a memory configured to store video data; and one or more processors. In this example, the one or more processors are configured to obtain reconstructed samples of a current block of the video data; and selectively filter the reconstructed samples of the current block to generate a filtered current block. In this example, to selectively filter the reconstructed samples of the current block, the one or more processors are configured to refrain from filtering at least one reconstructed sample of the current block such that the filtered current block includes at least one non-filtered sample and at least one filtered sample.

In another example, an apparatus for filtering a reconstructed block of video data includes means for obtaining reconstructed samples of a current block of the video data; and means for selectively filtering the reconstructed samples of the current block to generate a filtered current block. In this example, the means for selectively filtering the reconstructed samples of the current block are configured to refrain from filtering at least one reconstructed sample of the current block such that the filtered current block includes at least one non-filtered sample and at least one filtered sample.

In another example, a computer-readable storage medium stores instructions, that when executed, cause one or more processors of a device for filtering a reconstructed block of video data to obtain reconstructed samples of a current block of the video data, and selectively filter the reconstructed samples of the current block to generate a filtered current block. In this example, the instructions that cause the one or more processors to selectively filter the reconstructed samples of the current block comprise instructions that cause the one or more processors to refrain from filtering at least one reconstructed sample of the current block such that the filtered current block includes at least one non-filtered sample and at least one filtered sample.

DETAILED DESCRIPTION

Video coders (e.g., video encoders and video decoders) may perform various filtering operations on video data. For instance, to preserve edges and reduce noises, a video decoder may perform bilateral filtering on a sample of video data by replacing the sample with a weighted average of itself and its neighbors.

It may be generally desirable for a video coder to be able to process multiple blocks of video data in parallel. For instance, a video decoder may reconstruct and filter the samples of several blocks of video data at the same time. By processing multiple blocks of video data in parallel, a video coder may reduce the amount of time required to decode pictures of video data. However, in some cases, it may not be possible to process some blocks of video data in parallel. For instance, if the decoding and/or reconstruction of samples of a current block depends on filtered samples of a neighboring block, it may decrease throughput since the decoding and/or reconstruction of samples of the current block needs to wait till the filtering process of the neighboring block is finished.

In accordance with one or more techniques of this disclosure, a video coder may selectively filter samples of a current block such that the filtering does not prevent parallel processing of neighboring blocks. For instance, a video decoder may bilaterally filter samples of a current block that may be not utilized by neighboring blocks for intra prediction and refrain from bilaterally filtering samples of a current block that may be utilized by neighboring blocks for intra prediction. In this way, a video coder may still obtain some of the benefits of filtering while still being able to process neighboring blocks in parallel.

FIG. 1is a block diagram illustrating an example video encoding and decoding system10that may utilize techniques of this disclosure. As shown inFIG. 1, system10includes a source device12that provides encoded video data to be decoded at a later time by a destination device14. In particular, source device12provides the video data to destination device14via a computer-readable medium16. Source device12and destination device14may comprise any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called “smart” phones, tablet computers, televisions, cameras, display devices, digital media players, video gaming consoles, video streaming device, or the like. In some cases, source device12and destination device14may be equipped for wireless communication. Thus, source device12and destination device14may be wireless communication devices. Source device12is an example video encoding device (i.e., a device for encoding video data). Destination device14is an example video decoding device (i.e., a device for decoding video data).

In the example ofFIG. 1, source device12includes a video source18, storage media19configured to store video data, a video encoder20, and an output interface22. Destination device14includes an input interface26, a storage media28configured to store encoded video data, a video decoder30, and display device32. In other examples, source device12and destination device14include other components or arrangements. For example, source device12may receive video data from an external video source, such as an external camera. Likewise, destination device14may interface with an external display device, rather than including an integrated display device.

The illustrated system10ofFIG. 1is merely one example. Techniques for processing video data may be performed by any digital video encoding and/or decoding device. Although generally the techniques of this disclosure are performed by a video encoding device, the techniques may also be performed by a video encoder/decoder, typically referred to as a “CODEC.” Source device12and destination device14are merely examples of such coding devices in which source device12generates coded video data for transmission to destination device14. In some examples, source device12and destination device14may operate in a substantially symmetrical manner such that each of source device12and destination device14include video encoding and decoding components. Hence, system10may support one-way or two-way video transmission between source device12and destination device14, e.g., for video streaming, video playback, video broadcasting, or video telephony.

Video source18of source device12may include a video capture device, such as a video camera, a video archive containing previously captured video, and/or a video feed interface to receive video data from a video content provider. As a further alternative, video source18may generate computer graphics-based data as the source video, or a combination of live video, archived video, and computer-generated video. Source device12may comprise one or more data storage media (e.g., storage media19) configured to store the video data. The techniques described in this disclosure may be applicable to video coding in general, and may be applied to wireless and/or wired applications. In each case, the captured, pre-captured, or computer-generated video may be encoded by video encoder20. Output interface22may output the encoded video information to a computer-readable medium16.

Output interface22may comprise various types of components or devices. For example, output interface22may comprise a wireless transmitter, a modem, a wired networking component (e.g., an Ethernet card), or another physical component. In examples where output interface22comprises a wireless receiver, output interface22may be configured to receive data, such as the bitstream, modulated according to a cellular communication standard, such as 4G, 4G-LTE, LTE Advanced, 5G, and the like. In some examples where output interface22comprises a wireless receiver, output interface22may be configured to receive data, such as the bitstream, modulated according to other wireless standards, such as an IEEE 802.11 specification, an IEEE 802.15 specification (e.g., ZigBee™), a Bluetooth™ standard, and the like. In some examples, circuitry of output interface22may be integrated into circuitry of video encoder20and/or other components of source device12. For example, video encoder20and output interface22may be parts of a system on a chip (SoC). The SoC may also include other components, such as a general purpose microprocessor, a graphics processing unit, and so on.

Input interface26of destination device14receives information from computer-readable medium16. The information of computer-readable medium16may include syntax information defined by video encoder20of video encoder20, which is also used by video decoder30, that includes syntax elements that describe characteristics and/or processing of blocks and other coded units, e.g., groups of pictures (GOPs). Input interface26may comprise various types of components or devices. For example, input interface26may comprise a wireless receiver, a modem, a wired networking component (e.g., an Ethernet card), or another physical component. In examples where input interface26comprises a wireless receiver, input interface26may be configured to receive data, such as the bitstream, modulated according to a cellular communication standard, such as 4G, 4G-LTE, LTE Advanced, 5G, and the like. In some examples where input interface26comprises a wireless receiver, input interface26may be configured to receive data, such as the bitstream, modulated according to other wireless standards, such as an IEEE 802.11 specification, an IEEE 802.15 specification (e.g., ZigBee™), a Bluetooth™ standard, and the like. In some examples, circuitry of input interface26may be integrated into circuitry of video decoder30and/or other components of destination device14. For example, video decoder30and input interface26may be parts of a SoC. The SoC may also include other components, such as a general purpose microprocessor, a graphics processing unit, and so on.

Storage media28may be configured to store encoded video data, such as encoded video data (e.g., a bitstream) received by input interface26. Display device32displays the decoded video data to a user, and may comprise any of a variety of display devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

In some examples, video encoder20and video decoder30may operate according to a video coding standard such as an existing or future standard. Example video coding standards include, but are not limited to, ITU-T H.261, ISO/IEC MPEG-1 Visual, ITU-T H.262 or ISO/IEC MPEG-2 Visual, ITU-T H.263, ISO/IEC MPEG-4 Visual and ITU-T H.264 (also known as ISO/IEC MPEG-4 AVC), including its Scalable Video Coding (SVC) and Multi-View Video Coding (MVC) extensions. In addition, a new video coding standard, namely High Efficiency Video Coding (HEVC) or ITU-T H.265, including its range and screen content coding extensions, 3D video coding (3D-HEVC) and multiview extensions (MV-HEVC) and scalable extension (SHVC), has been developed by the Joint Collaboration Team on Video Coding (JCT-VC) as well as Joint Collaboration Team on 3D Video Coding Extension Development (JCT-3V) of ITU-T Video Coding Experts Group (VCEG) and ISO/IEC Motion Picture Experts Group (MPEG). Ye-Kui Wang et al., “High Efficiency Video Coding (HEVC) Defect Report,” Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 14thMeeting, Vienna, AT, 25 Jul.-2 Aug. 2013, document JCTVC-N 1003_v1, is a draft HEVC specification.

ITU-T VCEG (Q6/16) and ISO/IEC MPEG (JTC 1/SC 29/WG 11) are now studying the potential need for standardization of future video coding technology with a compression capability that significantly exceeds that of the current HEVC standard (including its current extensions and near-term extensions for screen content coding and high-dynamic-range coding). The groups are working together on this exploration activity in a joint collaboration effort known as the Joint Video Exploration Team (JVET) to evaluate compression technology designs proposed by their experts in this area. The JVET first met during 19-21 Oct. 2015. Jianle Chen et al., “Algorithm Description of Joint Exploration Test Model 3,” Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 3rdMeeting, Geneva, CH, 26 May-1 Jun. 2016, document JVET-C1001, is an algorithm description of Joint Exploration Test Model 3 (JEM3).

In HEVC and other video coding specifications, video data includes a series of pictures. Pictures may also be referred to as “frames.” A picture may include one or more sample arrays. Each respective sample array of a picture may comprise an array of samples for a respective color component. In HEVC, a picture may include three sample arrays, denoted SL, SCb, and SCr. SLis a two-dimensional array (i.e., a block) of luma samples. SCbis a two-dimensional array of Cb chroma samples. SCris a two-dimensional array of Cr chroma samples. In other instances, a picture may be monochrome and may only include an array of luma samples.

As part of encoding video data, video encoder20may encode pictures of the video data. In other words, video encoder20may generate encoded representations of the pictures of the video data. An encoded representation of a picture may be referred to herein as a “coded picture” or an “encoded picture.”

To generate an encoded representation of a picture, video encoder20may encode blocks of the picture. Video encoder20may include, in a bitstream, an encoded representation of the video block. For example, to generate an encoded representation of a picture, video encoder20may partition each sample array of the picture into coding tree blocks (CTBs) and encode the CTBs. A CTB may be an N×N block of samples in a sample array of a picture. In the HEVC main profile, the size of a CTB can range from 16×16 to 64×64, although technically 8×8 CTB sizes can be supported.

A coding tree unit (CTU) of a picture may comprise one or more CTBs and may comprise syntax structures used to encode the samples of the one or more CTBs. For instance, each a CTU may comprise a CTB of luma samples, two corresponding CTBs of chroma samples, and syntax structures used to encode the samples of the CTBs. In monochrome pictures or pictures having three separate color planes, a CTU may comprise a single CTB and syntax structures used to encode the samples of the CTB. A CTU may also be referred to as a “tree block” or a “largest coding unit” (LCU). In this disclosure, a “syntax structure” may be defined as zero or more syntax elements present together in a bitstream in a specified order. In some codecs, an encoded picture is an encoded representation containing all CTUs of the picture.

To encode a CTU of a picture, video encoder20may partition the CTBs of the CTU into one or more coding blocks. A coding block is an N×N block of samples. In some codecs, to encode a CTU of a picture, video encoder20may recursively perform quad-tree partitioning on the coding tree blocks of a CTU to partition the CTBs into coding blocks, hence the name “coding tree units.” A coding unit (CU) may comprise one or more coding blocks and syntax structures used to encode samples of the one or more coding blocks. For example, a CU may comprise a coding block of luma samples and two corresponding coding blocks of chroma samples of a picture that has a luma sample array, a Cb sample array, and a Cr sample array, and syntax structures used to encode the samples of the coding blocks. In monochrome pictures or pictures having three separate color planes, a CU may comprise a single coding block and syntax structures used to code the samples of the coding block.

Furthermore, video encoder20may encode CUs of a picture of the video data. In some codecs, as part of encoding a CU, video encoder20may partition a coding block of the CU into one or more prediction blocks. A prediction block is a rectangular (i.e., square or non-square) block of samples on which the same prediction is applied. A prediction unit (PU) of a CU may comprise one or more prediction blocks of a CU and syntax structures used to predict the one or more prediction blocks. For example, a PU may comprise a prediction block of luma samples, two corresponding prediction blocks of chroma samples, and syntax structures used to predict the prediction blocks. In monochrome pictures or pictures having three separate color planes, a PU may comprise a single prediction block and syntax structures used to predict the prediction block.

Video encoder20may generate a predictive block (e.g., a luma, Cb, and Cr predictive block) for a prediction block (e.g., luma, Cb, and Cr prediction block) of a CU. Video encoder20may use intra prediction or inter prediction to generate a predictive block. If video encoder20uses intra prediction to generate a predictive block, video encoder20may generate the predictive block based on decoded samples of the picture that includes the CU. If video encoder20uses inter prediction to generate a predictive block of a CU of a current picture, video encoder20may generate the predictive block of the CU based on decoded samples of a reference picture (i.e., a picture other than the current picture).

Furthermore, video encoder20may decompose the residual blocks of a CU into one or more transform blocks. For instance, video encoder20may use quad-tree partitioning to decompose the residual blocks of a CU into one or more transform blocks. A transform block is a rectangular (e.g., square or non-square) block of samples on which the same transform is applied. A transform unit (TU) of a CU may comprise one or more transform blocks. For example, a TU may comprise a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax structures used to transform the transform block samples. Thus, each TU of a CU may have a luma transform block, a Cb transform block, and a Cr transform block. The luma transform block of the TU may be a sub-block of the CU's luma residual block. The Cb transform block may be a sub-block of the CU's Cb residual block. The Cr transform block may be a sub-block of the CU's Cr residual block. In monochrome pictures or pictures having three separate color planes, a TU may comprise a single transform block and syntax structures used to transform the samples of the transform block.

Video encoder20may apply one or more transforms a transform block of a TU to generate a coefficient block for the TU. A coefficient block may be a two-dimensional array of transform coefficients. A transform coefficient may be a scalar quantity. In some examples, the one or more transforms convert the transform block from a pixel domain to a frequency domain. Thus, in such examples, a transform coefficient may be a scalar quantity considered to be in a frequency domain. A transform coefficient level is an integer quantity representing a value associated with a particular 2-dimensional frequency index in a decoding process prior to scaling for computation of a transform coefficient value.

In some examples, video encoder20skips application of the transforms to the transform block. In such examples, video encoder20may treat residual sample values may be treated in the same way as transform coefficients. Thus, in examples where video encoder20skips application of the transforms, the following discussion of transform coefficients and coefficient blocks may be applicable to transform blocks of residual samples.

After generating a coefficient block, video encoder20may quantize the coefficient block. Quantization generally refers to a process in which transform coefficients are quantized to possibly reduce the amount of data used to represent the transform coefficients, providing further compression. In some examples, video encoder20skips quantization. After video encoder20quantizes a coefficient block, video encoder20may generate syntax elements indicating the quantized transform coefficients. Video encoder20may entropy encode one or more of the syntax elements indicating the quantized transform coefficients. For example, video encoder20may perform Context-Adaptive Binary Arithmetic Coding (CABAC) on the syntax elements indicating the quantized transform coefficients. Thus, an encoded block (e.g., an encoded CU) may include the entropy encoded syntax elements indicating the quantized transform coefficients.

Video encoder20may output a bitstream that includes encoded video data. In other words, video encoder20may output a bitstream that includes an encoded representation of video data. For example, the bitstream may comprise a sequence of bits that forms a representation of encoded pictures of the video data and associated data. In some examples, a representation of a coded picture may include encoded representations of blocks.

The bitstream may comprise a sequence of network abstraction layer (NAL) units. A NAL unit is a syntax structure containing an indication of the type of data in the NAL unit and bytes containing that data in the form of a raw byte sequence payload (RBSP) interspersed as necessary with emulation prevention bits. Each of the NAL units may include a NAL unit header and encapsulates a RBSP. The NAL unit header may include a syntax element indicating a NAL unit type code. The NAL unit type code specified by the NAL unit header of a NAL unit indicates the type of the NAL unit. A RBSP may be a syntax structure containing an integer number of bytes that is encapsulated within a NAL unit. In some instances, an RBSP includes zero bits.

Video decoder30may receive a bitstream generated by video encoder20. As noted above, the bitstream may comprise an encoded representation of video data. Video decoder30may decode the bitstream to reconstruct pictures of the video data. As part of decoding the bitstream, video decoder30may parse the bitstream to obtain syntax elements from the bitstream. Video decoder30may reconstruct pictures of the video data based at least in part on the syntax elements obtained from the bitstream. The process to reconstruct pictures of the video data may be generally reciprocal to the process performed by video encoder20to encode the pictures. For instance, video decoder30may use inter prediction or intra prediction to generate one or more predictive blocks for each PU of the current CU may use motion vectors of PUs to determine predictive blocks for the PUs of a current CU. In addition, video decoder30may inverse quantize coefficient blocks of TUs of the current CU. Video decoder30may perform inverse transforms on the coefficient blocks to reconstruct transform blocks of the TUs of the current CU. In some examples, video decoder30may reconstruct the coding blocks of the current CU by adding the samples of the predictive blocks for PUs of the current CU to corresponding decoded samples of the transform blocks of the TUs of the current CU. By reconstructing the coding blocks for each CU of a picture, video decoder30may reconstruct the picture.

A slice of a picture may include an integer number of CTUs of the picture. The CTUs of a slice may be ordered consecutively in a scan order, such as a raster scan order. In HEVC, a slice is defined as an integer number of CTUs contained in one independent slice segment and all subsequent dependent slice segments (if any) that precede the next independent slice segment (if any) within the same access unit. Furthermore, in HEVC, a slice segment is defined as an integer number of coding tree units ordered consecutively in the tile scan and contained in a single NAL unit. A tile scan is a specific sequential ordering of CTBs partitioning a picture in which the CTBs are ordered consecutively in CTB raster scan in a tile, whereas tiles in a picture are ordered consecutively in a raster scan of the tiles of the picture. As defined in HEVC and potentially other codecs, a tile is a rectangular region of CTBs within a particular tile column and a particular tile row in a picture. Other definitions of tiles may apply to types of blocks other than CTBs.

Video encoder20and/or video decoder30may perform various filtering operations on video data. For instance, as discussed in greater detail below, video decoder30may perform bilateral filtering on a sample of video data by replacing the sample with a weighted average of itself and its neighbors. However, performing bilateral filtering on samples of a current block may reduce the throughput of video decoder30because the reconstruction of samples of neighboring blocks of the current block may depend on unfiltered samples of the current block.

In accordance with one or more techniques of this disclosure, video encoder20and video decoder30may selectively filter samples of a current block such that the filtering does not prevent parallel processing of neighboring blocks. For instance, video decoder30may bilaterally filter on samples of a current block that may be utilized by neighboring blocks for intra prediction and refrain from bilaterally filtering samples of a current block that may be not utilized by neighboring blocks for intra prediction. In this way, video decoder20and video decoder30may still obtain some of the benefits of filtering while still being able to process neighboring blocks in parallel.

FIG. 2is a block diagram illustrating an example video encoder200that may perform the techniques of this disclosure. Video encoder200represents one example of video encoder20ofFIG. 1, though other examples are possible.FIG. 2is provided for purposes of explanation and should not be considered limiting of the techniques as broadly exemplified and described in this disclosure. For purposes of explanation, this disclosure describes video encoder200in the context of video coding standards such as the HEVC video coding standard and the H.266 video coding standard in development. However, the techniques of this disclosure are not limited to these video coding standards, and are applicable generally to video encoding and decoding.

In the example ofFIG. 2, video encoder200includes video data memory230, mode selection unit202, residual generation unit204, transform processing unit206, quantization unit208, inverse quantization unit210, inverse transform processing unit212, reconstruction unit214, filter unit216, decoded picture buffer (DPB)218, and entropy encoding unit220.

In this disclosure, reference to video data memory230should not be interpreted as being limited to memory internal to video encoder200, unless specifically described as such, or memory external to video encoder200, unless specifically described as such. Rather, reference to video data memory230should be understood as reference memory that stores video data that video encoder200receives for encoding (e.g., video data for a current block that is to be encoded). Video data memory230may also provide temporary storage of outputs from the various units of video encoder200.

Video encoder200may include arithmetic logic units (ALUs), elementary function units (EFUs), digital circuits, analog circuits, and/or programmable cores, formed from programmable circuits. In examples where the operations of video encoder200are performed by software executed by the programmable circuits, video data memory230may store the object code of the software that video encoder200receives and executes, or another memory (not shown) may store such instructions.

FIG. 3is a conceptual diagram illustrating a typical example of the Intra prediction for a 16×16 image block. As shown inFIG. 3, with Intra prediction, the 16×16 image block (in the heavy dashed square) may be predicted by the above and left neighboring reconstructed samples (reference samples) along a selected prediction direction (as indicated by the arrow).

In HEVC, for the Intra prediction of a luma block includes 35 modes, including the Planar mode, DC mode and 33 angular modes.FIGS. 4A and 4Bare conceptual diagrams illustrating example of the Intra prediction modes. In HEVC, after the intra prediction block has been generated for VER (vertical) and HOR (horizontal) intra modes, the left-most column and top-most row of the prediction samples may be further adjusted, respectively.

To capture finer edge directions presented in natural videos, the directional intra modes is extended from 33, as defined in HEVC, to 65. The new directional modes are depicted as dashed arrows inFIG. 4B, and the Planar and DC modes remain the same. These denser directional intra prediction modes apply for all block sizes and both luma and chroma intra predictions.

In addition, four-tap instead of two-tap intra interpolation filters may be utilized to generate the intra prediction block which improves the directional intra prediction accuracy. The boundary filter in HEVC may be further extended to several diagonal intra modes, and boundary samples up to four columns or rows are further adjusted using a two-tap (for intra mode 2 & 34) or a three-tap filter (for intra mode 3-6 & 30-33).

Position dependent intra prediction combination (PDPC) is a post-processing for Intra prediction which invokes a combination of HEVC Intra prediction with un-filtered boundary reference samples. In adaptive reference sample smoothing (ARSS), two low pass filters (LPF) are used to process reference samples:3-tap LPF with the coefficients of [1, 2, 1]/45-tap LPF with the coefficients of [2, 3, 6, 3, 2]/16

CCLM is a new chroma prediction method wherein the reconstructed luma blocks and the neighboring chroma block are utilized to derive the chroma prediction block. Additional information about PDPC, ARSS, and CCLM may be found in JVET-D1001, 4th Meeting: Chengdu, CN, 15-21 Oct. 2016 (hereinafter, “JVET-D1001”).

Filter unit216may perform one or more filter operations on reconstructed blocks. For example, filter unit216may perform deblocking operations to reduce blockiness artifacts along edges of CUs. As illustrated by dashed lines, operations of filter unit216may be skipped in some examples.

Video encoder200stores reconstructed blocks in DPB218. For instance, in examples where operations of filter unit216are not needed, reconstruction unit214may store reconstructed blocks to DPB218. In examples where operations of filter unit216are needed, filter unit216may store the filtered reconstructed blocks to DPB218. Motion estimation unit222and motion compensation unit224may retrieve a reference picture from DPB218, formed from the reconstructed (and potentially filtered) blocks, to inter-predict blocks of subsequently encoded pictures. In addition, intra-prediction unit226may use reconstructed blocks in DPB218of a current picture to intra-predict other blocks in the current picture.

As discussed above, filter unit216may perform one or more filter operations on reconstructed blocks. In some examples, such as in HEVC, filter unit216may employ two in-loop filters, including a de-blocking filter (DBF) and a Sample adaptive offset (SAO) filter.

Input to the de-blocking filter coding tool is a reconstructed image after prediction (e.g., intra or inter prediction, but other prediction modes are possible). The deblocking filter performs detection of the artifacts at the coded block boundaries and attenuates the artifacts by applying a selected filter. As described in Norkin et al., “HEVC Deblocking Filter”, IEEE Trans. Circuits Syst. Video Technol., 22(12): 1746-1754 (2012), compared to the H.264/AVC deblocking filter, the HEVC deblocking filter has lower computational complexity and better parallel processing capabilities while still achieving significant reduction of the visual artifacts.

Input to the SAO filter is a reconstructed image after invoking deblocking filtering. The concept of SAO is to reduce mean sample distortion of a region by first classifying the region samples into multiple categories with a selected classifier, obtaining an offset for each category, and then adding the offset to each sample of the category, where the classifier index and the offsets of the region are coded in the bitstream. In HEVC, the region (the unit for SAO parameters signaling) is defined to be a coding tree unit (CTU). Two SAO types that can satisfy the requirements of low complexity are adopted in HEVC: edge offset (EO) and band offset (BO). An index of SAO type is coded (which is in the range of [0, 2]).

According to the selected EO pattern, five categories denoted by edgeIdx in Table 1 are further defined. For edgeIdx equal to 0˜3, the magnitude of an offset may be signaled while the sign flag is implicitly coded, i.e., negative offset for edgeIdx equal to 0 or 1 and positive offset for edgeIdx equal to 2 or 3. For edgeIdx equal to 4, the offset is always set to 0 which means no operation is required for this case.

For BO, the sample classification is based on sample values. Each color component may have its own SAO parameters. BO implies one offset is added to all samples of the same band. The sample value range is equally divided into 32 bands. For 8-bit samples ranging from 0 to 255, the width of a band is 8, and sample values from 8 k to 8 k+7 belong to band k, where k ranges from 0 to 31. The average difference between the original samples and reconstructed samples in a band (i.e., offset of a band) is signaled to the decoder. There may be no constraint on offset signs. Offsets of four consecutive bands (and in some examples, only offsets of four consecutive bands) and the starting band position may be signaled to the decoder.

To reduce side information, multiple CTUs can be merged together (either copying the parameters from above CTU (through setting sao_merge_left_flag equal to 1) or left CTU (through setting sao_merge_up_flag equal to 1) to share SAO parameters.

In addition to the modified DB and HEVC SAO methods. JEM has included another filtering method, called Geometry transformation-based Adaptive Loop Filtering (GALF). GALF aims improve the coding efficiency of ALF studied in HEVC stage by introducing several new aspects. ALF is aiming to minimize the mean square error between original samples and decoded samples by using Wiener-based adaptive filter. Samples in a picture are classified into multiple categories and the samples in each category are then filtered with their associated adaptive filter. The filter coefficients may be signaled or inherited to optimize the tradeoff between the mean square error and the overhead. A GALF scheme may further improve the performance of ALF, which introduces geometric transformations, such as rotation, diagonal and vertical flip, to be applied to the samples in filter support region depending on the orientation of the gradient of the reconstructed samples before ALF. Input to ALF/GALF is the reconstructed image after invoking SAO.

GALF was proposed in Karczewicz et al., “EE2.5: Improvements on adaptive loop filter”, Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Doc. JVET-B0060, 2ndMeeting: San Diego, USA, 20 Feb.-26 Feb. 2016, and Karczewicz et al., “EE2.5: Improvements on adaptive loop filter”, Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11. Doc. JVET-C0038, 3rdMeeting: Geneva, CH, 26 May-1 Jun. 2016, the Geometric transformations-based ALF (GALF). GALF has been adopted to the most recent version of JEM, i.e., JEM3.0. In GALF, the classification is modified with the diagonal gradients taken into consideration and geometric transformations could be applied to filter coefficients. Each 2×2 block is categorized into one out of 25 classes based on its directionality and quantized value of activity. The details are described in the following sub-sections.

As described in C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images”, in Proc. of IEEE ICCV, Bombay, India, January 1998, Bilateral filtering was may help avoid undesirable over-smoothing for pixels in the edge. The main idea of bilateral filtering is the weighting of neighboring samples takes into account the pixel values themselves to weight more those pixels with similar luminance or chrominance values. A sample located at (i, j) is filtered using its neighboring sample (k, l). The weight ω(i, j, k, l) is the weight assigned for sample (k, l) to filter the sample (i, j), and it is defined as:

ω⁡(i,j,k,l)=e(-(i-k)2+(j-l)22⁢⁢σd2-I⁡(i,j)-I⁡(k,l)22⁢⁢σr2)(1)
In equation (1) above, I(i, j) and I(k, l) are the intensity values of samples (i, j) and (k, l) respectively, ad is the spatial parameter, and σris the range parameter. Definitions of the spatial parameter and range parameter are provided below. The filtering process with the filtered sample value denoted by ID(i, j) may be defined in accordance with equation (2) below.

The properties (or strength) of the bilateral filter are controlled by these two parameters. Samples located closer to the sample to be filtered, and samples having smaller intensity difference to the sample to be filtered, will have larger weight than samples further away and with larger intensity difference.

As described in Jacob Ström et al., “Bilateral filter after inverse transform”, JVET-D0069, 4th Meeting: Chengdu, CN, 15-21 Oct. 2016 (hereinafter, “JVET-D0069”), each reconstructed sample in the transform unit (TU) is filtered using its direct neighboring reconstructed samples only. The filter has a plus sign shaped filter aperture centered at the sample to be filtered, as depicted inFIG. 6.FIG. 6is a conceptual diagram illustrating current block600that includes current sample602and neighboring samples604-610utilized in bilateral filtering process. The spatial parameter (i.e., σd) may be set based on the transform unit size and the range parameter (i.e., σr) may be set based on the QP used for current block400. Equations (3) and (4) provide one example of how the spatial and range parameters may be determined.

As described in Jacob Ström et al., “Bilateral filter strength based on prediction mode”, JVET-E0032, 5th Meeting: Geneva, CH, 12-20 Jan. 2017 (hereinafter, “JVET-E0032”), to further reduce the coding loss under low delay configuration, the filter strength is further designed to be dependent on the coded mode. For intra-coded blocks, the above equation (3) is still used. While for inter-coded blocks, the following equation (5) is applied.

It is noted that the proposed bilateral filtering method may only applied to luma blocks with at least one non-zero coefficients. For chroma blocks and luma blocks with all zero coefficients, the bilateral filtering method may always be disabled.

For samples located at a TU top and left boundaries (i.e., top row and left column), only neighboring samples within current TU are used to filter current sample.FIG. 7is a conceptual diagram illustrating how neighboring samples within a current TU (e.g., a 4×4 TU) may be used to filter a current sample.FIG. 7illustrates current TU700as including current sample700and neighboring samples704-710. As shown inFIG. 7, left neighboring sample710of current sample702is not included in current TU700. As such, left neighboring sample710may not be used in the filtering process of current sample702.

Filter unit216may apply a bilateral filter in accordance with the techniques of this disclosure. For instance, filter unit216may apply a bilateral filter on reconstructed samples of a current block generated by reconstruction unit214in accordance with equation (2), above. After applying the bilateral filter to the reconstructed samples of the current block, filter unit216may store a filtered version of the current block in decoded picture buffer218. The filtered version of the current block may be used as a reference picture in encoding another picture of the video data, as described elsewhere in this disclosure.

The design of bilateral filtering in JVET-D0069 and JVET-E0032 may have the following potential issues. In particular, the bilateral filter is applied right after the reconstruction of one block. Therefore, video encoder20may have to wait until the filtering process of a current block is finished for the next neighboring block to be coded. Such a design may decrease the pipeline throughput, which may be undesirable.

The techniques of this disclosure may address the potential issue mentioned above. Some of the proposed techniques may be combined together. The proposed techniques may be applied to other in-loop filtering methods which depend on certain known information to implicitly derive adaptive filter parameters, or filters with explicit signaling of parameters.

In accordance with one or more techniques of this disclosure, filter unit216may selectively filter samples of a current block such that the filtering does not prevent parallel processing of neighboring blocks. For instance, filter unit216may categorize samples of the current block as either “to be filtered” or “not to be filtered” and only perform the bilateral filtering on samples categorized as to be filtered (i.e., filter unit216may refrain from bilaterally filtering samples categorized as not to be filtered). In this way, filter unit216may still obtain some of the benefits of filtering while still being able to process neighboring blocks in parallel.

Filter unit216may categorize samples of the current block as either to be filtered or not to be filtered in a variety of ways. As one example, filter unit216may perform the categorization based on whether the samples may be used for predicting samples of neighboring blocks. As another example, filter unit216may perform the categorization based on whether the samples are located in a pre-defined region of the current block. As another example, filter unit216may perform the categorization based on whether the samples are actually used for predicting neighboring blocks.

FIG. 8is a conceptual diagram illustrating one example of how samples may be categorized, in accordance with one or more techniques of the current disclosure. As shown inFIG. 8, picture800includes current block810, bottom neighboring block820, and right neighboring block830.

As discussed above, filter unit216may categorize the samples of the current block based on whether the samples may be used to predict samples in neighboring blocks (e.g., in intra prediction or LM mode). For instance, filter unit216may categorize, as not to be filtered, all samples of the current block that may possibly used by one or more neighboring blocks for intra prediction without evaluating whether the samples are/will actually used for intra prediction. To illustrate, if a first sample of the current block may be used by neighboring blocks for intra prediction, filter unit216may categorize the first sample as not to be filtered and refrain from performing bilateral filtering on the first sample. On the other hand, if a second sample of the current block may not be used by neighboring blocks for intra prediction, filter unit216may categorize the second sample as to be filtered and perform bilateral filtering on the second sample. In some examples, filter unit216may determine that samples located in a right most column or a bottom row of the current block (assuming horizontal raster scan order, it is understood that the right most column and bottom row are interpreted as the “leading” column/row and that other columns/rows may be used with other scan orders) may be utilized by neighboring blocks for intra prediction. For instance, in the example ofFIG. 8, filter unit216may categorize samples in right most column812and samples in bottom row814as do not filter because it is possible for neighboring blocks820and830to use samples in right most column812and samples in bottom row814for intra prediction.

As discussed above, filter unit216may categorize the samples of the current block based on whether the samples are located in a pre-defined region of the current block. This technique may be similar to, and overlap with in certain circumstances, the categorization based on whether the samples may be used by neighboring blocks for intra prediction. For instance, the pre-defined region of the current block may include the right most column and bottom row of the current block.

As discussed above, filter unit216may perform the categorization based on whether the samples are actually used for prediction of neighboring blocks. To determine which samples of the current block are utilized by neighboring blocks, filter unit216may determine, based on information received from mode selection unit202, whether the neighboring blocks of the current block are coded with intra mode. Responsive to determining that a right neighboring block (e.g., block830) of the current block is coded using intra prediction, filter unit216may determine that samples of the current block that are located in a right most column (e.g., samples in column812) of the current block are utilized by neighboring blocks for intra prediction. However, responsive to determining that the right neighboring block (e.g., block830) of the current block is not coded using intra prediction (e.g., is coded using inter prediction), filter unit216may determine that samples of the current block that are located in the right most column (e.g., samples in column812) of the current block are not utilized by neighboring blocks for intra prediction. Similarly, responsive to determining that a bottom neighboring block (e.g., block820) of the current block is coded using intra prediction, filter unit216may determine that samples of the current block that are located in a bottom row (e.g., samples in row814) of the current block are utilized by neighboring blocks for intra prediction. However, responsive to determining that the bottom neighboring block (e.g., block820) of the current block is not coded using intra prediction, filter unit216may determine that samples of the current block that are located in the bottom row (e.g., samples in row814) of the current block are not utilized by neighboring blocks for intra prediction.

In some examples, as discussed above, video encoder20may use a cross-component linear model (CCLM) prediction mode to predict samples of video data. In CCLM, video encoder20may utilize the luma samples of the whole block when performing the chroma intra prediction process of a chroma block. As such, where a neighboring block of the current block relied on luma reconstruction samples (e.g., if the neighboring block is coded using CCLM), filter unit216may determine that all of the samples of the current block are actually used for prediction of neighboring blocks. In such examples, filter unit216may refrain from performing bilateral filtering on any samples of the current block.

When categorizing samples based on whether the samples may be used for predicting samples of neighboring blocks or based on whether the samples are located in a pre-defined region of the current block, filter unit216may avoid having to actually determine which, if any, samples of the current block are actually used for predicting neighboring blocks. By not determining which samples of the current block are actually used for predicting neighboring blocks, filtering unit216may reduce the complexity of the filtering process. However, by determining which samples of the current block are actually used for predicting neighboring blocks and only refraining from filtering the samples that are actually used, filtering unit216may filter a larger number of samples, which may improve quality/artifact reduction.

In some examples, as opposed to selectively filtering some samples of a current block, filter unit216may perform bilateral filtering on all samples of the current block and store two sets of reconstruction blocks/sub-blocks. For instance, filter unit216may store a first set that includes non-bilaterally filtered samples of the current block and a second set that includes bilaterally filtered samples of the current block. In some examples, the second set may include samples that are bilaterally filtered but not yet filtered by other in-loop filters, such as deblocking filter.

In some examples, intra prediction unit226may always use the first set for performing an intra luma prediction process. In some examples, intra prediction unit226may select the first set or the second set for performing luma intra prediction of neighboring blocks based on the intra prediction mode information. For instance, if a neighboring block of current block is coded with the PDPC or ARSS mode or boundary filter is enabled, intra prediction unit226may select the first set for the luma intra prediction process of the neighboring block. In some examples, if the chroma mode relies on luma reconstruction samples, e.g., cross-component linear model (CCLM) prediction mode, intra prediction unit226may utilize the first set of the corresponding luma block when performing the chroma intra prediction process of a chroma block.

Similarly, the filtering process to the reconstruction of a block/sub-block may be applied after all the intra prediction to the next-coded block is done. Here, the intra prediction may include but not limited to, 1) traditional normal intra prediction using casual reconstructed samples, 2) cross-component linear model (CCLM) prediction.

Video encoder200represents an example of a device configured to encode video data, the device including a memory configured to store the video data (e.g., decoded picture buffer218) and one or more processors configured to obtain reconstructed samples of a current block of the video data; and selectively bilaterally filter the reconstructed samples of the current block to generate a filtered current block, wherein selectively bilaterally filtering the reconstructed samples of the current block comprises refraining from bilaterally filtering at least one reconstructed sample of the current block such that the filtered current block includes at least one non-bilaterally filtered sample.

FIG. 9is a block diagram illustrating an example video decoder300that may perform the techniques of this disclosure. Video decoder300represents one example of video decoder30ofFIG. 1, though other examples are possible.FIG. 9is provided for purposes of explanation and is not limiting on the techniques as broadly exemplified and described in this disclosure. For purposes of explanation, this disclosure describes video decoder300is described according to the techniques of JEM and HEVC. However, the techniques of this disclosure may be performed by video coding devices that are configured to other video coding standards.

In the example ofFIG. 9, video decoder300includes coded picture buffer (CPB) memory320, entropy decoding unit302, prediction processing unit304, inverse quantization unit306, inverse transform processing unit308, reconstruction unit310, filter unit312, and decoded picture buffer (DPB)314. Prediction processing unit304includes motion compensation unit316and intra-prediction unit318. Prediction processing unit304may include addition units to perform prediction in accordance with other prediction modes. As examples, prediction processing unit304may include a palette unit, an intra-block copy unit (which may form part of motion compensation unit316), an affine unit, a linear model (LM) unit, or the like. In other examples, video decoder300may include more, fewer, or different functional components.

Filter unit312may perform one or more filter operations on reconstructed blocks. For example, filter unit312may perform deblocking operations to reduce blockiness artifacts along edges of the reconstructed blocks. As illustrated by dashed lines, operations of filter unit312are not necessarily performed in all examples.

Filter unit312may generally perform a filtering process in a matter that is substantially similar to that described with respect to filter unit216(FIG. 1). For instance, filter unit312may selectively filter samples of a current block such that the filtering does not prevent parallel processing of neighboring blocks. For instance, filter unit312may categorize samples of the current block as either “to be filtered” or “not to be filtered” and only perform the bilateral filtering on samples categorized as to be filtered (i.e., filter unit312may refrain from bilaterally filtering samples categorized as not to be filtered). In this way, filter unit312may still obtain some of the benefits of filtering while still being able to process neighboring blocks in parallel.

Video decoder300may store the reconstructed blocks in DPB314. For instance, filter unit312may store filtered reconstructed blocks in DPB314. As discussed above, DPB314may provide reference information, such as samples of a current picture for intra-prediction and previously decoded pictures for subsequent motion compensation, to prediction processing unit304. Moreover, video decoder300may output decoded pictures from DPB for subsequent presentation on a display device, such as display device32ofFIG. 1.

FIG. 10is a flowchart illustrating an example process for filtering a reconstructed block of video data, in accordance with one or more techniques of this disclosure. For purposes of explanation, the method ofFIG. 10is described below as being performed by video decoder30/300and the components thereof (e.g., illustrated inFIGS. 1 and 9), though the method ofFIG. 10may be performed by other video decoders or video encoders. For instance, the method ofFIG. 10may be performed by video encoder20/200(e.g., illustrated inFIGS. 1 and 2).

Video decoder30may reconstruct samples of a current block of video data (1002). For instance, reconstruction unit310may add samples of a residual block (generated by inverse transform processing unit308) to corresponding samples of a prediction block (generated by prediction processing unit304) to reconstruct the samples of the current block.

Video decoder30may categorize samples of the current block as to be filtered or not to be filtered (1004). As discussed above, filter unit216may categorize samples of the current block as either to be filtered or not to be filtered in a variety of ways. As one example, filter unit216may perform the categorization based on whether the samples may be used for predicting samples of neighboring blocks. As another example, filter unit216may perform the categorization based on whether the samples are located in a pre-defined region of the current block. As another example, filter unit216may perform the categorization based on whether the samples are actually used for predicting neighboring blocks. In some examples, categorizing a sample may be interpreted as determining whether to filter. For instance, filter unit216may categorize a particular sample by determining whether or not to filter the particular sample and need not assign a value to some attribute or variable for the particular sample.

Video decoder30may filter samples of the current block that are categorized as to be filtered (1006). For instance, filter unit216may perform a bilateral filtering process on each sample categorized as to be filtered in accordance with equation (2) above. In particular, filter unit216may by replacing each sample categorized as to be filtered with a weighted average of itself and its neighbors.

Video decoder30may store the filtered samples of the current block (1008). For instance, filter unit216may store a filtered current block (that includes the filtered samples of the current block along with unfiltered samples categorized as not to be filtered) in decoded picture buffer314. Moreover, video decoder30may output decoded pictures from DPB for subsequent presentation on a display device, such as display device32ofFIG. 1.

Certain aspects of this disclosure have been described with respect to the video coding standards for purposes of illustration. However, the techniques described in this disclosure may be useful for other video coding processes, including other standard or proprietary video coding processes not yet developed.

The techniques described above may be performed by video encoder200and/or video decoder120, both of which may be generally referred to as a video coder. Likewise, video coding may refer to video encoding or video decoding, as applicable.

It should be understood that all of the techniques described herein may be used individually or in combination. This disclosure includes several signaling methods which may change depending on certain factors such as block size, palette size, slice type etc. Such variation in signaling or inferring the syntax elements may be known to the encoder and decoder a-priori or may be signaled explicitly in the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), slice header, at a tile level or elsewhere.

It is to be recognized that depending on the example, certain acts or events of any of the techniques described herein can be performed in a different sequence, may be added, merged, or left out altogether (e.g., not all described acts or events are necessary for the practice of the techniques). Moreover, in certain examples, acts or events may be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors, rather than sequentially. In addition, while certain aspects of this disclosure are described as being performed by a single module or unit for purposes of clarity, it should be understood that the techniques of this disclosure may be performed by a combination of units or modules associated with a video coder.

While particular combinations of various aspects of the techniques are described above, these combinations are provided merely to illustrate examples of the techniques described in this disclosure. Accordingly, the techniques of this disclosure should not be limited to these example combinations and may encompass any conceivable combination of the various aspects of the techniques described in this disclosure.