Source: http://www.patentsencyclopedia.com/app/20130177068
Timestamp: 2017-12-12 21:05:09
Document Index: 134592031

Matched Legal Cases: ['application No. 61', 'application No. 61', 'application No. 61', 'application No. 61', 'application No. 61', 'application No. 61', 'application No. 61']

DEVICES AND METHODS FOR SAMPLE ADAPTIVE OFFSET CODING - Patent application
Patent application title: DEVICES AND METHODS FOR SAMPLE ADAPTIVE OFFSET CODING
Patent application number: 20130177068
In one embodiment, a method for decoding a video bitstream having a plurality of pictures, the bitstream generated by a video coding system with sample adaptive offset (SAO), is provided, the method comprising the steps of: obtaining processed video data from a video bitstream; partitioning the processed video data into blocks, wherein each of the blocks is equal to or smaller than a picture and each block is comprised of a plurality of pixels; applying a first SAO compensation to each of the pixels in a processed video block; and applying a second SAO compensation to each of the pixels in the processed video block.
1. A method for decoding a video bitstream having a plurality of pictures, the bitstream generated by a video coding system with sample adaptive offset (SAO), the method comprising the steps of: obtaining processed video data from a video bitstream; partitioning the processed video data into blocks, wherein each of the blocks is equal to or smaller than a picture and each block is comprised of a plurality of pixels; applying a first SAO compensation to each of the pixels in a processed video block; and applying a second SAO compensation to each of the pixels in the processed video block.
2. The method of claim 1, wherein the first SAO compensation and second SAO compensation are predetermined.
3. The method of claim 1, further comprising: deriving an SAO type from the video bitstream for each of the blocks.
4. The method of claim 3, wherein if the derived SAO type is edge offset, the first SAO compensation is based on a horizontal edge offset (EO) type and the second SAO compensation is based on a vertical EO type.
5. The method of claim 3, wherein if the derived SAO type is edge offset, the first SAO compensation and second SAO compensation are orthogonal to each other.
6. The method of claim 3, further comprising: determining an SAO sub-class associated with the SAO type for each of the pixels in each of the blocks; and deriving intensity offset from the video bitstream for the sub-class associated with the SAO type; wherein at least one of the first or second SAO compensations is based on the derived intensity offset information.
7. The method of claim 1, wherein the first and second SAO compensations are applied to pixels that have been deblocked.
8. The method of claim 1, wherein at least one of the first and second SAO compensations are applied to pixels that have been deblocked and have previously had SAO compensation applied.
9. The method of claim 8, wherein the pixels that have been deblocked and have previously had SAO applied are used in determining an SAO sub-class associated with the SAO type for each of the pixels in each of the blocks and deriving intensity offset from the video bitstream for the sub-class associated with the SAO type.
10. The method of claim 1, further comprising: applying a third SAO compensation to each of the pixels in the processed video block.
11. The method of claim 10, wherein two of the compensations relate to edge offset (EO) types and wherein one of the compensations relate to a band offset (BO) type.
12. The method of claim 1, wherein each color component in a processed video block has its own SAO process for the first and second compensations.
13. The method of claim 1, wherein the method is implemented on a computer having a processor and a memory coupled to said processor, wherein at least some of steps are performed using said processor.
14. An apparatus for decoding a video bitstream having a plurality of pictures, the bitstream generated by a video coding system with sample adaptive offset (SAO), the apparatus comprising a video decoder configured to: obtain processed video data from a video bitstream; partition the processed video data into blocks, wherein each of the blocks is equal to or smaller than a picture and each block is comprised of a plurality of pixels; apply a first SAO compensation to each of the pixels in a processed video block; and apply a second SAO compensation to each of the pixels in the processed video block.
15. The apparatus of claim 14, wherein the apparatus comprises at least one of: an integrated circuit; a microprocessor; and a wireless communication device that includes the video decoder.
16. The apparatus of claim 14, wherein the video decoder is further configured to: derive an SAO type from the video bitstream for each of the blocks.
17. The apparatus of claim 16, wherein if the derived SAO type is edge offset, the first SAO compensation and second SAO compensation are orthogonal to each other.
18. The apparatus of claim 14, wherein the first and second SAO compensations are applied to pixels that have been deblocked.
19. The apparatus of claim 14, wherein at least one of the first and second SAO compensations are applied to pixels that have been deblocked and have previously had SAO compensation applied.
20. The apparatus of claim 14, wherein the video decoder is further configured to: apply a third SAO compensation to each of the pixels in the processed video block.
21. A method of encoding video data having a plurality of pictures, using sample adaptive offset (SAO), the method comprising the steps of: partitioning video data into blocks, wherein each of the blocks is equal to or smaller than a picture and each block is comprised of a plurality of pixels; applying a first SAO compensation to each of the pixels in a processed video block; and applying a second SAO compensation to each of the pixels in the processed video block.
22. The method of claim 21, further comprising: selecting an SAO type for the video data for each of the blocks; selecting an SAO sub-class associated with the SAO type for each of the pixels in each of the blocks; and deriving intensity offset for the video data for the sub-class associated with the SAO type.
23. The method of claim 22, wherein at least one of the first or second SAO compensations is based on the derived intensity offset information.
24. The method of claim 21, wherein the first and second SAO compensations are applied to pixels that have been deblocked.
25. The method of claim 21, wherein at least one of the first and second SAO compensations is applied to pixels that have been deblocked and have previously had SAO compensation applied.
26. The method of claim 21, further comprising: applying a third SAO compensation to each of the pixels in the processed video block.
27. The method of claim 21, wherein the method is implemented on a computer having a processor and a memory coupled to said processor, wherein at least some of steps are performed using said processor.
28. An apparatus for encoding a video data having a plurality of pictures, using sample adaptive offset (SAO), the apparatus comprising a video encoder configured to: partition video data into blocks, wherein each of the blocks is equal to or smaller than a picture and each block is comprised of a plurality of pixels; apply a first SAO compensation to each of the pixels in a processed video block; and apply a second SAO compensation to each of the pixels in the processed video block.
[0001] The present application claims the benefit of U.S. provisional patent application No. 61/583,555, entitled "Coding and Selection of SAO Parameters" filed Jan. 5, 2012, U.S. provisional patent application No. 61/589,297, entitled "Modified SAO Edge Class and Offsets" filed Jan. 21, 2012, U.S. provisional patent application No. 61/589,298, entitled "Modified SAO Band Offset Type and Classes" filed Jan. 21, 2012, U.S. provisional patent application No. 61/597,041, entitled "Modifications to SAO Band Offset and Edge Offset" filed Feb. 9, 2012, U.S. provisional patent application No. 61/616,373, entitled "Modifications to SAO Band Offset" filed Mar. 27, 2012, U.S. provisional patent application No. 61/619,916, entitled "Modifications to SAO Band Offset" filed Apr. 3, 2012, and U.S. provisional patent application No. 61/639,998, entitled "Multipass SAO" filed Apr. 30, 2012, which are incorporated herein by reference in their entirety.
[0002] The disclosure relates generally to the field of video coding, and more specifically to systems, devices and methods for sample adaptive offset (SAO) coding.
[0013] FIG. 7 illustrates an example of a segmented line formed by three pixel values or points L, C, and R according to embodiments of the disclosure;
[0014] FIG. 8 illustrates an example of a segmented line formed by three pixel values or points L, C, and R according to embodiments of the disclosure;
[0015] FIG. 9 illustrates an example band offset specification according to embodiments of the disclosure;
[0016] FIG. 10 illustrates an example band offset specification having a distribution of values according to embodiments of the disclosure;
[0017] FIG. 11 illustrates an example band offset specification according to embodiments of the disclosure;
[0018] FIG. 12 illustrates an example architecture for coding of offsets according to embodiments of the disclosure;
[0019] FIG. 13 illustrates an example band offset specification according to embodiments of the disclosure; and
[0020] FIG. 14 illustrates an example block diagram showing a generic multipass SAO process according to embodiments of the disclosure.
[0021] Accordingly, there is provided herein systems and methods that improve video quality by selection, coding, and signaling of parameters in a sample adaptive offset (SAO) process. The methods and systems described herein generally pertain to video processing such as video encoders and decoders.
[0022] In a first aspect, a method for decoding a video bitstream having a plurality of pictures, the bitstream generated by a video coding system with sample adaptive offset (SAO), is provided, the method comprising the steps of: obtaining processed video data from a video bitstream; partitioning the processed video data into blocks, wherein each of the blocks is equal to or smaller than a picture and each block is comprised of a plurality of pixels; applying a first SAO compensation to each of the pixels in a processed video block; and applying a second SAO compensation to each of the pixels in the processed video block. In an embodiment of the first aspect, the first SAO compensation and second SAO compensation are predetermined. In an embodiment of the first aspect, the method further comprises: deriving an SAO type from the video bitstream for each of the blocks. In an embodiment of the first aspect, if the derived SAO type is edge offset, the first SAO compensation is based on a horizontal edge offset (EO) type and the second SAO compensation is based on a vertical EO type. In an embodiment of the first aspect, if the derived SAO type is edge offset, the first SAO compensation and second SAO compensation are orthogonal to each other. In an embodiment of the first aspect, the method further comprises: determining an SAO sub-class associated with the SAO type for each of the pixels in each of the blocks; and deriving intensity offset from the video bitstream for the sub-class associated with the SAO type; wherein at least one of the first or second SAO compensations is based on the derived intensity offset information. In an embodiment of the first aspect, the first and second SAO compensations are applied to pixels that have been deblocked. In an embodiment of the first aspect, at least one of the first and second SAO compensations are applied to pixels that have been deblocked and have previously had SAO compensation applied. In an embodiment of the first aspect, the pixels that have been deblocked and have previously had SAO applied are used in determining an SAO sub-class associated with the SAO type for each of the pixels in each of the blocks and deriving intensity offset from the video bitstream for the sub-class associated with the SAO type. In an embodiment of the first aspect, the method further comprises: applying a third SAO compensation to each of the pixels in the processed video block. In an embodiment of the first aspect, two of the compensations relate to edge offset (EO) types and wherein one of the compensations relate to a band offset (BO) type. In an embodiment of the first aspect, each color component in a processed video block has its own SAO process for the first and second compensations. In an embodiment of the first aspect, the method is implemented on a computer having a processor and a memory coupled to said processor, wherein at least some of steps are performed using said processor.
[0023] In a second aspect, an apparatus for decoding a video bitstream having a plurality of pictures, the bitstream generated by a video coding system with sample adaptive offset (SAO), is provided, the apparatus comprising a video decoder configured to: obtain processed video data from a video bitstream; partition the processed video data into blocks, wherein each of the blocks is equal to or smaller than a picture and each block is comprised of a plurality of pixels; apply a first SAO compensation to each of the pixels in a processed video block; and apply a second SAO compensation to each of the pixels in the processed video block. In an embodiment of the second aspect, the apparatus comprises at least one of: an integrated circuit; a microprocessor; and a wireless communication device that includes the video decoder. In an embodiment of the second aspect, the video decoder is further configured to: derive an SAO type from the video bitstream for each of the blocks. In an embodiment of the second aspect, if the derived SAO type is edge offset, the first SAO compensation and second SAO compensation are orthogonal to each other. In an embodiment of the second aspect, the first and second SAO compensations are applied to pixels that have been deblocked. In an embodiment of the second aspect, at least one of the first and second SAO compensations are applied to pixels that have been deblocked and have previously had SAO compensation applied. In an embodiment of the second aspect, the video decoder is further configured to: apply a third SAO compensation to each of the pixels in the processed video block.
[0024] In a third aspect, a method of encoding video data having a plurality of pictures, using sample adaptive offset (SAO), is provided, the method comprising the steps of: partitioning video data into blocks, wherein each of the blocks is equal to or smaller than a picture and each block is comprised of a plurality of pixels; applying a first SAO compensation to each of the pixels in a processed video block; and applying a second SAO compensation to each of the pixels in the processed video block. In an embodiment of the third aspect, the method further comprises: selecting an SAO type for the video data for each of the blocks; selecting an SAO sub-class associated with the SAO type for each of the pixels in each of the blocks; and deriving intensity offset for the video data for the sub-class associated with the SAO type. In an embodiment of the third aspect, at least one of the first or second SAO compensations is based on the derived intensity offset information. In an embodiment of the third aspect, the first and second SAO compensations are applied to pixels that have been deblocked. In an embodiment of the third aspect, at least one of the first and second SAO compensations is applied to pixels that have been deblocked and have previously had SAO compensation applied. In an embodiment of the third aspect, the method further comprises: applying a third SAO compensation to each of the pixels in the processed video block. In an embodiment of the third aspect, the method is implemented on a computer having a processor and a memory coupled to said processor, wherein at least some of steps are performed using said processor.
[0025] In a fourth aspect, an apparatus for encoding a video data having a plurality of pictures, using sample adaptive offset (SAO), is provided, the apparatus comprising a video encoder configured to: partition video data into blocks, wherein each of the blocks is equal to or smaller than a picture and each block is comprised of a plurality of pixels; apply a first SAO compensation to each of the pixels in a processed video block; and apply a second SAO compensation to each of the pixels in the processed video block.
[0026] In this disclosure, the term "coding" refers to encoding that occurs at the encoder or decoding that occurs at the decoder. Similarly, the term coder refers to an encoder, a decoder, or a combined encoder/decoder (CODEC). The terms coder, encoder, decoder and CODEC all refer to specific machines designed for the coding (encoding and/or decoding) of video data consistent with this disclosure.
[0035] Video encoding systems achieve compression by removing redundancy in the video data, e.g., by removing those elements that can be discarded without adversely affecting reproduction fidelity. Because video signals take place in time and space, most video encoding systems exploit both temporal and spatial redundancy present in these signals. Typically, there is high temporal correlation between successive frames. This is also true in the spatial domain for pixels which are close to each other. Thus, high compression gains are achieved by carefully exploiting these spatio-temporal correlations.
[0036] A high-level description of how video data gets encoded and decoded by the encoders 116 and the decoders 138 and 140 in an embodiment of the disclosure will now be provided. In this embodiment, the encoders and decoders operate according to a High Efficiency Video Coding (HEVC) method. HEVC is a block-based hybrid spatial and temporal predictive coding method. In HEVC, an input picture is first divided into square blocks, called LCUs (largest coding units) or CTUs (coding tree units), as shown in FIG. 2A. Unlike other video coding standards, in which the basic coding unit is a macroblock of 16×16 pixels, in HEVC, the LCU can be as large as 128×128 pixels. An LCU can be divided into four square blocks, called CUs (coding units), which are a quarter of the size of the LCU. Each CU can be further split into four smaller CUs, which are a quarter of the size of the original CU. The splitting process can be repeated until certain criteria are met. FIG. 3A shows an example of LCU partitioned into CUs.
[0038] A CU that is not split (e.g., a CU corresponding a terminal, or "leaf" node in a given quadtree) may include one or more prediction units (PUs). In general, a PU represents all or a portion of the corresponding CU, and includes data for retrieving a reference sample for the PU for purposes of performing prediction for the CU. Thus, at each leaf of a quadtree, a final CU of 2N×2N can possess one of four possible patterns (N×N, N×2N, 2N×N and 2N×2N), as shown in FIG. 2B. While shown for a 2N×2N CU, other PUs having different dimensions and corresponding patterns (e.g., square or rectangular) may be used. A CU can be either spatially or temporally predictive coded. If a CU is coded in intra mode, each PU of the CU can have its own spatial prediction direction. If a CU is coded in inter mode, each PU of the CU can have its own motion vector(s) and associated reference picture(s). The data defining the motion vector may describe, for example, a horizontal component of the motion vector, a vertical component of the motion vector, a resolution for the motion vector (e.g., one-quarter pixel precision or one-eighth pixel precision), a reference frame to which the motion vector points, and/or a reference list (e.g., list 0 or list 1) for the motion vector. Data for the CU defining the one or more PUs of the CU may also describe, for example, partitioning of the CU into the one or more PUs. Partitioning modes may differ between whether the CU is uncoded, intra-prediction mode encoded, or inter-prediction mode encoded.
[0040] In general, in inter-prediction encoding, video sequences have high temporal correlation between frames, enabling a block in the current frame to be accurately described by a region in the previous coded frames, which are known as reference frames. Inter-prediction utilizes previously encoded and reconstructed reference frames to develop a prediction using a block-based motion estimation and compensation technique.
[0041] Following intra-predictive or inter-predictive encoding to produce predictive data and residual data, and following any transforms (such as the 4×4 or 8×8 integer transform used in H.264/AVC or a discrete cosine transform (DCT)) to produce transform coefficients, quantization of transform coefficients may be performed. Quantization generally refers to a process in which transform coefficients are quantized to possibly reduce the amount of data used to represent the coefficients, e.g., by converting high precision transform coefficients into a finite number of possible values. These steps will be discussed in more detail below.
[0042] Each CU can also be divided into transform units (TUs) by application of a block transform operation. A block transform operation tends to decorrelate the pixels within the block and compact the block energy into the low order coefficients of the transform block. In some embodiments, one transform of 8×8 or 4×4 may be applied. In other embodiments, a set of block transforms of different sizes may be applied to a CU, as shown in FIG. 5A where the left block is a CU partitioned into PUs and the right block is the associated set of transform units (TUs). The size and location of each block transform within a CU is described by a separate quadtree, called RQT. FIG. 5B shows the quadtree representation of TUs for the CU in the example of FIG. 5A. In this example, 11000 is coded and transmitted as part of the overhead.
[0046] In general, blocks and the various partitions thereof (e.g., sub-blocks) may be considered video blocks. In addition, a slice may be considered to be a plurality of video blocks (e.g., macroblocks, or coding units), and/or sub-blocks (partitions of macroblocks, or sub-coding units). Each slice may be an independently decodable unit of a video frame. Alternatively, frames themselves may be decodable units, or other portions of a frame may be defined as decodable units. Furthermore, a GOP, also referred to as a group of pictures, may be defined as a decodable unit.
[0048] There are several possible spatial prediction directions that the spatial prediction module 129 can perform per PU, including horizontal, vertical, 45-degree diagonal, 135-degree diagonal, DC, Planar, etc. Including the Luma intra modes, an additional mode, called IntraFromLuma, may be used for the Chroma intra prediction mode. A syntax indicates the spatial prediction direction per PU.
[0050] The prediction PU may then be subtracted from the current PU, resulting in the residual PU, e. The residual PU, e, may then be transformed by a transform module 117, one transform unit (TU) at a time, resulting in the residual PU in the transform domain, E. To accomplish this task, the transform module 117 may use e.g., either a square or a non-square block transform.
[0052] The quantized coefficients may then be entropy coded by an entropy coding module 120, resulting in the final compression bits. The specific steps performed by the entropy coding module 120 will be discussed below in more detail.
[0056] The SAO filter process is applied after the DBF and is made to allow for better reconstruction of the original signal amplitudes by using e.g., a look up table that includes some parameters that are based on a histogram analysis made by the encoder. The SAO filter has two basic types which are the edge offset (EO) type and the band offset (BO) type. One of the SAO types can be applied per coding tree block (CTB). The edge offset (EO) type has four sub-types corresponding to processing along four possible directions (e.g., horizontal, vertical, 135 degree, and 45 degree). For a given EO sub-type, the edge offset (EO) processing operates by comparing the value of a pixel to two of its neighbors using one of four different gradient patterns. An offset is applied to pixels in each of the four gradient patterns. For pixel values that are not in one of the gradient patterns, no offset is applied. The band offset (BO) processing is based directly on the sample amplitude which is split into 32 bands. An offset is applied to pixels in 16 of the 32 bands, where a group of 16 bands corresponds to a BO sub-type. The SAO filter process was designed to reduce distortion compared to the original signal by adding an offset to sample values. It can increase edge sharpness and reduce ringing and impulse artifacts. Further detail on the SAO process will be discussed below with reference to FIGS. 6-14.
[0058] The operation of the entropy coding module 120 (FIG. 4A) according to an embodiment will now be described in more detail. The entropy coding module 120 takes the quantized matrix of coefficients received from the quantizer module 118 and uses it to generate a sign matrix that represents the signs of all of the quantized coefficients and to generate a significance map. A significance map may be a matrix in which each element specifies the position(s) of the non-zero quantized coefficient(s) within the quantized coefficient matrix. Specifically, given a quantized 2D transformed matrix, if the value of a quantized coefficient at a position (y, x) is non-zero, it may be considered as significant and a "1" is assigned for the position (y, x) in the associated significance map. Otherwise, a "0" is assigned to the position (y, x) in the significance map.
[0059] Once the entropy coding module 120 has created the significance map, it may code the significance map. In one embodiment, this is accomplished by using a context-based adaptive binary arithmetic coding (CABAC) technique. In doing so, the entropy coding module 120 scans the significance map along a scanning line and, for each entry in the significance map, the coding module chooses a context model for that entry. The entropy coding module 120 then codes the entry based on the chosen context model. That is, each entry is assigned a probability based on the context model (the mathematical probability model) being used. The probabilities are accumulated until the entire significance map has been encoded.
[0060] The value output by the entropy coding module 120 as well as the entropy encoded signs, significance map and non-zero coefficients may be inserted into the bitstream by the encoder 116 (FIG. 1A). This bitstream may be sent to the decoders 138 and 140 over the HFC network 122.
[0061] It should be noted that the prediction, transform, and quantization described above may be performed for any block of video data, e.g., to a PU and/or TU of a CU, or to a macroblock, depending on the specified coding standard.
[0062] When the decoders 138 and 140 (FIG. 1A) receive the bitstream, they perform the functions shown in e.g., FIG. 4B. An entropy decoding module 146 of the decoder 145 may decode the sign values, significance map and non-zero coefficients to recreate the quantized and transformed coefficients. In decoding the significance map, the entropy decoding module 146 may perform the reverse of the procedure described in conjunction with the entropy coding module 120--decoding the significance map along a scanning pattern made up of scanning lines. The entropy decoding module 146 then may provide the coefficients to a dequantizer module 147, which dequantizes the matrix of coefficients, resulting in E'. The dequantizer module 147 may provide the dequantized coefficients to an inverse transform module 149. The inverse transform module 149 may perform an inverse transform operation on the coefficients resulting in e'. Filtering and spatial prediction may be applied in a manner described in conjunction with FIG. 4A.
[0063] In an SAO process, an offset is added to each pixel to reduce the distortion of the reconstructed pixel relative to the original pixel. In one embodiment, for a partition in a luma or chroma component, an encoder categorizes the pixels into one of six possible types (both types and sub-types are collectively referred to as types here): four edges offset (EO) types E0, E1, E2, E3 and two band offset (BO) types B0, B1. For the EO types, the pixels are further sub-categorized into one of five possible sub-classes based upon local behavior along the EO type direction. These five sub-classes are described in further detail below. For the BO types, the pixels are further sub-categorized into one of sixteen possible sub-classes based upon intensity. In some embodiments, for a given sub-class of pixels within an SAO type, the same offset is applied. For example, if the offset for sub-class i is oi, then the SAO output corresponding to an input of pi will be pi+oi. The encoder typically selects the SAO type per sub-class to minimize a cost function. For example, if the distortion for a given type t and set of offsets ot,i is Dt,i and the corresponding bitrate is Rt,i, then the cost function can be Jt,i=Dt,i+lambda*Rt,i, where lambda is a weighting factor. The encoder may signal to the decoder the SAO type per partition and the corresponding offsets per sub-class, and the decoder may perform the classification for the SAO type and applies the offsets per sub-class to each pixel. The SAO type can be signaled per color component, or a given type can be signaled and used for more than one color component. In some embodiments, it is also possible for the encoder to not use or turn off SAO, and this can also be signaled to the decoder.
[0064] For coding of SAO type, there are generally two coding methods: high efficiency (HE) and low complexity (LC). In LC, variable length codewords (VLCs) or binarized codewords are assigned to the SAO types; while in HE, the binarized codeword typically assigned to the type is followed by context-based adaptive binary arithmetic coding (CABAC). For the HE case, an encoder may signal the SAO type using a unary code, for example (0's and 1's can be interchanged) as shown in Table 1:
[0065] In Table 1, when SAO type is Off, no SAO is applied and the corresponding codeword is 0. The other codewords correspond to the other EO and BO types.
[0066] It may be noted that the units or digits within a codeword may be referred to as "bits" for LC and "bins" for HE. The difference in terminology is a result of applying CABAC to the codeword in the HE method. As used herein, "units" includes both bins and bits in codewords.
[0067] As described above, there are four possible EO types or classes, and five possible sub-classes per type. As used here, EO type or class refers to the direction along where pixels will be processed, and sub-class refers to the categorization of pixel values according to the gradient pattern along the EO type or class direction. The 5 possible EO sub-classes per type (e.g., sub-class 0-4) are illustrated in FIG. 6, where the 3 pixels indicate whether neighboring pixels are greater than, less than, or equal to the current pixel. The specific numbering of sub-classes in FIG. 6 is shown for illustration purposes only and other numberings are also contemplated. For example, referring to sub-class 0, at Line 1, Line 1 is defined by three points or pixel values: left neighbor pixel value (L), current pixel value (C) and right neighbor pixel value (R). Line 1, therefore illustrates a current pixel value C with a left neighboring pixel L having a smaller value and a right neighboring pixel R having a greater value than C. Line 2 illustrates a current pixel value C with a left neighboring pixel L having a higher value and a right neighboring pixel R having a smaller value than C. Line 3 illustrates a current pixel value C with a left neighboring pixel L having an equal value and a right neighboring pixel R having an equal value to C.
[0068] Generally, for sub-class 0, no offset is applied to the current pixel value C. For sub-class 1 and sub-class 2, the current pixel value C is generally lower than its neighboring pixels L and R, so a positive offset value to increase the value of C may be applied. For sub-class 3 and sub-class 4, the current pixel value C is generally greater than its neighboring pixels L and R, so a negative offset value to decrease the value of C may be applied. Generally, the offset (O) for a given sub-class is applied to all pixels in the sub-class, and the output (V) of the E0 operation is V=C+O.
[0069] In some embodiments, a predicted or interpolated current pixel value (I) may be generated using a weighted combination of its two neighbors, L and R. For example, if 2*C>(L+R), then the interpolated pixel value (I) can be computed as (L+R+1)>>1 (round to +infinity). If 2*C<(L+R), then the interpolated pixel value (I) can be computed as (L+R)>>1 (round to -infinity). In this example, the interpolated current pixel value I is an average of its two neighbors, but in general it can be a weighted combination or function of one or more neighboring values. FIG. 7 illustrates an example of a segmented line formed by L, C, and R and the relationship of I to the segmented line.
[0070] In other examples, if 2*C>(L+R), then the interpolated pixel value (I) can be computed as (L+R+1)>>1 (I is rounded towards C). If 2*C<(L+R), then the interpolated pixel value (I) can be computed as (L+R)>>1 (I is rounded towards C). For the case of 2*C=(L+R), C is effectively already on the interpolated line so that C=I, no offset is applied, and V=C.
[0071] While these weighted combinations use a relatively simple function, any function that takes into account one or more neighboring pixel values to generate I may be used. Additionally, the neighboring pixel values need not be the immediate neighboring pixels.
[0072] In some embodiments, if C=I, then no offset is applied, and the output (V)=C. If the offset (O) applied to C brings the pixel value away from I, or away from the segmented line formed by (L, I, R), then the offset is applied and V=C+O. If the offset (O) applied to C brings the pixel value towards I (towards the line) but does not pass I, then the offset is applied and V=C+O. If the offset (O) applied to C brings the pixel value towards I and it passes I, then the output pixel value is set to I, that is, V=I. In other words, if the offset (O) is applied to C and the result passes I, then the output value V is set to I, thereby limiting O.
[0073] In some embodiments, using the interpolated current pixel value (I) serves as a smoothing function because it operates as a limit defined by a function (e.g., average) of its neighbors. This, in turn, allows the offset (O) to serve efficiently both as a noise threshold to reduce variations caused by noise and as a signal enhancer to restore signal values.
[0074] Generally, offsets (O) may be allowed to be applied both towards and away from the segmented line formed by (L, I, R or L, C, R), such that the offset can be defined to be a signed value, where the sign can indicate the direction towards or away from the line, and the magnitude indicates the amount of movement.
[0075] In some embodiments, offsets are only allowed to be applied in the direction towards the segmented line formed by (L, I, R), thus allowing a single sign classification (e.g., defining the offset as non-negative). In such instances, only the offset magnitude needs to be transmitted.
[0076] In some embodiments, it may be beneficial to define the interpolated pixel value (I) differently than above, such as the rounding for I be away from C. For example, if 2*C>(L+R), then the interpolated pixel value (I) can be computed as (L+R)>>1 (away from C). If 2*C<(L+R), then the interpolated pixel value (I) can be computed as (L+R+1)>>1 (away from C). With this definition for I, the application of the offset O can proceed as described above. For example, if 2*C=(L+R), C is effectively already on the interpolated line so that C =I, no offset is applied, and V=C. With such a definition for I, it allows for some sharpening to be performed using an offset when C is close to but not equal to I.
[0077] In some embodiments, the current pixel value (C) may not be available or is hard to retrieve e.g., LCU boundary pixels. In such instances, the predicted or interpolated pixel value (I) can be derived from the available pixels e.g., along the same direction defined by EO type. The EO offset direction can be defined according to the predicted or interpolated pixel value. In some embodiments, the predicted pixel or interpolated value can be used in classifying the current pixel value, or a combination of the predicted pixel value together with the neighboring pixels can be used in the classification.
[0078] Although it was mentioned above that the interpolated pixel value (I) can be a weighted combination of its two neighbors, more generally I can be computed based on different neighboring values as well as on C and other parameters or functions. The parameters can include e.g., coefficients or weights for an M-tap filter or an offset, and the functions can include e.g., linear or non-linear combinations. For example, if the current and two neighboring values are C, L, and R, then the output value V can be computed as V=L+R+W*C+O, where the weight W and offset O can be signaled. These parameters can be signaled to the decoder per partition, LCU, slice, picture, group of pictures, or sequence, or be known to both encoder and decoder without further signaling.
[0079] Referring back to FIG. 6, it has been observed that for sub-class 1 and 2 pixels, the offset tends to be negative (towards the line). For sub-class 3 and 4 pixels, the offset tends to be positive (towards the line). Since in both cases, the offset is applied towards the line, the offset may be defined to be positive if it is in the direction of the line, and negative if away from the line (and zero for no change). With such a definition, entropy coding of the offset can take advantage of the higher probability that the offset will be positive (or non-negative). Consequently, shorter codewords may be assigned to the positive offset values, thereby achieving bit rate savings.
[0080] Alternatively, in some embodiments, the negative offsets are ignored or discarded, thus simplifying the coding of offsets. Ignoring the negative offsets may have some advantages for e.g., subjective quality of reconstructed frames. If negative offsets are not used or allowed, then the entropy coding can be designed appropriately. For example, in some embodiments, when negative offsets are not allowed, the most frequent offset in not zero, but rather a positive offset (e.g., 1). In this instance, by assigning the shortest codeword to this positive offset, bit rate savings can be achieved. In other words, assigning the shortest codeword to a most probable non-negative offset may result in greater efficiencies.
[0081] In conventional SAO, no offset is applied to sub-class 0 pixels, shown in FIG. 6. In contrast, the present disclosure provides applying an offset to these sub-class 0 pixels as described herein.
[0082] As explained above and shown in FIG. 7, a segmented line may be formed by L, R, and C. If the current sub-class 0 pixel value is below the line (e.g., C has a lower value than the average value of its neighbors L, R), the pixel is re-categorized or placed into sub-class 1 or 2 or a new sub-class, e.g., sub-class 5. If the current sub-class 0 pixel value is above the line (e.g., C has a greater value than the average value of its neighbors L, R), the pixel is re-categorized or placed into sub-class 3 or 4 or a new sub-class, e.g., sub-class 6. If the current sub-class 0 pixel is on the line, then it remains as sub-class 0 where no offset is applied. It should be appreciated that using this offset application technique, offsets may be applied to more pixels to improve performance without transmitting additional offsets.
[0083] In some embodiments, additional offset classes or sub-classes can be defined using e.g., distance thresholds, where a different offset is applied based on distances between the current pixel value, the predicted or interpolated pixel value, and/or the neighboring pixel values. For example, FIG. 8 illustrates an example segmented line formed by L, C, and R and the relationship of I to the segmented line. As shown, the current pixel value C is a very large distance from the interpolated pixel value I. Using conventional classification, C would be classified as belonging to sub-class 1, with a standard or predetermined offset O being applied. However, from inspecting FIG. 8, it is apparent that C is approximately the distance of three offsets O from I. Consequently, applying an offset that is larger than the predetermined offset O may be beneficial. In other words, different offsets may be applied depending on the distance between C and I.
[0084] In such instances, additional sub-classes can be defined based upon the distance of C from I, and additional offsets can be signaled for each sub-class. Alternatively, a scaled offset can be applied, where the scale factor depends on the distance between C and I.
[0085] In some embodiments, when C is close to I, such as within a distance T, an offset that moves in either direction (e.g., a signed offset) is provided to allow for both sharpening or smoothing, where the offset brings C away from or closer to I, respectively. In such instances, a new EO sub-class (EN) may be defined as provided herein.
[0086] The current pixel value C may be classified into sub-class EN if |C-I|<T, where T represents a closeness threshold between C and I, and an offset can be transmitted for the new sub-class EN. Otherwise, C may be categorized into other EO sub-classes (e.g., sub-class 0-4) such as before. For example, when T=I, then sub-class EN corresponds to the case where C=I. The entropy coding (e.g., VLC or binarization and CABAC) of sub-class offsets can be modified to better match the statistics of the offset values. For example, in some embodiments, the statistics of the offset values may at least partially depend on or relate to the value of the closeness threshold (T) chosen.
[0087] In some embodiments, the number of conventional sub-classes (e.g., 5) may be reduced. For example, sub-class 1 and 2 pixels may be combined into a new sub-class 1' and sub-class 3 and 4 pixels may be combined into a new sub-class 2', resulting in two offsets per EO type. For non-sub-class 0 pixels, the new sub-class 1' pixels are such that the two neighbors are both greater than or equal to the current pixel, and for non-sub-class 0 pixels, the new sub-class 2' pixels are such that the two neighbors are both less than or equal to the current pixel. A generalization of defining sub-classes to reduce the overall number of sub-classes and inclusion of pixels in current sub-class 0 can be achieved by defining two (three total) sub-classes of pixels: One group has pixel values higher than interpolated value I, and a second group has pixel values lower than interpolated value I (and the third sub-class has pixels that are equal to interpolated value I). By reducing the number of EO sub-classes, complexity may be reduced and fewer offsets need to be transmitted.
[0088] Further reduction of sub-classes for the purpose of optimizing offsets can be achieved by classifying pixels into two categories. The first category may include all the pixels with the same value as the interpolated value I. The second category may include all the other pixels. In such instances, two offsets (one for each of the two sub-classes) may be signaled, or one offset may be signaled for the second category. The signaled offset value for the first category, if signaled, may be a signed value while the signaled offset value for the second category may be unsigned and the sign of actual offset, to be applied to each pixel, may be derived from the relative position of that pixel relative to the interpolated value I.
[0089] As described above, there are four possible EO types or classes, and five possible sub-classes per type. As used here, EO type or class refers to the direction along where pixels will be processed, and sub-class refers to the categorization of pixel values according to the gradient pattern along the EO type or class direction. In some embodiments, the number of EO sub-classes may be extended to a total of nine sub-classes, where each pixel is classified depending on whether it is smaller, equal, or larger than the two neighboring pixels along the direction indicated by EO type or class.
[0090] It should be appreciated that although the number of EO sub-classes was described as including nine, any suitable increased number (e.g., greater than five) may be used. Because of the additional number of sub-classes, more offsets may need to be sent to the decoder. Although more offsets may need to be sent for the additional sub-classes, the reduction in distortion may improve performance.
[0091] In some embodiments, one or more of the above EO modifications can be combined to improve overall performance. It should be appreciated that the SAO (EO) sub-classes and offsets described herein can be signaled at a partition, LCU, slice, picture, group of pictures, or sequence level. The SAO (EO) sub-classes and offsets can also be combined with band offset types and offsets signaled at the partition, LCU, slice, picture, group of pictures, or sequence level.
Modifications to BO Types and Sub-Classes
[0092] Currently, SAO uses two fixed band types, B0 and B1, covering the entire intensity range, with each band further dividing the respective intensity range into 16 equal sub-classes. An offset can be signaled for each of the sub-classes. Because the statistics of a given picture may not fall nicely into one of the two existing band types, B0 and B1, it may be preferable to combine or merge the bands. In some embodiments, one band type can be used, where the range of values to apply an offset can be specified, and a number of sub-classes for the range can be specified, e.g., using a uniform sub-partitioning. An example of such partitioning using a single band type is illustrated in FIG. 9. In some embodiments, the range of values define the one or more sub-classes.
[0093] In some embodiments, the range of values where the offset is applied can be determined based on the data and on rate-distortion considerations. The offsets may generally be applied to values where the distortion can be reduced.
[0094] In some embodiments, SAO selection type need not be performed, such as when there is a single band type and no other SAO type. In such instances, the single band type is used without the additional steps associated with SAO selection.
[0095] As shown in FIG. 9, the start of the band is specified by bs, and Ns sub-classes of width ws can be used. FIG. 9 shows one embodiment where four (Ns=4) sub-classes of equal width (ws) adjoining each other, where the first sub-class starts at bs. In this case, four offsets can be signaled to the decoder for the four sub-classes. In one example, if the last sub-class exceeds the maximum intensity range, the last sub-class can end at the maximum value or wrap around to zero. Additional discussion on BO sub-classes can be found in U.S. patent application Ser. No. 13/672,476, entitled "Devices and Methods for Sample Adaptive Offset Coding and/or Signaling," filed on Nov. 8, 2012, and U.S. patent application Ser. No. 13/672,484, entitled "Devices and Methods for Sample Adaptive Offset Coding and/or Signaling," filed on Nov. 8, 2012, incorporated herein by reference in their entirety.
[0096] In some embodiments, bs is transmitted from the encoder to the decoder. In some embodiments, Ns is transmitted from the encoder to the decoder. In some embodiments, ws is transmitted from the encoder to the decoder.
[0097] Alternatively, a fixed set of values of bs, Ns and/or ws can be specified and agreed upon at the encoder and/or decoder. In such embodiments, only some parameters (e.g., the unspecified values) may need to be transmitted from the encoder to decoder. For example, these parameters can be signaled to the decoder and can be determined for e.g., per partition, LCU, slice (or other unit), picture, group of pictures, sequence, etc.
[0098] The specified values need not be transmitted to the decoder, assuming that both encoder and decoder are using the same unit. As used herein, a unit refers to data that both the encoder and decoder have and are configured to derive SAO parameters from. For example, a unit may refer to an LCU, slice, picture, etc. The unit may be specified implicitly (e.g., fixed, slice-dependent, prediction list-based, etc.) or explicitly (e.g., in sequence or slice header, etc.). In such instances, the range of reconstructed values can be determined after the unit is encoded or decoded, and the BO type may be derived from this range.
[0099] In some embodiments, the unit can be signaled from the encoder to the decoder, or derived from other coding parameters. Examples of the unit can include an LCU, slice, picture, or group of pictures.
[0100] For each BO type, the set of values for which SAO is applied can be determined from the unit or a portion of the unit. For example, the set of values can be determined from a subset of samples in the unit, e.g. alternating samples, the first quarter LCU pixel samples (e.g., top left quarter of the unit), the four corner and center samples of the unit, pixel samples that are not affected by the deblocking filter, etc.
[0101] It has been discovered that because natural images tend to exhibit high spatial correlation, a smaller unit may tend to have a smaller dynamic range of intensity values, thereby requiring fewer sub-classes or offsets. In other words, an SAO unit size may be selected that is suitable for typical data, which yields good performance given the number of offsets applied, without requiring too much buffer, delay, etc. In order to strike a balance between coding efficiency, parallel processing, delay, buffering, etc., an LCU unit in the range of 16×16 to 128×128 may be generally suitable for one or more processes described herein.
[0102] For example, in some embodiments, the encoder and decoder can decode an LCU and determine the range of values [min, max] in the LCU. A BO type (or types) can then be defined based on the range of values [min, max]. That is, the range of values [bs, es] in FIG. 9 for a BO type can be derived based on [min, max], with bs denoting the start of the BO type and es denoting the end of the BO type. In one example, the values can be chosen to be bs=min and es=max. Once bs and es are determined, the band can be divided into Ns sub-classes (note that Ns can be different for different BO types), for example, Ns sub-classes of equal width ws. The number of sub-classes Ns or width ws may be agreed upon at the encoder and decoder, as described above, or it can be transmitted for the LCU, at LCU level, slice level picture level, etc. for one or more color planes.
[0103] More generally, one or more BO types can be defined based on [min, max] and/or other statistical features of pixels within a corresponding area. For example, they can be defined based upon the statistical distribution of values within the range, such as illustrated in FIG. 10, which plots a distribution (e.g., histogram) of values. In FIG. 10, bs=min and es<max, so that no offset is applied to values larger than es. Four BO types are defined by B0, B1, B2, and B3 (in general BO types can cover overlapping or non-overlapping pixel values.). For each type Bi, Ni sub-classes can be defined by uniform or non-uniform partitioning of the Bi range. For the case of B1, the sub-classes can be distributed among the two intervals shown. The additional information about the BO types and sub-classes shown in FIG. 10 can either be transmitted to the decoder or agreed upon the encoder and decoder. Alternatively, they can be derived (and not transmitted) based upon statistical properties of the values, such as the mean, variance, etc.
[0104] In some embodiments, for a set of values specified by the range [min, max], the bs and es values can be obtained from the minimum and maximum values in the subset of samples. Alternatively, in some embodiments, the bs and es values may be determined based on a function or transformation of the samples. For example, a transformation such as a DCT can be used, and bs and es can be derived from the transformed values, e.g. DC coefficient, AC coefficient, etc. More generally, bs and es, or the range of values for the BO type, can be derived from samples wi and vi, for example bs=f(w0, w1, . . . , wn) and es=g(v0, v1, . . . , vm), where f(wi) and g(vi) are functions of the values wi and vi. The values wi and vi can be, for example, DCT coefficients. By determining the range of values based on a smaller subset of values in the unit, less computation may be required.
[0105] In some embodiments, other statistics of the values in a unit can be used to specify a BO type. For example, the mean of the values may be used to specify the location of sub-class intervals for a BO type. In some embodiments, the mean may be computed on a subset of values within the unit (e.g., LCU) and the subset of values can be in a certain region in the unit, alternating samples (or every nth sample horizontally and mth sample vertically), etc. A benefit of computing the mean value is that only one addition per pixel is performed in the subset of unit (e.g., LCU) and a division operation, or if the number of samples is 2n, a shift of the final total by n bits. For example, if the number of samples over which the mean (average) value is computed is a power of two, then the division by that number can be achieved by a bit shift, which can require less computation than a general division operation.
[0106] Once the mean value is computed or obtained, a BO type and/or set of sub-classes can then be derived based on the mean value. For example, a number of sub-classes can be specified relative to the mean value. The sub-classes can be centered or shifted about the mean and can cover a limited range or extended to cover the entire range (e.g., by extending the range of the first and last sub-classes as explained in further detail below).
[0107] FIG. 11 shows an example where a BO type is specified using the mean value (m) and a width (ws) for a case of 4 sub-classes with extended first and last sub-classes. For the case of only one sub-class (one offset) with a limited range or span, the mean can be the center (or approximately the center) of that span. For more sub-classes, the mean can indicate the center (or approximately the center) of all sub-classes. In another example, the entire range may be divided into fixed intervals. For the case of one sub-class, the sub-class can be defined as the interval in which the mean value is located. For additional sub-classes, adjacent intervals can be chosen, for example, centered about the interval with the mean value.
[0108] In some embodiments, it may be preferable to perform operations by shift, e.g., by mapping a pixel value to a band index. This mapping may be achieved using a look-up table or by other means (e.g., binary-shift of pixels values). In an example, if the 0-255 pixel range is divided into 32 uniformly spaced bands (corresponding to 32 offsets), then each pixel value may be shifted by 3 binary positions to the right to get the band-index. As is known, the band-index is useful is determining which offset should be applied to each pixel.
[0109] In other words, to categorize a pixel value into a sub-class to determine what offset should be applied, one can perform a simple bit shift operation and the result can be used to index the offset value. Merging can be used to combine bins to share a given number of offsets. Consequently, in some embodiments, a combination of pixel value shift and merging offsets may be used to map pixel values in non-uniform and/or discontinuous situations. In such instances, for example, the finest granularity of offset-intervals may be defined by shift and they may then be merged based on min-max or some other conditions to include only N contiguous offset. In one example, 4 offsets are transmitted, to be applied to 4 regions which are not necessarily uniform or contiguous. Each region may be formed by merging uniform bands on a finer granularity (in the finest granularity, there are 32 uniformly spaced bands). Based on statistics (e.g., min-max) or instructions, the 4 offsets are duplicated to some of the finer-uniform bands and assigned an offset of zero to the rest of finer-uniform bands. By this approach, the pixel values may be shifted to find the offset without the need to send all offsets at finest granularity precision (e.g., in this example transmitting 4 offsets instead of 32 offsets). It should be appreciated, for the case of only one sub-class (N=1), there is no need to calculate min and max and only the offset needs to be transmitted.
[0110] By determining the set of values for the BO type based on the unit or portion of the unit, the BO type parameters need not be transmitted, thereby saving on overhead bits. In some embodiments, the BO parameters for a current unit can be determined based on a unit previously available to the encoder and decoder. Using previously available unit parameters may reduce latency so that SAO processing can begin on the first sample in the unit without needing to process the unit to determine the BO type parameters. In one example, a collocated unit or motion-compensated unit from a previously coded picture can be used to determine the BO parameters for a current unit.
No Offset for Empty Sub-Classes
[0111] In some embodiments, for the existing B0 and B1 band offset types and/or for a single merged band offset type, there may be many sub-classes with no pixels in the respective intensity range (e.g., also known as empty sub-classes). Although it is possible to encode these sub-classes with a zero offset, in some embodiments, only the offset values for those sub-classes that have pixel intensity values are encoded and signaled. Such encoding of sub-classes that have pixel intensity values may be achieved by additionally encoding an escape code or end-of-offset code to signal no more offset values. This escape code can be, for example, a value that is larger than the maximum offset value used. This approach can be beneficial when there are many empty sub-classes; however, in cases where there are not many empty sub-classes, a combination of only encoding sub-classes having intensity pixel values and encoding sub-classes with a zero offset may be implemented. The approach can be used for signaling of offsets in both band offset and edge offset types. For the case of edge offset types, an empty sub-class corresponds to the case where there are no pixels with the respective gradient pattern. Additional discussion on offset for empty sub-classes can be found in U.S. patent application Ser. Nos. 13/672,476 and 13/672,484, previously incorporated by reference in their entirety.
[0112] As is appreciated, in one embodiment the decoder receives information on a band offset specification type such as shown in FIG. 9. The decoder classifies the reconstructed pixel values into the sub-classes according to their intensities. When the decoder receives the sequences of offset values, it can assign the offsets to each sub-class according to where pixel intensities exist in the sub-class.
[0113] In some embodiments, sub-classes where there are no pixel intensities will have no offset signaled. FIG. 12 illustrates this as an example. FIG. 12 shows an example of BO with eight sub-classes 0-7. The locations of the eight sub-classes or range of pixel amplitudes can be signaled to the decoder using methods previously described. In the example, there are only pixel intensities in sub-classes 1 and 6, while there are no pixel intensities in sub-classes 0, 2, 3, 4, 5, and 7. The latter sub-classes are empty and so no offsets need to be signaled. The offset value of 2 for sub-class 1 and value of -1 for sub-class 6 can be signaled, followed by an optional escape value signaling no more offset values. If the escape value is not signaled, then it is assumed that the decoder performs pixel classification into sub-classes prior to parsing the offset values. After the decoder receives the information specifying the BO sub-classes using methods such as previously described, it can classify the pixel intensities. After classifying the pixel intensities, the decoder assigns the first offset value of 2 to the first non-empty sub-class of 1 and the second offset value of -1 to the second non-empty sub-class of 6.
Extensions of Sub-Classes
[0114] Referring back to FIG. 9, a BO type is specified by bs, es, and Ns (e.g. Ns=4), which represent the start of the BO, end of the BO, and the number of sub-classes, respectively. The values for bs and es can be determined from min and max, respectively, e.g. bs=min and es=max. The interval can then be partitioned into sub-classes. FIG. 9 shows the sub-classes of equal intervals, although this need not be the case.
[0115] In FIG. 9, no offset is transmitted or applied for values outside the range from bs to es. However, values may occur outside the range if bs and es are determined from a subset of samples in the unit, while the offset is applied to all the samples in the unit or to samples outside the subset. In such instances, it may be beneficial to apply an offset (e.g., non-zero) to these values outside the range. FIG. 11 illustrates a method for applying an offset to values outside the bs and es ranges, where the first and last sub-classes are extended to the smallest and largest possible values (e.g. 0 and 255, respectively). This is one example of non-uniform interval BO sub-classes, and other examples are possible. The offsets for each sub-class can be determined based on e.g., RD optimization of a subset of samples or for all the samples in the unit.
[0116] In addition to possible RD performance improvements with the proposed approach, the complexity of SAO BO can be reduced. In FIG. 9, it may be necessary to check whether a pixel value is inside the BO offset range or not. On the other hand, in the scheme illustrated in FIG. 13, the additional out of range checks are not needed.
[0117] This is because, in some embodiments, the first and last sub-classes in a BO type can be extended to cover more pixel values with the potential to increase coding efficiency and/or reduce complexity. For example, in one embodiment, the "first" boundary (after 0) can be specified as b_s+w_s, and the "last" boundary (before 255) can be specified as e_s-w_s, with N_s-2 intervals in between.
[0118] For example, in one implementation, the band offset oi corresponding to FIG. 13 for 4 sub-classes can be applied to a pixel value p with the following logic statement, where a0, a1, and a2 represent the three boundaries, and a0=bs+ws, a1=bs+2ws, and a2=bs+3ws:
TABLE-US-00002 if (p < a0) apply offset o0 else if (p < a1) apply offset o1 else if (p < a2) apply offset o2 else apply offset o3
[0119] In this case of 4 sub-class BO, at most 3 comparison operations are needed, as opposed to the 5 operations corresponding to a scheme represented in FIG. 9. Because the comparisons are generally performed per pixel, the reduction in operations/time can be significant. For the case of 2 sub-class BO, only 1 comparison operation is needed as opposed to 3 operations, and the operations may be reduced by 67% to a single comparison per pixel. This is because without the sub-class extension, 3 comparison operations are needed, whereas with the extension, only 1 comparison operation is needed.
[0120] The modifications described herein may allow for less overhead, computation, and latency for specification and processing using SAO BO. Note that the BO can be specified and applied to any or all color components, and the number of offsets can be different for each color component. For example, luma BO may have 4 offsets and chroma BO may have 2 offsets.
[0121] It should also be appreciated that the SAO band types and offsets described herein can be signaled at a partition, LCU, slice, picture, group of pictures, or sequence level. They can also be combined with EO types and offsets signaled at the partition, LCU, slice, picture, group of pictures, or sequence level.
Multipass Approach to Applying Offsets
[0122] As explained above with reference to FIG. 4A, a deblocking filter (DBF) operation may be performed on the reconstructed PU, x'', first to reduce blocking artifacts. A sample adaptive offset (SAO) process may be conditionally performed after the completion of the deblocking filter process for the decoded picture, which compensates the pixel value offset between reconstructed pixels and original pixels.
[0123] FIG. 14 illustrates a general block diagram of an SAO process 1400, with the incoming data from the deblocking filter shown in block 1410 and a first data set after a first SAO process 1420 is applied shown in block 1430. There is optionally shown a second set of data after a second SAO process 1440 is applied shown in block 1450.
[0124] In general, the incoming data in block 1410 contains signal and noise distortion relative to the original data. In contrast, the distortion has been reduced in the first data set in block 1430 from applying offsets from the SAO process 1420.
[0125] For SAO, the classification (e.g., edge type or band type) is usually determined using a rate-distortion process where the different types are applied and the one with the smallest cost is selected. Thereafter, the selected SAO type and offset information can be signaled to the decoder and applied to the data in block 1410.
[0126] However, in some embodiments, instead of passing information to define which offset type is applicable to a set of pixels, a set of offsets may be assumed for more than one SAO type and applied in a e.g., sequential manner. When more than one offset (e.g., a set of offsets) is applied, such as in a sequential manner, the process may be referred to as "multipass."
[0127] For example, a signal can be sent to specify either EO type or BO type. In some embodiments, if EO type is selected, then two sets of offsets can be sent, where the first set of offsets is applied along horizontal classification and the second set of offsets is applied based on vertical direction classification. By performing EO operations in more than one direction, it is expected that distortion can be further reduced along more edge orientations. Any two directions can be used, although two orthogonal directions may provide the best distortion reduction.
[0128] In some embodiments, no classification type is signaled, but three sets of offsets are signaled for three sequential SAO type operations. In each of the three passes, the pixels are categorized according to the SAO type and sub-class, and the offsets are applied. The first set of offsets may be applied using EO based on horizontal classification. After processing the first offsets, the second set of offsets may be applied in conjunction with vertical-edge classification, and the resulting pixels may then be used for BO classification, where the third set of offsets may be applied accordingly.
[0129] When using multipass SAO, in some embodiments, the first SAO classification 1420 and the second SAO classification 1440 use pixel values that have been decoded and deblocked (e.g., raw data in block 1410). In other embodiments, pixel values that have been decoded, deblocked and SAO has been applied (e.g., filtered data in block 1430) may be used for classification if they are available (e.g., such as for second SAO processing 1440). In LCU-based processing, for example, these decoded+deblocked+SAO values can occur on the left and upper boundaries of an LCU.
[0130] In one example, after a first SAO process, a second SAO process is applied. In this case, after a first SAO pass, the decoded+deblocked+SAO values from the current LCU in the first pass may be used for classification in the second SAO pass. Because the SAO offsets can be specified to decrease distortion, or keep it the same for the case of a zero offset or SAO off, performance may improve (or not degrade) with additional applications of SAO. In some embodiments, if overall rate-distortion performance is not improved, additional SAO passes can be disabled.
[0131] In some embodiments, two or more SAO passes can be applied, and a combination of edge offset and/or band offset can be applied. For example, EO can be applied to pixels in the first pass, and then BO can be applied to the resulting pixels from the EO pass. As is known by one of skill in the art, color is constructed from one luma component and two chroma components. Accordingly, each color component can have a different number of SAO passes and the SAO type for each pass per component can be different.
[0132] In general, multipass SAO can be applied starting with decoded pixel values, or after decoded+other_processing, where other processing can include deblocking, loop filtering, etc. Benefits of multipass SAO may include improved visual quality and coding efficiency. In addition, when SAO classification is predetermined, the classification type need not be transmitted, thus resulting in bit savings. For example, in some embodiments "predetermined" SAO classification refers to classifications that have been agreed upon between the encoder and decoder (e.g., perform two EO passes in two agreed upon directions).
[0133] Multipass may be applied using at least two or more passes with associated SAO types. In some embodiments, the SAO types are of the same type (e.g., horizontal EO type). In other embodiments, the SAO types are of a similar type but different sub-type (e.g., first is horizontal EO type and second is vertical EO type). In some embodiments, the SAO types are of a different type (e.g., first offset is EO and second offset is BO). With the number of SAO types and sub-types in fluctuation, there is a number of different possible combinations for desirable multipass processes.
[0134] Additionally, it should be appreciated that any of the EO and BO modifications may be applied as part of the multipass process. For example, a two-pass EO process can be applied using a new sub-class EN with one value of T used for EO classification and processing first along the horizontal direction, followed by EO classification and processing along the vertical direction with a different value of T, where T is a threshold relating a current pixel value C and an interpolated pixel value I.
[0135] The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles described herein can be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, it is to be understood that the description and drawings presented herein represent exemplary embodiments of the disclosure and are therefore representative of the subject matter which is broadly contemplated by the present disclosure. It is further understood that the scope of the present disclosure fully encompasses other embodiments and that the scope of the present disclosure is accordingly limited by nothing other than the appended claims.
2013-07-25 Flexible band offset mode in sample adaptive offset in hevc
2013-10-31 Systems, devices, and methods for continuous time signal processing
2013-05-09 Video decoder with enhanced sample adaptive offset