Intra-estimation for high performance video encoders

An example method of encoding a video includes selecting blocks of pixels in a frame of the video, the blocks having luminance (Y) blocks, red color difference (Cr) blocks, and blue color difference (Cb) blocks; performing intra-estimation based on reconstructed pixels of at the blocks of pixels to generate predicted blocks and then subtracting the predicted blocks from the blocks of pixels to generate residual data, the residual data comprising respective residual data for the Y-blocks interleaved with respective residual data for the Cr-blocks and the Cb-blocks; and generating new reconstructed pixels using a pipeline of a video encoder by processing the residual data for the blocks.

TECHNICAL FIELD

Examples of the present disclosure generally relate to electronic circuits and, in particular, to intra estimation for high performance video encoders.

BACKGROUND

Video codecs, such as H.264, HEVC (High Efficiency Video Coding), VP9 and AV1, use a hybrid approach to get the high degree of compression. For example, inter prediction is done to exploit the temporal correlation between the frames and intra prediction to exploit the spatial dependencies. Transform is done to take advantage of co-relation that exists in residual signal. Because of multiple available coding choices, video encoders have an estimation stage, where estimation of cost for various Intra/Inter modes and transform choices are tested fora given block, also referred as Coding Unit (CU). After deciding the best choice, it is sent to the next module (encode stage), which does the actual encoding. A CU includes three different components Y, Cb, Cr, where Y is Luminance, Cr is red color difference and Cb is blue color difference. There are three different color formats generally supported in codecs, YCbCr 4:2:0, 4:2:2, and 4:4:4.

SUMMARY

Techniques for providing intra-estimation for high performance video encoders are described. In an example, a method of encoding a video includes: selecting blocks of pixels in a frame of the video, the blocks having luminance (Y) blocks, red color difference (Cr) blocks, and blue color difference (Cb) blocks; performing intra-estimation based on reconstructed pixels to generate residual data for the blocks, the residual data comprising respective residual data for the Y-blocks interleaved with respective residual data for the Cr-blocks and the Cb-blocks; and generating new reconstructed pixels using a pipeline of a video encoder by processing the residual data for the blocks.

In another example, a video encoder includes: an estimation circuit configured to receive video frames; an encoder circuit, coupled to the estimation circuit, configured to receive the video frames and output of the estimation circuit; and an intra-estimation pipeline configured to: select blocks of pixels in a frame of the video, the blocks having luminance (Y) blocks, red color difference (Cr) blocks, and blue color difference (Cb) blocks; perform intra-estimation based on reconstructed pixels to generate residual data for the blocks, the residual data comprising respective residual data for the Y-blocks interleaved with respective residual data for the Cr-blocks and the Cb-blocks; and generate new reconstructed pixels using a pipeline of a video encoder by processing the residual data for the blocks.

In another example, method of encoding a video includes: selecting blocks of pixels in a frame of the video, the blocks having luminance (Y) blocks, red color difference (Cr) blocks, and blue color difference (Cb) blocks; performing intra-estimation for a first intra-mode based on reconstructed pixels to generate first residual data for the blocks; performing intra-estimation for a second intra-mode based on reconstructed pixels to generate second residual data for the blocks; and generating new reconstructed pixels using a pipeline of a video encoder by processing the first residual data interleaved with the second residual data.

DETAILED DESCRIPTION

Techniques for intra-estimation processing for high performance video encoders are described. The techniques are provided for accelerating the video encoding pipeline processing. Most of the time, during intra block processing, some of the blocks are sitting idle because of the dependency on neighboring block's data. During intra block processing, neighboring reconstructed pixels are required for prediction of the current block. Hence, the processing of the current block cannot start until the neighboring blocks' boundary pixels are fully reconstructed. The techniques described herein offer the strategy to keep the encoding pipeline running by interleaving color components, various modes, and transform sizes to create a large amount of non-dependent data. By using these techniques, performance for video encoders can be increased without any increase in hardware resources or loss of compression efficiency. The techniques are applicable to hardware encoders as well as multi-core software encoders. These and further aspects are discussed below with respect to the drawings.

FIG. 1Ais a block diagram depicting a video encoder100according to an example. The video encoder100includes an estimation circuit102and an encoding circuit104. The estimation circuit102receives input video data (e.g., YUV data). An output of the estimation circuit102is coupled to an input of the encoding circuit104. The estimation circuit102provides a best encoding choice to the encoding circuit104. Another input of the encoding circuit104receives the input video data (e.g., the YUV data). An output of the encoding circuit104provides a compressed bitstream. Another output of the encoding circuit104provides feedback to the estimation circuit102. The estimation circuit102is configured to test various intra/inter modes and transform sizes for a given block of video data. The estimation circuit102sends the best encoding choices to the encoding circuit104, which is configured to encode the input video data based on the selected video encoding technique using the selected encoding choices of the estimation circuit102.

FIG. 2is a block diagram depicting a video frame200according to an example. The input video data (e.g., YUV data) is divided into frames. Each frame is divided into blocks. During processing by the video encoder100, some blocks of a frame are already reconstructed; other blocks of the frame are being encoded; and still other blocks of the frame are yet to be encoded. Each block includes an array of image pixels (e.g., a 4×4 array of image pixels). During prediction, neighboring left, above-left, and above-right image pixels are used for a given block.

Intra block coding can choose between multiple Intra prediction modes and multiple transform sizes. For example, VP9 specification has 10 intra prediction modes and 4 (4/8/16/32) transform sizes. The estimation circuit102can employ Rate Distortion Optimization (RDO) for selection of best intra mode and transform size to achieve high coding efficiency. Various combinations of modes and transforms make the estimation circuit102highly compute intensive. RDO is based on Lagrange multiplier method:
J=D+λ*R

Where λ is Lagrangian multiplier, D is distortion calculated as Sum of Squared Difference(SSD) between the reconstructed pixels and original pixels, and R is the number of bits taken to encode residue coefficients and mode bits. J is generally referred as RDO cost and the chosen mode has minimum RDO cost. Lower distortion signifies lesser deviation from original source input hence better quality, whereas lesser bits signify better compression. Difference between the reconstructed and original pixels is caused by the quantization of transform coefficients. Quantization step is determined by the rate control algorithm, which is a key step for achieving target bitrates in video encoders.

Due to high complexity of the RDO process, most of the real time Video encoders performs Intra estimation in two steps, Coarse Intra Estimation (CIE) and Fine Intra Estimation (FIE). During CIE, actual RDO is not performed and a list of 2-4 winner Intra modes is prepared by using some low-cost method. This list of winner Intra modes is provided to FIE step, where actual RDO process is performed, to find out the best Intra mode and transform size. FIE step is highly compute intensive process and generally creates the bottlenecks in encoder's performance due to dependency on neighboring data. Proposed techniques described herein reduce the dependencies and achieve better performances as described further below.

The benefit of Intra prediction in video coding is well known and it has been used in all advanced video coding schemes such as H264, VP8, HEVC, VP9, AV1 etc. Of-course it differs in number of modes (directions), transform sizes and prediction pixel computation (fir-filtering) in different specifications, but in terms of implementation constraints affecting performance, challenges are same—dependency on neighboring blocks for prediction data. Requirement of neighboring pixels creates the data dependency between the blocks. Current block(C) has dependency on the pixels of left block(L), left-above block (LA), above block (A) and right-above block (RA) for its prediction. So, encoder processing of block ‘C’ can only start after availability of all neighboring block's reconstructed pixels. This dependency on reconstructed pixels of prior blocks adds latency for start of next block's processing and eventually most of the time some of the encoding blocks are idle and waiting for reconstructed data to be available.

FIG. 1Bis a block diagram depicting a pipeline101for intra-estimation (IE) according to an example. The IE pipeline101can be in the estimation circuit102, the encoding circuit104, or both. The IE pipeline includes various blocks. An Intra Prediction (IP) circuit106is connected to a subtractor118, which is in turn connected to a Transformation Frequency Domain (TFD) circuit108. IP circuit106generates predicted pixels. Subtractor118feeds the residual (difference of source pixels from the input YUV and prediction pixels) to the TFD circuit108, which transforms the residual into frequency domain. Transform coefficients are quantized in Quantization (Q) circuit110. After that, quantized coefficients are inverse quantized in Inverse Quantization (IQ) circuit116and Inverse transformed by the inverse transform (IT) circuit114to generate the reconstructed residual. A reconstruction (R) circuit120adds the reconstructed residual and predicted pixels (from the IP106) to generate the reconstructed pixels. Thereafter, these reconstructed pixels are used by the IP circuit106to generate the prediction buffer for next block. In summary, residual data in pipeline101traverses through the chain of processing circuits IP→TFD→Q→IQ→IT→R→IP. This chain starts with IP circuit106and terminates at the R circuit120and since the prediction unit for a given block is waiting for reconstructed pixels of previous block, most of the time some of the processing blocks in pipeline remain idle. This leads to underutilization of hardware/processing resources and impacts overall encoder's performance. Rate Distortion Optimization (RDO) circuit122computes the distortion and the bits estimation of a block to be encoded for a given encoding choice. It takes the data from the TFD circuit108and IQ circuit116to compute the distortion of the block. It also receives the quantized coefficients from Q circuit110to estimate the bits required to encode the given block. From distortion and bits estimate it compute the final cost (J) according to the Lagrange multiplier method as described with regard toFIG. 2. Final cost (J) is sent to the Decision circuit124which compares the cost of all available choices and finally selects the choice which has minimum encoding cost (J).

In the techniques described herein, various Intra modes, transform sizes and color components (Y,Cr,Cb) are arranged in a special order to have minimal pipeline stalled blocks. For example, Luma and Chroma data has no dependency on each other, so they can be pushed in consecutive cycles in the encoding pipeline. The scheme of interleaving Luma and Chroma blocks is named as Luma Chroma Interleave (LCI). Similarly, during estimation stage many Intra modes are tested to determine the best Intra mode in sequential order. In the proposed method, different Intra Modes are also interleaved along with color components. This scheme is named as Intra Mode Interleave (IMI). Both the schemes are explained below in detail. Similarly, many transform sizes are tested to determine the best transform size in sequential order. In the proposed method, different transform sizes are also interleaved along with color components (LCI) and intra modes (IMI).

FIG. 3depicts a processing order for pixel blocks according to an example. InFIG. 3, a diagonal processing order is shown for sixteen 4×4 blocks (e.g, a 16×16 block of pixels). In Table-1, a grouping of blocks is shown for which no wait is required, and they can be pushed consecutively in the pipeline101. It is also shown that when diagonal processing is performed, the pipeline101is stalled 12 times (Luma and Chroma both), while in the LCI scheme the pipeline101is stalled only 6 times. That is, in Table 1, the different steps are divided based on data dependency between the blocks. Blocks in the same step do not have data dependency. In the diagonal order case, all the blocks are processed in 13 steps, meaning the pipeline is stalled 12 times. In the techniques described herein, all the blocks are processed in 7 steps, meaning the pipeline is stalled only 6 times. The LCI scheme exploits the non-dependency of color components.

FIG. 4is a flow diagram depicting a method400of processing YUV data according to an example. The method400performs the LCI scheme described above. At step402, Y-, Cr-, and Cb-blocks are selected in the video frame. For example, Y0, Cr0, and Cb0. At step404, intra-prediction is performed based on reconstructed pixels of left and top neighboring blocks to generate residual data for Y-blocks interleaved with Cr- and Cb-blocks. At step406, the pipeline101generates reconstructed pixels by processing the residual data. Since there is no dependency among the Y-, Cr-, and Cb-blocks, the blocks are processed in consecutive cycles of the pipeline such that there are no stalls in the pipeline. At step408, a determination is made as to whether there are more blocks in the frame to be processed. If so, the method400returns to step402and repeats. Otherwise, the method400ends at step410, where the intra-estimated data from the frame is output.

Intra Mode Interleave (IMI) Scheme

In the LCI scheme, the non-dependency of color components is exploited to provide for efficient use of the pipeline. In the IMI scheme, non-dependency of various intra-modes is exploited. The IMI scheme for four intra-modes is shown in Table 2 below.

As shown in Table 2, encoding cycles C0-C8 are shown for the different pipeline stages of transform (T), quantize (Q), inverse quantize (IQ), inverse transform (IT), and pixel reconstruction (R). The intra-estimation process is performed for four different intra-modes. Since the different intra-modes do not depend on each other, the residual data for the different intra-modes is processed in consecutive cycles (C0-C3) of the pipeline without stalling.

FIG. 5is a flow diagram depicting a method500of processing YUV data according to an example. The method500performs the LCI scheme described above along with the IMI scheme. At step502, Y-, Cr-, and Cb-blocks are selected in the video frame. At step504, intra-estimation is performed for a first intra-mode based on reconstructed pixels to generate first residual data for Y-blocks interleaved with Cr- and Cb-blocks. At step506, intra-estimation is performed for a second intra-mode based on reconstructed pixels to generate second residual data for Y-blocks interleaved with Cr- and Cb-blocks. In some examples, steps504and506can be performed concurrently. At step508, the pipeline101generates next reconstructed pixels by processing the first residual data interleaved with the second residual data. Since there is no dependency among the intra-modes, the first and second residual data are processed in consecutive cycles of the pipeline such that there are no stalls in the pipeline. At step510, a determination is made as to whether there are more blocks in the frame to be processed. If so, the method500returns to step502and repeats. Otherwise, the method500ends at step512, where the intra-estimated data from the frame is output. While the method500is described with respect to two intra-modes, it is to be understood that the method500can be extended to perform intra-estimation using the IMI scheme for any number of intra-modes (e.g., four intra-modes as shown in Table 2).

Transform Size Interleave Scheme

FIG. 6is a flow diagram depicting a method600of processing YUV data according to an example. The method600performs a transform size interleave scheme. At step602, Y-, Cr-, and Cb-blocks are selected in the video frame. At step604, intra-prediction is performed based on reconstructed pixels of left and top neighboring blocks to generate residual data for the blocks using a plurality of transform sizes. At step606, the pipeline101generates reconstructed pixels by processing the residual data. Since there is no dependency among the blocks using different transform sizes, the blocks are processed in consecutive cycles of the pipeline such that there are no stalls in the pipeline. At step608, a determination is made as to whether there are more blocks in the frame to be processed. If so, the method600returns to step602and repeats. Otherwise, the method600ends at step610, where the intra-estimated data from the frame is output.

FIG. 7Ais a block diagram depicting a programmable device54that can be used to implement the intra-estimation techniques described herein according to an example. The programmable device54includes a plurality of programmable integrated circuits (ICs)1, e.g., programmable ICs1A,1B,1C, and1D. In an example, each programmable IC1is an IC die disposed on an interposer51. Each programmable IC1comprises a super logic region (SLR)53of the programmable device54, e.g., SLRs53A,53B,53C, and53D. The programmable ICs1are interconnected through conductors on the interposer51(referred to as super long lines (SLLs)52).

FIG. 7Bis a block diagram depicting a programmable IC1according to an example. The programmable IC1can be used to implement one of the programmable ICs1A-1D in the programmable device54. The programmable IC1includes programmable logic (PL)3(also referred to as a programmable fabric), configuration logic25, and configuration memory26. The programmable IC1can be coupled to external circuits, such as nonvolatile memory27, DRAM28, and other circuits29. The programmable logic3includes logic cells30, support circuits31, and programmable interconnect32. The logic cells30include circuits that can be configured to implement general logic functions of a plurality of inputs. The support circuits31include dedicated circuits, such as transceivers, input/output blocks, digital signal processors, memories, and the like. The logic cells and the support circuits31can be interconnected using the programmable interconnect32. Information for programming the logic cells30, for setting parameters of the support circuits31, and for programming the programmable interconnect32is stored in the configuration memory26by the configuration logic25. The configuration logic25can obtain the configuration data from the nonvolatile memory27or any other source (e.g., the DRAM28or from the other circuits29). In some examples, the programmable IC1includes a processing system (PS)2. The processing system2can include microprocessor(s), memory, support circuits, IO circuits, and the like. In some examples, the programmable IC1includes a network-on-chip (NOC)55and data processing engine (DPE) array56. The NOC55is configured to provide for communication between subsystems of the programmable IC1, such as between the PS2, the PL3, and the DPE array56. The DPE array56can include an array of DPE's configured to perform data processing, such as an array of vector processors.

FIG. 7Cis a block diagram depicting an SOC implementation of the programmable IC1according to an example. In the example, the programmable IC1includes the processing system2and the programmable logic3. The processing system2includes various processing units, such as a real-time processing unit (RPU)4, an application processing unit (APU)5, a graphics processing unit (GPU)6, a configuration and security unit (CSU)12, a platform management unit (PMU)122, and the like. The processing system2also includes various support circuits, such as on-chip memory (OCM)14, transceivers7, peripherals8, interconnect16, DMA circuit9, memory controller10, peripherals15, and multiplexed10(MIO) circuit13. The processing units and the support circuits are interconnected by the interconnect16. The PL3is also coupled to the interconnect16. The transceivers7are coupled to external pins24. The PL3is coupled to external pins23. The memory controller10is coupled to external pins22. The MIO13is coupled to external pins20. The PS2is generally coupled to external pins21. The APU5can include a CPU17, memory18, and support circuits19.

Referring to the PS2, each of the processing units includes one or more central processing units (CPUs) and associated circuits, such as memories, interrupt controllers, direct memory access (DMA) controllers, memory management units (MMUs), floating point units (FPUs), and the like. The interconnect16includes various switches, busses, communication links, and the like configured to interconnect the processing units, as well as interconnect the other components in the PS2to the processing units.

The OCM14includes one or more RAM modules, which can be distributed throughout the PS2. For example, the OCM14can include battery backed RAM (BBRAM), tightly coupled memory (TCM), and the like. The memory controller10can include a DRAM interface for accessing external DRAM. The peripherals8,15can include one or more components that provide an interface to the PS2. For example, the peripherals15can include a graphics processing unit (GPU), a display interface (e.g., DisplayPort, high-definition multimedia interface (HDMI) port, etc.), universal serial bus (USB) ports, Ethernet ports, universal asynchronous transceiver (UART) ports, serial peripheral interface (SPI) ports, general purpose10(GPIO) ports, serial advanced technology attachment (SATA) ports, PCIe ports, and the like. The peripherals15can be coupled to the MIO13. The peripherals8can be coupled to the transceivers7. The transceivers7can include serializer/deserializer (SERDES) circuits, multi-gigabit transceivers (MGTs), and the like.

FIG. 7Dillustrates a field programmable gate array (FPGA) implementation of the programmable IC1that includes the PL3. The PL3shown inFIG. 6Dcan be used in any example of the programmable devices described herein. The PL3includes a large number of different programmable tiles including transceivers37, configurable logic blocks (“CLBs”)33, random access memory blocks (“BRAMs”)34, input/output blocks (“IOBs”)36, configuration and clocking logic (“CONFIG/CLOCKS”)42, digital signal processing blocks (“DSPs”)35, specialized input/output blocks (“I/O”)41(e.g., configuration ports and clock ports), and other programmable logic39such as digital clock managers, analog-to-digital converters, system monitoring logic, and so forth. The PL3can also include PCIe interfaces40, analog-to-digital converters (ADC)38, and the like.

In some PLs, each programmable tile can include at least one programmable interconnect element (“INT”)43having connections to input and output terminals48of a programmable logic element within the same tile, as shown by examples included at the top ofFIG. 6D. Each programmable interconnect element43can also include connections to interconnect segments49of adjacent programmable interconnect element(s) in the same tile or other tile(s). Each programmable interconnect element43can also include connections to interconnect segments50of general routing resources between logic blocks (not shown). The general routing resources can include routing channels between logic blocks (not shown) comprising tracks of interconnect segments (e.g., interconnect segments50) and switch blocks (not shown) for connecting interconnect segments. The interconnect segments of the general routing resources (e.g., interconnect segments50) can span one or more logic blocks. The programmable interconnect elements43taken together with the general routing resources implement a programmable interconnect structure (“programmable interconnect”) for the illustrated PL.

In an example implementation, a CLB33can include a configurable logic element (“CLE”)44that can be programmed to implement user logic plus a single programmable interconnect element (“INT”)43. A BRAM34can include a BRAM logic element (“BRL”)45in addition to one or more programmable interconnect elements. Typically, the number of interconnect elements included in a tile depends on the height of the tile. In the pictured example, a BRAM tile has the same height as five CLBs, but other numbers (e.g., four) can also be used. A DSP tile35can include a DSP logic element (“DSPL”)46in addition to an appropriate number of programmable interconnect elements. An10B36can include, for example, two instances of an input/output logic element (“IOL”)47in addition to one instance of the programmable interconnect element43. As will be clear to those of skill in the art, the actual I/O pads connected, for example, to the I/O logic element47typically are not confined to the area of the input/output logic element47.

Some PLs utilizing the architecture illustrated inFIG. 7Dinclude additional logic blocks that disrupt the regular columnar structure making up a large part of the PL. The additional logic blocks can be programmable blocks and/or dedicated logic. Note thatFIG. 7Dis intended to illustrate only an exemplary PL architecture. For example, the numbers of logic blocks in a row, the relative width of the rows, the number and order of rows, the types of logic blocks included in the rows, the relative sizes of the logic blocks, and the interconnect/logic implementations included at the top ofFIG. 7Dare purely exemplary. For example, in an actual PL more than one adjacent row of CLBs is typically included wherever the CLBs appear, to facilitate the efficient implementation of user logic, but the number of adjacent CLB rows varies with the overall size of the PL.