Source: https://patents.google.com/patent/US8369411B2/en
Timestamp: 2018-11-18 19:25:34
Document Index: 738608209

Matched Legal Cases: ['art 10', 'Application No. 2', 'Application No. 2', 'Application No. 2', 'Application No. 2', 'Application No. 2', 'Application No. 2', 'Application No. 2', 'Application No. 08', 'Application No. 08', 'Application No. 08', 'Application No. 08', 'Application No. 08', 'Application No. 08', 'Application No. 08']

US8369411B2 - Intra-macroblock video processing - Google Patents
Intra-macroblock video processing Download PDF
US8369411B2
US8369411B2 US11693473 US69347307A US8369411B2 US 8369411 B2 US8369411 B2 US 8369411B2 US 11693473 US11693473 US 11693473 US 69347307 A US69347307 A US 69347307A US 8369411 B2 US8369411 B2 US 8369411B2
US11693473
US20080240253A1 (en )
Au James
A system for processing sub-blocks of a macroblock of a video frame. In an example embodiment, the system includes a first module that is adapted to process each sub-block of the macroblock, wherein each sub-block is associated with a predetermined position in a first sequence. The processing of certain sub-blocks in the first sequence requires results of processing of one or more previously processed sub-blocks in the first sequence. A controller selectively enables the first module to process each sub-block of a second sequence that is altered from the first sequence so that the first module implements parallel or pipelined processing of certain sub-blocks of the macroblock. In a more specific embodiment each sub-block in the first sequence of sub-blocks is consecutively numbered 0-15 according to H.264 standards.
This disclosure relates generally to image or video processing and more specifically relates to processing pixels and associated sub-blocks within a macroblock of an image or video frame. Video processing systems are employed in various demanding applications, including high-definition television, missile guidance systems, and Internet video streaming. Such applications often demand compact cost-effective systems for encoding, transmitting, and decoding high quality compressed video relatively accurately, quickly, and efficiently.
Standards such as H.264 define specific ways of doing video encoding and decoding. H.264 is a digital video codec standard written by the Joint Video Team (JVT) comprising the International Telecommunication Union (ITU)—Telecommunication Standardization Sector (T) and the International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) Moving Picture Experts Group (MPEG). The H.264 standard is also called, the ISO/IEC MPEG-4 Part 10 standard or the ITU-T H.264 standard.
In an example H.264 system, pixels of a video image are logically grouped into 16×16 blocks of pixels called macroblocks. Each macroblock is grouped into sixteen 4×4 blocks of pixels, called sub-blocks. The image is partitioned into groups of contiguous macroblocks in coding order, called slices, each containing several macroblocks. When this example H.264 system processes a slice, information from one or more previously processed macroblocks is typically required before a subsequent macroblock is processed. Similarly, when processing sub-blocks within a macroblock, information from one or more previously processed sub-blocks is typically required before a subsequent sub-block is processed. Improvements in the speed and efficiency of implementing H.264 functions such as processing macroblocks, sub-blocks, or other data can lead to a better performing, lower cost, or otherwise more desirable system or product.
FIG. 1 illustrates an example video-encoding system.
FIG. 3 a is a more detailed diagram illustrating example components of the intra prediction module of the encoding module of FIG. 2 b.
FIG. 3 b is a more detailed diagram illustrating example components of the intra macroblock processor included in the encoding modules of FIGS. 2 a and 2 b and used for parallel processing of video macroblock sub-blocks.
FIG. 4 is a diagram illustrating an alternative intra macroblock processor that employs parallel engines and that is usable with the inter/intra processing module of FIGS. 3 a and 3 b.
FIG. 5 is a diagram of art example frame illustrating a macroblock, constituent sub-blocks, and neighboring pixels of adjacent sub-blocks used to process an example sub-block.
FIG. 6 is a diagram illustrating example labeling of certain rows of pixels of sub-blocks of the macroblock of FIG. 5 that are employed by the intra macroblock processors of FIGS. 3 b and 4 to process other sub-blocks of the macroblock.
FIG. 9 is a second timing diagram illustrating a second example of when certain sub-blocks of an example macroblock of FIGS. 5-7 are processed by the intra macroblock processors of FIGS. 2-4.
FIG. 10 is a flow diagram of an example method suitable for use with the intra macroblock processors of FIGS. 3 b and 4 and accompanying inter/intra processing module FIGS. 1-2.
A particular embodiment of the invention is discussed with respect to an implementation of an H.264 system. However, it should be apparent that aspects of the invention may be used to implement different standards or may be used in association in any other suitable data processing application. Nothing in the H.264 specifications should be construed as limiting the scope of the invention unless so noted. Many of the same terms that appear in the H.264 specification are used in the present specification but these terms are intended to have different meanings. For example, a macroblock may be any grouping of pixels in a frame of data. A sub-block may be any grouping of pixels within a macroblock. A frame may be any set of data that includes plural pixels of information, where a pixel may be a value associated with a data point. A video frame or an image frame may be any collection of data points or pixels that are associated with an image or something to be displayed or viewed. The terms video frame and image frame are employed interchangeably herein. An image may be any thing or representation of a thing that can be viewed. A slice of an image frame may be any grouping of macroblocks for processing purposes.
An example of a macroblock is the macroblock discussed in the H.264 Main Profile (MP) and High Profile (HiP) standards, which includes a 16×16 group of pixels. An example of a sub-block is a 4×4 grouping of pixels. An example of documentation that provides details of the H.264 standard is ISO/IEC 14496-10 (ITU-T H.264), International Standard (2005), Advanced video coding for generic audiovisual services; which is hereby incorporated by reference as if set forth in fell in this specification for all purposes. For the purposes of generating a prediction frame, macroblocks within a slice are predicted based on previously predicted macroblocks so that values associated with one macroblock within the slice depend on values associated with one or more other macro-blocks within the slice. The arrangement and size of a group of data can vary, as desired. For example, in other applications the groups can be of any dimensions and need not necessarily be square or even rectangular groupings.
An example embodiment of a system for processing sub-blocks of a macroblock includes a first module that is adapted to process each sub-block of the macroblock. Each sub-block is associated with a predetermined position in a first sequence. The processing of certain sub-blocks in the first sequence requires results of processing of one or more previously processed sub-blocks in the first sequence. A controller selectively enables the first module to process each sub-block of a second sequence that is altered from the first sequence so that the first module implements parallel processing of certain sub-blocks of the macroblock.
Use of this embodiment to facilitate parallel processing of sub-blocks of a macroblock may improve processing speeds, enabling video frames to be processed in a single slice, which may greatly increase video quality. This is unlike certain conventional video processing systems that must partition each video frame into multiple slices due to use of relatively inefficient methods, serial processing, and hardware speed and performance limitations.
In an H.264-compliant embodiment, each sub-block in the first sequence of sub-blocks is consecutively numbered 0-15 according to H.264 standards. The controller facilitates feeding sub-blocks numbered 0, 1, 2, 3, 8, 9, 10, 11, in order, or sub-blocks numbered 0, 1, 4, 5, 8, 9, 12, 13, in order, to the first module. Furthermore, the controller feeds the sub-blocks 4, 5, 6, 7, 12, 13, 14, 15, in order, or sub-blocks 2, 3, 6, 7, 10, 11, 14, 15, in order, to the first module. Note that such sequence orders are merely illustrative and that other arrangements are possible.
The first module includes a first mechanism for processing sub-blocks of certain pairs of sub-blocks in a pipelined fashion. The certain pairs of sub-blocks include sub-blocks numbered 2 and 4; 3 and 5; 8 and 6; 9 and 7; 10 and 12; and 11 and 13. The first module includes a second mechanism for processing sub-blocks of certain pairs of sub-blocks in parallel. Note that the order of individual sub-blocks within each pair of sub-blocks indicated above may be changed without departing from the scope of the present teachings. Parallelism in processing can be achieved by any manner known in the art such as by using multiple processors, pipelined stages, etc.
In an illustrative embodiment, the second mechanism includes a first engine and a second engine. The second engine is fed sub-blocks for processing in the following order after sub-block numbered 1 is processed by the first engine: 4, 5, 6, 7, 12, 13, 14, 15. Alternatively, second engine is fed sub-blocks in the following order after sub-block numbered 1 is processed by the first engine: 2, 3, 6, 7, 10, 11, 14, 15. In this example embodiment, the controller includes a demultiplexer that is adapted to feed requisite processing information pertaining to one or more sub-blocks to the first module in a desired sequence or in parallel in response to a predetermined control signal.
For the purposes of the present discussion, a sub-block is said to be fed to a processor if information pertaining to the sub-block is delivered to the processor to enable processing of the sub-block by the processor.
Processes implemented via the parallel engines may include implementing intra prediction for a sub-block. For the purposes of the present discussion, intra prediction may be any process wherein one or more pixels in an image frame are predicted or estimated based on information associated with the image frame. Intra prediction may include the process wherein values for a predicted sub-block are predicted or ascertained from one or more other sub-blocks within a macroblock of a frame that includes or is to include the macroblock.
Hence, unlike a conventional system that must process sub-blocks of a macroblock serially via a single engine, certain embodiments discussed herein enable use of multiple engines to process sub-blocks of a macroblock within an image frame. This may result in improved encoder performance, image quality, and so on.
In operation, the front end 22 and formats input from the video receiver 14, which is accessible by the display module 24 and the encoding module 20. The controller 26 provides control input to the display module, the front end 22, and the encoding module 20 to facilitate controlling the operation of the encoder. Input video may be displayed via the display monitor 18 after the video data is formatted and prepared as needed via the display module 24.
The encoding module 20 may be implemented to perform processing which is in accordance with H.264 standards. The front end 22, display module 24, and controller 26 may be implemented via various technologies, including, but not limited to Field Programmable Gate Array (FPGA) and Digital Signal Processor (DSP) technologies.
FIG. 2 a is a more detailed diagram illustrating example components 30-52 of the encoding module 20 included in the video-encoding system 10 of FIG. 1. The encoding module 20 includes an input-frame memory 30 for receiving and selectively storing video frames from the front end 22 of FIG. 1. The output of the input-frame memory 30 is input to a subtracter 32 and to an inter/intra processing module 50. The subtracter 32 also receives input from the inter/intra processing module 50 and provides output to a transformation module 36. An output of the transformation module 36 is coupled to a quantization module 38, also called a quantizer, an output of which is coupled to inputs of a dequantization module 40 and a entropy encoder 34. In the present example embodiment, the entropy encoder 34 implements Context Adaptive Binary Arithmetic Coding (CABAC), and consequently, is also called a CABAC module. The CABAC module 34 provides feedback to a rate control module 52, which provides a rate-control signal to the quantizer 38.
The dequantizer 40 is further coupled to an inverse transform module 42, an output of which is coupled to an input of an adder 44. An output of the adder 44 is coupled to an input of a deblocking filter 46. An output of the deblocking filter 46 is coupled to as input of a reference frame memory 48. An output of the reference frame memory 48 is coupled to an input of the inter/intra processing module 50. An output of the inter/intra processing module 50 is coupled to an input of the subtracter 32 and to an input of the adder 44. The inter/intra processing module 50 further includes a macroblock processor 54 for processing macroblocks within a slice of a video image frame, and an intra macroblock processor 60 for processing sub-blocks of macroblocks processed by the macroblock processor 54. The intra macroblock processor 60 may act as a nested processing loop within a loop governed by the macroblock processor 54 as discussed more fully below.
In either mode, a predicted macroblock output by the inter/intra processing module 50 is subtracted from the corresponding macroblock in the input frame that is output from the input-frame memory 30. The resulting macroblock is called a residual or difference macroblock. The difference macroblock is then transformed from a time domain to a frequency domain, such as via a block transform, by the transformation module 36. The resulting transformed macroblock is quantized by the quantizer 38; dequantized by the dequantizer 40; and then transformed back to the time domain via the inverse transform module 42. The resulting residual macroblock is added to a corresponding macroblock from the prediction frame that is output from the inter/intra processing module 50 before the resulting added macroblock, called a reconstructed macroblock, is input back to the inter/intra processing module 50 as a reconstructed macroblock.
The output of the quantizer 38 represents a set of quantized transform coefficients, also called residuals. These coefficients are then entropy encoded via the CABAC 34. Entropy encoded coefficients and information required to decode a macroblock, such as prediction mode, quantizer step size, and so on, are output from the CABAC 34 via one or more bitstreams. The number of bits employed by the CABAC during a given operation may be fed back to the rate control module 52 to facilitate controlling the amount of information removed by the quantizer, thereby controlling the final bit rate.
Hence, the rate control module 52 generates rate control signals to control the quantizer 38 and accompanying quantization parameters based on bit-production feedback from a previous frame, picture complexity, current bit buffer levels. The inter/intra processing module 50 may facilitate performing prediction through intra search mode-decision operations; may implement a mode decision function that selects a best prediction mode for each macroblock to be processed; and may perform prediction via intra compensation to form sample intensity predictions. The terms intra compensation and intra prediction may be employed interchangeably herein. Intra compensation involves predicting a macroblock based on information from the current frame in which the macroblock is a part.
The subtracter 32 outputs residuals representing differences between input data samples from the input-frame memory 30 and prediction samples from the inter/intra processing module 50. The transform module 36 converts residuals to the frequency domain. The quantizer 38 quantizes frequency coefficients, effectively discarding certain information to reduce entropy in the residuals. The dequantizer 40 and inverse transform module 42 are adapted to reconstruct transformed and quantized residuals through dequantization and inverse transform processes. The adder 44 facilitates adding reconstructed residuals output from the inverse transform module 42 to prediction samples output from the inter/intra processing module 50 to reconstruct decoded samples for a given macroblock. The deblocking filter 46 is adapted to remove blocking artifacts from decoded samples output from the adder 44. The CABAC 34 is adapted to implement entropy coding in accordance with H.264 main/high profile. The CABAC 34 codes macroblock modes, prediction information, and residuals into H.264-compliant bitstreams.
For illustrative purposes, the controller 26 is shown communicating with the inter/intra processing module 50. The controller 26 may communicate with more modules, different modules, or no modules in FIG. 2 without departing from the scope of the present teachings. Furthermore, the rate-control module 52 and the controller 26 may be implemented in a common control module without departing from the scope of the present teachings.
FIG. 2 b is more detailed diagram illustrating example components of a second embodiment of an encoding module 20′ that is usable with the video-encoding system 10 of FIG. 1. The encoding module 20′ of FIG. 2 b may be substituted for the encoding module 20 of FIGS. 1 and 2 a.
The alternative encoding module 20′ includes the input-frame memory 30, which may receive input video data from the encoder front end 22 of FIG. 1 and provide input video data to an intra prediction module 54′ and to an inter prediction module 57. The reference frame memory 48 receives filtered reconstructed pixel data from the deblocking filter 46. The filtered reconstructed pixel data represents reference video data corresponding to reference frames. The reference frames are provided to the inter prediction module 57 for use in inter prediction calculations.
The main mode decision module 59 includes instructions for selecting a preferred prediction method based on output from the intra prediction module 54′ and the inter prediction module 57. For example, in certain situations, such as when a video changes scenes or otherwise changes suddenly, output from the intra prediction module 54′ may be chosen as the output of the main mode decision module. In other situations, such as when successive frames of a video are very similar, output from the inter prediction module 57 may be chosen as the output of the main mode decision module 59. Generally, intra prediction is often performed when encoding so-called I-frames, which are defined in accordance with H.264 standards. Inter prediction is often performed when encoding P-frames, which are also defined in accordance with H.264 standards. When encoding P-frames. It is possible to encode a frame using a combination of inter prediction mid intra prediction. Exact details of methods implemented by the main mode decision module 59 are application specific. Those skilled in the art with access to the present teachings may readily implement appropriate methods to meet the needs of a given application, without undue experimentation.
The intra prediction module 54′ and the inter prediction module 57 include TQR loops as discussed more fully below. The TQR loops facilitate reconstructing pixels and providing residual coefficients to the main mode decision module 59.
FIG. 3 a is a more detailed diagram illustrating example components of the intra prediction module 54′ of the encoding module 20′ of FIG. 2 b. The intra prediction module 54′ includes an intra 4×4 search engine 60′, also called an intra macroblock processor. For illustrative purposes, the intra macroblock processor 60′ of FIG. 3 a is similar to the intra macroblock processor 60 of FIG. 2 a, with the exception that the macroblock processor 60′ of FIG. 3 a includes the TQR loop 66. However, a substantially different macroblock processor may be employed in FIG. 3 a with out departing from the scope of the present teachings.
The intra prediction, module 54′ further includes an optional intra 8×8 search engine 69, and an intra 16×16 search engine 69. The search engines 60′, 69, 73 include individual TQR loops 66, 71, 75. The search engines 60′, 69, 73 receive input macroblock data from an input frame memory, such as the memory 30 of FIG. 2 b, and also receive reconstructed pixel information from the output of the main mode decision module 59 of FIG. 2 b. The search engines 60′, 69, 73 then perform intra 4×4 prediction, intra 8×8 prediction, and 16×16 prediction, respectively, to provide corresponding coefficients, SAD values, and reconstructed pixels to an intra mode decision module 67. The intra mode decision module 67, then performs implements instructions for selecting a preferred intra search method. In the present specific embodiment, intra mode decision module 67 is adapted to select the output of the search engines 60′, 69, 73 that employs the fewest number of bits to encode a given macroblock.
The search engines 60′, 69, 73 are also called sub-engines, since they are within the intra prediction module 54′. The output from the search engine 60′, 69, 73 that is selected by the intra mode decision module 67 is forwarded to main mode decision 59 of FIG. 2 b.
The intra 4×4 search engine 60′ is adapted to process sub-blocks of a macroblock in parallel, such as in a pipelined fashion, as discussed more fully below. The intra 4×4 search engine 60′ is further adapted to strategically control the flow of information pertaining to processed sub-blocks of each macroblock, to achieve various efficiencies, such as obviating the need for a large memory, and to facilitate parallel processing, as discussed more fully below.**
FIG. 3 b is a more detailed diagram illustrating example components of the intra macroblock processor 60′ of FIG. 3 a, which is included in the intra prediction module 54′ of the encoding module 20′ of FIG. 2 b and used for parallel processing of video macroblock sub-blocks. The intra macroblock processor 60′ may be employed as the intra macroblock processor 60 of FIG. 2 a by moving the TQR loop 66 outside of the macroblock processor 60′ of FIG. 3 b, as shown in FIG. 2 a.
For the purposes of the present discussion, the terms macroblock sub-block, sub-block, and intra block are employed interchangeably. A macroblock sub-block may be any grouping of pixels comprising a portion of a macroblock. An example of a sub-block is a 4×4 block of adjacent pixels in a macroblock.
The example intra prediction module 54′ of FIG. 2 b includes the intra macroblock processor 60′ of FIG. 3 a. The intra macroblock processor 60′ includes a intra prediction module 62, which communicates with a neighbor sub-block memory 64 that stores previously processed sub-blocks and information associated therewith, such as processing modes used to process each of the sub-blocks. The neighbor sub-block memory 64 is coupled between an output of the TQR loop 66 and an input of the intra prediction module 62. The macroblock processor 54 includes a neighbor macroblock memory 56, which is also coupled between an output of the TQR loop 66 and an input of the intra prediction module 62.
The intra prediction module 62 receives current macroblock information pertaining to a current macroblock to be processed from input-frame memory 30 of FIG. 4. In addition, the intra prediction module 62 receives neighbor macroblock data (e.g., ntop, nleft, etc.) from the TQR loop 66 and receives processed pixel data representing reconstructed pixels pertaining to requisite sub-blocks of the current macroblock, from the neighbor sub-block memory 64.
In operation, intra prediction module 62 runs one or more instructions for implementing pixel prediction and processing-mode decisions. The intra prediction module 62 may include an intra 4×4 neighbor component to determine neighboring pixels for each 4×4 sub-block to be used in the prediction of subsequent sub-blocks, and an intra 4×4 search component to perform prediction calculations and comparisons for each sub-block. The TQR loop 66 facilitates reconstructing pixels within a macroblock and providing reconstructed pixels as feedback to the intra prediction module 62. Data pertaining to the reconstructed pixels, which are also called predicted pixels or intra predicted pixels, are stored in the neighbor sub-block memory 64 for use by the pixel reconstruction module 62 to process subsequent sub-blocks. Pixel reconstruction, as discussed with respect to the present example, includes implementing intra prediction calculations in compliance with H.264 output standards.
Hence, the neighbor sub-block memory 64 stores results of the processing of certain sub-block pixels for use in processing subsequent sub-blocks. Similarly, the neighbor macroblock memory 56 stores results of the processing of certain macroblocks for use in processing subsequent macroblocks and sub-blocks therein. Reconstructed pixels associated with neighboring macroblocks may come from intra (4×4, 8×8, or 16×16) or inter prediction operations. Reconstructed pixels associated with neighboring sub-blocks within a macroblock come from intra prediction operations, such as those performed by the intra prediction module 62. The sub-block memory 64 and the neighbor macroblock memory 56, which are shown as separate memories for illustrative purposes, may be implemented a single memory without departing from the scope of the present teachings.
The loop formed between the intra prediction module 62 and accompanying neighbor sub-block memory 64 may be considered as a nested loop formed within a macroblock-processing loop comprising the macroblock. TQR loop 66, the neighbor macroblock memory 56, and the intra macroblock processor 60′.
The reconstruction module 70 is configured to accept back-to-back sub-blocks for processing in a pipelined fashion. For the purposes of the present discussion, two sub-blocks are said to be processed in a pipelined fashion by a processor, engine, or computer, if at any given time during processing, the processor, engine, or computer works on processing both sub-blocks. The pipelined processing may be staggered so that the processor, engine, or computer completes processing of one sub-block before completing processing of the other sub-block. Two sub-blocks are said to be processed in parallel if at least a portion of each of the two sub-blocks are being processed simultaneously by one or more processors, engines, or computers. Hence, two sub-blocks or macroblocks that are being processed in a pipelined fashion are also being processed in parallel but not necessarily vice versa. Parallel processing may occur via separate processors, engines, or computers that are working or processing simultaneously or via one or more pipelined processors.
FIG. 4 is a diagram illustrating an alternative intra macroblock processor 70 that employs parallel engines 80, 72 and that is usable with the intra macroblock processors 60, 60′ of FIGS. 2 a, 3 a, and 3 b. The alternative intra macroblock processor 70 may be inserted in place of the intra macroblock processors 60, 60′ in FIGS. 2 a and 3 b.
The intra microblock processor 70 of FIG. 4 includes a first engine 80 and a second engine 72, which communicate with a shared neighbor sub-block memory 74. An output of the shared sub-block memory 74 is input to a first demultiplexer (DEMUX) 76. Outputs of the engines 80, 72 are coupled to inputs of an output multiplexer (MUX) 78, an output of which is coupled to the macroblock processor 54, the inter/intra processing module 50, and/or the macroblock TQR loop 66 of FIG. 2.
The controller 26 controls the delivery of requisite vectors, motion costs, modes, and reconstructed pixels from the input of the DEMUX 76 to each engine 80, 72 to control the processing of a given sub-block within a macroblock by a given engine 80, 72. For example, the controller 26 includes one or more instructions for determining when the processing of a given macroblock sub-block is complete; when certain requisite previous sub-blocks have been processed to enable processing of a subsequent sub-block; and for delivering requisite reconstructed pixel information to the appropriate engine 80, 72. While in the present embodiment, the controller 26 is shown implemented separately from the engines 80, 72, similar sequencing functionality may be incorporated within the engines 80, 72 or elsewhere without departing from the scope of the present teachings. Furthermore, the controller 26 shown in FIG. 4 may be implemented via the same controller 26 as shown in FIGS. 1-2 or via a different controller.
While in tire present example embodiment, sub-blocks of macroblocks are processed in parallel, macro blocks may also be processed in parallel in accordance with the teachings of U.S. patent application Ser. No. 11/693,506, entitled PARALLEL OR PIPELINED MACROBLOCK PROCESSING, which is assigned to the assignee of the present invention and which is hereby incorporated by reference herein. Those skilled in the art with access to the present teachings may readily implement a video encoder employing parallel processing of macroblocks and macroblock sub-blocks in accordance with the present teachings, without undue experimentation.
FIG. 5 is a diagram of an example frame 100 illustrating a macroblock 102 wife sixteen constituent Sub-Blocks (SBs 0→15) numbered 0-15 in accordance with H.264 standards. FIG. 5 further illustrates neighboring pixels (pixels A→M) of adjacent sub-blocks (SBs 8, 2, 3, 6) used to process an example sub-block (SB9). The macroblock 102 represents a 16×16 square of pixels, which are partitioned into sixteen squares of sub-blocks, where each sub-block includes a 4×4 square of sixteen pixels (labeled a→p).
With reference to FIGS. 3-5, in the present example, pixels A→M of SBs 2, 3, 6, 8 are employed by the intra macroblock processors 60, 60′, 70 to predict pixels a→p of SB 9. For the purposes of the present discussion, SB9 is said to be processed when the constituent pixels a→p are predicted. Predicted pixels are also said to be reconstructed.
Similarly, other sub-blocks of the macroblock 102 use previously predicted, i.e., processed pixels of sub-blocks to the left, upper left, above, and to the upper right of the sub-blocks. For example, the processing of SB3 requires previously processed pixels from SBs 0, 2, 1, and 4. As another example, the processing of SB0 uses previously processed pixels from neighboring macroblocks. In particular, SB0 would use pixels from the tenth and eleventh sub-block of a macroblock (not shown) above the macroblock 102; would use pixels from the fifteenth sub-block of the macroblock to the upper left of the macroblock 102; would use pixels from fifth sub-block of the macroblock to the left of the macroblock 102. Pixel information from neighboring macroblocks is retrieved or delivered to the intra macroblock processors 60, 60′, 70 from the neighbor macroblock memory 56 of FIG. 3 b.
Note that the successful processing of SB9 according to certain H.264 standards depends upon the results of the processing of SBs 2, 3, 6, 8. Conventionally, sub-blocks are processed serially due to such dependencies on previously processed sub-blocks. However, in certain example embodiments discussed herein, sub-blocks are selectively processed in parallel, such as via the pipelined processor 60′ of FIGS. 3 a and 3 b or the parallel engines 80, 72 of FIG. 4, by taking advantage of the ordering of sub-blocks within a macroblock, such as the macroblock 102, as discussed more fully below.
Sub-block used to process the
(SB) indicated sub-block
3 0, 1, 2, 4
9 2, 3, 6, 8
11 8, 9, 10, 12
12 3, 6, 7, 9
13 6, 7, 12
14 9, 11, 12, 13
15 12, 13, 14
FIG. 6 is a diagram illustrating example labeling of certain rows of pixels of sub-blocks (SBs 0→15) of the macroblock 102 of the frame 100 of FIG. 5 that are employed by the intra macroblock processors 60, 60′, 70 of FIGS. 2-4 to process other sub-blocks of the macroblock 102. Pixels in a row of pixels directly above SB0, which are positioned in the macroblock above the macroblock 100, are labeled ntop0 pixels. Pixels in a row of pixels directly above SB1 are labeled ntop1. Pixels in a row of pixels directly above SB4 are labeled ntop2. Similarly, pixels in a row of pixels directly above SB5 are labeled ntop3. In addition, pixels in a bottom row of a sub-block of a macroblock diagonally to the upper right of the macroblock 100 are labeled ntop4 pixels. As an example, with reference to FIG. 5, the ntop0 pixels and the ntop1 pixels correspond to A→D and E→H pixels for SB0, respectively.
Rows of pixels within the macroblock 102 that are used for the processing of other sub-blocks are labeled bottom0→bottom15, as they are positioned in the bottom portions of SBs 0→15, respectively. As an example, with reference to FIGS. 5 and 6, bottom3 and bottom6 pixels represent pixels A→H used by SB9.
FIG. 7 is a diagram illustrating example labeling of certain columns of pixels of SBs 0→15 of the macroblock 102 of FIGS. 5 and 6 that are employed by the intra macroblock processors 60, 60′, 70 of FIGS. 2-4 to process other sub-blocks of the macroblock 102. Pixels in a column of pixels directly to the left of SB10, which are positioned in a macroblock directly to the left the macroblock 102, are labeled nleft0 pixels. Pixels in a column of pixels directly to the left of SB8 are labeled nleft1. Pixels directly to the left of SB2 are labeled nleft2. Similarly, pixels in a column of pixels directly to the left of SB0 are labeled nleft3.
Columns of pixels within the macroblock 102 that are used for the processing of other sub-blocks are labeled right0→right15, as they are positioned in SBs 0→15, respectively. As an example, with reference to FIGS. 3-7, right8 pixels and the bottom pixel of the right2 pixels are used by the intra macroblock processors 60, 60′, 70 of FIGS. 2-4, in addition to the bottom3 and bottom6 pixels of FIG. 6, to process SB9. The right8 pixels correspond to pixels I→L of FIG. 5. The bottom pixel of the right2 pixels corresponds to pixel M of FIG. 5.
Sub- Pixels Pixels Pixels
Block A, B, C, D E, F, G, H I, J, K, L Pixel M
0 ntop0 ntop1 nleft3 nc
1 ntop1 ntop2 right0 ntop0
2 bottom0 bottom1 nleft2 nleft3
3 bottom1 bottom4 right2 bottom0
4 ntop2 ntop3 right1 ntop1
5 ntop3 ntop4 right4 ntop2
6 bottom4 bottom5 right3 bottom1
7 bottom5 right pixel of right6 bottom4
8 bottom2 bottom3 nleft1 nleft2
9 bottom3 bottom6 right8 bottom2
10 bottom8 bottom9 nleft0 nleft1
11 bottom9 bottom12 right10 bottom8
12 bottom6 bottom7 right9 bottom3
13 bottom7 right pixel of right12 bottom6
14 bottom12 bottom13 right11 bottom9
15 bottom13 right pixel of right14 bottom12
FIG. 8 is a first example timing diagram 110 illustrating a first example of when certain sub-blocks (SBs 0→15) of the example macroblock 102 of FIGS. 5-7 are processed by the intra macroblock processors 60, 60′, 70 of FIGS. 2-4. Illustrative processing intervals for the SBs 0→15 are plotted along a horizontal time axis 112. Note that in practice, the processing intervals shown in FIG. 8 may be staggered and processed via a pipelined engine instead of the two engines 80, 72 shown in FIG. 8.
With reference to FIGS. 3-7 and Tables 1 and 2 above, in one operative scenario, the first engine 90 processes SBs 0, 1, 4, 5, 8, 9, 12, and 13 in sequence. The second engine 72 begins processing SBs 2, 3, 6, 7, 10, 11, 14, and 15 in sequence after the first engine 80 completes processing SB0 so that when the first engine 80 is processing SB4, the second engine 72 is processing SB2. The processing is staggered so that the following pairs of sub-blocks are processed in parallel: SB4 and SB2, SB5 and SB3, SB8 and SB6, SB9 and SB7, SB12 and SB10, SB13 and SB11. Note that if sub-blocks are processed in the sequences given in FIG. 8, by the time any given sub-block is processed, the pixels of the other sub-blocks used to process the given sub-block have already been processed. Hence, by capitalizing on the sub-block processing dependencies and the ordering of sub-blocks within a macroblock, efficient parallel processing of sub-blocks is achieved.
In a second operative scenario, the pipelined processor 60′ of FIGS. 3 a and 3 b, also called a pipelined engine, processes SBs 0 and 1 before SB4 is processed. SB2 is fed into the pipelined processor 60′ directly behind SB4 so that the pipelined processor 60′ begins processing SB2 before the processing of SB4 is complete. Alternatively, the pipelined processor 60′ begins processing SB2 just before it begins processing SB4. Either way, the processing of SB2 and 4 by the pipelined processor 60′ overlap so that parallel processing is achieved. Similarly, the other pairs of sub-blocks (SB5 and SB3, SB8 and SB6, SB9 and SB7, SB12 and SB10, SB13 and SB11) are processed by the pipelined processor 60′ in a pipelined fashion.
FIG. 9 is a second timing diagram 120 illustrating a second example of when certain sub-blocks (SBs 0→15) of the macroblock 102 of FIGS. 5-7 are processed by the intra macroblock processors 60, 60′, 70 of FIGS. 2-4. Processing intervals for the SBs 0→15 are plotted along the horizontal time axis 112. The timing diagram 120 of FIG. 9 is similar to the timing diagram of FIG. 8 with the exception that the first engine 80 of the parallel intra macroblock processor 70 of FIG. 4 processes sub-blocks in the following sequence: SB0, SB1, SB2, SB3, SB5, SB9, SB10, SB11. In addition, the second engine 72 of the parallel intra macroblock processor 70 of FIG. 4 processes sub-blocks in the following sequence: SB4, SB5, SB6, SB7, SB12, SB13, SB14, SB15 after the first engine has processed SBs 0 and 1. Accordingly, the following pairs of sub-blocks are processed in parallel by the parallel processor 70 or the pipelined processor 60′ of FIGS. 3 a and 3 b: SB2 and SB4, SB3 and SB5, SB6 and SB5, SB7 and SB9, SB10 and SB12, SB11 and SB13.
FIG. 10 is a flow diagram of an example method 150 suitable for use with, the intra macroblock processors 60, 60′, 70 of FIGS. 3-4 and accompanying inter/intra processing module 50 of FIG. 1-2.
In a first step 152, the method 150 includes receiving information, such as neighboring pixel information, to process a first sub-block. A second step 154 includes receiving information to process a second sub-block.
A third, step 154 determines whether sufficient information associated with one or more previously processed sub-blocks is simultaneously available for processing the first sub-block and the second sub-block. If sufficient information is not available, steps 150 and 152 are performed as needed. When sufficient information is available a fourth step 158 is performed.
The fourth step 158 includes processing a first portion of the second sub-block simultaneously with a second portion of tire first sub-block. Subsequently, the method 150 completes.
In certain implementations, the first portion of the second sub-block may include the entire second sub-block, and the second portion of the first sub-block may include the entire first sub-block, so that the entire first sub-block and the entire second sub-block are processed in parallel. Such implementations may involve use of the parallel engines 80, 72 of FIG. 4. Alternatively, pairs of macroblocks are processed slightly staggered, so that the processing of the first sub-block may complete before the processing of the second sub-block. Such implementations may involve use of the pipelined engine 60′ of FIGS. 3 a and 3 b or the parallel engines 80, 72 of FIG. 4.
Although embodiments of the invention are discussed primarily with respect to an H.264-compliant encoder, embodiments of the present invention may be adapted to any video encoder wherein parallel engines or a pipelined engine may be useful to process macroblocks and/or sub-blocks of the macroblocks in a frame. Furthermore, any acceptable architecture, topology, protocols, or other network and digital processing features can be employed. In general, certain modules and components discussed herein can be implemented in hardware, software, or via any device with processing ability or other requisite functionality. Techniques described herein may be suitable for use with other types of information processing. For example, the processing can operate on previously compressed or encoded image information, on three-dimensional image data, on non-visual information, etc.
Arrowheads shown, on signal paths between various modules are for illustrative purposes only. For example, various communication paths or connecting lines, which appear to be unidirectional in the drawings, may be bidirectional without departing from the scope of the present teachings.
Although, processes of the present, invention and the hardware executing, the processes may be characterized by language common to a discussion of video processing (e.g., “FPGA,” “CABAC,” etc.), it should be apparent that operations of an embodiment of the present invention can execute on any type of suitable hardware in any communication relationship to another device on any type of link or network.
Although a process of the present invention may be presented as a single entity, such as software or hardware executing on a single machine, such software can readily be executed on multiple machines. That is, there may be multiple-instances of a given software program, a single program may be executing on two or more processors in a distributed processing environment, parts of a single program may be executing on different physical machines, etc. Furthermore, two different programs, such as a transformation program and a quantization program, can be executing in a single module, or in different modules.
Although the invention has been discussed with respect to specific example embodiments thereof these embodiments are merely illustrative, and not restrictive, of the invention. In the description herein, numerous specific details are provided, such as examples of components and/or methods, to provide a thorough understanding of discussed example embodiments. One skilled, in the relevant art will recognize, however, that certain embodiments can be practiced without one or more of the specific details, or with other apparatus, systems, assemblies, methods, components, materials, parts, and/or the like, in other instances, well-known structures, materials, or operations are not specifically shown or described in detail to avoid obscuring aspects of the example embodiments discussed herein.
A “machine-readable medium” or “computer-readable medium” may be any medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with, the instruction execution system, apparatus, system or device. The computer readable medium can be, by way of example only but not by limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, system, device, propagation medium, or computer memory.
Reference throughout this specification to “one embodiment”, “an example embodiment”, or “a specific embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment not necessarily included in all possible example embodiments. Thus, respective appearances of the phrases “in one embodiment”, “in an embodiment”, or “in a specific embodiment” in various places throughout this specification are not necessarily referring to the same embodiment. Furthermore, the particular features, structures, or characteristics of any specific embodiment or example embodiment discussed herein may be combined in any suitable manner with one or more other embodiments. It is to be understood that other variations and modifications of the embodiments described and illustrated herein are possible in light of the teachings herein, and the variations are to be considered as part of the spirit and scope of the present teachings.
Example embodiments discussed herein may be implemented in whole or in part, by using a programmed general purpose digital computer; by using application specific integrated circuits, programmable logic devices, FPGAs, optical, chemical, biological, quantum or nanoengineered systems or mechanisms; and so on. In general, the functions of various embodiments can be achieved by any means as is known in the art. Distributed or networked systems, components, and/or circuits can be used. Communication, or transfer of data may be wired, wireless, or by any other means.
It will also be appreciated that, one or more of the elements depicted in the drawings/figures can also be implemented in a more separated or integrated manner, or even removed or rendered as inoperable in certain cases, as is useful in accordance with a particular application. It is also within the spirit and scope of the present invention to implement a program or code that can be stored in a machine-readable medium to permit a computer to perform any of the methods described above.
1. A system for processing sub-blocks of a macroblock, the system comprising:
process each sub-block of the macroblock, wherein each sub-block is associated with a predetermined position in a first sequence, wherein the processing of certain sub-blocks in the first sequence requires results of processing of one or more previously processed sub-blocks in the first sequence, wherein the processing of one or more previously processed sub-blocks in the first sequence comprises the processor operative to form a prediction sub-block based on information pertaining to a current macroblock to be processed; and
selectively process each sub-block of the macroblock in a second sequence, wherein the second sequence is altered from the first sequence to enable the first module to process certain sub-blocks of the macroblock in parallel; and
a controller for controlling a demultiplexer to dispatch the sub-blocks of the macroblock and requisite processing information pertaining to the sub-blocks of the macroblock to the first sequence and the second sequence, wherein the sub-blocks are dispatched on successive clock ticks.
2. The system of claim 1, wherein the processor is further operative to implement an intra prediction of pixels within each sub-block.
3. The system of claim 1, further comprising first means for processing sub-blocks of certain pairs of sub-blocks in parallel.
4. The system of claim 1, first means for processing sub-blocks of certain pairs of sub-blocks in a pipeline fashion.
5. The system of claim 3, wherein the first means includes: a first engine and a second engine.
6. The system of claim 1, wherein the demultiplexer is adapted to feed the requisite information pertaining to one or more sub-blocks to the processor in a desired sequence or in parallel in response to a predetermined control signal to enable processing of the one or more sub-blocks.
7. A method for processing a sub-block of a macroblock, the method comprising: receiving, at a processor, information to process a first sub-block, where the receiving information to process a first sub-block comprises receiving at least one previously processed sub-blocks in the first sub-block operative to form a prediction sub-block based on information pertaining to a current macroblock to be processed; receiving, at the processor, information to process a second sub-block; processing a first portion of the second sub-block simultaneously with a second portion of the first sub-block when information associated with one or more previously processed sub-blocks is simultaneously available for processing the first sub-block and the second sub-block; and
dispatching, by a multiplexer circuit, the first sub-block, the second sub-block and requisite processing information pertaining to the sub-blocks of the macroblock to the first sequence and the second sequence, wherein the sub-blocks are dispatched on successive clock ticks.
8. An apparatus for facilitating processing a macroblock, the apparatus comprising:
a processor operative to selectively stagger the delivery of information pertaining to a first sequence of sub-blocks and a second sequence of sub-blocks to the processor so that one or more sub-blocks of the first sequence or the second sequence that are required for processing certain sub-blocks of the first sequence and second sequence in parallel, are available to the processor before the processor processes the certain sub-blocks, where the information pertaining to the first sequence of sub-blocks comprises at least one previously processed sub-block in the first sequence of sub-blocks operative to form a prediction sub-block based on information pertaining to a current macroblock to be processed; and
a controller circuit for controlling a demultiplexer to dispatch sub-blocks of the macroblock and the information pertaining to the sub-blocks of the macroblock to the first sequence and the second sequence, wherein the sub-blocks are dispatched on successive clock ticks.
9. The apparatus of claim 8, wherein the certain sub-blocks include a certain pair of sub-blocks.
10. The apparatus of claim 8, further comprising an output operative to selectively deliver, from the processor, certain processed pairs of sub-blocks, wherein the processed pairs were processed in parallel by the processor.
11. A system for processing a macroblock, the system comprising:
process a sub-block; and
selectively enable the processor to begin processing a first sub-block and enable the processor to begin processing a second sub-block without waiting for the processor to complete processing of the first sub-block, when information is available to process the second sub-block, wherein the information available to process the second sub-block comprises a prediction sub-block based on the second sub-block; and
a controller for controlling a demultiplexer to dispatch sub-blocks of the macroblock and requisite processing information pertaining to the sub-blocks of the macroblock to the first sequence and the second sequence, wherein the sub-blocks are dispatched on successive clock ticks.
12. The system of claim 1, wherein the prediction sub-block is formed based on neighbor sub-block information and processed pixel data representing reconstructed pixels pertaining to requisite sub-blocks of the current macroblock.
13. The system of claim 1, wherein the neighbor sub-block information include information regarding top, top right and left sub-blocks in the macroblock.
transform the residual macroblock from time domain to frequency domain to form a transformed macroblock;
quantize and then dequantize the transformed macroblock; and
transform the dequantized transformed macroblock from the frequency domain to time domain to form a resulting residual macroblock.
15. The system of claim 14, wherein the processor is further operative to add the resulting residual macroblock to the predicted sub-block to form a reconstructed macroblock.
16. The system of claim 14, wherein the processor is further operative to track a bit production rate wherein bit production is a number of bits processed during quantization of the transformed macroblock.
17. The system of claim 14, wherein the sub-blocks and the requisite processing information pertaining to the sub-blocks are dispatched to the first sequence and the second sequence based on at least one of the following: a picture complexity, current bit buffer levels and the bit production rate from the previous sub-blocks.
18. The system of claim 14, wherein the processor is further operative to determine a sum absolute difference (SAD) value, wherein the SAD value represent a measure of quality, efficiency and accuracy of the prediction sub-block.
19. The system of claim 18, wherein the accuracy of the prediction sub-block is a measure of how closely the predicted sub-block matches with corresponding original sub-block in unprocessed macroblock.
20. The system of claim 1, wherein the processor is further operative to subtract the predicted sub-block from the corresponding macroblock to form a residual macroblock.
US11693473 2007-03-29 2007-03-29 Intra-macroblock video processing Active 2031-01-24 US8369411B2 (en)
US11693473 US8369411B2 (en) 2007-03-29 2007-03-29 Intra-macroblock video processing
PCT/US2008/058343 WO2008121667A3 (en) 2007-03-29 2008-03-27 Intra-macroblock video processing
EP20080732889 EP2132939B1 (en) 2007-03-29 2008-03-27 Intra-macroblock video processing
CA 2682449 CA2682449C (en) 2007-03-29 2008-03-27 Intra-macroblock video processing
US20080240253A1 true US20080240253A1 (en) 2008-10-02
US8369411B2 true US8369411B2 (en) 2013-02-05
ID=39734872
US11693473 Active 2031-01-24 US8369411B2 (en) 2007-03-29 2007-03-29 Intra-macroblock video processing
US (1) US8369411B2 (en)
EP (1) EP2132939B1 (en)
CA (1) CA2682449C (en)
WO (1) WO2008121667A3 (en)
US20100309978A1 (en) * 2009-06-03 2010-12-09 Fujitsu Limited Video encoding apparatus and video encoding method
US20150091914A1 (en) * 2013-09-27 2015-04-02 Apple Inc. Processing order in block processing pipelines
US8848779B2 (en) 2010-07-15 2014-09-30 Sharp Laboratories Of America, Inc. Method of parallel video coding based on block size
US8837577B2 (en) 2010-07-15 2014-09-16 Sharp Laboratories Of America, Inc. Method of parallel video coding based upon prediction type
US8855188B2 (en) 2010-07-15 2014-10-07 Sharp Laboratories Of America, Inc. Method of parallel video coding based on mapping
US8873617B2 (en) 2010-07-15 2014-10-28 Sharp Laboratories Of America, Inc. Method of parallel video coding based on same sized blocks
US8879619B2 (en) 2010-07-15 2014-11-04 Sharp Laboratories Of America, Inc. Method of parallel video coding based on scan order
US8548062B2 (en) 2010-07-16 2013-10-01 Sharp Laboratories Of America, Inc. System for low resolution power reduction with deblocking flag
US9313523B2 (en) 2010-07-16 2016-04-12 Sharp Laboratories Of America, Inc. System for low resolution power reduction using deblocking
US8767828B2 (en) 2010-07-16 2014-07-01 Sharp Laboratories Of America, Inc. System for low resolution power reduction with compressed image
US20140169481A1 (en) * 2012-12-19 2014-06-19 Ati Technologies Ulc Scalable high throughput video encoder
WO2001052538A1 (en) 2000-01-13 2001-07-19 Accord Networks Ltd. Method and system for compressed video processing
US20020021234A1 (en) 1998-03-25 2002-02-21 Taichi Yanagiya Encoding, decoding, and probability estimation method
US20020196853A1 (en) 1997-06-04 2002-12-26 Jie Liang Reduced resolution video decompression
EP1296520A2 (en) 2001-09-14 2003-03-26 Polycom Israel Ltd. Method and system for multimedia video processing
EP1355499A2 (en) 2002-04-08 2003-10-22 Broadcom Corporation System and method for multi-row decoding of video with dependent rows
US6887590B2 (en) * 2000-05-15 2005-05-03 Thyssenkrupp Stahl Ag Galvannealed metal sheet and method for manufacture of such metal sheet
US20060088099A1 (en) 2002-07-22 2006-04-27 Wen Gao Bit-rate control Method and device combined with rate-distortion optimization
US20060294174A1 (en) 2005-06-27 2006-12-28 Haque Munsi A Hardware-based CABAC decoder
US20080075376A1 (en) 2006-09-26 2008-03-27 James Wilson Iterative process with rotated architecture for reduced pipeline dependency
US20080117965A1 (en) 2005-01-25 2008-05-22 Collaboration Properties, Inc. Multiple-Channel Codec and Transcoder Environment for Gateway, Mcu, Broadcast, and Video Storage Applications
US20080162911A1 (en) 2006-12-29 2008-07-03 Karthik Vaithianathan High performance renormalization for binary arithmetic video coding
US20080240587A1 (en) 2007-03-29 2008-10-02 James Au Selective information handling for video processing
US20080240254A1 (en) 2007-03-29 2008-10-02 James Au Parallel or pipelined macroblock processing
US196853A (en) * 1877-11-06 Improvement in tube-machines
US75376A (en) * 1868-03-10 William s
US138045A (en) * 1873-04-22 Improvement in school desks and seats
US117965A (en) * 1871-08-15 Improvement in epergnes for fruits and flowers
US162911A (en) * 1875-05-04 Improvement in the manufacture of earthenware and pottery
US94729A (en) * 1869-09-14 Improved ltfnch-box
EP1746889B1 (en) * 2004-05-12 2009-07-22 Bayer CropScience AG Plant growth regulation
US20080291940A1 (en) 2002-09-27 2008-11-27 Leblanc Wilf Switchboard For Dual-Rate Single-Band Communication System
WO2008039321A2 (en) 2006-09-26 2008-04-03 Analog Devices, Inc. Iterative process with rotated architecture for reduced pipeline dependency
WO2008121667A2 (en) 2007-03-29 2008-10-09 Scientific-Atlanta, Inc. Intra-macroblock video processing
WO2008121664A2 (en) 2007-03-29 2008-10-09 Scientific-Atlanta, Inc. Video processing architecture
WO2008121662A2 (en) 2007-03-29 2008-10-09 Scientific-Atlanta, Inc. Parallel or pipelined macroblock processing
WO2008121663A2 (en) 2007-03-29 2008-10-09 Scientific-Atlanta, Inc. Entropy coding for video processing applications
WO2008127855A2 (en) 2007-03-29 2008-10-23 Scientific-Atlanta, Inc. Selective information handling for video processing
US7953284B2 (en) 2007-03-29 2011-05-31 James Au Selective information handling for video processing
Canadian Office Action dated Jan. 11, 2012 cited in Application No. 2,682,449, 3 pgs.
Canadian Office Action dated Jan. 5, 2012 cited in Application No. 2,682,461, 3 pgs.
Canadian Office Action dated Jul. 4, 2011 cited in Application No. 2,682,590.
Canadian Office Action dated Jun. 21, 2012 cited in Application No. 2,682,461, 4 pgs.
Canadian Office Action dated Jun. 4, 2012 cited in Application No. 2,682,590, 2 pgs.
Canadian Office Action dated Mar. 26, 2012 cited in Application No. 2,682,436, 2 pgs.
Canadian Office Action dated May 7, 2012 cited in Application No. 2,682,315, 3 pgs.
Detlev Marpe et al., "Context-Based Adaptive Binary Arithmetic Coding in the H.264/AVC Video Compression Standard," IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, No. 7, Jul. 2003, pp. 621-636.
European Office Action dated Dec. 17, 2010 cited in Application No. 08 732 886.0.
European Office Action dated Feb. 11, 2010 cited in Application No. 08 732 886.0.
European Office Action dated Feb. 25, 2010 cited in Application No. 08 780 490.2.
European Office Action dated Mar. 18, 2010 cited in Application No. 08 732 889.4.
European Office Action dated Mar. 18, 2010 cited in Application No. 08 744 419.6.
European Office Action dated May 29, 2012 cited in Application No. 08 732 886.0, 7 pgs.
European Office Action dated Nov. 3, 2011 cited in Application No. 08 732 886.0, 6 pgs.
Genhua Jin et al , A parallel and pipelined exexuation of H.264/AVC intra prediction, 2006, IEEE comptuer society, p. 1-6. *
Genhua Jin, et al., "A Parallel and Pipelined Execution of H.264/AVC Intra Prediction", 6 pages, Computer and Information Technology, 2006.
Hassan Shojania et al., "A High Performance CABAC Encoder," IEEE-NEWCAS Conference, 2005, 4 pgs.
Hassan Shojania et al., "A VLSI Architecture for High Performance CABAC Encoding," Visual Encoding and Image Processing, 2005, Proc. of SPIE vol. 5960, 11 pgs.
Hendrik Eeckhaut et al., "Optimizing the critical loop in the H.264/AVC CABAC Decoder," Field Programmable Technology, 2006, pp. 113-118.
International Preliminary Report dated Sep. 29, 2009 cited in Application No. PCT/US2008/058334.
International Search Report dated Dec. 3, 2008 cited in Application No. PCT/US2008/058349.
International Search Report dated Nov. 10, 2008 cited in Application No. PCT/US2008/058335.
International Search Report dated Nov. 14, 2008 cited in Application No. PCT/US2008/058334.
Jian-Wen Chen et al., "Introduction to H.264 Advanced Video Coding," Design Automation, Asia and South Pacific Conference on Jan. 24, 2006, pp. 736-741.
Lingfeng Li et al., "A CABAC Encoding Core with Dynamic Pipeline for H.264/AVC Mail Profile," Circuits and Systems, 2006, pp. 760-763.
Mo Li et al., "A High Throughput Binary Arithmetic Coding Engine for H.264/AVC," Solid-State and Integrated Circuit Technology, 2006, 5 pgs.
Ron Wilson, "The Right Video Architecture Can Make All the Difference," Acquired at: http://www.edn.com/contents/images/6363920.pdf, Sep. 1, 2006, 9 pgs.
Sato K., et al., "Consideration on Intra Prediction for Pipeline Processing in H.264/MPEG-4 AVC", 10 pages, Applications of Digital Image Processing XXVII, Proceedings of SPIE, vol. 5558, 2004.
U.S. Final Office Action dated Dec. 16, 2011 cited in U.S. Appl. No. 11/693,526, 21 pgs.
U.S. Final Office Action dated Dec. 29, 2011 cited in U.S. Appl. No. 11/693,567, 19 pgs.
U.S. Final Office Action dated Jan. 4, 2012 cited in U.S. Appl. No. 11/693,506, 19 pgs.
U.S. Final Office Action dated Oct. 18, 2010 cited in U.S. Appl. No. 11/693,439.
U.S. Office Action dated Jul. 23, 2010 cited in U.S. Appl. No. 11/693,439.
U.S. Office Action dated Jun. 16, 2011 cited in U.S. Appl. No. 11/693,526.
U.S. Office Action dated Jun. 16, 2011 cited in U.S. Appl. No. 11/693,567.
U.S. Office Action dated Jun. 27, 2011 cited in U.S. Appl. No. 11/693,506.
Wonjae Lee et al , Piplelined intra prediction using shuffled encoding order for H 264/AVC, 2006 IEEE and p. 4. *
Wonjae Lee, et al., "Pipelined Intra Prediction Using Shuffled Encoding Order for H.264/AVC", 4 pages, Tencon 2006, 2006 IEEE Region Conference.
Yu-Wen Huang, et al., "Hardware Architecture Design for H.264/AVC Intra Frame Coder", 4 pages, Circuits and Systems, 2004.
US9106917B2 (en) * 2009-06-03 2015-08-11 Fujitsu Limited Video encoding apparatus and video encoding method
US9218639B2 (en) * 2013-09-27 2015-12-22 Apple Inc. Processing order in block processing pipelines
CA2682449A1 (en) 2008-10-09 application
EP2132939A2 (en) 2009-12-16 application
EP2132939B1 (en) 2013-08-21 grant
WO2008121667A3 (en) 2008-11-27 application
WO2008121667A2 (en) 2008-10-09 application
CA2682449C (en) 2014-02-18 grant
US20080240253A1 (en) 2008-10-02 application
US20070086528A1 (en) 2007-04-19 Video encoder with multiple processors
US20110206113A1 (en) 2011-08-25 Data Compression for Video
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:AU, JAMES;MOSS, BARRY;REEL/FRAME:019867/0189