Multi-core decompression of block coded video data

Apparatus for and a method of decompression of block coded video data in a multi-core processor. The processor cores decode respective coded groups of blocks of video data independently, in parallel and deblock respective decoded groups of blocks of video data independently and in parallel with the decode operations and with other deblock operations.

BACKGROUND OF THE INVENTION

The present invention is directed to video data decoding and, more particularly, to an apparatus for and a method of multi-core decompression of block coded video data.

Video compression is the reduction of the quantity of data used to represent digital video images and includes spatial image compression and temporal motion estimation. Typically, in block coded video compression, blocks of neighboring pixels, often called macro-blocks, are compared and the video compression encoding scheme retains the differences between blocks. Video data may be intra-frame encoded by registering spatial differences within a frame and/or inter-frame encoded by registering temporal differences between frames. Various techniques exist for video compression, such as H.264/MPEG-4 AVC, which is one widely used block-coded motion-compensation-based compression/decompression (‘codec’) standard.

The greater the degree of compression obtained, the greater the computational load on decompression. Moreover, high definition video and other video formats with increased data content, such as stereoscopic video for example, also increase the computational load of decompression. Multi-core processors may be used, for real-time decompression for example, in which multiple processor cores participate in parallel in decompression processing. However, such multi-core parallel processing has posed issues of load balancing and computational overhead associated with each additional core.

The embodiments of the invention described below are applicable to decompressing video data that is block-coded in compliance with the standard H.264/MPEG-4 AVC. The standard H.264/MPEG-4 AVC is based on motion-compensated discrete cosine transform (‘DCT’) coding. Each picture is compressed by partitioning it as groups of macro-blocks of luma samples and corresponding chroma samples referred to as slices. However, embodiments of the invention are also applicable to other video data coding techniques and standards.

FIG. 1illustrates an apparatus100for multi-core decompression of block coded video data in accordance with one embodiment of the invention, given by way of example. The apparatus100comprises a plurality of processor cores102and a task management module104for controlling the processor cores to decode respective coded groups of blocks of video data independently in parallel and to deblock respective decoded groups of blocks of video data independently in parallel.

Decoding is the process of recovering detailed picture data for the video blocks from the compressed picture content data and from parameter data common to a set of video blocks. Deblocking is a process applied to decoded video blocks to improve visual quality and prediction performance by smoothing the sharp edges which can form between adjacent video blocks.

If the tasks of the processor cores102were partitioned functionally, so that one or more processor cores were dedicated to decoding coded blocks of video data, and one or more other processor cores were dedicated to deblocking decoded blocks, a large amount of data would have to be exchanged between processor cores, posing issues of cache coherency, computational overhead data for the multi-core processor, and high memory requirements.

In the apparatus100, each of the coded groups of blocks of video data may be a slice of a video frame that is coded separately from other slices and can be decoded independently. In the standard H.264/MPEG-4 AVC, a slice is a sequence of macro-blocks, which are processed in a scan order (left to right and top to bottom in the case of a television picture). The multi-core apparatus100can decode a slice that is made up of contiguous macro-blocks.

The task management module104is in communication with the processor cores102and controls the processor cores102to decode respective coded slices of video data independently in parallel. The task management module104controls at least one of the processor cores102to deblock one or more decoded groups of blocks of video data independently in parallel with at least one other of the processor cores decoding one or more of the coded groups of blocks of video data. The task management module104controls the processor cores102to decode respective coded groups of blocks with equal weighting, or to deblock respective decoded groups of blocks of video data subject to dependencies. The task management module104can allocate a higher priority to deblock tasks than to decode tasks but deblock tasks may be dependent on prior completion of decode tasks. The task management module104also can merge a plurality of deblock tasks and control at least one of the processor cores to perform the merged tasks.

The apparatus100may include non-cacheable memory110which is shared by the processor cores102for task management buffers, core states and inter-core communication controlling messages. The apparatus100may include shared memory112which is cacheable in the processor cores for reference pictures, currently decoded picture, deblocking filter parameters, motion vectors and reference picture list buffers, the processor cores including respective local cacheable memories for current coding parameters and pointers to shared cacheable and shared non-cacheable memories. The processor cores102may include respective local cacheable memories for current coding parameters and pointers to shared cacheable and shared non-cacheable memories. One of the processor cores102may initialize the reference picture lists and perform memory management control operations for a new video access unit, and the initialized reference picture lists may be copied locally by other processor cores, which re-order the locally copied reference picture lists independently. The processor cores102may verify whether a packet received from said task management module104is from the current access unit, decode the packet if it is from the current access unit, and signal to the task management module104if the packet is not from the current access unit. In order to detect lost packets, the task management module104may verify whether more packets are available to be processed, verify whether available packets are from the current access unit and, if one of the cores102is in the WAITING state and all the others are in the TERMINATE state, verify whether all packets of the current access unit have been successfully decoded with no packet loss detection.

In more detail, the multi-core processor102includes N processor cores CORE1, CORE2, CORE3, CORE4to CORE N. The apparatus100is scalable, that is to say that the same basic structure of hardware and software is efficient and practical when applied to different numbers of processor cores. Each of the processor cores CORE1to CORE N includes a respective local cacheable memory.

The core state and task management module104defines the states of the processor cores CORE1to CORE N as WAITING, WORKING or TERMINATED. Initially all the cores are placed in the WAITING state. When a task is assigned to a core, its state gets converted to WORKING. If no task is assigned to a core then it remains in, or reverts to the WAITING state and is available for assignment of other tasks. If the process meets exit criteria, for example decompression of a frame being complete or no more data remaining to process, the core state is converted to TERMINATED. The core state and task management module104receives compressed video data from a source106and the multi-core processor102provides decompressed, decoded and deblocked video data to a user module108, the source106and the user module108forming part of a framework environment109.

The core state and task management module104includes the memory110which is shared by all the cores CORE1to CORE N but is non-cacheable, and which is used mainly for task assignment and communication with the cores. The memory110is locked and always accessed using semaphores. All memory allocations inside the core state and task management module104are done from the shared non-cacheable memory110. Examples for such allocations are task management buffers, core state and control flags, inter-core communications controlling messages and semaphores.

The apparatus100also includes the memory112which is shared by all the active processor cores CORE1to CORE N and is cacheable. In order to maintain cache coherency among cores, proper memory synchronization operations (flush and invalidate) are performed before accessing these memories for read and/or write into caches. Examples of data stored in the cacheable shared memory112are reference pictures, the currently decoded picture, deblocking filter parameters, motion vectors and reference picture list buffers, for example.

The local memories in the cores CORE1to CORE N do not require any coherency operations. Examples for such buffers are current coding parameters for any given slice and pointers to the shared cacheable and shared non-cacheable memories112and110. One of the processor cores102may initialize the reference picture lists and perform memory management control operations for a new video access unit, and the initialized reference picture lists may be copied locally by other processor cores, which re-order the locally copied reference picture lists independently. The processor cores102may verify whether a packet received from said task management module104is from the current access unit, decode the packet if it is from the current access unit, and signal to the task management module104if the packet is not from the current access unit. In order to detect lost packets, the task management module104may verify whether more packets are available to be processed, verify whether available packets are from the current access unit and, if one of the cores102is in the WAITING state and all the others are in the TERMINATE state, verify whether all packets of the current access unit have been successfully decoded with no packet loss detection.

The standard H.264/MPEG-4 AVC is applicable to a variety of applications, including broadcasting such as cable, satellite, cable modem, digital subscriber line (‘DSL’), terrestrial or interactive broadcasting, or serial storage on optical and magnetic devices, conversational services, video-on-demand or multimedia streaming, and multimedia messaging services, for example. The standard H.264/MPEG-4 AVC provides for a “Network Abstraction Layer” (‘NAL’). The NAL formats the Video Coding Layer (‘VCL’) representation of the video image and provides header information in a manner appropriate for conveyance by a variety of transport layers or storage media. The coded video data is organized into NAL units, each of which is effectively a packet that contains an integer number of bytes. The first byte of each NAL unit is a header byte that contains an indication of the type of data in the NAL unit, and the remaining bytes contain payload data of the type indicated by the header. Non-VCL NAL units may contain any associated additional information such as parameter sets (important header data that can apply to a large number of VCL NAL units) and supplemental data.

FIG. 2illustrates an example200of tasks which may be performed by the apparatus100in operation. The apparatus100receives coded NAL units NAL0, NAL1, NAL2. The core state and task management module104assigns decoding tasks DECODING TASK0, DECODING TASK1, DECODING TASK2to the processor cores CORE1to CORE N. Each of the decoding tasks DECODING TASK0to DECODING TASK2may include entropy decoding, inverse transform, and inter-frame motion compensation and intra-frame prediction, for example. The resulting decoded video data is shown schematically at202and comprises decoded slices SLICE #0, SLICE #1, SLICE #2corresponding to the coded NAL units NAL0, NAL1, NAL2. Each of the slices SLICE #0to SLICE #2comprises a respective sequence of macroblocks, an example of the boundaries between the slices being indicated by bold lines.

The core state and task management module104also assigns deblocking tasks DEBLOCKING TASK0, DEBLOCKING TASK1, DEBLOCKING TASK2to the processor cores CORE1to CORE N. The deblocking tasks smooth the sharp edges which can form between adjacent video macroblocks, as indicated at204. The decoding tasks DECODING TASK0to DECODING TASK2are assigned to the cores CORE1to CORE N in parallel with equal weighting, that is to say without dedicating or otherwise specializing one or more cores for this function. The deblocking tasks DEBLOCKING TASK0to DEBLOCKING TASK2are assigned to any available one or more of the cores CORE1to CORE N in parallel with other deblocking tasks and in parallel with the decoding tasks. The multi-core processor102may execute multiple decode tasks and multiple deblock tasks in parallel if deblocking across slices is disabled. The multi-core processor102may execute multiple decode tasks and single deblock tasks in parallel if deblocking across slices is enabled.

A decode task does not have any dependencies and can be executed at any time, depending upon the task priority. A deblock task has higher priority than a decode task. However, deblocking of the nthrow can only start after the deblocking of the n−1throw and after decoding of the nthrow. In other words, a deblock task can be executed only if associated dependencies are resolved. The core state and task management module104maintains a common task list for decode tasks and deblock tasks. Each of the cores CORE1to CORE N can be assigned a task from the shared task list or add a task to the same list. A deblocking task is added to the task list only when all its dependencies are resolved and it is available to deblock. Accordingly, no post-processing is needed. The core state and task management module104may also merge deblock tasks. If more than one deblocking task is available in the task list then the core state and task management module104merges the deblocking tasks, combining the tasks of deblocking different continuous rows as a single deblocking task which it adds to the task list. This avoids resource conflict and false searching for tasks.

FIG. 3illustrates a method300of decompression of block coded video data in a multi-core processor comprising a plurality of processor cores in accordance with one embodiment of the invention, given by way of example, and which may be performed by the apparatus100, for example, or by other apparatus. The method300comprises controlling the processor cores to decode respective coded groups of blocks of video data independently in parallel and to deblock respective decoded groups of blocks of video data independently in parallel.

In the method300, each of the coded groups of blocks of video data may be a slice of a video frame which is coded separately from other slices and the processor cores may decode respective coded slices of video data independently in parallel. At least one of the processor cores may deblock one or more decoded groups of blocks of video data independently in parallel with at least one other of said processor cores decoding one or more of said coded groups of blocks of video data. The processor cores may decode respective coded groups of blocks with equal weighting, or deblock respective decoded groups of blocks of video data subject to dependencies. A plurality of deblock tasks may be merged and at least one of said processor cores may perform the merged tasks.

In the method300, the processor cores may share a non-cacheable memory for task management buffers, core states and inter-core communication controlling messages. The processor cores may share a memory which is cacheable for reference pictures, currently decoded picture, deblocking filter parameters, motion vectors and reference picture list buffers, the processor cores including respective local cacheable memories for current coding parameters and pointers to shared cacheable and shared non-cacheable memories. One of the processor cores may initialize the reference picture lists and perform memory management control operations for a new video access unit, and the initialized reference picture lists may be copied locally by other processor cores, which re-order the locally copied reference picture lists independently. The processor cores may verify whether a packet received from said task management module is from the current access unit, decode the packet if it is from the current access unit, and signal to said task management module if the packet is not from the current access unit. Detecting lost packets may include verifying whether more packets are available to be processed, verifying whether available packets are from the current access unit and, if one core is in the WAITING state and all the others are in the TERMINATE state, verifying whether all packets of the current access unit have been successfully decoded with no packet loss detection.

In more detail, the method300starts at302by defining tasks for a first one of the processor cores. At304, the first core verifies whether a new access unit is starting or not while other cores remain in the WAITING state at306until the first core is ready for synchronization. At304, if no new access unit is starting then the first core does not do any initialization operation. However, if a new access unit is starting at304, the process resets at308all the management data, picture data and parameters which are not relevant to the new access unit. At310, the process then decodes non-VCL NALs and partially decodes slice headers of a single VCL NAL from the initial NAL units. At312one of the processor cores initializes the reference lists and performs memory management control operations for the new video access unit.

The cores are synchronized at314and the initialized reference picture lists are copied locally by other processor cores. The process verifies the core states at316until the core is put in the TERMINATE state. For those cores which are in the WAITING state, the task management process is engaged in succession at318. As indicated at320, the task management process may result in the current core being put in the TERMINATE state if the exit criteria are met, for example decompression of a frame being complete or no more data remaining to process. Otherwise at318and320, a DECODE task may be added to the task list, a task may be assigned to the current core, two or more tasks may be merged in the task list, or the lost packet process may be engaged, for example, which may result in the current core being put in the WORKING state.

At322, the process depends on the nature of the task assigned to the current core. If the task assigned is DEBLOCK, the deblock process is performed at324and the process reverts to core state verification and task management at316and318. If the task assigned is DECODE, the processor core verifies at326whether the unit to be processed corresponds to a new access unit (a new frame) and, if not, the reference lists in the local memory of the current core are reordered at328, the DECODE process is performed at330, a DEBLOCK task is added to the task list at332and the process reverts to core state verification and task management at316and318. The process also reverts to core state verification and task management at316and318if at326the unit to be processed corresponds to a new access unit, and if at322the current core no longer has a task to perform.

On-the-fly new frame detection at326checks whether a DECODE task belongs to the next access unit or not; and if so raises a next access unit flag which is used at steps304,336and408(FIG. 4) for decisions. If the DECODE task belongs to the next access unit, then it is not processed and, the processor core identifies to the task management module and the framework that it relates to a FUTURE NAL which is to be processed in the next access unit, and the process reverts to core state verification and task management at316and318. If the DECODE task belongs to the current access unit, the task is processed and intimated to framework as a FREE NAL, and the process reverts to core state verification and task management at316and318.

If at316the process meets the criteria for the TERMINATE state, the process ends at334for all cores except for the last one. For the last core at336, if the next access unit flag set at326is found to be FALSE the process for the current access unit also ends for the last core. If at336next access unit flag is found to be TRUE then the process verifies at338whether frame is complete or incomplete. If the frame is complete, the process proceeds to memory management control operation (‘MMCO’) or sliding window at340. If the frame is incomplete, the process identifies the frame locations corresponding to the lost access units (packets) and performs error concealment algorithms at342before proceeding to MMCO or sliding window at340. The decompressed frame is then obtained at344.

FIG. 4illustrates in more detail an example of a method400to detect lost access units (lost packets) at320. The method400starts at402and at404, the process verifies whether more packets are available to be processed. If not, the process verifies core state at406and unless one core is in the WAITING state and all the others are in the TERMINATE state, the process proceeds to task management at318so that the other active cores can complete their tasks. If at404more packets are available to be processed, the process verifies at408whether the next access unit flag is set to TRUE or FALSE at step326. If at408the next access unit flag is found to be FALSE, the process proceeds to task management at318to process the packets. If at408the next access unit flag is found to be TRUE, the process reverts to core state verification at406.

If at406one core is in the WAITING state and all the others are in the TERMINATE state, the process verifies at410whether all macroblocks for the current access unit have been successfully decoded. If at410all macroblocks for the current access unit have been successfully decoded with no packet loss detection, the method400ends at412and the process proceeds to task management at318to perform MMCO or sliding window340. If at410successfully decoded macroblocks for the current access unit are incomplete, the process identifies the frame locations corresponding to lost packets and performs error concealment algorithms at342and then MMCO or sliding window340, the method400ending at412.

FIG. 5illustrates an example500of the relative timing of tasks performed in two different cores CORE1and CORE2during the process300. Task management activity is shown by forward hatched areas. Time spent by a core waiting for task management activity is shown by cross hatched areas. Reference picture management activity is shown by reverse hatched areas. Core WAITING time is shown by dotted areas.

At502, both cores CORE1and CORE2are assigned by the framework environment109to be used in decoding. At504, both cores CORE1and CORE2are being initialized, and their local memories are being reset. At506, CORE1is engaged in reference picture management and CORE2is in the WAITING state. At508, CORE1is being assigned a task by the task manager and CORE2is waiting for task management activity. At510, CORE2is being assigned a task by the task manager.

At512and514, cores CORE1and CORE2are engaged in decoding activity in parallel. CORE1is decoding NAL unit #0and CORE2is independently decoding NAL unit #1. At516, CORE2has completed decoding NAL unit #1and is being assigned a task by the task manager. At518, CORE2is engaged in deblocking activity in parallel with the decoding activity512of NAL unit #0in CORE1. At520, CORE1is being assigned a task by the task manager and at522CORE1is decoding NAL unit #2, initially in parallel with the deblocking activity518in CORE2. At524, CORE2has completed the deblocking activity518and is being assigned a task by the task manager. At526, CORE2is engaged in deblocking activity in parallel with the decoding activity522of NAL unit #2in CORE1. Similarly, at528to534, CORE2has completed deblocking activities526and530and is assigned a task by the task manager at528and532. At526,530and534, CORE2is engaged in deblocking activity in parallel with the decoding activity522of NAL unit #2in CORE1.

At536, the decoding activity522in CORE1and the deblocking activity534in CORE2have been completed, CORE1is being assigned a task by the task manager and CORE2is waiting for task management activity. At538, CORE1is engaged in deblocking activity. At540, CORE2is being assigned a task by the task manager and at542CORE2is engaged in framework activity. At544, CORE1has completed the deblocking activity538and is being assigned a task by the task manager. At546, CORE1is engaged in reference picture management and at548CORE1is engaged in framework activity.

The example500illustrates how the method300enables the decoding tasks to be assigned to any available one or more of the cores in parallel with equal weighting, that is to say without dedicating or otherwise specializing one or more cores for this function. The deblocking tasks may be assigned to any available one or more of the cores in parallel with other deblocking tasks and in parallel with the decoding tasks.

In an example of utilization of the method300running in an apparatus100on two typical video scenes, the use of a 6-core processor instead of a single core processor reduced the latency from 0.64 seconds to 0.13 seconds for a computational overhead of 2.7% per core for one video sequence, and reduced the latency from 1.49 seconds to 0.32 seconds for a computational overhead of 3.6% per core for the other scene. Load balance between the cores was close to optimal.

The invention may be implemented at least partially in a computer program that runs on a computer system, having code portions for performing steps of a method according to the invention when run on a programmable apparatus, such as a computer system or enabling a programmable apparatus to perform functions of a device or system according to the invention. A computer program is a list of instructions such as a particular application program and/or an operating system. The computer program may include one or more of: a subroutine, a function, a procedure, an object method, an object implementation, an executable application, an applet, a servlet, source code, object code, a shared library/dynamic load library (DLL) and/or other sequence of instructions designed for execution on a computer system.

The computer program may be stored internally on computer readable storage medium or transmitted to the computer system via a computer readable transmission medium. All or some of the computer program may be provided on computer readable media permanently, removably or remotely coupled to an information processing system. The computer readable media may include, for example, magnetic storage media including disk and tape storage media; optical storage media such as compact disk media (e.g., CD-ROM, CD-R, etc.) and digital video disk storage media; nonvolatile memory storage media including semiconductor-based memory units such as FLASH memory, EEPROM, EPROM, ROM; ferromagnetic digital memories; MRAM; volatile storage media including registers, buffers or caches, main memory, RAM, etc.; and data transmission media including computer networks, point-to-point telecommunication equipment, and carrier wave transmission media, just to name a few.

Also for example, the examples, or portions thereof, may be implemented as soft or code representations of physical circuitry or of logical representations convertible into physical circuitry, such as in a hardware description language of any appropriate type.