Power and computational load management techniques in video processing

Techniques for managing power consumption and computational load on a processor during video processing and decoding are provided. One representative embodiment discloses a method of processing a data stream that includes video data. According to the method, one or more protocols used to create the data stream are identified. The various parsing and decoding operations required by the protocol are then identified and managed based on the available electrical power or available processing power. Another representative embodiment discloses a method of processing a data stream that includes video data. According to the method, one or more protocols used to create the data stream are identified. The various parsing and decoding operations required by the protocol are then identified and managed based on a visual quality of the video or a quality of experience.

TECHNICAL FIELD

The present disclosure relates generally to the field of video processing and, more specifically, to techniques for power and computational load management in video processing and decoding.

BACKGROUND

The amounts of digital information included in video data are massive and tend to increase along with advances in performance of video cameras. Processing of the video data places large demands on power and computational resources of video-enabled devices and, in particular, wireless communication devices such as cellular phones, personal digital assistants (PDAs), laptop computers, and the like.

Although video compression primarily reduces spatial and temporal redundancy, there are several pre-processing and post-processing operations that are required after the source video has been captured (or extracted from storage as the case may be) and before the reconstructed video is rendered (consumed) at the display. Video Processing places large demands on memory (stored and data transfer) and computational load primarily due to the required arithmetic operations which are directly proportional to power requirements (battery, talk time, etc).

Given the amount of redundancy in video, a proportional reduction in the quantity of such operations should be expected. Since compression ratios are many orders of magnitude (100:1 to 1000:1), a significant reduction in the amount of video data to be processed can be achieved in spite of implementation overheads. Spatio-temporal redundancy can be identified using compression metadata and correspond to a reduction in redundant operations, which saves power. Different levels of redundancy translate to different levels of consumed power and computational loading on a processor.

There is therefore a need for techniques for power and computational load management in video processing and decoding.

SUMMARY

Techniques for managing power consumption and computational load on a processor in video processing and decoding are described herein. In one configuration, an apparatus is provided that comprises a processor having a set of instructions operative to extract and compile information from a bitstream containing video data. The processor is operative to identify one or more protocols used to create the bitstream, and then identify various parsing and decoding operations required by the protocols. Once identified, the parsing and decoding operations for the bitstream can then be managed based on the amount of electrical power available to the apparatus, or based on the available processing power. According to one example, the identified parsing operations and decoding operations of the bitstream may not be carried out in their entirety, but instead may be selectively carried out based on the available amount of electrical power or processing power.

According to another configuration, an apparatus is provided that comprises a processor having a set of instructions operative to extract and compile information from a bitstream containing video data. The processor is operative to identify one or more protocols used to create the bitstream, and then identify various parsing and decoding operations required by the protocols. Once identified, the parsing and decoding operations for the bitstream are managed based on a visual quality of the video or a quality of user experience. The apparatus also includes a memory coupled to the processor.

In another aspect, an integrated circuit (IC) comprising a processor having a set of instructions operative to extract and compile information from a bitstream having video is provided. The processor is operative to identify one or more protocols used to create the bitstream, identify various parsing and decoding operations required by the protocols, and then manage the parsing and decoding operations for the bitstream based on the amount of electrical power available or available processing power. According to another configuration, the parsing and decoding operations are managed based on a visual quality of the video or a quality of user experience. The integrated circuit also includes a memory coupled to the processor.

In another configuration, a computer program product including a computer readable medium having instructions for causing a processor to extract and compile information from a bitstream containing video data is provided. The instructions further cause the processor to identify one or more protocols used to create the bitstream as well as the various parsing and decoding operations required by the protocols, and then manage the parsing and decoding operations based on the amount of electrical power available or available processing power. According to another configuration, the parsing and decoding operations are managed based on a visual quality of the video or a quality of user experience.

A further aspect of the configurations includes a processor that organizes the various parsing and decoding operations into groups, with each group having a projected electrical power requirement or projected processing power requirement. The processor then selects a group such that the groups projected electrical power or processing power requirement meets or does not exceed the available electrical power or available processing power. Alternatively, the various parsing and decoding operations can be organized into groups based on a visual quality of the video or quality of user experience provided by the parsing and decoding operations associated with that group.

The summary is neither intended nor should it be construed as being representative of the full extent and scope of the present disclosure, which these and additional aspects will become more readily apparent from the detailed description, particularly when taken together with the appended drawings.

To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures, except that suffixes may be added, when appropriate, to differentiate such elements. The images in the drawings are simplified for illustrative purposes and are not depicted to scale. It is contemplated that features configurations may be beneficially incorporated in other configurations without further recitation.

The appended drawings illustrate exemplary configurations of the disclosure and, as such, should not be considered as limiting the scope of the disclosure that may admit to other equally effective configurations.

DETAILED DESCRIPTION

The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any configuration or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other configurations or designs, and the terms “core”, “engine”, “machine”, “processor” and “processing unit” are used interchangeably.

The techniques described herein may be used for wireless communications, computing, personal electronics, handsets, etc. An exemplary use of the techniques for wireless communication is described below.

FIG. 1shows a block diagram of a configuration of a wireless device10in a wireless communication system. The wireless device10may be a handset. The wireless device10or handset may be a cellular or camera phone, a terminal, a wirelessly-equipped personal digital assistant (PDA), a wireless communications device, a video game console, a laptop computer, a video-enabled device or some other wirelessly-equipped device. The wireless communication system may be a Code Division Multiple Access (CDMA) system, a Global System for Mobile Communications (GSM) system, or some other system.

The wireless device10is capable of providing bi-directional communications via a receive path and a transmit path. On the receive path, signals transmitted by base stations are received by an antenna12and provided to a receiver (RCVR)14. The receiver14conditions and digitizes the received signal and provides samples to a digital section20for further processing. On the transmit path, a transmitter (TMTR)16receives data to be transmitted from the digital section20, processes and conditions the data, and generates a modulated signal, which is transmitted via the antenna12to the base stations.

The digital section20includes various processing, interface and memory units such as, for example, a modem processor22, a video processor24, a controller/processor26, a display processor28, an ARM/DSP32, a graphics processing unit (GPU)34, an internal memory36, and an external bus interface (EBI)38. The modem processor22performs processing for data transmission and reception (e.g., modulation and demodulation). The video processor24performs processing on video content (e.g., still images, moving videos, and moving texts) for video applications such as camcorder, video playback, and video conferencing. The video processor24performs video encoding and decoding or codec operations. The video encoding and decoding operations may be performed by another processor or shared over various processors in the digital section20. The controller/processor26may direct the operation of various processing and interface units within digital section20. The display processor28performs processing to facilitate the display of videos, graphics, and texts on a display unit30. The ARM/DSP32may perform various types of processing for the wireless device10. The graphics processing unit34performs graphics processing.

The GPU34may be compliant, for example, with a document “OpenGL Specification, Version 1.0,” Jul. 28, 2005, which is publicly available. This document is a standard for 2D vector graphics suitable for handheld and mobile devices, such as cellular phones and other referred to above wireless communication apparatuses. Additionally, the GPU34may also be compliant with OpenGL2.0, OpenGL ES2.0, or D3D9.0 graphics standards.

The techniques described herein may be used for any of the processors in the digital section20, e.g., the video processor24. The internal memory36stores data and/or instructions for various units within the digital section20. The EBI38facilitates the transfer of data between the digital section20(e.g., internal memory36) and a main memory40along a bus or data line DL.

The digital section20may be implemented with one or more DSPs, micro-processors, RISCs, etc. The digital section20may also be fabricated on one or more application specific integrated circuits (ASICs) or some other type of integrated circuits (ICs).

The techniques described herein may be implemented in various hardware units. For example, the techniques may be implemented in ASICs, DSPs, RISCs, ARMs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and other electronic units.

Raw video data may be compressed in order to reduce the amount of information that must be transmitted to or processed by wireless device10or other video-enabled device. Compression may be performed using, for example, video coding techniques compliant with one or more of industry-adapted video compression and communication standards, including those standards by ISO/IEC's Moving Picture Expert Group MPEG-2 and MPEG-4, ITU-T's H.264/AVC, or others (AVC stands for Advanced Video Coding). Video coding techniques compliant with non-standard compression methods such as VP6 used in Adobe Flash player may also be used to generate the compressed video data. In the configurations, the raw and compressed video data may be transmitted to, from, or within the wireless device10or other video-enabled device using wireless or wired interfaces or a combination thereof. Alternatively, the compressed data may be stored in media such as DVDs.

The compressed video data is encapsulated in a payload format for transmission using transport protocols using, for example, Internet Protocol (IP) as defined by IETF in Real Time Transport Protocol specifications.

FIG. 2Ashows a block diagram of a data stream and the corresponding protocols that must be transmitted or processed by wireless device10or other video-enabled device. A data stream,2141, comprised of transport layer data2142, for example, encapsulation as specified by the transport protocol specification,2145, and video layer data,2143. The transport layer data follows format or syntax or semantics of data representation as specified in the corresponding transport protocol and video layer data follows format or syntax or semantics for representation of video data as specified in video coding protocol,2144, such as the compression standards.

A Transport Protocol2145encapsulates video layer data for transmission or storage, e.g. file format like MP4 or transport format such as RTP or UDP or IP. A Video Coding protocol2144can be a video coding standard such as MPEG-2 or MPEG-4 or H.264/AVC or any other video codec such as Real Video or Windows Media etc. The syntax and semantics of the transport layer data is governed or specified by the transport protocol and syntax and semantics of the video layer data is governed or specified by the video coding protocol.

FIG. 2Bshows the format of the video layer data,2143. The video layer data comprises a sequence or group of pictures (GOP) or a picture layer data,2243, a slice or macroblock (MB) layer data,2254and a block layer data,2247.

At the receiver, when the data stream is received, in traditional systems, the video processor parsers and decodes the data stream in the order specified by the corresponding transport protocol specification and the video coding protocol or standard specification. The transport parser unwraps the encapsulation in an order corresponding to the transport protocol specification herein referred to as normal parsing operation. The video decoder parsers and decodes the video layer data in an order specified by the video coding protocol or standard specification herein referred to as normal decoding operation.

In the described system and methods below, the video processor selectively parses and/or decodes or processes parts of the data stream and the order of the parsing and/or decoding and processing operations is based on available power, available computational processing power or visual quality.

FIG. 2Cshows a general MPEG packet format50. An MPEG packet format is an example of a data stream,2141. The MPEG packet format50includes a plurality of MPEG layers52,54,56,58,60,62and64. The MPEG layers include a transport layer52, a sequence layer54, a group of pictures (GOP) layer56, a picture layer58, a slice layer60, a macroblock (MB) layer62and a block layer64. InFIG. 2A, the layers are shown stacked to represent a hierarchical order of layers that require decoding and processing. For the purposes of description herein, the sequence and picture layers54and58are grouped together and called a video sequence/picture layer (VS/PL)70for the purposes of power load management described herein. In some standards, only a sequence layer may be present or a picture layer or a combination of layers. Additionally, the slice and macroblock (MB) layers60and62are grouped together to form a slice/MB layer (S/MBL)72for the purposes of power load management described herein. In some standards, one or more of the layers may be omitted or combined.

In MPEG compression, video frames may be coded and formatted into a group of pictures (GOP) which may include one or more of an intra-coded (I) frame, a predictive-coded (P) frame, and a bidirectionally predictive-coded (B) frame. Some B-frames may be reference frames. Non-reference B-frames may be designated as b-frames. As can be appreciated, describing all the frames and arrangement of frames in the standards is prohibitive.

FIG. 2Dshows a general MPEG bitstream with decodable units. The bitstream includes, at the sequence layer54, a sequence header54A followed by sequence data54B. The sequence layer54is a decodable unit. The sequence data54B includes the picture layer58which includes a plurality of pictures denoted as picture1, picture2, picture3, . . . , picture (N−1) and picture N. Each picture is a decodable unit. Each picture includes a picture header58A and picture data58B. The picture data58B includes the slice layer60. The slice layer60includes a plurality of slices denoted as slice1, slice2, slice3, slice (M−1) and slice M. Each slice is a decodable unit. The slice includes a slice header60A followed by slice data60B. The slice data60B of a slice includes the macroblock layer62. The macroblock layer62includes a plurality of macroblocks denoted as MB1, MB2, MB3, . . . , MB (P−1) and MB P. Each macroblock is a decodable unit. Each macroblock includes a MB header62A and MB data62B. Some decodable units are dependent on another decodable unit. Thus, prioritization will take into consideration dependent decodable units. Moreover, one or more of the decodable units in each layer are divisible.

FIG. 3Ashows a block diagram of a power management module100and video encoder and decoder engines102and104. The power management module100has a multi-level low power mode set generator114. The multi-level mode set generator114has a plurality of low power modes arranged in accordance with the hierarchical (tiered) layers of the MPEG format. The plurality of low power modes are based on prioritized power management (PM) sequences of decodable units that may be selectively decoded for improved granularity and/or visual quality at each layer. Granularity may refer to the extent of parsing or decoding operations that can be executed to maximize the resulting visual quality for a given power consumption target. PM sequences are sequences of decoding or parsing operations that facilitate power management. PM sequences attempt to maximize visual quality for a given power through look-ahead processing of selective decode and/or parsing operations. The multi-level low power mode set generator114has a plurality of layer modes. In this configuration, the plurality of layer modes includes a TL mode, a VS/PL mode and a SL/MB mode. As can be appreciated, the techniques described herein are not limited to the MPEG format but may be used with other video compression and/or transport protocol formats.

In one embodiment, information from a data stream including video data is extracted and compiled and based on this information, the sequences of decoding and parsing operations for the data stream that facilitates power management (PM sequences) is prioritized.

In another embodiment, the prioritization is based on look-ahead processing of at least one of decoding and parsing operations. In yet another embodiment, projections of at least one of power and computational loading for each of the prioritized PM sequences is calculated. In another embodiment, the prioritizing of power management sequences is based on at least one of visual quality and granularity.

The embodiments further comprise generating a hierarchical list of low power modes or quality modes to selectively decode the prioritized power management sequences, based on the prioritization. Different low power modes or quality modes correspond to different degrees of visual quality. The selection of a low power mode may be in response to available power or computational loading. In addition, the selective decoding of one or more of the prioritized power management sequences may be in response to the selected low power mode. In another embodiment, the selective decoding may be based on calculating projections of at least one of power and computational loading for the prioritized power management sequences.

In the exemplary configuration, degrees of redundancy indicated by prediction modes, for example, yields a graduated set of layers which in turn can be mapped to a graduated set of low/reduced power operational modes. One format using H.264 prediction modes is based on the fact that the level of redundancy in video in decreasing order corresponding to inter and intra prediction modes includes: Skip, Direct, Inter, and Intra prediction modes. The order of modes also corresponds to differing degrees of visual quality when compromised (when inaccuracies are introduced in decoding and reconstruction of MBs corresponding to these modes). These concepts can be extended to other video coding standards and formats.

To exploit the redundancy in video toward power optimized video processing, several aspects involving the decoder engine104only, encoder engine102only or coordinated across the encoder and decoder engines may be employed for power load management. In the case of a decoder engine only (DO) solution, the DO solution may be applied during decoding or rendering at the device10and are encoder agnostic. The solutions may be are divided into conformant and non-conformant categories. A conformant category solution would output a video stream which maintains standards conformance. Here, strict conformance requirements are to be met. In a non-conformant solution, an advantage of this solution is flexibility and larger reduction (compared to conformant) in complexity for minimal impact to visual quality.

In the case of an encoder engine102only (EO) solution, all complexity reduction methods are incorporated during encoding and are decoder agnostic. In the EO solution all encoding functions are biased from the perspective of processing power. Optionally, a cost function for processing power is included in rate-distortion (RD) optimizations referred to as RD-power optimizations.

In the case of a joint encoder-decoder engine (JED) solution, power reduction methods are incorporated or adopted during the encoding and the decoder engine performs appropriate reciprocal actions to provide increased reduction (in power/load/cost). In the JED solution, the encoder engine is aware of the capabilities of the decoder engine to apply DO solution methods described above and incorporates indicators for appropriate actions in the bitstream (user field or supplemental enhancement information (SEI) messages) or side channel for use at the decoder engine. A previously agreed upon protocol based on a set of power reduction would may be adopted by both encoder and decoder engines for increased reduction in power/load/cost.

DO solutions apply to open ended applications where the decoder engine is unaware of the encoding process. Examples would include Mobile TV, Video-On-Demand (VOD), PMP, etc. The EO solutions would find application in video servers where power friendly bitstreams are required to drive low power devices. The EO solution is also useful in scenarios where multiple coded versions of a source are generated and a network server adaptively selects/switches between them based on network/channel conditions. JED solutions provide the most gain in terms of power reduction for a given quality compared to DO or EO solution methods. The JED solution apply to closed or conversational applications where a communications/control path (real-time or apriori) is possible.

The description below is directed to DO solutions and provides a multi-layer framework configuration to regulate implementation and operational complexity in video decoding. Load management in video decoding and rendering operations are possible where an extended video playback is required by various applications such as Mobile TV, portable multimedia player (PMP), (movie/DVD players), etc. The techniques described herein embodied in the multi-layer framework configuration may be extended to any video or multimedia application.

Load or power management refers to regulation of run-time complexity including but not limited to delays, power consumption and million instruction per second or processor cycles (MIPS) availability. Regulation includes optimizing the user experience, in particular, video quality given the available processing, power and time resources. Regulation may also be done to optimize a quality of experience (QoE) of the user, which may encompass other performance factors besides video quality, such as for example, audio quality, response time, quality of graphics, etc. The multi-layer framework configuration allows such regulation at various levels of granularity for both precautionary and reactionary responses to instantaneous demands on the video decoder implementation by the application(s). Alternate execution or control/data flow paths are recommended based on available information and power (battery) levels or desired visual quality. In view of the foregoing, the description provided herein is primarily directed to DO operations as performed by the decoder engine104. The video decoding by the decoder engine104may be followed by rendering, performed by a rendering stage28A (FIG. 23) in the display processor28. Decoding/rendering does not have to be a serial process. For example, multiple slices could be decoded in parallel, rows of an image may be rendered in parallel or in a waveform fashion which may not be considered serial. Nonetheless, although decoding followed by rendering in serial order is the norm, in one configuration these operations are parallelized. Rendering typically includes post-processing (color-space conversion, scaling, etc.) followed by compositing the image to be display and the display process (transferring to a display buffer, reading from this buffer and writing to display). For the sake of example, decoding is followed by rendering in a serial process and occurs in chronological order (timing is based on decode time stamps for decoding and presentation time stamps for rendering/displaying). However the input to this process (decoding and rendering) is a video bitstream (except maybe in the case of the viewfinder) which is not necessarily prioritized by order of importance in terms of visual quality (i.e. Instantaneous Decoder Refresh (IDR), intraframe (I) and predicted (P) frames are interspersed). Also, the transport layer protocol that delivers the video bitstream to the decoder engine104does so in packets which are presented to the decoder engine104in order of packet/sequence numbers. Processing the bitstream in the received order may result in frame drops and does not allow for throttling the quality of the output video for low power operations (either user-initiated to conserve battery or modulated by the system based on available or allocated power). Lack of processor cycles, MIPS and/or accumulated delays and latencies may result in key frames, typically larger in size, being dropped causing video to stall for long durations.

The encoder engine102acquires or generates and compresses video data in compliance with MPEG standards, H.264 or other standards. The video data is processed to extract a selected portion of video information so that encoding meets image quality, power, and/or computational load requirements of the device10and/or transmission capabilities of an output video interface, bandwidth or other characteristics of the device10or video-enabled device (for example, wireless or wired interface). Additionally, encoding may be such that decoding (by a recipient) meets quality, power and/or computational requirements and capabilities of the recipient's decoder engine or receiver14.

FIG. 3Bshows a block diagram of the decoder engine104for use with the power management module100. The decoder engine104includes a standard (normal) sequence decoding processing unit105to decode the bitstream when power management or a low power mode is not necessary. The decoder engine104also includes a transport layer (TL) parser and processing unit106, a video sequence/picture layer (VS/PL) parser and processing unit108, a slice/MB layer (S/MBL) parser and processing unit110, a block layer parser and processing unit112. In the exemplary configuration, power and computation load management at the block layer64is not described.

As will be seen from the description below, the TL parser and processing unit106parses and processes the transport layer52. The VS/PL parser and processing unit108parses and processes at least the sequence layer54and the picture layer58. The combination of the sequence layer54and the picture layer58is herein after referred to as a video sequence/picture layer (VS/PL)70. However, the VS/PL parser and processing unit108may also parse and process the GOP layer56or some other parser and processing unit may be employed for the GOP layer56. Thus, the line from the reference numeral70to the GOP layer56is shown in phantom. The S/MBL parser and processing unit110parses and processes the slice layer60and the macroblock (MB) layer62. The block layer parser and processing unit112parses and processes the block layer64in order to decode the video or programming in the MPEG format.

One or more of the parser and processing units106,108,110, and112may be employed to operate in parallel, separately or in a combined relationship to carryout the power and computational load management functions described herein. Furthermore, one or more of the power and computational load management functions of the parser and processing units106,108,110, and112may be omitted. Nonetheless, in the exemplary configuration, the parser and processing units106,108and110are selectively actuated as necessary to provide a tiered power and computational load management function which controls the visual quality, trading visual quality for power loading and granularity in any one of the tiers so as to also maintain or enhance the user's experience while using power efficiently.

FIG. 4shows a flowchart of a process120for projecting power and computational loads for decoding prioritized power management (PM) sequences of decodable units. In order to prioritize the bitstream and the consequent power management (PM) sequences of decodable units for selective decoding, a three-phase (3-phase) process120is provided for each hierarchical (tiered) layer to provide hierarchically arranged low power operational modes. The process120relies on non-causality in the video bitstream and its efficiency depends on the amount of look-ahead (decoder engine's input buffer depth).

The process120begins at block122where parsing and extracting layer information takes place. Block122is followed by block124where those PM sequences (not to be confused with the sequence layer) of decodable units requiring decoding are prioritized. For illustrative purposes, the prioritized PM sequences of decodable units are shown in the form of a list, as will be described in more detail later. The term “prioritized PM sequences of decodable units,” will hereinafter sometimes be referred to as “prioritized PM sequences.” However, each sequence includes one or more divisible decodable units. A decodable unit comprises one or more or groups of a picture, a slice, and macroblocks, as will be seen from the following description.

Block124is followed by block126where the power and computation load for the prioritized PM sequences are projected. In the exemplary configurations, the computational load is projected as a function of the number of million instructions per second (MIPS). A corresponding MIPS requires a projected or predetermined power.

At block126, there is a correlation of the prioritized PM sequences to corresponding MIPS and, subsequently, the power required to decode one or more of the PM sequences. Blocks122and124are described below with respect also the H.264 standard. Block126may use the results from power analysis corresponding to typical scenarios (e.g. test bitstreams) and, optionally, feedback driven training or run-time updates for use in the projection. Once a bitstream is known, the power and computation loads (processing power) may be projected to inform the user whether there is not enough power in the device10to decode the bitstream to completion. Thus, if the power (battery or electrical power) would be depleted prior to the bitstream being completely decoded (such as during playback), the user has an option to select a power mode that would allow the bitstream to be completed.

As previously mentioned, the3-phase process120can be repeated for the different layers in the compression data format. After each block126(corresponding to different layers), a hierarchical set of low power modes are generated for the decoder engine104to utilize for power and computational load management of the decoding operations. This can happen on the fly or it may be predetermined for a set of appropriately chosen bitstreams and the decoder calibrated/programmed in advance prior to real-time operations.

FIG. 5shows a transport layer (TL) parser and processing unit106. The TL parser and processing unit106includes a TL information extractor and compiler150(FIG. 6), a TL prioritized PM sequences generator152(FIG. 8) and a TL decoding MIPS and power projector154(FIG. 9). The transport layer (TL) parser and processing unit106will carryout the three-phase process120for use in power and computational load management of the decoding operations for the transport layer52.

FIG. 6shows a TL information extractor and compiler150. The TL information extractor and compiler150depends on the transport protocol over which the video bitstream is received. An example, of a portion of a received time slice190is shown inFIG. 7. In the exemplary configuration, the various information in the bitstream can be extracted and compiled by the TL information extractor and compiler150. The TL information extractor and compiler150will parse the received time slice190shown inFIG. 7. The TL information extractor and compiler150includes a random access point (RAP) extractor160, a state information extractor166, a bitstream characteristics extractor176and a transport layer information compiler186. The RAP extractor160may extract information162having location, size, presentation time stamp (PTS), etc. of packets/slices flagged as acquisition. The RAP extractor160may also extract the sequence of RAPs164in the transport header (e.g. entry-point header in the real-time transport protocol (RTP) payload format). In the exemplary configuration, the extracted state information, by the state information extractor166, includes information regarding a channel change170and a user viewing preferences172(such as in broadcast or mobile TV applications). The extracted state information166may also include a change in application174(e.g. resolution/preview mode or picture-in-picture mode), etc.

The bitstream characteristics extractor176extracts a bitrate178, a frame rate180, a resolution182, an application (stored vs. streaming)184, etc. In the case of the bitrate178, values are readily available in some cases (e.g. MPEG file format). In other cases, the bitrate is computed, such as by the transport layer information complier186, based on the size of bitstream over a second indicated by time stamps after transport headers and padding are removed. The TL layer information extractor and compiler150includes a TL information compiler186to compute the information that is not directly extractable from the transport layer of the received time slice. Example calculations for a packet size and bitrate are described in relation toFIG. 9.

FIG. 7shows the received time slice190in this example with RAP1191, RAP2196, . . . RAPN198in the received data. (It could optionally be a section of the bitstream extracted from a stored file). Each RAP, such as RAP1191, has a header192followed by a payload193having data pertaining the coded frame that is the random access point, such as an I-frame. The header includes a plurality of fields, one of which is a PTS1interval194. The absolute RAP locations (packets) and PTS values from the PTS1interval194for each RAP are computed (e.g. in RTP, derived from random access (RA) count and reference time and PTS offset). RAP2196has a header followed by a payload having data pertaining to the coded frame that is the random access point, such as an I-frame. The header includes a plurality of fields, one of which is a PTS2interval.

An interval denoted as RAP-GOP is defined as a group of pictures that begins with a RAP frame until the next RAP frame. At the transport layer, the RAP is a decodable unit. Furthermore, the RAP-GOP may be a decodable unit for the transport layer. Based on the application, more data is retrieved or requested if needed. For example, during playback of stored video, it may be possible to seek through file format headers for a few seconds (2-5 seconds) worth of data to assess the bitrate and frame rate. Then, based on the available power, decide on decoding all of the data or decode toward a reduced frame rate.

The received time slice190may be a superframe for MediaFLO™ or a time slice for digital video broadcast (DVB) such as DVB-H (where H stands for handheld).

FIG. 8shows TL prioritized PM sequences generator152. The information extracted above is processed to create a list of TL prioritized PM sequences of decodable units200. The TL prioritized PM sequences generator152derives absolute parameter values. Assume at the transport layer52, a plurality of packets RAP1, RAP2, . . . RAPN191,196and198with corresponding GOPs have been received. Thus, the TL prioritized PM sequences of decodable units to be decoded by the decoder engine104begins with block202. Here the decodable units are RAP1at block202, the rest of RAP-GOP1at block204, RAP2at block206, the rest of RAP-GOP2at block208. The prioritizing of TL prioritized PM sequences continues until the decodable unit RAPNat block210and the rest of the RAP-GOPNat block212. The above description of TL prioritized PM sequences of decodable units is just one example of an arrangement of sequences.

FIG. 9shows a TL decoding MIPS and power projector154. At the transport level52, a first level of power/computation load reductions are possible. This level may provide coarse power/computational load reductions. For example, for the lowest power mode setting or when a battery level of device10has been depleted to <10%, only the RAP packets (denoted by blocks202,206and210) would be decoded and, while rendering by the rendering stage28A, optionally, the graphics processing unit34may be triggered to create transition effects between the I-frames. The transition effects provide a low cost “video” instead of a slide show effect. Based on the low power mode, other video compensations may be used to compensate for skipped decodable units. For example, image morphing may be employed. Another example of compensation may employ optical flow.

The TL decoding MIPS and power projector154generates data representative of a projection (column4) for the MIPS to decode one, more or all of the PM sequences of decodable units in the list of TL prioritized PM sequences200. For illustrative and descriptive purposes only, a MIPS projection table230is shown. The table has a plurality of columns. In column C1, the transport layer information or the TL prioritized PM sequences of decodable units are itemized. In column2, the packet size to decode the some or all of the decodable units is identified. In column C3, the bitrate calculation is identified. In column C4, the projected MIPS to decode the decodable units is provided. As can be appreciated, the bitrate in column C3and the packet size in column C2may have been derived during the prioritization phase.

In this example, row R1identifies a first decodable unit in the list of TL prioritized PM sequences of decodable units200. In this instance, the first decodable unit is RAP1. The packet size of RAP1can be calculated based on the extracted and compiled information from the TL extractor and compiler150. In the exemplary configuration, the decode packet size for RAP1corresponds to the size of the transport packet for RAP1—the size of (transport header192plus the payload193). The decode packet size for RAP2corresponds to the size of the transport packet for RAP2—the size of (transport header plus the payload). Likewise, the decode packet size for RAPNcorresponds to the size of the transport packet for RAPN—the size of (transport header plus the payload). In row RN+1 (the last row) corresponds to the entire received time slice, such as slice190. Thus, a projection of the MIPS is calculated for each decodable unit and all decodable units at row RN+1 for the entire transport layer52of the received time slice190.

In column3, the bit rate is calculated or extracted. In this case, the bitrate is calculated based on the sizes of RAP, GOP1, PTS2, PTS1according to the size of the interval (RAP-GOP1) divided by the size of the interval (PTS2−PTS1) or (PTS2minus PTS1). The bitrate for RAP2, . . . RAPNis calculated in a similar manner as RAP1. In row RN+1, the bitrate is the size of the received time slice/interval (PTS2−PTS1).

In column4, in row R1, the projected MIPS to decode RAP1has two values. The first value is a function of the I-frame size for RAP1. The second value is a function of that portion of the bitstream of size (RAP-GOP1) for the given codec. The information for the projection of the MIP is available from the transport headers (RAP and the corresponding PTS). Thus, the decodable units divisible and are not fully decoded when projecting the MIPS. Instead, only the header or a portion thereof needs to be decoded to extract the necessary information, as will be described in more detail below. In row RN+1, the projected MIPS to decode the entire time slice is projected according to bitstream size (for the time slice) for the given codec. It should be noted that the MIPS projection to decode for the specified quantity is a function of power profiling and analysis.

For each of the MIPS projection in column C4, a corresponding power requirement can be determined. The corresponding power can be calculated as needed or may be pre-stored in a lookup table. This will generally complete the third phase of the three-phase process120.

FIG. 10illustrates a process240for decoding with power and computational load management. Given the MIPS requirement to decode one or more of the decodable units, and the available MIPS at a given instant, (or power requirement vs. available power/amps), the decision to decode all or part of the received time slice190can be made. The process240is illustrated with the third phase of process120shown in phantom. The third phase of the process120provides the necessary projections for computational loads and/or power necessary to decode the transport layer52. Thus, at block242, the MIPS are projected. At block244, the power corresponding to the projected MIPS is determined. While the exemplary configuration provides for a MIPS and power relationship, other values which affect the power and computational loads may be employed.

Block244ends the third phase. Block244is followed by blocks246where the available MIPS (computational load) for a given instant is determined. Block244is also followed by block248where the available power is determined for a given instant. Blocks246and248are shown in parallel. Nonetheless, in various configurations, the blocks of the process240and other processes described herein are performed in the depicted order or at least two of these steps or portions thereof may be performed contemporaneously, in parallel, or in a different order.

Block246is followed by block250where a determination is made whether the projected MIPS is greater than the available MIPS. If the determination is “No,” meaning the available computation load at the instant is sufficient, then all of the transport layer52can be decoded at block254. However, if the determination at block250is “Yes,” meaning the available computation load is insufficient, then part of the transport layer52can be decoded in accordance with any of the modes identified in the list of low power mode settings260(FIG. 11) at block256.

Block248is followed by block252where a determination is made whether the projected power is compared to the available power. If the determination at block252is “No,” meaning that the available power is sufficient, then all of the transport layer52may be decoded. However, if the determination at block252is “Yes,” meaning the available power is insufficient, then part of the transport layer52can be decoded at block256in accordance with any of the modes identified in list of low power mode settings260(FIG. 11). All of the transport layer52would be decoded if both conditions from blocks250and252are No. The transport layer52would be decoded in part for all other cases. The blocks248,252are shown in phantom to denote that they are also optional.

FIG. 11shows a multi-layer low power mode set generator114during the TL mode. The multi-layer low power mode set generator114generates a list of selectable low power mode settings260. In the exemplary configuration ofFIG. 11, there are a plurality of transport layer low power modes denoted as mode1in row R1, mode1A in row2and mode2in row3. The transport layer mode1corresponds, for example, to a slideshow using all the RAPs (hereinafter referred to as “SS-RAP”). The transport layer mode1A corresponds to the SS-RAP with transition effects by the rendering stage28A. Thus, mode1A differs from mode1in that mode1A provides an enhanced visual quality over mode1. The transport layer mode2corresponds to selectively decoding RAP-GOPs based on the available power. The list in column C2would provide the necessary instruction to cause the decoder engine104to selectively decode one or more of the decodable units at the transport layer52.

The power management module100during the TL mode makes a determination based on the projected MIPS and/or power which one low power mode1,1A or2can be afforded to the user for the decoding of the bitstream. Mode2may be selected if there is available power which may be further conserved based on managing the power of other layers of decodable units, as will be described in relation to the video sequence/picture layer.

If TL mode1A is selected, normal decoding of the SS-RAPs (I-frames) with transition effects takes place. However, if TL mode1is selected, the power management module100may proceed to VS/PL mode3for further upgrades in visual quality.

FIG. 12shows a video sequence/picture layer (VS/PL) parser and processing unit108. The VS/PL parser and processing unit108includes a VS/PL information extractor and compiler280(FIG. 13), a VS/PL prioritized PM sequences generator282(FIG. 15) and a VS/PL decoding MIPS and power projector284(FIG. 17). The VS/PL parser and processing unit108will carryout the three-phase process120for use in power/computational load management of the decoding operations for the VS/PL70.

FIG. 13shows a VS/PL information extractor and compiler282. The VS/PL information extractor and compiler282depends on the VS/PL format of the video bitstream. An example, of a received time slice330according to the VS/PL70is shown inFIG. 14. Based on the video codec (encoder engine and decoder engine), information is extracted at the sequence layer54. In the case of MPEG-2 and MPEG-4, the video sequence layer parameters are extracted. This requires an interface into the video decoder engine104. The extraction will be described later in relation toFIG. 27 and 28.

If some parameters listed at the transport layer52could not be retrieved (e.g. I-frame locations or packet ID), such information may be extracted at the sequence layer54. The VS/PL information extractor and compiler280extracts the I-frame location284and the packet ID286. The VS/PL information extractor and compiler282also extracts a profile290, a level292, and parameter constraints (constrained_set_flags)294from the sequence parameter set (SPS) such as for the H.264 standard or the sequence layer54. The picture parameter set (PPS) may also be used.

The VS/PL information extractor and compiler282may also extract or compile picture information296. The picture information may include a number of reference frames298, resolution300, frame rate302(if not already retrieved), display parameters (VUI), etc. to assess the computational load required to decode/process the data. Additional information includes information regarding reference location304, reference picture size306, PTS and reference picture information308. Non-reference picture locations310, non-reference picture size312, non-reference picture information314and the PTS also may be extracted or compiled. An information that is compiled is complied by the information compiler316. In order to extract the VS/PL information, the sequence header and all picture headers are decoded only. The payload of the picture is left un-decoded, as will be described in more detail inFIGS. 27 and 28.

FIG. 15shows a VS/PL prioritized PM sequences generator282. At the VS/PL prioritized PM sequences generator282absolute parameter values are derived from the extracted information. The list of VS/PL prioritized PM sequences of decodable units360is populated with more details for improved granularity. The prioritization is similar to that discussed in relation toFIG. 8. At this level or layer, however, the plurality of packets RAP1, RAP2, . . . RAPN(blocks362,382,388) identified in the transport layer52are further qualified or prioritized based on the type of I-frame such as IDR and I-frame in the case of H.264. Alternatively, all I-frames are identified using the picture header information and are then prioritized.

In the exemplary configuration, the block or interval RAP-GOP1364is further sub-divided into other VS/PL decodable units. These VS/PL decodable unit are further prioritized such that IDRs (or I-frames at the beginning of a closed GOP in MPEG-2) are followed by a non-IDR I-frames (open GOP). Hence, prioritization may be set so that IDR-frames366are followed by I-frames368. I-frames366are followed by P-frames370which are followed by reference B-frames372. The reference B-frames372are then followed by non-reference B-frames denoted as b-frames374.FIG. 14shows a received time slice330indicating frame types (P, B and b).

Thus, the VS/PL prioritized PM sequences to be decoded by the decoder engine104begins with RAP1at block362which is followed the RAP-GOP1at block364. The RAP-GOP1is further prioritized according to blocks366,368,370,372and374. The interval corresponding to the RAP-GOP1can be further prioritized based on the b-frame. In the exemplary configuration, the VS/PL prioritized PM sequence is further prioritized using size information for b-frames at blocks376-380. For example, b-frames having a size information larger than the FRUC threshold, denoted as, FRUC_THR, (block376) may have a higher priority than those b-frames with a size which is less than the FRUC threshold. Additionally, b-frames smaller than a Drop threshold, denoted as DROP_THR, may be flagged and dropped entirely with no FRUC. Thus, at block378, the prioritization criteria may be set as DROP_THR<b<FRUC_THR. At block380, the prioritization criteria may be set as b<DROP_TH. These thresholds can be mapped to percentage reduction in processing cycles/power required.

Block382sets the prioritization for decoding RAP2. Block384is followed by block382where the prioritization for the rest of the RAP-GOP2is prioritized similar to blocks366,368,370,372,374,376,378and380above. The prioritization of the VS/PL prioritized PM sequences continue at block386until block388for prioritizing the decoding of RAPN. Block388is followed by block390where the rest of the RAP-GOPNis prioritized for decoding.

Depending on a status of the computational load, the sequence of decoding operations may be reduced or modified through elimination of an appropriate number of low priority sequences or selectable decodable units.

FIG. 16shows a flowchart of a process400to project MIPS by the VS/PL by the VS/PL decoding MIPS and power projector284. The process400begins with block402where MIPS to decode an IDR-frame sizes are determined. Block402is followed by block404where the MIPS to decode all I-frames sizes are determined. Block404is followed by block406where the MIPS to decode all the P-frames sizes are determined. Block406is followed by block408where the MIPS to decode all B-frames sizes are determined. Block408is followed by block410where the MIPS to decode all b-frames sizes is determined with various conditions. For example, if some of the b-frames (such as a b1frame and b2frame) are dropped, the projected MIPS to decode is set to 0.

FIG. 17shows a VS/PL decoding MIPS and power projector284. At the frame level; frame type (such as IDR, I, P, B, b . . . ), size (306or312) and the frame rate302are key factors (other qualifiers may be included), that can be used to assess the amount or proportion of processor cycles required to decode them. Power profiling and analysis using specific test bitstreams can be used to derive a relationship between the amount of processor cycles and frame size based on frame type (IDR, I, P, B, b . . . ). These relationships may be arranged in a lookup table for later use in the MIPS and power projections. Other conditions may be fixed during this analysis and extrapolated later. For example, the mapping may be derived for1-reference picture scenario and relative complexity vs. 5-reference pictures may be extrapolated based on independent analysis. In the case of the H.264 standard, frame level information is not available until slice headers are parsed.

InFIG. 17, the VS/PL decoding MIPS and power projector284generates a list of projected MIPS to decode440. The list is generated for illustrative and descriptive purposes. At row R1, the VS/PL decoding MIPS and power projector284projectors for the IDR-frames based on the size(IDR) of each IDR. At row R2, the VS/PL decoding MIPS and power projector284generates the projected MIPS for all I-frames based on a sequence of sizes I-frame sizes (size(I1), size(I2), . . . ). At row R3, the VS/PL decoding MIPS and power projector284generates the projected MIPS for all P-frames based on P-frame sizes (size(P1), size(P2), . . . ). At row R4, the VS/PL decoding MIPS and power projector284generates the projected MIPS for all B-frames based on B-frame sizes (size(B1), size(B2), . . . ) At row R5, the VS/PL decoding MIPS and power projector284generates the projected MIPS for all B-frames (non-reference B-frames) based on B-frame sizes (size(b1), size(b2), . . . ). When projecting the MIPS for all b-frames, a determination is made whether b1and b2are dropped. If so, then the projected MIPS is set to zero (0). There is also a projection in relation to FRUC in place of b1, b2, . . . , etc.

In relation toFIGS. 10 and 17, for each of the MIPS projections in the list of projected MIPS to decode450, a corresponding power requirement is applied. Given the MIPS requirement, and the available MIPS at a given instant, (or power requirement vs. available power/amps), the decision to decode all or selected frames (part), which are decodable units, can be made in a similar manner as described above in relation toFIG. 10.

At the end of sequence/picture level processing, medium granularity power reduction modes are possible (reductions from 0-60% in steps of 5% are possible—assuming that an I-frame typically constitutes 30% of bits in a GOP and the number of bits is proportional to the MIPS requirement). Depending on feedback on current status of processor load and power levels, the sequence of operations is shortened through elimination of appropriate number of low priority entities. Modes possible at the sequence/picture layer are listed inFIG. 18in order of increasing power requirements.

FIG. 18shows a multi-layer low power mode set generator114during the VS/PL mode. The multi-layer low power mode set generator114generates a list of low power modes450. The list is generated for illustrative purposes. The list includes a VS/PL Layer Mode3corresponding to instructions to decode a Slideshow using RAPs and all I-frames. Thus, if mode1A is selected, the power management module100would evaluate if additional MIPS are available such that all of the I-frames may also be decoded for improved visual quality or granularity. VS/PL Layer Mode3is followed by a VS/PL Layer Mode4A corresponding to instructions to decode based on a reduced frame rate using all I-frames and P-frames only. VS/PL Layer Mode4A is followed by VS/PL Layer Mode4B corresponding to instructions to decode based on a reduced frame rate using all I-frames and P-frames only with selective FRUC in place of B-(and b) frames. At mode4B, the I and P-frames are decoded using normal decoding. However, the B-frames are not decoded. Instead, selective FRUC is substituted in place of each B or b-frame for all B or b-frames. VS/PL Layer Mode4B is followed by VS/PL Layer Mode4C corresponding to instructions to selectively decode RAP-GOPs based on available power such as the I-frames and P-frames as above. However, as an alternate operation, the selective FRUC is used in place of each B or b-frame for a selective number of B or b-frames. VS/PL Layer Mode4C is followed by VS/PL Layer Mode4D corresponding to instruction to decode based on a reduced frame rate (higher than mode4C) using all I and P-frames. All B-frames may be also included. Alternately, and the selective FRUC is used in place of each B-frame (optionally for selective number of B-frames) and no operation is used for the b-frames. Alternately, the b-frames may be skip or by-passed. VS/PL Layer Mode4D is followed by VS/PL Layer Mode5corresponding to instructions to decode all received frames (I, P, B and b).

If the VS/PL Layer mode3or5was selected, a further alternate operation would be to replace with skip macroblocks (MBs). Furthermore, from mode2, further enhanced visual quality or granularity may be achieved by the refinements afforded by the modes4A-4D and5.

FIG. 19shows a slice/macroblock layer (S/MBL) parser and processing unit110. The S/MBL parser and processing unit110includes a S/MBL information extractor and compiler460(FIG. 20), a S/MBL prioritized PM sequences generator462(FIG. 21) and a S/MBL decoding MIPS and power estimator464. The S/MBL parser and processing unit110will carryout the three-phase process120for use in power/load management of the decoding operations for the S/MBL72.

FIG. 20shows a S/MBL information extractor and compiler460. The S/MBL information extractor and compiler460depends on the protocol or standard for which the video bitstream has been compressed. Here, the S/MBL information extractor and compiler460extracts slice information470and MB information472. Slice information470and MB headers are parsed for the pictures corresponding to those identified in the prioritized sequence fromFIG. 15. A select portion of the frames from the prioritized sequence (FIG. 15) in the previous layer VS/PL70might be flagged to be decoded. Note that, decoding may continue for all pictures if finer granularity of power management is required. Thus, only the slice headers and MB headers are decoded.

If headers are detected to be in error, coefficient/MB data for the MB or entire slice may be discarded. Optionally, zero MV concealment may be applied and refined later with more sophisticated error correction (EC). The MB information472includes MB type474, motion vectors (MVs)476, mode478, size480, other information482and MB maps per frame484.

An exemplary MB maps per frame is described in patent application Ser. No. 12/145,900 filed concurrently herewith and is incorporated herein by reference as if set forth in full below.

FIG. 21shows a S/MBL prioritized PM sequences generator462. The list of S/MBL prioritized PM sequences490uses the slice information and MB maps to estimate the complexity of each frame in the prioritized list inFIG. 15. In one configuration, only P-frame slices at block492and B-frame slices at block502are further prioritized. At the previously layer, all I-frames are to be decoded for any selected mode in the VS/PL. The P-frame slices are further prioritized based on the ROI MBs per slice at block494and Non-ROI MBs with mode smoothing at block496. The mode smoothing is further prioritized according to forced uniform motion at block498and forced P-skips at block500. The B-frame slices are further prioritized based on the ROI MBs per slice at block504and Non-ROI MBs with mode smoothing at block506. The mode smoothing is further prioritized according to forced uniform motion at block507and forced B-skips at block508.

Mode smoothing may be applied to group MBs with similar characteristics. For every 3×3 or 5×5 window of MBs, the windows of MBs are assessed for uniformity of modes. Outliers (MB with mode that is different from remaining MBs) in the window are identified. If outliers are marginally different, they are forced to a mode of the window. Otherwise, the mode for the outlier is maintained. For example, if in a 3×3 MB window, one MB is inter mode while the others are skip and if the residual (indicated by CBP or MB size) of the inter MB is less than a Skip_threshold, the MB is forced to skip mode. After mode smoothing, the proportion of skip vs. direct/inter mode MBs is computed and included as a factor of complexity. Additionally, connected regions of skip MBs may be combined as tiles through MB dilation and MB erosion (as in the patent application Ser. No. 12/145,900. Tiles can then be qualified as skip/static, non-static, uniform motion, region-of-interest (ROI, based on relative MB/tile size) etc. In the case of uniform motion tile, MVs of the MBs may be quantized and the tile may be forced to one MV (note that this may be done provided the residual/CBP of these MBs are zero or almost zero). Another option is where only non-static or ROI tiles are decoded and rest are forced to be skipped. In this case, some of the non-ROI MBs may be of modes other than skip but will be forced to skip such as in blocks500and508.

FIG. 22shows a multi-layer low power mode set generator114during the S/MBL mode. The multi-layer low power mode set generator114generates a hierarchical list of low power modes650. The ability to manipulate which of the MBs in a frame and which of the received frames are processed provides a significant level of granularity in managing the decoding and rendering process. In addition, the above described MB level power optimization may be performed during decoding (on-the-fly). Again, detailed profiling and power analysis is required to map the proportion of reduction in power to the corresponding low power mode.

The examples of modes are described as S/MBL mode6A,6B,6C,7A,7B,7C and8. In mode6A, the I-frames are decoded per normal decoding of the I-frames. However, additional alternate operations may take place to improve visual quality or granularity as power permits. For example, in mode6A, the P-frames with non-ROI MBs may be forced to P_Skips and B- and b-frames may be forced to selective FRUC. In mode6B, I-frames are decoded as per normal decoding of I-frames. Alternate operations in mode6B may include forcing P-frames with mode smoothing to P_Skips and B- and b-frames to selective FRUC. In mode6C, the I-frames are decoded according to the normal decoding process. However, as an alternate operation, P-frames with mode smoothing may be forced uniform motion and B- and b-frames may be forced to selective FRUC. In mode7A, the I and P-frames are decoded according to the normal decoding process. However, as an alternate operation, B-frames with non-ROI MBs are forced to Skips. In mode7B, the I and P-frames are decoded according to the normal decoding process. However, as an alternate operation, B-frames with mode smoothing are forced to Skips. In mode7C, the I and P-frames are decoded according to the normal decoding process. However, as an alternate operation, B-frames with mode smoothing are force uniform motion. In mode8, all received frames (I, P, B and b) are decoded.

FIG. 23shows a high level block diagram of the power management operations. The block diagram includes the decoder engine104in communication with a rendering stage28A of the display processor28. Thus, the power management (PM) module100processes the bitstreams at the TL mode, VS/PL mode and the S/MBL modes. The PM module100controls the decoder engine104to decode the decodable units according to the low power mode selected. The processing required during rendering is also derived from the prioritized sequence of operations from the framework in any of the low power mode of operations described above. Furthermore, the output of the decoder engine114may be sent to other devices, memory, or apparatus. The output from the decoder may be forwarded to another video enabled apparatus for eventual storage or consumption (display). The graphics processing unit34is also in communication with the display processor28

FIG. 24illustrates an exemplary standard (normal) decoding process700in sequential order. The process700is also described in relation toFIG. 2D. Beginning at the video sequence and picture layer, the standard (normal) decoding process700will decode a sequence header54A at block702followed by decoding a picture header58A of picture1at block704. After decoding the picture header of picture1, the picture data58B is decoded which includes the slice and macroblock information. Thus, when the picture1data decoded, all of the slice units are decoded denoted by slice1-M. Since each slice is decoded similarly, only one slice will be described in more detail.

When slice1is decoded, the slice header60A of slice1of picture1is decoded at block706. Then, the MB header62A of macroblock (MB)1of slice1of picture1is decoded at block708. After, the MB header62A of the MB1is decoded, the related MB data for MB1of slice1of picture1is decoded at block710. Then the next macroblock is obtained. Thus, the MB header62A of macroblock (MB)2of slice1of picture1is decoded at block712. After, the MB header of the MB2is decoded, the related MB data62B for MB2of slice1of picture1is decoded at block714. The decoding of the macroblock header followed by the related MB data62B continues for all remaining MBs in the slice. In this example, there are N MBs. Thus, the decoding of slice1of picture1would end with decoding the MB header for MB N of slice1of picture1at block716followed by decoding the related MB data62B of MB N of slice1of picture1at block718.

Thus, the process700would continue decoding the picture1information by decoding each of the remaining slices in a similar manner as described above for slide1. In this example, there are M slices. Thus, at block720, the slice M is decoded in the manner as described above in relation to blocks706-718.

Next, the process700would decode the next picture frame such as picture2. To decode picture2, the process700would decode the picture header58A of picture2at block722to derive the locations for slices1-M, Thus, at block724, the slice1is decoded in the manner as described above in relate to blocks706-718. All remaining slices of picture2are decoded similarly. Thus, at block726, the slice M is decoded in the manner as described above in relation to blocks706-718to complete the decoding of picture2.

The process700repeats the decoding of all the picture frames sequentially in a similar manner until the last picture Z. In this example, there are pictures1-Z. Hence to decode the last picture, picture Z, the process700would decode the picture header58A of picture Z at block728followed by slice1decoding at block730. Each slice is decoded in picture Z until slice M at block732.

FIG. 25illustrates a flowchart of a TL decoding process800with power management operations. The process800begins with block802where TL information extraction and TL prioritization of the PM sequences of decodable units takes place, as described in relation toFIGS. 6an8. Block802is followed by block804where the MIPS and/or power loading are projected for the TL prioritized PM sequences. Block804is followed by block806where the MIPS for the TL low power mode set is projected. Block806is followed by block808where a determination is made whether there is enough power to decode the bitstream. If the determination is “YES,” then the normal decoding process700may take place at block810in accordance with the procedure described in relation toFIG. 24. However, if the determination is “NO,” then as an option, the user may be notified that there is insufficient power such as to playback a video at block812. The user would be given low power mode options corresponding to modes1,1A and2from which to select. Alternately, the low power mode may be selected automatically. Block812is followed by block814where a TL low power mode is selected whether by a user or automatically. The flowchart ofFIG. 25proceeds toFIG. 26.

FIG. 26illustrates a flowchart of a VS/PL decoding process900with power management operations. The process900begins with performing at block902the VS/PL information extraction and VS/PL prioritizing the PM sequences, such as described inFIGS. 13 and 15. At block904, the MIPS for all frames types (decodable units) are projected as described in related toFIGS. 16 and 17. At block906, the MIPS and/or power loading for each VS/PL low mode set, as shown inFIG. 18, is projected based on the VS/PL prioritized PM sequences. Based on the projected MIPS, PM sequences are grouped together based on visual quality and granularity. Some of the sequences (such as all sequences) may not be decoded because of insufficient power. Thus, at block908, a ranked sub-set of low power modes from the VS/PL low power mode set that is below the maximum available power may be generated. The ranking is a function of improved visual quality and/or granularity. At block910, optionally, the user may be notified of the limited power and provided a selection of low power mode options. At block912, the best ranked low power mode of the sub-set may be selected or the low power mode selected by the user. At block914, based on the selected VS/PL low power mode, decoding begins by interjecting the decoding operations back into the normal decoding process700ofFIG. 24as appropriate.

In one configuration, after each frame is decoded based on one selected low power mode, the MIPS may be re-projected. Thereafter, the next frame or other un-decoded frames in the bitstream may be decoded using a different selected mode. Thus, the low power mode may be dynamically changed or generated on the fly during the decoding of the bitstream.

FIG. 27illustrates a block diagram of a VS/PL information extraction protocol902A which diverges from the normal decoding process700ofFIG. 24. InFIG. 27, the VS/PL information extraction protocol902A will decode the sequence header54A at block950. Thus, the location of the pictures1-N is derived as denoted by the arrow above each block denoted for pictures1-N. At block952, the picture header58A of picture1is decoded. At block954the picture header58A of picture2is decoded. All picture headers are decoded. At block956, the picture header for picture Z (the last picture) is decoded. Thus, the decoding of the picture headers58A allows the PM sequences of decodable units to be derived for a particular bitstream and the MIPS projected for the PM sequences of decodable units.

FIG. 28illustrates a block diagram of the VS/PL decoded units from the bitstream in accordance with the VS/PL information extraction protocol902A. The sequence header54A is shown hatched to denote that the sequence header54A has been decoded. Furthermore, the picture headers58A for each of the pictures1-N are shown hatched to denote that the picture headers58A have been decoded. The picture data58B remains unhatched to denote that it remains undecoded at this point. The sequence data54B also remains un-decoded. The decoding of the picture headers58A allows the necessary slice locations to be obtained for the slice and macroblock layer without decoding the picture data.

FIG. 29illustrates a flowchart of a S/MBL decoding process1000with power management operations. The process1000begins at block1002with performing the S/MBL information extraction and S/MBL prioritizing of the PM sequences of decodable units, such as described inFIGS. 20 and 21. In one configuration, only information for the P and B-frames are extracted and prioritized. At block1004, the MIPS for each of the S/MBL low power modes are re-projected. At block1006, a ranked sub-set of low power modes from the S/MBL low power mode set that is below the maximum available power is generated. The ranking is a function of improved visual quality and/or granularity. At block1008, optionally, the user may be notified of the limited power and provided a selection of low power mode options. At block1010, the best ranked low power mode of the sub-set may be selected or the low power mode selected by the user. At block1012, based on the selected S/MBL low power mode, P and B-frame slice and MB data decoding begins by interjecting the decoding operations back into the normal decoding process700ofFIG. 24as appropriate. After block1012, the MIPS may be re-projected after one or more frames have been decoded so that the low power mode may be upgraded or downgraded according to the remaining available power.

FIG. 30illustrates a block diagram of a S/MBL information extraction protocol1002A which diverges from the normal decoding process700ofFIG. 24.FIG. 30will be described in conjunction withFIG. 31.FIG. 31illustrates a block diagram of a S/MBL decoded units from the bitstream in accordance with the S/MBL information extraction protocol1002A. The S/MBL information extraction protocol1002A will decode the picture data58B for the first picture1. To decode the picture data58B, only the slice header60A and MB header62B are decoded until a low power mode can be selected. The arrows above the blocks for pictures1-N indicate a location of the picture. The black shading of the arrows denotes the selection of the picture based on a low power mode. The non-shaded arrows denotes a picture that has not been selected. At block1050, the slice header60A for picture1is decoded at block1050. At block1052, the MB header62A of MB1of slice1of picture1is decoded. At block1054, the MB header62A of MB2of slice1of picture1is decoded. All macroblock headers for slice1of picture1are decoded where at block1056, the MB header for MB N of slice1of picture1is decoded. The macroblock data62B is not decoded. The hatching of picture data58B, slice header60A and MB header62A denotes decoding thereof.

The slice header60A of each slice is decoded followed by the decoding of the MB header of each MB of a slice. The, at block1058, slice M (the last slice) of picture1is decoded in a similar manner as block1050-1056. All remaining pictures are decoded in a similar manner. At block1060, the picture Z (the last picture) is decoded as described above in relation to blocks1050-1058. Thus, the decoding of the slice and macroblock headers allows the PM sequences of decodable units to be derived for a particular bitstream and the MIPS projected for the PM sequences of decodable units.

FIG. 32illustrates a block diagram of the final slice and macroblock decoding according to a selected power management mode. The slice data60B and the MB data62B are decoded according to the PM sequences to be decoded for the selected low power mode (such as modes6A-6C and7A-7C). The slice data60B and the MB data62B are shown hatched to indicate decoding thereof.

FIG. 33illustrates a block diagram of a hierarchical arrangement of the multi-layer power management modes. The TL PM processing1200begins the first tier of power management at the transport layer52. As the result of the TL PM processing1200, a plurality of low power modes are established based on the projected MIPS. In one configuration, modes1,1A and2are proposed. The power management operations continue to a second tier at VS/PL PM processing1202. The second tier of power management is conducted at the sequence and picture layer70. The VS/PL PM processing1202produces a plurality of low power modes as a function of projected MIPS and visual quality and/or granularity. In one configurations, modes3,4A-4D are generated. Mode5is a power mode but may not be a low power mode if all frames are decoded. Nonetheless, the power management operations continue to a third tier at S/MBL PM processing1204. The third tier of power management is conducted at the slice and macroblock layer72. The S/MBL PM processing1204produces a plurality of low power modes as a function of projected MIPS and visual quality and/or granularity. In one configurations, modes6A-6C and7A-7C are generated. Mode8allows all the frames to be decoded if power permits. Furthermore, mode8may be used after a part of the bitstream has been decoded and the re-projection of the MIPS indicates that all remaining frames may be decoded.

The previous description of the disclosed configurations is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to these configurations will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other configurations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the configurations shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.