Method and apparatus for dynamic allocation of scalable selective enhanced fine granular encoded images

A method and system for dynamically selectively enhancing desired area or areas of a video image which are FGS encoded. The method comprising the steps of determining at least one of the FGS encoded macroblocks in each of the FGS encoded bit-planes associated with the desired area or portion of the video image, determining an order of transmission of each of the determined FGS encoded macroblocks within the transmission sequence and advancing each of the determined FGS encoded macroblocks in the transmission sequence order corresponding to a known level of enhancement, wherein the advanced FGS encoded macroblocks are contained in a bit-plane having a higher transmission priority. In one aspect of the invention, the desired area may be selected by interactively by a user. In another aspect, the desired area or areas may be selected automatically.

FIELD OF THE INVENTION

This invention relates generally to video encoding and more specifically to dynamically allocating selective enhancement Fine Granular Scalable encoded video data.

BACKGROUND OF THE INVENTION

The MPEG-4 Fine-Granular Scalability (FGS) framework allows for different levels of compression for different parts of a video image by using an adaptive quantization technique, referred to as Selective Enhancement. Selective Enhancement of FGS encoded video images is more fully disclosed in U.S. Patent Application Ser. No. 60/217,827, entitled, “System And Method For Fine Granular Scalable Video With Selective Quality Enhancement,” filed on Jul. 12, 2000. An improvement to the transmission efficiency of selectively enhanced FGS video signals is more fully disclosed in U.S. patent application, Ser. No. 09/887,747, entitled “Method And Apparatus For Improved Efficiency In Transmission Of Fine Granular Scalable Selective Enhanced Images, filed on Jun. 22, 2001.

Utilizing selective enhancement designated areas of an FGS encoded video image may be transmitted to achieve a higher quality level than non-designated areas of the image. As disclosed in the referred to patents applications, a higher quality level or higher resolution of a transmitted image may be achieved by “shifting” specific or designated areas or regions of FGS encoded image elements to a higher transmission priority level. The selectively enhanced images are thus transmitted out of their normal sequence. The specific or designated areas or regions may be associated with a specific pixel, pixel arrays or sets or pixel arrays, referred to herein as macroblocks. An indication of enhancement factor or shift factor is also associated with each shifted FGS encoded image elements, i.e., macroblocks.

One disadvantage of the current selective enhancement method is that the shifting factors and the area or areas selected for selected enhancement are essentially pre-determined and allocated during the FGS encoding processing. Such selection of enhanced areas may be preformed by an automatic system based on some predetermined rules. For example, data blocks associated with the slowest movement within the image may be selected for enhancement. Alternatively, faces within the image may be more enhanced than the background. However, these rule-based automatic systems often fail to enhance precisely what the user or viewer is interested in. Hence, that is the area or region of interest selected by the automatic system may not coincide with the area or region that is of interest to the user or viewer.

Hence, there is a need to provide a system that allows the user or viewer to select or allocate an area or areas or interest for transmission as enhanced FGS encoded data.

SUMMARY OF THE INVENTION

A method and system to dynamically select or allocate an area or areas within FGS encoded video images is presented. The method comprising the steps of determining at least one of the FGS encoded macroblocks in each of the FGS encoded bit-planes associated with the desired area or portion of the video image, determining an order of transmission of each of the determined FGS encoded macroblocks within the transmission sequence and advancing each of the determined FGS encoded macroblocks in a transmission sequence order corresponding to a known level of enhancement, wherein the advanced FGS encoded macroblocks are contained in a bit-plane having a higher transmission priority. In one aspect of the invention, the desired area may be selected interactively by a user. In another aspect, the desired area or areas may be selected automatically.

It is to be understood that these drawings are solely for purposes of illustrating the concepts of the invention and are not intended as a level of the limits of the invention. It will be appreciated that the same reference numerals, possibly supplemented with reference characters where appropriate, have been used throughout to identify corresponding parts.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1aillustrates a conventional Fine Granular Scalability (FGS) video encoding system100employing pre-determined selective enhancement technology. As illustrated, this system is composed of a base layer encoder102and an enhancement layer encoder104. Original video signal106is digitally encoded and quantized by base layer encoder102which produces base layer (BL)110. Base layer110contains sufficient information that is representative of a minimally acceptable video signal. Base layer signal110may also include motion compensation information. Motion compensation is well known in the art and need not be discussed herein. Base layer signal110is then provided to buffer/rate transmitter170for subsequent transmission over network, e.g., Internet,180.

Original video signal106is also provided to enhancement layer encoder104along with a digitized and quantized version of base layer110. Enhancement layer encoder104determines a residual error between the original video signal and the quantized base layer110. Enhancement layer encoder104creates quality improvement (SNR) enhancement layer150containing information items, which when applied to transmitted base layer110removes the errors of quantization and improves the transmitted image quality. SNR Enhancement layer signal150is then provided to buffer170for subsequent transmission through rate controller175over network, e.g., Internet,180.

Also illustrated is temporal FGS encoded enhancement layer (FGST)155, which includes motion compensation information regarding the base layer110and enhancement layer150. Temporal enhancement layer signal155is also provided to buffer170for subsequent transmission through rate controller175over network180.

As is understood in transmission of FGS encoded video images, the quantity of information items within enhancement layers150,155that are transmitted depends upon the bandwidth available. Hence, each information item within enhancement layer150,155may not be transmitted during a frame. Consequently, those areas of a transmitted image that are transmitted first, or at a higher priority, tend to have higher quality or greater resolution than those transmitted latter.

Selective Enhancement device108within FGS encoder104provides for designated areas within an image to be transmitted at a higher transmission priority than other, non-designated, areas or regions of the image. Selective enhancement is more fully disclosed in the referred to patent applications, which are incorporated by reference herein.

FIG. 1billustrates a conventional FGS encoded image200with predetermined selective enhancement generated in accordance with the encoder illustrated inFIG. 1a.As shown, image200is composed of a base layer110, an enhancement layer150and a shift factor layer160. As discussed more fully in the referred to patent applications, video images FGS encoded are formulated into blocks, which are conventionally representative of image information contained within six 8×8 data blocks, conventionally referred to as macroblocks. The encoded image is stored in levels of information, referred to as bit planes. Each bit-plane contains progressively more detailed information (e.g., non-zero DCT residual coefficients) regarding the video information of each macroblock. For example, a portion of a video image may be stored in base layer block112and progressively finer image resolution data is stored in enhancement layer block132,142, etc. Hence, in accordance with FGS encoding principles, a minimum resolution video image is stored in base layer110, composed of information blocks, macroblocks or data blocks,112,114,116,118, etc., while, progressively finer resolution information regarding corresponding portions of the video image are stored in enhancement layer150composed of bit-planes120,130and140, etc. Thus, in this illustrative case, bit-plane130includes information regarding the most significant bit or byte of corresponding macroblocks, while bit-plane140, includes information regarding the least significant bit or byte of corresponding macroblocks.

Bit-plane120contains information regarding selectively enhanced macroblocks scaled to achieve a higher transmission priority. In this exemplary illustration, the information regarding the image is scaled such that the most significant bit or byte of corresponding macroblock114, represented as E′e, is transmitted prior to the most significant bit or byte corresponding to first macroblock110, represented as E′o. Similarly, information regarding the next most significant bit or byte corresponding to macroblock114, represented as E″eis “shifted” or a scaled such that it is transmitted prior to the next most significant bit or byte of first macroblock110, represented as E″O. Information regarding the shifting or scaling of corresponding macroblocks is located in shift factor layer160, which is composed, in this illustrated example, of blocks162,164,166,168, that provide information for each corresponding macroblock. As is known, shifting factors in160are not transmitted all at the same time, but rather before enhancement layer150. They are transmitted for each macroblock (MB) independently, as soon as that MB becomes significant (i.e., non-zero DCT residual coefficients). Hence, either a MB will be transmitted when it is significant or alternatively, a non-zero-macroblock symbol (NULL) will be transmitted to indicate that that MB is still insignificant. The first time a MB becomes significant, the corresponding Selective Enhancement (SE) shifting factor160will also be transmitted.

Each FGS encoded macroblock information item stored in bit-planes corresponding to associated frames of video source is then stored in device170. Device170may be a permanent or semi-permanent medium, such as a Read-only Memory (ROM), Random Access Memory (RAM), magnetic or optical disk, etc., for transmission over a communication network, such as the Internet, at a latter time. Although an exemplary selective enhanced composition image200is shown, it would be further understood that multiple versions of video source106may be concurrently stored on at least one permanent or semi-permanent medium for transmission in accordance with different bandwidth or bit rate conditions. Although not shown, it would be understood that FGST enhancement layer155would similarly be composed of macroblocks and shifted accordingly.

FIG. 2illustrates an exemplary encoding system in accordance with the principles of the present invention. In this exemplary system, base layer110, FGS enhancement layer150and FGST enhancement layer155are stored in buffer170in a conventional, non-selectively enhanced configuration. Macroblock data is then provided to dynamic selective enhancement module210to dynamically determine the transmission sequence when selective enhancement is requested. Whether macroblock data or information is or is not selectively enhanced, macroblock data or information is provided to rate controller175for subsequent transmission over network180.

FIG. 3aillustrates the bit-plane structure of a conventional FGS encoded video image, which is simply represented as being composed of four macroblocks. Further, the video images are FGS encoded over six bit-planes, wherein bit plane310, composed of macroblocks311,312,313and314, is representative of base layer110and the subsequent bit planes320–360, respectively, are representative of enhancement layer150. Thus, macroblocks,321,331,341, etc., contain information regarding progressively finer resolution to the video information contained in macroblock311.

In this example, the video image is transmitted using a conventional row raster-type scan370, which, in this illustrative example, transmits macroblock data in odd rows from left to right and in even row from right to left. Hence, data in macroblock311is first transmitted, then the data in macroblock312followed by the data in macroblocks314and313, respectively. Then, if sufficient bandwidth is available, data in macroblocks321,322,324and323, which are representative of the first level of enhancement layer data, is next transmitted. Data in bit-planes330–360may also be transmitted if sufficient bandwidth is available. Although a row raster-type scan is illustrated, it would be understood, that horizontal and/or vertical progressive/interlaced scans are also contemplated to be within the scope of the invention.

FIG. 3billustrates an exemplary transmission sequence order380corresponding to the FGS encoded bit-plane structure depicted inFIG. 3a.In the exemplary transmission sequence order shown, pointers to, or vectors associated with, each macroblock within each bit-plane is stored in the order of transmission. Thus, for the exemplary horizontal raster scan shown inFIG. 3a,a pointer to, or vector associated with, macroblock311is stored as a first entry in transmission sequence order380and corresponding pointers to, or vectors associated with, macroblocks312,314, and313are then sequentially stored in sequence control table380. Similarly, corresponding pointers to, or vectors associated with, each macroblock in each bit-plane are stored in the order of their transmission. As would be understood, rate controller175, illustrated inFIG. 1aor2, includes a process, as will be further described, that sequentially proceeds through each element of transmission sequence order380to determine the first/next macroblock to be transmitted. As would be understood, an alternate order of macroblock transmission would cause an alternative order of storage of associated macroblock pointers or vector. Hence, in accordance with the principles of the invention, the exemplary transmission sequence of macroblock rather than being fixed, or “hard-wired,” is determined by the order selected in transmission sequence order380.

FIG. 4aillustrates an adaptation of transmission sequence order380in response to a request to selectively enhance a desired portion of an image corresponding to that shown in macroblock313. In this case, corresponding pointers to, or vectors associated with, macroblock313and macroblocks in subsequent bit-planes, i.e., macroblocks323,333,343,353,363, are shifted or re-positioned in transmission table order380to accommodate the requested enhancement. This shifting or re-positioning of the order of transmission provides a higher transmission priority for the requested image portion. Thus, information regarding macroblock313represented as pointer or vector394is positioned prior to information regarding macroblock311represented as pointer or vector395. Similarly, information regarding macroblock363represented as pointer or vector397is positioned prior to information regarding macroblock361represented as pointer or vector398.

It would be understood that to provide compatibility with many existing receiving systems and limit the need for special receiving or transmission equipment, a conventional raster-type transmission sequence is maintained. In this case, pointers or vectors391,392,393, are introduced into transmission sequence order380to compensate for selective enhancement or higher transmission priority of the requested macroblock311. In this illustrative example, the pointers or vectors are set to a “not significant” macroblock or block. As would be appreciated, the pointer or vector may similarly be set to an unused value, such as “0”, which would be indicative of a “not significant” character or block. “Not significant” macroblocks are transmitted to indicate to a corresponding receiving system to zero fill the received macroblock. Similarly, to maintain the transmission order after all selective enhanced macroblocks are transmitted, “0” pointers are introduced in the bit-planes of the selectively enhanced macroblocks. Hence, “0” pointers are placed in transmission control table380at locations corresponding to the shifted macroblock information. In this case, “0” pointer are placed at location399to indicate a set of zero blocks will be transmitted.

FIGS. 4band4ccollectively illustrate an alternate embodiment of a transmission sequence order and an exemplary alteration using a link-list structure.FIG. 4billustrates a transmission sequence order similar to that shown inFIG. 3b.However, in this case, a first pointer or vector is employed to determine a macroblock for transmission, similar that shown inFIG. 3b.A second pointer or vector,415,420, . . .440, etc., is employed to determine a subsequent macroblock for transmission. Thus, rather than sequentially accessing and determining the next macroblock as is done inFIG. 3b,the second pointer or vector is used to determine the next macroblock for transmission. To determine a first entry for transmission a separate value410containing a pointer or vector is maintained.

FIG. 4cillustrates an adaptation of the transmission sequence order for selective enhancement or advancement in transmission priority of macroblock313ofFIG. 3a.In this case, initial value pointer410is altered to access a data block455of “not significant” values. Data block455is employed, in this case, to maintain a known transmission order. The last entry of data block455includes a pointer or vector460that selects that entry in the transmission sequence that corresponds to macroblock313. The next macroblock pointer465associated with macroblock313is then altered to select macroblock311for next transmission. Macroblock312is then selected as the next macroblock for transmission as the next macroblock pointer associated with macroblock311is unaltered. Similarly, macroblock314is then selected as the next macroblock for transmission as the next macroblock pointer470associated with macroblock312is unaltered. However, the next macroblock pointer475associated with macroblock312is altered to select macroblock324for transmission. To complete and maintain the transmission sequence order, a last or final block480, which includes a “not significant” block may be accessed by a pointer485associated with the last transmitted macroblock364.

As would be appreciated, data block455and480are employed to maintain a uniform number of macroblocks for each bit-plane. However, it would be understood that the use of either data block455or480is optional in that non-uniform number of macroblocks per bit-plane may also be maintained and transmitted.

FIGS. 3b,4a,4b, and4cillustrate a linear or sequential order of transmission. It would be known of those skilled in the art to use initial pointer and linked-list and/or double-list to maintain and dynamically alter a transmission sequence. Further, alternate non-sequential transmission sequences may be maintained using linked-list or double-linked lists.

Although only one macroblock is selectively enhanced to one higher level of transmission priority, it would be understood that any number of macroblocks may be shifted to any higher level of transmission priority. For example, a plurality of macroblocks may be shifted or re-positioned such that the shifted macroblocks may be completely transmitted prior to the transmission of the most significant bit or byte of the unenhanced image.

The dynamic selective enhancement process is further illustrated by the following example. Assuming that an original FGS residual image is encoded over six (6) bit-planes and each bit-plane includes four (4) macroblocks. Further, the most significant byte of the first macroblock is six, the most significant byte of the second macroblock is3, the most significant byte of the third macroblock is4and the most significant byte of the fourth macroblock is 5. The original FGS encoded bit-planes may be represented as:

Where “Coded-bpx” refers to the coded bit-plane corresponding to the macroblock; and

“0”indicates a not significant value.

Assuming that macroblock3is selectively enhanced by, for example, a factor of three (3), then the number of bit-planes is increased to seven (7) and the selectively FGS encoded macroblocks are represented as:

FIG. 5illustrates an exemplary transmission system500utilizing the principles of the present invention. In this exemplary system transmission unit510is composed of video frame source106, which receives video frame image information, video encoding unit100, similar to that illustrated inFIG. 1awithout selective enhancement units108and in encoder buffer170, which stores the FGS encoded video image. Transmission rate controller175accesses the stored FGS encoded video images contained within encoder buffer170and transmits the stored image data over data network180in the order stored in an associated transmission sequence table. At receiving system517, the received data frames are stored in decoder buffer518and provided to video decoder520. The decoded information is then presented on video display522.

Further illustrated is processor577, which is comparable to dynamic selective enhancement device210illustrated inFIG. 2. In this exemplary embodiment, dynamic selective enhancement is integrated within transmission rate controller175. In this illustrative case, controller175dynamically determines those macroblocks that encompasses a desired portion of an image, which are to be selectively enhanced, adapts transmission sequence order for next/subsequent frame transmissions and transmits the image data over network180at a rate appropriate for the available bandwidth. Process577may further receive information over network180regarding user directed or requested selective enhancement. In this case, processor577responsive to the user requests for selective enhancement, which may be provided by I/O processor525, may dynamically determine those macroblocks that encompass that requested image portion and adapt a transmission sequence order for next/subsequent transmission.

As would be appreciated, processor577or I/O processor525may be any means, such as general purpose or special purpose computing system, or may be a hardware configuration, such as a dedicated logic circuit, integrated circuit, Programmable Array Logic (PAL), Application Specific Integrated Circuit (ASIC), that provides known outputs, i.e., macroblock data, in response to known inputs, i.e., transmission sequence order.

In a alternative aspect of the invention, transmission controller175, may automatically (absent user requests) selectively enhance designated areas of the transmitted image in order to provide better quality to receiving systems based on the available bandwidth. For example, a face-based enhancement rule, a motion-based segmentation rule, an importance mask-based enhancement rule, a center-based enhancement rule, etc., may automatically determine an area or areas of enhancement.

FIG. 6illustrates a flow chart600of an exemplary process for transmission controller175. In this exemplary process, a first pointer is selected at block610. A determination is made at block615whether the selected pointer corresponds to a macroblock that is not significant. If the answer is in the affirmative, then a data block representative of a “0” pointer is selected for transmission, at block620.

If, however, the answer is in the negative, then information items within the selected data block are selected for transmission, at block630.

At block640a next pointer is selected. At block650a determination is made whether the selected next pointer is an end-of-frame marker. If the answer is the negative, then processing returns to the determination at block615to determine whether the selected next pointer is associated with a “null” block.

If, however, the answer is in the affirmative, then the processing for the selected frame transmission sequence table is completed.

FIG. 7illustrates a flow chart700of an exemplary process for dynamically adapting a frame transmission sequence table in accordance with the principles of the present invention. In this case, processor577determines that a designated area of an image is capable of being selectively enhanced. For example, processor577may receive a user request for selective enhancement of a portion of an image, at block710, as is illustrates. In an alternative embodiment, processor577may determine, based on available bandwidth, that selective enhancement is appropriate.

At block720macroblocks associated with or encompass the selected image portion requested to be selectively enhanced are determined. Similarly, the level of enhancement is determined at block730. At block740each next frame transmission sequence table is selected. Within each selected next frame transmission sequence table, the determined macroblocks associated with the requested selected enhancement area are adjusted, i.e., shifted or re-positioned, to enable the determined macroblocks to be assigned a higher transmission priority at block750. At block760, locations within each of the selected sequence tables not containing pointer data is filled to allow for a data block transmission that is representative of “0” data.

FIG. 8illustrates an exemplary embodiment of a processor577which may be used for implementing the principles of the present invention. Processor577may represent a desktop, laptop or palmtop computer, a personal digital assistant (PDA), a video/image storage device such as a video cassette recorder (VCR), a digital video recorder (DVR), a TiVO device, etc., as well as portions or combinations of these and other devices. Processor577includes one or more input/output devices802, a processor803and a memory804. Memory804may be a semiconductor medium, such as RAM, ROM, Flash, Cache, a magnetic media, such as magnetic disk, or optical media, such a CD-ROM.

Processor577may access, through I/O device802one or more data sources801, e.g. buffer170, which may be received over a network (not shown). The source(s)801may alternatively represent one or more network connections for receiving information from a server or servers over, e.g., a global computer communications network such as the Internet, a wide area network, a metropolitan area network, a local area network, a terrestrial broadcast system, a cable network, a satellite network, a wireless network, or a telephone network, as well as portions or combinations of these and other types of networks.

The input/output devices802, processor803and memory804may communicate over a communication medium805. The communication medium805may represent, e.g., a bus, a communication network, one or more internal connections of a circuit, circuit card or other device, as well as portions and combinations of these and other communication media. Data from source(s)801is processed in accordance with one or more software programs stored in memory804and executed by processor803to alter transmission sequence order and to generate an output that may be supplied over a network (not shown) or to a display device806.

In a preferred embodiment, the coding and decoding employing the principles of the present invention may be implemented by computer readable code executed by the system. The code may be stored in the memory804or read/downloaded from a memory medium such as a CD-ROM or floppy disk. In other embodiments, hardware circuitry may be used in place of, or in combination with, software instructions to implement the invention. For example, the elements illustrated herein may also be implemented as discrete hardware elements.

Although the invention has been described and pictured in a preferred form, it is, however, understood that the present disclosure has been made only by way of example, and that numerous changes in the details may be made without departing from the spirit and scope of the invention as hereinafter claimed. For example, the indicators described may be designated by individual settings that describe a specific method employed in a transmission frame. Or the indicators may be coded values with a fixed number of transmission bits within a transmission frame. Or the indicators may be a single setting that specifies the presence of a specific method employed in a transmission frame. It is intended that the patent shall cover by suitable expression in the appended claims, those features of patentable novelty that exists in the invention disclosed.