Source: http://www.google.com/patents/US7961784?dq=7,177,838
Timestamp: 2015-05-07 09:48:00
Document Index: 141007226

Matched Legal Cases: ['Application No, 2', 'art 2', 'art 2', 'art 2', 'art 2', 'Application No. 2003', 'Application No. 2', 'Application No. 10', 'Application No. 200805819']

Patent US7961784 - Method and system for improving compressed image chroma information - Google PatentsSearch Images Maps Play YouTube News Gmail Drive More »Sign inAdvanced Patent SearchPatentsMethods, systems, and computer programs for improving compressed image chroma information. In one aspect of the invention, a resolution for a red color component of a color video image is used that is higher than the resolution for a blue color component of the color video image. Another aspect includes...http://www.google.com/patents/US7961784?utm_source=gb-gplus-sharePatent US7961784 - Method and system for improving compressed image chroma informationAdvanced Patent SearchPublication numberUS7961784 B2Publication typeGrantApplication numberUS 12/176,267Publication dateJun 14, 2011Filing dateJul 18, 2008Priority dateJul 12, 2001Fee statusPaidAlso published asCA2452506A1, CA2452506C, CA2621107A1, CN1250003C, CN1526236A, EP1405515A2, EP1405515A4, EP1988712A2, EP1988712A3, EP2239943A1, EP2239943B1, EP2493178A1, EP2493179A1, EP2493180A1, EP2493181A1, EP2493182A1, EP2493183A1, EP2493184A1, EP2493185A1, EP2493186A1, EP2698991A2, EP2698991A3, US8737466, US20030112863, US20080273809, US20110038411, US20110286517, US20110286518, US20110286519, US20120033729, US20120087408, US20120087409, WO2003007126A2, WO2003007126A3Publication number12176267, 176267, US 7961784 B2, US 7961784B2, US-B2-7961784, US7961784 B2, US7961784B2InventorsGary A. DemosOriginal AssigneeDolby Laboratories Licensing CorporationExport CitationBiBTeX, EndNote, RefManPatent Citations (89), Non-Patent Citations (58), Referenced by (8), Classifications (20), Legal Events (3) External Links: USPTO, USPTO Assignment, EspacenetMethod and system for improving compressed image chroma information
US 7961784 B2Abstract
Methods, systems, and computer programs for improving compressed image chroma information. In one aspect of the invention, a resolution for a red color component of a color video image is used that is higher than the resolution for a blue color component of the color video image. Another aspect includes utilizing a lower or higher value of a quantization parameter (QP) for one or more chroma channels as compared to the luminance channel. Another aspect is use of a logarithmic representation of a video image to benefit image coding. Another aspect uses more than two chroma channels to represent a video image.
1. A method for coding a color space representation of video macroblocks in a computerized video system having at least an encoder and a decoder, the method comprising:
at the encoder,
utilizing, for a macroblock, a luminance quantization parameter (QPy) and a first chroma bias value (QPcb1) and a second chroma bias value (QPcb2);
deriving a first chrominance quantization parameter (QPc1) by subtracting the first chroma bias value from the luminance quantization parameter, wherein the first chrominance quantization parameter differs from the luminance quantization parameter;
deriving a second chrominance quantization parameter (QPc2) by subtracting the second chroma bias value from the luminance quantization parameter, wherein the second chrominance quantization parameter differs from the luminance quantization parameter;
encoding, the macroblock with differential coding using the luminance quantization parameter, the first chrominance quantization parameter, and the second chrominance quantization parameter; and
transmitting the first and second chroma bias values from the encoder to the decoder for the macroblock, wherein the first chroma bias value and the second chroma bias value differ from each other for the macroblock, and
at the decoder,
receiving the first and second chroma bias values;
reconstructing, with the decoder, the first and second chrominance quantization parameters for the macroblock, wherein the reconstructing comprises:
determining the first chrominance quantization parameter using the received first chroma bias value; and
determining the second chrominance quantization parameter using the received second chroma bias value; and
decompressing the macroblock using the luminance quantization parameter and the reconstructed first and second chrominance quantization parameters.
2. The method of claim 1, further comprising configuring the first chrominance quantization parameter to differ from the second chrominance quantization parameter for the macroblock. Description
This application is a divisional of and claims priority to U.S. application Ser. No. 09/905,039, filed on Jul. 12, 2001, the disclosure of which is incorporated by reference.
This invention relates to video compression, and more particularly to methods, systems, and computer programs for improving compressed image chroma information in MPEG-like video compression systems.
MPEG-2 and MPEG-4 are international video compression standards defining a video syntax that provides an efficient way to represent image sequences in the form of more compact coded data. The language of the coded bits is the �syntax.� For example, a few tokens can represent an entire block of samples (e.g., 64 samples for MPEG-2). Both MPEG standards also describe a decoding (reconstruction) process where the coded bits are mapped from the compact representation into an approximation of the original format of the image sequence. For example, a flag in the coded bitstream signals whether the following bits are to be preceded with a prediction algorithm prior to being decoded with a discrete cosine transform (DCT) algorithm. The algorithms comprising the decoding process are regulated by the semantics defined by these MPEG standards. This syntax can be applied to exploit common video characteristics such as spatial redundancy, temporal redundancy, uniform motion, spatial masking, etc. In effect, these MPEG standards define a programming language as well as a data format. An MPEG decoder must be able to parse and decode an incoming data stream, but so long as the data stream complies with the corresponding MPEG syntax, a wide variety of possible data structures and compression techniques can be used (although technically this deviates from the standard since the semantics are not conformant). It is also possible to carry the needed semantics within an alternative syntax.
These MPEG standards use a variety of compression methods, including intraframe and interframe methods. In most video scenes, the background remains relatively stable while action takes place in the foreground. The background may move, but a great deal of the scene is redundant. These MPEG standards start compression by creating a reference frame called an �intra� frame or �I frame�. I frames are compressed without reference to other frames and thus contain an entire frame of video information. I frames provide entry points into a data bitstream for random access, but can only be moderately compressed. Typically, the data representing I frames is placed in the bitstream every 12 to 15 frames (although it is also useful in some circumstances to use much wider spacing between I frames). Thereafter, since only a small portion of the frames that fall between the reference I frames are different from the bracketing frames, only the image differences are captured, compressed, and stored. Two types of frames are used for such differences�predicted or P frames, and bi-directional interpolated or B frames.
P frames generally are encoded with reference to a past frame (either an frame or a previous P frame), and, in general, are used as a reference for subsequent P frames. P frames receive a fairly high amount of compression. B frames provide the highest amount of compression but require both a past and a future reference frame in order to be encoded. Bi-directional frames are never used for reference frames in standard compression technologies.
MPEG Color Space Representation
MPEG-1, MPEG-2, and MPEG-4 all utilize a Y, U, V color space for compression. There is a choice of luminance equation, hut a typical conversion transformation between RGB (red-green-blue) to a YUV representation is expressed as:
Y=0.59G+0.29R+0.12B U=R−Y V=B−Y The Y luminance factors for green range from 0.55 up to 0.75, depending upon the color system. The factors for red range from 0.2 to 0.3, and the factors for blue range from 0.05 to 0.15.
This transformation can be cast as a matrix transformation, which is a linear operator intended for use on linear signals. However, this simple transformation is performed in MPEG 1, 2, and 4 in the non-linear video space, yielding various artifacts and problems.
It is typical in MPEG to reduce the resolution of the U and V chroma channels to achieve higher compression. The most commonly used reduction of resolution is to use half resolution both vertically and horizontally. MPEG-2 supports full resolution chroma, as well as half resolution horizontally. However, the most commonly used MPEG-2 profiles, Main Profile at Main Level (MP@ ML) and Main Profile at High Level (MP@ HL), use half resolution horizontally and vertically. MPEG-4 versions 1 and 2 use half resolution vertically and horizontally. Note that full chroma resolution is often called 4:4:4, half chroma horizontal resolution is often called 4:2:2, and half vertical and horizontal resolution is often called 4:2:0. (It should be noted that the 4:x:x nomenclature is flawed in its meaning and derivation, but it is common practice to use it to describe the chroma resolution relationship to luminance.)
The filter which reduces the horizontal and vertical chroma resolution under the various MPEG standards is applied to non-linear video signals as transformed into the U and V color representation. When the inverse transformation is applied to recover RGB, the non-linear signals and the filters interact in such a way as to produce artifacts and problems. These problems can be generalized as �crosstalk� between the Y luminance and the U and V chroma channels, along with spatial aliasing.
Further information on linear versus non-linear representations and transformations may be found in �The Use of Logarithmic and Density Units for Pixels� by Gary Demos, presented at the October 1990 SMPTE conference, and published in the SMPTE Journal (October 1991, vol. 100, no. 10). See also �An Example Representation for Image Color and Dynamic Range which is Scalable, Interoperable, and Extensible� by Gary Demos, presented at the October 1993 SMPTE conference and published in the proceedings and preprints. These papers describe the benefits of logarithmic and linear spaces at various stages of the image compression processing pipeline, and are hereby incorporated by reference.
The reason for reducing chroma resolution for U and V is that the human visual system is less sensitive to changes in U and V than it is to changes in luminance, Y. Since Y is mostly green, and U and V are mostly red, and blue respectively, this can also be described as a human visual sensitivity being higher for green than for red and blue. However, although U and V are treated the same in MPEG-1, MPEG-2, and MPEG-4, the human visual system is more sensitive to U (with its red component) than to V (with its blue component).
This difference in chroma sensitivity is embodied in the 1951 NTSC-2 color standard that is used for television. NTSC-2 uses a YIQ color space, where I and Q are similar to U and V (with slightly different weightings). That is, the I channel primarily represents red minus luminance and the Q channel primarily represents blue minus luminance. In NTSC-2, the luminance is given 4.5 MHz of analog bandwidth, and the I chroma channel is given 1.5 MHz of analog bandwidth. The Q channel, representing the blue-yellow axis, is given only 0.5 MHz of analog bandwidth.
Thus, the NTSC-2 television system allocates three times as much information to the I channel than it does to the Q channel, and three times as much information to the Y luminance channel than to the I channel. Therefore, the bandwidth ratio between the Y luminance channel and the Q (blue minus luminance) channel is nine. These MPEG YUV and NTSC-2 relationships are summarized in the following table:
NTSC-2
Red, U, and I pixels to Y
Blue, V, and Q pixels to Y
Ratio of Chroma Resolution to Luminance
Clearly there is a greater difference in treatment between the luminance channel and the U and V channels under the MPEG standards than the luminance and I and Q channels in the NTSC-2 standard.
The invention is directed to methods, systems, and computer programs for improving compressed image chroma information.
More particularly, in one aspect of the invention, a color video image may be improved by increasing the red resolution for an RGB representation (or the U resolution for a YUV representation) above the resolution used for blue (or V). Using lower resolution for the blue color component means less information needs to be compressed, such as in a motion compensated color video image compression system. This aspect of the invention includes a method, system, and computer program for compressing image chroma information of a color video image in a video image compression system by selecting a resolution for a red color component of the color video image that is higher than the resolution for a blue color component of the color video image.
Another aspect of the invention is a technique for reducing the level of chroma noise that results from any given value of the quantization parameter (QP) used during compression, thereby improving image quality. This is accomplished by utilizing a lower value of QP for the U (=R−Y) channel than for the Y channel. Similarly, the quality of the V (=B−Y) channel may also be improved by utilizing a lower QP value for the V channel than for the Y channel.
Another aspect of the invention is a technique useful when higher compression is required. In this aspect, a positive QP bias is applied to the QP value for the Y channel for use with either or both of the U and V chroma channels.
Another aspect of the invention is use of a logarithmic representation to benefit image coding. Logarithmic coding, when feasible, can improve coding efficiency for YUV color space representations of images originally represented as linear RGB pixel values. At other processing steps, a conversion to and from linear representations can be beneficial.
Another aspect of the invention is a method for improving the video characteristics of a color video image in a video compression system, including: selecting a set of image channels to represent the color video image, including a luminance channel and n chroma channels, where n is at least three; and compressing the luminance channel and the n additional chroma channels to a compressed video image.
FIG. 1 is a flowchart showing an illustrative method (which may be computer implemented) for increasing the resolution for U above the resolution used for V in a YUV color space representation.
FIG. 2 is a flowchart showing an illustrative method (which may be computer implemented) for applying a QP bias for chroma channels.
FIG. 3 is a flowchart showing an illustrative method (which may be computer implemented) for logarithmic coding of luminance and chroma information.
FIG. 4 is a flowchart showing an illustrative method (which may be computer implemented) for coding additional chroma channels in an image compression system.
Improved Color Coding Precision
As the quality of images improves with respect to the attributes of reduced noise, extended dynamic range, and extended color range, human sensitivity to color also increases. In particular, it has been observed that red in an RGB representation (or U in a YUV representation) often requires higher precision and clarity than is commonly used in video compression.
Unless blue is being used for processing (such as blue-screen special effects compositing or image analysis), human sensitivity to the blue-yellow chroma axis, as embodied by either blue or V, is adequately addressed by half resolution sampling horizontally and vertically. Thus, one quarter of the total number of pixels of an image provides sufficient quality for representing the blue or V chroma axis. However, unlike blue and V, one-half resolution coding of red and/or U is sometimes insufficient in quality with respect to large wide-dynamic range displays and projectors.
Thus, an image may be improved by increasing the red resolution for an RGB representation (or the U resolution for a YUV representation) above the resolution used for blue (or V). Using lower resolution for the blue color component means less information needs to be compressed, such as in a motion compensated color video image compression system.
In accordance with the invention, there are three preferred methods of maintaining increased red (or U) resolution with respect to a downfiltered blue (or V) resolution:
1) Use full resolution for red and/or U;
2) Use one-half resolution on only one chroma axis, either vertically or horizontally, for red and/or U; or
3) Use a filtered resolution between full size and one-half, such as ⅔ or �, on one or both chroma axes for red and/or U.
FIG. 1 is a flowchart showing an illustrative method (which may be computer implemented) utilizing higher resolution for U than the resolution used for V in a YUV color space representation (a similar method may be applied to an RGB color space representation):
Step 101: In an image compression system utilizing a YUV color space representation, downsize filter the V (=B−Y) channel of an input image to one-half resolution horizontally, and optionally to one-half resolution vertically.
Step 102: Downsize filter the U (=R−Y) channel of the image to a resolution higher than the V (=B−Y) channel, preferably being one of:
a) full resolution;
b) between one-half and full resolution horizontally, but full resolution vertically;
c) between one-half and full resolution horizontally and vertically;
d) between one-half and full resolution vertically, but full resolution horizontally.
Step 103: Compress the YUV image (having luminance Y and the downsize filtered U and V chroma information) using an MPEG-like compression system.
Step 104: Decompress the images into Y, U, and V channels (usually in a different computer).
Step 105: Convert the U and V channels to full resolution, using the appropriate resolution increase (i.e., the reciprocal of the downsize filter factor used in Step 101 above for V and Step 102 above for U).
Step 106: Optionally, convert the YUV picture to an RGB image for viewing, analysis, or further processing.
Differential QP Bias for Chroma
Co-pending U.S. patent application Ser. No. 09/798,346, entitled �High Precision Encoding and Decoding of Video Images� and assigned to the assignee of the present invention (which is hereby incorporated by reference), teaches various aspects of the use of the quantization parameter (QP) during compression. Another aspect of the present invention is a technique for reducing the level of chroma noise that results from any given value of the quantization parameter (QP) used during compression, thereby improving image quality. This is accomplished by utilizing a lower value of QP for the U (=R−Y) channel than for the Y channel. Similarly, the quality of V (=B−Y) may also be improved by utilizing a lower QP value for the V channel than for the Y channel.
A simple method of implementing a reduced chroma QP value is to subtract a constant value from the QP value used for the Y (luminance) channel. It is thus useful to expand the range of QP in the general case. However, note that use of high precision intermediate encoding and/or decoding in conjunction with a direct correspondence between the range of QP and QP values (i.e., a value x is the same as the representation number x; thus, value 14 equals representation number 14) can result in high quality compression and decompression.
It is desirable to maintain the small number of steps in QP (such as 32 or 64 values, or some similar small number) if variation of QP within a frame is desired, since the bits required to code QP variations per macroblock are limited to 2 units in MPEG-4. If QP is varied only once per frame, or once per slice or similar large structure, then the number of values for QP can be large.
If needed, the effective range of QP can be extended to a wide range of determined values (i.e., the values of QP actually applied during compression and decompression) within a small number of representational codes by taking advantage of the perceptually logarithmic nature of dynamic range. Accordingly, a non-linear lookup table can be used to map a small number of representational QP codes (such as the range of 1 to 32, or 1 to 128) into a much wider range of determined QP values (such as 1 to 128, 1 to 256, 1 to 1024, or 1 to 4096). In such a table, the low QP code entries would map nearly one-to-one to determined QP values. For example, QP codes 1 to 4 might map to determined QP output values of 1 to 4. However, the mapping will gradually become steeper, in a logarithmic model, such that QP code 8 might map to determined QP value 16, QP code 16 might map to determined QP value 64, and QP code 32 might map to determined QP value 256.
Alternatively, a separate constant value (lower than the QP value for Y) might be used for each of U and V. For example, �2� might be subtracted from the QP value for Y to yield the QP value for U, and �1� might be subtracted for the QP value for Y to yield the QP value for V. Any useful value of the amount to subtract can be used, limited only by a minimum value of �1� for the applied QP value.
This method works for constant QP values (variable bit rate). It also works as well for variable QP values (e.g., in both constant and variable bit rate motion compensated compression systems), since the instantaneous QP value can be biased by subtracting a specified difference value from the QP value for Y to yield a QP value for each of U and V.
Further, the range of these differential chroma-biased QP values can be extended using the extended QP range function or lookup, as described in the �High Precision Encoding and Decoding of Video Images� Patent Application referenced above.
It is necessary to signal the U and V bias values from the encoder to the decoder unless a pre-arranged value is used. These can be specified once, for example, for each session, group of pictures (GOP), frame, or image region.
FIG. 2 is a flowchart showing an illustrative method (which may be computer implemented) for applying a QP bias for chroma channels:
Step 201: In an image compression system, reduce the QP value for each of the U and V chroma channels by a selected value (which may be different for each channel).
Step 202: Utilize this reduced QP value for the U and V chroma channel compressions, respectively.
Step 203: Optionally, if variable QP values are used, ensure that the reduced U and V QP value is at least �1�.
Step 204: Unless a pre-set bias is to be used, signal or convey the QP value reduction amount to the decoder as often as it may change (once at a minimum).
Step 205: Decompress (usually in a different computer) the signal using the appropriate QP value for U and V (again ensuring that the reduced QP value is at least �I�).
Step 206: Optionally, view the decompressed images, or use the images for additional processing or analysis.
Another aspect of the invention is a technique useful when higher compression is required. In this aspect, a positive QP bias is applied to the QP value for the Y channel for use with either or both of the U and V chroma channels (preferably checking against a QP maximum value of a compression system, if any). Separate bias can be used for each of the U and V channels. Otherwise, the steps of such an embodiment would be similar to those shown in FIG. 2.
Logarithmic Coding of Luminance and Chroma
The paper entitled �The Use of Logarithmic and Density Units for Pixels,� referenced above, describes the benefits of a logarithmic representation for dynamic range. Log representations of a matching dynamic range are somewhat similar to commonly used video transfer functions. Even though similar, the logarithmic representation is more optimal in extensibility, calibration usage, and in orthogonality of color channels than are the various commonly used video representations.
Another aspect of the invention is use of a logarithmic representation to benefit image coding. It has been discovered that logarithmic coding, when feasible, can improve coding efficiency for YUV color space representations of images originally represented as linear RGB pixel values (such as at the sensor of a camera). At other processing steps, a conversion to and from linear representations can be beneficial.
As described in the �High Precision Encoding and Decoding of Video Images� Patent Application referenced above, chroma crosstalk with luminance is minimized when:
Y log=Log(Wr*R+Wg*G+Wb*B)
U=Log(R)−Y log
V=Log(B)−Y log
where Wr, Wg, and Wb are the linear weightings for the red, green, and blue components of luminance, and where R, G, and B represent a linear light space. These relationships are useful in applying this aspect of the invention.
FIG. 3 is a flowchart showing an illustrative method (which may be computer implemented) for logarithmic coding of luminance and chroma information:
Step 301: In an image compression system, perform the following transformation on input (e.g., directly from a video camera) linear R, G, and B pixel values:
where Wr, Wg, and Wb are the linear weightings for the red, green, and blue components of luminance.
Step 302: Optionally, reduce the resolution of the U and V chroma channels (as described above).
Step 303: Perform motion-compensated compression on this Y, U, and V representation of the moving image.
Step 304: Decompress the compressed images to restore Y, U, and V color components of the moving image (usually in a different computer).
Step 305: If optional Step 302 was applied, reverse the resolution reduction to restore full U and V resolution.
Step 306: Restore the linear R, G, and B pixel values using the following transformation:
R=anti-log(Y+U)
B=anti-log(Y+V)
G=(anti-log(Y)−Wr*R−Wb*B)/Wg Step 307: Optionally, convert to other video RGB representations (alternatively, may be done in lieu of Step 306 rather than in addition to Step 306).
Additional Chroma Axes
In extended dynamic range and extended contrast range images, it may be beneficial to augment visible wavelength channels with additional channels of image information, both visible and non-visible.
The range of colors available from any given set of red, green, and blue primaries does not include all possible visible colors. The combining of proportions of red, green, and blue primary colors to create other visible colors such as yellow, orange, cyan, and brown, is a property of the human visual system known as the �metamerism�.
As pointed out in the paper entitled �An Example Representation for Image Color and Dynamic Range which is Scalable, Interoperable, and Extensible�, referenced above, it is possible to add additional color primaries to the three primaries of red, green, and blue. In particular, cyan, magenta, and yellow color primaries help to extend the color gamut beyond the range available from most common red, green, and blue primary values. Further, violet and ultraviolet (which brightens phosphorescent colors) can also be conveyed.
Beyond the visible colors, invisible infrared wavelengths have proven useful in penetrating clouds and haze, and in seeing in the dark. Ultraviolet wavelengths can also be useful for seeing low-amplitude visible image details, such as fingerprints and surface coatings.
Further, even in the visible wavelengths, various materials (e.g., smog and underwater algae) often reduce the amount of contrast or dynamic range of some wavelengths. This is why smog can appear brown, giving a brown tint to all objects in the distance, having reduced the blue contrast and dynamic range. This is also why underwater photography can appear green, blue-green, or blue, since the red end of the visible spectrum is reduced in contrast and dynamic range.
The logarithmic relationships between Y, U, and V, as described above, will optimize the coding of color relationships for visible light.
In this aspect of the invention, additional chroma channels are added to the channels encoding three primary wavelengths, typically embodied by RUB or YUV representations. Further, when using a YUV color space, it is also possible to change the makeup of the Y (luminance) channel to favor the highest amplitude image signals. Thus, for example, the green visible channel might be coded using its own chroma channel, with luminance moving to other wavelength regions. This concept can be extended to where Y luminance is infrared, with red, green, and blue (and perhaps other visible and non-visible primaries) each having their own chroma channels.
In accordance with this aspect of the invention, for each new chroma channel, the following should be determined:
1) Should the channel be coded differentially from one or more other channels (usually from luminance, such as U=R−Y)?
2) Should the channel be given full resolution with respect to luminance, or can resolution be reduced without impairing the image quality for a given intended usage?
The determination in 1) is based upon the correlation of each coded channel with other channels. For example, ultraviolet or far-infrared wavelength images may be relatively uncorrelated to visible wavelengths, or to each other. In such a case, these channels might be coded without reference to other channels. However, any visible wavelengths are highly correlated, and thus can almost always benefit from being coded with respect to each other.
Based upon these determinations, a set of image channels can be selected, usually exceeding (or replacing and exceeding) the three primary channels (e.g., YUV). For example, the set of selected image channels may comprise a Y′ luminance channel, and n chroma channels, such as a U′ first chroma channel, a V′ second chroma channel, and an X′ third chroma channel.
Using this example, and applying motion compensated compression, the selected value of Y′ would be coded with full resolution, and the various other chroma channels (U′, V′, X′) would be differentially or independently coded. All channels can utilize the same motion vector and macroblock motion compensation structure as would be used for conventional YUV representations, except that there would be additional channels. Each such channel would utilize an appropriate resolution with respect to Y (as determined in step 2 above). In addition, a QP bias (as described above) can be independently applied to each chroma channel, to ensure that the desired compression chroma quality is achieved.
Even when applied only to visible wavelengths, additional chroma channels can ensure not only extended color range and more accurate color, but also allow additional clarity, detail, and noise fidelity to be applied to such highly visible colors as magenta, orange, yellow, and aqua-cyan. These benefits can be particularly significant for wide-dynamic range and wide-contrast range images.
FIG. 4 is a flowchart showing an illustrative method (which may be computer implemented) for coding additional chroma channels in an image compression system:
Step 401: In an image compression system, determine an optimal luminance representation for an image, selected based upon widest dynamic range and highest resolution, including optional non-visible wavelength image signals.
Step 402: Determine n additional chroma channels to represent the image, where n is at least three.
Step 403: Optionally, for each chroma channel, determine whether it is beneficial to code differentially with respect to luminance and/or one or more other chroma channels.
Step 404: Determine the resolution desired for each chroma channel image signal from an input with respect to the luminance image signal, such resolution being equal to or less than the resolution of the luminance, and optionally apply a resolution reduction.
Step 405: Compress the Y+n chroma image signals using motion compensated compression.
Step 406: Decompress the Y+n chroma images (usually in a different computer).
Step 407: If resolution reduction was applied, restore the original resolutions of the chroma channels.
Step 408: Combine each chroma channel with its differential counterpart, if any, from Step 403 above.
Step 409: Optionally, perform any of the following:
a) Convert the chroma channels to a viewing space, such as RGB, or to spaces having more than three primaries, and view as a true-color image;
b) Perform the conversion of a) but view as a false-color image (such as mapping infrared to green);
c) Use the chroma channels without conversion for processing and/or analysis.
As another option, each chroma channel may have a biased QP value applied (either increasing or decreasing), relative to the QP value used for the luminance channel, to achieve a desired level of quality for each chroma channel (i.e., trading off chroma noise versus higher degree of compression).
Patent CitationsCited PatentFiling datePublication dateApplicantTitleUS4654696Apr 9, 1985Mar 31, 1987Grass Valley Group, Inc.Video signal formatUS4903317Jun 23, 1987Feb 20, 1990Kabushiki Kaisha ToshibaImage processing apparatusUS5073820Oct 23, 1990Dec 17, 1991Olympus Optical Co., Ltd.Image data coding apparatus and coding methodUS5132783Apr 17, 1990Jul 21, 1992Canon Kabushiki KaishaDevice for adjusting white balance by peak detection and smoothingUS5231484Nov 8, 1991Jul 27, 1993International Business Machines CorporationMotion video compression system with adaptive bit allocation and quantizationUS5294974Jul 24, 1992Mar 15, 1994Matsushita Electric Corporation Of AmericaHigh-definition video encoding system having color-sensitive quantizationUS5374958Jun 25, 1993Dec 20, 1994Sony CorporationImage compression based on pattern fineness and edge presenceUS5408270Jun 24, 1993Apr 18, 1995Massachusetts Institute Of TechnologyAdvanced television systemUS5426463Feb 22, 1993Jun 20, 1995Rca Thomson Licensing CorporationApparatus for controlling quantizing in a video signal compressorUS5438374Dec 10, 1993Aug 1, 1995At&T Corp.System and method for filtering video signalsUS5442407Mar 22, 1994Aug 15, 1995Matsushita Electric Corporation Of AmericaVideo signal noise reduction system using time-varying filter coefficientsUS5446498Jan 24, 1994Aug 29, 1995Matsushita Electric Industrial Co., Ltd.Scaling interlaced images for encoding and decoding of digital video signals of multiple scanning standardsUS5465119Feb 22, 1991Nov 7, 1995DemografxPixel interlacing apparatus and methodUS5475435Oct 28, 1993Dec 12, 1995Sony CorporationLayer encoding apparatus and layer decoding apparatus for input non-interlace video signalUS5555030Jan 5, 1993Sep 10, 1996Accom, Inc.Component video signal to composite video signal encoding system, control system and methodUS5661524Mar 8, 1996Aug 26, 1997International Business Machines CorporationMethod and apparatus for motion estimation using trajectory in a digital video encoderUS5668600Oct 28, 1996Sep 16, 1997Daewoo Electronics, Co., Ltd.Method and apparatus for encoding and decoding a video signal using feature point based motion estimationUS5737032Sep 5, 1995Apr 7, 1998Videotek, Inc.Serial digital video processing with concurrent adjustment in RGB and luminance/color differenceUS5742343Aug 19, 1996Apr 21, 1998Lucent Technologies Inc.Scalable encoding and decoding of high-resolution progressive videoUS5748903Sep 29, 1995May 5, 1998Intel CorporationEncoding images using decode rate controlUS5812194Jun 17, 1997Sep 22, 1998Sony CorporationMotion compensated video processingUS5825421Dec 27, 1996Oct 20, 1998Matsushita Electronic Industrial Co., Ltd.Video coding method and decoding method and devices thereofUS5825680Jun 21, 1996Oct 20, 1998Digital Equipment CorporationMethod and apparatus for performing fast divisionUS5835498Jun 14, 1996Nov 10, 1998Silicon Image, Inc.System and method for sending multiple data signals over a serial linkUS5852565Jan 30, 1996Dec 22, 1998DemografxTemporal and resolution layering in advanced televisionUS5877754Apr 8, 1994Mar 2, 1999Intel CorporationProcess, apparatus, and system for color conversion of image signalsUS5920353Dec 3, 1996Jul 6, 1999St Microelectronics, Inc.Multi-standard decompression and/or compression deviceUS5930386Feb 1, 1995Jul 27, 1999Canon Kabushiki KaishaImage processing apparatus and methodUS5933360Sep 18, 1996Aug 3, 1999Texas Instruments IncorporatedMethod and apparatus for signal compression and processing using logarithmic differential compressionUS5988863Dec 21, 1998Nov 23, 1999DemografxTemporal and resolution layering in advanced televisionUS6031937May 19, 1994Feb 29, 2000Next Software, Inc.Method and apparatus for video compression using block and wavelet techniquesUS6069914Sep 19, 1996May 30, 2000Nec Research Institute, Inc.Watermarking of image data using MPEG/JPEG coefficientsUS6115421Apr 25, 1997Sep 5, 2000Matsushita Electric Industrial Co., Ltd.Moving picture encoding apparatus and methodUS6163574May 2, 1995Dec 19, 2000Sony CorporationMotion picture encoding system for either intra-frame encoding or inter-frame encodingUS6172768Feb 5, 1998Jan 9, 2001Canon Kabushiki KaishaHalftoning with changeable error diffusion weightsUS6215824May 1, 1998Apr 10, 2001Boom CorporationTranscoding method for digital video networkingUS6236727Jun 24, 1997May 22, 2001International Business Machines CorporationApparatus, method and computer program product for protecting copyright data within a computer systemUS6268886Nov 12, 1998Jul 31, 2001Lg Electronics Inc.Apparatus for converting picture format in digital TV and method of the sameUS6301304Jun 17, 1998Oct 9, 2001Lsi Logic CorporationArchitecture and method for inverse quantization of discrete cosine transform coefficients in MPEG decodersUS6327602Apr 7, 1999Dec 4, 2001Lg Electronics Inc.Inverse discrete cosine transformer in an MPEG decoderUS6332194Jun 5, 1998Dec 18, 2001Signafy, Inc.Method for data preparation and watermark insertionUS6430222Jul 6, 1999Aug 6, 2002Sharp Kabushiki KaishaMoving picture coding apparatusUS6434196Apr 2, 1999Aug 13, 2002Sarnoff CorporationMethod and apparatus for encoding video informationUS6442203Nov 5, 1999Aug 27, 2002DemografxSystem and method for motion compensation and frame rate conversionUS6507615Apr 28, 1999Jan 14, 2003Sony CorporationVideo signal processing apparatus and video signal processing methodUS6629104Nov 22, 2000Sep 30, 2003Eastman Kodak CompanyMethod for adding personalized metadata to a collection of digital imagesUS6650708Oct 10, 2000Nov 18, 2003Matsushita Electric Industrial Co., Ltd.Video signal encoding apparatusUS6654417Jan 26, 1998Nov 25, 2003Stmicroelectronics Asia Pacific Pte. Ltd.One-pass variable bit rate moving pictures encodingUS6654420Oct 27, 2000Nov 25, 2003Koninklijke Philips Electronics N.V.Video encoding-methodUS6728317Apr 7, 2000Apr 27, 2004Dolby Laboratories Licensing CorporationMoving image compression quality enhancement using displacement filters with negative lobesUS6816552Jul 11, 2001Nov 9, 2004Dolby Laboratories Licensing CorporationInterpolation of video compression framesUS6957350Apr 3, 2000Oct 18, 2005Dolby Laboratories Licensing CorporationEncrypted and watermarked temporal and resolution layering in advanced televisionUS7113152Jun 7, 2001Sep 26, 2006Genoa Color Technologies Ltd.Device, system and method for electronic true color displayUS20010028725Dec 29, 2000Oct 11, 2001Toshiyuki NakagawaInformation processing method and apparatusUS20020154693Mar 2, 2001Oct 24, 2002Demos Gary A.High precision encoding and decoding of video imagesUS20020186766Dec 22, 2000Dec 12, 2002Joseph AdelaideDevice for image encoding and decodingUS20030112871Jul 11, 2001Jun 19, 2003Demos Gary A.Interpolation of video compression framesUS20030112873Jul 11, 2001Jun 19, 2003Demos Gary A.Motion estimation for video compression systemsUS20040005004Jun 28, 2002Jan 8, 2004Demos Gary A.Interpolation of video compression framesUS20050254649Jul 21, 2005Nov 17, 2005Demos Gary AEncrypted and watermarked temporal and resolution layering in advanced televisionUS20060002467 *Jun 29, 2005Jan 5, 2006At&T Corp.System, method and computer-readable medium for encoding a signal into macroblocksCA2127151A1Jun 30, 1994Mar 22, 1995Atul PuriSpatially scalable video encoding and decodingDE19714886A1Apr 10, 1997Nov 6, 1997Philips PatentverwaltungGamma correction process for colour film video signalsEP0511778A2Apr 23, 1992Nov 4, 1992AT&amp;T Corp.Multiple frame motion estimationEP0531041A1Aug 26, 1992Mar 10, 1993Matsushita Electric Industrial Co., Ltd.Adaptive DCT/DPCM video signal coding methodEP0534350A2Sep 21, 1992Mar 31, 1993Matsushita Electric Industrial Co., Ltd.Derivation and use of motion vectors in a differential pulse code modulation systemEP0634871A2Jul 6, 1994Jan 18, 1995AT&amp;T Corp.Scalable encoding and decoding of high-resolution progressive videoEP1369820A2May 27, 2003Dec 10, 2003Microsoft CorporationSpatiotemporal prediction for bidirectionally predictive (B) pictures and motion vector prediction for multi-picture reference motion compensationGB2266635A Title not availableJP2001008209A Title not availableJP2001045476A Title not availableJP2001086356A Title not availableJP2001177731A Title not availableJP2001252088A Title not availableJPH104497A Title not availableJPH04311195A Title not availableJPH04323960A Title not availableJPH06165150A Title not availableJPH06350995A Title not availableJPH07170516A Title not availableJPH11239351A Title not availableJPS62264784A Title not availableJPS62283776A Title not availableWO1995004433A1Jul 29, 1994Feb 9, 1995British TelecommProcessing image dataWO1997028507A1Jan 24, 1997Aug 7, 1997DemografxTemporal and resolution layering in advanced televisionWO1999020040A1Oct 9, 1998Apr 22, 1999Faroudja Lab IncFilm source video detectionWO2003007119A2Jul 11, 2002Jan 23, 2003DemografxInterpolation of video compression framesWO2003041041A2Oct 29, 2002May 15, 2003Pioneer CorpDispl ay panel driving apparatus with reduced power lossWO2004004310A2Jun 27, 2003Jan 8, 2004Dolby Lab Licensing CorpImproved interpolation of video compression frames* Cited by examinerNon-Patent CitationsReference1"Color Science: Concepts and Methods, Quantitative Data and Formulae," Wyszecki and Stiles, John Wiley and Sons, 2nd Edition, p. 485-489 (1982).2"IEEE Standard Specification for the Implementations of 8�8 Inverse Discrete Cosine Transforms," IEEE Std 1180-1990, The Institute of Electrical and Electronics Engineers, Inc.; United States of America, 13 pages (1991).3Aravind, R. et al., "Packet Loss Resilience of MPEG-2 Scalable Video Coding Algorithms," IEEE Transactions on Circuits and Systems for Video Technology 6(5): 426-435 (Oct. 1996).4Australian Examination Report, Application Serial No. 2002316676, dated Oct. 4, 2006, 2 pages.5Australian Examination Report, Application Serial No. 2007203288, dated Aug. 27, 2007, 4 pages.6Australian Examination Report, Application Serial No. 2007203288, dated Sep. 20, 2007, 2 pages.7Australian Notice of Acceptance, Application Serial No. 2007203288, dated Nov. 21, 2007, 16 pages.8Bloomfield, L., "Copy Protection-d�j� vu," Broadcast Engineering 40(11): 14-15 (Oct. 1998).9Bloomfield, L., "Copy Protection�d�j� vu," Broadcast Engineering 40(11): 14-15 (Oct. 1998).10Canadian Office Action issued in 2,631,107 on Nov. 3, 2009, 3 pages.11Canadian Office Action, Application Serial No. 2439727, dated Aug. 15, 2007, 3 pages.12Canadian Patent Grant, Canadian Application No, 2.452,506, dated Apr. 8, 2008, 3 pages.13Chinese Patent Publication; Patent No. ZL 02813792.2, dated Apr. 5, 2006, 25 pages.14Deed of Letters Patent and the published specification for the granted patent for Australian Patent No. AU 20022316676 B2; Granted Sep. 6, 2007, 41 pages.15Demos, G., "An Example Representation for Image Color and Dynamic Range Which is Scalable, Interoperable, and Extensible," 135th Technical Conference, Society of Motion Picture and Televison Engineers, Oct. 1993, Los Angeles, CA, 22 pages.16Demos, G., "Temporal and Resolution Layering in Advanced Television," SPIE 2663: 52-68 (Nov. 1995).17Demos, G., "The Use of Logarithmic and Density Units for Pixels," SMPTE Journal 100(10): Oct. 1990, pp. 805-816 (1990).18Demos, G., A Comparison of Hierarchical High Definition Imagry Coding Schema,: DemoGraFX Corp. IEEE, pp. 68-75 (1992).19English language abstract for JP 06165150, published Jun. 10, 1994, entitled: "Dynamic Picture Coding/Decoding Device".20European Search Report and Opinion issued on Sep. 1, 2010 in EP 10007124.0.21European Search Report issued on Mar. 8, 2010, in EP 08075656.2, 7 pages.22European Search Report issued on Mar. 8, 2010, in EP 08075656.2.23European Search Report, Application Serial No. 02747000.4, dated May 13, 2008, 6 pages.24H.261, ITU-T Telecommunication Standardization Sector of ITU, Line Transmission of non-telephone signals. Video Codec for Audiovisual Services at p. X64 kbits, (Mar. 1993), 32 pages.25H.263, Appendix III, ITU-T Telecommunication Standardization Sector of ITU, Series H: Audiovisual and Multimedia Systems, Infrastructure of audiovisual services-coding of moving video. Video coding for low bit rate communication, Appendix III: Examples for H.263 encoder/decoder implementations, (Jun. 2001), 48 pages.26H.263, Appendix III, ITU-T Telecommunication Standardization Sector of ITU, Series H: Audiovisual and Multimedia Systems, Infrastructure of audiovisual services�coding of moving video. Video coding for low bit rate communication, Appendix III: Examples for H.263 encoder/decoder implementations, (Jun. 2001), 48 pages.27H.263, ITU-T Telecommunication Standardization Sector of ITU, Series H: Audiovisual and Multimedia Systems, Infrastructure of audiovisual services-coding of moving video. Video coding for low bit rate communication, (Jan. 2005), 226 pages.28H.263, ITU-T Telecommunication Standardization Sector of ITU, Series H: Audiovisual and Multimedia Systems, Infrastructure of audiovisual services�coding of moving video. Video coding for low bit rate communication, (Jan. 2005), 226 pages.29ISO/IEC 14496-2 International Standard, Information technology-coding of audio-visual objects-Part 2: visual, 2nd Edition, Amendment 2: Streaming video profile, Feb. 1, 2002, 64 pages.30ISO/IEC 14496-2 International Standard, Information technology�coding of audio-visual objects-Part 2: visual, 2nd Edition, Amendment 2: Streaming video profile, Feb. 1, 2002, 64 pages.31ISO/IEC 14496-2 International Standard, Information technology-coding of audio-visual objectsPart 2: visual, 2nd Editionm Dec. 1, 2001, 536 pages.32ISO/IEC 14496-2 International Standard, Information technology�coding of audio-visual objectsPart 2: visual, 2nd Editionm Dec. 1, 2001, 536 pages.33Japanese Office Action with translation, Application No. 2003-512824, dated Oct. 15, 2008, 4 pages.34Japanese Office Action, Application Serial No. 2003-512824, dated Mar. 18, 2008, 46 pages.35Lim, Jae S., "A migration path to a better digital television system," SMPTE Journal 103(1): 2-6 (Jan. 1, 1994).36Notice of Allowance, Canadian Patent Application Serial No. 2452506, dated Dec. 6, 2007, 2 pages.37Office Action issued in U.S. Appl. No. 09/905,039 on Apr. 23, 2008, 10 pages.38Office Action issued in U.S. Appl. No. 09/905,039 on Apr. 5, 2004, 7 pages.39Office Action issued in U.S. Appl. No. 09/905,039 on Apr. 8, 2005 ,8 pages.40Office Action issued in U.S. Appl. No. 09/905,039 on Aug. 8, 2007, 11 pages.41Office Action issued in U.S. Appl. No. 09/905,039 on Dec. 23, 2004, 7 pages.42Office Action issued in U.S. Appl. No. 09/905,039 on Feb. 16, 2006, 6 pages.43Office Action issued in U.S. Appl. No. 09/905,039 on Jan. 15, 2004, 6 pages.44Office Action issued in U.S. Appl. No. 09/905,039 on Jul. 17, 2003, 9 pages.45Office Action issued in U.S. Appl. No. 09/905,039 on Sep. 2, 2004, 5 pages.46Office Action issued on Apr. 21, 2009 in corresponding Canadian Patent Application No. 2,621,107, 3 pages.47Office Action issued on Jan. 20, 2009, in corresponding Korean Patent Application No. 10-2004-7000367, and English translation (8 pages).48Office Action, U.S. Appl. No. 09/798,346, dated Apr. 3, 2006, 12 pages.49Office Action, U.S. Appl. No. 09/798,346, dated Aug. 23, 2007, 8 pages.50Office Action, U.S. Appl. No. 09/798,346, dated Nov. 27, 2006, 7 pages.51Patent Abstracts of Japan vol. 1999, No. 13 (Nov. 30, 1999) re: JP 11239351.52Patent Abstracts of Japan, vol. 1995, No. 03 (Apr. 28, 2005) re: JP 06350995.53Pinkerton, Janet, "Digital video stymied by content protection," Dealscope Consumer Electronics Marketplace, Philadelphia, Jan. 1999, vol. 4, issue 1, p. 32.54Puri et al., "Temporal Resolution Scalable Video Coding," Image Processing. 1994 International Conference, IEEE, pp. 947-951 (1994).55Search and Examination Report issued on Dec. 2, 2009 in Singapore Patent Application No. 200805819-0.56Singapore Office Action, Application Serial No. 200601486-4, dated Aug. 28, 2006, 12 pages.57Singapore Office Action, Application Serial No. 200601486-4, dated Sep. 7, 2007, 7 pages.58Vincent, A., et al., "Spatial Prediction in Scalable Video Coding," International Broadcasting Convention, IEEE Conference Publication No. 413, RAI International Congress and Exhibition Centre, Amsterdam, The Netherlands, Sep. 14-18, 1995, pp. 244-249.Referenced byCiting PatentFiling datePublication dateApplicantTitleUS8374237 *Mar 2, 2001Feb 12, 2013Dolby Laboratories Licensing CorporationHigh precision encoding and decoding of video imagesUS8488670 *Feb 12, 2013Jul 16, 2013Dolby Laboratories Licensing CorporationHigh precision encoding and decoding of video imagesUS8576907Jan 7, 2011Nov 5, 2013Dolby Laboratories Licensing CorporationHigh precision encoding and decoding of video imagesUS8594188Jan 7, 2011Nov 26, 2013Dolby Laboratories Licensing CorporationHigh precision encoding and decoding of video imagesUS8737466Aug 3, 2011May 27, 2014Dolby Laboratories Licensing CorporationMethod and system for improving compressed image chroma informationUS8947449Feb 21, 2012Feb 3, 2015Google Inc.Color space conversion between semi-planar YUV and planar YUV formatsUS20020154693 *Mar 2, 2001Oct 24, 2002Demos Gary A.High precision encoding and decoding of video imagesUS20100201870 *Feb 9, 2010Aug 12, 2010Martin LuessiSystem and method for frame interpolation for a compressed video bitstream* Cited by examinerClassifications U.S. Classification375/240.03International ClassificationH04N7/30, H04N7/12, H04N11/04, G06T3/40, G06T9/00, H04N7/26Cooperative ClassificationH04N19/61, H04N19/126, H04N19/186, G06T3/4015, G06T9/007, H04N7/12European ClassificationH04N7/26A4Q2, H04N7/50, G06T9/00T, H04N7/26A6C8, H04N7/12, H04N7/26A8U, G06T3/40CLegal EventsDateCodeEventDescriptionDec 15, 2014FPAYFee paymentYear of fee payment: 4Jul 29, 2008ASAssignmentOwner name: DOLBY LABORATORIES LICENSING CORPORATION, CALIFORNFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:DOLBY LABORATORIES, INC.;REEL/FRAME:021305/0038Effective date: 20030905Owner name: DOLBY LABORATORIES, INC., CALIFORNIAFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:DEMOGRAFX;REEL/FRAME:021305/0014Effective date: 20030417Jul 28, 2008ASAssignmentOwner name: DEMOGRAFX, CALIFORNIAFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:DEMOS, GARY A.;REEL/FRAME:021302/0612Effective date: 20010925Owner name: DOLBY LABORATORIES LICENSING CORPORATION, CALIFORNFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:DOLBY LABORATORIES, INC.;REEL/FRAME:021302/0840Effective date: 20030905Owner name: DOLBY LABORATORIES, INC., CALIFORNIAFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:DEMOGRAFX;REEL/FRAME:021302/0668Effective date: 20030417RotateOriginal ImageGoogle Home - Sitemap - USPTO Bulk Downloads - Privacy Policy - Terms of Service - About Google Patents - Send FeedbackData provided by IFI CLAIMS Patent Services