Source: http://www.google.com/patents/US7885458?dq=6,757,710
Timestamp: 2014-07-11 17:31:30
Document Index: 273657854

Matched Legal Cases: ['art 60', 'art 60', 'art 60', 'art 60', 'art 60', 'art 60', 'art 60', 'art 60', 'art 60', 'art 60', 'art 60']

Patent US7885458 - Illuminant estimation using gamut mapping and scene classification - Google PatentsSearch Images Maps Play YouTube News Gmail Drive More »Sign in<nobr>Advanced Patent Search</nobr>PatentsMethods and systems for processing image data are described. A scene classifier determines the probabilities that various scene classes are associated with the image data. A gamut mapper determines the probabilities that various combinations of illuminant color and scene class are associated with the...http://www.google.com/patents/US7885458?utm_source=gb-gplus-sharePatent US7885458 - Illuminant estimation using gamut mapping and scene classificationAdvanced Patent SearchPublication numberUS7885458 B1Publication typeGrantApplication numberUS 11/262,107Publication dateFeb 8, 2011Filing dateOct 27, 2005Priority dateOct 27, 2005Publication number11262107, 262107, US 7885458 B1, US 7885458B1, US-B1-7885458, US7885458 B1, US7885458B1InventorsShang-Hung LinOriginal AssigneeNvidia CorporationExport CitationBiBTeX, EndNote, RefManPatent Citations (43), Non-Patent Citations (2), Referenced by (3), Classifications (7), Legal Events (1) External Links: USPTO, USPTO Assignment, EspacenetIlluminant estimation using gamut mapping and scene classificationUS 7885458 B1Abstract Methods and systems for processing image data are described. A scene classifier determines the probabilities that various scene classes are associated with the image data. A gamut mapper determines the probabilities that various combinations of illuminant color and scene class are associated with the image data. The probabilities from the scene classifier are used to weight the probabilities from the gamut mapper. The weighted results can be used to select an illuminant color. The image data can be adjusted to compensate for the selected illuminant color.
accessing a frame of image data for an illuminant color selected from a plurality of illuminant colors and for each scene class in a plurality of scene classes, accessing first values that correspond to probabilities that combinations of said illuminant color and said scene classes are associated with said frame of image data, each of said first values corresponding to a combination of said illuminant color and a scene class;
weighting said first values with respective second values to produce weighted values, wherein said weighted values correspond to said illuminant color, and wherein said second values correspond to probabilities that said scene classes are associated with said frame of image data, each of said second values corresponding to one of said scene classes;
summing said weighted values to produce a score for said illuminant color;
repeating said accessing, said weighting, and said summing for each of the remaining said illuminant colors to produce a score for each of said illuminant colors; and
selecting a particular illuminant color of said illuminant colors based on the scores for said illuminant colors.
2. The method of claim 1 wherein said selecting comprises using the maximum of said scores to select said particular illuminant color.
3. The method of claim 1 further comprising adjusting said image data to compensate for said particular illuminant color.
4. The method of claim 1 wherein said first values are accessed from scene class-dependent lookup tables.
5. The method of claim 1 wherein said processing is performed per pixel of said image data.
6. The method of claim 1 wherein said image data is segmented into segments according to color, wherein a segment comprises pixels of substantially the same color and wherein said processing is performed per segment.
7. The method of claim 6 wherein said pixels are selected regardless of spatial dependence.
8. The method of claim 1 wherein each of said first values comprises a product of i) a probability that said image data is produced for said combination of said illuminant color and said scene class and ii) a probability that said illuminant color is associated with said scene class.
9. The method of claim 8 wherein said product is divided by a probability that said image data is associated with said scene class.
10. A non-transitory computer-readable medium having computer-executable program code embodied therein for causing a computer system to perform a method of processing image data, said method comprising:
for a plurality of illuminant colors and for a plurality of scene classes, identifying a plurality of respective first values, wherein a first value corresponds to a probability that a combination of an illuminant color and a scene class is associated with a frame of image data;
for said plurality of scene classes, identifying a plurality of respective second values, wherein a second value corresponds to a probability that a scene class is associated with said frame of image data;
for said plurality of illuminant colors, weighting said plurality of first values using said plurality of second values to determine a plurality of third values, wherein a third value corresponds to a probability that an illuminant color is associated with said frame of image data; and
selecting a particular illuminant color based on said plurality of third values.
11. The computer-readable medium of claim 10 wherein said method further comprises adjusting said image data to compensate for said particular illuminant color.
12. The computer-readable medium of claim 10 wherein said method further comprises using the maximum value of said plurality of third values to select said particular illuminant color.
13. The computer-readable medium of claim 10 wherein said identifying a plurality of respective first values further comprises, for said plurality of scene classes, accessing a respective scene class-dependent lookup table to identify a plurality of respective fourth values, wherein a fourth value corresponds to a probability of obtaining said image data given an illuminant color.
14. The computer-readable medium of claim 10 wherein said processing is performed per pixel.
15. The computer-readable medium of claim 10 wherein said image data is segmented into segments according to color, wherein a segment comprises pixels of substantially the same color and wherein said processing is performed per segment.
a processor coupled to said memory and for processing said image data, said processor operable for determining first values that correspond to probabilities that respective combinations of act illuminant colors and scene classes are associated with a frame of said image data, for determining second values that correspond to probabilities that respective said scene classes are associated with said frame of image data, for weighting said first values with respective said second values, and for adjusting said image data to compensate for a selected illuminant color that is selected from said illuminant colors based on said weighting.
17. The imaging system of claim 16 wherein said first values and said second values are determined for each of said scene classes to generate a plurality of illuminant color-dependent weighted values, wherein said weighted values are summed for each of a plurality of illuminant colors to determine a plurality of summed values, wherein said plurality of summed values is used to choose said selected illuminant color.
18. The imaging system of claim 17 wherein the maximum of said summed values is used to choose said selected illuminant color.
19. The imaging system of claim 16 wherein said first values are determined using scene class-dependent lookup tables comprising probabilities that said image data would be obtained given an illuminant color.
20. The imaging system of claim 16 wherein said processing is performed per pixel.
21. The imaging system of claim 16 wherein said processor is also operable for segmenting said image data into segments according to color, wherein a segment comprises pixels of substantially the same color and wherein said processing is performed per segment. Description
FIELD OF THE INVENTION Embodiments of the present invention generally relate to the processing of image data.
BACKGROUND ART The human visual system perceives the same range of colors under a wide variety of scene illuminations. A white piece of paper remains resolutely white independent of the color of the illuminant (the color of the light under which the piece of paper is viewed). In contrast, color imaging systems (e.g., digital cameras) are less color constant in that they will often incorrectly infer the color of the illuminant. Unless the influence of the color of the illuminant is compensated, the digital camera cannot acceptably reproduce actual scene colors. The process of �correcting� the image data to compensate the effect of the illuminant color is commonly referred to as white balancing.
In white balancing, the color of the scene illumination either is measured or is estimated from the image data, and then the image data is adjusted to compensate for the effect of the illuminant. Because it is not practical to equip each camera with a dedicated illumination sensor and then expect users to calibrate to a white reference every time a picture is taken, conventional cameras typically estimate the illuminant color from the image data.
According to one conventional white balancing method, all of the pixel color values in an image are averaged, and the image data is then adjusted so that the average of the pixel color values is gray. According to another conventional white balancing method, the brightest spot (a specularity) in an image is presumed to be the light source, the color of that specularity is presumed to be the color of the illuminant, and the image data is adjusted to compensate for the color of the illuminant. These types of methods are simple to implement but unfortunately they fail in many scenarios because they are based on assumptions that do not always hold true. For example, the average color of a scene may not actually be gray, and a specularity may not appear in an image (and when present may be difficult to find).
Other attempts to address the illuminant estimation problem use a probabilistic framework. Given an input image Cim, the objective of a probabilistic approach is to recover Pr(E|Cim)�the probability that E was the scene illuminant given Cim. The illuminant hypothesis Ei that produces the largest Pr(E|Cim) is the estimated scene illuminant. According to Bayes' Rule:
Pr ⁡ ( E ❘ C im ) = Pr ⁡ ( C im ❘ E ) ⁢ Pr ⁡ ( E ) Pr ⁡ ( C im ) ; ( 1 ) where Pr(E) is the probability that the scene illuminant is E, and Pr(Cim|E) is the probability of observing the image Cim under illuminant E.
Pr(Cim) is a constant regardless of the illuminant, so it can be omitted from equation (1) without changing the outcome of the maximum probability estimation:
Pr(E|C im)=Pr(C im |E)Pr(E); (2)
where Pr(E) is the probability of illuminant E. In most cases, Pr(E) is assigned an equal value for all illuminant colors. The likelihood function Pr(Cim|E) is typically estimated by assuming that the chromaticity values of the pixels in an image are independent of each other, so that Pr(Cim|E) can be replaced in equation (2) by the product of the conditional probabilities of the pixel given illuminant E:
Pr ⁢ ( E ❘ C im ) = [ ∏ ∀ c ⁢ ∈ C im ⁢ ⁢ Pr ⁡ ( c ❘ E ) ] ⁢ Pr ⁡ ( E ) ; ( 3 ) where c is the color (chromaticity) of a pixel in image Cim, and Pr(c|E) indicates how �probable� it is that the camera would capture a color c under the illuminant color E (that is, given an illuminant color E, what is the probability that color c would be observed in the image).
In order to generate Pr(c|E), a correlation matrix is created to correlate possible image colors with potential colors of scene illuminants. The correlation matrix characterizes, for each illuminant color considered, the range of possible image colors that can be observed under that illuminant color. In other words, a �camera gamut� for each potential illuminant color is constructed.
To implement a correlation matrix in a camera, training image data is used to create a two-dimensional (2D) or even three-dimensional (3D) color histogram for each illuminant color considered, and the histograms are installed in the camera's memory as lookup tables. If pixel color is represented using eight (8) bits per pixel, and the probabilistic score Pr(c|E) is represented in an 8-bit format, a 2D lookup table requires 64,000 bytes of memory per illuminant color, and a 3D lookup table requires 16,000,000 bytes of memory per illuminant color. Because multiple illuminant colors are typically considered, an extensive amount of memory can be consumed by a conventional correlation matrix.
SUMMARY OF THE INVENTION Thus, one problem with conventional white balancing approaches is that they can consume an extensive amount of a camera's memory. Another problem with conventional approaches is that they are heavily biased by the training image database that is used to generate the correlation matrix. If the training image database is not carefully selected, estimation of the illuminant color can be adversely affected. For example, it is important to match the training image database with the type of image data expected to be encountered when the camera is put to use. Studies on this subject have demonstrated that conventional training image databases are problematic.
Consequently, methods and/or systems that can be used for white balancing, but that consume less memory, would be advantageous. Methods and/or systems that can accomplish this with an improved training image database would also be advantageous. Embodiments in accordance with the present invention provide these and other advantages.
In overview, embodiments in accordance with the present invention utilize an integrated framework to improve the performance of probabilistic or correlation-based illuminant estimation methods. In one embodiment, incoming image data goes through a scene classifier and a gamut mapper. The scene classifier determines the probabilities that various scene classes (e.g., indoor, landscape, sky, portrait, etc) are associated with the image data. The gamut mapper determines the probabilities that various combinations of illuminant color and scene class are associated with the image data. The probabilities from the scene classifier are used to weight the probabilities from the gamut mapper. The weighted results can be used to select an illuminant color. The image data can be adjusted to compensate for the selected illuminant color.
According to embodiments of the present invention, the correlation matrix used by the gamut mapper is scene class-dependent. As a result, the process of preparing the training image database can be better focused. Also, because construction of the correlation matrix focuses on the most probable colors that can occur within the scene classes considered, the effective camera gamut is contracted and thus the memory requirement for the correlation matrix can be reduced. These and other objects and advantages of the various embodiments of the present invention will be recognized by those of ordinary skill in the art after reading the following detailed description of the embodiments that are illustrated in the various drawing figures.
FIG. 1 is a block diagram showing an image processor according to one embodiment of the present invention.
FIG. 2 is a data flow diagram showing information flow through an image processor according to one embodiment of the present invention.
FIG. 3 is a data flow diagram showing operations performed by an image processor according to one embodiment of the present invention.
FIG. 4 is also a data flow diagram showing operations performed by an image processor according to one embodiment of the present invention.
FIG. 5 is a block diagram of an imaging system upon which embodiments in accordance with the present invention may be implemented.
FIG. 6 is a flowchart of a method for processing image data according to one embodiment of the present invention.
It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present invention, discussions utilizing terms such as �determining,� �weighting,� �selecting,� �repeating,� �summing,� �using,� �identifying,� �adjusting,� �accessing� or the like, refer to actions and processes (e.g., flowchart 60 of FIG. 6) of a computer system or similar electronic computing device or processor. The computer system or similar electronic computing device manipulates and transforms data represented as physical (electronic) quantities within the computer system memories, registers or other such information storage, transmission or display devices. The present invention is well suited to use with other computer systems.
FIG. 1 is a block diagram showing an image processor 12 according to one embodiment of the present invention. Raw image data 11 is captured (that is, for example, a picture is taken with a digital camera), then processed by image processor 12 to produce processed image data 13. In the present embodiment, image processor 12 incorporates a white balance module 14 for illuminant estimation and compensation. Image processor 12 may perform other functions. In one embodiment, image processor 12 is implemented as part of a digital camera (see FIG. 5, for example).
The image data 11 and 13 (FIG. 1) can be referred to as a frame of image data. The image data 11 and 13 may represent a �still� picture. The image data 11 and 13 may also represent a video or movie, because a video is essentially a sequence of still images.
FIG. 2 is a data flow diagram showing information flow through image processor 12 (FIG. 1) according to one embodiment of the present invention. In the present embodiment, raw image data 11 (C or Cim may be used interchangeably herein) is forwarded to scene classifier 22, gamut mapper 23 and image adjustor 25.
Scene classifier 22 determines the probability that SN is the scene class given the raw image data 11. This probability is expressed herein as Pr(SN|C) or Pr(SN|Cim). For example, there may be N scene classes, and scene classifier 22 determines, given the raw image data 11, a first probability that S1 is the scene class, a second probability that S2 is the scene class, and so on.
Scene classification is a technique used for image database indexing and content retrieval. Scene classification is known in the art. The classification can be done by image segmentation or by qualitative scene structure encoding. Several predetermined scene classes, such as but not limited to portrait, landscape, beach, sky, ski, sunset and indoor, are assigned and trained in the scene classifier a priori. The output of the scene classifier is the confidence level (expressed herein as a probability) that the given image resembles a certain class of scene.
If scene classifier 22 categorizes the images into N different scene classes, then Pr(E|Cim)�the probability that E was the scene illuminant given Cim�can be expressed as:
Pr ⁢ ( E ❘ C im ) = ∑ N ⁢ ⁢ Pr ⁡ ( S ❘ C im ) ⁢ Pr ⁡ ( E ❘ C im , S ) ; ( 4 ) where Pr(S|Cim) is the probabilistic output score (e.g., the confidence level or probability) of the scene classifier 22 for the scene class S, and Pr(E|Cim,S) is the probability that E was the scene illuminant given Cim for the scene class S. Pr(E|Cim,S) can be further expanded by Bayes' Rule as follows:
Pr ⁢ ( E ❘ C im ) = ∑ N ⁢ ⁢ Pr ⁡ ( S ❘ C im ) ⁢ p ⁡ ( C im ❘ E , S ) ⁢ Pr ⁡ ( E ❘ S ) p ⁡ ( C im ❘ S ) ; ( 5 ) where p(Cim|E,S) is the probability of observing Cim given E and S (that is, what is the probability of obtaining the image data Cim if E is the illuminant color and S is the scene class), Pr(E|S) is the probability that the illuminant is E given the scene class S, and p(Cim|S) is the probability of observing Cim given S (that is, what is the probability of obtaining the image data Cim if S is the scene class).
In one embodiment, p(Cim|E,S), Pr(E|S) and p(Cim|S) are implemented as lookup tables (LUTs) 1, 2 and 3, respectively. The LUTs are scene class-dependent; thus, there is a lookup table for each scene class S. In one embodiment, gamut mapper 23 determines P(E|C,S) for each of the M illuminant colors E being considered, where P(E|C,S) is the probability that E is the illuminant color given the scene class S and the image data C.
The probability Pr(S|Cim) from scene classifier 22 can be viewed as a weighting factor that is applied by illuminant estimator 24 to the scene class-dependent gamut mapping results P(E|C,S) from gamut mapper 23. The probability Pr(S|Cim) can be obtained directly from the output of the scene classifier 22, because Pr(S|Cim) represents the probability of scene class S if the incoming image is Cim. This is illustrated further by FIGS. 3 and 4, below.
With reference to FIG. 2, illuminant estimator 24 determines Pr(E|Cim)�the probability that E was the scene illuminant given Cim�for each of the M illuminant colors being evaluated. In one embodiment, the illuminant color E that corresponds to the largest value of Pr(E|Cim) is selected. Using this information, image adjustor 25 adjusts the raw image data 11 to compensate for the effect of the illuminant color E, thereby generating processed image data 26. Processed image data 26 may be displayed to the user or downloaded to a personal computer or printer, for example.
The correlation matrix p(Cim|E,S) can be implemented as a 2D or 3D histogram (with a smaller data range relative to conventional art), or by other clustering/classification schemes, such as but not limited to VQ (vector quantization), SVM (support vector machine), and neural network.
The value of p(Cim|S) is independent of the illumination color, and is implemented as a constant value for each scene class S.
Pr(E|S) is a pre-programmed value. As mentioned above, it indicates how probable it is that illuminant E can occur if the scene class is S. For example, if the image is categorized as landscape, the score for daylight illuminants would be higher than incandescent or fluorescent illuminants.
Significantly, instead of a large multipurpose correlation matrix as applied in equation (3), embodiments in accordance with the present invention decompose the correlation matrix into several scene class-dependent lookup tables. One of the advantages of creating scene class-dependent LUTs is that the preparation of the training image database is more focused and guided. For example, to build a correlation matrix for the portrait scene class, the skin tone color becomes a major color cue, as the blue tone is for the sky/sea classes and the green tone is for the landscape/vegetation classes. The major color cues will be the major contributor for building the correlation matrix of the corresponding scene class. More major color samples are collected, finer resolution at the major color zone is prepared, and a higher weighting factor is assigned. The non-major colors can either be de-emphasized or even omitted. Down-playing the non-major colors reduces the chance of a false positive error. It is also expected to save memory space, because the correlation matrix covers only the spread of the major colors instead of the whole color spectrum.
There could be more than one major color for a scene class. For example, the landscape class may use green, blue, and gray as major colors, and the sunset class can use yellow and red.
Equation (5) is implemented on a per-pixel basis. If the scene classifier 22 applies image segmentation techniques, the likelihood function p(Cim|E,S) of equation (5) can be further decomposed to:
p ⁡ ( C im ❘ E , S ) ≈ ∏ K ⁢ ⁢ p ⁡ ( K ❘ E , S ) ; ( 6 ) where K represents an image segment of similar color. Thus, instead of processing (white balancing) the image data pixel-by-pixel, the processing can be performed on a per-segment basis. Image segmentation techniques are known in the art.
If the scene classifier does not use an image segmentation technique, equation (6) may still be applicable if the definition of K is changed from an image segment to a major or non-major color. In other words, in one instance, segmentation is performed by grouping pixels of a similar color where the pixels in a segment are spatially dependent, and in another instance, segmentation is performed by grouping pixels of a similar color where the pixels in the segments are spatially independent. For example, in the former instance, if the image data represents an image of a landscape, with an expanse of blue sky in the background and a similarly colored blue object in the foreground, the pixels associated with the blue sky would be grouped in one segment and the pixels associated with the foreground object would be grouped in another segment. However, in the latter instance, the pixels associated with the blue sky and the pixels associated with the foreground object would be grouped in the same segment.
FIG. 3 is a data flow diagram showing operations performed by an image processor 12 of FIG. 1 according to one embodiment of the present invention. More specifically, in one embodiment, the operations of FIG. 3 are performed by illuminant estimator 24 of FIG. 2.
For each of the M illuminant colors being considered, and for each of the N possible scene classes (that is, for each possible combination of M and N), gamut mapper 23 determines a value of P(E|C,S) that is input to illuminant estimator 24. For each of the N possible scene classes, scene classifier 22 determines a value for Pr(S|Cim) that is input to illuminant estimator 24.
With references to FIGS. 2 and 3, at multipliers 31 and 32, for M=1, 2, . . . , M, illuminant estimator 24 multiplies each value of P(E|C,S) and a respective value of Pr(S|Cim). That is, for M=1, 2, . . . , M, the value of P(EM|C,S) for a particular scene class is multiplied by the value of Pr(S|Cim) for the same scene class. For example, for M=1, 2, . . . , M, the value of P(EM|C,S1) is multiplied by the value of Pr(S1|Cim); the value of P(EM|C,S2) is multiplied by the value of Pr(S2|Cim); and so on, for N=1, 2, . . . , N.
For each illuminant color being evaluated, illuminant estimator 24 then sums the results of the multiplication operations to determine Pr(EM|C), M=1, 2, . . . , M: That is, the results of the multiplication operations include a probability value for each illuminant color for each scene class. Thus, for example, there is a first set of probability values for illuminant E1 for all of the scene classes S1, S2, . . . , SN; a second set of probability values for illuminant E2 for all of the scene classes S1, S2, . . . , SN; and so on, for M=1, 2, . . . , M. At adder 33, the probability values within each set are added. Thus, for example, the individual values within the first set (associated with illuminant E1) are added to determine a probability value Pr(E1|C). In one embodiment, the illuminant color E that is associated with the largest value of Pr(EM|C) is selected as the illuminant color that in turn may be used by image adjustor 25 (FIG. 2) to compensate for the effects of the illuminant on the raw image data 11.
FIG. 4 is a data flow diagram also showing operations performed by an image processor 12 (FIG. 1) according to one embodiment of the present invention. More specifically, in one embodiment, the operations of FIG. 3 are performed by illuminant estimator 24 of FIG. 2.
For each scene class N=1, 2, . . . , N, gamut mapper 23 determines a probability value P(E1|C,S). For example, for scene class S=1, gamut mapper 23 accesses the appropriate scene class-dependent set of LUTs 1, 2 and 3 to fetch p(C|E1,S1), Pr(E1|S1) and p(C|S1) and determines P(E1|C,S1). At multiplier 41, for scene class S=1, P(E1|C,S1) is weighted by multiplying P(E1|C,S1) and Pr(S1|C) from scene classifier 22.
The fetch and multiply operations described in the preceding paragraph are repeated for each scene class N=1, 2, . . . , N for illuminant color E1. At adder 43, the results of the multiply operations are added. Thus, the output of adder 43 is the probability Pr(E1|C) that E1 is the illuminant color given the raw image data C.
The operations described in the above discussion of FIG. 4 are also repeated for each illuminant color EM (M=1, 2, . . . , M). Thus, for each illuminant color EM being evaluated, the probability Pr(E1|C) that EM is the illuminant color given the raw image data C is determined for all scene classes. In one embodiment, the illuminant color that is associated with the maximum value of Pr(E|C) is chosen and input to image adjustor 25 (FIG. 2).
FIG. 5 is a block diagram of an image capture and processing system 50 (e.g., a digital camera or a digital video recorder) upon which embodiments in accordance with the present invention may be implemented. In the example of FIG. 5, system 50 includes an image capture device 51 (e.g., a detector), which may be a charged coupled device (CCD) imager or a complementary metal-oxide semiconductor (CMOS) imager. The image capture device 51 may include lenses and other subcomponents. System 50 also includes an image processor 12 that includes the hardware, software and/or firmware necessary for implementing the image data processing methodology described herein. In the present embodiment, system 50 also includes image memory 53 that stores image data before, during and after processing of that data by image processor 12. In one embodiment, system 50 also includes a display 54 for displaying the image captured and represented by the image data, and for displaying menus and commands as part of a user interface.
FIG. 6 is a flowchart 60 of a method for processing image data according to one embodiment of the present invention. Although specific steps are disclosed in flowchart 60, such steps are exemplary. That is, the present invention is well suited to performing various other steps or variations of the steps recited in flowchart 60. It is appreciated that the steps in flowchart 60 may be performed in an order different than presented and that the steps in flowchart 60 are not necessarily performed in the sequence illustrated. In one embodiment, flowchart 60 is implemented as program instructions stored in a computer-readable memory unit of an imaging system 50 and executed by image processor 12 (FIG. 5). In FIG. 6, C is equivalent to Cim.
In block 601 of FIG. 6, an illuminant hypothesis value E is selected.
In block 602, a scene class S is selected.
In block 603, for the current values of E and S, the probability P(E|C,S) that E is the illuminant color given S and the raw image data C is determined. In one embodiment, the probability P(E|C,S) is determined by gamut mapper 23 of FIGS. 2-4. In one embodiment, the probability P(E|C,S) is determined using information fetched or read from scene-class specific LUTs.
In block 604 of FIG. 6, for the current value of S, the probability Pr(S|C) that S is the scene class given the raw image data C is determined. In one embodiment, the probability Pr(S|C) is determined by scene classifier 22 of FIGS. 2-4.
In block 605 of FIG. 6, the probability P(E|C,S) is weighted by the probability Pr(S|C) to determine Pr(E|C) for the current values of E and S.
At block 606, if there is another scene class to consider, then flowchart 60 returns to block 602. Otherwise, flowchart 60 proceeds to block 607.
At block 607, the values of Pr(E|C) for the current value of E and all values of S are summed.
At block 608, if there is another illuminant color to consider, then flowchart 60 returns to block 601. Otherwise, flowchart 60 proceeds to block 609.
At block 609, the maximum of the summed probabilities for all values of E (the summed probabilities determined in block 607 for each value of E) is identified. The value of E associated with the maximum value is selected. The selected value of E is thus identified as the most likely color of the illuminant that was used to illuminate a scene at the time the image data representing an image of the scene was captured. Accordingly, the selected value of E can be applied to white balance the image data.
In summary, embodiments of the present invention provide a hybrid approach of gamut mapping and scene classification. More specifically, according to the present invention, the results of a scene classification step are used to weight the results of a gamut mapping step.
Furthermore, the correlation matrix used by the gamut mapper of the present invention is scene class-dependent. Consequently, the process of preparing the training image database can be better focused. Also, because construction of the correlation matrix focuses on the most probable colors that can occur within the scene classes considered, the effective camera gamut is contracted and thus the memory requirement for the correlation matrix can be reduced. Embodiments in accordance with the present invention thus provide methods and systems that can be used for white balancing, but that consume less memory and utilize an improved training image database relative to conventional techniques.
Patent CitationsCited PatentFiling datePublication dateApplicantTitleUS3904818Feb 28, 1974Sep 9, 1975Rca CorpRemoval of dark current spikes from image sensor output signalsUS4253120Dec 5, 1979Feb 24, 1981Rca CorporationDefect detection means for charge transfer imagersUS4642684 *May 6, 1985Feb 10, 1987Eastman Kodak CompanyMethod and apparatus for detecting and printing color photographic images of scenes exposed with narrow band illuminantUS4739495Sep 25, 1985Apr 19, 1988Rca CorporationSolid-state imager defect correctorUS4920428Jul 8, 1988Apr 24, 1990Xerox CorporationOffset, gain and bad pixel correction in electronic scanning arraysUS5652621Feb 23, 1996Jul 29, 1997Eastman Kodak CompanyFor processing a digitized image signalUS5667944 *Apr 22, 1996Sep 16, 1997Eastman Kodak CompanyDigital process sensitivity correctionUS5850470 *Aug 30, 1995Dec 15, 1998Siemens Corporate Research, Inc.Neural network for locating and recognizing a deformable objectUS6038339 *Nov 14, 1997Mar 14, 2000Hewlett-Packard CompanyWhite point determination using correlation matrix memoryUS6377702 *Jul 18, 2000Apr 23, 2002Sony CorporationColor cast detection and removal in digital imagesUS6683643Mar 18, 1998Jan 27, 2004Konica Minolta Holdings, Inc.Electronic camera capable of detecting defective pixelUS6724932Jul 27, 2000Apr 20, 2004Fuji Photo Film Co., Ltd.Image processing method, image processor, and storage mediumUS6785814Jul 28, 1999Aug 31, 2004Fuji Photo Film Co., LtdInformation embedding method and apparatusUS6873727 *Jul 23, 2001Mar 29, 2005Hewlett-Packard Development Company, L.P.System for setting image characteristics using embedded camera tag informationUS6900836Feb 19, 2001May 31, 2005Eastman Kodak CompanyCorrecting defects in a digital image caused by a pre-existing defect in a pixel of an image sensorUS7009639May 30, 2000Mar 7, 2006Sony CorporationColor imaging by independently controlling gains of each of R, Gr, Gb, and B signalsUS7092018Oct 25, 2000Aug 15, 2006Sanyo Electric Co., Ltd.Image signal processor and deficient pixel detection methodUS7133072Jun 29, 2001Nov 7, 2006Canon Kabushiki KaishaImage processing apparatus having an image correction circuit and its processing methodUS7305148Jul 27, 2005Dec 4, 2007Stmicroelectronics S.R.L.Color interpolation using data dependent triangulationUS7486844Nov 17, 2005Feb 3, 2009Avisonic Technology CorporationColor interpolation apparatus and color interpolation method utilizing edge indicators adjusted by stochastic adjustment factors to reconstruct missing colors for image pixelsUS7502505Mar 15, 2004Mar 10, 2009Microsoft CorporationHigh-quality gradient-corrected linear interpolation for demosaicing of color imagesUS7671910Mar 15, 2004Mar 2, 2010Samsung Electronics Co., Ltd.Interpolator, method, and digital image signal processor for adaptive filtering of Bayer pattern color signalUS20020015111Jun 29, 2001Feb 7, 2002Yoshihito HaradaImage processing apparatus and its processing methodUS20020044778Sep 5, 2001Apr 18, 2002Nikon CorporationImage data processing apparatus and electronic cameraUS20020105579Jan 10, 2002Aug 8, 2002Levine Peter AlanAddressable imager with real time defect detection and substitutionUS20020167602Mar 20, 2001Nov 14, 2002Truong-Thao NguyenSystem and method for asymmetrically demosaicing raw data images using color discontinuity equalizationUS20030007076 *Jun 28, 2002Jan 9, 2003Minolta Co., Ltd.Image-processing apparatus and image-quality control methodUS20030194125 *Apr 10, 2002Oct 16, 2003Hubel Paul M.White point estimation using color by convolutionUS20030222995Jun 4, 2002Dec 4, 2003Michael KaplinskyMethod and apparatus for real time identification and correction of pixel defects for image sensor arraysUS20040001234Jul 1, 2002Jan 1, 2004Xerox CorporationDigital de-screening of documentsUS20040032516Aug 16, 2002Feb 19, 2004Ramakrishna KakaralaDigital image system and method for combining demosaicing and bad pixel correctionUS20040120572 *Aug 25, 2003Jun 24, 2004Eastman Kodak CompanyMethod for using effective spatio-temporal image recomposition to improve scene classificationUS20050069201 *Sep 30, 2003Mar 31, 2005Sharp Laboratories Of America, Inc.Systems and methods for correcting image color balanceUS20050238225Mar 16, 2005Oct 27, 2005Young-Mi JoDigital signal processing apparatus in image sensorUS20050248671Jun 3, 2004Nov 10, 2005Dialog Semiconductor GmbhSingle line bayer RGB bad pixel correctionUS20050264658 *May 5, 2005Dec 1, 2005Ray Lawrence AFace detecting camera and methodUS20060176375Feb 4, 2005Aug 10, 2006Hau HwangConfidence based weighting for color interpolationUS20070091188Oct 17, 2006Apr 26, 2007Stmicroelectroncs, Inc.Adaptive classification scheme for CFA image interpolationUS20070247532Apr 11, 2007Oct 25, 2007Megachips CorporationImage processing apparatusUS20090010539Jul 2, 2008Jan 8, 2009Stmicroelectronics S.R.L.Method and relative device of color interpolation of an image acquired by a digital color sensorUS20090116750May 30, 2007May 7, 2009Ho-Young LeeColor interpolation method and deviceCN1275870AMay 30, 2000Dec 6, 2000索尼公司Color imaging device and methodWO2004063989A2Jan 15, 2004Jul 29, 2004Blur Technologies Ltd DCamera with image enhancement functions* Cited by examinerNon-Patent CitationsReference1"Method of Color Interpolation in a Single Sensor Color Camera Using Green Channel Separation" Weerasighe, et al. Visual Information Processing Lab, Motorola Australian Research Center pp. IV-3233-IV3236.2Kuno et al, "New Interpolation Method Using Discriminated Color Correlation for Digital Still Cameras" IEEE Transac. on Consumer Electronics, vol. 45, No. 1, Feb. 1999, pp. 259-267.Referenced byCiting PatentFiling datePublication dateApplicantTitleUS8564687May 7, 2007Oct 22, 2013Nvidia CorporationEfficient determination of an illuminant of a sceneUS8698917Jun 4, 2007Apr 15, 2014Nvidia CorporationReducing computational complexity in determining an illuminant of a sceneWO2013189925A2 *Jun 18, 2013Dec 27, 2013St-Ericsson SaDigital image analysis* Cited by examinerClassifications U.S. Classification382/167, 382/162, 382/274International ClassificationG06K9/40, G06K9/00Cooperative ClassificationH04N9/73European ClassificationH04N9/73Legal EventsDateCodeEventDescriptionOct 27, 2005ASAssignmentOwner name: NVIDIA CORPORATION, CALIFORNIAFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:LIN, SHANG-HUNG;REEL/FRAME:017165/0776Effective date: 20051026RotateOriginal ImageGoogle Home - Sitemap - USPTO Bulk Downloads - Privacy Policy - Terms of Service - About Google Patents - Send FeedbackData provided by IFI CLAIMS Patent Services©2012 Google