Objective metric relating to perceptual color differences between images

Techniques described herein may determine an objective metric that relates to the color difference that may be perceived by humans viewing two images of the same visual scene. In one implementation, a method may include receiving first and second images; determining a first histogram based on hue values associated with pixels in the first image; and determining a second histogram based on hue values associated with pixels in the second image. A color difference metric may be determined based on a comparison between the first and second histograms. The color difference metric may relate to an objective measure of color differences between the first and second images.

BACKGROUND

A color model is a model describing the way colors can be represented as sets of numbers. Examples of color models include the RGB color model. The RGB color model is an additive color model that models the addition of red, green, and blue light to reproduce colors. Other examples of color models are the HSL and HSB (also called HSV) color models. The HSL and HSB models relate to cylindrical-coordinate representations of points in the RGB color model. In an HSL/HSB cylinder, the angle around the central vertical axis corresponds to the “hue” (H), the distance from the axis corresponds to the “saturation” (S), and the distance along the axis corresponds to the coordinate with which the model attempts to represent perceived luminance in relation to the saturation (the “lightness” (L) or “brightness” (B)).

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Techniques described herein may determine an objective metric that relates to the color difference that may be perceived by humans viewing two images of the same visual scene. For example, an image may be encoded using two different encodings (e.g., the images may be encoded to have different resolutions). A human viewing the images may perceive color differences between the images. With the techniques described herein, the two images may be analyzed to obtain an objective metric (called a “color difference metric” herein), such as a single scalar value, that relates to the perceived color differences. In one implementation, the color difference metric may be expressed as a percentage value (e.g., between zero and 100 percent), in which larger values indicate that a human is likely to perceive a greater difference in the coloration of the two images.

The color difference metric may be used in a variety of applications. For example, a video may be encoded using two different encodings (e.g., high and low resolution encodings). Corresponding images in the encoded videos may be sampled and compared based on the color difference metric. Values above a threshold may trigger review by a technician. As another example, displays produced by a manufacturer may be tested for uniformity by capturing an image of the display operating to display a test image. Images displayed by different displays may be captured using a camera in a controlled image capture environment (e.g., the same lighting may be used). The color difference metric may be calculated for each captured image, relative to a reference image, to determine whether a particular display produces an unacceptable amount of color variation. The color difference metric described herein may be used in other applications, some of which will be described in more detail below.

FIG. 1is a diagram illustrating an example of an overview of concepts described herein. Assume it may be desirable to determine the value of the color difference metric for the two images labeled as “Image A” and “Image B.” The images may correspond to, for example, corresponding images (e.g., at the same time point) from corresponding video files that were recorded using different encodings. The two images may be input to a color difference component, which may generally operate to convert each image to a histogram of cumulative modified hue intensities, in which each modified hue intensity relates to a hue value (e.g., from the HSB color model) that is scaled by one or more scaling factors that are determined based on corresponding saturation and brightness values. The color difference component may obtain the color difference metric, for the two images, by comparing differences in the histograms. In one implementation, the centers of gravity may be calculated for the histograms. The color difference metric may be calculated based on a comparison of the centers of gravity. The operation of the color difference component will be described in more detail below.

In the example ofFIG. 1, the color difference metric is illustrated as having the value of 1%. This may indicate that “Image A” and “Image B” are perceptually close to one another from a color perspective. For example, a human viewing “Image A” and “Image B” may not perceive notable differences in the coloration between the two images.

FIG. 2is a diagram illustrating an example environment200, in which systems and/or methods described herein may be implemented. As illustrated, environment200may include perceptual color difference component210and alert component220. Color difference component210may generate the color difference metric for input images230-1and230-2. In some implementations, the color difference metric may be used by alert component220to generate alerts to indicate that the perceived color differences between images230-1and230-2are likely to be above a threshold. The alert may be received by a user240, such as an administrator or technician. In some implementations, the alert may be automatically processed by another computing device.

Color difference component210may include one or more computing devices to receive images230-1and230-2and, based on the received images, calculate a value for the color difference metric. In some implementations, color difference component210may be implemented as a network service that provides calculation of the color difference metric for one or more user devices that may communicate with color difference component210over a network. Alternatively, color difference component210may be implemented locally relative to the application or system that uses the results of color difference component210.

Alert component220may include one or more computing devices to receive the calculated color difference metrics. In one implementation, color component220may be configured, such as by a technician, with one or more threshold values. When the received color difference metric is above one of the threshold values, alert component220may output an indication (an “alert”) that the perceived color differences between images230-1and230-2may be greater than the acceptable limit for the particular application. Examples of applications of the use of the color difference metric are described more detail below with reference toFIGS. 7and8. In some implementations, alert component220may be omitted, and the generated color difference metric may be directly outputted to a user or directly used by another application or system.

Images230-1and230-2may be images corresponding to the same scene. For example, as previously discussed, in one implementation, images230-1and230-2may be two images corresponding to the same point in a video that is encoded using two different encoding formats (e.g., using two different encoding resolutions). In another possible implementation, images230-1and230-2may be two images that are to be compared to determine a color similarity between the two images (e.g., as part of an image matching application). In yet another possible implementation, images230-1may be an image taken by a camera, in a controlled lighting environment, of the output of a display that is being tested during manufacture. In this case, image230-2may be a reference image that is used to test the color output of the display.

AlthoughFIG. 2illustrates example components of environment200, in other implementations, environment200may contain fewer components, different components, differently arranged components, or additional components than those depicted. Alternatively, or additionally, one or more components of environment200may perform one or more other tasks described as being performed by one or more other components of environment200.

FIGS. 3A-3Care diagrams conceptually illustrating the HSB color model.FIGS. 3A-3Cgraphically illustrate the relationship between hue (H), saturation (S), and brightness (B) in the HSB color model.

InFIG. 3A, cylinder300represents possible states in the HSB color model. In cylinder300, angle310around the central axis may correspond to the hue value in the HSB color model, the radial distance320from the central axis may correspond to the saturation in the HSB color model, and the distance330along the axis (the height of the cylinder) may correspond to the brightness value in the HSB color model.

FIG. 3Bis a diagram illustrating a circle340that may correspond to a slice of cylinder300at a constant brightness value. As described herein, hue values in the HSB color model may be expressed in the range from zero to 360 degrees. For example, a hue value of zero degrees may correspond to red, a hue value of 180 degrees may correspond to cyan, etc. The saturation values may range from zero to one, where a saturation value of zero may correspond to no color saturation (i.e., white) and a saturation value of one may correspond to full saturation of the corresponding hue. The brightness values may range from zero to one, where a brightness value of zero may correspond to black and a brightness value of one may correspond to 100% brightness.FIG. 3Cis another diagram conceptually illustrating colors in the HSB color model. InFIG. 3C, a cone, such as a cone extracted from cylinder300, is illustrated. As shown, increasing brightness values may correlate with lighter color and increasing color saturation may correlate with movement away from the center axis.

The ranges of the hue, saturation, and brightness values illustrated inFIGS. 3A-3Care examples of possible ranges. For example, saturation and brightness could equivalently be expressed on a scale of zero to 100 instead of zero to one. Similarly, hue may be expressed on a different scale, such as zero to one.

FIG. 4is a flowchart illustrating an example process400relating to determining the color difference metrics for two images. Process400may be performed by, for example, color difference component210.

Process400may include receiving a pair of images (block410). Received images that are not in the format of the HSB color model may be converted to the HSB color model (block410). Although the color difference metric is described herein as being calculated in terms of the HSB color model, in alternative implementations, other color models could be used.

Process400may include detecting edges in each of the images (block420). Edge detection may refer to techniques for identifying points in an image at which the image brightness changes sharply or at which the image has discontinuities. A number of edge detection techniques are known. In one implementation, the Sobel edge detection technique may be used to detect edges in each of the images. In other implementations, other edge detection techniques, such as the Canny or Kayyali edge detection techniques, may be used.

Each pixel, in each edge-detected image, may be further processed in blocks430-470. In particular, process400may further include determining whether a particular pixel is part of an edge (block430). The determination of whether a pixel is part of an edge may be based on the edges detected in block420. For example, the output of block420may include, for each image, an indication of whether each pixel is associated with an edge or is not associated with an edge.

Pixels that are part of an edge may not be further processed. In particular, when a pixel is part of an edge (block430—YES), the next pixel in the image may be processed (block440).

For each pixel that is not part of an edge (block430—NO), process400may further include calculating a hue modification coefficient (block450). The term “hue modification coefficient,” as used herein, may refer to a value by which the hue value for a pixel is to be boosted or attenuated. The hue modification coefficient, for a particular pixel, may be based on the saturation and/or brightness of the pixel. In general, the hue modification coefficient may be based on the goal of increasing the impact, on the final color difference metric, for pixels that are prominent (in the sense of color) and decreasing the impact of pixels that are less prominent (e.g., pixels that appear washed out or black due to lack of brightness, over brightness, under saturated, or oversaturated).

In one implementation, the hue modification coefficient, Hc, may be determined as the product of a “saturation coefficient,” Sc, and a “brightness coefficient,” Bc. Thus Hc may be calculated as: Hc=Bc*Sc. Sc may be calculated as the difference between the peak possible saturation value (1.0 in the HSB color model) and the saturation value, S, of the pixel. Accordingly, Sc may be calculated as: Sc=1.0−S. Bc may be calculated as twice the distance between the brightness value, B, of the pixel, and the midrange of all possible brightness values (0.5 in the HSB color model). Accordingly, Bc may be calculated as: Bc=2*abs(B−0.5), where “abs” represents the absolute value.

Process400may further include modifying the hue, of the particular pixel, based on the calculated hue modification coefficient, to obtain the “hue intensity” for the pixel (block460). Thus, the hue intensity may be calculated as: hue_intensity=H*Hc, where H represents the hue value for the pixel. As previously mentioned, in the HSB color model, the hue value may be represented as an angular portion of a circle (i.e., between zero and 360 degrees).

The above description of blocks450and460described one possible technique for assigning a hue intensity for each (non-edge) pixel in an image. The hue intensity value may be calculated with the goal of quantifying the perceptual color impact, for a viewer, of the particular pixel. In other implementations, other techniques can be used to calculate the hue intensity for the pixels in an image. For example, thresholding based on brightness and saturation may be applied, such as if threshold values for B and/or S are exceeded, the hue intensity may be calculated by decrementing or incrementing H, of the pixel, by a particular amount.

Blocks430-470may be repeated for each pixel in the pair of images. When all pixels have been processed (block470—Yes), process400may include determining, for each image, a histogram of the cumulative hue intensities (block480). In one implementation, the hue values for each pixel may be rounded or otherwise mapped to integer values between one and 360 (e.g., 360 degrees). The histogram, for each image, may be divided into 360 bins, each corresponding to one of the 360 possible integer hue values. The hue intensities, for each image and corresponding to a particular bin of the histogram, may be summed to obtain the cumulative hue intensity for the particular bin of the histogram. The color difference metric, as will be described with respect to blocks490and495, may be calculated based on the determined histograms.

FIG. 5is a diagram illustrating an example histogram500for an image, such as for image230-1or image230-2. As illustrated, histogram500includes a cumulative hue intensity peak at approximately 270 degrees, which may indicate that pixels of this hue (violet) have a relatively high perceptual value in the corresponding image.

Referring back toFIG. 4, process400may further include, for each histogram, calculating the center of gravity of the histogram (block490). In one implementation, the center of gravity for the histogram may be calculated based on the histogram being “wrapped” around a frictionless axis of an HSB color wheel (as illustrated inFIG. 3B). The center of gravity of the histogram may be calculated as the hue value corresponding to the angle at which the histogram aligns with an assigned direction of “gravity” (assuming the wrapped histogram is of constant density).

FIGS. 6A-6Care diagrams graphically illustrating an example of the center of gravity for a histogram. InFIG. 6A, an HSB color wheel610is illustrated. Assume that the top of the wheel corresponds to a hue value of 360 degrees (red). Wheel610may have an arbitrary radius (e.g., radius of one), no mass, and have a frictionless axle620at the center. Further, assume the direction of “gravity” to be down.

InFIG. 6B, the histogram of one of the images is illustrated as wrapped around wheel610to obtain wrapped wheel630. In this example, the histogram wrapped around wheel610may correspond to the histogram shown inFIG. 5.

As shown inFIG. 6C, wrapped wheel630is conceptually illustrated as having rotated around axle620until wrapped wheel630comes to rest (due to the operation of gravity). The angle at which wrapped wheel630comes to rest, illustrated as approximately 280 degrees inFIG. 6C, may be determined as the center of gravity of the corresponding histogram.

In one implementation, the center of gravity for wrapped wheel630may be defined as the angle, with respect to axle620, of the centroid of wrapped wheel630. In other words, wrapped wheel630may be assumed to be a planar object. In this case, the centroid may be estimated as the arithmetic mean of all points in the shape defined by wrapped wheel630. InFIG. 6C, an example of the calculated centroid is given at point640. The angle of point240with respect to axle620may be 280 degrees.

Referring back toFIG. 4, the color difference metric may be calculated based on differences in the centers of gravities between the pair of images (block495). In one implementation, the color difference metric may be scaled or otherwise normalized to generate a final color difference metric for outputting. For example, the difference between the two centers of gravities, if greater than 180, may be shifted to obtain a value between zero and 180 (i.e. 180 may be subtracted from the difference between the two centers of gravities when the difference is greater than 180). This value, which will range between zero and 180, may be scaled to a percentage value between zero and 100% (e.g., by dividing by 180). As an example of the calculation of the color difference metric, assume that the centers of gravity for the two images are determined to be 280 degrees and 270 degrees. The color difference metric may be calculated as: (280−270)/180*100, which equals approximately 5.6%.

In some implementations, the shifted value between zero and 180 may be divided by a number less than 180 (e.g., 135) and the result limited to between zero and 100%. This may be useful to accommodate the situation that while 180 degrees may numerically be the maximum color separation in the HSB color model, no color on the color wheel would be described as similar to another if separated by more than a threshold amount (e.g., 135 degrees). In this example, any color separated by more than the threshold amount (e.g., 135 degrees) from another color may be considered to be perceptually different.

Process400may further include outputting or storing the final color difference metric (block497). The final color difference metric may be, for example, as described previously, a value scaled to be between zero and 100%. In other implementations, the final color difference metric may be scaled or expressed in another manner.

Color difference metrics computed by color difference component210may be used in a number of applications. Two example applications in which color difference component210may be used will next be described with respect toFIGS. 7 and 8.

FIG. 7is a diagram illustrating an implementation of color difference component in the context of encoding or transcoding videos. InFIG. 7, assume that a video is to be encoded or transcoding into two different encoding formats. As illustrated, base video encoding710may represent a base encoding or raw format of the video. Base video encoding710may be encoded by encoders720(“Encoder1”) and730(“Encoder2”). Encoders720and730may transcode base video encoding710into different encoded files based on the use of different encoding techniques and/or different parameters applied to a single encoding technique. The different encoded versions of base video encoding710may be stored by content storage740. Content storage740may represent one or more content storage or distribution servers to distribute video to end-users.

In operation, pairs of images, corresponding to images of the same scene from the videos encoded by encoders720and730, may be input to color difference component210. Color difference component210may, as described above, calculate the color difference metric between the two images. The calculated color difference metric may be input to alert component220, which, as described previously, may generate alerts or other indications, such as to a technician, when the value of the color difference metric is above a threshold.

FIG. 8is a diagram illustrating an implementation of color difference component in the context of the production of display devices by a manufacture. The display devices may include, for example, displays for computer monitors, displays for televisions, displays for mobile phones, or other displays that may be used to display images to users. InFIG. 8, assume that displays810are manufactured in an assembly line and are tested to ensure uniform color presentation. As part of the test, a known image may be shown on each display. A camera820may take a picture of the display showing the known image. The picture may be taken in a controlled environment, such as one in which the focus of the camera the ambient lighting is controlled to be uniform for pictures on different displays810.

The picture taken by camera820(“Image”) may be input to color difference component210along with a reference image. The reference image may include an image, previously taken by camera820, that is known to correspond to a properly working display810. Color difference component210may, as described above, calculate the color difference metric between the image and the reference image. The calculated color difference metric may be input to alert component220, which, as described previously, may generate alerts or other indications, when the value of the color difference metric is above a threshold. In this example, an alert may indicate that the corresponding monitor810may be defective.

It can be appreciated that the color difference metric may be used in applications other than those described with respect toFIGS. 7 and 8. For example, the color difference metric may be used in video rendering, by video rendering software developers, to ensure video is correctly reproduced on the viewing screen. As another example, the color difference metric may be used as part of automated image identification and object recognition, such as in facial recognition and “Identify Friend or Foe” (IFF) applications.

FIG. 9is a diagram of example components of device900. One or more of the devices described above (e.g., with respect to illustrated inFIGS. 1,2,7, and/or8) may include one or more devices900. Device900may include bus910, processor920, memory930, input component940, output component950, and communication interface960. In another implementation, device900may include additional, fewer, different, or differently arranged components.

Bus910may include one or more communication paths that permit communication among the components of device900. Processor920may include a processor, microprocessor, or processing logic that may interpret and execute instructions. Memory930may include any type of dynamic storage device that may store information and instructions for execution by processor920, and/or any type of non-volatile storage device that may store information for use by processor920.

Input component940may include a mechanism that permits an operator to input information to device900, such as a keyboard, a keypad, a button, a switch, etc. Output component950may include a mechanism that outputs information to the operator, such as a display, a speaker, one or more light emitting diodes (“LEDs”), etc.

Communication interface960may include any transceiver-like mechanism that enables device900to communicate with other devices and/or systems. For example, communication interface960may include an Ethernet interface, an optical interface, a coaxial interface, or the like. Communication interface960may include a wireless communication device, such as an infrared (“IR”) receiver, a Bluetooth radio, or the like. The wireless communication device may be coupled to an external device, such as a remote control, a wireless keyboard, a mobile telephone, etc. In some embodiments, device900may include more than one communication interface960. For instance, device900may include an optical interface and an Ethernet interface.

The foregoing description of implementations provides illustration and description, but is not intended to be exhaustive or to limit the possible implementations to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations. For example, while a series of blocks has been described with regard toFIG. 4, the order of the blocks may be modified in other implementations. Further, non-dependent blocks may be performed in parallel.

Further, while certain connections or devices are shown, in practice, additional, fewer, or different, connections or devices may be used. Furthermore, while various devices and networks are shown separately, in practice, the functionality of multiple devices may be performed by a single device, or the functionality of one device may be performed by multiple devices.