Apparatus and methods for boosting dynamic range in digital images

A method for increasing dynamic range of original image data representing an image comprises applying an expansion function to generate from the original image data expanded data having a dynamic range greater than that of the original image data and, obtaining an expand map comprising data indicative of a degree of luminance of regions associated with pixels in the image. The method then combines the original image data and the expanded data according to the expand map to yield enhanced image data. Apparatus for boosting the dynamic range of image data comprises a dynamic range expander that produces expanded data, a luminance analyzer that produces an expand map and a combiner that combines the original and expanded data according to a variable weighting provided by the expand map.

TECHNICAL FIELD

This invention relates to digital imaging. The invention relates specifically to methods and apparatus for increasing the dynamic range of digital images.

BACKGROUND

The human eye is sensitive light over a very wide range of intensities. Images must have high dynamic ranges to accurately reproduce real scenes. High-performance image sensors, such as high-performance CCD arrays are capable of acquiring images having high dynamic ranges. There are displays, such as the displays available from Dolby Canada Corporation are capable of displaying high dynamic range images. However, many computer displays, televisions and the like have limited dynamic ranges and are incapable of displaying such high dynamic range images.

Some image data has low dynamic range because of the way that the image data is acquired or generated. In other cases, the dynamic range of image data may be reduced to match the characteristics of displays on which image of the image data will be reproduced. A tone mapping operator can be applied to higher dynamic range image data to reduce the dynamic range of the image data. This may be done, for example, to provide image data that matches the dynamic range of a type of display or a particular image data format.

There is a vast amount of existing image data that has a dynamic range that is lower than the dynamic range that can be displayed by available high dynamic range displays and/or appreciated by the human eye.

There is a need for methods and apparatus that can boost the dynamic range of lower dynamic range image data.

SUMMARY

One aspect of the invention provides a method for increasing the dynamic range of original image data representing an image. The method comprises, in any order: applying an expansion function to generate from the original image data expanded data having a dynamic range greater than that of the original image data; and, obtaining an expand map comprising data indicative of a degree of luminance of regions associated with pixels in the image. The method combines the original image data and the expanded data according to the expand map to yield enhanced image data.

Another aspect of the invention provides apparatus for expanding the dynamic range of original image data. The apparatus comprises: a dynamic range expander connected to receive original image data and to output expanded data having a dynamic range greater than that of the original image data; a luminance distribution analyzer configured to generate an expand map indicative of the luminance of regions associated with pixels in the image of the original image data; and an image combiner configured to combine the original image data with the expanded data from the dynamic range expander according to the expand map to yield enhanced image data.

Another aspect of the invention provides a method for enhancing the dynamic range of original image data, the method comprising processing luminance values for pixels of the original image data according to the inverse tone mapping operator given by:

α2Lwhite2⁢L_w2⁢Lw2⁡(x,y)+αLw⁢(1-Ld⁡(x,y))⁢Lw⁡(x,y)-Ld⁡(x,y)=0
or a mathematical equivalent thereof, where α, Lwhite, andLware parameters and Ldis a luminance value corresponding to a pixel in the original image data.

In addition to the exemplary aspects and embodiments described above, further aspects and embodiments are described below and/or will become apparent by reference to the drawings and by study of the following detailed description.

DESCRIPTION

This invention provides methods and apparatus for boosting the dynamic range of image data. The methods and apparatus may be applied, for example, to increase the dynamic range of legacy images (which may be still or video images, for example) for display on high dynamic range displays.

FIG. 1shows a method20according to an embodiment of the invention. Method20acts on original image data23obtained in block24to expand the dynamic range of the image data23. In block26, method20expands original image data23to obtain expanded image data25. Expanding maps pixel luminosity values from a first range to a second range. The second range provides more possible values than the first range. For example, the first range may permit values in the range of 0 to 255 while the second, expanded, range may permit values in the range of 0 to 1023 or values in the range of 0.00 to 1.00 to some specified precision. In some embodiments, the luminosity value of a pixel in expanded image data25is a function of the luminosity value of the corresponding pixel in original image data23.

Expanding may involve any of: linear scaling, non-linear scaling or applying a more complicated expansion function such as the inverse of a tone mapping function. Where block26applies an inverse tone-mapping function, the inverse tone-mapping function applied in block26is not necessarily the inverse of any particular tone mapping function used in the creation of original image data23. Indeed, in some cases, original image data23may have been obtained without the application of a tone mapping function or through the application of a tone mapping function different from the inverse of the tone mapping function applied in block26.

One example tone mapping function is thePhotographic Tone Reproductiondescribed in Reinhard et al.,Photographic tone reproduction for digital images, ACM Trans. Graph., 21, 3, 267-276 (2002). Other tone mapping functions are described, for example, in Smith et al,Beyond tone mapping: Enhanced depiction of tone mapped HDR images., Computer Graphics Forum 25, 3 (2006); and Ledda et al.Evaluation of tone mapping operators using a high dynamic range display, ACM Trans. Graph., 24, 3, 640-648 (2005).

The photographic tone reduction tone mapping operator scales pixel luminosity values based on a geometric average, which is or approximates the key of the scene and then compresses the resulting values. The scaling may be given by:

Lm⁡(x,y)=αL_w⁢Lw⁡(x,y)(1)
where Lmis the scaled value, α is a user parameter, Lwis the luminosity of the pixel in original image data23. If original image data23is in an RGB format then Lw, may be given by:
Lw=0.213Rw+0.715Gw+0.072Bw(2)
where Rw, Gwand Bware respectively the red, green and blue pixel values for the pixel in the RGB color space.Lwis the geometric average defined by:

L_w=exp⁡(1N⁢∑x,y⁢⁢log⁡(δ+Lw⁡(x,y)))(3)
where δ is a small non-negative value and N is a number of pixels in the image.

The compression maybe provided by a function which takes input values within a first range and produces output values within a second, narrower range. For example, compression could be provided by:

Ld⁡(x,y)=Lm⁡(x,y)1+Lm⁡(x,y)(4)
where Ldis the compressed value for the pixel at (x, y). In a more flexible embodiment, the compression is provided by:

Equation (5) can be inverted by solving the quadratic equation:

Lm2⁡(x,y)Lwhite2+Lm⁡(x,y)⁢(1-Ld⁡(x,y))-Ld⁡(x,y)=0(6)
In Equation (5), Lmcan be replaced by the value from Equation (1) to yield:

α2Lwhite2⁢L_w2⁢Lw2⁡(x,y)+αLw⁢(1-Ld⁡(x,y))⁢Lw⁡(x,y)-Ld⁡(x,y)=0(7)
Equation (7) can be solved for Lwusing the quadratic formula to obtain the largest positive-valued solution. This can be performed for each pixel in an image to obtain expanded image data25.

To apply the solution of Equation (7) for inverse tone-mapping one must assign values to the parameters α, Lwhite, Ldand the geometric averageLw. Unless one knows what tone-mapping operator (if any) was applied to obtain the lower dynamic range image being worked on, these parameters will not be known. In methods according to some embodiments, a user may set values for these parameters. In some embodiments, the parameters may be automatically set or predetermined. In some embodiments, some or all of the parameters are set automatically to initial values and a user can vary the parameter values from those initial values, if desired.

One way to assignLwis to use the geometric average luminance of the lower dynamic range image being processed. It has been found that the geometric average luminance of a higher- and lower-dynamic range images of the same scene are typically quite similar (unless the lower-dynamic-range image is significantly over- or under-exposed). Certain tone-mapping operators tend to change the geometric average luminance. Where such tone mapping operators have been used in the generation of original image data23it may be desirable to use a function of the geometric average luminance of the lower dynamic range image being processed forLw. The function may be chosen based upon knowledge of the tone-mapping operator used to generate original image data23or may be determined empirically.

One way to assign a value to the parameter Ld(x,y) is to realize that Ldis the luminance of the lower-dynamic range image being processed.

The parameters α and Lwhitemay be set by the user. The meaning of the parameter α is somewhat enigmatic. Therefore, the inventors prefer to define a parameter Lmax′ such that:

Lmax′=Lwhite⁢L_wα(8)
Lmax′ is the maximum luminance value expected in the inverse tone-mapped image. Lwhiteaffects the expansion of the original low- and medium-luminance values. If Lwhiteis very high, those values are mapped to very low luminance values. If Lwhiteis very low, the inverse-tone-mapped image will have luminance values similar to those in the original lower-dynamic range image scaled by the factor Lmax′. In typical applications, setting Lwhiteand Lmax′ to have values that are equal or of the same order tends to produce reasonable results.

The expansion function applied in block26may produce an expanded image that is not completely acceptable. If the expansion function produces output luminance values that are high then the resulting image may be “blocky” in appearance.

Method20obtains an output image33by combining expanded image25with original image23according to an expand map29generated in block28. Expand map29identifies higher-luminance and lower-luminance areas of original image23. Using expand map29, method20bases output image33more heavily on expanded image25in higher-luminance areas and bases output image33more heavily on original image23in lower-luminance areas. Block28may use any suitable method for evaluating the luminance level of an area to which a pixel belongs. For example, block28may compute an average luminance or weighted average luminance of pixels in an area to which each pixel belongs. Expand map29includes weights associated with each pixel. The weights indicate the relative degree to which original image23and expanded image25contribute to the value for that pixel in enhanced image33.

In the illustrated embodiment, block28applies a median cut algorithm in block28A. The median cut algorithm is described, for example, in Debevec, P.,A median cut algorithm for light probe sampling, ACM Siggraph 2005 posters (2005). The median cut algorithm identifies a set of point light sources that are clustered near areas of high luminance in an image. The number and intensity of such light sources in the vicinity of a pixel is used to create expand map29in some embodiments.

The median cut algorithm divides an image into 2nregions of similar light energy. These areas may be identified by subdividing the image along the longest dimension such that the luminance is equally apportioned between the resulting regions. The process is repeated for the resulting regions. A light source is placed at the centroid of each of the 2nregions obtained by iterating the process of dividing the image into regions n times. The colour of each light source is set to an average value across the region (for example, the colour may be set to equal a sum of pixel values within the region.

In some embodiments of the invention n is at least 9 (corresponding to 512 light sources). In some embodiments n is 10 or more.

In some implementations, the point light sources may be stored in a data structure comprising a 2D tree to facilitate nearest-neighbour searches that may be performed in creation of expand map29.

It is not mandatory that the original image23be used for identification of higher- and lower-luminance areas. The distribution of higher- and lower-luminance areas will be similar in original image data23and expanded image data25. The median cut algorithm may be performed on expanded image data25.

One way to obtain a set of weights from the light-sources identified by the median cut algorithm performed in block28A is to determine for each pixel (x,y) the density of light sources within an area surrounding the pixel. The area may conveniently comprise a circular area having some radius r for example. Other area shapes could also be used. Density estimation is described in Duda et al.Pattern Classification2ndEdition, Wiley Interscience (2001).

A basic formula that may be used for density estimation is:

Λ⁡(X,r)=∑p∈P⁢⁢Ψpπ⁢⁢rmax2(9)
where: Λ is the density; X is the location (x.y) in the image; Ψpis the luminance value for a light source at point p; and, P is the set of points within the area (a circle having radius r and centered at X in this example) that correspond to light sources identified by the median cut algorithm. r

The density estimation can be improved in a number of ways including:iterating the median cut algorithm to obtain a greater number of light sources (i.e. making n larger);applying a smoothing filter to the results of the density estimation;requiring that at least a threshold number of light sources (for example, where there are 1024 or more light sources—n=10 the threshold number could be 4 or more—in some cases 4-6 light sources) be within a region of influence of a pixel (e.g. within a radius r of the pixel) before assigning a non-zero density Λ to the pixel.

A smoothing filter may comprise a Gaussian smoothing filter. For example, a prototype embodiment applied a Gaussian filter defined by the weight of the kernel given by:

wgp=γ[1-1-ⅇ-β⁢rp22⁢rmax21-ⅇ-β](10)
where: wgpis the kernel; γ and β are parameters. This filter is described, for example in Pavicic,Convenient Anti-Aliasing Filters that Minimize Bumpy Sampling, Morgan Kaufmann, (1990). Example values for γ and β are γ=0.918 and β=1.953. This filter is normalized and can be applied by scaling the luminances by wgpwhen computing the density estimation as described above. It can be seen that this filter weights light sources that are closer to the pixel more deavily than light sources that are farther from the pixel.

In block30, original data23and expanded data25are combined using expand map29to yield enhanced data33. In an illustrative example, expand map29provides a value in the range [0,1] for each pixel. This value can be used as weights for a linear interpolation between original data23and expanded data25. For example:
Lfinal(x,y)=Λ(x,y)Lw(x,y)+(1−Λ)(x,y))Ld(x,y)  (11)
where: Lfinalis a pixel luminance value in enhanced data33for a pixel at location (x,y); Lwis a luminance value for the pixel in expanded data25; Ldis a luminance value for the pixel in original data23; and Λ is a weight for the pixel in the range [0,1] from expand map29.

Block30could optionally combine original data23and expanded data25in other ways. For example, the interpolation could be non-linear.

The methods described herein may be applied to enhance the dynamic range of digital still images or video images, for example. Enhanced data33may be saved, as indicated in block34or displayed on a display, as indicated in block36.

The inverse tone mapping operator described above with reference to Equations (6) and (7) has application outside of method20. For example, the inverse tone mapping operator could be applied directly to increase the dynamic range of frames in a video. In such embodiments, the image of each frame of the video may be processed by the inverse tone mapping operator to obtain an expanded frame. The expanded frames may be stored and/or played back to provide video having a dynamic range higher than that of the original video.

Typical images contain hundreds of thousands of pixels and, more typically, millions of pixels. The methods described herein are performed using automated apparatus, such as specialized hardware and/or programmed computer systems.

FIG. 2shows schematically apparatus40for producing images having expanded dynamic ranges from original image data23.

Apparatus40comprises a dynamic range expander44that processes original image data23to yield expanded data25. In some embodiments, dynamic range expander44comprises a software module that takes original image data23and applies a dynamic range expansion function to each luminance value in original image data23to yield expanded data25.

Apparatus40comprises a luminance distribution analyzer46that processes original image data23(or optionally expanded data25) to yield expand map29. Luminance distribution analyzer46determines the degree to which pixels in the original image data23belong to high-luminance and low-luminance areas of the image represented by original image data23.

Combiner48combines original image data23and expanded data25to yield enhanced data33. The relative degree to which each pixel of enhanced data33is based upon the value for the corresponding pixel of original image data23and expanded data25depends upon the value of the corresponding pixel in expand map29.

Each of dynamic range expander44, luminance distribution analyzer46and combiner48may comprise a hardware module, a combination of hardware and software, or configurable hardware, such as one or more suitably configured field-programmable gate arrays (FPGAs). In some embodiments, apparatus40is provided in a high dynamic range electronic display system capable of displaying still and/or video images. In such embodiments, apparatus40may be activated to enhance legacy images and/or video images having dynamic ranges lower than a dynamic range that the display system is capable of reproducing.

FIG. 3illustrates apparatus50according to another embodiment of the invention. Apparatus50has a user interface52which permits a user to control values of parameters31in a data store54. Parameters31control the operation of a dynamic range boosting system56that processes original image data23to obtain enhanced image data33as described herein. Enhanced image data33is displayed on a display60controlled by a high dynamic range display driver58. In the illustrated embodiment, a user can view the effect of a particular set of parameters31on the image displayed on display60and then alter the values of one or more parameters31by way of user interface52to achieve a desired appearance of the image. The user can then save the enhanced image data33for later display on display60or on other high dynamic range displays.

Certain implementations of the invention comprise computer processors which execute software instructions which cause the processors to perform a method of the invention. For example, one or more processors in an image-processing or image display system may implement the methods ofFIG. 1by executing software instructions in a program memory accessible to the processors. The invention may also be provided in the form of a program product. The program product may comprise any medium which carries a set of computer-readable signals comprising instructions which, when executed by a data processor, cause the data processor to execute a method of the invention. Program products according to the invention may be in any of a wide variety of forms. The program product may comprise, for example, physical media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, or the like. The computer-readable signals on the program product may optionally be compressed or encrypted.

While a number of exemplary aspects and embodiments have been discussed above, those of skill in the art will recognize certain modifications, permutations, additions and sub-combinations thereof. For example:Application of the invention is not limited to any particular formats for representing image data or to any particular colour spaces. Although luminance values are processed, it is not necessary that the original image data23or the enhanced image data33be in a LUV or other format in which luminance values are explicitly represented. The invention can be practiced with other image formats that contain information from which luminance values can be derived. For example, where image data is represented in a RGB format, luminance values can be derived through the application of Equation (2) or other suitable relationship which produces a value related to luminance from values for individual colours in the image.

It is therefore intended that the following appended claims and claims hereafter introduced are interpreted to include all such modifications, permutations, additions and sub-combinations as are within their true spirit and scope.