Abstract:
Preferred embodiments of an image display system achieve mapping of high dynamic range image data to render on a lower dynamic range display device a corresponding image characterized by stable global intensity levels and visually perceptible local area detail. The high dynamic range image data include representations of relatively low intensity contrast, high spatial frequency details and relatively low spatial frequency intensities. Data derived from the high dynamic range image data are applied to a nonlinear intensity transform. The nonlinear intensity transform preserves or enhances the low intensity contrast, high spatial frequency details and maintains a visually perceptible representation of the relatively low spatial frequency intensities to thereby provide visually perceptible local area detail. An exemplary embodiment derives high dynamic range image data from a thermal infrared camera for use with aircraft.

Description:
TECHNICAL FIELD  
       [0001]    This disclosure describes a system and method by which arbitrary images produced by a high dynamic range camera are displayed on a dynamic range-limited display device. The system and method process image data in such a way as to maintain or enhance local contrast while limiting overall image dynamic range. 
       BACKGROUND INFORMATION 
       [0002]    The output of sensors with a high dynamic range can be difficult to render on typical imaging displays. The need to render large differences in intensity and the need to achieve sufficient contrast in areas of relatively uniform intensity compete with each other, given the limited dynamic range of available display devices. An example is a thermal infrared image of a warm airport runway that includes areas of clear sky. The runway will be hot relative to non-runway ground areas, and very hot relative to the sky areas. In this case, reliably representing the relative thermal differences among these three areas along with minute thermal differences within each of these areas would be impossible without use of some sort of non-linear, spatially sensitive intensity transform. 
         [0003]    In addition to dynamic range limitations, typical display devices have characteristic intensity curves that result in differences in perceived intensity with varying input intensity. A particular intensity difference may, for example, be more perceptible if the average intensity is in the middle, as opposed to the extreme low end or high end, of the output range of the display. 
         [0004]    The problem to be solved is the stabilizing of global intensity levels of the displayed image while optimizing local area detail. There exist a number of approaches to solving the problem, many of them under the category of histogram equalization (HE) (or “histogram stretching”) reviewed in Pier, Stephen M.; Amburn, E. Philip; Cromartie, Robert; et al., Adaptive histogram equalization and its variations,  Computer Vision, Graphics, and Image Processing , vol. 39, issue 3, pp. 355-68, September 1987, and in Reza, Ali M., Realization of the Contrast Limited Adaptive Histogram Equalization (CLAHE) for Real-Time Image Enhancement,  Journal of VLSI Signal Processing Systems , vol. 38, no. 1, pp. 35-44, August 2004. In this technique, the brightness values of image pixels are reassigned based on a histogram of their input intensities. In the simplest “flattening” approach, the broad goal is to assign an equal number of pixels to each possible brightness level. In more sophisticated processors, the process is “adaptive” in that the nonlinear transformations are applied on a local area or multi-scale basis. Other operations such as minimum, maximum, and median filters, as well as clipping, may be applied. 
         [0005]    Another approach is gradient domain dynamic range compression (GDDRC), which is described in Fattal, Raanan; Lischinski, Dani; Werman, Michael, Gradient domain high dynamic range compression,  ACM Transactions on Graphics  (TOG), vol. 21 no. 3, July 2002. The GDDRC technique works in the logarithmic domain to shrink large intensity gradients more aggressively than small gradients. This serves to reduce the global contrast ratio, while preserving local detail. 
         [0006]    Histogram manipulations are effective for the particular problem of fine details in dark image regions. However, good quality image details can actually be degraded by naïve implementation of these algorithms. Neither HE nor GDDRC constitutes a seamless, esthetically pleasing and information-preserving solution to widely varying levels and contrasts over arbitrary image scenes. 
       SUMMARY OF THE DISCLOSURE  
       [0007]    The preferred embodiments disclosed achieve mapping of high dynamic range image data to render on a lower dynamic range display device a corresponding image characterized by stable global intensity levels and visually perceptible local area detail. The high dynamic range image data include representations of relatively low intensity contrast, high spatial frequency details and relatively low spatial frequency intensities. Data derived from the high dynamic range image data are applied to a nonlinear intensity transform. The nonlinear intensity transform preserves or enhances the low intensity contrast, high spatial frequency details and maintains a visually perceptible representation of the relatively low spatial frequency intensities to thereby provide visually perceptible local area detail. Saturation of image detail is avoided, as is the formation of artifacts such as “halos” around high spatial frequency image features. The computations are relatively simple and hence may be implemented on an economical processing platform. While the airborne application described above is of interest, the approach is appropriate across a wide range of thermal imaging systems in which mapping from a high dynamic range camera to a much lower dynamic range display is a challenge. The preferred embodiments implement an elegant and practical solution to the global/local dynamic range compression problem, while correcting for artifacts arising from spatial frequency manipulations. 
         [0008]    Additional aspects and advantages will be apparent from the following detailed description of preferred embodiments, which proceeds with reference to the accompanying drawings. 
     
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0009]      FIG. 1  is a block diagram of an embodiment of a system that implements a nonlinear intensity transform to enable display of high dynamic range images on a dynamic range-limited display device. 
           [0010]      FIG. 2  is an arbitrary example of a high dynamic range (HDR) image data waveform applied to the system of  FIG. 1 . 
           [0011]      FIGS. 3 ,  4 ,  5 ,  6 ,  7 ,  8 , and  9  are waveforms produced at the outputs of their associated processing unit modules of the system of  FIG. 1  in response to application of the HDR waveform of  FIG. 2 . 
           [0012]      FIG. 10  is an example of a transfer curve stored in a look-up table (LUT) processing unit module of the system of  FIG. 1 . 
           [0013]      FIGS. 11 and 12  show for two different airport runway scenes comparative relationships of images reduced in dynamic range by least significant bits truncation (left side images) and by nonlinear intensity transform processing in accordance with the disclosure (right side images). 
       
    
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
       [0014]    The preferred embodiments include a number of modular processing units existing as computer algorithms implemented in a general processing unit or as hardware constructs in, for instance, a field programmable gate array (FPGA), as arranged in a system  10  shown in  FIG. 1 . System  10  receives a high dynamic range image data waveform produced by a high dynamic range imaging device, such as a thermal infrared camera (not shown).  FIG. 2  shows an arbitrary high dynamic range (HDR) image data waveform  14  representing, for example, a 10-bit input signal. HDR waveform  14  has three discontinuities  16 ,  18 , and  20 .  FIGS. 3 ,  4 ,  5 ,  6 ,  7 ,  8 , and  9  show the waveforms produced at the outputs of their associated processing units of system  10  in response to application of HDR waveform  14 . Each of these drawing figures depicts along the Y-axis the full extent of the dynamic range of intensity and represents along the X-axis a series of hypothetical pixels along a single line of the image data represented by HDR waveform  14 . Each of three discontinuities  16 ,  18 , and  20  in intensity of HDR waveform  14  in  FIG. 2  is considered to be of high frequency for purposes of this example. 
         [0015]    In a first embodiment, HDR waveform  14  is applied to the inputs of a blurring spatial filter  30 , a summing unit  32 , a statistics unit  34 , and a clamping unit  36 . In an alternative, second embodiment HDR waveform  14  is applied to the inputs of blurring spatial filter  30 , summing unit  32 , and statistics unit  34 ; and the output of blurring spatial filter  30  is applied to the input of clamping unit  36  (as shown in dashed lines in  FIG. 1 ). The following description is directed to the first embodiment. A signal inverting unit  40  receives from spatial filter  30  an output signal and delivers an inverted version of it to a second input of summing unit  32 .  FIG. 3  shows that blurring spatial filter  30  provides a waveform  42  representing a blurred version of HDR waveform  14 , and  FIG. 4  shows that inverting unit  40  provides a waveform  44  representing an inverted version of waveform  42 . 
         [0016]    Blurring spatial filter  30 , signal inverting unit  40 , and summing unit  32  combine to form a high pass filter to process the incoming high bandwidth data represented by HDR waveform  14 . Summing unit  32  adds the raw image data of HDR waveform  14  and the blurred and inverted image data of waveforms  42  and  44  and divides the result by two to maintain the same dynamic range as that of the raw image data. The desired effective kernel size of the high pass filter is fixed and is dependent upon the high dynamic range imaging device.  FIG. 5  shows a waveform  50  that is developed at the output of summing unit  32 . Waveform  50  represents a generally flat line signal trace, except for intensity spikes  16   a ,  18   a , and  20   a  corresponding to, respectively, discontinuities  16 ,  18 , and  20  of HDR waveform  14 . Intensity spikes  16   a ,  18   a , and  20   a  each have rising and falling edges and are positioned generally in the middle of the dynamic range of intensity. 
         [0017]    The output of summing unit  32  is delivered to a dynamic look-up table (LUT)  52 , which applies an intensity transform to the high-pass filtered image data produced by summing unit  32 . This transform is designed to minimize visible artifacts of the high pass filter, most specifically spatial halos around imaged objects of very high or low intensity relative to their surroundings. A typical transform curve is shown in  FIG. 10 . The X-axis represents the absolute difference between the high pass image input to LUT  52  and the implicit average value of those data that will always be one-half of the dynamic range. 
         [0018]    The actual values of this transform depend upon the input image data of HDR waveform  14  characteristics. LUT  52  has a control signal input  53  that determines, from a library, which transform curve to apply. This curve is chosen based on the dynamic range of the raw image input data of HDR  14 . If that dynamic range is low, then a curve or look-up table with a higher output to input ratio (gain) may be selected. The subjective goal is to produce an output image, the dynamic range of which covers at least one-fourth of the dynamic range of an output display device. The maximum output value of LUT  52  is preferably no more than one-half of the dynamic range of the output display device. The gain implicit in LUT  52  is partly determined by the characteristic response of the high dynamic range imaging device and is, therefore, determined experimentally. The transform curve selected from LUT  52  may be changed between successive images. Generally, the most common stimuli are represented by input values that fall below the asymptotic limit, which is approximately  255  for the example of LUT  52 , shown in  FIG. 10 .  FIG. 6  shows a waveform  54  produced at the output of LUT  52 , which is programmed to transform its input waveform  50  as described above. Intensity spikes  16   a ′,  18   a ′, and  20   a ′ of waveform  52  corresponding to the respective intensity spikes  16   a ,  18   a , and  20   a  of waveform  50  cover a larger portion of the dynamic range than that covered by waveform  50  in  FIG. 5 . In this example, the maximum intensity of waveform  52  is no greater than one-fifth of the dynamic range. 
         [0019]    Statistics unit  34  calculates the mean of the high-dynamic range input image data and transmits that mean value to clamping unit  36 . Clamping unit  36  limits the intensity extent of the high-dynamic range image data to a certain amount around the mean value calculated by statistics unit  34 .  FIG. 7  shows a waveform  56  produced at the output of clamping unit  36 . In the second alternative embodiment clamping unit  36  limits the intensity extent of the blurred image data produced by blurring spatial filter  30 . 
         [0020]    A dynamic gain and level unit  60  determines and applies a gain and level intensity transform to the clamped image data produced by clamping unit  36 . This transform determines the minimum and maximum intensity extent of the incoming image data. These limits, along with the mean calculated by statistics unit  34 , are used to calculate a gain that is then applied to the incoming image data. The gain is preferably determined as follows: 
         [0000]                                            If (mean−min) &lt; (max−mean), then             Gain = low−range / [(mean−min)*2]           Else             Gain = low−range / [(max−mean)*2]           End,                        
where ‘Gain’ is the gain applied to the incoming image data intensity values, ‘low-range’ is the number of possible low-dynamic range output intensities, ‘mean’ is the mean input intensity value calculated by statistics unit  34 , ‘min’ is the minimum input intensity observed by dynamic gain and level unit  60 , and ‘max’ is the maximum input intensity observed by dynamic gain and level unit  60 .  FIG. 8  shows a waveform  62  produced at the output of dynamic gain and level unit  60 . Waveform  62  exhibits coverage of a wider portion but at a lower level of the dynamic range than that of waveform  56  of  FIG. 7 . Clamping unit  36  and dynamic gain and level unit  60  together function as a reference image unit  70 .
 
         [0021]    A variable summing unit  64  combines the high frequency data from LUT  52  with the low frequency data from gain and level unit  60 . Variable summing unit  64  has a control signal input  66  that determines the ratio of high-frequency to low-frequency data. This is a subjective measure that may be determined by an observer. The outputs of LUT  52 , dynamic gain and level unit  60 , and variable summing unit  64  produce waveforms representing low dynamic range (LDR) image data.  FIG. 9  shows a waveform  68  produced at the output of variable summing unit  64 . Intensity spikes  16   b ,  18   b , and  20   b  of waveform  68  correspond to the respective discontinuities  16 ,  18 , and  20  of HDR waveform  14 . Waveform  68  demonstrates that this approach ensures that the mean value of the high-dynamic range image is always represented in the low-dynamic range scene as the mid-range intensity of that low range. 
         [0022]    An alternative determination of the gain is as follows: 
         [0000]      Gain=low-range/(max−min). 
       The difference between the alternative method and the preferred method is that the former does not perform the “centering” of the output image intensity. 
       [0023]      FIGS. 11 and 12  show practical examples of an image before and after being processed using the present invention. The left side of each image has been reduced in dynamic range by simply dropping the least significant bits, while the right side has been processed using the method described herein. 
         [0024]    It will be obvious to those having skill in the art that many changes may be made to the details of the above-described embodiments without departing from the underlying principles of the invention. The scope of the present invention should, therefore, be determined only by the following claims.