Abstract:
A method and apparatus involve electronically transforming a monochromatic infrared source image of a source scene into a natural daylight color image of the source scene as a function of reference information, the only information derived from the source scene and utilized in the transforming being the monochromatic infrared source image. According to a different aspect, an apparatus includes an infrared image detector that provides a monochromatic infrared source image of a source scene, and includes a transformation section that transforms the monochromatic infrared source image into a natural daylight color image of the source scene as a function of reference information, the only information derived from the source scene and utilized in the transforming being the monochromatic infrared source image.

Description:
FIELD OF THE INVENTION 
     This invention relates in general to colorization of images and, more particularly, to colorization of infrared images. 
     BACKGROUND 
     In a situation involving night visibility, or seeing through fog and smoke, it may be difficult or impossible to generate a useful image in the visible light range. Moreover, even if a visible light image is generated under such circumstances, it is often a low-contrast image, in which details are hard to identify. 
     In contrast, infrared (IR) imagery is very useful for enhanced night visibility, and/or for seeing through fog and smoke. However, the interpretation of IR images is inherently unintuitive, because the IR luminances of real-world objects are significantly different from what the human eye is normally used to seeing in the visible light range. It would be easier to interpret IR images if the IR color scheme was mapped into natural daylight coloring. However, in the case of a single-band IR image, it has previously been believed that such an image does not contain sufficient information to permit colorization thereof in a manner yielding a natural daylight color image in which the colorization is reasonably accurate. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       A better understanding of the present invention will be realized from the detailed description that follows, taken in conjunction with the accompanying drawings, in which: 
         FIG. 1  is a block diagram of an apparatus that can colorize a monochrome infrared image of a source scene using reference information derived from a reference scene. 
         FIG. 2  is a diagrammatic view of an example of a source scene. 
         FIG. 3  is a diagrammatic view of an example of a reference scene corresponding to the source scene of  FIG. 2 . 
         FIG. 4  is a diagram showing how a transformation section in the apparatus of  FIG. 1  can generate the reference information from an infrared reference image and a color reference image that represents the reference scene of  FIG. 3 . 
         FIGS. 5 ,  6  and  7  are each a diagrammatic view showing, in an enlarged scale, a respective portion of the infrared reference image of  FIG. 4 . 
         FIG. 8  is a diagram showing how the transformation section in the apparatus of  FIG. 1  utilizes the reference information to colorize a monochrome infrared source image representing the source scene of  FIG. 2 . 
         FIG. 9  is a high-level flowchart showing a colorization process that is described in association with  FIGS. 4 and 8 . 
     
    
    
     DETAILED DESCRIPTION 
       FIG. 1  is a block diagram of an apparatus  10  that can colorize a monochrome infrared (IR) image of a source scene  12  using reference information derived from a reference scene  13 . The source scene  12  and the reference scene  13  are similar but different. In this regard,  FIG. 2  is a diagrammatic view of an example of a source scene  12 , and  FIG. 3  is a diagrammatic view of an example of a corresponding reference scene  13 . The scenes  12  and  13  in  FIGS. 2 and 3  are different (in that they are not identical), but they are also similar (in that they each include a yard with grass, a building, and at least one tree). 
     In  FIG. 1 , the apparatus  10  includes a monochrome IR camera or image detector  16  that can produce a monochromatic IR source image of the source scene  12 . The camera  16  is a device of a type known in the art, and is therefore not illustrated and described here in further detail. In the disclosed embodiment, the camera  16  is a single-band IR camera that is responsive to IR radiation in only one of the near IR band, the intermediate IR band, and the far IR band. 
     The apparatus  10  includes a transformation section  21  that receives and processes images from the camera  16 . The transformation section  21  includes computer hardware in the form of a conventional and commercially-available computer system of the type commonly known as a personal computer. For example, the hardware could be a personal computer obtained commercially from Dell Inc. of Round Rock, Tex. Alternatively, the hardware of the transformation section  21  could be a microcontroller of a known type, or any other suitable hardware configuration. 
     The transformation section  21  includes a processor  22  of a known type. For example, the processor  22  could be a device that is commercially available under the tradename CORE™ 2 Duo from Intel Corporation of Santa Clara, Calif. Alternatively, the processor  22  could be implemented using an appropriately-configured, field-programmable gate array (FPGA). For example, the FPGA could be an IGLOO® Nano available commercially from Actel Corporation of Mountain View, Calif., or alternatively a Cyclone® III LS available commercially from Altera Corporation of San Jose, Calif. In the disclosed embodiment, the transformation section  21  also includes a memory  26  that is shown diagrammatically in  FIG. 1 , in that it collectively represents several different types of memory that happen to be present. For example, the memory  26  may include any or all of a hard disk drive (HDD), a volatile random access memory (RAM), a “flash” RAM, a read-only memory (ROM), or any other type of memory component suitable for use within the transformation section  21 . The memory  26  stores a computer program  28  that is executed by the processor  22 , and that causes the transformation section  21  to operate in a manner described in more detail later. The memory  26  also stores data, including reference information  31  that is used by the program  28  to colorize monochrome IR images from the camera  16 , in a manner described in more detail later. 
     The apparatus  10  includes a display  36  of a well-known type. The display  36  can receive an image from the transformation section  21 , and then display that image. For example, the transformation section  21  can (1) receive from the camera  16  a monochrome IR source image of the source scene  12 , (2) colorize that image to produce a natural daylight color image of the source scene, and then (3) supply the colorized image to the display  36 , where it is displayed. 
     The apparatus  10  also includes a further monochrome IR camera or image detector  41 , and an RGB (red, green, blue) color camera or image detector  43  that is responsive to visible light. The cameras  41  and  43  are optional, and are shown in  FIG. 1  for purposes of completeness. In the disclosed embodiment, the monochrome IR camera  41  is identical to the camera  16 . In fact, the camera  16  could optionally perform dual duty, and provide an image of the source scene  12  and also an image of the reference scene  13 . However, for clarity,  FIG. 1  shows two separate cameras at  16  and  41 . Although the camera  41  in  FIG. 1  is identical to the camera  16 , it would alternatively be possible to use any suitable camera at  41 . The RGB camera  43  is also a camera of a known type, and produces a natural daylight color image of the reference scene  13 . 
     The cameras  41  and  43  are configured to produce images that are registered with respect to each other. For purposes of this disclosure, the term “registered” means (1) that an image from the camera  41  and a corresponding image from the camera  43  each have the same number of pixels both vertically and horizontally, and (2) that each pixel in one image represents exactly the same portion of the reference scene  13  as the corresponding pixel in the other image. Images from the cameras  41  and  43  are used to generate the reference information  31 , in a manner described later. 
     The cameras  41  and  43  are shown in broken lines in  FIG. 1  because, as mentioned above, they may or may not be part of the apparatus  10 . If the cameras  41  and  43  are present in the apparatus  10 , then the transformation section  21  can use images from these cameras to generate the reference information  31 , in a manner described later. On the other hand, the cameras  41  and  43  could alternatively be coupled to some other not-illustrated computer system that generates the reference information  31 , and then the reference information  31  could be transferred from that other computer system to the memory  26  in the transformation section  21  in any suitable manner, for example through a computer network, or using a portable memory device such as a Universal Serial Bus (USB) drive. 
     A portion of the apparatus  10  is designated by a broken line  51 . This portion of the apparatus  10  could be incorporated into a device such as a weapon sight, or a night vision aid. This device would take a monochrome IR image of a source scene (for example when visible light is low, or when smoke or fog is present), and then colorize and display that image so that a user sees a natural daylight color image. Previously, it was believed that a single-band IR image of the type produced by camera  16  does not contain sufficient information to permit colorization thereof in a manner yielding a natural daylight color image with reasonably accurate colors. However, as discussed in more detail below, the apparatus  10  is capable of doing this, and in particular is capable colorizing a single-band IR image of the type produced by camera  16  in a manner yielding a natural daylight color image with relatively accurate colors. 
       FIG. 4  is a diagram showing how images from the cameras  41  and  43  ( FIG. 1 ) are used by the transformation section  21  to generate the reference information  31 . In more detail,  FIG. 4  shows at  101  a monochrome IR reference image produced by the camera  41 , and shows at  102  a color reference image produced by the camera  43 . The images  101  and  102  each represent the same scene (shown in  FIG. 3 ), except that image  101  is a single-band IR image, and image  102  is a visible-light color image. In  FIG. 4 , each of the images  101  and  102  is depicted as an array of 24×24 pixels, but the 24×24 array of pixels is used here for simplicity and clarity. The actual number of pixels could be higher or lower, and would typically be much higher. As received from the camera  41 , each pixel in the image  101  has a respective luminance value associated with it. 
     The two images  101  and  102  are registered with respect to each other. As discussed earlier, this means (1) that the images  101  and  102  have the same number of pixels horizontally and the same number of pixels vertically, and (2) that each pixel in the image  101  represents the same portion of the reference scene  13  as the corresponding pixel in the image  102 . As one example, image  101  has a pixel  106  that represents an end portion of a building roof, and image  102  has a corresponding pixel  107  that represents the same portion of the building roof. 
     As indicated diagrammatically by a block  111  in  FIG. 4 , the computer program  28  ( FIG. 1 ) determines, for each pixel in image  101 , a standard deviation value. In this regard,  FIG. 5  is a diagrammatic view showing a portion  114  of the image  101 . For an arbitrary pixel  116 , the block  111  determines a standard deviation value for that pixel using a 7×7 array of pixels, where the pixel  116  is at the center of the array. Although the discussion here is based on use of a 7×7 array of pixels, it would alternatively be possible to determine standard deviation using some other configuration of pixels in the neighborhood of the pixel of interest. 
     In  FIG. 5 , stippling is used to identify the 49 pixels in the 7×7 array for pixel  116 . Each of these 49 pixels has a respective luminance value. The computer program  28  takes the 49 luminance values for these 49 pixels, and determines a standard deviation for these values (based on the standard mathematical definition for standard deviation). 
     It will be recognized that, for some pixels located near an edge of the image  101 , the standard deviation will be determined by less than a 7×7 array of pixels. For example,  FIG. 6  is a diagrammatic view of a portion  121  of the image  101 , where a pixel  123  is located along one edge of the image. The standard deviation value for the pixel  123  is determined using a 4×7 array of pixels that is designated by stippling in  FIG. 6 . Similarly,  FIG. 7  is a diagrammatic view of a portion  126  of the image  101 , where a pixel  128  is located in a corner of the image. The standard deviation value for the pixel  128  will be determined using a 4×4 array of pixels that is designated by stippling in  FIG. 7 . In this manner, a respective standard deviation value is determined for each of the pixels in the IR reference image  101  ( FIG. 4 ). 
     Referring again to  FIG. 4 , the computer program  28  generates a table  141  having N rows, where N is the total number of pixels in the image  101 , and where each row in the table corresponds to a respective pixel. For each pixel in the image  101 , the corresponding row of the table is populated with (1) the original luminance value for that pixel, and (2) the standard deviation value (STD DEV) calculated for that pixel in block  111 . 
     The computer program  28  generates a second table  146  that also has N rows, each corresponding to a respective pixel in the image  102 . Each row in table  146  contains the red value, the green value and the blue value for the pixel from RGB image  102  that is associated with the row. 
     After generating the two tables  141  and  146 , the computer program  28  scans the table  141 , looking for two rows that happen to have identical luminance and standard deviation values. For example, it will be noted in  FIG. 4  that rows  1  and  5  (representing pixels  1  and  5 ) each have a luminance value of  63  and a standard deviation value of  77 . In the other table  146 , the computer program  28  (1) averages the two red values in rows  1  and  5 , (2) averages the green values in these two rows, (3) averages the blue values in these rows, (4) inserts these three average values into one of the two rows, and then (5) deletes the other of these rows. For example, the computer program  28  could take the red values of  67  and  69  for rows  1  and  5 , average them to obtain an average value  68 , and then substitute this average value  68  for the original value  67  in row  1 . In a similar manner, the program could calculate an average green value of  63  and substitute this into row  1 , and calculate an average blue value of  65  and substitute this into row  1 . The program  28  could then delete row  5  from each of the tables  141  and  146 . 
     Alternatively, it would be possible to use other methods of averaging the color values in rows  1  and  5  of table  146 , with the aim of improving the naturalness of the result. For example, the red, green and blue values could be transformed into a uniform perceptual color space, and then averaged there. If color averaging is done in an alternate color space, then the resulting color information must be transformed from that space back into its corresponding red, green and blue values for use in the table  146 . 
       FIG. 8  is a diagram showing how the computer program  28  in the transformation section  21  of  FIG. 1  utilizes the reference information  31  to colorize a monochrome IR image from the camera  16 . In  FIG. 8 , reference numeral  201  designates a monochrome IR source image that has been produced by the camera  16 , and that represents the source scene  12  of  FIG. 2 . In  FIG. 8 , the image  201  is shown as an array of 24×24 pixels, but this is merely for simplicity and clarity. The actual number of pixels could be higher or lower, and would typically be much higher. As discussed earlier, the reference images  101  and  102  in  FIG. 4  are registered, and must have the same number of pixels. However, the source image  201  does not need to have the same number of pixels as the reference images  101  and  102 , and could have a larger or smaller number of pixels. 
     As indicated diagrammatically by a block  211  in  FIG. 8 , the computer program  28  calculates a respective standard deviation value for each pixel in the source image  201 . This calculation is carried out in the same manner described above for block  111 . 
     One specific example of the reference information  31  from  FIG. 1  is shown in more detail at the center of  FIG. 8 . It will be noted that the tables  141  and  146  from the bottom of  FIG. 4  have been merged into a single table that serves as the reference information  31 . It will also be noted that, as described above, row  5  is no longer present, and the red, green and blue values in row  1  are each an average of the original values from rows  1  and  5 . Due in part to the fact that the values for some pixels have been combined, the left column of the table is labeled “index” rather than “pixel”. 
     The computer program  28  uses the reference information  31  to colorize the monochrome IR source image  201 , in order to thereby generate a corresponding color source image  218 . The color source image  218  has the same number of pixels as the IR source image  221 , both vertically and horizontally. The colorization process is carried out in the following manner. The computer program  28  takes a pixel  221 , and determines a standard deviation value for that pixel at  211 , based on luminance values for the pixel  221  and several neighboring pixels. The computer program takes the luminance value and the calculated standard deviation value for that pixel  221 , and searches the reference information  31  for a row with either an exact match or, in the absence of an exact match, the closest match. The computer program then takes the red, green and blue values from that row, and assigns them to the corresponding pixel  222  in the color source image  218 . The computer program then repeats this process for each of the other pixels in the image  201 . The process shown in  FIG. 8  is a fast and compact technique for colorizing a monochrome IR image. 
       FIG. 9  is a high-level flowchart showing the process that was described above in association with  FIGS. 4 and 8 . This process begins at block  301  and proceeds to block  302 , where the registered monochrome IR reference image  101  and color reference image  102  are obtained for a reference scene  13 . Then, in block  303 , for each pixel of the IR reference image  101 , a determination is made at ill ( FIG. 4 ) of a statistical value for luminance in the region of that pixel. Color information is then determined for a corresponding portion of the color reference image  102 . For each pixel, this information is used to populate a respective row in each of the tables  141  and  146  ( FIG. 4 ). 
     In block  306 , the monochrome IR source image  201  ( FIG. 8 ) is generated for a source scene. Then, in block  307 , for each pixel of the IR source image  201 , a determination is made of a statistical value for luminance in the region of that pixel. The luminance and statistical value for that pixel are then matched as closely as possible to the luminance and statistical value for a pixel of the IR reference image  101 . Then, the color information that was derived from the color reference image  102  and that is associated with the matching pixel is used as the color information for a corresponding pixel of the color source image  218 . The process of  FIG. 9  then concludes at block  308 . 
     Although a selected embodiment has been illustrated and described in detail, it should be understood that a variety of substitutions and alterations are possible without departing from the spirit and scope of the present invention, as defined by the claims that follow.