Abstract:
A method and apparatus that improves the quality of digital images. When provided a plurality of digital images, the method and apparatus aligns the digital images to a reference coordinate system, and synthesizes an enhanced image such that the regions of image overlap between any two images have substantially improved quality.

Description:
The invention relates to digital image processing systems and, more particularly, relates to a method and apparatus for improving the resolution of digital images. 
     BACKGROUND OF THE DISCLOSURE 
     Digital images frequently suffer from a range of impairments due to low digital capture resolution, low capture dynamic range, pixel defects (drop-outs), or other sensor or scanner limitations. Also, because compression techniques such as JPEG are frequently used to reduce the volume of data associated with digital images, image quality is often impaired due to compression artifacts. These artifacts include “blockiness”, loss of visual texture, spurious contouring, and edge distortions. While the visibility of some artifacts can be reduced by digital filtering or other post processing techniques applied to a single image, these techniques may add further distortions. 
     Therefore, there is a need in the art for a method and apparatus that improves image quality without relying upon extensive post processing of a single image. 
     SUMMARY OF THE INVENTION 
     The disadvantages associated with the prior art are overcome by the present invention of a method and apparatus that improves the quality of digital images. The inventive technique is useful for imaging situations when multiple images having similar picture content are available for processing. When provided a plurality of digital images, the invention aligns the digital images to a reference coordinate system, and synthesizes an enhanced image such that the regions of image overlap between any two images have substantially improved quality. The synthesis process, in essence, combines information in overlapping regions to form an enhanced image that corrects many of the image impairments that result from image sampling and compression such as low digital capture resolution, low capture dynamic range, pixel defects (drop-outs) and the like. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The teachings of the present invention can be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which: 
     FIG. 1 depicts a general purpose computer system that is used to execute a routine to implement the invention; 
     FIG. 2 depicts a flow diagram of a routine that represents the present invention; and 
     FIG. 3 depicts a detailed flow diagram of an embodiment of the present invention used to enhance images produced by a hand-held still image camera. 
    
    
     To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. 
     DETAILED DESCRIPTION 
     FIG. 1 depicts a block diagram of an image processing system  100  comprising at least one image sensor (e.g., a camera  102 , although other sensors may be used) and an image processor  104 . The camera  102  is assumed to produce a plurality of digital still images or digital video frames, i.e., two-dimensional arrays of pixel values. However, the camera may be an analog sensor that produces an analog signal and additional circuitry may be used to convert the analog signals into digital images for processing. To process the digital images, the image processor  104  comprises a central processing unit (CPU)  106 , a memory device  110 , conventional CPU support circuits  112 . The image processor may be a stand-alone computer system that is used to process images in accordance with the invention or the image processor may be embedded in a product such as a digital camera. When the invention is implemented using a general purpose computer system, the image processor  104  is generally coupled to conventional input/output (I/O) peripherals  108  such as a keyboard, a mouse, and a display. However, if the invention is implemented using an embedded processor in, for example, a digital camera, the general I/O peripherals may not be used. The CPU  106  is a general purpose computer that, when executing specific routines that are recalled form memory  110 , becomes a specific purpose computer, e.g., an image processor. The CPU can be any high-performance processor such as a PENTIUM II {circle around (8)}  processor manufactured by Intel Corporation or a POWER PC {circle around (2)}  processor manufactured by Motorola Inc. The memory  110  can be random access memory (RAM), read only memory (ROM), a hard disk drive, a floppy disk drive or any combination thereof. The support circuits  212  include various conventional circuits such as frame grabber circuits, analog-to-digital (A/D) circuits, clock circuits, cache, power supplies, I/O drivers, and the like. The I/O peripherals  108  generally include a keyboard, a mouse, and a display, but may also include a video tape recorder, a video disk player, and the like. The images that are processed by the image processor  104  may not be sourced directly from a sensor (camera  102 ), but may also be sourced from pre-recorded images such as would be provided by a video tape recorder or other image storage device. 
     The present invention is embodied in an executable image processing routine  114  that, when executed by the CPU  108 , provides an image resolution enhancement method and apparatus. Although the embodiment of the invention is described as a software implementation, those skilled in the art will realize that the invention can be implemented in hardware or as a combination of hardware and software. Thus each of the functional aspects of the routine can be wholly or partially implemented in one or mare hardware devices. 
     FIG. 2 depicts a simplified flow diagram of the routine  114  of the present invention. The routine  114  begins at step  202  by capturing (or otherwise having supplied) a plurality of source images. These images are generally stored in the system RAM while being processed. The term captured should be broadly understood as to include recalling images from memory such as a hard disk drive, directly scanning images into RAM, or capturing images using a digital camera and storing the images in RAM, and the like. At step  204 , the routine aligns the source images with a reference coordinate system. The coordinate system is generally the coordinate system of the output image (the enhanced image). Lastly, at step  206 , the routine  114  synthesizes an enhanced image using the aligned source images. 
     To enhance the images, the invention assumes that any noise contained in the images is uncorrelated from one image to another while the scene information in the images is correlated. As such, by averaging multiple images, the noise content of the images is reduced while the signal content is enhanced relative to the noise content. Although the present invention can enhance any two images containing common subject matter, the best results occur when the two images are slightly different, i.e., each of the images is produced with a slightly different perspective. Having additional images sampled at slightly different locations is similar (once the images are aligned) to having a greater number of samples of the scene. Additionally, such artifacts as fixed pattern noise and compression effects usually occur at fixed image locations with respect to the sampling array. Having multiple images in which these errors are displaced with respect to each other means that, with high probability, the combined image will show reduced error amplitude. 
     FIG. 3 depicts a more detailed flow diagram of an illustrative embodiment  300  of the invention, where the source images are produced by a hand-held digital camera. At step  302 , the source images are captured in rapid succession while the camera is pointed steadily at a target (scene), but with a small amount of motion between frames due to slight unsteadiness of the hand-held camera. At step  304 , the first image (image  0 ) in the sequence is designated the “reference image” and, at step  306 , the image information in each of the subsequent images is aligned to this image using an affine transformation. These affine transformations are estimated by analyzing each subsequent image together with the reference image using a multiresolutional direct estimation process described in U.S. Pat. No. 5,649,032 issued Jul. 15, 1997 and hereby incorporated herein by reference. 
     Once aligned, the reference image, at step  308 , is spatially interpolated to double its sample density using a fourth order Langragian interpolation method. The affine transformations (the parameters are supplied to step  310  via path  312 ) are then used in step  310  to warp each respective source image to create temporary images all of which are represented with respect to the interpolated reference image coordinate system. The temporary images are temporarily stored in memory at step  314 . Finally, at step  316 , these temporary images are recalled from memory and combined using a “trimmed mean” process in which the pixel values at a location in each temporary (and reference) image are sorted and averaged after the largest and the smallest are discarded. This mode of combination provides some of the noise reduction effects of averaging while also allowing rejection of large amplitude artifacts that would distort a simple average. At step  318 , the enhanced image is displayed (and/or stored). The process of the present invention corrects many of the image impairments that generally result from image sampling and compression such as low digital capture resolution, low capture dynamic range, pixel defects (drop-outs) and the like. 
     Although the source images are discussed as being captured by a hand-held camera having inherent movement, the movement could be intentionally added by intentionally moving the camera. Additionally, the reference coordinate system was arbitrarily chosen to be the first image coordinate system. However, the coordinate system can be that associated with any of the source images or any arbitrary coordinate system. Furthermore, the illustrative embodiment used, at step  306 , an affine transformation to align the images. This transformation should be considered illustrative of one particular alignment process. Any alignment process that achieves sub-pixel alignment of the source images would be appropriate. For example, if the images are captured using a large camera rotation, a projective transformation would be more appropriate. 
     Although various embodiments which incorporate the teachings of the present invention have been shown and described in detail herein, those skilled in the art can readily devise many other varied embodiments that still incorporate these teachings.