Method and apparatus for improving image resolution

A method and apparatus that improves the quality of digital images. When provided a plurality of digital images, the method and apparatus aligns the digital images to a reference coordinate system, and synthesizes an enhanced image such that the regions of image overlap between any two images have substantially improved quality.

The invention relates to digital image processing systems and, more
 particularly, relates to a method and apparatus for improving the
 resolution of digital images.
 BACKGROUND OF THE DISCLOSURE
 Digital images frequently suffer from a range of impairments due to low
 digital capture resolution, low capture dynamic range, pixel defects
 (drop-outs), or other sensor or scanner limitations. Also, because
 compression techniques such as JPEG are frequently used to reduce the
 volume of data associated with digital images, image quality is often
 impaired due to compression artifacts. These artifacts include
 "blockiness", loss of visual texture, spurious contouring, and edge
 distortions. While the visibility of some artifacts can be reduced by
 digital filtering or other post processing techniques applied to a single
 image, these techniques may add further distortions.
 Therefore, there is a need in the art for a method and apparatus that
 improves image quality without relying upon extensive post processing of a
 single image.
 SUMMARY OF THE INVENTION
 The disadvantages associated with the prior art are overcome by the present
 invention of a method and apparatus that improves the quality of digital
 images. The inventive technique is useful for imaging situations when
 multiple images having similar picture content are available for
 processing. When provided a plurality of digital images, the invention
 aligns the digital images to a reference coordinate system, and
 synthesizes an enhanced image such that the regions of image overlap
 between any two images have substantially improved quality. The synthesis
 process, in essence, combines information in overlapping regions to form
 an enhanced image that corrects many of the image impairments that result
 from image sampling and compression such as low digital capture
 resolution, low capture dynamic range, pixel defects (drop-outs) and the
 like.

To facilitate understanding, identical reference numerals have been used,
 where possible, to designate identical elements that are common to the
 figures.
 DETAILED DESCRIPTION
 FIG. 1 depicts a block diagram of an image processing system 100 comprising
 at least one image sensor (e.g., a camera 102, although other sensors may
 be used) and an image processor 104. The camera 102 is assumed to produce
 a plurality of digital still images or digital video frames, i.e.,
 two-dimensional arrays of pixel values. However, the camera may be an
 analog sensor that produces an analog signal and additional circuitry may
 be used to convert the analog signals into digital images for processing.
 To process the digital images, the image processor 104 comprises a central
 processing unit (CPU) 106, a memory device 110, conventional CPU support
 circuits 112. The image processor may be a stand-alone computer system
 that is used to process images in accordance with the invention or the
 image processor may be embedded in a product such as a digital camera.
 When the invention is implemented using a general purpose computer system,
 the image processor 104 is generally coupled to conventional input/output
 (I/O) peripherals 108 such as a keyboard, a mouse, and a display. However,
 if the invention is implemented using an embedded processor in, for
 example, a digital camera, the general I/O peripherals may not be used.
 The CPU 106 is a general purpose computer that, when executing specific
 routines that are recalled form memory 110, becomes a specific purpose
 computer, e.g., an image processor. The CPU can be any high-performance
 processor such as a PENTIUM II.sup.8 processor manufactured by Intel
 Corporation or a POWER PC.sup.2 processor manufactured by Motorola Inc.
 The memory 110 can be random access memory (RAM), read only memory (ROM),
 a hard disk drive, a floppy disk drive or any combination thereof. The
 support circuits 212 include various conventional circuits such as frame
 grabber circuits, analog-to-digital (A/D) circuits, clock circuits, cache,
 power supplies, I/O drivers, and the like. The I/O peripherals 108
 generally include a keyboard, a mouse, and a display, but may also include
 a video tape recorder, a video disk player, and the like. The images that
 are processed by the image processor 104 may not be sourced directly from
 a sensor (camera 102), but may also be sourced from pre-recorded images
 such as would be provided by a video tape recorder or other image storage
 device.
 The present invention is embodied in an executable image processing routine
 114 that, when executed by the CPU 108, provides an image resolution
 enhancement method and apparatus. Although the embodiment of the invention
 is described as a software implementation, those skilled in the art will
 realize that the invention can be implemented in hardware or as a
 combination of hardware and software. Thus each of the functional aspects
 of the routine can be wholly or partially implemented in one or mare
 hardware devices.
 FIG. 2 depicts a simplified flow diagram of the routine 114 of the present
 invention. The routine 114 begins at step 202 by capturing (or otherwise
 having supplied) a plurality of source images. These images are generally
 stored in the system RAM while being processed. The term captured should
 be broadly understood as to include recalling images from memory such as a
 hard disk drive, directly scanning images into RAM, or capturing images
 using a digital camera and storing the images in RAM, and the like. At
 step 204, the routine aligns the source images with a reference coordinate
 system. The coordinate system is generally the coordinate system of the
 output image (the enhanced image). Lastly, at step 206, the routine 114
 synthesizes an enhanced image using the aligned source images.
 To enhance the images, the invention assumes that any noise contained in
 the images is uncorrelated from one image to another while the scene
 information in the images is correlated. As such, by averaging multiple
 images, the noise content of the images is reduced while the signal
 content is enhanced relative to the noise content. Although the present
 invention can enhance any two images containing common subject matter, the
 best results occur when the two images are slightly different, i.e., each
 of the images is produced with a slightly different perspective. Having
 additional images sampled at slightly different locations is similar (once
 the images are aligned) to having a greater number of samples of the
 scene. Additionally, such artifacts as fixed pattern noise and compression
 effects usually occur at fixed image locations with respect to the
 sampling array. Having multiple images in which these errors are displaced
 with respect to each other means that, with high probability, the combined
 image will show reduced error amplitude.
 FIG. 3 depicts a more detailed flow diagram of an illustrative embodiment
 300 of the invention, where the source images are produced by a hand-held
 digital camera. At step 302, the source images are captured in rapid
 succession while the camera is pointed steadily at a target (scene), but
 with a small amount of motion between frames due to slight unsteadiness of
 the hand-held camera. At step 304, the first image (image 0) in the
 sequence is designated the "reference image" and, at step 306, the image
 information in each of the subsequent images is aligned to this image
 using an affine transformation. These affine transformations are estimated
 by analyzing each subsequent image together with the reference image using
 a multiresolutional direct estimation process described in U.S. Pat. No.
 5,649,032 issued Jul. 15, 1997 and hereby incorporated herein by
 reference.
 Once aligned, the reference image, at step 308, is spatially interpolated
 to double its sample density using a fourth order Langragian interpolation
 method. The affine transformations (the parameters are supplied to step
 310 via path 312) are then used in step 310 to warp each respective source
 image to create temporary images all of which are represented with respect
 to the interpolated reference image coordinate system. The temporary
 images are temporarily stored in memory at step 314. Finally, at step 316,
 these temporary images are recalled from memory and combined using a
 "trimmed mean" process in which the pixel values at a location in each
 temporary (and reference) image are sorted and averaged after the largest
 and the smallest are discarded. This mode of combination provides some of
 the noise reduction effects of averaging while also allowing rejection of
 large amplitude artifacts that would distort a simple average. At step
 318, the enhanced image is displayed (and/or stored). The process of the
 present invention corrects many of the image impairments that generally
 result from image sampling and compression such as low digital capture
 resolution, low capture dynamic range, pixel defects (drop-outs) and the
 like.
 Although the source images are discussed as being captured by a hand-held
 camera having inherent movement, the movement could be intentionally added
 by intentionally moving the camera. Additionally, the reference coordinate
 system was arbitrarily chosen to be the first image coordinate system.
 However, the coordinate system can be that associated with any of the
 source images or any arbitrary coordinate system. Furthermore, the
 illustrative embodiment used, at step 306, an affine transformation to
 align the images. This transformation should be considered illustrative of
 one particular alignment process. Any alignment process that achieves
 sub-pixel alignment of the source images would be appropriate. For
 example, if the images are captured using a large camera rotation, a
 projective transformation would be more appropriate.
 Although various embodiments which incorporate the teachings of the present
 invention have been shown and described in detail herein, those skilled in
 the art can readily devise many other varied embodiments that still
 incorporate these teachings.