Abstract:
A lens system ( 100 ) and image plane ( 104 ) are used to capture a number of sample images ( 400 ) of a three dimensional scene, each sample image ( 400 ) corresponding to a different object distance ( 106 ). The in-focus portions of each of the sample images ( 400 ) are then merged into a composite image ( 620 ) which appears to have a depth of field greater than any of the sample images ( 400 ). In one embodiment, the lens system ( 100 ) is movably attached to a camera housing ( 302 ) such that a motor ( 304 ) can move the lens system ( 100 ) in the direction of the central axis of the lens system ( 100 ). A two dimensional array ( 300 ) of photo-sensors is mounted in the camera along the central axis of the lens system ( 100 ). As the lens system ( 100 ) is traversed along the central axis, points at various distances in front of the lens system ( 100 ) pass in and out of focus on the array ( 300 ) of photo-sensors. Image information is captured from the array ( 300 ) as it is traversed, producing a series of depth-differentiated sample images ( 400 ). The in-focus portion of each sample image ( 400 ) is identified through contrast analysis. The in-focus portions identified are added to produce a composite image ( 620 ) of the scene exhibiting an apparently large depth of field.

Description:
FIELD OF INVENTION 
     This invention pertains to the field of image focus enhancement. More specifically, this invention pertains to using information from multiple images to construct an image with an apparently enhanced depth of field. The invention is preferably implemented in a digital computer. 
     BACKGROUND OF THE INVENTION 
     Depth of field is a measurement of the range of depth along a view axis corresponding to the in-focus portion of a three dimensional scene being imaged to an image plane by a lens system. Several parameters of a lens system influence the depth of field of that lens system. In general, optical systems with high magnification, such as microscopes, have small depths of field. Also, optical systems which use large aperture lens systems to capture more light generally have small depths of field. 
     In some situations it is desirable to have the benefits of a larger depth of field without giving up those optical qualities which generally result in small depths of field. For example, some analyses of microscopic specimens would be aided by the availability of a high magnification microscope with a relatively large depth of field. Such a microscope could be used to more clearly image the full structure of a microscopic object which is three dimensional in nature. Ordinary microscopes generally allow the clear viewing of a thin section of such a three dimensional specimen, due to the small depth of field of those microscopes. Portions of the specimen which are on either side of the in-focus section will be out of focus, and will appear blurry. The ability to clearly see the full three dimensional structure of a specimen would aid in the understanding of the structure of that specimen. This would be especially useful when used in conjunction with biojective microscopes which allow a user to view a specimen stereoscopically. 
     Another situation in which a small depth of field can pose problems is the low light photography of a scene with large depth variations. An example of this is a landscape scene including foreground objects photographed at night. In order to get sufficient light onto the film at the image plane of the camera, a large aperture lens must generally be used. A large aperture lens, however, will result in a relatively small depth of field. Because of the small depth of field, only a portion of the scene being photographed will be in focus. 
     A conventional method of imaging the depth information of a three dimensional microscopic scene is confocal microscopy. In confocal microscopy a single photodetector is situated behind a pinhole in an opaque screen. An objective lens focuses light from an illuminated point onto the pinhole, and the screen masks out any non-focused light. The illuminated point is generally illuminated by an intense, focused light source, such as a laser. The illuminating light source and the pinhole must be scanned over a microscopic specimen, either one point at a time or in a series of lines, in order to build up information for the whole region of interest. Depth information can be extracted from the data recorded by the photodetector. The information obtained from a confocal microscope can be used to image the three dimensional structure of microscopic specimens, but such a system is too complex and expensive for typical microscopy. Also, confocal microscopy is limited to situations in which microscopic specimens are being imaged, and is not practical for imaging macroscopic scenes. 
     What is needed is a system capable of producing an image of a three dimensional scene with enhanced focus over a large depth of field, without sacrificing optical qualities which ordinarily require a small depth of field. 
     SUMMARY OF THE INVENTION 
     A lens system ( 100 ) and image plane ( 104 ) are used to capture a number of sample images ( 400 ) of a three dimensional scene, each sample image ( 400 ) having a is depth of field which is smaller than desired. Each sample image ( 400 ) corresponds to a different object distance ( 106 ), which is the distance from the objective lens of the lens system ( 100 ) to the plane in the scene that is focused on the image plane ( 104 ). The in-focus portions of each of the sample images ( 400 ) are merged into a composite image ( 620 ) that appears to have a depth of field greater than any of the sample images ( 400 ). Because the sample images ( 400 ) can each have an arbitrarily small depth of field, the lens system ( 100 ) can have a large aperture, and the scene can be highly magnified by the lens system ( 100 ). The optical qualities which ordinarily result in a small depth of field are retained in an image ( 620 ) having apparently deep focus. 
     In one embodiment, the lens system ( 100 ) is movably attached to a camera housing ( 302 ) such that a motor ( 304 ) can move the lens system ( 100 ) in the direction of the central axis of the lens system ( 100 ). A two dimensional array ( 300 ) of photo-sensors is mounted in the camera ( 308 ) along the central axis of the lens system ( 100 ). As the lens system ( 100 ) is traversed along the central axis, points at various distances in front of the lens system ( 100 ) pass in and out of focus on the array ( 300 ) of photo-sensors. Image information is captured from the array ( 300 ) as it is traversed, producing a series of depth-differentiated sample images ( 400 ). 
     The in-focus portion of each sample image ( 400 ) is identified through contrast analysis. The in-focus portions thus identified are combined to produce a composite image ( 620 ) of the scene exhibiting an apparently large depth of field. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     FIG. 1 is an illustration of a lens system  100  focusing light from an object  102  onto an image plane  104 . 
     FIG. 2 is an illustration of two examples of an object  102  which is out of focus on an image plane  104 . 
     FIGS. 3 a  and  3   b  illustrate a camera  308  used in one embodiment of the present invention. 
     FIGS. 4 a  and  4   b  illustrate images  400   a  and  400   b , which were captured by the camera  308  shown in FIGS. 3 a  and  3   b.    
     FIG. 5 illustrates a typical variation of image contrast which accompanies a change in image plane distance  110 . 
     FIG. 6 illustrates an embodiment of the present invention. 
     FIGS. 7 a  and  7   b  illustrate measurement regions  702  as subsets of pixels  704  of images  400 . 
     FIG. 8 illustrates a series of images  400   a -d used to construct composite image  620 . 
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     Referring now to FIG. 1, converging lens system  100  is shown focusing light from object  102  onto image plane  104 . Lens system  100  can be a single convex lens or a series of lenses. Alternately, lens system  100  can be a focusing mirror system which performs essentially the same function of an imaging lens system  100 . Object  102  is located a distance  106  in front of lens system  100 . Distance  106  is referred to as object distance  106 , or d o . Light from object  102  is focused on image plane  104 , which is distance  108  behind lens system  100 . Distance  108 , which is dependent upon the characteristics of lens system  100  and object distance  106 , is referred to as image distance  108 , or d i . Light from an object  102  an infinite distance in front of lens system  100  will be focused on image plane  104  when it is a distance behind lens system  100  which corresponds to the focal length of lens  100 . Focal length is represented by the variable f, and is a fixed property of lens system  100 . The mathematical relationship between d o , d i  and f is expressed in Equation 1:                  1     d   i       +     1     d   o         =     1   f             Eq.  1                                
     As used herein, focus distance  112  is the sum of d o  and d i , represented by d f . 
     As illustrated in FIG. 2, for an object  102  at a particular object distance  106 , if image plane  104  is closer to or further from lens system  100  than image distance  108 , light from object  102  will not be focused on image plane  104 . Image plane  104   a  is closer than image distance  108  to lens system  100 , and image plane  104   b  is further from lens system  100  than image distance  108 . The light from object  102  in both cases is spread out over an area on image planes  104   a  and  104   b  which is larger than in the case where the distance between image plane  104  and lens system  100  is image distance  108 . Due to this lack of focus, the image of object  102  on image planes  104   a  and  104   b  has less sharpness, and therefore less contrast, than the image of object  102  which is focused on image plane  104  in FIG.  1 . The actual distance between lens system  100  and image plane  104  is image plane distance  110 . Distance  110  is the same as distance  108  when object  102  is focused on image plane  104 . 
     Contrast in an image of object  102  will be highest, in general, when object  102  is focused, because light coming from a point on object  102  is spread over a larger area in less focused images. If the contrast at a single region of image plane  104  is monitored as image plane distance  110  is varied, the measured contrast will generally reach a maximum when image plane distance  110  equals image distance  108  for an object  102  being imaged at that region of image plane  104 . 
     In an exemplary embodiment of the present invention illustrated in FIGS. 3 a  and  3   b , a two dimensional array  300  of photo-sensitive cells is mounted in camera housing  302 . In one embodiment, array  300  is composed of Charge-Coupled Device (CCD) cells, such as the 1152 by 1728 cell CCD incorporated in the Canon EOS D2000 digital camera. Lens system  100  is mounted in housing  302  such that motor  304  can move lens system  100  along its central axis. Motor  304  is preferably an ultrasonic motor, such as the UA 80  ultrasonic motor produced by Canon Inc. Array  300  is mounted perpendicular to the central axis of lens system  100 , in order to receive light that has passed through lens system  100 . The components thus assembled in housing  302  constitute camera  308 . In the exemplary embodiment described, array  300  is an array of electronic photosensitive elements. In an alternate embodiment, array  300  can be replaced by photographic film which records a series of images  400  which correspond to unique focal distances  112 . These photographic images  400  can either be interrogated optically, or scanned into computer  306  and examined in the same manner presented here. Also, wavelengths of light other than the visible spectrum, such as X-rays, can be detected by array  300 . 
     Array  300  senses the intensity and color of light striking each of its cells and transmits this information to computer  306 , which is coupled to camera  308 . The information transmitted from array  300  to computer  306  indicates the pattern of light striking each cell of array  300  at a sample point in time. This pattern forms an image  400  associated with the sample time. Computer  306  can store several sample images  400  from array  300 . 
     Computer  306  is also coupled to motor  304 , and computer  306  can send signals to motor  304  which cause lens system  100  to traverse along its central axis, moving closer to, or farther away from array  300 . As lens system  100  moves relative to array  300 , image plane distance  110  changes, causing object distance  106  and focus distance  112  to change as well. In FIG. 3 a , light from small object  102  is focused on array  300 , and forms image  400   a  in FIG. 4 a . Object  102  appears as object image  402   a  in FIG. 4 a . In FIG. 3 b , lens system  100  is moved such that light from object  102  is not focused on array  300 . FIG. 4 b  illustrates image  400   b , which corresponds to the image on array  300  of FIG. 3 b . Object image  402   b  is noticeably larger than object image  402   a  in FIG. 4 a , due to the lack of focus. Unfocused object image  402   b  will in general exhibit lower contrast than focused object image  402   a . A typical relationship between image plane distance  110 , image distance  108 , and contrast is illustrated in FIG.  5 . 
     Referring now to FIG. 6, the operation of an exemplary embodiment is described. Computer  306  includes central processing unit (CPU)  602 . CPU  602  is coupled to image memory  604 , composite image memory  606 , camera  308 , and program disk  610 . Program disk  610  in the exemplary embodiment is a magnetic storage medium, and contains program instructions for CPU  602 . Other forms of storage for the computer instructions will be apparent to those skilled in the art, including optical disk media, random access memory (RAM), and read only memory (ROM) modules. Also, any of the components making up computer  306  can be incorporated directly into camera  308 . The program instructions stored on program disk  610  cause CPU  602  to command motor  304  to sweep lens system  100  between two positions. Several times during the sweep, array  300  communicates an image  400  to CPU  602 , which stores that image  400  in image memory  604 . In FIG. 6, four images  400   a-d  are shown stored in image memory  604 . The number of images  400  that can be stored is a function of the size of images  400  and the size of image memory  604 . 
     After the sweep, CPU  602  is used to examine the images  400  stored in image memory  604 . In alternate embodiments, this examination could be performed concurrently with the sweeping of lens system  100 . Also, a processor other than the processor responsible for commanding motor  304  can perform this examination. As illustrated in FIGS. 7 a  and  7   b , during the examination, each image  400  is divided into a number of measurement regions  702 , each measurement region  702  encompassing one or more pixel locations  704 . Each pixel location  704  ordinarily corresponds to one pixel in each image  400 . Each pixel corresponding to a single pixel location  704  represents the same portion of the scene being imaged. Ordinarily, each image  400  will have the same number of pixels, and a single pixel location  704  will correspond to the same pixel offset in each image  400 . Where camera  308  moves during the sweeping of lens system  100 , or where movement of lens system  100  alters the magnification of the image, each pixel location  704  generally corresponds to different pixel offsets within each image  400 . In such cases, either each image  400  is altered, using conventional image processing techniques, to put pixels and pixel locations  704  into one-to-one correspondence, or the examination takes into account the varying pixel offsets in each image  400 . 
     In an exemplary embodiment, the image contrast in each measurement region  702  is calculated. One example of such a calculation is to assign to each measurement region  702  a contrast value which is based on the ratio of the brightest pixel value in region  702  to the darkest pixel value in region  702 . The calculated contrast is assigned to image  400  at a location corresponding to the centroid of measurement region  702 . In this embodiment, the calculated contrast value is assigned to the pixel location  704  which coincides with the centroid  706  of measurement region  702 , indicated in FIGS. 7 a  and  7   b  by the shaded pixel location. In other embodiments, the calculated contrast value can be assigned to a region which is larger than one pixel  704 . 
     After this is done for one measurement region  702 , it is repeated for as many other measurement regions  702  as there are in image  400 . A measurement region  702  can contain some of the same pixels  704  as another measurement region  702 . In FIG. 7 b , measurement region  702   b  contains many of the same pixels as measurement region  702   a  in FIG. 7 a . The offset between measurement regions  702   a  and  702   b  in FIGS. 7 a  and  7   b  is just one pixel location  704 , because in this embodiment contrast values are calculated at a resolution of one contrast value per pixel location  704 , even though measurement regions  702  are much larger than one pixel location  704 . Different size measurement regions  702  can be used, as different images  400  have different contrast characteristics. For example, with an image  400  of many thin dark lines against a light background, the most accurate results will generally be obtained from small measurement regions  702 . An image  400  of larger objects with less contrast will generally yield better results with larger measurement regions  702 . 
     This calculation of contrast for each measurement region  702  is performed for each image  400  stored in image memory  604 . Then, for each measurement region  702 , that image  400  with the maximum contrast for the measurement region  702  is determined. This image  400  is assumed to be the image  400  which was captured when image plane distance  110  was most nearly equal to image distance  108  for the object  102  imaged at measurement region  702 . Referring now to FIG. 8, the regions of each image  400   a  -d which are determined to be most in focus are identified as regions  402  with solid lines, and out-of-focus regions are illustrated with dotted lines. Some images, such as  400   a  and  400   d , do not contain any in-focus regions  402 . Other images, such as  400   b  and  400 c, do contain in-focus regions  402  which comprise part of the scene being imaged. The in-focus regions  402  are combined by CPU  602  in composite image  620 , which is stored in composite image memory  606 . 
     An alternate embodiment of the invention uses measurement regions  702  which each comprise only one pixel location  704 . Because one pixel location  704  does not contain enough information for a contrast calculation, the brightness of the pixel in that pixel location  704  for each image  400  is examined. As previously discussed, at the point where image plane distance  110  equals image distance  108 , there will generally be maximum contrast at measurement region  702 . This point of maximum contrast will generally coincide with a point of either maximum or minimum brightness. In this embodiment, an in-focus region  402  includes that collection of pixel values in an image  400  which are either a maximum or a minimum brightness compared to pixels of the same pixel location  704  in other images  400 . These pixel values are added to composite image  620  at their pixel locations  704 . 
     In one embodiment, each image  400  corresponds to a single focus distance  112 . This requires motor  304  to start and stop repeatedly, pausing for an image  400  to be captured at each focus distance  112 . In another embodiment, motor  304  moves lens system  100  in a single, smooth movement with images  400  being captured while it is moving. The images  400  thus captured do not correspond to a single focus distance  112 , since focus distance  112  changes while image  400  is being captured, but pixels near each other do correspond to similar focal distances  112 . If the difference in focal distances  112  for pixel locations  704  of the same measurement region  702  is small, contrast measurements can be relatively accurate. In general, smaller measurement regions  702  will be less susceptible to skew in focal distance  112  due to the movement of lens system  100 . If differences in focal distances  112  are too large, in-focus regions  402  can be calculated using brightness information for each pixel location  704 , rather than using multi-pixel measurement regions  702 . 
     The above description is included to illustrate the operation of an exemplary embodiment and is not meant to limit the scope of the invention. The scope of the invention is to be limited only by the following claims. From the above description, many variations will be apparent to one skilled in the art that would yet be encompassed by the spirit and scope of the present invention. For example, the infocus regions  402  can be identified through other methods, such as stereoscopic disparity measurements and object-based methods.