Patent Publication Number: US-8988520-B2

Title: Method and apparatus for improving depth of field (DOF) in microscopy

Description:
BACKGROUND 
     1. Field of the Invention 
     Embodiments of the present invention generally relate to processing images pertaining to digital microscopy and, more specifically, to a method and apparatus for improving depth of field (DOF) in microscopy. 
     2. Description of the Related Art 
     A digital microscope images a specimen at different depths to create a sequence of digital images. Each image represents a slice of the specimen at a particular depth of focus. Thus, in a single slice of a specimen only a fraction of the entire specimen is in focus. The sequence of digital images is stacked along a Z dimension corresponding to depth, referred to as a Z-stack. Each image in a Z-stack is focused at a different depth in the captured specimen. The optics of digital microscopes impose a very narrow depth of field, too small to capture the entire specimen in focus with a single picture. Thus, all images in the Z-stack are of the same specimen, but with different focus for each image. 
     Existing methods for improving DOF of images, i.e., extended depth of field (EDOF) images, have a common underlying rationale. In existing methods, the best image for each specified location in an output image is found by identifying the digital image at a depth where the identified digital image gives the best focus at the specified location. As a result, an output pixel in the output image acquires a value based on the best image found. Existing methods differ in defining (i.e. determining) best focus and enhancing consistency in estimating depth-map. However, performance of these methods is often poor and the methods are problematic due to their high computational complexity. 
     Thus, there is a need for method and apparatus for improving depth of field (DOF) in microscopy. 
     SUMMARY OF THE INVENTION 
     An apparatus and/or method for improving depth of field (DOF) in microscopy, substantially as shown in and/or described in connection with at least one of the figures, as set forth more completely in the claims. 
     Various advantages, aspects and novel features of the present disclosure, as well as details of an illustrated embodiment thereof, will be more fully understood from the following description and drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       So that the manner in which the above recited features of the present invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this invention and are therefore not to be considered limiting of its scope, for the invention may admit to other equally effective embodiments. 
         FIG. 1  depicts a block diagram of an apparatus for improving depth of field (DOF) in microscopy, in accordance with exemplary embodiments of the present invention; 
         FIG. 2  depicts an example of a Z-stack, in accordance with exemplary embodiments of the present invention; 
         FIG. 3  depicts five individual digital images of  FIG. 2 , in accordance with exemplary embodiments of the present invention; 
         FIG. 4  depicts a flow diagram of a method for improving DOF, in microscopy, in accordance with exemplary embodiments of the present invention. 
         FIG. 5  depicts a graph illustrating an example of focus measure, in accordance with exemplary embodiments of the present invention; and 
         FIG. 6  depicts processed images, in accordance with the exemplary embodiments of the present invention. 
     
    
    
     DETAILED DESCRIPTION 
     Embodiments of the present disclosure generally include a method and apparatus for improving depth of field (DOF) in microscopy. The apparatus for improving DOF in microscopy comprises a digital microscope for capturing microscope images of a specimen at different depths for creating a sequence of images and a computing device for combining a sequence of images captured from different focal distances to form a single all-focus image. The step of combining a sequence of images captured from different focal distances to form a single all-focus image comprises computing a focus measure at every pixel, finding a plurality of candidate values for an all-focus pixel based on the focus measure and blending the candidate values together according to the focus measure in order to determine a final pixel value for the all-focus image. 
     In certain embodiments, a method for Extended Depth of Field (EDOF) in microscopy, which combines a set of images taken with different focal distances to form a single all-focus image, is disclosed. It is possible to use the method to reduce a large stack of microscopy images to a single all-focus image, or to reduce a large stack of microscopy images to a smaller stack of microscopy images. Various embodiments of the method for EDOF in microscopy enable substantial data reduction in reducing a large stack of microscopy images to a single all-focus image or a smaller stack of microscopy images. According to one embodiment, the method first computes a focus measure at every pixel, and then finds multiple candidate values for the all-focus pixel based on the focus measure. The candidates are blended together according to the focus measure in order to determine the final all-focus image. A byproduct of the method is a depth-map, which may also be useful in certain applications. 
       FIG. 1  depicts a functional block diagram of an apparatus  100  for improving DOF in microscopy using a digital microscope  102 , in accordance with exemplary embodiments of the present invention. The apparatus  100  comprises a digital microscope  102  and a computing device  104 . In certain embodiments the digital microscope  102  is coupled to the computing device  104 . In other embodiments, a stack of images is created by the microscope  102  and transferred to the computing device  104 , e.g., via memory stick, compact disk, network connection or the like. 
     The digital microscope  102  captures digital images of a specimen at different depths of focus creating a sequence of images, where each image represents a “slice.” The digital microscope  102  images the specimen at different depths to create the sequence of digital images. For purposes of illustration, the sequence of digital images is represented by the following expression 1:
 
 I ( x, y, z ),   [Expression 1]
 
where (x, y) is the spatial index with x=0 to X−1 and y=0 to Y−1 and z is the depth index with z=0 to Z−1.
 
     The computing device  104  comprises a Central Processing Unit (CPU)  106 , support circuits  108 , a memory  110  and an I/O device  112 . The CPU  106  may comprise one or more commercially available microprocessors or microcontrollers that facilitate data processing and storage. The various support circuits  108  facilitate the operation of the CPU  106  and include one or more clock circuits, power supplies, cache, input/output circuits, and the like. The memory  110  comprises at least one of Read Only Memory (ROM), Random Access Memory (RAM), disk drive storage, optical storage, removable storage and/or the like. 
     The memory  110  comprises an Operating System (OS)  114  and an EDOF generation module  116 . 
     The EDOF generation module  116  comprises an all-focus image generator  118  and a depth-map generator  120 . The EDOF generation module  116  facilitates combination of the sequence of digital images captured at different focal distances by the digital microscope  102  to form an all-focus image. The EDOF generation module  116  facilitates generation of a depth-map. 
     As used herein, the term “all-focus image” refers to the reduction of a Z-stack from three dimensions to two dimensions such that result contain the entire specimen in focus. 
     According to an exemplary embodiment, the all focus-image generator  118  generates an all focus-image. The depth-map generator  120  generates a depth-map as a by-product of the overall process of generating an all-focus image. In one embodiment, the depth map is used for data compression of a Z-stack. According to other embodiments the depth map is used to quickly “snap” the Z-dimension such that it shows the image most in-focus at a desired pixel without scrolling through all images manually as described in commonly assigned U.S. patent application Ser. Nos. 13/553,434 and 13/553,428. In operation, the computing device  104  executes the EDOF generation module  116 . The EDOF generation module  116  is coupled with the all focus-image generator  118 . The EDOF generation module  116  facilitates combining of the sequence of digital images captured by the digital microscope  102  at different focal distances to form an all-focus image. The all focus-image generator  118  computes a focus measure at every pixel of each of the sequence of digital images. The all focus-image generator  118  finds multiple candidate values for an all-focus pixel based on the focus measure. The all focus-image generator  118  blends the candidates together according to the focus measure to create a pixel value for an all-focus image. 
       FIG. 2  depicts an example Z-stack  200  generated by the digital microscope  102 . Each image in the Z-stack is focused at a different depth in the specimen. The digital microscope  102  captures digital images of a specimen at different focus depths to create a sequence of images. Thus, focus depth is incremented throughout the depth of the specimen to capture the entire specimen in focus in at least one of the digital images. The sequence of digital images is stacked along the Z dimension corresponding to depth, in addition to spatial dimensions X and Y, in the Z-stack. 
       FIG. 3  depicts five individual images of the Z-stack  200  shown in  FIG. 2 , in accordance with exemplary embodiments of the present invention.  FIG. 3  illustrates images  302 ,  304 ,  306 ,  308  and  310 , each at different Z depths. For example, when a user of a digital microscope initially views a slide of a specimen, image  302  appears. If the user wishes to adjust the focus to a different Z depth, image  304  appears, and so on, forming the Z-stack  200  shown in  FIG. 2 . 
       FIG. 4  is a flow diagram of a method  400  for improving DOF in microscopy, as performed by the EDOF generation module  116  of  FIG. 1 , in accordance with exemplary embodiments of the present invention. 
     The method  400  is implemented by the EDOF generation module  116  executed by the CPU  106  on the computing device  104 . The EDOF generation module  116  comprises an all-focus image generator  118  and a depth-map generator  120 . The EDOF generation module  116  facilitates combination of the sequence of digital images captured at different focal distances by the digital microscope  102  to form an all-focus image. The EDOF generation module  116  also facilitates generation of a depth-map. 
     The method begins at step  402  and proceeds to step  404 . 
     At step  404 , the all focus-image generator  118  computes a focus measure at every pixel of each of the sequence of digital images. 
     The focus measure is represented by following Expression:
 
 H ( x, y, z )   [Expression 2].
 
The all focus-image generator  118  computes the focus measure by convolving a sequence of digital images with a two-dimensional (2-D) high-pass filter at step  404 ( a ). The sequence of digital images and the 2-D high-pass filter are represented by following Expressions:
 
 I ( x, y, z )  [Expression 3]
 
and
 
 F   1 ( x, y )  [Expression 4].
 
The all focus-image generator  118  converts the result of convolution of the sequence of digital images with the 2-D high-pass filter to a positive signal by taking the absolute value at each pixel position in the convolved result at step  404 ( b ).
 
     In certain alternate embodiments, the square of the absolute value, instead of the absolute value, is considered for converting the result of convolution. The all focus-image generator  118  convolves the result of conversion with a 2-D low-pass filter (not shown here explicitly), producing a focus measure H(x,y,z). The 2-D low-pass filter is represented by following Expression:
 
 f   2 ( x, y )  [Expression 5].
 
     The method  400  then proceeds to step  406 , where the all-focus image generator finds the largest peaks of one-dimensional function applied to the focus measure after smoothing as candidates at each position in the focus measure. The focus measure is a one-dimensional (1-D) function of z at each position in the focus measure. The 1-D function of z is represented by following Expression:
 
 H   x, y  ( z )= H ( x, y, z )  [Expression 6].
 
Based on the assumption that the focus measure is a One-Dimensional (1-D) function of z at each position in the focus measure, the all focus-image generator  118  finds peaks at maximum in H x, y  (z). According to some embodiments, a discrete 1-D function f[t] has a peak at t p  if f[t p ]&gt;f[t p +1] and f[t p ]&gt;f[t p −1], i.e., a 1-D function can have multiple peaks
 
     The peaks (local maxima) at maximum in the smoothed version of H x, y  (z) are represented by the following sequence of terms: d 1 (x, y), d 2 (x, y) . . . d p (x, y). At step  406 ( a ), the peaks are smoothed and refined. The all focus-image generator  118  smoothes the 1-D function of z, i.e., H x, y  (z), by using a low-pass filter (not shown here explicitly) represented by the following Expression:
 
 f   3 ( z )  [Expression 7].
 
Smoothing prevents noise from yielding false peaks. The all focus-image generator  118  finds the peaks in smoothed version of H x, y  (z). In certain optional embodiments, the all focus-image generator  118  refines the peaks found by finding the peak in the unsmoothed version of H x, y  (z) in a small neighborhood of the peak found from the smoothed version of H x, y  (z). The all focus-image generator  118  generates an improved DOF image, i.e. an EDOF image, at location (x, y). The EDOF image is a blended composite of pixels from original Z-stack I(x, y, z) at depths corresponding to the peaks found.
 
     At step  408 , the all focus-image generator  116  blends the candidates together according to the focus measure to determine an all-focus image corresponding to the peaks found in step  406  according to the following formula:
 
 EDOF ( x,y )=Σ i=1   p γ i   I ( x,y,d   i ( x,y )).
 
     At step  408 ( a ), the weights γ i  are chosen such that a higher focus measure value leads to higher weight in the final weighted sum. There are many ways to choose weights that satisfy this general principle. According to one embodiment, the weights are chosen as follows: For i=1 . . . p, set initial values {circumflex over (γ)} i =H x,y (d i (x,y))−min z H x,y (z). Subtracting the minimum value resets the zero according to a “noise floor”. The weights are then specified as: 
     
       
         
           
             
               γ 
               i 
             
             = 
             
               
                 
                   γ 
                   ^ 
                 
                 i 
                 2 
               
               
                 
                   ∑ 
                   
                     j 
                     = 
                     1 
                   
                   p 
                 
                 ⁢ 
                 
                   
                     γ 
                     ^ 
                   
                   j 
                   2 
                 
               
             
           
         
       
     
     According to one embodiment, for the high-pass filter f 1 (x,y), a Gaussian-based filter with parameter β is used according to the following formula: 
     
       
         
           
             
               
                 f 
                 1 
               
               ⁡ 
               
                 ( 
                 
                   x 
                   , 
                   y 
                 
                 ) 
               
             
             = 
             
               
                 δ 
                 ⁡ 
                 
                   ( 
                   
                     x 
                     , 
                     y 
                   
                   ) 
                 
               
               - 
               
                 
                   1 
                   
                     2 
                     ⁢ 
                     π 
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     
                       β 
                       2 
                     
                   
                 
                 ⁢ 
                 
                   exp 
                   ( 
                   
                     - 
                     
                       
                         
                           x 
                           2 
                         
                         + 
                         
                           y 
                           2 
                         
                       
                       
                         2 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           β 
                           2 
                         
                       
                     
                   
                   ) 
                 
               
             
           
         
       
     
     In this embodiment, β=2.0, although this value may be tuned to account for different circumstances in alternate embodiments. Note that δ(x, y) is the impulse function, defined as 1 at the origin and 0 elsewhere. 
     For the low-pass filter f 2 (x, y), a Gaussian-based filter with parameter α is used according to the following formula: 
     
       
         
           
             
               
                 f 
                 2 
               
               ⁡ 
               
                 ( 
                 
                   x 
                   , 
                   y 
                 
                 ) 
               
             
             = 
             
               
                 1 
                 
                   2 
                   ⁢ 
                   π 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   
                     α 
                     2 
                   
                 
               
               ⁢ 
               
                 exp 
                 ( 
                 
                   - 
                   
                     
                       
                         x 
                         2 
                       
                       + 
                       
                         y 
                         2 
                       
                     
                     
                       2 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         α 
                         2 
                       
                     
                   
                 
                 ) 
               
             
           
         
       
     
     According to one embodiment, α=2.0, although this value may be tuned to appropriately for different circumstances according to other embodiments. For the low-pass filter f 3 (z), a Gaussian-based filter with parameter σ is used according to the following formula: 
     
       
         
           
             
               
                 f 
                 3 
               
               ⁡ 
               
                 ( 
                 z 
                 ) 
               
             
             = 
             
               
                 1 
                 
                   
                     
                       2 
                       ⁢ 
                       π 
                     
                   
                   ⁢ 
                   σ 
                 
               
               ⁢ 
               
                 exp 
                 ( 
                 
                   - 
                   
                     
                       z 
                       2 
                     
                     
                       2 
                       ⁢ 
                       
                         σ 
                         2 
                       
                     
                   
                 
                 ) 
               
             
           
         
       
     
     In this embodiment, σ values that depend on the number of slices are used. For z-stacks with many slices, stronger smoothing is necessary (larger σ value), whereas for z-stacks with fewer slices weaker smoothing is necessary (smaller σ value). The exact value of σ may be adjusted appropriately for different circumstances. 
     According to other embodiments, simplified filters are used to achieve similar results as with Gaussian filters. According to one embodiment, averaging (mean) filters are used instead of the Gaussian filters. A byproduct of the method  400  is a depth map, generated by the depth map generator  120 . The depth map is simply the depths d i (x,y), and according to some embodiments, due to multiple peaks, there are more than one depth that has in-focus content at a particular position (x,y). Once the steps have been performed for all pixels, the method  400  proceeds to step  410  and ends. 
       FIG. 5  shows an example of focus measure, i.e. One-Dimensional (1-D) function of z, represented by H x, y  (z) in the Expression 6 in a 2-D graphical representation on one “circle” of H x, y  (z) versus z. Several points in the graph are illustrated using circles which represent the focus measures H x,y (z) at each of the images of the Z-stack, for one particular pixel in the image. Curve  500  represents the result of low-pass filtering with f 3 (z), showing more clearly the two peaks in the focus measure. 
       FIG. 6  shows example results of processing according to some embodiments of this invention with two peaks, i.e., p=2. The original data set, i.e. Z-stack, contains 50 images, several of which were shown in  FIGS. 2 and 3 . In the  FIG. 6 , image  602  is the all-focus image, and the image  604  is a depth-map composed of dominant peak at each position. An additional depth-map is generated by considering secondary peaks. In an alternative embodiment, the Z-stack of M images is used to create a plurality of all-focus images for a subset of the Z-stack. M images are merged to form N all-focus images. 
     The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the present disclosure and its practical applications, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as may be suited to the particular use contemplated. 
     While the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.