Abstract:
A method and an arrangement for optimizing the image quality of movable subjects imaged with a microscope system are proposed. The microscope system encompasses at least one objective that defines an image window. Motions of the subjects being observed are captured in the image frame. A computer system, having a means for determining a respective displacement vector field from a comparison of the respective pixels of two chronologically successive images, generates a trajectory from the synopsis of the displacement vector field of all the acquired images. A means for applying an operation to the image data along a trajectory is also provided.

Description:
CROSS REFERENCE TO RELATED APPLICATIONS  
         [0001]    This application claims priority of the German patent application 102 35 657.2 which is incorporated by reference herein.  
         FIELD OF THE INVENTION  
         [0002]    The invention concerns a method for optimizing the image quality of image sequences of movable subjects acquired with a microscope. The invention further concerns an arrangement for optimizing the image quality of image sequences of movable subjects acquired with a microscope.  
         BACKGROUND OF THE INVENTION  
         [0003]    In the observation of living and movable subjects, artifacts occur in image production because the subjects move. This on the one hand results in unsharpness (motion generates artifacts similar to those of defocusing), and on the other hand, in confocal microscopy the images exhibit poor quality (signal-to-noise ratio) because methods such as image averaging cannot be applied to a pixel when motion is present. With averaging, for example, motion would cause subject pixels to be mixed with non-subject pixels.  
         SUMMARY OF THE INVENTION  
         [0004]    It is the object of the invention to create a method with which it is possible to generate high-quality images of movable subjects, and to make possible efficient application of operations such as averaging and filtering, even to movable subjects.  
           [0005]    The object is achieved by means of a method for optimizing the image quality of movable subjects imaged with a microscope system, comprising the following steps:  
           [0006]    a) acquiring a plurality of images having a plurality of pixels;  
           [0007]    b) determining a respective displacement vector field from a comparison of the pixels of each two chronologically successive images;  
           [0008]    c) identifying a trajectory for each pixel of the image from the displacement vector fields; and  
           [0009]    d) applying an operation to the image data along a trajectory.  
           [0010]    A further object of the invention is to create an arrangement with which it is possible to generate high-quality images of movable subjects, and to make possible efficient application of operations such as averaging and filtering, even to movable subjects.  
           [0011]    The object is achieved by way of an arrangement for optimizing the image quality of movable subjects imaged with a microscope system, the microscope system comprising: at least one objective defining an image window, a detector unit for acquiring a plurality of images each having a plurality of pixels, a computer system, which encompasses a means for determining a respective displacement vector field from a comparison of the respective pixels of at least two chronologically successive images, a means for identifying a trajectory for each pixel of the image from the displacement vector fields, and a means for applying an operation to the image data along a trajectory.  
           [0012]    In order to solve the problem associated with these objects, it is advantageous that a trajectory, which records displacements and thus subject motions, is determined for each pixel of the image. The displacements and subject motions are advantageously determined as displacement vector fields which evaluate in their totality all of the motions within the scene. The displacement vector field results from a comparison of the pixels of, in each case, at least two chronologically successive images. The use of more than two images of a sequence may result in better convergence. Such displacement fields are determined by solving a flow problem, a pixel change model being formulated as a differential equation and fitted numerically to the image data using a minimum description length (MDL) method. Probably the most prominent representative of such models is the modeling of the motion of solid bodies in video technology, for which the synonym “optical flow method” has already become established. Further representatives may be found, for example, in climate modeling, where liquid bodies (from clouds to water) are modeled. Although the “optical flow” designation is not common here, this text uses the term synonymously. A trajectory is constructed by tracking the local displacement vectors from pixel to pixel, which can easily be accomplished with a computer algorithm. The trajectory determined in this fashion is a so-called guideline for the application of operations. Operations along the trajectory that is determined can be, for example (with no limitation as to generality), a deconvolution, a smoothing, or an averaging filter. An extension to the entire class of image processing classes operating in time-lateral fashion is included in this application in this context, and is left to the imagination of one skilled in the art in terms of implementing a system.  
           [0013]    A peculiarity of these new operations is that ambiguities occur as a result of the motion and the displacement vector field. For example, a subject can migrate into the image segment, and a filter with memory must treat that new pixel differently from another, more-static pixel in the same scene. Another example is the splitting of a subject into several subjects (trajectory source). Yet another is the combination of individual pixels into one (trajectory sink). This is solved by way of intelligent trajectory management.  
           [0014]    Further advantageous embodiments of the invention are evident from the dependent claims.  
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0015]    The subject matter of the invention is depicted schematically in the drawings and will be explained below with reference to the Figures, in which:  
         [0016]    [0016]FIG. 1 schematically depicts a scanning microscope;  
         [0017]    [0017]FIG. 2 schematically depicts the image frame imaged through the microscope, and the manner in which it is subdivided into individual regions or pixels;  
         [0018]    [0018]FIG. 3 schematically depicts the processing of the data obtained from the observation of living and movable subjects;  
         [0019]    [0019]FIG. 4 is a block diagram of the method according to the present invention;  
         [0020]    [0020]FIG. 5 depicts an example of a situation in which a subject leaves the image, and the identified trajectory ends at the edge;  
         [0021]    [0021]FIG. 6 depicts an example of a situation in which a subject comes into the image;  
         [0022]    [0022]FIG. 7 depicts a situation in which a subject splits and several trajectories result therefrom; and  
         [0023]    [0023]FIG. 8 depicts a situation in which several subjects combine into one subject, and the trajectories of the individual subjects end at one point. 
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0024]    [0024]FIG. 1 schematically shows an exemplary embodiment of a confocal scanning microscope system with which the method according to the present invention can be carried out. Other microscope systems can likewise be used. A detector unit  19  is equipped with a video system or a CCD sensor for acquiring images.  
         [0025]    This is not, however to be construed as a limitation of the invention. It is sufficiently clear to one skilled in the art that the invention can also be carried out with conventional microscopes with digital image production. Illuminating light beam  3  coming from at least one illumination system  1  is directed by a beam splitter or a suitable deflection means  5  to a scanning module  7 . Before illuminating light beam  3  strikes deflection means  5 , it passes through an illumination pinhole  6 . Scanning module  7  comprises a gimbal-mounted scanning mirror  9  that guides illuminating light beam  3  through a scanning optical system  12  and a microscope objective  13 , over or through a subject  15 . In the case of nontransparent subjects  15 , illuminating light beam  3  is guided over the subject surface. With biological subjects  15  (preparations) or transparent subjects, illuminating light beam  3  can also be guided through subject  15 . For that purpose, non-luminous preparations are optionally prepared with a suitable dye (not depicted, since established existing art). The dyes present in the subject are excited by illuminating light beam  3  and emit light in a characteristic spectral region peculiar to them. This light proceeding from subject  15  defines a detected light beam  17 . The latter travels through microscope optical system  13  and scanning optical system  12  and via scanning module  7  to deflection means  5 , passes through the latter and arrives, through a detection pinhole  18 , at at least one detector unit  19 , which is equipped in the exemplary embodiment depicted here with at least one photomultiplier as detector. It is clear to one skilled in the art that other detectors, for example diodes, diode arrays, photomultiplier arrays, CCD chips, or CMOS image sensors, can also be used. Detected light beam  17  proceeding from or defined by subject  15  is depicted in FIG. 1 as a dashed line. In detector  19 , electrical detected signals proportional to the power level of the light proceeding from subject  15  are generated. Since, as already mentioned above, light of more than one wavelength is emitted from subject  15 , it is useful to insert in front of detector unit  19  a selection means  21  for the spectrum proceeding from the sample. The data generated by detector unit  19  are forwarded to a computer system  23 . At least one peripheral unit  27  is associated with computer system  23 . The peripheral unit can be, for example, a display on which the user receives instructions for adjusting the scanning microscope and can also view the present setup and also the image data in graphical form. Also associated with computer system  23  is an input means comprising, for example, a keyboard  28 , an adjusting apparatus  29  for the components of the microscope system, and a mouse  30 .  
         [0026]    [0026]FIG. 2 schematically depicts an image frame  41  acquired with microscope  100 . Image frame  41  is defined by the image window determined by microscope  100 . Image frame  41  is subdivided into individual regions or pixels  39 . Movable subject  40  is located within image frame  41 . Pixels  39  can be embodied as two-dimensional regions of image frame  41 , or also as three-dimensional regions of image frame  41 .  
         [0027]    [0027]FIG. 3 shows the observation of living and movable subjects  40  and the processing of data obtained from the observation of living and movable subjects  40 . For the observation of living and movable subjects  40 , several images or image frames  41   1 ,  41   2 ,  41   3 , . . . ,  41   n  are acquired consecutively, for example using scanning microscope  100  described in FIG. 1, each image frame  41   1 ,  41   2 ,  41   3 , . . . ,  41   n  defining an XY plane or an acquired specimen volume XYZ. Between each two successive images, e.g.  41   1 ,  41   2  or  41   2 ,  41   3 , or  41   n−1 ,  41   n , a respective displacement vector field  42   1 ,  42   2 , . . . ,  42   n−1  is determined. The displacement vector field between two successive images, e.g.  41   2  and  41   3 , can be determined from a comparison of the individual mutually corresponding pixels of the two images. Proceeding from a first image  41   1  having N pixels, it is thus possible to ascertain the new positions in the next image  42   2  by way of the displacement. An even more accurate model can also be fitted for a trajectory  43 , with sub-pixel accuracy, from the discrete displacements. Advantageously, more than one successive image is then used for this accuracy-enhancing operation. Trajectory  43  for the movable subject is obtained from the plurality of displacement vector fields  42   1 ,  42   2 , . . . ,  42   n−1  by tracking the displacement vector fields of the individual images  41   1 ,  41   2 ,  41   3 , . . . ,  41   n . In the graphical depiction of trajectory  43 , the moving subjects are represented by at least one trajectory through XYt space  44 .  
         [0028]    A video contains a three-dimensional space-time (two spatial dimensions XY, one time dimension t). The pixels of a movable subject  40  thus move along a curved path (trajectory) within this space-time. Trajectory  43  that is determined defines this curved path unequivocally, and data concerning the motion of subject  40  are thereby obtained. Operations that are to be applied to the moving subject can thus be performed along trajectory  43 . For example, data about said trajectory  43  can be fed to an averaging filter, yielding an image of higher quality that takes into account the motion of subject  40 , specifically in that the signal-to-noise ratio is better. This approach is of course also possible for sequences of volumes (four-dimensional space-time), and can be transferred to any kind of operation, e.g. filters (deconvolution, smoothing). In order to produce these filters, instead of the simple summing formulas common in image processing, the continuous operation equation must be discretized to the trajectory in the curved space-time, incorporating the present geometry. Such methods are established in numerical mathematics, and are existing art in simulation technology.  
         [0029]    [0029]FIG. 4 is a block diagram of the method according to the present invention. The first step is image acquisition  50  of a series of images. As already described above, acquisition is accomplished using detector unit  19  of the microscope or scanning microscope. The data representing each image are stored in a first image memory  52 . From image memory  52 , the images are sequentially conveyed to an optical flow calculator  53 . Parallel therewith, the data of each image are conveyed to a nonlinear filter  54 . From optical flow calculator  53 , the data modified by calculator  53  are conveyed to a trajectory tracker  55  and then to a trajectory memory  56 . The data present in trajectory memory  56  are also made available to nonlinear filter  54  in order to allow discretization. As already mentioned above, any desired operations are applied to the acquired image data, taking place in nonlinear filter  54  in consideration of the stored trajcctory  43 . The data modified in this fashion travel into a second image memory  58  and can be retrieved from there, for example, for presentation on a display.  
         [0030]    [0030]FIGS. 5 through 8 depict various events that result in respectively distinguished trajectories. FIG. 5 depicts the situation in which subject  40  leaves image frame  41  during the recording of N image frames. Upon recording of the (N+1)th image frame, the subject can no longer be captured by the microscope. Trajectory  43  resulting from the N captured image frames ends at the edge of XYt space  44 . It can be deleted from trajectory memory  56  by trajectory tracker  55 .  
         [0031]    [0031]FIG. 6 depicts the situation in which a subject  40  is present in first image frame  41   1 . During the recording of N image frames, a further subject  60  enters the region of the image frame, so that it can be captured by the microscope. Subject  60  can also be captured by the microscope when the Nth image frame is recorded. In addition to trajectory  43  for subject  40 , a further trajectory  63  is added in XYt space  44  for subject  60  that has just entered the image frame of the microscope.  
         [0032]    [0032]FIG. 7 depicts the situation in which a subject  70  is present in first image frame  41   1 . By the time the Nth image frame is recorded, subject  70  has split into, for example, four subjects  70   1 ,  70   2 ,  70   3 , and  70   4 . Subjects  70   1 ,  70   2 ,  70   3 , and  70   4  can also be captured by the microscope upon recording of the Nth image frame. In addition to trajectory  43  for subject  70 , four further trajectories  73   1 ,  73   2 ,  73   3 , and  73   4  are added at an end point of trajectory  43  at a certain time t, representing the motions of four subjects  70   1 ,  70   2 ,  70   3 , and  70   4 .  
         [0033]    [0033]FIG. 8 depicts the situation in which four subjects  80   1 ,  80   2 ,  80   3 , and  80   4  are present in first image frame  41   1 . Upon recording of the Nth image frame, subjects  80   1 ,  80   2 ,  80   3 , and  80   4  have combined into one subject  80 . Trajectories  83   1 ,  83   2 ,  83   3 , and  83   4  end in XYt space  44  at a point  84 .  
         [0034]    The invention has been described with reference to a particular exemplary embodiment. It is self-evident, however, that changes and modifications can be made without thereby leaving the range of protection of the claims below.