Abstract:
The invention relates to improving the spatial resolution of images captured using multi-beam scanning confocal imaging systems by developing the mechanisms required in a variety of multibeam confocal scanner formats that enable the data capture requirements of the prior art calculations to be met.

Description:
TECHNICAL FIELD 
       [0001]    The present invention relates to improving the spatial resolution of images captured using multi-beam scanning confocal imaging systems by developing the mechanisms required in a variety of multi-beam confocal scanner formats that enable the data capture requirements of the prior art calculations to be met. 
       DESCRIPTION OF THE PRIOR ART 
       [0002]    Attempts at improving spatial image resolution beyond the Abbe resolution limit in wide field and confocal imaging systems have used structured illumination methods combined with manipulation of the Fourier representation of those images (Mueller and Enderlein, Physical Review Letters 104, 198101 (2010)). These attempts have been targeted at traditional single beam scanning confocal systems and have not been applied to multi-beam confocal scanning systems because the latter systems do not inherently contain the necessary elements or the precision of step by step scanning or synchronisation between elements to obtain the data required for the calculations. 
         [0003]    In the prior art a single point scanning confocal system is made to scan a focused laser illumination beam step by step over a sample; and an image of a region of interest (ROI) Spq is captured at each stationary beam scan position producing an array of pixel intensities I(Spq) for each scan position Rj,k. The distance between spatially adjacent scan positions on each coordinate axis is substantially less than the native optical resolution of the confocal system and typically a sub-multiple of the pixel dimensions of the detector. The data from the multiple ROIs is processed to produce an effective image Ieff(Rjk)=summation of intensities for all p,q of I(Rjk−(Spq/2), Spq); the effective image Ieff(Rjk) is further processed by Fourier transformation, filtered with a weighting function equivalent to an enhanced point spread function (PSF), then reverse Fourier transformed into a spatial equivalent to produce a final image which exhibits enhanced spatial resolution. Full details of this data processing are given in the reference (Mueller and Enderlein). 
       BACKGROUND OF THE INVENTION 
       [0004]    There is a class of confocal imaging systems which use multi-beam scanning techniques to increase the speed of image capture over the traditional single beam scanners. Such systems use multiple parallel beams that scan the sample simultaneously. Multi-beam scanners come in a variety of forms, for example, those based on Nipkow spinning discs (for example U.S. Pat. No. 5,428,475), 1-D array (sometimes called swept field) (for example U.S. Pat. No. 6,856,457) or 2-D array (for example PCT WO 03/075070) scanners. 
         [0005]    Nipkow disc based systems may use a single disc, spinning on its central axis, and contain an array of apertures placed in a conjugate image plane such that the illumination source is divided into multiple scanning beams and in which the return fluorescent or reflected light that passes through the same, or an identical set of apertures (as in tandem scanning confocal systems, for example U.S. Pat. No. 4,802,748), forms a confocal image. An enhancement of the basic Nipkow disc arrangement patented by Yokogawa Electric Corporation, Japan, adds a second disc to the first such that the second disc contains an array of microlenses, positioned such that each aperture has an associated microlens. This second disc is inserted between the first disc and the light source, such that each of the microlenses collects the illumination light from a larger area and focuses it into its associated aperture, thus improving the efficiency of the illumination path. A dichromatic mirror, or beam splitter, placed between the two discs separates the emission, fluorescent or reflected, light from the illumination light and reflects it out from between the two discs. 
         [0006]    Swept field scanners use a 1-dimensional array of illumination apertures, with or without an associated microlens array, which is scanned rapidly back and forth across the sample while a piezo transducer oscillates the scanning lines of the one dimensional array along the axis of the array with an amplitude equal to the separation of the apertures in the 1-D array. The period of the piezo oscillation is typically half that of the framing rate of the image capture device, such that each half period scans the whole sample field of view. 
         [0007]    2-D array scanners use a single oscillating galvanometer mirror to scan a 2-D array of illumination apertures, with or without an associated microlens array, back and forth across the sample; each sweep completing a full scan of the sample. 
         [0008]    The fluorescent light returned in both the swept field and 2-D array scanners is de-scanned by the scanning mirror, separated from the illumination light by a suitable dichroic mirror and focused on to an array of imaging apertures matching the pattern positions of the array of illumination apertures in a conjugate focal plane. These systems also make use of microlens arrays positioned in the illumination path such that the focus plane of the points of light so generated are focused as diffraction limited spots in the conjugate sample plane with or without passing through apertures in the illumination path. 
         [0009]    Multi-beam scanning systems offer several useful advantages over single spot scanners, for example, faster scanning of the sample area enabling higher image capture rates to be achieved; and reduced photo-bleaching of the sample material with consequent reduction in phototoxic effects especially when applied to the imaging of living cells in life science research. Consequently improved spatial image resolution in these systems is also highly desirable. This invention shows how multi-dimensional confocal scanners can be adapted to meet the stringent scanning requirements and collect and process the data into the form required by the prior art for two-dimensional image resolution enhancement. 
       SUMMARY OF THE INVENTION 
       [0010]    This invention provides for improving the spatial resolution of images captured using multi-beam confocal scanning systems by developing the mechanisms required in a variety of multi-beam confocal scanner formats that enable the data capture requirements of the prior art calculations to be met. 
         [0011]    Each of the beams in a multi-beam scanning system can be considered as an individual single beam scanner, provided that there is sufficient spatial separation between any pair of adjacent beams and the beams are focused into diffraction limited spots in the sample focal plane. These conditions have to be met for successful normal operation of such confocal systems and therefore can be assumed to exist in all practical multi-beam confocal scanning systems. 
         [0012]    Each beam, or focused scanning spot of illumination, forms a special case of a structured illumination in which all Fourier components supported by the imaging optics are present (Mueller and Enderlein). In normal operation, the scanning of these multiple beams or spots is continuous and the fluorescent, or reflected, light returned from the illuminated spots in the sample focus plane is focused on to a set of confocal apertures arranged such that every illumination spot has its own confocal aperture in a conjugate focal plane and the light which passes through the confocal apertures forms a confocal image in yet another conjugate focal plane. Typically, a 2-dimensional detector, for example, a CCD camera, is placed in this focal plane (or in an optically relayed duplicate of it) to record the confocal image. Such confocal images are substantially limited in resolution by the Abbe resolution limit. 
         [0013]    If the array of illumination spots is made to sequentially step through a scan pattern rather than scan continuously, and an image is captured at each step position, each of the images so captured can be divided into an array of sub images or ROIs, a sub array or ROI being centered on the current image position of each of the illumination spots. 
         [0014]    The present invention defines the means by which this image data may be collected for processing by the methods defined in the prior art by Mueller and Enderlein in that the intensity data contained in each ROI from a series of images that together form a complete scan of the sample is processed to create an intermediate image having an improved resolution. The intermediate image is Fourier transformed, filtered with a weighting function based on an improved point spread function (PSF) and reverse Fourier transformed to create the final image containing further resolution enhancement. 
         [0015]    The requirement of the prior art is to scan the sample point by point with increments smaller than the optical resolution limit of the optical system in two orthogonal directions in the sample focal plane of the confocal scanning microscope system. This requirement is not met by current multibeam confocal scanning microscope systems. The present invention describes methods by which the various forms of multibeam confocal scanning systems may be modified to achieve this requirement. 
         [0016]    1-D or swept field confocal scanners require additional means to produce the required positional accuracy of the scanning mechanisms on both axes and to obtain the level of synchronisation required between the scanning mechanisms and the image capture by the 2-D detector. 
         [0017]    Nipkow disc and 2-D array confocal scanners require additional means to move the scan orthogonal to their basic scanning direction in order to generate the additional data required for the calculations required to create the improved resolution images. Such additional means can take various forms and has to deflect the scan in sub optical resolution dimensions with a total range of movement of the same order of magnitude as the native optical resolution of the target system. 
         [0018]    The inclusion of devices to move both the microlens and pinhole arrays by equivalent amounts provides one method which suits both the 1-D and 2-D array scanners. 
         [0019]    Moving the scan of a Nipkow disc scanner in a radial direction across the sample field of view can be achieved by a variety of methods, for example moving the center of rotation of the rotating disc(s) using displacement devices such as a piezo displacement device or a linear motor. An alternative solution is to deflect the scanning beams and this can be achieved by inserting an optical window into the optical path between the confocal scanner and the microscope and tilting the window such that the scanning beams are deflected by refraction at the air/window and window/air interfaces by an amount determined by the angle of tilt and the thickness of the window. Yet another solution is the insertion and removal of a tilted, or tilting, window in the optical path to provide the necessary displacements orthogonal to the scan direction. Due to the reciprocal nature of the optical path, the fluorescent, or reflected, light returned from the sample on the microscope is deflected by an identical amount and hence no change in position of the microlens or pinhole arrays relative to the optical path is required providing that compensation means are included for chromatic aberrations. 
         [0020]    1-D and 2-D array scanners may also use the technique of including a tilting optical window in the optical path between the confocal scanner and the microscope to shift the scanning beams. 
         [0021]    Another solution requires that the array of points in the spinning disc and in the 2-D array scanner are arranged such that subsets of scanned points within the scan pattern are displaced by sub native optical resolution distances orthogonal to the scanning direction, in this case no additional displacement mechanism is required, it remains to identify the scan positions of the relative subsets of points so that the appropriate ROI images can be used in the subsequent calculations. This solution requires no additional optics in the beam path and is computationally efficient when the sub-optical resolution scanning increment is arranged to match a sub-multiple of the detector pixel size. 
         [0022]    In the Nipkow disc type of confocal scanner, the rotational position of the discs must be known to a sufficiently precise degree such that the relative image points required by the prior art calculations can be identified with sufficient accuracy. The angular position can be identified by techniques such as adding an encoding mechanism to the rotation of the discs. This can be a separate encoding module attached to the shaft or its motor drive, or an encoding pattern (optical, mechanical, magnetic, etc.,) placed directly on one or both discs, and combined with a suitable encoder detection device. 
         [0023]    In the 2-D array scanner additional positional accuracy can be obtained by implementing such methods as closed loop control techniques to the drive of the scanning mirror. It is also necessary to ensure that the drive signal to the scanning mirror has the necessary resolution and accuracy to enable sufficiently small incremental scanning steps. 
         [0024]    Using the above techniques the required image data can be collected by methods such as stepping the scan, position by position, over the sample area and capturing an image at each step; alternatively, images may be collected while scanning ‘on the fly’ by strobing the illumination and/or gating the detector at each required scan position. 
         [0025]    The ability to collect the intensity data in parallel from multiple ROIs when using multi beam scanners significantly improves the speed of image (i.e. data) capture. The ability of modern computer systems with parallel operation of multiple and multi-core CPUs and multi-threaded software provides suitable environments for efficient use of the increased data capture rates and offers the potential for real time imaging with enhanced spatial resolution. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0026]      FIG. 1A  Nipkow disc confocal configuration showing encoder arrangement on the rotating disc and a tilting window 
           [0027]      FIG. 1B  Nipkow disc confocal configuration showing encoder on rotating shaft and a galvanometer 
           [0028]      FIG. 2A  Illustrates spiral parameters for spinning disc confocal configuration 
           [0029]      FIG. 2B  Illustration of spiral arrangements 
           [0030]      FIG. 3A  Illustrates 1-D swept field parameters 
           [0031]      FIG. 3B  Illustrates a configuration for a 1-D confocal scanner with tilting window 
           [0032]      FIG. 4  Illustrates phase shifted patterns in 2-D array confocal configuration 
           [0033]      FIG. 5A  Illustrates tilted mirror in 2D array confocal configuration 
           [0034]      FIG. 5B  Illustrates additional galvanometer in 2D array confocal configuration 
           [0035]      FIG. 5C  Illustrates moving the arrays in 2D array confocal configuration 
           [0036]      FIG. 6  Identifies the references for data manipulation 
       
    
    
     DESCRIPTION OF PREFERRED EMBODIMENTS 
       [0037]    In a typical Nipkow disc configuration ( FIGS. 1A and 1B ) an expanded laser beam illuminates the field of view of the sample  170  after passing through a microlens array disc  120 , a dichroic mirror  150  mounted between the microlens and pinhole discs, a pinhole array disc  121  and the microscope optics  160 . The matching patterns of microlenses on  120  and pinholes on  121  are constructed as a number of interleaved spiral patterns ( FIG. 2B ) where the separation distance  210  between the start and end of one spiral is filled with N other identical spirals. Therefore in one rotation of the discs, the image is scanned N times. The microlenses and pinholes are typically arranged on the spirals such that the separation distance between them along the spiral is the same as the separation distance between spirals to maintain a uniform area density of microlenses and pinholes. The positions of the microlenses and pinholes on each spiral can be described by the polar coordinates r,phi ( FIG. 2B ). To achieve sub native optical resolution positioning of the microlenses and pinholes the rotation angle Theta ( FIG. 2A ) must be monitored and used to determine when an image should be captured. The rotation angle can be determined by several methods.  FIG. 1A  shows an encoder track  101 , which may be magnetic or optical and read by a suitable detector  100 .  FIG. 1B  shows an encoder  110  mounted on the drive shaft or motor driving the discs. 
         [0038]    Sub native optical resolution spacing between the scanned tracks of the microlenses and pinholes can be obtained by using techniques such as that described in  FIG. 1A  where a thick optical window  130  is inserted into the optical path between the confocal head and the microscope and tilts about an axis perpendicular to the radius of the disc at the center of the field of view. The optical window can be tilted by devices such as a precision stepper or piezo motor  131 , thus shifting the path of the spirals by sub native optical resolution spacing. Another method is to insert a galvanometer mirror into the beams between the confocal scanner and the microscope, other positions of the galvo are possible provided that the optical path is suitably modified. One such implementation is described in  FIG. 1B  showing the beam path extended by an array of mirrors ( 140 ) which includes a galvanometer mirror  141 . Yet another method is to make the spiral patterns identical but shifted in phase such that the tracks made by the microlenses and pinholes in one spiral do not duplicate the tracks made by the microlenses and pinholes in any other spiral. This introduces N−1 additional tracks with the same spiral separation, which is sufficient to introduce sub native optical resolution δd between adjacent tracks, where δd is equal to original track spacing L divided by N tracks. In practical spinning disc systems the number of spirals is typically 12, shifting the phase for each one would provide spacing of L/12, however, it would also be practical to have 2 repeats of 6 phase shifted spirals, or 3 repeats of 4 phase shifted spirals, etc., thus reducing the rate at which the disc needs to spin if the resulting separation of the tracks is still sub native optical resolution. 
         [0039]    For a 1-D array scanning confocal configuration the scans are illustrated in  FIG. 3A  (only partial scans are shown for clarity). The spatial filter array  310  (represented by the microlens/pinhole plate in  FIG. 3B ) is rapidly scanned back and forth across the image in direction  320  (for example by a Galvo,  FIG. 3B ) producing scan lines  340  separated by distance L equal to the aperture or microlens spacing in the spatial filter. The scanning of array  310  is also combined with an oscillation along its axis in direction  330  (for example by a piezo controlled mirror,  FIG. 3B ) by an amount just less than the aperture separation (L−dL) such that the field of view is fully scanned in one capture interval of the image detection device. The distance dL and the separation between the oscillations can be identical and sub native optical resolution of the system and are conveniently set to be an integral sub-multiple of the microlens and pinhole spacing L. Scanning of the 1-D arrays may be made in a variety of ways, such as reflecting the arrays off two orthogonal galvanometer mirrors, by reflecting off piezo controlled mirrors, by physical displacement of the arrays in the confocal head, by adjustment of a motorized  362  tilted window  361  in the optical path between the confocal head and a microscope, or combinations of these methods. The illumination light path is from the laser input via mirror  351 , piezo mirror, galvo mirror, dichroic  352 , optional tilting window  361  to the microscope. The fluorescent or reflected light from the sample is reflected at the beam splitter or dichroic mirror  352 , the mirror  353 , de-scanned by the Galvo and Piezo mirror, mirror  354 , passed through the pinholes in the Microlens/pinhole plate and via mirror  355  rescanned by the Piezo mirror and Galvo via mirror  356  to form an image of the sample at the Image Plane ( FIG. 3B ). 
         [0040]    In a 2-D array scanning confocal system, the sub native optical resolution increments along the normal scanning direction can be set by precise control of the image scanning galvanometer. If only a single pattern of microlenses and pinholes is used for scanning then an additional mechanism is required to shift the scan in sub native optical resolution steps in the direction perpendicular to the normal scan direction. This can use techniques such as an additional galvanometer mirror  520  (FIG.  5 B) or an adjustable tilting thick optical window  510  ( FIG. 5A ) placed in the optical path between the confocal head and a microscope and tilting about an axis  511  perpendicular to the normal scan direction; such tilted window can be driven by devices such as a galvanometer, stepper or piezo motor. 
         [0041]    An alternative method requires a change to the microlens and pinhole arrays such that the pattern of microlenses and pinholes is extended, either to include additional pinholes that interleave with the existing pattern or including additional pattern repeats  410 ,  411 ,  412  ( FIG. 4 ), each repeat being phase shifted so as to produce interleaved scan lines whose separation is sub native optical resolution. Yet another method of achieving sub native optical resolution stepping is to physically displace both the microlens and pinhole arrays in a direction orthogonal to the normal scan direction ( FIG. 5C ). For example, piezo, stepper, linear or rotational motors  531 ,  532  could be used to achieve this displacement. The potential disadvantage is that the movement of both arrays must be extremely precise to avoid relative displacements between the arrays which would seriously affect the confocal and resolution performance of the system. An electronic or computer control system  550  drives the precision motors  531 ,  532  in synchronism with the Galvo. 
         [0042]    Each of the embodiments requires a means to synchronize the image capture to a stationary pattern of illumination spots in the sample. In the case of spinning disc configurations it is not trivial to drive the disc to a specific rotation angle and to step it to an adjacent angle quickly and accurately due to the inertial mass of the disc and its drive motor, etc., so a means of apparently stopping the disc motion is required. This may take the form of a stroboscopic control of the light source, where the illumination source, typically but not exclusively a laser, is turned on and off extremely rapidly. A laser light source may be modulated by direct modulation of a laser diode, by passing the laser beam through an electro-optic or acousto-optic device such as an AOM or AOTF, or a high speed shutter. An alternative method is to illuminate the sample continuously and to gate the detector, such that the detector exposure time is very short and effectively ‘stops’ the disc motion. The cue for each image capture is derived from the rotational position of the discs provided by an encoder device attached directly to the shaft of the discs or to the drive motor, or via an encoder track (magnetic or optical) formed on at least one of the discs and a suitable detection device that reads the encoded track. 
         [0043]    The 1-D array scanner can be driven step by step on each axis, using any of the scan moving mechanisms detailed in the embodiments previously described, thus the image capture can be readily synchronized with these movements. It is also possible to continuously scan with the 1-D array scanner and use stroboscopic illumination or detector gating as described for the spinning disc embodiments. 
         [0044]    The 2-D array scanner can also be driven step by step on each axis, using any of the scan moving mechanisms detailed in the embodiments previously described, thus the image capture can be readily synchronized with these movements. It is also possible to continuously scan with the 2-D array scanner and use stroboscopic illumination or detector gating as described for the spinning disc embodiments. 
         [0045]    Each embodiment contains an electronic or computer subsystem that coordinates the movement of the scan, control of the illumination, the capture of the images, and the data processing steps. 
         [0046]    The data processing steps are described with reference to  FIG. 6  which represents an embodiment of the two-dimensional array scanner. The one-dimensional array scanner data is processed in the same manner as that from the two-dimensional array scanner, the difference is that the position of beams in the spatial filter has a single fixed value for one axis, say x. The spinning disc scanner can also be processed in the same manner, the difference being a conversion from the polar coordinates of the scanning and spatial filter positions into rectangular coordinates prior to testing for inclusion in the data to be passed to the prior art algorithm. 
         [0047]    The sample in the field of view has coordinates j,k corresponding to the coordinates j,k in the prior art reference. 
         [0048]    The multibeam spatial filter has coordinates of x,y for the positions of the microlens or pinhole elements in the filter. 
         [0049]    The spatial filter scan position coordinates are u,v 
         [0050]    For each scan position u,v an image is captured in which an array of ROIs T(x,y) is recorded. Each ROI has dimensions p,q and is centered on the coordinates x,y. 
         [0051]    For each of the multi-beam scanner types, the ROIs centered on each beam position are captured simultaneously and the intensity data from the ROIs is loaded into the appropriate elements of the four-dimensional array I(R(u+x,v+y),Spq) according to the scan position u,v and ROI center position x,y after testing that the values of u+x and v+y lie within the image field of view dimensions j and k respectively. Such computational processing may be carried out in parallel, thus improving the speed of creation of the array I(R(u+x,v+y),Spq) which is equivalent to the array I(Rjk,Spq) of the prior art reference. This equivalent array is processed according to the prior art to produce an image with enhanced spatial resolution. 
         [0052]    In the case of spinning disc scanners the ROIs are centered on positions r,phi in the spatial filter. The position of the spatial filter is given by d, theta. These positions are converted from polar to rectangular coordinates before testing if they lie within j,k.