Patent Description:
In conventional microscopy methods, a spatial control of laser usage in confocal imaging software is generally limited to manually drawn regions of interest (ROIs), i.e. to ROIs which have been determined by a user in advance. Accordingly, defining complex ROI structures is time-consuming and might be the limiting factor in specific applications, especially in in vivo experiments.

The generation of binary masks using thresholds as well as image segmentation is a tool that is successfully used during image processing. However, such binary masks have not yet been used in imaging experiments.

Regarding prior art, reference is made to documents <CIT>, <CIT>, and <CIT> disclosing microscopy methods in which preview images are generated, these preview images being used to mark an ROI. Further, document <CIT> discloses a method in which an electronic image of at least one picture detail of a preparation is captured, and a picture detail is processed analytically. Subsequently, an object to be cut out is automatically determined, and a cutting line is automatically drawn around said object to be cut out. Further, <NPL> discloses an imaging software automatically detecting ROIs.

Document <CIT> discloses a method for recording the activity of neurons with a laser scanning microscope. A high-resolution image of a sample is captured. From the image one or more ROls are determined by image processing. Based on the ROI, a modulation pattern is determined. An arbitrary waveform generator driving an excitation light source encodes the modulation pattern as a sequence of intensity pulses of excitation light. The excitation light is then used to illuminate the sample.

<CIT> discloses a microscope comprising an image capturing unit having a CCD, and a laser unit for scanning the sample with laser light. The microscope is connected to a computer having a region specifying unit for determining a region of interest (ROI) in an image captured by the capturing unit.

<NPL> discloses an image acquisition method based on FRAP. In the method a first image is acquired at a low magnification. Using a MATLAB script, cells in the first image that are suitable for FRAP are selected, i.e. multiple ROI are selected. The selected cells are then bleached and imaged again.

It is an object of the present invention to provide a method and a microscope for imaging a sample facilitating spatial control of a light beam applied to the sample. The afore-mentioned object is achieved by the subject-matter of the independent claims.

The proposed method for imaging a sample comprises the steps according to claim <NUM>.

According to this method, a sample information is automatically generated based on an analysis of a first image, wherein the sample information enables a controlled modulation of the light beam while the light beam is scanning the sample. In this way, the controlled modulation based on the extracted sample information can be used for providing an effective spatial control of the laser light usage in a specific imaging experiment. For instance, the extracted sample information may be used to turn the laser beam on and off while scanning the sample. In this case, the extracted sample information may be considered as a virtual binary mask transmitting and blocking the light beam during its scan movement corresponding to ON and OFF states of the light beam, respectively. As another example for modulating the light beam, in order to implement a virtual grey-scaled mask, the intensity of the light beam may be varied continuously or in a plurality of steps rather than to simply turn on and off the light beam.

Usage of a virtual mask being represented by the extracted sample information for controlling light application in a subsequent manipulation of the sample allows for a large variety of experiments. Further, it may also be considered to generate the sample information for modulating the light beam prior to the actual experiment, i.e. off-line, for instance by creating binary images. These images may be used to spatially control the light beam, enabling the application of arbitrary light patterns on the sample.

The modulated light beam used in the method according to the present invention may be any type of beam, in particular a laser beam. However, the modulated light beam is not a light beam causing stimulated emission depletion.

The extracted sample information comprises a modulation map determining a relationship between at least one parameter of the light beam to be modulated and positions within the sample to be scanned with the modulated light beam. Such a modulation map, which corresponds to the afore-mentioned virtual mask, may be considered as a function between a position vector within the sample and a data vector describing data specifying the modulation of the light beam.

The modulation map may correspond to at least a part of the first image, said part being preferably more than <NUM>%, more preferably more than <NUM>%, and most preferably more than <NUM>% of the first image. The larger the map, the more accurately the light beam can be controlled. On the other hand, if the size of the map is reduced, the sample information can be extracted faster.

Preferably, the map comprises an intensity profile of at least a part of the first image, said intensity profile being generated using one or more intensity threshold values. Using such intensity threshold values enables the modulation map to be created very fast.

The extracted sample information is generated at least partly by applying an image processing algorithm increasing the resolution and/or the contrast of the first image of the sample. A suitable processing algorithm is e.g. a deconvolution algorithm and/or a deblurring algorithm. This embodiment provides for a two-stage process for extracting the sample information which is used to modulate the light beam. Thus, in a first, essentially hardware-based stage, image data is provided in form of the first image which may be a raw image with relatively low resolution. Accordingly, the first image may be taken e.g. by applying a fast, non-scanning process. In a second, essentially software-based stage, the spatial resolution is increased by applying the afore-mentioned image processing algorithm. As a result, a virtual mask corresponding to the extracted sample information can be created with high resolution despite the relatively low quality of the first image.

Another advantage can be seen in generally improving the quality of the virtual mask. Thus, by applying an enhancing image processing algorithm, any interfering artefacts may be avoided from occurring in the first place when the mask is created e.g. by applying a threshold. As a result, it may not be necessary to apply an artefact correcting segmentation algorithm at a later stage.

In a specific embodiment, the extracted sample information corresponds to a black-and-white image. In such a case, the sample information can be generated very fast.

In a further embodiment, the extracted sample information is generated by applying an image segmentation algorithm. For instance, such an image segmentation algorithm may be used to reasonably cluster black and white regions.

Further, the image segmentation algorithm may comprise a machine learning algorithm, preferably a deep learning algorithm.

In a further preferred embodiment, the method comprises the steps of displaying at least a part of the extracted sample information to a user, and modifying the extracted sample information in response to user input before recording the second image. For instance, in case that the virtual mask generated by using a threshold contains undesired selection areas, an erase tool may be used to allow the user to delete these undesired selection areas from the mask.

Preferably, the segmentation algorithm may utilize a neural network, and the user input may be used to train the neural network. In this way, the sample information corresponding to the virtual mask can be generated even more efficiently.

The light beam may be modulated in terms of at least one beam property selected from a group comprising intensity, wavelength, and polarization. The sample is deliberately bleached by the light beam. However, any other type of sample manipulation may be considered for implementing the invention, as well as any type of imaging.

The method may be used for fluorescence recovery after photo bleaching (FRAP) or fluorescence loss in photo bleaching (FLIP) experiments. In FLIP and FRAP experiments as well as in any other experiment e.g. based on photo conversion or photo activation using laser light, the method of the present invention may be used to derive ROIs from complex sample structures as nuclei, endoplasmatic reticulum etc..

The first image and/or the second image of the sample may be recorded using a scan microscope and/or a widefield microscope and/or a light sheet microscope. In particular, the first and second images may be recorded by using different microscopy methods. For instance, the first image may be generated by using a widefield method, whereas the second image may be generated by using a confocal scan method.

Preferably, the sample is scanned with the light beam by means of a scanner configured to variably deflect the light beam. Such a scanner may be formed by one or more galvanometer mirrors or by a digital micromirror device (DMD).

According to another aspect, a microscope for imaging a sample is provided, comprising the features according to claim <NUM>.

The afore-mentioned first and second imaging devices may apply the same or different imaging methods. For example, the first imaging device may be implemented as a widefield microscope, whereas the second imaging device may be implemented by a confocal scan microscope. Further, the first and second imaging device may also be formed by a single device commonly used to record the first and second images.

According to another aspect, a computer program with a program code for performing the afore-mentioned method is provided as defined by claim <NUM>.

<FIG> is a schematic diagram showing a microscope <NUM> according to an embodiment, the microscope <NUM> being configured to image a sample <NUM>.

The microscope <NUM> comprises a first imaging device <NUM> which is configured to record a first image I1 which is represented by image data ID1. The microscope <NUM> may comprise a processor <NUM> including an analyzer <NUM>. The analyzer <NUM> serves to analyze the image data ID1 in order to extract sample information SI from the first image I1.

The microscope <NUM> shown in <FIG> may further comprise an illumination unit <NUM> including a light source <NUM>, a scanner <NUM>, and a modulator <NUM>. The light source <NUM> is preferably formed by a laser device emitting a light beam LB toward the scanner <NUM>. The scanner <NUM> is e.g. formed by one or more galvanometer mirrors or a digital micromirror device and configured to variably deflect the light beam LB so that the light beam LB performs a scanning movement on the sample <NUM>. The modulator <NUM> serves to modulate the light beam LB based on the sample information SI provided by the analyzer <NUM>. Preferably, the modulator <NUM> is configured to modulate the intensity of the light beam LB depending on the sample information SI. Additionally or alternatively, the light beam <NUM> may also be modulated in terms of its wavelength and/or polarization.

It is to be noted that the configuration being comprised of the light source <NUM>, the scanner <NUM>, and the modulator <NUM> as shown in <FIG> is purely schematic. In particular, the afore-mentioned configuration is not restricted to the specific order in which the afore-mentioned components <NUM>, <NUM>, <NUM> are arranged in series, as schematically depicted in <FIG>. Thus, the modulator <NUM> may also be integrated with the light source <NUM> in order to perform the desired modulation of the light beam LB in accordance with the sample information SI generated by the analyzer <NUM>. In any case, the light source <NUM>, the scanner <NUM> and the modulator <NUM> interact with each other such that the light beam LB emitted from the illumination unit <NUM> scans the sample <NUM> while being modulated in accordance with the sample information provided by the analyzer <NUM>. In this way, at least one beam property, e.g. intensity, wavelength, and/or polarization of the light beam LB varies in a controlled manner as the light beam moves across the sample <NUM>. In other words, modulating the light beam LB results in a spatial control of the light output towards the sample <NUM>.

The microscope <NUM> may further comprise a second imaging device <NUM>. The second imaging device <NUM> serves to record a second image I2 of the sample <NUM> wherein recording is performed during and/or after scanning the sample <NUM> with the modulated light beam LB. In case that recording of the second image I2 is performed after scanning, the modulated light beam LB may be used to manipulate the sample <NUM> in preparation for a subsequent image acquisition. For instance, the light beam LB may be used to deliberately bleach the sample <NUM> in a FRAP or FLIP process. On the other hand, when recording is performed during scanning, the modulated light beam may be used to induce fluorescence within the sample to be detected by the second imaging device <NUM> during image acquisition.

The analyzer <NUM> may be configured to extract the sample information SI in form of a modulation map determining a relationship between at least one parameter of the light beam LB to be modulated and positions within the sample <NUM> to be scanned with the modulated light beam LB. For instance, in order to generate the afore-mentioned map, the analyzer <NUM> may use one or more intensity threshold values to generate an intensity profile of the first image I1 or a part thereof. Further, the analyzer <NUM> may apply an image processing algorithm, e.g. a deconvolution algorithm and/or a deblurring algorithm to the image data ID1 such that the resolution and/or the contrast of the first image I1 increases. In other words, whereas the first image I1 may be generated with relatively low resolution or contrast, the sample information SI can be generated with relatively high resolution or contrast. The modulation map may be considered as a virtual mask corresponding e.g. to a black-and-white image or a grey-scale image.

According to the embodiment shown in <FIG>, the microscope <NUM> may further comprise a display device <NUM> which is used to display at least a part of the extracted sample information SI to a user. By operating an input device <NUM>, the user may provide a user input UI in order to modify the sample information SI according to his or her preferences. For example, the input device <NUM> may be used as an erase tool allowing the user to delete undesired selection areas from the virtual mask corresponding to the sample information SI. In this case, the input device <NUM> outputs a modified sample information MSI to the analyzer <NUM> which subsequently uses the modified sample information MSF for modulating the light beam LB (as indicated by a dotted arrow in <FIG>).

<FIG> illustrate an example work flow of a fictional FRAP experiment generally performed for determining the kinetics of diffusion through tissue or cells.

In a first step illustrated in <FIG>, an image of a protein of interest is acquired (left). Further, an image of a reference dye is acquired (right).

In a second step illustrated in <FIG>, a threshold is applied to generate a virtual binary mask based on the image of the reference dye. In the illustrated example, the mask shows white areas at the positions of cell nuclei.

In a third step illustrated in <FIG>, the laser power is modulated such that only the fluorescent protein in the nucleus is bleached.

In a fourth step illustrated in <FIG>, a subsequent time-lapse recording is performed so that the diffusion of the fluorescently tagged protein into the bleached nucleus area can be monitored.

As can be understood from the above description, the imaging method according to the present invention takes much less time compared to manual ROI drawing. Especially in vivo samples tend to change shape rapidly thereby limiting the available time for ROI generation. Due to the automatic ROI generation as described herein, a plurality of ROIs exhibiting complex structures can be generated simultaneously. In contrast, drawing those structures manually usually results in less complex and less detailed structures to be imaged.

Claim 1:
A method for imaging a sample (<NUM>) using a microscope (<NUM>), comprising the steps of:
recording a first image (I1) of the sample (<NUM>), the first image (I1) being represented by image data (D1),
extracting sample information (SI) from the first image (I1) by analyzing the image data (D1) using an analyzer (<NUM>) configured to analyze the image data (D1),
scanning at least a part of the sample (<NUM>) with a light beam (LB) while modulating the light beam (LB) based on the extracted sample information (SI), wherein the sample (<NUM>) is deliberately bleached by the light beam (LB), and
recording a second image (l2) of the sample (<NUM>) during and/or after scanning the sample with the modulated light beam (LB),
wherein said extracted sample information (SI) comprises a modulation map determining a relationship between at least the intensity of the light beam (LB) and positions within the sample (<NUM>) to be scanned with the modulated light beam (LB),
wherein said extracted sample information (SI) is generated at least partly by applying an image processing algorithm increasing the resolution and/or the contrast of the first image (I1) of the sample (<NUM>).