Patent ID: 12228716

DETAILED DESCRIPTION

In an embodiment, the present invention provides a microscope and a method enabling fast imaging of a large object or a large number of objects without causing movement artefacts.

The microscope comprises a motorized object stage configured to move an object, an optical imaging system configured to form an optical image of a plane in which said object is to be optically imaged, an optical scanning unit configured to move said plane to be optically imaged by said optical imaging system relative to said optical imaging system, an image sensor configured to detect said optical image of said plane formed by said optical imaging system, and a controller configured to control said motorized object stage and said optical scanning unit for simultaneously moving said object and said plane in the same direction relative to said optical imaging system while said optical image being detected by said image sensor.

The afore-mentioned plane to be optically imaged by the optical imaging system represents an object plane from which an optical image is generated. In the present context, it should be noted that the term “plane” is not to be understand restrictively in a mathematical sense as a strict two-dimensional plane. Rather, the plane is to be understood as a more or less flatly extended area of the object from which light is received by the imaging optical system in order to form an optical image of the object area.

Accordingly, the microscope enables the plane to be imaged by the optical imaging system to track the object being moved by the motorized object stage. For this, the optical scanning unit is controlled cooperatively with the object stage for moving the afore-mentioned plane relative to the optical imaging system in a manner that the object and the plane perform a synchronized movement relative to the optical imaging system while the image sensor is detecting the optical image formed by the optical imaging system. The movement of the plane to be imaged caused by the optical scanning unit corresponds to a movement of the field of view synchronized with the movement of the object. Thus, any movement artefact otherwise caused by moving the object within the field of view of the optical imaging system can be avoided. As a result, an image field stabilization is achieved allowing an imaging of laterally extended objects or an increase in the number of objects to be processed. In particular, it is possible to move the object stage at a relatively high, preferably constant speed without having to stop the object stage during image acquisition. Specifically, the motorized object stage can be operated without being restricted by the image sensor exposure time which is a limiting factor in prior art configurations.

The optical scanning unit may be configured to move said plane perpendicularly to the optical axis of the optical imaging system.

Preferably, the controller is configured to control the motorized object stage and the optical scanning unit such that the plane is stationary relative to the object while the optical image being detected by the image sensor. In such an embodiment, the object stage and the scanning unit are operated in a manner enabling synchronous movements of the image and the plane to be imaged in terms of speed and moving direction. As the plane to be imaged remains stationary relative to the moving object, any movement artefacts can be avoided from occurring.

Preferably, the optical imaging system has a first field of view, and the image sensor has a second field of view, said second field of view being encompassed by said first field of view. In other words, the optical scanning unit is used for virtually moving the second field of view of the image sensor within the larger first of field of view of the optical imaging system, wherein said first field of view can be considered to be stationary.

In a preferred embodiment, the controller is configured to control the motorized object stage and the optical scanning unit such that the second field of view including the optical image of the plane is kept encompassed by the first field of view while the second field of view being detected by the image sensor and being moved thereon.

Preferably, an area of the first field of view currently not overlapped by the second field of view forms a field of view reserve to be used for enabling the second field of view to move within the first field of view. The afore-mentioned field of view reserve may be determined such that it allows a sufficient tracking movement of the plane to be imaged. In this respect, the exposure time of the image sensor may be taken into account for determining the field of view reserve.

In a preferred embodiment, the controller is configured to control the motorized object stage for moving the object at a constant speed. Operating the object stage at a constant speed facilitates to control the optical scanning unit in synchronization with the object stage.

Preferably, the controller is configured to control the image sensor for detecting the optical image in a sequence of consecutive images, each image of said sequence representing said optical image. In this embodiment, each image of the afore-mentioned sequence may be acquired while controlling the motorized object stage and the optical scanning unit in synchronization with each other as described above.

Preferably, the controller is configured to control the optical scanning unit for moving the plane at a constant speed while each image of said sequence being detected by the image sensor. Moving the plane at a constant speed facilitates to control the optical scanning unit, in particular in case that the motorized object stage is operated at a constant speed likewise.

Preferably, the controller is configured to control the optical scanning unit for moving the plane starting from an initial position to an end position while each image of said sequence being detected by the image sensor. The initial and end positions limiting the movement of the afore-mentioned plane may be determined taking into account the exposure time required by the image sensor for detecting a single image and the speed at which the object stage is being moved.

The initial position may be the same for all images of the sequence. Further, the afore-mentioned end position may be the same for all images of the sequence, likewise. In this case, after acquisition of a single image has been completed, the optical scanning unit is reset to an initial state when starting the next image acquisition, wherein said initial state is the same as in the previous image acquisition. Such an embodiment may be advantageously used for recording an image stack, the step size thereof being defined by the afore-mentioned initial and end positions of the plane which is moved by means of the optical scanning unit. Specifically, the step size Δs may be expressed by following equation:
Δs=−vt·τ,  (1)

wherein vtdesignates the speed of the object stage, and T designates the exposure time of the image sensor.

In a preferred embodiment, the controller is configured to let the afore-mentioned initial position drift within said sequence. In this case, the step size deviates from Δs as indicated in equation (1). Rather, a modified step size Δs' being smaller than Δs is applied, wherein Δs' results in a drift or residual shift for each image acquisition. This drift d may be expressed by following equation:
d=Δs−Δs′.(2)

The drift d represents a drift of the field of view of the image sensor within the field of view of the optical imaging system. In case that a field of view reserve is provided as described above, such a drift can be tolerated and compensated by the optical scanning unit provided that an accumulated drift D satisfies following condition:
D=l0(vtτ−1)<Fa(3)

wherein l0designates a dimension of the object to be imaged, and Fadesignates the field of view reserve. The dimension l0may be known in advance or at least limited. As already explained above, the field of view reserve Fais represented by an amount, by which the field of view of the optical imaging system is larger than the field of view of the image sensor (measured in direction of movement of the plane to be imaged).

In a preferred embodiment, the controller is configured to determine a predetermined starting condition for simultaneously moving the object and the plane, said object being moved and said plane being kept stationary relative to the optical imaging system before determining said starting condition. In this embodiment, dynamically positioning the field of view of the image sensor within the stationary field of view of the optical imaging system may be used to achieve a reduction of recorded image data before starting the actual image acquisition. Thus, in an exemplary situation in which it can be assumed that a certain portion of the object stage comprises only single regions of interest, e.g. single distinct objects being spatially separated from each other, the field of view reserve may be used for positioning the field of view of the image sensor at an edge of the field of view of the optical imaging system and for continuously recording image data into a buffer memory. In this case, the data rate may be reduced by binning, and the light exposure may be reduced by lowering illumination intensity. Then, the actual image acquisition is only started when said starting condition is determined. Thus, before determining the starting condition, an image acquisition based on lower data rate and/or lower illumination may be performed.

Preferably, the controller is configured to control the image sensor for detecting a test image and to analyze the test image for determining said predetermined starting condition, said predetermined starting condition indicating a region of interest of said object being captured by said test image. Capturing the afore-mentioned test image serves e.g. to determine whether or not an object to be imaged is detected. Once an object to be imaged is detected, the field of view reserve may be used in order to control the optical scanning unit for causing the tracking operation of the field of view of the image sensor as described above. While performing the tracking operation, the actual, high-quality image acquisition may be conducted. Thus, whereas the test image may be generated with low data rate, low illumination, and long exposure time, the actual image acquisition may be performed with high data rate, high illumination and short exposure time. In particular, before determining the starting condition, long exposure times are not considered to be detrimental as movement artefacts may be tolerated when generating the test image.

The high quality image acquisition may be finished when e.g. one of the following conditions are met: continuously recording the image data stream indicates that a complete image of the (contiguous) object is recorded; a given number of individual images and thus a given volume has been captured; the field of view reserve is used up in case of the afore-mentioned drift implementation. An exemplary application of this embodiment may be imaging of 3D cell cultures (e.g. multi-cell spheroids) in microtiter plates. Depending on the preparation, it may be assumed that the individual objects are connected and that there is only one single object in each microtiter cavity roughly centered therein. Due to the asymmetry of the field of view in an OPM or SCAPE configuration caused by the inclination of the plane to be imaged, this embodiment is particularly beneficial when using an OPM or SCAPE configuration for the imaging 3D cell cultures in microtiter plates.

Preferably, the microscope is formed by a light sheet microscope. For instance, such a light sheet microscope may comprise a single objective lens facing the object and used for both illumination and detection. In particular, the microscope may be provided in an OPM or SCAPE configuration. For instance, according to an OPM configuration, the microscope may comprise an optical transport system and an optical detection system forming the optical imaging system, as well as an optical illumination system. In this case, the optical axis of the optical transport system, the optical detection system and the optical illumination system converge into an intermediate image space, i.e. intersect each other therein. However, other configurations are possible, in particular in terms of coupling the illumination light into the system. For example, the invention may be applied to light sheet configurations as disclosed in U.S. Pat. No. 8,582,203 B2 and US 2012 014 0240 A1.

According to another aspect, a method for imaging an object using a microscope is provided, comprising the following steps: moving said object by means of a motorized object stage; forming an optical image of a plane in which said object is to be optically imaged by means of an optical imaging system; moving said plane to be optically imaged by said optical imaging system relative to said optical imaging system by means of an optical scanning unit; detecting said optical image of said plane formed by said optical imaging system by means of an image sensor; and controlling said motorized object stage and said optical scanning unit for simultaneously moving said object and said plane in the same direction relative to said optical imaging system while said optical image being detected by said image sensor.

Referring to the diagram ofFIG.1, a configuration of a microscope100according to an embodiment will be explained hereinafter.

According to the embodiment shown inFIG.1, the microscope100comprises an OPM configuration without being restricted thereto. Accordingly, the microscope100comprises an optical illumination system102, an optical transport system104, and an optical detection system106. Optical axes O1, O2, O3of the optical illumination system102, the optical transport system104, and the optical detection system106, respectively, converge into an intermediate image space116, i.e. intersect each other therein.

The microscope10further comprises a motorized object stage108holding an object110to be imaged by means of the microscope100. According to the specific embodiment shown inFIG.1, the object stage110is movable by a motor (not shown inFIG.1) in a lateral direction, i.e. in a direction perpendicular to the optical axis O2of the optical transport system. Accordingly, the object stage108is configured to move the object110in lateral direction parallel to a y-axis referring to an orthogonal coordinate system as indicated inFIG.1.

The optical illumination system102comprises a light source (not shown inFIG.1), a light sheet generating device112and an objective114facing the intermediate image space116. The light sheet generating device may comprise a cylinder lens which is configured to focus the illumination light emitted by the light source in only one direction to form a light sheet. Alternatively, the light sheet generating device may comprise a scanner causing a scanning movement of the illumination light to dynamically create the light sheet. Accordingly, the optical illumination system102serves to focus the light sheet in the intermediate image space.

The optical transport system104comprises an objective118facing the object110, a tube lens120, a scan lens122, an optical scanning unit124, a scan lens126, a tube lens128, and an objective130facing the intermediate image space116. The optical scanning unit124may comprise a galvanometer mirror which is tiltable around one or more axes, and is positioned in or close to a plane conjugate with the back focal plane of the objective118, which represents the pupil plane of the telecentric optical transport system104. According to the specific embodiment shown inFIG.1, it is assumed that the optical scanning unit124is tiltable around an axis lying perpendicular to the optical axis O2of the optical transport system104and perpendicular to the drawing sheet. Needless to say that such a configuration of the optical scanning124unit is only exemplary. Any other suitable configuration may be used, for instance a digital micromirror device (DMD).

The optical detection system106comprises an image sensor132, a tube lens134, and an objective136facing the intermediate image space116. The image sensor132is formed e.g. by a camera comprising a plurality of pixel elements which are configured to convert light received by the image sensor132into electrical signals.

The microscope100further comprises a controller138which may be configured to control the overall operation of the microscope100. In the present context, the controller138in particular serves to control the motorized object stage108, the optical scanning unit124, the light sheet generating device112, and the image sensor132.

As already mentioned above, the optical illumination system102, the optical transport system104, and the optical detection system106are arranged in a such a way that their optical axes O1, O2, and O3converge into the intermediate image space116. Thus, the light sheet focused by the optical illumination system102into the intermediate image space is imaged by the optical transport system104into the object110located on the motorized object stage108. According to the configuration shown inFIG.1, a plane OE within the object110is illuminated by the light sheet.

In the specific OPM configuration as shown inFIG.1, the plane OE is oriented obliquely relative to the optical axis O2of the optical transport system104. The plane OE illuminated by the light sheet is excited to emit fluorescent light which is captured by the objective118facing the object110. Accordingly, the optical transport system104images the plane OE within the object110in form of an intermediate image into the intermediate image space116. The intermediate image of the plane OE is imaged by the optical detection system106onto the image sensor132.

The optical transport system104and the optical detection system106form an optical image system which is configured to form an optical image of the plane OE on the image sensor132. Accordingly, the image sensor132detects the optical image of the plane OE formed by the afore-mentioned optical imaging system104,106. As can be understood from the above, the optical scanning unit124is configured to move the plane OE which is to be optically imaged by the optical imaging system104,106relative thereto. According to the specific example shown inFIG.1, the optical scanning unit124is used to move the plane OE in the lateral direction y relative to the objective118facing the motorized object stage108.

The object stage108is movable relative to the optical imaging system104,106to enable processing of larger objects or a large number of objects. In order to avoid movements artefacts from occurring due to shifting the object110relative to the optical imaging system104,106, the controller138is configured to control the motorized object stage108and the optical scanning unit124for simultaneously moving the object110and the plane OE in the same direction relative to the optical imaging system104,106while the optical image being detected by the image sensor132. In particular, the controller138operates the object stage108and the optical scanning unit124in such way that the plane OE is stationary relative to the object110moving along with the object stage108while the optical image of the plane OE is being detected by the image sensor132.

The diagram ofFIG.2illustrates how the optical imaging system104,106and the image sensor132may be adapted to each other in terms of their fields of view in order to enable the motorized object stage108and the optical scanning unit124to be controlled as described above. Thus,FIG.2shows a first field of view640associated with the optical imaging system104,106and a second field of view642associated with the image sensor132. A length of the first field of view640in direction y is referred to as Fo. Likewise, a length of the second field of view642in direction y is referred to as Fc.

FIG.2further illustrates a shift of the second field of view642associated with the image sensor132within the first field of view640associated with the optical imaging system104,106when the optical scanning unit124moves the plane OE to be imaged in direction y relative to the optical imaging system104,106. As can be seen fromFIG.2, the motorized object stage108and the optical scanning unit124are controlled in such a way that the second field of view642including the optical image of the plane OE is kept encompassed by the first field of view while the second field of view642being detected by the image sensor132and being moved thereon. The second field of view having moved from the right to the left inFIG.2is illustrated with dotted lines (referred to as644inFIG.2).

As further illustrated inFIG.2, the fields of view640,642of the optical imaging system104and the image sensor132are adapted to each other such that a field of view reserve Fa is provided in direction y. The field of view reserve Fa may be used for enabling the second field of view642to move within the first field of view640in direction y. Assumed that the second field of view642refers to an initial position of the plane OE not yet moved, an area of the first field of view640, which is not overlapped by the second field of view642in this initial position, represents the afore-mentioned field of view reserve Fa (referring to direction y).

FIG.3shows a time diagram illustrating a specific example for controlling the motorized object stage108and the optical scanning unit124in order to avoid movement artefacts from occurring. InFIG.3, the horizontal axis represents the time, wherein T designates the exposure time of the image sensor132. The vertical axis of the diagram shown inFIG.3represents a position in y direction.

According to the embodiment illustrated inFIG.3, the object stage108is moved at a constant speed in direction y as indicated by a stage trajectory350, the slope thereof representing the stage speed. In order to avoid any movement artefacts possibly caused by the stage movement, the optical scanning unit124is controlled such that the plane OE tracks the movement of the object stage108in each image acquisition. In other words, during exposure time T required for imaging the plane OE in a single image acquisition, the plane OE is kept stationary relative to the object110which is moved together with the object stage108. For this, the optical scanning unit124is operated to move the plane OE in direction y at a speed being equal to the stage speed vt when detecting a single image of the plane OE by means of the image sensor132. Keeping the plane OE stationary with respect to the object110is indicated by a plane trajectory352inFIG.3, said plane trajectory352illustrating the movement of the plane OE relative to the moving object110.

Before starting the next image acquisition, the optical scanning unit124is reset into an initial position which is the same as in the previous image acquisition as can be seen from a scanning trajectory354. Thus, the optical scanning unit124is reset by the amount Δs as defined in equation (1) explained above. The reset amount Δs can be derived from the plane trajectory352.

FIG.4shows a time diagram illustrating a modified example for controlling the motorized object stage108and the optical scanning unit124in order to avoid movement artefacts from occurring. The control method shown inFIG.4differs from the method ofFIG.3in that an amount Δs' is applied when resetting the optical scanning unit124before starting the next image acquisition. This can be seen from a plane trajectory452being different from the corresponding plane trajectory352shown inFIG.3. The reset amount Δs' differs from the amount Δs according to equation (1) by a drift d as defined in equation (2) mentioned above. In particular, the amount Δs' is reduced by the drift d. Referring toFIG.2, the drift d represents a drift of the field of view associated with the image sensor132(from642to644inFIG.2) utilizing the field of view reserve Fa in direction y. Specifically, the drift d refers to a single image acquisition as can be seen from a scanning trajectory454inFIG.4corresponding to the scanning trajectory354shown inFIG.3.

As shown inFIG.4, the drift d referring to a single image acquisition accumulates over a sequence of single images to a total drift D as defined in equation (3). In particular, the total drift D takes into account an object dimension10in direction y which is known in advance or at least limited. InFIG.4, the total drift D can be derived from the scanning trajectory454corresponding the scanning trajectory shown inFIG.4.

FIG.5shows a time diagram illustrating another modified example for controlling the motorized object stage108and the optical scanning unit124. The control method shown inFIG.5differs from the method ofFIG.4in that a starting condition is determined at a time t0before the actual image acquisition is performed. The afore-mentioned starting condition may be determined by analyzing a test image which may be generated with relatively low image quality, e.g. low data rate, low illumination and/or long exposure time. Specifically, an example may be considered in which a certain portion of the object stage108comprises only single regions of interest, e.g. distinct objects being spatially separated from each other. Thus, a single region of interest may be easily detected by analyzing the test image. As long as analyzing the test image does not indicate the occurrence of a single region of interest, as indicated by a period T1inFIG.5, the object stage108may be continuously moved at a constant speed while the plane OE is kept stationary relative to the optical imaging system104,106. In other words, the optical scanning unit124is not operated to move the plane OE relative to the optical imaging system104,106during the period T1. Only when a region of interest is detected at to, the optical scanning unit124starts moving the plane OE relative to the optical imaging system104,106during the subsequent period T2.

FIG.6is a flow diagram illustrating an exemplary process for performing an image acquisition according to the embodiment shown inFIG.5.

After starting the process in step S1, the object stage108is moved at a constant speed in step S2. While moving the object stage108, a low quality image is continuously captured without moving the plane OE relative to the optical imaging system104,106in step S3, said low quality image representing the afore-mentioned test image. Further, the test image is analyzed in order to detect a region of interest. In step S4, an inquiry is performed as to whether or not a region of interest has been detected. In case that a region of interest has not been detected in step S4, the process continues to move the object stage108(step S2) and to capture the test image (step S3) without moving the plane OE relative to the optical imaging system104,106. However, in case that a region of interest has been detected in step S4, the process proceeds to step S5in which a high quality image acquisition is performed while moving the plane OE relative to the optical imaging system104,106in order to avoid movement artefacts as described above. After the high quality image acquisition has been completed, the process returns to step S2in order start low quality image acquisition again.

Specific embodiments have been described above. Needless to say that the present invention shall not be limited to these embodiments. For instance, the microscope100shown inFIG.1forms a light sheet microscope in OPM configuration. However, any other type of microscope may be used as long as such a microscope enables a tracking movement of the plane to be imaged.

Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a processor, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some one or more of the most important method steps may be executed by such an apparatus.

Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a non-transitory storage medium such as a digital storage medium, for example a floppy disc, a DVD, a Blu-Ray, a CD, a ROM, a PROM, and EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.

Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may, for example, be stored on a machine readable carrier.

Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

A further embodiment of the present invention is, therefore, a storage medium (or a data carrier, or a computer-readable medium) comprising, stored thereon, the computer program for performing one of the methods described herein when it is performed by a processor. The data carrier, the digital storage medium or the recorded medium are typically tangible and/or non-transitionary. A further embodiment of the present invention is an apparatus as described herein comprising a processor and the storage medium.

A further embodiment of the invention is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may, for example, be configured to be transferred via a data communication connection, for example, via the internet.

A further embodiment comprises a processing means, for example, a computer or a programmable logic device, configured to, or adapted to, perform one of the methods described herein.

A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

In some embodiments, a programmable logic device (for example, a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus.

While subject matter of the present disclosure has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. Any statement made herein characterizing the invention is also to be considered illustrative or exemplary and not restrictive as the invention is defined by the claims. It will be understood that changes and modifications may be made, by those of ordinary skill in the art, within the scope of the following claims, which may include any combination of features from different embodiments described above.

The terms used in the claims should be construed to have the broadest reasonable interpretation consistent with the foregoing description. For example, the use of the article “a” or “the” in introducing an element should not be interpreted as being exclusive of a plurality of elements. Likewise, the recitation of “or” should be interpreted as being inclusive, such that the recitation of “A or B” is not exclusive of “A and B,” unless it is clear from the context or the foregoing description that only one of A and B is intended. Further, the recitation of “at least one of A, B and C” should be interpreted as one or more of a group of elements consisting of A, B and C, and should not be interpreted as requiring at least one of each of the listed elements A, B and C, regardless of whether A, B and C are related as categories or otherwise. Moreover, the recitation of “A, B and/or C” or “at least one of A, B or C” should be interpreted as including any singular entity from the listed elements, e.g., A, any subset from the listed elements, e.g., A and B, or the entire list of elements A, B and C.

LIST OF REFERENCE SIGNS

100microscope102optical illumination system104optical transport system106optical detection system108motorized object stage110object112light sheet generating device114objective116intermediate image space104optical transport system118objective120tube lens122scan lens124optical scanning unit126scan lens128tube lens130objective132image sensor134tube lens136objective138controller350stage trajectory352plane trajectory354scanning trajectory640first field of view642,644second field of viewFa field of view reserveFo length of first field of view in y directionFc length of first field of view in y direction