Patent Description:
Autofocus systems are often used in optical systems e.g. cameras or microscopes to generate sharp images of objects being observed. For automatic focusing, an appropriate focus setting is determined such that inherent optical elements of the optical system can be moved to set the focus.

In neurosurgery anatomical structures are often highly magnified and viewed beyond a visible spectrum such as ultraviolet or infrared fluorescence spectrum. Due to dispersion, autofocus systems often reach limits in such applications using high magnifications or light indicating a broad spectral range.

Some autofocus systems use the information of white light for focusing. There are difficulties in focusing e.g. fluorescing objects indicating a light spectrum beyond the visible wavelength range. To generate sharp images beyond white light, manual readjustment of the focus is usually applied. However, manual readjustments are often time-consuming and hence not suitable for fluorescence light emitted within a short time interval. Manuel focusing is also subjective and often not sufficient to find the best focus.

In other implementations, lasers are used to determine a distance to the observed object for focusing. These autofocus systems do not consider the spectrum of observed light.

Above, current optical systems have difficulties in generating sharp images for the entire spectral range rather than for a specific wavelength range of observed light.

Publication <CIT> proposes to use chromatic aberration for multiplanar image acquisition in an automated scanning cytometry system.

Publication <CIT> discloses that video recordings from two or more optical channels are produced, processed, and analyzed simultaneously in order to provide quantitative analysis of action potentials, calcium transients and ionic flux in excitable cells loaded with voltage or ion sensitive dyes with distinct excitation and emission wavelengths. Publication <CIT> proposes a multifunction autofocus for automated microscopy that includes automatic coarse focusing of an automated microscope by reflective positioning, followed by automatic image-based autofocusing of the automated microscope performed in reference to a coarse focus position.

Publication <CIT> proposes an automatic focus system for an optical microscope that facilitates faster focusing by using at least two cameras.

Publication <CIT> describes an optical system that uses unwanted light to perform autofocus functions. More particularly, one or more optical elements may be used to reflect unwanted light to one or more secondary image sensors associated with an autofocus function.

Hence, there is a desire for an improved autofocus system.

This desire is addressed by the subject matter of the independent claims. Optional features are defined in the dependent claims.

<FIG> shows an embodiment of an autofocus system <NUM> configured to receive a first signal 102a, a second signal 102b and a signal comprising an information on an operation mode <NUM> to generate the output signal <NUM>. The first signal 102a corresponds to a first wavelength range and the second signal 102b corresponds to a second wavelength range. For example, the first signal 102a may be generated by a first sensor being sensitive to light indicating the first wavelength range. Similarly, the second signal 102a may be generated by a second sensor being sensitive to light indicating the second wavelength range. The first and second sensor may generate the first and second signal according to an observed light being generated by an object to be observed.

The first and the second wavelength range can be different from each other or can partly or completely comprise the same range. For example, a wavelength range may correspond to visible light, to a particular color of visible light, to infrared light, to ultraviolet light or other light indicating broader or narrower wavelength ranges.

The first and the second signal can be used to determine the output signal <NUM>. The output signal <NUM> comprises a focus setting information which can be used for focusing the object appropriately by considering the two wavelength ranges separately. For example, the output signal <NUM> can be received by a controller of a microscope. The controller may change a configuration in the microscope such as by displacing optical elements relative to each other or relative to the observed object. By this means, the observed object can brought into focus regarding the first and the second wavelength range.

In the embodiment of <FIG>, signal <NUM> is also used to generate the output signal <NUM>. Signal <NUM> comprises information on an operation mode of the autofocus system <NUM>. The operation mode indicates a use of the first wavelength range or the second wavelength range and may determine a functionality of the autofocus system <NUM>. The autofocus system <NUM> may operate according to the first wavelength range or the second wavelength range. Information on the operation mode may be provided by signal <NUM> e.g. from a user interface. For focusing the object according to different specifications, the operation mode may be used to generate different output signals <NUM> comprising different focus setting information.

For example, the observed object generates light with a broad spectrum. The observed light can be focused according to different operation modes indicating a different use of the wavelength ranges. The first wavelength range may correspond to visible light and the second wavelength range may correspond to infrared light with a wavelength range closely around <NUM> due to Indocyanine green (ICG) fluorescence emission. By using a first operation mode, the autofocus system <NUM> can focus the object according to the visible light. By this means, the autofocus system <NUM> can be used to generate a sharp white light image of the object. By using a second operation mode, the autofocus system <NUM> can focus the object according to the visible light and the fluorescence light around <NUM>. By this means, the autofocus system <NUM> can be used to view a sharp image of the object being in focus for white light and for fluorescence light at around <NUM>. Further, the autofocus system <NUM> may function in a third operation mode wherein the object can be focused only according to the infrared light at <NUM> without considering the visible light.

In another embodiment, the autofocus system <NUM> may use three or more signals and the information on the operation mode for generating the output signal. The third signal may be used for focusing the object according to a third wavelength range.

In the embodiment shown in <FIG>, the first signal 102a comprises an image of an object in the first wavelength range and the second signal 102b comprises an image of the object in the second wavelength range. The image of the object in the first wavelength range, in the following referred to as the first image, can be generated by a first sensor and the image of the object in the second wavelength range, referred to as the second image, can be generated by a second sensor. The sensors may be able to detect the light from the object and to generate the signals 102a-b comprising an image of the object. Images can be white light images, binary images, fluorescence images or other types of images being generated by the sensor. The first and the second image can be similar or different from each other e.g. depending on a characteristic of the sensor such as spectral sensitivity or resolution.

In the described embodiment, the autofocus system <NUM> is further configured to use the first signal 102a to determine a first focus and the second signal 102b to determine a second focus. Further, the autofocus system <NUM> is configured to determine the focus setting information using the first focus and the second focus. In the embodiment, the first focus can be determined by analyzing the first signal 102a comprising the first image and the second focus can be determined by analyzing the second signal 102b comprising the second image. By this means, the autofocus system <NUM> is able to consider the first wavelength range independently from the second wavelength range for better focusing.

The separation into the first and the second focus may be advantageous to reduce the effect of dispersion when focusing an object. Imaging properties of optical elements such as lenses depend on the wavelength of light being projected. Thus, an observed object generating light with a broad wavelength range may not be focused appropriately if only one focus is considered. To be able to generate a sharp image of the object for a broad spectrum, the autofocus system <NUM> may use one focus for each wavelength range.

Focus differences can be greater for high magnifications. <FIG> illustrates an example of a diagram <NUM> comprising focal positions depending on the wavelength when imaging with a surgical stereomicroscope at high magnifications. The focus difference can amount up to several millimeters such as <NUM> between a wavelength of <NUM> and <NUM>. Hence, if only one focus is considered for a broad wavelength range, blurred images can be generated due to wavelength dependent optical characteristics.

The first focus and the second focus can be determined by the images of the signals 102a-b, respectively. For example, images may comprise contrast information according to an intensity distribution over pixels. Sharp edges can be identified for areas comprising high intensity differences between adjacent pixels or high intensity slopes within a certain vicinity of pixels. A focus of an image can then be determined by e.g. increasing the contrast of the image for the considered wavelength range.

In other implementations, the autofocus system <NUM> may use different approaches for determining the focuses such as phase detection. Generally, the autofocus system <NUM> is not restricted to these examples and may use other methods of e.g. image analysis to determine a focus. For better analysis, the first and the second image can optionally be converted to secondary images via image processing. By this means, characteristics of the images can be revealed or emphasized such as e.g. edges, shapes, contrasts, intensity offsets etc. The autofocus system <NUM> may use a criterion such as a threshold to evaluate whether the image is sufficiently focused or not.

The first and the second focus can be used for determining the focus setting information. The focus setting information may comprise instructions for controlling optical elements or hardware components of e.g. a microscope for setting the determined focusses.

The process of determining the first and second focus can be iterative to determine an appropriate focus setting information. For example, in a first iteration the first and the second image are analyzed according to a contrast information. The autofocus system <NUM> may initially determine a temporary focus setting information e.g. used to set an intermediate focus. In a second iteration, the images focused according to contrast enhancement can be further analyzed and adjusted according to another criterium to determine another focus setting information used to set e.g. a second intermediate focus. Further iterations may be performed until a final focus setting information for sufficient focusing is determined.

In other implementations, the first and second focus can be determined in a single iteration to determine the focus setting information.

In the described embodiment of the autofocus system <NUM>, the first wavelength range corresponds to visible light and the second wavelength range corresponds to fluorescence light emitted by the object. <FIG> illustrates schematically the wavelength range of visible light between approximately <NUM>-<NUM> and wavelengths for fluorescence light at <NUM> and <NUM> in the visible range and <NUM> in the near infrared range (NIR).

The observed spectrum comprising the illustrated wavelengths may be attributable to light caused by the observed object. For example, the observed object can be illuminated by a light source generating white light of appropriate intensity. White light indicating the first wavelength range might be reflected on the surface of the observed object. Further, the illuminating light of the light source may excite the object to emit fluorescence light indicating the second wavelength range.

The illustrated light spectrum of <FIG> comprises a broad wavelength range of around <NUM> and can be e.g. observed in surgical applications. For appropriate focusing it might be suitable to consider the spectrum regarding the first wavelength range of e.g. visible light and the second wavelength range at e.g. <NUM>. In other implementations, the wavelength ranges may correspond to different light such as ultraviolet light or light comprising an arbitrary wavelength range. For example, a wavelength range can be considered as narrow if a total interval of less than e.g. <NUM> is used or can be considered as broad if a total interval of more than <NUM> is used.

In a conventional implementation, the wavelength ranges may correspond to any light being detected. The spectrum is not restricted to the example given in <FIG> and may comprise lower, higher, broader, sharper or other ranges than illustrated.

According to an implementation, the autofocus system <NUM> can be used as a standalone system receiving the first and the second signal from a separate system. The output signal generated by the autofocus system <NUM> can be used by the separate system to provide autofocusing e.g. for better imaging.

According to another implementation, the autofocus system <NUM> may be a component of main system such as an optical system.

<FIG> shows an embodiment of an optical system <NUM> comprising an autofocus system <NUM> according to the previous description. The optical system <NUM> comprises three sensors 401a-c generating signals 402a-c, respectively. The first sensor 401a is configured to generate a first image in the first wavelength range corresponding to visible light. The second sensor 401b is configured to generate a second image in the second wavelength range corresponding to fluorescence light at <NUM>. The third sensor 401c is configured to generate a third image in the third wavelength range corresponding to fluorescence light at <NUM>. In other implementations, the wavelength ranges may be different from the discussed embodiment.

The optical system <NUM> further comprises an operation unit 404a generating a signal 404b comprising information on an operation mode. The autofocus system <NUM> can use the signals 402a-c from the sensors 401a-c and the signal 404b from the operation unit 404a to generate an output signal <NUM>. The output signal <NUM> can be used by a controller <NUM>.

In the embodiment, the controller <NUM> is configured to control a focus system of the optical system <NUM> using the output signal <NUM>. The focus system comprises a main focus system <NUM> and a fine focus system <NUM>. Optionally, the controller <NUM> may further control an illumination system <NUM> illuminating an object <NUM>.

In the embodiment of the optical system <NUM>, the main focus system <NUM> comprises a main lens 408a and an actuator 408b to control the main lens 408a. Further, the fine focus system comprises a fine focusing lens 409a and an actuator 409b to control the fine focusing lens 409a.

After receiving the output signal <NUM>, the controller <NUM> may send the actuator 408b of the main focus system <NUM> a control signal. The control signal may prompt the actuator 408b to move the main lens 408a according to the specifications to change a focus. For example, the main lens 408a can be moved for focusing the observed object <NUM> according to the first wavelength range. By adjusting the position of the main lens 408a, a sharp intermediate image <NUM> can be generated and observed via an ocular <NUM>.

Further, the controller <NUM> may send the control signal to the actuator 409b of the fine focus system <NUM>. The control signal may prompt the actuator 409b to move the fine focusing lens 409a according to the specifications to change a focus. For example, the fine focusing lens 409a can be moved for focusing the observed object <NUM> according to the second wavelength range. By adjusting the position of the fine focusing lens 409a, a sharp image can be generated on the surface of the second sensor 401b. As generating the intermediate image <NUM> uses a first optical path and generating the image on the surface of the second sensor 401b uses a second optical path, interaction during focus setting for both wavelength ranges can be avoided.

In other implementations, the optical system <NUM> may comprise more than one fine focusing system such as one fine focusing system for each sensor 401a-c. Each fine focusing system may be implemented and configured independently from each sensor. For example, each fine focusing system may focus an image on the surface of its corresponding sensor. Hence, each fine focusing lens can be moved for focusing the observed object <NUM> according to the specific wavelength range being considered.

According to another implementations, the main focus system <NUM> may be used for the first sensor and the fine focus system <NUM> may be used for the second sensor. The main focus system <NUM> can be also used for all sensors such as to set a focus which can be further fine-tuned by the fine focus system. The fine focus system <NUM> may be also used for the first sensor, the third sensor or for all sensors simultaneously. In a conventional implementation, the main focus system may be used to generate a focused intermediate image <NUM> for viewing and the fine focus system may be used to generate a fine-focused image on the surface of at least one sensor.

The main focus system <NUM> and fine focus system <NUM> are not restricted to this implementation and may comprise further optical elements such as filters, further lenses, apertures or further elements for controlling.

For better understanding, the optical path and further optical components of the optical system <NUM> are discussed hereafter.

The observed object <NUM> can be illuminated by the illumination system <NUM>. The illumination system <NUM> may generate light with different intensities or of different spectra. For example, the illumination system <NUM> comprises a bulb for generating light with a wavelength range of <NUM>-<NUM>, multiple, selectable white light and fluorescence excitation filters or other optical components for illumination or fluorescence excitation. The observed object <NUM> may reflect visible light and emit fluorescence light.

The main lens 408a of the main focus system <NUM> may project the incoming light to a zoom system <NUM> comprising e.g. movable lenses. The zoom system <NUM> can direct the light to a beam splitter <NUM> comprising e.g. semitransparent mirrors or prisms. The beam splitter <NUM> can generate two optical paths of the incoming light. As described above, the first optical path can be used for observing the intermediate image <NUM> via the ocular <NUM> and the second optical path can be used for imaging on the sensors 401a-c. For focusing the light from the beam splitter <NUM>, a tube lens <NUM> can be used. The light of the second optical path can be focused by the fine focusing lens 409b onto the surface of the sensors 401a-c.

To direct portions of the second beam path to several sensors 404a-c, further beam splitter <NUM> can be used. The beam splitter <NUM> may comprise a coating. Filters <NUM> can be arranged between beam splitters <NUM> and sensors 401a-c. The coating of the beam splitter <NUM> and the filters <NUM> may define the spectrum of light being received by each sensor 401a-c. The sensors 401a-c may comprise further filters such as RGB filters. According to an implementation, the sensor 401a may detect a broad range of visible light, sensor 401b fluorescence emissions at <NUM> and sensor 401c fluorescence emissions at <NUM>. For example, sensors 401a-c are camera or video sensors such as CCD or CMOS sensors generating the signals 402a-c comprising the image information. The signals 402a-c can comprise different information according to different narrow or broad spectral ranges of the detected light.

The signals 402a-c can be received and further processed by the autofocus system <NUM> to focus the observed object <NUM> according to the considered wavelength ranges. The autofocus system <NUM> can determine a focus setting information also by using the operation mode determined by a user of the optical system <NUM>.

According to the embodiment in <FIG>, the optical system <NUM> further comprises an imaging system <NUM>. The imaging system <NUM> can provide a single sharp image by combining two or more focused images each based on a specific wavelength range. A combination of an image being in focus for the first wavelength range with an image being in focus for the second wavelength range can be realized e.g. according to one of the following implementations:
In a conventional implementation, an image such as the second image generated on the surface of the second sensor can be injected into an observation beam path of the optical system <NUM>. The observation beam path may comprise the image plane in which the intermediate image <NUM> is generated for viewing via the ocular <NUM>. For example, if the sensor image being in focus according to the second wavelength range is overlaid to the intermediate image <NUM> being in focus according to the first wavelength range, a single sharp image can be observed via the ocular <NUM>.

The imaging system <NUM> may enable to observe a combined image indicating a visible and invisible light spectrum. For example, the signal of the second sensor using ultraviolet (UV) fluorescence light is injected to the visible intermediate image <NUM>. By this means, a visible representation of the UV fluorescence light, e.g. providing the fluorescence information in pseudo color, can be observed in combination with the visible intermediate image <NUM>. This combined information can be observed via the ocular <NUM> such that an observer can keep his perspective without turning his head towards a monitor.

In another implementation, the imaging system <NUM> is configured to combine the first image of the first sensor 401a and the second image of the second senor 401b into an observable image. The observable image can be generated digitally such that a 2D or 3D image can be displayed on one or more screens. In some other implantations, the imaging system <NUM> uses also the third image to generate the observable image.

According to the embodiment in <FIG>, the optical system <NUM> is a microscope such as a stereomicroscope comprising a second ocular for viewing. The microscope comprises further optical elements such as additional beam splitter, lenses, zoom systems, filters, prisms or other components to view a further intermediate image by the second ocular. The imaging system <NUM> may combine an image of a sensor with one or both intermediate images.

In another implementation, the optical system <NUM> is a surgical microscope. A physician may use the surgical microscope in a first operation mode focusing objects according to white light or in a second or third operation mode focusing objects according to different fluorescence light e.g. Protoporphyrin IX (PpIX) or Indocyanine green (ICG) emissions. The first operation mode of white light can be used for analyzing a topology or structure of e.g. tissue, organs, vessels or any anatomical or biological structure to be observed in vivo or in vitro. The second or third operation mode of fluorescence light may be used to analyze e.g. physiological properties of the observed object such as anomalous characteristics, areas of insufficient blood flow or areas comprising malignant cells such as tumor cells. As physiological processes such as blood flow characteristics are only observable during a short time range of e.g. <NUM>, the autofocus system <NUM> can be beneficial to provide fast, automatic and appropriate focusing of the object.

The discussed embodiments are described for better understanding and are not to be interpreted in a restrictive way. Similarly, the illustrated figures are given to better understand the features und function of the autofocus system or optical system only. The autofocus system <NUM> can also be used in other applications such as in camera systems, video systems, other microscope systems or other optical systems suitable for observing or focusing objects regarding different wavelength ranges. The autofocus system can be used in surgery, laboratory, photography or for any workflow comprising imaging or focusing on objects. The arrangement and orientation of the elements such as optical elements, units or systems can differ for from the positions as shown in the figures.

In the following, further possible implementations or applications of the autofocus system <NUM> or optical system <NUM> are described.

An operator of the optical system <NUM> can decide if the main lens 408a and his view will adapt the focus via the main focus system <NUM> or if he keeps his focus and just the image of the sensor is focused via the fine focus system <NUM>.

One or more cameras comprising sensors 404a-c with filters <NUM> or coatings can read different parts of light spectrum of observed light <NUM> and the generated signals can be used for determining the focus setting information.

The illumination system <NUM> can be coupled with the operation unit 404a. According to a chosen operation mode, characteristics of the illuminating light e.g. intensity or wavelength range may change.

Reading and analysis of the sufficient focus for each wavelength range can be done constantly. If the operation mode changes, the main lens 408a and/or the fine focusing lens 409a can be moved directly to the determined focal position with or without analyzing the image on the sensor.

The autofocus system <NUM> can restart image analysis if e.g. the operator switches from any operation mode to another. The autofocus system <NUM> can analyze important or main structures in e.g. the fluorescence image to focus on. If appropriate, image analysis can be restricted to a Region of Interest (ROI) so that the image can primarily be in focus at this region.

According to another implementation, the autofocus analysis is done when switching from any operation mode to another.

<FIG> illustrates a flowchart of an embodiment of a method <NUM> for an autofocus system. The method comprises receiving <NUM> a first signal corresponding to a first wavelength range and receiving <NUM> a second signal corresponding to a second wavelength range. Further, the method comprises determining <NUM> an output signal comprising a focus setting information using the first signal and the second signal.

Some embodiments relate to a microscope comprising a system as described in connection with one or more of the <FIG>. Alternatively, a microscope may be part of or connected to a system as described in connection with one or more of the <FIG>. <FIG> shows a schematic illustration of a system <NUM> configured to perform a method described herein. The system <NUM> comprises a microscope <NUM> and a computer system <NUM>. The microscope <NUM> is configured to take images and is connected to the computer system <NUM>. The computer system <NUM> is configured to execute at least a part of a method described herein. The computer system <NUM> may be configured to execute a machine learning algorithm. The computer system <NUM> and microscope <NUM> may be separate entities but can also be integrated together in one common housing. The computer system <NUM> may be part of a central processing system of the microscope <NUM> and/or the computer system <NUM> may be part of a subcomponent of the microscope <NUM>, such as a sensor, an actor, a camera or an illumination unit, etc. of the microscope <NUM>.

The computer system <NUM> may be a local computer device (e.g. personal computer, laptop, tablet computer or mobile phone) with one or more processors and one or more storage devices or may be a distributed computer system (e.g. a cloud computing system with one or more processors and one or more storage devices distributed at various locations, for example, at a local client and/or one or more remote server farms and/or data centers). The computer system <NUM> may comprise any circuit or combination of circuits. In one embodiment, the computer system <NUM> may include one or more processors which can be of any type. As used herein, processor may mean any type of computational circuit, such as but not limited to a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), multiple core processor, a field programmable gate array (FPGA), for example, of a microscope or a microscope component (e.g. camera) or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system <NUM> may be a custom circuit, an application-specific integrated circuit (ASlC), or the like, such as, for example, one or more circuits (such as a communication circuit) for use in wireless devices like mobile telephones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system <NUM> may include one or more storage devices, which may include one or more memory elements suitable to the particular application, such as a main memory in the form of random access memory (RAM), one or more hard drives, and/or one or more drives that handle removable media such as compact disks (CD), flash memory cards, digital video disk (DVD), and the like. The computer system <NUM> may also include a display device, one or more speakers, and a keyboard and/or controller, which can include a mouse, trackball, touch screen, voice-recognition device, or any other device that permits a system user to input information into and receive information from the computer system <NUM>.

Depending on certain implementation requirements, embodiments can be implemented in hardware or in software.

Some embodiments comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

Generally, embodiments can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer.

In other words, an embodiment is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

A further embodiment is, therefore, a storage medium (or a data carrier, or a computer-readable medium) comprising, stored thereon, the computer program for performing one of the methods described herein when it is performed by a processor. A further embodiment is an apparatus as described herein comprising a processor and the storage medium.

A further embodiment is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.

A further embodiment comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver.

Claim 1:
An optical system (<NUM>) comprising:
a first sensor (401a) configured to generate a first image in a first wavelength range;
a second sensor (401b) configured to generate a second image in a second wavelength range;
an autofocus system (<NUM>, <NUM>) configured to:
receive a first signal (102a, 402a) corresponding to the first wavelength range;
receive a second signal (102b, 402b) corresponding to the second wavelength range;
receive information on an operation mode (<NUM>, 404b), the operation mode indicating a use of the first wavelength range or the second wavelength range; and
use the information on the operation mode (<NUM>, 404b) to determine an output signal (<NUM>, <NUM>) comprising a focus setting information, the focus setting information using the first signal (102a, 402a) and the second signal (102b, 402b);
a controller (<NUM>) configured to control a focus system of the optical system (<NUM>) using the output signal (<NUM>, <NUM>); and characterized by
an imaging system (<NUM>) configured to combine the first image and the second image into an observable image.