Patent Description:
Most specimens that are observed with a microscope have small variations in height across their surfaces. While these variations are frequently not visible to the human eye, they can cause images of a portion of a specimen captured by a microscope to be out of focus.

The range in which a microscope can create a usable focused image is known as the depth of field. The microscope must keep a portion of a specimen within its depth of field to generate useful images. However, when transitioning from observing a first portion of a specimen to observing a second portion of the specimen, the small variations in height of the specimen may cause the second portion to be outside the depth of field.

Different sharpness measurements such as image contrast, resolution, entropy and/or spatial frequency content, among others, can be used to measure the quality of focus of images captured by a microscope. Generally, when a specimen is in focus, the captured image will exhibit the best sharpness quality (e.g., large contrast, a high range of intensity values and sharp edges). The different sharpness measurements that can be used to determine when a specimen is in focus usually require capturing a series of images and increasing or decreasing the distance between the microscope objective lens and the specimen until the image appears in focus. This increases the total microscopic scan time of each specimen, making methods using such measurement prohibitively slow for high throughput scanning applications.

<CIT> discloses an automatic focusing device which performs follow up focusing operation while transferring an observation object and a processing object, preliminarily restricts distance between an objective lens and the objects within a certain range, and shortens distance for performing the final focusing operation after standstill to shorten focal time. The automatic focusing device has a projection means for projecting two pin holes with different distance in the optical axis direction and in the direction perpendicular to the optical axis direction on a flat surface of an object to be measured via an objective lens and an operation means for calculating difference between focal positions of projected images on the side of the object to be measured based on level difference of the two pin holes and magnification of the objective lens.

Accordingly, it is desirable to find a suitable in-focus plane of a specimen using a smaller number of images.

Systems, methods, and media for automatic microscopic focus are provided.

In some embodiments, systems for automatically focusing a microscope are provided, the systems comprising: an objective; a stage for positioning a specimen on a first image forming conjugate plane; a first camera, configured for focusing, positioned on a second image forming conjugate plane; a second camera, configured for focusing, positioned at an offset distance from the second image forming conjugate plane; a primary illumination source that emits light in a first wavelength range, wherein the emitted light is received by the first camera; a secondary illumination source that emits light in a second wavelength range which is different from the first wavelength range and that projects light through a focusing pattern that is positioned on a third image forming conjugate plane, wherein the projected light is received by the second camera; and a hardware processor coupled to the first camera and the second camera that is configured to: determine, using the first camera, when a specimen is in focus based on a sharpness value; determine, using the second camera, a sharpness setpoint for the specimen when the specimen is determined to be in focus for the first camera; after movement of the specimen, determine, using the second camera, a first sharpness value of the specimen; determine whether the first sharpness value of the specimen is higher or lower than the sharpness setpoint; and adjust a distance between the objective and the stage so that a second sharpness value of the specimen determined using the second camera corresponds to the sharpness setpoint.

In some embodiments, systems for automatically focusing a microscope are provided, the systems comprising: an objective; a stage for positioning a specimen on a first image forming conjugate plane; a first camera, configured for taking images of the specimen when the specimen is determined to be in focus, positioned on a second image forming conjugate plane; a second camera, configured for focusing, positioned on a third image forming conjugate plane; a third camera, configured for focusing, positioned at an offset distance from the third image forming conjugate plane; a primary illumination source that emits light in a first wavelength range, wherein the emitted light is received by the first camera; a secondary illumination source that emits light in a second wavelength range which is different from the first wavelength range and that projects light through a focusing pattern that is positioned on a fourth image forming conjugate plane, wherein the projected light is received by the second and third camera; and a hardware processor coupled to the second camera and the third camera that is configured to: determine, using the second camera, when a specimen is in focus based on a sharpness value; determine, using the third camera, a sharpness setpoint for the specimen when the specimen is determined to be in focus for the second camera; after movement of the specimen, determine, using the third camera, a first sharpness value of the specimen; determine whether the first sharpness value of the specimen is higher or lower than the sharpness setpoint; and adjust a distance between the objective and the stage so that a second sharpness value of the specimen determined using the third camera corresponds to the sharpness setpoint.

In some embodiments, methods for automatically focusing a microscope are provided, the methods comprising: positioning a specimen on a stage on a first image forming conjugate plane; positioning a first camera, configured for focusing, on a second image forming conjugate plane; positioning a second camera, configured for focusing, at an offset distance from the second image forming conjugate plane; emitting, from a primary illumination source, light in a first wavelength range, wherein the emitted light is received by the first camera; emitting, from a secondary illumination source, light in a second wavelength range which is different from the first wavelength range, wherein the light is projected through a focusing pattern that is positioned on a third image forming conjugate plane, wherein the projected light is received by the second camera; determining, by a hardware processor using the first camera, when a specimen is in focus based on a sharpness value; determining, by the hardware processor using the second camera, a sharpness setpoint for the specimen when the specimen is determined to be in focus for the first camera; after movement of the specimen, determining, by the hardware processor using the second camera, a first sharpness value of the specimen; determining, by the hardware processor, whether the first sharpness value of the specimen is higher or lower than the sharpness setpoint; and adjusting, by the hardware processor, a distance between an objective and the stage so that a second sharpness value of the specimen determined using the second camera corresponds to the sharpness setpoint.

In some embodiments, methods for automatically focusing a microscope are provided, the methods comprising: positioning a specimen on a stage on a first image forming conjugate plane; positioning a first camera, configured for taking images of the specimen when the specimen is determined to be in focus, on a second image forming conjugate plane; positioning a second camera, configured for focusing, on a third image forming conjugate plane; positioning a third camera, configured for focusing, at an offset distance from the third image forming conjugate plane; emitting, from a primary illumination source, light in a first wavelength range, wherein the emitted light is received by the first camera; emitting, from a secondary illumination source, light in a second wavelength range which is different from the first wavelength range, wherein the light is projected through a focusing pattern that is positioned on a fourth image forming conjugate plane, wherein the projected light is received by the second and third camera; determining, by a hardware processor using the second camera, when a specimen is in focus based on a sharpness value; determining, by the hardware processor using the third camera, a sharpness setpoint for the specimen when the specimen is determined to be in focus for the second camera; after movement of the specimen, determining, by the hardware processor using the third camera, a first sharpness value of the specimen; determining, by the hardware processor, whether the first sharpness value of the specimen is higher or lower than the sharpness setpoint; and adjusting, by the hardware processor, a distance between the objective and the stage so that a second sharpness value of the specimen determined using the third camera corresponds to the sharpness setpoint.

In accordance with some embodiments of the disclosed subject matter, mechanisms (which can include systems, methods, devices, apparatuses, etc.) for automatic microscopic focus of specimens are provided.

<FIG> illustrates an example <NUM> of an automatic focus system according to some embodiments of the disclosed subject matter. At a high level, the basic components of automatic focus system <NUM>, according to some embodiments, include an illumination unit <NUM> for providing light, a focusing unit <NUM> for finding the in-focus plane of a specimen, a vertical illuminator <NUM>, an imaging camera <NUM>, an objective <NUM>, a stage <NUM>, and a control system <NUM> comprising hardware, software, and/or firmware.

Automatic focus system <NUM> can be implemented as part of any suitable type of microscope. For example, in some embodiments, system <NUM> can be implemented as part of an optical microscope that uses transmitted light or reflected light. More particularly, system <NUM> can be implemented as part of the nSpec® optical microscope available from Nanotronics Imaging, Inc. of Cuyahoga Falls, OH. Although the following description refers to a reflected light vertical illuminator <NUM>, the mechanisms described herein can be a part of microscopes that do not use a reflected light vertical illuminator.

According to some embodiments, the microscope can include, one or more objectives <NUM>. The objectives can have different magnification powers and/or be configured to operate with brightfield/darkfield microscopy, differential interference contrast (DIC) microscopy and/or any other suitable form of microscopy. The objective and/or microscopy technique used to inspect a specimen can be controlled by software, hardware, and/or firmware in some embodiments.

In some embodiments, a fine focus actuator <NUM> can be used to drive objective <NUM> in a Z direction towards and away from stage <NUM>. Fine focus actuator <NUM> can be designed for high precision and fine focus adjustment of objective <NUM>. Fine focus actuator <NUM> can be a stepper motor, servo motor, linear actuator, piezo motor, and/or any other suitable mechanism. For example, in some embodiments, a piezo motor can be used and can drive the objective <NUM> to <NUM> micrometers (µm), <NUM> to <NUM>, or <NUM> to <NUM>, and/or any other suitable range(s) of distances.

In some embodiments, an XY translation stage can be used for stage <NUM>. The XY translation stage can be driven by stepper motor, servo motor, linear motor, and/or any other suitable mechanism.

In some embodiments, focus unit <NUM>, comprising actuator <NUM>, can be used to adjust stage <NUM> in a Z direction towards and away from objective <NUM>. Actuator <NUM> can be used to make coarse focus adjustments of, for example, <NUM> to <NUM>, <NUM> to <NUM>, <NUM> to <NUM>, and/or any other suitable range(s) of distances. Actuator <NUM> can also be used to move stage <NUM> up and down to allow specimens of different thicknesses to be placed on the stage. Actuator <NUM> can also be used in some embodiments to provide fine focus of, for example, <NUM> to <NUM>, <NUM> to <NUM>, <NUM> to <NUM>, and/or any other suitable range(s) of distances. In some embodiments, focusing unit <NUM> can also include a location device <NUM>. The location device can be configured to store an absolute position of stage <NUM> (e.g., the position of the stage when a specimen is in focus), even upon reset and/or power cycling of autofocus system <NUM>. In some embodiments, the location device can be a linear encoder, a rotary encoder or any other suitable mechanism to track the absolute position of stage <NUM> with respect to the objective.

In some embodiments, automatic focus system <NUM>, when it is properly focused and aligned, can include a set of conjugate focal planes, for example an image-forming conjugate set, that occur along the optical pathway through the microscope. Each plane within the image-forming conjugate set is conjugate with the others in that set because the planes are simultaneously in focus and can be viewed superimposed upon one another when observing specimens through the microscope. The set of image-forming conjugate planes in automatic focus system <NUM> can include a primary focusing camera <NUM> image plane, an imaging camera <NUM> image plane, a focusing pattern <NUM> image plane, a field diaphragm (F-stop) <NUM> image plane and a specimen image plane. More specifically, all references herein to positioning a primary focusing camera <NUM> and imaging camera <NUM> on the image-forming conjugate planes refer to positioning the sensors within cameras <NUM> and <NUM> on the image-forming conjugate planes.

In some embodiments, focusing pattern <NUM> can be formed from opaque material, with a pattern cut out of the material. The cutout section of the material allows light to pass through to the specimen image plane, while the opaque material section blocks light from passing through. In other embodiments, focusing pattern <NUM> can be formed from clear material such as clear glass or clear plastic that has an opaque pattern thereon which causes an image to be projected on the specimen image plane by light passing through the clear glass or plastic.

In some embodiments, imaging camera <NUM> can include an image sensor <NUM> that is positioned on an image-forming conjugate plane of automatic focus system <NUM>. Imaging camera <NUM> can be used to capture images of a specimen once control system <NUM> determines that the specimen is in focus. Image sensor <NUM> can be, for example, a CCD, CMOS and/or any other suitable electronic device that allows images of a specimen to be captured and stored.

In some embodiments, control system <NUM>, comprising controller <NUM> and controller interface <NUM>, can control any settings of the components of automatic focus system <NUM> (e.g., actuators <NUM> and <NUM>, primary illumination source <NUM>, secondary illumination source <NUM>, focusing cameras <NUM> and <NUM>, stage <NUM>, focusing pattern <NUM>, imaging camera <NUM> and objective <NUM>), as well as communications, operations (e.g., taking images, turning on and off an illumination source, moving stage <NUM> and objective <NUM>, storing different values associated with a specimen) and calculations (e.g., sharpness calculations) performed by, and between, components of the automatic focus system. Control system <NUM> can include any suitable hardware (which can execute software in some embodiments), such as, for example, computers, microprocessors, microcontrollers, application specific integrated circuits (ASICs), and digital signal processors (any of which can be referred to as a hardware processor), encoders, circuitry to read encoders, memory devices (including one or more EPROMS, one or more EEPROMs, dynamic random access memory ("DRAM"), static random access memory ("SRAM"), and/or flash memory), and/or any other suitable hardware elements. In some embodiments, individual components within automatic focus system <NUM> can include their own software, firmware, and/or hardware to control the individual components and communicate with other components in automatic focus system <NUM>.

In some embodiments, communication <NUM> between the control system (e.g., controller <NUM> and controller interface <NUM>) and the components of automatic focus system <NUM> can use analog technologies (e.g., relay logic), digital technologies (e.g., using RS232, ethernet, or wireless) and/or any other suitable communication technologies.

In some embodiments, operator inputs can be communicated to the control system using any suitable input device (e.g., a keyboard, mouse or joystick).

<FIG> shows the general configuration of an embodiment of an illumination unit of the automatic focus system, in accordance with some embodiments of the disclosed subject matter. The illumination unit <NUM> can include two illumination sources, for example a primary illumination source <NUM> and a secondary illumination source <NUM>. The illumination sources can provide light beams in ranges of wavelengths that are different from each other.

In some embodiments, for example, primary illumination source <NUM> provides a light beam having a wavelength in the range of <NUM> to <NUM> nanometers (nm), while the secondary illumination source <NUM> provides a light beam having a wavelength that is higher or lower than the range of wavelengths used for the primary source. For example, the wavelength range of the primary illumination source <NUM> can be in the range of <NUM> to <NUM> and the wavelength range of the secondary illumination source can be in the range of <NUM> to <NUM>. Light of any wavelength range can be used for primary illumination source <NUM> as long as the value of the range is known and can be separated from other wavelengths using known filtering techniques. Similarly, light of any wavelength range can be used for secondary illumination source <NUM>, as long as the light is not in the same wavelength range as primary illumination source <NUM>.

In some embodiments, as shown in <FIG>, primary illumination source <NUM> is positioned so that its light is transmitted in a horizontal direction towards vertical illuminator <NUM>. Primary illumination source <NUM> can include a focusing lens <NUM> (e.g., a double convex lens) for focusing the primary light beam. The secondary illumination source <NUM> can be positioned at a suitable distance below focusing pattern <NUM> located on image-forming conjugate plane <NUM>. The diameter of focusing pattern <NUM> (e.g., <NUM>) can be adjusted so that a projection of focusing pattern <NUM> is smaller than the field of view (FOV) of focusing cameras <NUM> and <NUM>. Focusing pattern <NUM> can be any suitable geometric shape for example, a circle, rectangle, triangle, or hexagon. Focusing pattern <NUM> can also include a series of discrete openings, so that when light is transmitted through the discrete openings, the lines and spaces are projected across the field of view. In some embodiments, the location of primary illumination source <NUM> and secondary illumination source <NUM> can be switched.

In some embodiments, automatic focus system <NUM> can be configured so that light from secondary illumination source <NUM> is continuously transmitted through focusing pattern <NUM> in order to continuously project the focusing pattern image on a specimen that can be captured by focusing cameras <NUM> and <NUM>. The continuous projection of the focusing pattern image can facilitate sharpness focus of a specimen, especially for transparent specimens or for specimens that lack any visually recognizable features. Focusing pattern <NUM> can be used instead of, or in addition to, a field diaphragm, for sharpness focusing. For example, automatic focus system <NUM>, in addition to focusing pattern <NUM>, can also include a field diaphragm (F-stop) <NUM> that can be located in the vertical illuminator <NUM>. Field diaphragm <NUM> can also be positioned on an image-forming conjugate plane of automatic focus system <NUM>. In some embodiments, field diaphragm <NUM> controls the diameter of light emitted by illumination source <NUM> and <NUM> and transmitted to objective <NUM>. More specifically, in some embodiments, by reducing the size of the field diaphragm, the diameter of the light passing through is reduced. This creates a dark outline around the image of the specimen received by focusing cameras <NUM> and <NUM> and can be used to adjust the focus of the specimen (e.g., by moving the specimen and objective closer together or farther apart). At the point of greatest measured sharpness, the specimen is considered to be in-focus and the field diaphragm can be opened to a larger size to allow imaging of the specimen by imaging camera <NUM>. Reducing the field diaphragm and returning it to its original size, however, takes time (e.g., <NUM>-<NUM> seconds) and can slow down the scanning process and throughput.

Focusing pattern <NUM> can be positioned on an any suitable image-forming conjugate plane of automatic focus system <NUM> (e.g., above secondary illumination source <NUM> (as shown in <FIG>), or at field diaphragm <NUM>), as long as an appropriate filter is used, when necessary, to make sure that focusing pattern <NUM> is not projected onto imaging camera <NUM>. For example, if focusing pattern <NUM> is positioned on the field diaphragm <NUM> image forming conjugate plane, in place of field diaphragm <NUM>, then a filter would be necessary. In some embodiments, a band filter can be located on the field diaphragm image forming conjugate plane (in place of field diaphragm <NUM>) and a focusing pattern in the form of a pattern cutout can be created in the band filter. More specifically, a band filter can be selected that transmits light in the same wavelength range of primary illumination source <NUM> (e.g., greater than <NUM>) and blocks light in the same wavelength range of secondary illumination source <NUM> (e.g., less than or equal to <NUM>), except in the focusing pattern <NUM> region. In other words, light in the same wavelength range of secondary illumination <NUM> source would be blocked except in the region of focusing pattern <NUM>, which would allow the light from secondary illumination <NUM> to be transmitted through to focusing cameras <NUM> and <NUM>. As described below, in connection with <FIG>, optical filter <NUM> ensures that only light from the primary illumination source <NUM> is transmitted to imaging camera <NUM>.

Note that, in some embodiments, any suitable illumination source can be used with illumination unit <NUM>, such as a <NUM> ultraviolet collimated light-emitting diode (LED) for secondary illumination source <NUM> and a <NUM> white light collimated LED for primary illumination source <NUM>.

In some embodiments focusing lens <NUM> (e.g., a <NUM> focal length bioconvex lens) can be placed at a suitable distance between the secondary illumination source <NUM> and focusing pattern <NUM>. Further, another focusing lens <NUM> can be placed at a suitable distance on the other side of focusing pattern <NUM>. In some embodiments, the distance of the lenses <NUM> and <NUM> from focusing pattern <NUM> can be based on the optical characteristics of the microscope to ensure the focusing of the light and positioning of focusing pattern <NUM> to be in a conjugate image-forming plane.

In some embodiments, a dichroic <NUM> is placed in the optical pathway of both primary illumination source <NUM> and secondary illumination source <NUM> before the light travels to vertical illuminator <NUM>. Dichroic, as used herein, can refer to mirrors, beam splitters, filters or beam combiners that transmits light of a known, specified wavelength and combines the with a light of another known, specified wavelength. Note that a combination of the aforementioned devices can be used to reflect and transmit the desired illumination sources and wavelengths. In some embodiments, a dichroic having a specific cut-off wavelength is selected in order to reflect the wavelengths of light emitted by secondary illumination source <NUM> and to allow the wavelengths of light emitted from primary illumination source <NUM> to pass through. For example, if secondary illumination source <NUM> emits light in a wavelength range of <NUM>-<NUM> and primary illumination source <NUM> emits light in a wavelength range of <NUM>-<NUM>, then a <NUM> cutoff dichroic (i.e., a dichroic that reflects light with a wavelength of <NUM> and below and allows light with a wavelength greater than <NUM> to pass through thereby combining the beams) can be used to reflect light from secondary illumination source <NUM> and to allow light from primary illumination source <NUM> to pass through. Dichroic <NUM> can be designed for a <NUM>° angle of incidence, so that rejected light from secondary illumination source <NUM> is reflected at an angle of <NUM>° and travels parallel to the light path from primary illumination source <NUM>.

In some embodiments, primary illumination source <NUM> can be the light source used for imaging a specimen on imaging sensor <NUM> in imaging camera <NUM> and secondary illumination source <NUM> can be the light source used for imaging a specimen on focusing sensors <NUM> and <NUM> of focusing cameras <NUM> and <NUM>.

Note that, in some embodiments any suitable dichroic, illuminator, illumination source, focusing lens, sensor and focusing pattern can be used with illuminating unit <NUM>. In some embodiments, any suitable arrangement of these components can be used with illuminating unit <NUM>. In some embodiments, the components of illuminating unit <NUM> can be mounted to illuminator <NUM> in any suitable manner, such as by using guide rods in a similar manner to how focusing camera <NUM> is shown as being mounted to focusing housing <NUM> in <FIG> (described below), in order to allow variable geometry.

<FIG> shows an example of a general configuration of an embodiment of a focusing unit of the automatic focus system, in accordance with some embodiments of the disclosed subject matter. The focusing unit <NUM> can include two cameras: a primary focusing camera <NUM> and an offset focusing camera <NUM>. These cameras can include, for example, a charged coupled device (CCD) image sensor, a CMOS image sensor and/or any other suitable image sensor that allows images of a specimen to be captured. In some embodiments, the captured image is stored and analyzed by control system <NUM>.

The focusing unit <NUM> can be mounted in an area between vertical illuminator <NUM> and imaging camera lens tube <NUM>. This area is known as infinity space. In some embodiments, the focusing unit <NUM> can be mounted in other locations using appropriate components to adapt the selected location to the optical characteristics of the system.

Primary focusing camera <NUM> can include a sensor <NUM> that is positioned on an image-forming conjugate plane of automatic focus system <NUM> (as represented for example by line <NUM>).

An offset focusing camera <NUM> can include a sensor <NUM> that can be positioned at an offset to image-forming conjugate plane <NUM>. The offset can be either be in the positive direction <NUM> or the negative direction <NUM>. Offset focusing camera <NUM> can be located above or below primary focusing camera <NUM>. Offset focusing camera <NUM> can be movable along guide rods <NUM> or any other suitable structure in order to adjust an offset distance of offset camera <NUM>. The offset distance can be adjusted based on the calculated sharpness curves for offset focusing camera <NUM> at different distances from the image-forming conjugate plane <NUM>, as discussed below in connection with <FIG>.

The focusing unit <NUM> can also include two focusing lenses <NUM> and <NUM>. Focusing lens <NUM> can be placed in the same horizontal optical pathway as primary focusing camera <NUM> and focusing lens <NUM> can be placed in the same horizontal optical pathway as offset focusing camera <NUM>. In some embodiments, focusing lenses <NUM> and <NUM> achieve the same focal distance as microscope tube lens <NUM>, to ensure that sensors <NUM> and <NUM> are each in focus when they are positioned on the image-forming conjugate plane <NUM>. Microscope tube lens <NUM> can include a lens (not shown) for focusing an image of a specimen on sensor <NUM>, so that the specimen is in focus when sensor <NUM> is positioned on an image-forming conjugate plane of automatic focus system <NUM>.

Note that in some embodiments, lenses <NUM> and <NUM> can be double convex lenses or any other suitable type lenses. In some embodiments, the focal length of the lenses can be based on the optical characteristics of the microscope.

As also shown in <FIG>, focusing unit <NUM> can also include a cutoff dichroic <NUM> that is positioned above vertical illuminator <NUM> in the optical pathway of the light reflected off a specimen. The dichroic <NUM> is positioned so that the light reflected off the specimen that is below the cutoff of the dichroic is reflected at an angle of <NUM>° towards primary focusing camera <NUM>. A dichroic having a specific cut-off wavelength can be selected in order to reflect the wavelengths of light emitted by secondary illumination source <NUM> (the "focusing beam"). For example, if the focusing beam is in the range of <NUM> to <NUM>, then a <NUM> cut-off filter can be used with focusing unit <NUM> in order to reflect the focusing beam towards primary focusing camera <NUM>.

In some embodiments, focusing unit <NUM> can include a beam splitter <NUM> that can be positioned between dichroic <NUM> and primary focusing camera <NUM>. The beam splitter <NUM> can be, for example, a <NUM>/<NUM> beam splitter designed to send <NUM>% of the focusing light beam to primary focusing camera <NUM> and <NUM>% of the focusing light beam to offset focusing camera <NUM>. A mirror <NUM> can be placed at a distance directly above beam splitter <NUM> and can be designed to direct the beam of light from beam splitter <NUM> to offset focusing camera <NUM>.

In some embodiments, a cut-off filter <NUM> can be positioned between dichroic <NUM> and beam splitter <NUM> to filter out any light coming from primary illumination source <NUM> (the "imaging beam"). For example, if imaging beam has a wavelength in the range of <NUM> and above, then a <NUM> cutoff filter can be used to filter out the imaging beam and prevent the imaging beam from transmitting light to focusing cameras <NUM> and <NUM>. In other embodiments, two cut-off filters can be used and each filter can be placed, for example, in front of or behind lenses <NUM> and <NUM>.

Note that, in some embodiments any suitable dichroic, focusing camera, focusing lens, mirror, image sensor, beam splitter and cut-off filter can be used with focusing unit <NUM>. In some embodiments, any suitable arrangement of these components can be used with focusing unit <NUM>. The components of focusing unit <NUM> can be mounted to guide rods or any other suitable structure for connecting the components. Further, in some embodiments, primary focusing camera <NUM> is not necessary and the focusing operations described herein for primary focusing camera <NUM>, can instead be performed by imaging camera <NUM>.

<FIG> shows example optical pathways, represented by a pair of dashed lines, for automatic focus system <NUM>, in accordance with some embodiments of the disclosed subject matter. Automatic focus system <NUM> can be configured so that the light emitted from secondary illumination source <NUM> (the "focusing beam (FB)," as represented by the shorter dashed lines) is projected onto specimen S and then reflected to focusing cameras <NUM> and <NUM>. Autofocus system <NUM> can also be configured so that light emitted from primary illumination source <NUM> (the "imaging beam (IB)," as represented by the longer dashed lines) is projected onto specimen S and then reflected to imaging camera <NUM>.

More specifically, in some embodiments, the focusing beam can travel from illumination source <NUM> through focusing pattern <NUM> to dichroic <NUM>. Dichroic <NUM> can reflect the focusing beam towards vertical illuminator <NUM>.

The imaging beam can travel from primary illumination source <NUM>, pass through dichroic <NUM> to combine with the focusing beam.

The combined beam can then travel through vertical illuminator <NUM> to prism <NUM>. Prism <NUM> can reflect the light coming from the illumination sources at <NUM>° downwards through a nosepiece and objective <NUM> to a specimen S. Specimen S can reflect the combined beam upwards through objective <NUM>, which is then transmitted through prism <NUM> towards dichroic <NUM>. Dichroic <NUM> can separate the transmitted beam back into the imaging beam and focusing beam by, for example, reflecting the wavelengths of the focusing beam towards focusing cameras <NUM> and <NUM> and by allowing the wavelengths of the imaging beam to pass through towards camera <NUM>.

In some embodiments, the focusing beam that is reflected by dichroic <NUM> can pass through cutoff filter <NUM> to remove any light above the cutoff wavelength. The focusing beam can then travel to beam splitter <NUM>. Beam splitter <NUM> can send <NUM>% of the focusing beam towards primary focusing camera <NUM> by directing the light through focusing lens <NUM> located in focusing housing <NUM>. From there the focusing beam, can travel to a light sensor <NUM> (<FIG>) in camera <NUM>. The other <NUM>% of the focusing beam can be directed by beam splitter <NUM> upwards towards mirror <NUM>. Mirror <NUM> can reflect the focusing beam towards focusing lens <NUM> locating in focusing housing <NUM>. From there the focusing beam can be directed to sensor <NUM> (<FIG>) in offset camera <NUM>.

In some embodiments, the imaging beam that passes through dichroic <NUM> can pass through an optical filter <NUM> (e.g., a filter that transmits only the wavelengths from the imaging beam), up through tube lens <NUM>, and to camera sensor <NUM> located in imaging camera <NUM>.

In some embodiments, primary focusing camera <NUM> can be used to determine the in-focus point of a specimen. The focus of a specimen can be adjusted, for example, by moving the objective and stage closer together or farther apart along a Z axis (as shown in <FIG>). More particularly, primary focusing camera <NUM> can be used to obtain an image of a specimen at two or more Z positions (e.g., by moving stage <NUM> and/or objective <NUM> in a Z direction). From the resulting images, a relative sharpness value can be calculated by control system <NUM> for each Z position of the specimen to determine the quality of focus. Automatic focus system <NUM> can use any suitable sharpness equation to calculate the relative sharpness of the resulting images. One example equation that can be used by automatic focus system <NUM> to calculate a relative sharpness score is a measure of image variance V, normalized by the mean µ to account for intensity fluctuations: <MAT> where s(i,j) is the grayscale pixel value at coordinates (i,j) and N and M represent the number of pixels in the i and j directions respectively. Other example methods for calculating a relative sharpness value that can be used by automatic focus system <NUM> are described by Sivash <NPL>), which is hereby incorporated by reference herein in its entirety. The above disclosed methods are just examples and are not intended to be limiting.

<FIG> shows a graph comprising an X axis that represents the relative position of a specimen in a Z direction (the "Z position") and a Y axis representing a relative sharpness score. The relative Z position represents the distance between the top of stage <NUM> and the objective <NUM>. The Z position can be changed either by adjusting a stage <NUM> towards or away from objective <NUM> and/or by adjusting objective <NUM> towards or away from stage <NUM>. The sharpness curve shown in <FIG> compares, at each measurement point along the curve, the relative sharpness of an image captured by primary focusing camera <NUM> with the relative Z position. As shown in <FIG>, the sharpness value for a specimen can have a largest measured sharpness (e.g., sharpness score of <NUM> in <FIG>) at a given relative position (e.g., Z position <NUM>) (that can be referred to as the in-focus position) and may decrease symmetrically on each side of the in-focus position (e.g., Z position <NUM>). In some instances, the slope of the curve in <FIG> at the in-focus position can be zero or close to zero. It should be understood that the term "in focus" as used herein is intended to denote when the relative positioning of the objective and the stage are such that a sharpness measurement is at a point at or near the top of a sharpness curve. The term "in focus" is not intended to be limited to perfect or optimal focus.

The range of coarse Z movement is represented by lines <NUM> (e.g., at <NUM>) and <NUM> (e.g., at <NUM>). The range of fine focus Z movement is represented by lines <NUM> (e.g., at <NUM>) and <NUM> (e.g., at <NUM>). Note, that the range of Z movement refers to a practical range of movement to achieve different Z positions between objective <NUM> and stage <NUM>. The range of Z movement also refers to the range of Z movement where a sharpness calculation can be used to focus a specimen. Arrow <NUM> shows the sharpness score increasing to a maximum point at Z position <NUM> (indicating that the image is considered to be in focus as described above) as stage <NUM> and objective <NUM> move farther apart and arrow <NUM> shows the sharpness score decrease from maximum point at Z position <NUM> as stage <NUM> and objective <NUM> continue to move farther apart.

<FIG> shows an example sharpness curve for offset focusing camera <NUM>. Similar to <FIG>, the X axis of the graph represents the relative Z position, the Y axis represents a relative sharpness score and line <NUM> indicates the Z position where the maximum measured sharpness value for primary focusing camera <NUM> is found. The sharpness curve shown in <FIG> compares, at each point along the curve, the relative sharpness of an image captured by offset focusing camera <NUM> with the relative Z position. In some embodiments, automatic focus system <NUM> can use the same equation to calculate the sharpness curve for primary focusing camera <NUM> and offset focusing camera <NUM>.

As shown in the example of <FIG>, at the in-focus position (e.g., Z position <NUM>) determined using primary focusing camera <NUM> as described above in connection with <FIG>, the relative sharpness value for an image captured by offset focusing camera <NUM> is around <NUM> (as indicated by arrow <NUM>). This value (e.g., <NUM>) can be stored by control system <NUM> as the sharpness setpoint for offset camera <NUM> for that particular specimen, specimen class and/or any other suitable classification group. In some embodiments, a specimen class can be defined based on specimens made from materials of similar reflective qualities. Some example specimen classes can include, but are not limited to: a bare silicon wafer; a semiconductor wafer with a known pattern; and a biological specimen of the same known substances prepared consistently with a glass slide and cover slip.

As shown in <FIG>, the sharpness curve for the images taken by offset focusing camera <NUM> is constantly increasing (as represented by arrow <NUM>) between lines <NUM> and <NUM> (representing the range of Z movement). Once the sharpness setpoint for a specimen or a class of specimens is found, then offset focusing camera <NUM> can be used to determine whether to move stage <NUM> and objective <NUM> closer together or farther apart. For example, if the sharpness setpoint of a specimen is determined to be <NUM>, and stage <NUM> is translated in an X/Y plane perpendicular to the Z axis such that the specimen is no longer in focus, then the sharpness value of an image captured by the offset focusing camera <NUM> can be used, together with the sharpness setpoint and the sharpness curve, to bring the specimen back in focus. For example, if the sharpness setpoint for the image of the specimen, as captured by offset focusing camera <NUM>, is <NUM> as described above and a relative sharpness value of an image of the specimen, post stage translation, as captured by offset focusing camera <NUM> is <NUM> (for example), then, as evident from the sharpness curve in <FIG>, the distance between the stage and objective must be decreased (e.g., from <NUM> to <NUM>) to bring the specimen back into focus. On the other hand, if the relative sharpness value for the image of the specimen as captured by offset focusing camera <NUM> is less than <NUM> (e.g., at <NUM>), then, as evident from the sharpness curve in <FIG>, the distance between the stage and objective must be increased (e.g., from <NUM> to <NUM>) to bring the specimen back into focus. Since the relative sharpness curve of offset focusing camera <NUM>, together with the sharpness setpoint, indicates whether the distance between the stage and objective must be decreased or increased, fewer images of the specimen can be taken to bring a specimen back into focus.

This same information about whether to move the stage and objective closer together or farther apart cannot be gleaned from the sharpness curve shown in <FIG>. For example, as shown in <FIG>, if the largest measured sharpness value of a specimen, as captured by primary focusing camera <NUM>, is <NUM>, and the actual sharpness value of the specimen is measured to be <NUM>, then the sharpness curve shows that the relative Z position can be either to the right or the left of the in-focus point at Z position <NUM>. Because the relative Z position can be either to the right or the left of the in-focus point at Z position <NUM>, the sharpness curve cannot be used to determine whether to move the stage and objective closer together or farther apart.

<FIG> shows different example sharpness curves (i.e., sharpness curves A, B and C) for offset focusing camera <NUM> when it is positioned at different distances from image-forming conjugate plane <NUM> (as shown in <FIG>). Similar to <FIG>, the X axis of the graph in <FIG> represents the relative Z position, the Y axis represents a relative sharpness score and line <NUM> indicates the Z position where the maximum measured sharpness value for primary focusing camera <NUM> is found. Sharpness curves A, B and C compare, at each point along the curves, the relative sharpness of an image captured by offset focusing camera <NUM> with the relative Z position. Sharpness curve D compares, at each point along the curve, the relative sharpness of an image captured by primary focusing camera <NUM> with the relative Z position. In some embodiments, automatic focus system <NUM> uses the same sharpness equation to calculate sharpness curves A, B, C and D. Note that the positioning of offset focusing camera <NUM> herein refers to the positioning of sensor <NUM> in focusing camera <NUM>.

Out of the three sharpness curves, curve C represents the sharpness curve for offset focusing camera <NUM> when it is closest to image-forming conjugate plane <NUM>. Curve B represents a sharpness curve for offset focusing camera <NUM> when the offset camera is at a distance farther than its offset distance for curve C, but closer than its offset distance for curve A. Curve A represents a sharpness curve for offset focusing camera <NUM> when the offset camera is farthest away from image-forming conjugate plane <NUM>. The offset distance refers to the distance between sensor <NUM> of offset focusing camera <NUM> and image-forming conjugate plane <NUM>.

Lines A', B' and C' represent the slopes of the respective curves A, B and C at the Z position (e.g., line <NUM>) where the specimen is considered to be in focus (as described above in connection with <FIG>) for primary focusing camera <NUM>. The slopes of the curves become steeper as offset focusing camera <NUM> moves closer to image-forming conjugate plane <NUM>. A steeper slope, represents a larger change in sharpness versus a smaller change in Z height (also referred to as greater resolution). A steeper slope is desirable because it allows for finer focal adjustment and control.

In some embodiments, the range of Z movement necessary to bring a specimen in focus can determine the offset distance. The range of Z movement can be based on, for example: the thickness and/or any other suitable characteristic of a specimen; the specimen class and/or any other suitable grouping of the specimen; and/or the optical characteristics of the microscope (e.g., the magnification of the objective). The range of Z movement can also be chosen to encompass a wide range of specimen types to prevent having to constantly adjust the offset distance.

In some embodiments, to determine the appropriate offset distance for automatic focus system <NUM>, offset focusing camera <NUM> can be positioned at different offset distances. A sharpness curve can be calculated at each offset distance. The offset distance that produces a sharpness curve that represents a desired range of Z movement and is constantly increasing (represented by a positive slope) or decreasing (represented by a negative slope) in that range, can be selected.

More specifically, if the range of Z movement is large (e.g., the distance between lines <NUM> and <NUM> shown in <FIG>), then offset focusing camera <NUM> can be positioned farther away from image-forming conjugate plane <NUM> (as represented by curve A). For example, if the desired range of Z movement is between <NUM> and <NUM> (as represented by lines <NUM> and <NUM>), then the offset camera should not be positioned at the offset distance that produced curve B or curve C, because curves B and curves C both increase and decrease in the desired range of Z movement and cannot be used to determine whether to move the specimen and objective closer together or farther apart.

In some embodiments, the offset distance can also be based on the steepness of the sharpness curve at the position where the specimen is in optimum focus for primary focusing camera <NUM> (e.g., as represented by lines A', B' and C'). For example, if the range of Z movement necessary to bring a specimen in focus is small (e.g., between <NUM> - <NUM>), then offset focusing camera <NUM> can be placed closer to image-forming conjugate plane <NUM> (e.g., at the offset distance that produced curve C). Even though a larger offset distance can produce acceptable sharpness curves (e.g., curves B and A), the position of offset focusing camera <NUM> that produces curve C can be selected because that position has the steepest slope and greatest resolution compared to an offset distance farther away from image-forming conjugate plane <NUM>. In some embodiments, a position farther away from the image-forming conjugate plane <NUM> can be selected to accommodate a maximum range of Z movement for automatic focus system <NUM>, so that the offset focusing camera <NUM> does not have to constantly be repositioned for specimens of varying thicknesses.

Note that offset focusing camera <NUM> can be positioned at an offset distance to the right or to the left of image-forming conjugate plane <NUM>. The sharpness slope over the range of Z movement when positioned to the right or to the left will be moving in one direction and opposite of each other. For example, if offset focusing camera <NUM> is positioned to the right of image-forming conjugate plane <NUM> and the sharpness slope is increasing over the range of increasing Z values, then if the offset focusing camera <NUM> is positioned to the left of image-forming conjugate plane <NUM>, the sharpness slope will be the opposite (i.e., decreasing over the range of increasing Z values). In other words, the sign of the sharpness slope (i.e., whether the slope is positive or negative) depends on whether the offset focusing camera is to the right or left of the image forming conjugate plane. Therefore, if the relative position of the offset focusing camera (i.e., whether the offset focusing camera is to the right or left of the image forming conjugate plane), the sharpness setting, and a sharpness value are known, then whether to increase or decrease the relative positioning of the objective and the stage to achieve better focus can be inferred.

In some embodiments, the offset distance for offset focusing camera <NUM> can be set once for automatic focus system <NUM>. In other embodiments, the offset distance can vary to accommodate different objectives, different specimen thicknesses, different specimen classes or any other suitable criteria. For example, offset camera <NUM> can be moved closer to image-forming conjugate plane for higher magnification objectives to accommodate a smaller depth of field (focus) and smaller range of Z movement. In some embodiments the offset distance can be saved by control system <NUM> as an offset distance setpoint. The offset distance setpoint can be associated, for example, with the thickness and/or any other suitable characteristic of a specimen, the specimen class and/or any other suitable grouping of the specimen, and/or the optical characteristics of the microscope (e.g., the magnification of an objective). The offset distance setpoint can be used to automatically position offset focusing camera <NUM>.

<FIG>, with further reference to <FIG>, shows at a high level, an example of an automatic focus operation of automatic focus system <NUM>, in accordance with some embodiments of the disclosed subject matter. Automatic focus process <NUM> can use automatic focus system <NUM>.

At <NUM>, a specimen can be placed on stage <NUM>.

If automatic focus system <NUM> does not know a sharpness setpoint for the specimen (e.g., the value can be obtained by user input or a prior value stored by control system <NUM> and associated with a particular specimen, a particular specimen class and/or any other suitable classification group for the specimen), then at <NUM> control system <NUM> can move stage <NUM> and objective <NUM> closer together and/or farther apart until the control system determines, using a suitable sharpness algorithm (as discussed above in connection with <FIG>), that the images captured by primary focusing camera <NUM> are in focus (e.g., control system <NUM> determines the in-focus position (e.g., at Z position <NUM>, i.e., the Z position when the specimen is considered to be in focus for primary focusing camera <NUM> as shown in <FIG>). In some embodiments, imaging camera <NUM> can be used, instead of primary focusing camera <NUM>, to determine the largest measured sharpness value for a specimen. In some embodiments, a sharpness curve can be calculated for offset focusing camera <NUM> based on images captured by the offset focusing camera <NUM> at the various Z positions of the stage and objective during the focusing process. The sharpness curve for offset focusing camera <NUM> can be stored as a sharpness curve setpoint and associated with a particular specimen, a particular specimen class and/or any other suitable classification group for the specimen.

At <NUM>, in some embodiments, once the specimen is determined to be in focus, an in-focus image can be captured by imaging camera <NUM>.

At <NUM>, once the specimen is determined to be in focus by primary focusing camera <NUM> (or imaging camera <NUM>), an image of the specimen can be captured by offset focusing camera <NUM>. A sharpness value for the captured imaged can be calculated (e.g., using the same sharpness equation used for the primary focusing camera <NUM>) and stored by control system <NUM>. The stored value can be stored as the in-focus sharpness setpoint and associated with a particular specimen, a particular specimen class and/or any other suitable classification group for the specimen. In some embodiments, when the specimen is in focus for primary camera <NUM> or imaging camera <NUM>, the absolute position of: stage <NUM>; objective <NUM>; the top of the specimen on stage <NUM>; and/or the distance between the top of stage <NUM> and objective <NUM>, can be stored by control system <NUM> as a position setpoint. The position setpoint can be associated with a particular specimen, a particular specimen class and/or any other suitable classification group for the specimen.

At <NUM>, stage <NUM> can be moved in an X/Y plane perpendicular to the Z axis.

At <NUM>, in some embodiments, offset camera <NUM> can be used to capture an image of the specimen at the new X, Y position of stage <NUM> and control system <NUM> can calculate the sharpness value for that image. Based on the sharpness value of the image compared to the in-focus sharpness setpoint, control system <NUM> can determine whether the specimen is in focus at stage <NUM>'s new X, Y coordinates or the Z height needs to be adjusted, so that the specimen is brought back in focus. For example, based on the sharpness curve shown in <FIG>, if the calculated sharpness value is greater than the stored in-focus sharpness setpoint, then stage <NUM> and objective <NUM> can be brought closer together in a Z direction until the sharpness value of an image captured by offset camera <NUM> is calculated to be the same as the stored sharpness setpoint. Conversely, if the calculated sharpness value is less than the stored in-focus sharpness setpoint, then the stage and objective can be brought farther apart in a Z direction until the sharpness value of an image captured by offset camera <NUM> is calculated to be the same as the stored in-focus sharpness setpoint. The direction to adjust the Z position can be determined either from a sharpness curve captured for offset camera <NUM> during the focusing process at <NUM> or based on the position of offset focusing camera <NUM> in relation to the image-forming conjugate plane <NUM>. This process for calculating the sharpness value of an image by the offset focusing camera and comparing it to a stored in-focus sharpness setpoint can be repeated each time the X,Y coordinates of stage <NUM> changes.

At <NUM>, a new specimen can be placed on stage <NUM>. If control system <NUM> determines that there is a sharpness setpoint already associated with the specimen, the specimen class and/or any other suitable classification group for the specimen, then the control system can use images captured by offset focusing camera <NUM>, as described at <NUM>, to determine when the new specimen is in focus. For example, an image of the new specimen can be captured by offset focusing camera <NUM> and the sharpness value can be compared with the sharpness setpoint associated with the new specimen. The stage and objective can be brought closer together or farther apart in a Z direction until the sharpness value of an image captured by offset camera <NUM> is calculated to be the same as the stored sharpness setpoint.

In some embodiments, once the calculated sharpness value for a specimen, using offset focusing camera <NUM>, corresponds to the sharpness setpoint, primary focusing camera <NUM> can be used to fine tune the focus of the specimen and the sharpness setpoint of offset focusing camera <NUM>. For example, using primary focusing camera <NUM>, sharpness values can be calculated for at least two relative Z positions of the stage and objective to determine whether an estimated maximum sharpness has been achieved or the relative Z-position needs to be adjusted to achieve an estimated maximum sharpness (i.e., the point on the sharpness curve where the slope is <NUM> or close to <NUM>). Once an estimated maximum sharpness is achieved, the sharpness value of the specimen, using offset focusing camera, can be calculated and stored as the new sharpness setpoint.

In some embodiments, control system <NUM> can also determine whether there is a position setpoint associated with the new specimen, specimen class and/or any other suitable classification group for the specimen, and can position autofocus system <NUM> at that position setpoint before it begins the aforementioned focusing process. Knowing the relative Z position, reduces the relative Z distance that is needed to focus the specimen and allows the offset camera to be positioned closer to the image-forming conjugate plane. As discussed above in connection with <FIG>, the slope of the sharpness curve can become steeper as offset focusing camera <NUM> moves closer to the image-forming conjugate plane. A steeper slope, represents greater resolution or a larger change in sharpness versus a smaller change in Z height. A steeper slope can allows for finer focal adjustment and control.

The division of when the particular portions of process <NUM> are performed can vary, and no division or a different division is within the scope of the subject matter disclosed herein. Note that, in some embodiments, blocks of process <NUM> can be performed at any suitable times. It should be understood that at least some of the portions of process <NUM> described herein can be performed in any order or sequence not limited to the order and sequence shown in and described in the <FIG> in some embodiments. Also, some of the portions of process <NUM> described herein can be or performed substantially simultaneously where appropriate or in parallel in some embodiments. Additionally or alternatively, some portions of process <NUM> can be omitted in some embodiments.

Process <NUM> can be implemented in any suitable hardware and/or software. For example, in some embodiments, process <NUM> can be implemented in control system <NUM>.

Claim 1:
A system (<NUM>) for automatically focusing a microscope, comprising:
an objective (<NUM>);
a stage (<NUM>) for positioning a specimen on a first image forming conjugate plane;
a first camera (<NUM>), configured for focusing, positioned on a second image forming conjugate plane;
a second camera (<NUM>), configured for focusing, positioned at an offset distance from the second image forming conjugate plane;
a primary illumination source (<NUM>) that emits light in a first wavelength range, wherein the emitted light is received by the first camera;
a secondary illumination source (<NUM>) that emits light in a second wavelength range which is different from the first wavelength range and that projects light through a focusing pattern that is positioned on a third image forming conjugate plane, wherein the projected light is received by the second camera; and
a hardware processor coupled to the first camera (<NUM>) and the second camera (<NUM>) that is configured to:
determine, using the first camera (<NUM>), when the specimen is in focus based on a sharpness value;
capture, using the second camera (<NUM>), a relative sharpness value when the specimen is determined to be in focus for the first camera (<NUM>) and determine the relative sharpness value as a sharpness setpoint;
after movement of the specimen, determine, using the second camera, a first sharpness value of the specimen;
determine whether the first sharpness value of the specimen is higher or lower than the sharpness setpoint; and
adjust a distance between the objective (<NUM>) and the stage (<NUM>) so that a second sharpness value of the specimen determined using the second camera (<NUM>) corresponds to the sharpness setpoint.