Patent Description:
The invention relates to imaging blood cells, e.g., using electronic imaging systems.

Many imaging and scanning applications acquire images in an automated fashion. Interpreting such images may require identifying one or more objects within the images. While such identification may be performed manually, the process is time consuming and may be prone to errors. Document <CIT> discloses an automated system for cell identification and classification, wherein a primary classifier performs a basic screening function, and selects normal cells from among abnormal cells in a cell sample based on one set of features calculated by the computer. A secondary classifier indicates whether the selected cells are normal or have malignancy-associated changes based on a second set of computed features.

In summary, the independent claim <NUM> defines the invention and the dependent claims <NUM>-<NUM> defines preferred implementations of the invention.

Biological specimens can be imaged at multiple individual wavelengths and a set of images can be acquired. Characteristics of the biological specimens can be determined by processing one or more of the images. For instance, processing of low-magnification images can include counting of red blood cells (RBCs), locating and counting blood constituents (e.g., white blood cells (WBCs) and nucleated red blood cells (nRBCs)), and counting platelets,. For instance, processing of high-magnification images can include characterizing features of RBCs, calculating a WBC differential, and classifying WBCs. Calculating a WBC differential can include, for example, counting a number of different types of WBCs in the blood. In some implementations, calculating a WBC differential can include calculating percentages of different types of WBC in the blood.

<FIG> shows one embodiment of an imaging system <NUM> that employs the fast autofocusing methods described herein. Imaging system <NUM> includes imaging hardware <NUM> that is controlled by a computer <NUM>. The computer <NUM> generally includes a central processing unit <NUM>, a hard drive <NUM>, and random access memory <NUM>.

In the imaging system shown in <FIG>, a light source <NUM> illuminates a slide <NUM> comprising a biological specimen <NUM>. The light source <NUM> includes individual light sources of different colors. In one example, the light source <NUM> may include blue, green, yellow, and red light emitting diodes (LEDs). Other types of colored light sources can also be used. The light source <NUM> can be used for low magnification imaging, high magnification imaging, or both.

The slide is mounted on a first motorized stage <NUM> capable of moving in a horizontal plane (parallel to the surface of the slide that is imaged) such that any part of the slide <NUM> can be positioned under an objective lens <NUM>. A second motorized stage <NUM> moves the objective lens <NUM> up and down to facilitate focusing on the specimen <NUM> deposited on slide <NUM>. The distance between the slide <NUM> and the objective lens <NUM> is referred to as the "focal distance. " A reduction in focal distance implies, in this example, moving the objective lens <NUM> vertically towards the slide <NUM>. Alternatively, the focal distance can also be adjusted by moving the slide <NUM> (for example by moving the first motorized stage <NUM>) vertically towards the objective lens <NUM>. In some implementations, both the slide <NUM> and the objective lens <NUM> can move to facilitate adjusting the focal distance. The axes, with respect to which the first motorized stage <NUM> is moved in a horizontal plane, are typically referred to at the X and Y axes. The vertical axis along which the second motorized stage <NUM> moves the objective lens <NUM> is typically referred to as the Z axis. The three axes define a coordinate system that the system <NUM> utilizes to image any (x, y, z) point in space relative to the slide <NUM>.

Light from the source <NUM> passes through the slide <NUM> and is projected by the objective lens <NUM> onto the sensor of the camera <NUM>. The sensor may, for example, be a charge-coupled device (CCD) array. <FIG> depicts an example of "bright field" microscopy where objects on the slide are visible because they absorb light and are therefore darker in the image produced by the camera. The imaging hardware <NUM> can include one or more additional lenses. Other microscopic modes such as fluorescence, dark-field, or phase contrast can also generate images to which the fast auto-focus methods described herein can be applied.

If an image is acquired at a non-optimal focal distance, the image is blurry and typically unsuitable for many image processing applications. If the surface of the slide <NUM> were perfectly planar, the system <NUM> could acquire in-focus images simply by determining the proper z height corresponding to an in-focus image at three (x, y) locations and then fitting a plane to those (x, y, z) points. The plane equation would then provide an in-focus z height for any other (x, y) location on the slide. In practice, however, the focal distance for a given location may not be accurately determined from a plane fit as described above due to irregularities in the surface of the slide <NUM> and/or the stage <NUM>. Therefore, in general, the focal distance may need to be adjusted slightly for each image that is acquired at a different (x, y) location on the slide.

After image acquisition, camera <NUM> sends images to the computer <NUM> for processing. If the images are processed quickly enough, focal distance data from one image location may be used to adjust the focal distance at the next location for capturing an image. This allows the system to adjust to variations in focal distances corresponding to different locations and, in turn, produce more accurately focused images for display. For example, if one region of the slide <NUM> is slightly thicker than others and ten locations are imaged within that region, the change in thickness can be discovered after the first image is acquired and additional images taken at other locations within the region can be acquired at slightly adjusted focal distances to compensate for the change in thickness using the methods described herein.

In some examples, low magnification images can be acquired with a 10X objective, a <NUM>. 5X optical coupler, and a <NUM> megapixel CCD camera. In some examples, high magnification images can be acquired with a 50X objective, a 1X optical coupler, and a <NUM> megapixel, <NUM> frames per second camera.

In some examples, a set of black and white images (referred to herein as a "stack" of images) is acquired of the slide at an imaging location. Each image of the set acquired using a single wavelength of light (e.g., using each of the blue, green, yellow, and red LEDs of the light source <NUM>). If desired, color images, such as color JPEG images, can be created from the set of black and white images for display purposes. In some examples, objective lenses are not corrected for oil immersion or for the presence of a coverslip. Each image of the set is background corrected and aligned with each other image of the set. If more than one image is taken with a certain color at a particular location, one of those images is selected (e.g., the image with the best focus). An on-the-fly estimate of the focal plane of the slide can be updated during image acquisition. The set of images is processed by one or more imaging modules to yield measurements that can be used to determine blood count, white blood cell (WBC) differential, and other characteristics of the biological specimen.

Illuminating a specimen with different colors of light can result in different information being extracted from the acquired images. For example, in the case of imaging a specimen containing blood cells, cells may appear differently under different colored illumination, thereby facilitating easier identification, classification, or differentiation. For example, red blood cells absorb significant amounts of blue light due to the presence of hemoglobin, and cell nuclei stained with standard Romanowsky stains absorb yellow light.

Referring to <FIG>, image acquisition and processing is controlled by a control system <NUM>. In the depicted example, the control system <NUM> is implemented by a single computer <NUM>. In some examples, the control system <NUM> can be implemented by multiple computers. For instance, some modules of the control system <NUM> can be implemented by a first computer and other modules of the control system <NUM> can be implemented by a second computer.

A hardware module <NUM> sends commands to imaging hardware (e.g., controlled by an imaging module <NUM>), which in turn sends pulses to the camera <NUM> to trigger image acquisition. The acquired images are transferred to a computer (e.g., the computer <NUM>). A camera module <NUM> packages the acquired images into stacks.

While a slide is being imaged, a focus module <NUM> examines each stack of images and estimates at what height the stack is likely to have been acquired, e.g., based on the relative focus scores of the images of the stack. The focus module <NUM> reports these height estimates to a plane fit module <NUM>. Before each location is imaged, the hardware module <NUM> asks the plane fit module <NUM> for a focal plane estimate at that location. This mechanism of updating the focal plane estimate with image acquisition can be used by the imaging system <NUM> or the imaging module <NUM> or both to adapt to irregularities on the surface of a slide. In some instances, there may be a lag in the adaptive response to slide irregularities, e.g., due to pipelining of the focal plane estimates.

A registration module <NUM> and a pick registration frames module <NUM> can be used to match coordinate systems between low magnification images and high magnification images. Thus, for instance, any object found during low magnification imaging can be precisely located and imaged during high magnification imaging.

An alignment module <NUM> provides a capability to align images of a stack. In some instances, the raw images of a stack do not align with each other, for instance, due to slide movement, slightly off-center LED dies, optical effects of the lens, or other factors. Furthermore, under low magnification, images acquired under different illumination colors may each have a slightly different magnification if the lens is not perfectly color-corrected. Under high magnification, the scale of the four colors may be the same but the images may still be misaligned. In one example, to align low magnification images, an algorithm can be used that selects a set of high-contrast tie points (e.g., cells) in a target image, locates the same tie points (e.g., cells) in a second image, and transforms the second image to align the second image with the target image. In one example, to align high magnification images, an algorithm can try all offsets within a limited range of pixels and select the offset for each color that maximizes the R<NUM> correlation with the image of a target color (e.g., with a green image).

Images, results of analysis of the images, or both can be displayed on a user interface, such as a graphical user interface (GUI) <NUM>. Images, results of analysis of the images, or both can be logged by a logging module <NUM> and stored in a file, database <NUM>, or other storage location.

In some examples, a calibration can be performed when the imaging system <NUM> is initialized, e.g., with no slide in place. Calibration can include blank image acquisition and initial shutter adjustment. In some examples, for each slide, a focus search can be performed to determine a tilt of the slide, e.g., using a fast adjustment algorithm that takes only a fraction of a second.

Various types of image processing can be performed on stacks of images acquired using the imaging system <NUM>. For instance, processing of low-magnification images can include counting of red blood cells (RBCs), locating and counting blood constituents (e.g., white blood cells (WBCs) and nucleated red blood cells (nRBCs)), counting platelets, and other types of processing. For instance, processing of high-magnification images can include characterizing features of RBCs, calculating a WBC differential, classifying WBCs, and other types of processing.

Referring again to <FIG>, in some examples, image processing is executed by an image processing module <NUM> that is implemented by the computer <NUM> implementing the control system <NUM>. In some examples, the image processing module <NUM> is implemented by a different computer that does not implement any module of the control system <NUM>.

In one example, low magnification images can be processed to count RBCs. RBCs can have various sizes (e.g., RBCs can be large, small, or fragmented). RBCs can also have various appearances (e.g., RBCs can have large or small pallor or unusual shapes). To account for these variations, a filter (referred to herein as a "spot filter") is applied that transforms each RBC, regardless of its size or pallor, into a smaller, dark spot. The transformed spots can be easier to count than images of original, varied RBCs. In some instances, the spot filter can be applied only to the blue image of a stack of images, because white cells are nearly invisible in blue images. The filtered images can be processed to count the RBCs, including masking dust, debris, and other contaminants; smoothing the image; determining an average area of a RBC, counting the spots in the image; rejecting artifacts (e.g., stain debris); and adjusting the count based on the "dust area.

In one example, low magnification images can be processed to locate WBCs and nRBCs. WBCs can be located during low magnification imaging, and in some examples can be revisited during high magnification imaging. Referring to <FIG>, a low magnification image is processed by locating blue spots of approximately the right size to be a nucleus or nuclear fragment (<NUM>). Regions of the image that are the right color to be cytoplasm are located (<NUM>). A distance transform is performed on the nucleus images, constrained to within the cytoplasm boundaries (<NUM>). A watershed transform with a size limit is performed (<NUM>) to divide touching cytoplasms. Cell types can then be assigned (<NUM>), e.g., using a ten part classifier that identifies five normal WBC types, platelets, nRBCs, double WBCs, damaged WBCs, and artifacts (e.g., stain debris). Once the classification has been performed, a cell count is performed (<NUM>) and locations of interest for high magnification can be selected (<NUM>).

In one example, low magnification images can be processed to count platelets. Platelets can be located and counted using the filter counting approach described above for RBCs, but applying the spot filter to the yellow and blue images and skipping areas that are under objects already identified as WBCs. A classifier can be used to differentiate platelets from artifacts to determine the final count of platelets.

High magnification imaging provides the ability to examine cells more closely than in low magnification imaging. In some examples, a high magnification of a WBC can be used to classify the WBC. In some examples, a high magnification image of RBCs or platelets can be used to characterize the RBCs or platelets.

In one example, high magnification images can be processed to characterize features of RBCs, such as mean cell volume (MCV), hemoglobin content (MCH), reticulocyte percent, or other features, such as other elements of a complete blood count (CBC). For these characterizations, a large number of RBCs can be imaged under high magnification (e.g., at least about <NUM>,<NUM> RBCs). Under the assumption that RBC aggregation is random, high magnification images of single, non-overlapping RBCs can be selected for measurement.

MCV and MCH can be calculated from a weighted sum of nine features: the area of a RBC, an integrated optical density of the RBC for each of the four imaging colors, and a volume measurement of the RBC for each of the four imaging colors. The reticulocyte percentage can be determined by analyzing the level of blue content within each of the imaged RBCs. Other elements of a CBC can be determined from the RBC count (determined from low magnification imaging, e.g., as described above), the MCH, and the MCV of a sample.

Additional features of high magnification images of RBCs can be used to determine the potential presence of inclusions in the RBCs, to assess RBC shapes, or to perform other characterizations of the RBCs. Specific RBCs with potential inclusions, unusual shapes, or both, can be tagged for presentation to a user. For instance, RBCs with potential inclusions can be displayed in an RBC gallery. For instance, RBCs with unusual shapes can be assigned values corresponding to their shapes, allowing the RBCs in the gallery to be sorted by shape.

In one example, high magnification images can be processed to determine a WBC differential. An accurate WBC differential can be determined based on segmentation for WBC nuclei and for the cytoplasm. WBC nuclei generally stain the same color or similar colors and thus can be relatively easy to recognize. Cytoplasm can stain many different colors, and in some cases can be almost transparent. In addition, WBCs often aggregate; to obtain an accurate WBC differential, the WBCs in an aggregate can be counted individually.

Referring to <FIG>, to process high magnification images to determine a WBC differential, the cytoplasmic material is located (<NUM>). Aggregated WBCs are separated (<NUM>). The nuclei of the WBCs are segmented (<NUM>). To account for any variations in the stained color of the WBC nuclei, an adaptive threshold can be chosen (<NUM>) within each cytoplasm mask based on the yellow, green, and blue images. Two nuclear masks are created (<NUM>), one mask of the nucleus segmented as if it were a granulocyte and the other mask of the nucleus segmented as if it were agranular. The two nuclear masks are intersected (<NUM>) to produce a nuclear segmentation that is well suited to both granulocytes and agranular nuclei.

Features (e.g., over <NUM> features) are measured on the intersected cell masks (<NUM>). Some features can be measured only on the cytoplasm or only on the nucleus region of a WBC; other features can be measured on the entire cell. Area and shape features can be determined based upon just the nuclear and cytoplasmic masks. Color features can be determined based on the grey levels of individual pixels within the nuclear or cytoplasmic mask in different color images. Optical density and texture features can be determined based on the four different color images, within the nuclear or cytoplasmic masks. The measured features can be used to determine the WBC differential (<NUM>).

In one example, high magnification images can be processed to classify WBCs. A multi-stage "tree" classifier can be used. The classifier can have mostly linear discriminants (LDA). The classifier can also include a few non-LDA nodes, e.g., where the non-LDA nodes can be helpful for separating populations without a normal distribution. In some implementations, at least one of the non-LDA nodes can implement a Bayesian classification system.

Referring to <FIG>, the classifier sorts out any non-WBC objects in the images that may be similar to WBCs, such as clumps of platelets, giant platelets, micromegakaryocytes, or other non-WBC objects (<NUM>). The remaining objects are sorted into a rough five-way classification (<NUM>), e.g., along the biological lines of WBC maturation pathways. Any damaged or out-of-focus cells from the fringes of each sorted group are removed from the classification (<NUM>). Each of the five groups is processed to extract potential unclassified cells, such as immature cells, blasts, promonos, prolymphs, or other unclassified cells, and to correct mistakes in the classification of the normal types (<NUM>). In some examples, the classification scheme can be designed to minimize the number of false positive unclassified cells. For instance, each of the groups of unclassified cells (e.g., immature granulocytes, blasts, promonos, prolymphs) that were extracted is processed by a secondary classifier (<NUM>) to check for false positives. The cells that remain unclassified are grouped together into a single group (<NUM>).

A value indicating the "nuclear complexity" of neutrophils can be determined. A value indicating the "atypicality" of lymphocytes can also be determined (<NUM>).

The results of the classification can be displayed on a viewing station (<NUM>), such as a user interface of a computing device. In some examples, the group of unclassified cells can be displayed on the top line of a WBC tab on the user interface. In some examples, the unclassified cells can be displayed, e.g., on a front page of a WBC gallery, along with the equivalent of a hundred cell differential by choosing hundred cells at random from the normal WBC classification categories. Additional WBCs can be displayed responsive to instructions from a user. The nuclear complexity and atypicality values can also be displayed or made available to the user. For instance, a user can classify neutrophils as "banded" or "segmented" based on their nuclear complexity value and can sort neutrophils as banded or segmented based on this classification. A user can also classify lymphocytes as "atypical" based on their atypicality value and can sort lymphocytes as typical or atypical based on this classification.

A multi-stage classifier is used to classify white blood cells (WBCs) according to one or more features extracted from images of the biological sample. The multi-stage classifier performs multiple stages of classification. Different types of classifiers are used at the different stages. According to the invention, one stage of the classifier uses a linear discriminant analysis (LDA) based classifier and another stage uses a Bayesian classifier.

In general, the multi-stage classifier can have a complex topology that can combine topologies from two or more other classifiers, such as tree classifiers and ladder classifiers, to enable efficient, accurate classification of WBCs, or blood cells in general.

The classifier may classify objects into cell types, such as lymphocytes, monocytes, neutrophils, basophils, and eosinophils. The classifier may classify objects into specific other types of objects, such as various specific types of immature cells, or may classify these types of objects into a general category (e.g., an unclassified category).

Each stage of the multi-stage classifier can classify objects into a plurality of classifications. In some instances, the classification of an object at a particular stage may confirm the classification of the object from an earlier stage (for instance, a lymphocyte may remain classified as a lymphocyte). In some instances, the classification of an object at a particular stage may reclassify the object into another category into which the object had not been classified during an earlier stage (for instance, in a first stage, the object may have been classified as a lymphocyte rather than a neutrophil; in a second stage, the object may be reclassified as a neutrophil). In some instances, the classification of an object at a particular stage may classify the object into a category that was not available at an earlier stage (for instance, a lymphocyte may be classified as a blastocyst, which was not available for classification in earlier stages).

Objects can be classified based on feature vectors representing one or more features of the objects, such as cell area, cell shape, cell color, cell optical density, cell texture, and other features of the cells. In some examples, the feature vector used for classification can be substantially same for each stage of classification. In some examples, a different feature vector can be used for some or all stages of classification. For instance, feature vectors for later classification stages may have more features than feature vectors for earlier classification stages, enabling more accurate classification, or classification into a different set of classes.

Referring to <FIG>, in a general approach to multi-stage classification of objects in a biological sample, one or more images of a substrate on which the biological sample is disposed are acquired (<NUM>). A plurality of features of the objects is extracted from the one or more images (<NUM>), e.g., using one or more of the techniques described above.

Multiple classifications are performed for each object. For instance, a first classification is performed (<NUM>) and a second classification is performed (<NUM>). Each classification is based on a corresponding feature vector composed of one or more of the extracted features of the respective object. If the results of the second classification are the same as the results of the first classification (<NUM>), the classification of the object is maintained (<NUM>). If the results of the second classification differ from the results of the first classification (<NUM>), the classification of the object is changed (<NUM>). For instance, changing a previous classification of the object can include reclassifying the object into a category that was available during a previous classification. Changing a previous classification can include classifying the object into a category that was not available during a previous classification.

Any number of classifications can be performed.

Referring to <FIG>, an example topology of a multi-stage classifier <NUM> is depicted. The classifier depicted in this example classifies objects in multiple stages according to an LDA classification (depicted by circular nodes such as node <NUM>) and non-LDA (e.g., Bayesian) classification (depicted by diamond nodes such as node <NUM>).

At a first classification node <NUM>, f-platelets (referred to as "fplate") are removed from the set of "all" objects for classification. At a second classification node <NUM>, s-platelets ("splate") are also removed from the set of "all" objects for classification.

The f-platelets can be classified into platelets (referred to as "pit"), out-of-focus ("oof") material, damaged ("dam") material, or junk (node <NUM>). The s-platelets can be reclassified as "all" or classified into platelets, oof material, damaged material, or junk (node <NUM>). The objects classified as platelets at nodes <NUM>, <NUM> are further classified at nodes <NUM>, <NUM>. The objects reclassified as "all" are returned to the set of "all" objects for further classification.

The set of "all" objects can be further classified at node <NUM> into five cell types: lymphocytes ("lymph"), monocytes ("mono"), neutrophils ("neut"), basophils ("baso"), or eosinophils "(eo"). The objects in each of these classifications are classified to remove out-of-focus or damaged objects from the classification (nodes <NUM>, <NUM>, <NUM>, <NUM>, <NUM>).

The objects in each cell type are further classified. Some classifications can result in the classification of an object being maintained (e.g., a lymphocyte may remain classified as a lymphocyte). Some classifications can result in an object being reclassified as a different cell type (e.g., a lymphocyte may be reclassified as a neutrophil). Some classifications can result in an object being classified into a new classification (e.g., a lymphocyte may be classified as a blastocyst). In the example shown, the following classifications can occur:.

In subsequent classification stages, all objects in a particular classification are treated together, regardless of the nodes at which those objects were classified:.

Further classification stages also treat all objects in a particular classification together, regardless of the nodes at which those objects were classified, as illustrated by the following examples:.

The classifier results in the classification of objects as follows:.

This multi-stage classifier can provide a scheme that enables efficient and accurate classification of objects.

<FIG> is a schematic diagram of a computer system <NUM> that can be used to control the operations described in association with any of the computer-implemented methods described herein, according to one implementation. The system <NUM> includes a processor <NUM>, a memory <NUM>, a storage device <NUM>, and an input/output device <NUM>. Each of the components <NUM>, <NUM>, <NUM>, and <NUM> are interconnected using a system bus <NUM>. The processor <NUM> is capable of processing instructions for execution within the system <NUM>. In one implementation, the processor <NUM> is a single-threaded processor. In another implementation, the processor <NUM> is a multi-threaded processor. The processor <NUM> is capable of processing instructions stored in the memory <NUM> or on the storage device <NUM> to display graphical information for a user interface on the input/output device <NUM>.

In some implementations, the memory <NUM> is a computer-readable medium. The memory <NUM> can include volatile memory and/or non-volatile memory.

In general, the storage device <NUM> can include any non-transitory tangible media configured to store computer readable instructions.

In some implementations, the input/output device <NUM> includes a keyboard and/or pointing device. In some implementations, the input/output device <NUM> includes a display unit for displaying graphical user interfaces.

The features described can be implemented in digital electronic circuitry, or in computer hardware, firmware, or in combinations of them. The features can be implemented in a computer program product tangibly embodied in an information carrier, e.g., in a machine-readable storage device, for execution by a programmable processor; and features can be performed by a programmable processor executing a program of instructions to perform functions of the described implementations by operating on input data and generating output. The described features can be implemented in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. A computer program includes a set of instructions that can be used, directly or indirectly, in a computer to perform a certain activity or bring about a certain result.

Various software architectures can be used for implementing the methods and systems described in this application. For example, a publish/subscribe messaging pattern can be used in implementing the methods and systems described herein. In the case of publish/subscribe messaging, the system includes several hardware and software modules that communicate only via a messaging module. Each module can be configured to perform a specific function. For example, the system can include one or more of a hardware module, a camera module, and a focus module. The hardware module can send commands to the imaging hardware implementing the fast auto-focus, which in turn triggers a camera to acquire images.

A camera module can receive images from the camera and determine camera parameters such as shutter time or focus. Images can also be buffered in the computer's memory before being processed by the camera module. When performing the initial search for the tilt of the slide, the camera module can also send a message interrupting the hardware module when it has seen enough images to determine the proper shutter time or focus.

The system can also include a focus module that can be implemented as software, hardware or a combination of software and hardware. In some implementations, the focus module examines all the frames in a stack and estimates how far the stack is from the ideal or ideal focal distance. The focus module can also be responsible for assigning a focus score to each frame in a stack of images.

Computers include a processor for executing instructions and one or more memories for storing instructions and data.

Alternatively, the computer can have no keyboard, mouse, or monitor attached and can be controlled remotely by another computer.

Claim 1:
A method for classifying white blood cells (WBCs) in a biological sample on a substrate, the method comprising:
acquiring, by an image acquisition device, one or more images of the substrate; extracting a plurality of features of the white blood cells in the biological sample from the one or more images; and
performing a plurality of classifications for each white blood cell, each classification based on a corresponding feature vector composed of one or more of the extracted features of the respective white blood cell, each classification including either maintaining a previous classification of the white blood cell or changing a previous classification of the white blood cell,
wherein at least one of the classifications includes a linear discriminant analysis (LDA) classification and at least one of the classifications includes a Bayesian classification.