Patent ID: 12243231

DETAILED DESCRIPTION

In the following detailed description, for purposes of explanation and not limitation, specific details are set forth in order to provide a better understanding of the present disclosure. It will be apparent to one skilled in the art that the present disclosure may be practiced in other embodiments that depart from these specific details.

Cancer is the second leading cause of death in North American women. Of all types of cancer in women, breast cancer is the most common and the second leading cause of cancer death. Therefore, the accuracy of breast cancer treatment has a great impact on the lifespan and quality of life in the significant percentage of women that will be affected by breast cancer at some point in their lives.

Based on the expression of certain genes, breast cancers can be divided into different molecular subtypes. A commonly used classification scheme is as follows:1. Luminal A: ER+, PR+, HER2−2. Luminal B: ER+, PR−, HER2+3. Triple-Negative Breast Cancer (TNBC): ER−, PR−, HER2−4. HER2-enriched: HER2+, ER−, PR−5. Normal-like.

ER stands for estrogen receptor. PR stands for progesterone receptor. HER2 stands for human epidermal growth factor receptor 2.

We describe a computer-automated tumor finding method which detects and outlines invasive and in situ breast cancer cell nuclei automatically. The method is applied to a single input image, such as a WSI, or a set of input images, such as a set of WSIs. Each input image is a digitized, histological image, such as a WSI. In the case of a set of input images, these may be differently stained images of adjacent tissue sections. We use the term stain broadly to include staining with biomarkers as well as staining with conventional contrast-enhancing stains.

Since computer-automated outlining of tumors is much faster than manual outlining, it enables an entire image to be processed, rather than only manually annotating selected extracted tiles from the image. The proposed automatic tumor outlining should thus enable pathologists to compute a positivity (or negativity) percentage over all the tumor cells in the image, which should result in more accurate and reproducible results.

The proposed computer-automated method for tumor finding, outlining and classifying uses a convolutional neural network (CNN) to find each nuclear pixel on the WSI and then to classify each such pixel into one of a non-tumor class and one of a plurality of tumor classes, in our current implementation breast tumor classes.

The neural network in our implementation is similar in design to the VGG-16 architecture available at: <http://www.robots.ox.ac.uk/˜vgg,/research/very_deep> and described in Simonyan and Zisserman 2014, the full contents of which are incorporated herein by reference.

The input image is a pathology image stained with any one of several conventional stains as discussed in more detail elsewhere in this document. For the CNN, image patches are extracted of certain pixel dimensions, e.g. 128×128, 256×256, 512×512 or 1024×1024 pixels. It will be understood that the image patches can be of arbitrary size and need not be square, but that the number of pixels in the rows and columns of a patch conform to 2n, where n is a positive integer, since such numbers will generally be more amenable for direct digital processing by a suitable single CPU (central processing unit), GPU (graphics processing unit) or TPU (tensor processing unit), or arrays thereof.

We note that ‘patch’ is a term of art used to refer to an image portion taken from a WSI, typically with a square or rectangular shape. In this respect we note that a WSI may contain a billion or more pixels (gigapixel image), so image processing will typically be applied to patches which are of a manageable size (e.g. ca. 500×500 pixels) for processing by a CNN. The WSI will thus be processed on the basis of splitting it into patches, analyzing the patches with the CNN, then reassembling the output (image) patches into a probability map of the same size as the WSI. The probability map can then be overlaid, e.g. semi-transparently, on the WSI, or part thereof, so that both the pathology image and the probability map can be viewed together. In that sense the probability map is used as an overlay image on the pathology image. The patches analyzed by the CNN may be of all the same magnification, or may have a mixture of different magnifications, e.g. 5×, 20×, 50× etc. and so correspond to different sized physical areas of the sample tissue. By different magnifications, these may correspond to the physical magnifications with which the WSI was acquired, or effective magnifications obtained from digitally downscaling a higher magnification (i.e. higher resolution) physical image.

FIG.1Ais a schematic drawing of our neural network architecture. Layers C1, C2. . . C10are convolutional layers. Layers D1, D2, D3, D4, D5and D6are transpose convolution (i.e. deconvolutional) layers. The lines interconnecting certain layers indicate skip connections between convolutional, C, layers and deconvolutional, D, layers. The skip connections allow local features from larger dimension, shallower depth layers (where “larger” and “shallow” mean a convolutional layer of lower index) to be combined with the global features from the last (i.e. smallest, deepest) convolutional layer. These skip connections provide for more accurate outlines. Maxpool layers, each of which is used to reduce the width and height of the patch by a factor of 2, are present after layers C2, C4and C7, but are not directly shown in the schematic, although they are shown by implication through the consequential reducing size of the patch. In some implementations of our neural network the maxpool layers are replaced with 1×1 convolutions resulting in a fully convolutional network.

The convolutional part of the neural network has the following layers in sequence: input layer (RGB input image patch); two convolutional layers, C1, C2; a first maxpool layer (not shown); two convolutional layers C3, C4; a second maxpool layer (not shown); three convolutional layers, C5, C6, C7, and a third maxpool layer (not shown). The output from the second and third maxpool layers is connected directly to deconvolutional layers using skip connections in addition to the normal connections to layers C5and C8, respectively.

The final convolutional layer, C10, the output from the second maxpool layer (i.e. the one after layer C4) and the output from the third maxpool layer (i.e. the one after layer C7), are then each connected to separate sequences of “deconvolution layers” which upscale them back to the same size as the input (image) patch, i.e. convert the convolutional feature map to a feature map which has the same width and height as the input image patch and a number of channels (i.e. number of feature maps) equal to the number of tissue classes to be detected, i.e. a non-tumorous type and one or more tumor types. For the second maxpool layer, we see a direct link to the layer D6since only one stage of deconvolution is needed. For the third maxpool layer, two stages of deconvolution are needed, via intermediate deconvolution layer D4, to reach layer D5. For the deepest convolutional layer C10, three stages of deconvolution are needed, via D1and D2to layer D3. The result is three arrays D3, D5, D6of equal size to the input patch.

A simplified, albeit probably less-well performing, version of what is illustrated inFIG.1Acould omit the skip connections, in which case layers D4, D5and D6would not be present and the output patch would be computed solely from layer D3.

FIG.1Bshows in more detail how the final steps in the neural network architecture ofFIG.1Aare carried out. Namely, global feature map layer D3and local feature map layers D5, D6are combined to generate a feature map that predicts an individual class for each pixel of the input image patch. Specifically,FIG.1Bshows how the final three transpose convolution layers D3, D5, D6are processed to the tumor class output patch.

We now discuss how the above-described approach differs from a known CNN used currently in digital pathology. This known CNN assigns one class selected from multiple available classes to each image patch. Examples of such type of CNN are in the papers by Wang et al 2016, Liu et al 2017, Cruz-Roa et al 2017 and Vandenberghe et al 2017. However, what we have just described is that, within a given image patch, one class selected from multiple available classes is assigned to each and every pixel. Therefore, instead of generating a single class label for each image patch, our neural network outputs a class label for each individual pixel of a given patch. Our output patch has a one-to-one pixel-to-pixel correspondence with the input patch such that each pixel in the output patch has assigned to it one of the multiple available classes (non-tumor, tumor1, tumor2, tumor3etc.).

In such known CNNs, to assign a single class to each patch, a series of convolutional layers is employed followed by one or several fully connected layers, followed by an output vector which has as many values as there are classes to detect. The predicted class is determined by the location of the maximum value in the output vector.

A trained CNN will take, as input, pixels from a digital slide image and return a vector of probabilities for each pixel (Goodfellow, Bengio, and Courville2016). The vector is of length N where N is the number of classes the CNN has been trained to detect. For example, if a CNN has been trained to distinguish between three classes, invasive tumor, in situ tumor and non-tumor, the vector v will be of length 3. Each coordinate in the vector indicates the probability that the pixel belongs to a specific class. So v[0] may indicate the probability that the pixel belongs to the invasive tumor class, v[1] the probability it belongs to the in situ class and v[2] the probability it belongs to the non-tumor class. The class of each pixel is determined from the probability vector. A simple method of assigning a pixel to a class is to assign it to the class for which it has the highest probability.

To predict the class of individual pixels, our CNN uses a different architecture following the convolutional layers. Instead of a series of fully connected layers, we follow the convolutional layers with a series of transpose convolutional layers. The fully connected layers are removed from this architecture. Each transpose layer doubles the width and height of the feature maps while at the same time halving the number of channels. In this manner, the feature maps are upscaled back to the size of the input patch.

In addition, to improve the prediction, we use skip connections as described in Long et al 2015, the full contents of which is incorporated herein by reference.

The skip connections use shallower features to improve the coarse predictions made by upscaling from the final convolutional layer C10. The local features from the skip connections contained in layers D5and D6ofFIG.1Aare concatenated with the features generated by upscaling the global features contained in layer D3ofFIG.1Afrom the final convolutional layer. The global and local feature layers D3, D5and D6are then concatenated into a combined layer as shown inFIG.1B.

From the concatenated layer ofFIG.1B(or alternatively directly from the final deconvolutional layer D3in the case that skip connections are not used), the number of channels is reduced to match the number of classes by a 1×1 convolution of the combined layer. A softmax operation on this classification layer then converts the values in the combined layer into probabilities. The output patch layer has size N×N×K, where N is the width and height in pixels of the input patches and K is the number of classes that are being detected. Therefore, for any pixel P in the image patch there is an output vector V of size K. A unique class can then be assigned to each pixel P by the location of the maximum value in its corresponding vector V.

The CNN thus labels each pixel as non-cancerous or belonging to one or more of several different cancer (tumor) types. The cancer of particular interest is breast cancer, but the method is also applicable to histological images of other cancers, such as cancer of the bladder, colon, rectum, kidney, blood (leukemia), endometrium, lung, liver, skin, pancreas, prostate, brain, spine and thyroid.

Our specific neural network implementation is configured to operate on input images having certain fixed pixel dimensions. Therefore, as a preprocessing step, both for training and prediction, patches are extracted from the WSI which have the desired pixel dimensions, e.g. N×N×n pixels, where n=3 in the case that each physical location has three pixels associated with three primary colors—typically RGB, when the WSI is a color image acquired by a conventional visible light microscope. (As mentioned further below ‘n’ may be 3 times the number of composited WSIs in the case the two or more color WSIs are combined.) Moreover ‘n’ would have a value of one in the case of a single monochrome WSI. To make training faster the input patches are also centered and normalized at this stage.

Our preferred approach is to process the entire WSI, or at least the entire area of the WSI which contains tissue, so the patches in our case are tiles that cover at least the entire tissue area of the WSI. The tiles may be abutting without overlap, or have overlapping edge margin regions of for example 1, 2, 3, 4, 5, 6, 7, 8, 9 or 10 pixels wide so that the output patches of the CNN can be stitched together taking account of any discrepancies. Our approach can however, if desired, also be applied to a random sample of patches over the WSI which are of the same or different magnification, as in the prior art, or as might be carried out by a pathologist.

Our neural network is similar in design to the VGG-16 architecture of Simonyan and Zisserman 2014. It uses very small 3×3 kernels in all convolutional filters. Max pooling is performed with a small 2×2 window and stride of 2. In contrast to the VGG-16 architecture, which has a series of fully connected layers after the convolutional layers, we follow the convolution layers with a sequence of “deconvolutions” (more accurately transpose convolutions) to generate segmentation masks. This type of upsampling for semantic segmentation has previously been used for natural image processing by Long et al 2015, the full contents of which are incorporated herein by reference.

Each deconvolutional layer enlarges the input feature map by a factor of two in the width and height dimensions. This counteracts the shrinking effect of the maxpool layers and results in class feature maps of the same size as the input images. The output from each convolution and deconvolutional layer is transformed by a non-linear activation layer. At present, the non-linear activation layers use the rectifier function ReLU (x)=max (0, x)ReLU(x)=max(0,x). Different activation functions may be used, such as ReLU, leaky ReLU, eLU, etc. as desired.

The proposed method can be applied without modification to any desired number of tissue classes. The constraint is merely the availability of suitable training data which has been classified in the manner that it is desired to replicate in the neural network. Examples of further breast pathologies are invasive lobular carcinoma or invasive ductal carcinoma, i.e. the single invasive tumor class of the previous example can be replaced with multiple invasive tumor classes. The accuracy of the neural network is mostly dictated by the number of images available for each class, how similar the classes are, and how deep the neural network can be made before running into memory restrictions. In general, high numbers of images per class, deeper networks and dissimilar classes lead to higher network accuracy.

A softmax regression layer (i.e. multinomial logistic regression layer) is applied to each of the channel patches to convert the values in the feature map to probabilities.

After this final transformation by the softmax regression, a value at location location (x, y) in a channel C in the final feature map contains the probability, P(x, y), that the pixel at location location (x, y) in the input image patch belongs to the tumor type detected by channel C.

It will be appreciated that the number of convolution and deconvolution layers can be increased or decreased as desired and subject memory limitations of the hardware running the neural network.

We train the neural network using mini-batch gradient descent. The learning rate is decreased from an initial rate of 0.1 using exponential decay. We prevent neural network overfitting by using the “dropout” procedure described by Srivastava et al 2014 [2017], the full contents of which are incorporated herein by reference. Training the network may be done on a GPU, CPU or a FPGA using any one of several available deep learning frameworks. For our current implementation, we are using Google Tensor-flow, but the same neural network could have been implemented in another deep learning framework such as Microsoft CNTK.

The neural network outputs probability maps of size N×N×K, where N is the width and height in pixels of the input patches and K is the number of classes that are being detected. These output patches are stitched back together into a probability map of size W×H×K, where W and H are the width and height of the original WSI before being split into patches.

The probability maps can then be collapsed to a W×H label image by recording the class index with maximum probability at each location (x, y) in the label image.

In its current implementation, our neural network assigns every pixel to one of three classes: non-tumor, invasive tumor and in situ tumor.

When multiple tumor classes are used, the output image can be post-processed into a simpler binary classification of non-tumor and tumor, i.e. the multiple tumor classes are combined. The binary classification may be used as an option when creating images from the base data, while the multi-class tumor classification is retained in the saved data.

While the above description of a particular implementation for our invention has concentrated on a specific approach using a CNN, it will be understood that our approach can be implemented in a wide variety of different types of convolutional neural network. In general, any neural network that uses convolution to detect increasingly complex features and subsequently uses transpose convolutions (“deconvolutions”) to upscale the feature maps back to the width and height of the input image should be suitable.

Example 1

FIGS.2A-2Dare color images in operation and show the difference between patch-level predictions such as those generated by Google's CNN solution for the Camelyon competition (Liu et al. 2017) and the pixel-level predictions for tumor class generated by the presently described CNN.

FIGS.2A and2Bare copied fromFIG.7of Liu et al 2017, whereasFIG.2CandFIG.2Dare comparable tiles for an example according to the presently described CNN as applied to classify by tumor tissue class (not including ink/no-ink).

FIG.2Ais a patch from an H&E-stained WSI in which the cluster of larger, dark purple cells in the bottom right quadrant is a tumor, while the smaller dark purple cells are lymphocytes.

FIG.2Bis a tumor probability heatmap generated by the CNN of Liu et al 2017 which the authors state accurately identifies the tumor cells, while ignoring the connective tissue and lymphocytes.

FIG.2Cis a raw image patch from an example WSI to which the CNN method embodying the invention is applied. As well as the raw image,FIG.2Cshows outlines drawn manually by a pathologist (solid red perimeter lines). In addition, with reference toFIG.2D,FIG.2Calso shows results from the presently described CNN method (first areas shaded pink with pink perimeter lines correspond to a first tumor type, i.e. the tumor type shown red inFIG.2D; second areas shaded yellow with pink perimeter lines correspond to a second tumor type, i.e. the tumor type shaded blue inFIG.2D).

FIG.2Dis a tumor probability heatmap generated by our CNN. It can be seen how our approach of pixel-level prediction produces areas with smooth perimeter outlines. For our heatmap, different (arbitrarily chosen) colors indicate different classes, namely green for non-tumor, red for a first tumor type and blue for a second tumor type.

Example 2

FIGS.3A-3Bare color images in operation and show an example of the input RGB image patch (FIG.3A) and the final output tumor probability heat map (FIG.3B).

FIG.3Aadditionally shows the pathologist's manual outlining of invasive tumors (red outlines) and in addition overlays of our neural networks predictions (shaded pink and yellow areas) as separately shown inFIG.3B.

FIG.3Bis a tumor probability heatmap generated by our CNN. For our heatmap, different (arbitrarily chosen) colors indicate different classes, namely green for non-tumor, reddish-brown for invasive tumor (shown pink inFIG.3A), and blue for in situ tumor (shown yellow inFIG.3A). Once again, it can be seen how our approach of pixel-level prediction produces areas with smooth perimeter outlines. Moreover, it can be seen how the CNN predictions are compatible with the pathologist's manual marking. In addition, the CNN provides a further distinction between invasive and non-invasive (in situ) tissue which was not carried out by the pathologist, and is inherently part our multi-channel CNN design which can be programmed to and trained for classifying tissue into any number of different types as desired and clinically relevant.

Acquisition & Image Processing

The starting point of the method is that a tissue sample has been sectioned, i.e. sliced, and adjacent sections have been stained with different stains. The adjacent sections will have very similar tissue structure, since the sections are thin, but will not be identical, since they are of different layers.

For example, there could be 5 adjacent sections, each with a different stain, such as ER, PR, p53, HER2, H&E and Ki-67. A microscope image is then acquired of each section. Although the adjacent sections will have very similar tissue shapes, the stains will highlight different features, e.g. nucleus, cytoplasm, all features by general contrast enhancement etc.

The different images are then aligned, warped or otherwise pre-processed to map the coordinates of any given feature on one image to the same feature on the other images. The mapping will take care of any differences between the images caused by factors such as slightly different magnifications, orientation differences owing to differences in slide alignment in the microscope or in mounting the tissue slice on the slide, and so forth.

It is noted that with a coordinate mapping between different WSIs of a set comprising differently stained adjacent sections, the WSIs can be merged into a single composite WSI from which composite patches may be extracted for processing by the CNN, where such composite patches would have dimensions N×N×3m, where ‘m’ is the number of composited WSIs forming the set.

Some standard processing of the images is then carried out. These image processing steps may be carried out on the WSI level or at the level of individual image patches. The images may be converted from color to grayscale if the CNN is configured to operate on monochrome rather than color images. The images may be modified by applying a contrast enhancement filter. Some segmentation may then be performed to identify common tissue areas in the set of images or simply to reject background that does not relate to tissue. Segmentation may involve any or all of the following image processing techniques:1. Variance based analysis to identify the seed tissue areas2. Adaptive thresholding3. Morphological operations (e.g. blob analysis)4. Contour identification5. Contour merging based on proximity heuristic rules6. Calculation of invariant image moments7 Edge extraction (e.g. Sobel edge detection)8. Curvature flow filtering9. Histogram matching to eliminate intensity variations between serial sections10. Multi-resolution rigid/affine image registration (gradient descent optimizer)11. Non-rigid deformation/transformation12. Superpixel clustering

It will also be understood that image processing steps of the above kind can be carried on the WSIs or on individual patches after patch extraction. In some cases, it may be useful to carry out the same type of image processing both before and after patch extraction, i.e. as CNN pre-processing and CNN post-processing respectively. That is, some image processing may be done on the WSI before patch extraction and other image processing may be done on a patch after its extraction from the WSI.

These image processing steps are described by way of example and should not be interpreted as being in any way limitative on the scope of the invention. For example, the CNN could work directly with color images if sufficient processing power is available.

Training & Prediction

FIG.4is a flow diagram showing the steps involved in training the CNN.

In Step S40, training data is retrieved containing WSIs for processing which have been annotated by a clinician to find, outline and classify tumors. The clinician's annotations represent the ground truth data.

In Step S41, the WSIs are broken down into image patches, which are the input image patches for the CNN. That is, image patches are extracted from the WSI.

In Step S42, the image patches are pre-processed as described above. (Alternatively, or in addition, the WSIs could be pre-processed as described above prior to Step S41.)

In Step S43, initial values are set for the CNN weights, i.e. the weights between layers.

In Step S44, each of a batch of input image patches is input into the CNN and processed to find, outline and classify the patches on a pixel-by-pixel basis as described further above with reference toFIGS.1A and1B. The term outline here is not necessarily strictly technically the right term to use, since our method identifies each tumor (or tumor type) pixel, so it is perhaps more accurate to say that the CNN determines tumor areas for each tumor type.

In Step S45, the CNN output image patches are compared with the ground truth data. This may be done on a per-patch basis. Alternatively, if patches have been extracted that cover the entire WSI, then this may be done at the WSI level, or in sub-areas of the WSI made up of a contiguous batch of patches, e.g. one quadrant of the WSI. In such variants, the output image patches can be reassembled into a probability map for the entire WSI, or contiguous portion thereof, and the probability map can be compared with the ground truth data both by the computer and also by a user visually if the probability map is presented on the display as a semi-transparent overlay to the WSI, for example.

In Step S46, the CNN then learns from this comparison and updated the CNN weights, e.g. using a gradient descent approach. The learning is thus fed back into repeated processing of the training data as indicated inFIG.4by the return loop in the process flow, so that the CNN weights can be optimized.

After training, the CNN can be applied to WSIs independently of any ground truth data, i.e. in live use for prediction.

FIG.5is a flow diagram showing the steps involved in prediction using the CNN.

In Step S50, one or more WSIs are retrieved for processing, e.g. from a laboratory information system (LIS) or other histological data repository. The WSIs are pre-processed, for example as described above.

In Step S51, image patches are extracted from the or each WSI. The patches may cover the entire WSI or may be a random or non-random selection.

In Step S52, the image patches are pre-processed, for example as described above.

In Step S53, each of a batch of input image patches is input into the CNN and processed to find, outline and classify the patches on a pixel-by-pixel basis as described further above with reference toFIGS.1A and1B. The output patches can then be reassembled as a probability map for the WSI from which the input image patches were extracted. The probability map can be compared with the WSI both by the computer apparatus in digital processing and also by a user visually, if the probability map is presented on the display as a semi-transparent overlay on the WSI or alongside the WSI, for example.

In Step S54, the tumor areas are filtered excluding tumors that are likely to be false positives, for example areas that are too small or areas that may be edge artifacts.

In Step S55, a scoring algorithm is run. The scoring is cell specific and the score may be aggregated for each tumor, and/or further aggregated for the WSI (or sub-area of the WSI).

In Step S56, the results are presented to a pathologist or other relevantly skilled clinician for diagnosis, e.g. by display of the annotated WSI on a suitable high-resolution monitor.

In Step S57, the results of the CNN, i.e. the probability map data and optionally also metadata relating to the CNN parameters together with any additional diagnostic information added by the pathologist, are saved in a way that is linked to the patient data file containing the WSI, or set of WSIs, that have been processed by the CNN. The patient data file in the LIS or other histological data repository is thus supplemented with the CNN results.

Margin Assessment

FIG.6is a flow diagram showing the steps involved in prediction using the CNN.

In Step S1, one or more WSIs of inked and stained tissue samples are retrieved for processing, e.g. using a laboratory information system (LIS) to retrieve the data from a virtual slide library or other histological data repository.

In Step S2, the WSIs are pre-processed, for example as described above.

In Step S3, a first CNN is applied to perform the 3-way tissue type classification (non-tumor, invasive tumor, in situ tumor). Image patches are extracted from the one or more WSI. The patches may cover the entire WSI or may be a random or non-random selection. The image patches may be pre-processed, for example as described above. Each of a batch of input image patches is input into the CNN and processed to classify the patches on a pixel-by-pixel basis as described further above with reference toFIGS.1A and1B. The output patches are reassembled as a probability map of the tumor locations.

In Step S4, a second CNN is applied to perform the binary classification to recognize ink pixels (ink, no-ink). A probability map is generated as described for Step S3, where this time the probability map identifies the ink locations, i.e. the boundary of the tissue mass.

In Step S5, some post-processing of the two probability maps is performed to filter the probability values according to simple thresholding or some more complex numerical processing. For example, segmentation may be applied to the probability values to identify an ink boundary and then adjust the probability values of pixels of ink categories accordingly. Post-processing may also use segmentation to exclude tumor areas that are likely to be false positives, for example areas that are too small or areas that may be edge artifacts. If desired the two probability maps output from Steps S3and S4could be merged or combined into a single probability map and the post-processing applied to the merged probability map.

As described further above, Steps S3and S4are effectively merged in other embodiments in which a single CNN performs a 6-way classification and generates a single probability map, in which case the post-processing of Step S5would be performed on the single probability map. (It is also noted that the probability maps may also be used to generate visualizations in a viewing application, e.g. as overlays for the histological image, which could be presented to a pathologist together with the margin assessment data output from the process flow ofFIG.6.)

In Step S6, a mask is created by assigning each pixel to one of the six classes. (It is also noted that the mask may also be used to generate visualizations in a viewing application, e.g. as one or more overlays for the histological image, such as an ink boundary overlay or a tumor pixel overlay. These overlays could be presented to a pathologist together with the margin assessment data output from the process flow ofFIG.6.)

In Step S7, it is determined whether there is more than a threshold number c of pixels in the tumor-and-ink classes. If ‘no’, the margin status is set as ‘negative’ in Step S8and the process flow proceeds to compute margin distance, whereas if ‘yes’ the margin status is set as ‘positive’ in Step S12and the process flow proceeds to compute margin involvement, i.e. the extent to which tumorous tissue has encroached on the boundary of the tissue mass excised by the resection surgery. The threshold text could apply a single threshold to the sum of pixels in all tumor-and-ink classes, or separate thresholds for separate sums of each tumor-and-ink class. For example, in the latter case, if either the “invasive tumor and ink” or the “in situ tumor on ink” classes are detected at a level higher than a respective threshold values (which may be the same or different), the threshold test result is ‘yes’ and the margin status is “positive”, whereas if the level is equal to or lower in both threshold comparisons, then the test result is no and the margin status is “negative”. The threshold(s) can be set to empirically defined value(s) or any other suitable value(s).

Following the negative branch, in Step S9, margin distance is computed and, in Step S10, tested against a suitable threshold. If pixels in the tumor-on-ink classes are detected at a level below the threshold c then there is a margin and thus the margin status must be either “clear” or “close” depending on the margin distance. In Step S11, if margin distance is below the threshold, then the status is set to negative/close. In Step S14, if margin distance is below the threshold, then the status is set to negative/clear. We define margin distance as the closest distance between tumor tissue and the inked surface. One can define the set S1as containing the coordinates (xt, yt)∈R2of all the tumor pixels and the set S2as containing the coordinates(xm, ym)∈R2of all the margin pixels. Therefore, the problem of computing the margin distance is the problem of computing the minimum distance between two sets of planar points. It has been shown that this measure can be computed in O(n Log n) time and can therefore be computed quickly for even large sets of points (Toussaint and Bhattacharya 1983). If the margin distance Dm is below a specified threshold θ, as may be set by the user, the margin status is returned as “close”. If Dm≥θ, the margin status is returned as “clear”. Of course, the actual computed margin distance may be output. This may be a simple number representing the smallest distance, or a more sophisticated output providing additional data such as separate distances for smallest and largest margin distances, statistical values such as average margin distance and its probability distribution, e.g. in terms of standard deviations.

Following the positive branch, in Step S13margin involvement is computed. Since pixels in the tumor-and-ink classes have been detected at a level at or above the, or one of the, thresholds c, then there is no tumor-free margin, i.e. there is a positive margin. For positive margins, the margin involvement may be computed as the total length of tumor cells on the ink surface. This is equivalent to finding the maximum distance between two pixels belonging to each tumor and summing the result. Since this is problem is equivalent to the minimum distance problem it also can be solved in O(n Log n) time (Vaidya 1989). The current College of American Pathologists guidelines only ask for approximate extent to be reported. One scheme for classifying the extent of margin involvement based on the total lengths is as follows:Unifocal: one focal area of carcinoma at the margin with a maximum distance less than a threshold of, for example, <4 mmMultifocal: two or more foci of carcinoma at the margin, with a sum of their maximum distances being less than the unifocal thresholdMinimal/moderate: as for unifocal or multifocal but with the sum of the maximum distances being less than a second threshold that is greater than the threshold employed for the unifocal and multifocal classifications, for example <5 mm.Extensive: carcinoma present at the margin over a broad front (sum of maximum distances >5 mm)

If desired, unifocal and multifocal may be merged into a single class labeled focal.

Finally, the process flow concludes with Step S15where the margin status is reported, i.e. output, e.g. to a display and/or by saving to a file.

In summary, we have described a computer-automated method for image processing a histological image of a tissue section obtained from a tumor mass extracted by resection and painted with an ink to highlight its surface region. One or two convolutional neural networks are trained to distinguish between tissue classes that distinguish between ink and no-ink as well as tumor and no-tumor. The trained neural network or networks are applied to the histological image to generate an output image whose pixels are assigned to the tissue classes. Positive or negative margin status of the tissue section can then be determined dependent respectively on the presence or absence of tumor-and-ink classified pixels. In addition, the positive and negative margin statuses can be examined further to generate sub-statuses for margin involvement (positive sub-statuses) and margin distance (negative sub-statuses) by computing additional parameters based on classification-specified inter-pixel distances.

Tumor Area Filtering & Ranking

FIG.7is a flow diagram of according to an embodiment of the disclosure.

Step S71provides an image data file containing image data of a WSI, as may have been generated by a slide scanner. It will be appreciated the image data file may include multiple images, e.g. one for each of a plurality of stains, or one for each of a different depth in the sample (a so-called z-stack) obtained by stepping the focal plane of the microscope through a transparent or semi-transparent sample of finite depth.

Step S72is an optional step where some CNN pre-processing may be performed, as described by way of example further above, such as variance-based analysis, adaptive thresholding, morphological operations and so forth.

Step S73runs the above-described CNN, in particular as described with reference to Steps S51to S54ofFIG.5. A pixel-by-pixel classification of tissue type is performed to mark tumor pixels, followed by segmentation to outline tumors (i.e. tumor areas). The tissue type is a classification by carcinoma type. For the segmentation, it is generally the case that contiguous tumor pixels, i.e. ones that are touching each other or in close proximity to each other, belong to a common tumor. More complex segmentation criteria will however usually be included to improve reliability, e.g. to identify two touching tumors of different pixel classifications, e.g. associated with two different cancerous cell classifications. The CNN assigns each pixel a probability, which is the probability vector representing the probability of the pixel belonging to each of the N classes that the CNN has been trained to detect. For example, in the case of a CNN trained to distinguish between invasive, in situ and non-tumor areas, each pixel will be assigned a vector of length 3. A pixel at location k may have a probability vector [0.1, 0.2, 0.7] indicating there is a 10% probability the pixel is in an invasive area, a 20% probability it is in an in situ area and a 70% probability it is in a non-tumor area.

In Step S74, the data generated by the tumor-finding CNN in Step73, i.e. the tumor-specific data, is used to compute a set of summary statistics for each tumor. For example, for each tumor, a score may be computed as the mathematical average of the above-mentioned probability values for all the pixels contained in that tumor (area). Some other summary statistic such as median, weighted average, etc. may also be used to compute a score. Other measures may be used for example dimensional or morphological attributes of a tumor, such as tumor area as measured by the number of pixels in the tumor or shape of tumor area or prevalence of a certain pixel classification such as invasive tumor and in situ tumor. Usually for each tumor the average and standard deviation of tumor probability, tumor area and length of the tumors' greatest dimension will be included. Tumor areas are not necessarily from a single slide; they may belong to separate slides, e.g. the tissue samples of two slides may be stained with different stains and thus highlight different classes of tumor cells, so that some tumors are identified in a first slide and other tumors in a second slide. Optionally, more sophisticated scores can be computed. For example, it is possible to predict patient risk using a CNN trained on a combination of histology image data (i.e. tumors identified in image data) and patient-specific genetic (i.e. genomic) data. A CNN of this kind is described in Mobadersany et al 2018. In other implementations, the score can be computed using traditional image processing techniques (e.g. as listed further above in connection with segmentation) applied to the tumors identified by the CNN. For example, shape and texture measures can be combined with genetic data to create a set of statistical measures to include in the summary statistics. A support vector machine or random forest algorithm may use these features to predict risk of metastasis. We define the risk of metastasis as the probability that cells from this tumor will metastasize to other parts of the body. Either way a metastasis risk score will be computed and associated with each tumor area.

Step S75carries out a filtering of the tumors based on the summary statistics computed and compiled in Step S74. The filter may operate by a simple pass through the list of tumors comparing a the value of a single one of the parameters in the summary statistics, or a compound value derived from a logical combination and/or formula including multiple ones of the parameters present in the summary statistics, with a threshold value and then removing or keeping the tumor object based on the comparison result. For example, the filter, which may be configured by the pathologist or be pre-set, may choose to pass only tumors with a maximum dimension above a threshold value, e.g. 100 micrometers, that have an average probability above a threshold value, e.g. 50%. Another example filter is to pass only tumors classified as invasive and among invasive tumors only those with an average probability above a threshold value of, say, 80%.

Step S76takes the set of tumors passed by the filter in Step S75and ranks them in order. A ranking of the tumors can then be carried out based on the summary statistics and applying a set of criteria. Pre-sets of standard ranking approaches may be provided to the user, allowing the user to select one of the pre-sets. Moreover, the user may be provided with a user interface to define which criteria are to be applied. In the case of scalar valued criteria, e.g. based on a length dimension or area or an integer count (e.g. cell count), the user may set threshold values, or value ranges for these criteria. The ranking order may be based on a composite score indicating importance for patient survival, e.g. 5-year survival probability, or be a simple single parameter ranking, e.g. based on a size parameter of the tumor such as area or maximum dimension, or a morphological parameter such as roundness.

Step S77generates a visualization of the slide image and displays it to the user in a GUI window of a display device. The visualization takes account of the filtering and also the ranking. In particular, what are deemed to be the most important tumors in the WSI are displayed in a clinically relevant way with their summary statistics also being available either in all cases as part of the overall display, or selectively responsive to a GUI command from the user, e.g. ‘clicking’ with a cursor on a tumor of interest in the WSI to generate a pop-up window showing the statistics for that tumor in tabular or other suitable form. This approach allows potentially significant tumors to be highlighted and presented to the user in a way that provides the user with ranking information among the potentially significant tumors as well as statistical summaries of these tumors. The statistical summaries can present individual parameter values for each tumor, in particular those used as filtering and/or ranking criteria, as well as compound parameters such as a ranking number or significance score that are computed from formulaic and/or Boolean logical combinations of multiple filtering and/or ranking criteria.

Typically, the image displayed will either be in the form of a combined overlay view or a multi-tile view. In an overlay view the raw data (possibly processed) is displayed with the segmentation data overlaid on top, where the segmentation data is translated for the visualization into a shading and/or outline of each tumor. The shading or outlining may be color-coded, e.g. by tumor classification. Non-tumorous areas of tissue may not be marked at all or may be shaded with a color wash of high transparency, e.g. a blue wash. In a multi-tile view, what were the different layers in the overlay view are displayed side-by-side as tiles, so there will be a tile showing raw image data (possibly processed) and segmentation data of the filtered tumor areas. If desired a separate segmentation data tile may be displayed for each tumor classification type. The presentation of the tumor areas in the display takes account of the filtering performed in Step S75and preferably also the ranking performed in Step S76. Factors such as the tumor scores, the classified tumor type and any other parameters in the summary statistics associated with tumor areas can be used singly or in combination to configure the display.

There are several options for displaying the detected, filtered and ranked tumors to the user.

In a WSI view, one way to display the tumor information is to overlay a series of markers, which may comprise thumbnail images and/or text information, over a low-resolution image of the WSI. The markers can be sorted by, or expressly indicate, a ranking. A pathologist is provided with suitable GUI tools for selecting a tumor through selecting its marker, and optionally also for navigating through the tumors by ranking, both from higher ranked tumors to lower ranked tumors and the other way. Suitable GUI tools include: keyboard shortcuts; keyboard up and down, or left and right, arrow keys, keyboard page up and page down keys, mouse navigation (e.g. scrolling up or down with a scroll wheel) or other input devices (voice navigation, multitouch gesture in a touch sensor, etc.). Selecting the ranking marker in the GUI may prompt display of that tumor's summary statistics, or a subset thereof, and/or display of a higher resolution view of the tumor either in a pop-up linked to the display of the tumor in the low-resolution image, or in a separate, higher resolution viewing pane. The user may cause display of a high, e.g. full native, resolution image of the tumor by a suitable GUI command, such as a keyboard command, a mouse command (e.g. double-click) or other suitable input. A corresponding GUI command is provided to navigate back to a low-resolution WSI view. The visualization application preferably provides GUI controls that permit the user to step through the tumors up or down by ranking order in both low-resolution and high-resolution views.

One example WSI view would be for all tumors that have passed the filter of Step S76to be displayed (i.e. their segmentation data displayed), together with a ranking marker label1,2,3etc. Clicking on the ranking marker label (or the tumor) may then generate a pop-up listing a selected set of the summary statistics, in particular those that are used by the filter and/or a thumbnail view of the tumor at a resolution higher than that in the WSI view. Alternatively, the view may be a split-screen view with the overlay image or image tiles being displayed in one part of the screen and a table of the filtered tumors in another part of the screen. The table may be presented initially sorted by ranking, but the GUI may also have the facility to be user re-sortable by any other column, or combination of multiple columns, where the other columns may be any criterion from the summary statistics or filter criteria, such as tumor area, tumor classification, metastatic probability etc. For example, the sorting could be by tumor classification followed by tumor area. In the case of a multi-slide image file, the sorting could be by slide number followed by some other parameter(s).

In a multi-resolution view comprising one viewing pane at lower resolution (e.g. 10× magnification), typically reproducing a WSI, and another viewing pane at higher resolution (e.g. 60× magnification, i.e. 6× or 600% zoom relative to the 10× view). For example, the initial view presented may be a WSI low resolution image pane and a high-resolution image pane centered on the tumor area with the top ranking. A step-through down arrow (or pair of down and up arrows) or other suitable GUI button or button combination, such as a physical or virtual scroll wheel, may then allow the user to step through the filtered tumor areas by ranking one by one. The GUI may allow the user to adjust the resolution in the high-resolution image pane through user input. The GUI may also select the initial resolution for the high-resolution image pane so that the tumor is sized to substantially fill the high-resolution viewing pane.

The visualization application may thus determine what is displayed to the user, what tumors are highlighted with segmentation data and summary statistics and, in the case of a selective view, the time sequence, i.e. order, in which the tumors are displayed to the user.

Another way of displaying the tumor areas that is particularly suited to slide sets is to create a montage where low-resolution images of the tumors along with their summary statistics are displayed as sorted tiles. The sorting may be by way of displaying a one-dimensional (1D) list, or a two-dimensional (2D) grid. It is even possible to arrange the tumors in three dimensions, e.g. with the aid of virtual reality goggles. The user may navigate the 1D list or 2D tile array using keystrokes, mouse scrolling or other input modalities (voice, touch, etc.). Once a tile is selected the pathologist can quickly navigate to a high-resolution version of the tumor in the appropriate slide to make a more in-depth verification of the tumor optionally with comparison to the results of the CNN analysis as presented by the summary statistics and segmentation data, and optionally also perform additional processing on the tumor by applying one or more further analysis algorithms to the tumor that may assist diagnosis.

In Step S78, after viewing the filtered and ranked tumors, the pathologist has the further option of selecting any one of these tumors (or any subset of the ranked tumors, or indeed the full set of filtered tumors) for further study.

In Step S79, this further study is performed. By further study we mean applying one or more additional algorithms to provide further information on the selected tumors to assist with their diagnosis. This may be to help confirm a provisional diagnosis or to make an initial diagnosis for the pathologist to consider as a starting point.

For example, for breast cancer, in Step S78a pathologist may select for tumors containing invasive breast carcinomas and then in Step S79apply an algorithm that computes the mitotic count of a tumor. That is, the visualization application includes such an algorithm. Another breast cancer example relevant for image data from an IHC stained slide is to apply an algorithm to compute scores for the expression of one or more diagnostically relevant genes, such as ER, PR, or HER2 in a selected tumor.

In summary, the visualization is configured to prompt the user to examine areas in the image that contain potentially significant tumors. In particular, compared with a traditional visualization application, it is not necessary for the user to perform a manual visual scan over the whole slide area at low resolution, zooming in to, and back out of, a higher resolution view whenever a potential tumor area is encountered. The automated pre-processing of the visualization presented to the pathologist based on CNN-aided filtering and ranking as described above thus reduces the amount of time needed by the pathologist to review a slide and make appropriate diagnoses as well as reducing the chance that an important tumor is missed by human error.

Example Embodiment

In one embodiment, a system for identifying tumors in a histological image is configured to perform a method to identify tumors in a histological image. The method includes receiving a histological image including a two-dimensional array of pixels and applying a convolutional neural network to generate an output image having a two-dimensional array of pixels and a mapping of the pixels in the output image to the pixels of the histology image. The output image is generated by assigning one of a plurality of tissue classes to each pixel, wherein the plurality of tissue classes includes at least one class representing non-tumorous tissue and at least one class representing tumorous tissue.

Next, the system generates a segmentation mask from the output image. The segmentation mask identifies areas occupied by individual tumors. The system also computes summary statistics for each tumor and the system is configured to filter the segmentation mask to select and deselect individual tumors according to the to the summary statistics for each tumor. The system is also configured to create a visualization of the histological image in accordance with the edited segmentation mask and receive a selection of an individual tumor in the filtered segmentation mask and then coordinate execution of an additional computational diagnostic process on the currently selected tumor.

In one embodiment, the additional computational diagnostic process comprises sending a processing task to the additional computational diagnostic process and receiving results of the processing task from the additional computational diagnostic. The additional computational diagnostic process can be executed locally or on a remote processor enabled device communicatively coupled via a data communication network. A portion of the processing task may include the histological image. The processing task may also include metadata related to the histological image and in some circumstances, the metadata related to the histological image may include the segmentation mask. Additionally, the metadata related to the histological image may also include one or more of patient information, stain information, staining protocol information, and scanning protocol information.

CNN Computing Platform

The proposed image processing may be carried out on a variety of computing architectures, in particular ones that are optimized for neural networks, which may be based on CPUs, GPUs, TPUs, FPGAs and/or ASICs. In some embodiments, the neural network is implemented using Google's Tensorflow software library running on Nvidia GPUs from Nvidia Corporation, Santa Clara, California, such as the Tesla K80 GPU. In other embodiments, the neural network can run on generic CPUs. Faster processing can be obtained by a purpose-designed processor for performing CNN calculations, for example the TPU disclosed in Jouppi et al 2017, the full contents of which is incorporated herein by reference.

FIG.8shows the TPU of Jouppi et al 2017, being a simplified reproduction of Jouppi'sFIG.1. The TPU100has a systolic matrix multiplication unit (MMU)102which contains 256×256 MACs that can perform 8-bit multiply-and-adds on signed or unsigned integers. The weights for the MMU are supplied through a weight FIFO buffer104that in turn reads the weights from a memory106, in the form of an off-chip 8 GB DRAM, via a suitable memory interface108. A unified buffer (UB)110is provided to store the intermediate results. The MMU102is connected to receives inputs from the weight FIFO interface104and the UB110(via a systolic data setup unit112) and outputs the 16-bit products of the MMU processing to an accumulator unit114. An activation unit116performs nonlinear functions on the data held in the accumulator unit114. After further processing by a normalizing unit118and a pooling unit120, the intermediate results are sent to the UB110for resupply to the MMU102via the data setup unit112. The pooling unit120may perform maximum pooling (i.e. maxpooling) or average pooling as desired. A programmable DMA controller122transfers data to or from the TPU's host computer and the UB110. The TPU instructions are sent from the host computer to the controller122via a host interface124and an instruction buffer126.

It will be understood that the computing power used for running the neural network, whether it be based on CPUs, GPUs or TPUs, may be hosted locally in a clinical network, e.g. the one described below, or remotely in a data center.

Network & Computing & Scanning Environment

The proposed computer-automated method operates in the context of a laboratory information system (LIS) which in turn is typically part of a larger clinical network environment, such as a hospital information system (HIS) or picture archiving and communication system (PACS). In the LIS, the WSIs will be retained in a database, typically a patient information database containing the electronic medical records of individual patients. The WSIs will be taken from stained tissue samples mounted on slides, the slides bearing printed barcode labels by which the WSIs are tagged with suitable metadata, since the microscopes acquiring the WSIs are equipped with barcode readers. From a hardware perspective, the LIS will be a conventional computer network, such as a local area network (LAN) with wired and wireless connections as desired.

FIG.9shows an example computer network which can be used in conjunction with embodiments of the invention. The network150comprises a LAN in a hospital152. The hospital152is equipped with a number of workstations154which each have access, via the local area network, to a hospital computer server156having an associated storage device158. A LIS, HIS or PACS archive is stored on the storage device158so that data in the archive can be accessed from any of the workstations154. One or more of the workstations154has access to a graphics card and to software for computer-implementation of methods of generating images as described hereinbefore. The software may be stored locally at the or each workstation154or may be stored remotely and downloaded over the network150to a workstation154when needed. In other example, methods embodying the invention may be executed on the computer server with the workstations154operating as terminals. For example, the workstations may be configured to receive user input defining a desired histological image data set and to display resulting images while CNN analysis is performed elsewhere in the system. Also, a number of histological and other medical imaging devices160,162,164,166are connected to the hospital computer server156. Image data collected with the devices160,162,164,166can be stored directly into the LIS, HIS or PACS archive on the storage device156. Thus, histological images can be viewed and processed immediately after the corresponding histological image data are recorded. The local area network is connected to the Internet168by a hospital Internet server170, which allows remote access to the LIS, HIS or PACS archive. This is of use for remote accessing of the data and for transferring data between hospitals, for example, if a patient is moved, or to allow external research to be undertaken.

FIG.10is a block diagram illustrating an example computing apparatus500that may be used in connection with various embodiments described herein. For example, computing apparatus500may be used as a computing node in the above-mentioned LIS or PACS system, for example a host computer from which CNN processing is carried out in conjunction with a suitable GPU, or the TPU shown inFIG.8.

Computing apparatus500can be a server or any conventional personal computer, or any other processor-enabled device that is capable of wired or wireless data communication. Other computing apparatus, systems and/or architectures may be also used, including devices that are not capable of wired or wireless data communication, as will be clear to those skilled in the art.

Computing apparatus500preferably includes one or more processors, such as processor510. The processor510may be for example a CPU, GPU, TPU or arrays or combinations thereof such as CPU and TPU combinations or CPU and GPU combinations. Additional processors may be provided, such as an auxiliary processor to manage input/output, an auxiliary processor to perform floating point mathematical operations (e.g. a TPU), a special-purpose microprocessor having an architecture suitable for fast execution of signal processing algorithms (e.g., digital signal processor, image processor), a slave processor subordinate to the main processing system (e.g., back-end processor), an additional microprocessor or controller for dual or multiple processor systems, or a coprocessor. Such auxiliary processors may be discrete processors or may be integrated with the processor510. Examples of CPUs which may be used with computing apparatus500are, the Pentium processor, Core i7 processor, and Xeon processor, all of which are available from Intel Corporation of Santa Clara, California. An example GPU which may be used with computing apparatus500is Tesla K80 GPU of Nvidia Corporation, Santa Clara, California.

Processor510is connected to a communication bus505. Communication bus505may include a data channel for facilitating information transfer between storage and other peripheral components of computing apparatus500. Communication bus505further may provide a set of signals used for communication with processor510, including a data bus, address bus, and control bus (not shown). Communication bus505may comprise any standard or non-standard bus architecture such as, for example, bus architectures compliant with industry standard architecture (ISA), extended industry standard architecture (EISA), Micro Channel Architecture (MCA), peripheral component interconnect (PCI) local bus, or standards promulgated by the Institute of Electrical and Electronics Engineers (IEEE) including IEEE 488 general-purpose interface bus (GPM), IEEE 696/S-100, and the like.

Computing apparatus500preferably includes a main memory515and may also include a secondary memory520. Main memory515provides storage of instructions and data for programs executing on processor510, such as one or more of the functions and/or modules discussed above. It should be understood that computer readable program instructions stored in the memory and executed by processor510may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in and/or compiled from any combination of one or more programming languages, including without limitation Smalltalk, C/C++, Java, JavaScript, Perl, Visual Basic, .NET, and the like. Main memory515is typically semiconductor-based memory such as dynamic random access memory (DRAM) and/or static random access memory (SRAM). Other semiconductor-based memory types include, for example, synchronous dynamic random access memory (SDRAM), Rambus dynamic random access memory (RDRAM), ferroelectric random access memory (FRAM), and the like, including read only memory (ROM).

The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

Secondary memory520may optionally include an internal memory525and/or a removable medium530. Removable medium530is read from and/or written to in any well-known manner. Removable storage medium530may be, for example, a magnetic tape drive, a compact disc (CD) drive, a digital versatile disc (DVD) drive, other optical drive, a flash memory drive, etc.

Removable storage medium530is a non-transitory computer-readable medium having stored thereon computer-executable code (i.e., software) and/or data. The computer software or data stored on removable storage medium530is read into computing apparatus500for execution by processor510.

The secondary memory520may include other similar elements for allowing computer programs or other data or instructions to be loaded into computing apparatus500. Such means may include, for example, an external storage medium545and a communication interface540, which allows software and data to be transferred from external storage medium545to computing apparatus500. Examples of external storage medium545may include an external hard disk drive, an external optical drive, an external magneto-optical drive, etc. Other examples of secondary memory520may include semiconductor-based memory such as programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), or flash memory (block-oriented memory similar to EEPROM).

As mentioned above, computing apparatus500may include a communication interface540. Communication interface540allows software and data to be transferred between computing apparatus500and external devices (e.g. printers), networks, or other information sources. For example, computer software or executable code may be transferred to computing apparatus500from a network server via communication interface540. Examples of communication interface540include a built-in network adapter, network interface card (MC), Personal Computer Memory Card International Association (PCMCIA) network card, card bus network adapter, wireless network adapter, Universal Serial Bus (USB) network adapter, modem, a network interface card (NIC), a wireless data card, a communications port, an infrared interface, an IEEE 1394 fire-wire, or any other device capable of interfacing system550with a network or another computing device. Communication interface540preferably implements industry-promulgated protocol standards, such as Ethernet IEEE 802 standards, Fiber Channel, digital subscriber line (DSL), asynchronous digital subscriber line (ADSL), frame relay, asynchronous transfer mode (ATM), integrated digital services network (ISDN), personal communications services (PCS), transmission control protocol/Internet protocol (TCP/IP), serial line Internet protocol/point to point protocol (SLIP/PPP), and so on, but may also implement customized or non-standard interface protocols as well.

Software and data transferred via communication interface540are generally in the form of electrical communication signals555. These signals555may be provided to communication interface540via a communication channel550. In an embodiment, communication channel550may be a wired or wireless network, or any variety of other communication links. Communication channel550carries signals555and can be implemented using a variety of wired or wireless communication means including wire or cable, fiber optics, conventional phone line, cellular phone link, wireless data communication link, radio frequency (“RF”) link, or infrared link, just to name a few.

Computer-executable code (i.e., computer programs or software) is stored in main memory515and/or the secondary memory520. Computer programs can also be received via communication interface540and stored in main memory515and/or secondary memory520. Such computer programs, when executed, enable computing apparatus500to perform the various functions of the disclosed embodiments as described elsewhere herein.

In this document, the term “computer-readable medium” is used to refer to any non-transitory computer-readable storage media used to provide computer-executable code (e.g., software and computer programs) to computing apparatus500. Examples of such media include main memory515, secondary memory520(including internal memory525, removable medium530, and external storage medium545), and any peripheral device communicatively coupled with communication interface540(including a network information server or other network device). These non-transitory computer-readable media are means for providing executable code, programming instructions, and software to computing apparatus500. In an embodiment that is implemented using software, the software may be stored on a computer-readable medium and loaded into computing apparatus500by way of removable medium530, I/O interface535, or communication interface540. In such an embodiment, the software is loaded into computing apparatus500in the form of electrical communication signals555. The software, when executed by processor510, preferably causes processor510to perform the features and functions described elsewhere herein.

I/O interface535provides an interface between one or more components of computing apparatus500and one or more input and/or output devices. Example input devices include, without limitation, keyboards, touch screens or other touch-sensitive devices, biometric sensing devices, computer mice, trackballs, pen-based pointing devices, and the like. Examples of output devices include, without limitation, cathode ray tubes (CRTs), plasma displays, light-emitting diode (LED) displays, liquid crystal displays (LCDs), printers, vacuum florescent displays (VFDs), surface-conduction electron-emitter displays (SEDs), field emission displays (FEDs), and the like.

Computing apparatus500also includes optional wireless communication components that facilitate wireless communication over a voice network and/or a data network. The wireless communication components comprise an antenna system570, a radio system565, and a baseband system560. In computing apparatus500, radio frequency (RF) signals are transmitted and received over the air by antenna system570under the management of radio system565.

Antenna system570may comprise one or more antennae and one or more multiplexors (not shown) that perform a switching function to provide antenna system570with transmit and receive signal paths. In the receive path, received RF signals can be coupled from a multiplexor to a low noise amplifier (not shown) that amplifies the received RF signal and sends the amplified signal to radio system565.

Radio system565may comprise one or more radios that are configured to communicate over various frequencies. In an embodiment, radio system565may combine a demodulator (not shown) and modulator (not shown) in one integrated circuit (IC). The demodulator and modulator can also be separate components. In the incoming path, the demodulator strips away the RF carrier signal leaving a baseband receive audio signal, which is sent from radio system565to baseband system560.

If the received signal contains audio information, then baseband system560decodes the signal and converts it to an analog signal. Then the signal is amplified and sent to a speaker. Baseband system560also receives analog audio signals from a microphone. These analog audio signals are converted to digital signals and encoded by baseband system560. Baseband system560also codes the digital signals for transmission and generates a baseband transmit audio signal that is routed to the modulator portion of radio system565. The modulator mixes the baseband transmit audio signal with an RF carrier signal generating an RF transmit signal that is routed to antenna system570and may pass through a power amplifier (not shown). The power amplifier amplifies the RF transmit signal and routes it to antenna system570where the signal is switched to the antenna port for transmission.

Baseband system560is also communicatively coupled with processor510, which may be a central processing unit (CPU). Processor510has access to data storage areas515and520. Processor510is preferably configured to execute instructions (i.e., computer programs or software) that can be stored in main memory515or secondary memory520. Computer programs can also be received from baseband processor560and stored in main memory510or in secondary memory520or executed upon receipt. Such computer programs, when executed, enable computing apparatus500to perform the various functions of the disclosed embodiments. For example, data storage areas515or520may include various software modules.

The computing apparatus further comprises a display575directly attached to the communication bus505which may be provided instead of or addition to any display connected to the I/O interface535referred to above.

Various embodiments may also be implemented primarily in hardware using, for example, components such as application specific integrated circuits (ASICs), programmable logic arrays (PLA), or field programmable gate arrays (FPGAs). Implementation of a hardware state machine capable of performing the functions described herein will also be apparent to those skilled in the relevant art. Various embodiments may also be implemented using a combination of both hardware and software.

Furthermore, those of skill in the art will appreciate that the various illustrative logical blocks, modules, circuits, and method steps described in connection with the above described figures and the embodiments disclosed herein can often be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled persons can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the invention. In addition, the grouping of functions within a module, block, circuit, or step is for ease of description. Specific functions or steps can be moved from one module, block, or circuit to another without departing from the invention.

Moreover, the various illustrative logical blocks, modules, functions, and methods described in connection with the embodiments disclosed herein can be implemented or performed with a general purpose processor, a digital signal processor (DSP), an ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor can be a microprocessor, but in the alternative, the processor can be any processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

Additionally, the steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium including a network storage medium. An exemplary storage medium can be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can also reside in an ASIC.

A computer readable storage medium, as referred to herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

Any of the software components described herein may take a variety of forms. For example, a component may be a stand-alone software package, or it may be a software package incorporated as a “tool” in a larger software product. It may be downloadable from a network, for example, a website, as a stand-alone product or as an add-in package for installation in an existing software application. It may also be available as a client-server software application, as a web-enabled software application, and/or as a mobile application.

Embodiments of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

The computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The illustrated flowcharts and block diagrams illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

Apparatus and methods embodying the invention are capable of being hosted in and delivered by a cloud computing environment. Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a provider of the service. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

Characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with the service's provider.

Broad network access: capabilities are available over a network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to demand. There is a sense of location independence in that the consumer generally has no control or knowledge over the exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elastically provisioned, in some cases automatically, to quickly scale out and rapidly released to quickly scale in. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be purchased in any quantity at any time.

Measured service: cloud systems automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency for both the provider and consumer of the utilized service.

Service Models are as follows:

Software as a Service (SaaS): the capability provided to the consumer is to use the provider's applications running on a cloud infrastructure. The applications are accessible from various client devices through a thin client interface such as a web browser (e.g., web-based e-mail). The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but has control over the deployed applications and possibly application hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, deployed applications, and possibly limited control of select networking components (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for an organization. It may be managed by the organization or a third party and may exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by several organizations and supports a specific community that has shared concerns (e.g., mission, security requirements, policy, and compliance considerations). It may be managed by the organizations or a third party and may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the general public or a large industry group and is owned by an organization selling cloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technology that enables data and application portability (e.g., cloud bursting for load-balancing between clouds).

A cloud computing environment is service oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

It will be clear to one skilled in the art that many improvements and modifications can be made to the foregoing exemplary embodiment without departing from the scope of the present disclosure.

FIG.11Ais a block diagram illustrating an example processor enabled device551that may be used in connection with various embodiments described herein. Alternative forms of the device551may also be used as will be understood by the skilled artisan. In the illustrated embodiment, the device551is presented as a digital imaging device (also referred to herein as a scanner system or a scanning system) that comprises one or more processors556, one or more memories566, one or more motion controllers571, one or more interface systems576, one or more movable stages580that each support one or more glass slides585with one or more samples590, one or more illumination systems595that illuminate the sample, one or more objective lenses600that each define an optical path605that travels along an optical axis, one or more objective lens positioners630, one or more optional epi-illumination systems635(e.g., included in a fluorescence scanner system), one or more focusing optics610, one or more line scan cameras615and/or one or more area scan cameras620, each of which define a separate field of view625on the sample590and/or glass slide585. The various elements of the scanner system551are communicatively coupled via one or more communication busses560. Although there may be one or more of each of the various elements of the scanner system551, for simplicity in the description that follows, these elements will be described in the singular except when needed to be described in the plural to convey the appropriate information.

The one or more processors556may include, for example, a central processing unit (“CPU”) and a separate graphics processing unit (“GPU”) capable of processing instructions in parallel or the one or more processors556may include a multicore processor capable of processing instructions in parallel. Additional separate processors may also be provided to control particular components or perform particular functions such as image processing. For example, additional processors may include an auxiliary processor to manage data input, an auxiliary processor to perform floating point mathematical operations, a special-purpose processor having an architecture suitable for fast execution of signal processing algorithms (e.g., digital signal processor), a slave processor subordinate to the main processor (e.g., back-end processor), an additional processor for controlling the line scan camera615, the stage580, the objective lens225, and/or a display (not shown). Such additional processors may be separate discrete processors or may be integrated with the processor556.

The memory566provides storage of data and instructions for programs that can be executed by the processor556. The memory566may include one or more volatile and persistent computer-readable storage mediums that store the data and instructions, for example, a random access memory, a read only memory, a hard disk drive, removable storage drive, and the like. The processor556is configured to execute instructions that are stored in memory566and communicate via communication bus560with the various elements of the scanner system551to carry out the overall function of the scanner system551.

The one or more communication busses560may include a communication bus560that is configured to convey analog electrical signals and may include a communication bus560that is configured to convey digital data. Accordingly, communications from the processor556, the motion controller571, and/or the interface system576via the one or more communication busses560may include both electrical signals and digital data. The processor556, the motion controller571, and/or the interface system576may also be configured to communicate with one or more of the various elements of the scanning system551via a wireless communication link.

The motion control system571is configured to precisely control and coordinate XYZ movement of the stage580and the objective lens600(e.g., via the objective lens positioner630). The motion control system571is also configured to control movement of any other moving part in the scanner system551. For example, in a fluorescence scanner embodiment, the motion control system571is configured to coordinate movement of optical filters and the like in the epi-illumination system635.

The interface system576allows the scanner system551to interface with other systems and human operators. For example, the interface system576may include a user interface to provide information directly to an operator and/or to allow direct input from an operator. The interface system576is also configured to facilitate communication and data transfer between the scanning system551and one or more external devices that are directly connected (e.g., a printer, removable storage medium) or external devices such as an image server system, an operator station, a user station, and an administrative server system that are connected to the scanner system551via a network (not shown).

The illumination system595is configured to illuminate a portion of the sample590. The illumination system may include, for example, a light source and illumination optics. The light source could be a variable intensity halogen light source with a concave reflective mirror to maximize light output and a KG-1 filter to suppress heat. The light source could also be any type of arc-lamp, laser, or other source of light. In one embodiment, the illumination system595illuminates the sample590in transmission mode such that the line scan camera615and/or area scan camera620sense optical energy that is transmitted through the sample590. Alternatively, or in combination, the illumination system595may also be configured to illuminate the sample590in reflection mode such that the line scan camera615and/or area scan camera620sense optical energy that is reflected from the sample590. Overall, the illumination system595is configured to be suitable for interrogation of the microscopic sample590in any known mode of optical microscopy.

In one embodiment, the scanner system551optionally includes an epi-illumination system635to optimize the scanner system551for fluorescence scanning. Fluorescence scanning is the scanning of samples590that include fluorescence molecules, which are photon sensitive molecules that can absorb light at a specific wavelength (excitation). These photon sensitive molecules also emit light at a higher wavelength (emission). Because the efficiency of this photoluminescence phenomenon is very low, the amount of emitted light is often very low. This low amount of emitted light typically frustrates conventional techniques for scanning and digitizing the sample590(e.g., transmission mode microscopy). Advantageously, in an optional fluorescence scanner system embodiment of the scanner system551, use of a line scan camera615that includes multiple linear sensor arrays (e.g., a time delay integration (“TDI”) line scan camera) increases the sensitivity to light of the line scan camera by exposing the same area of the sample590to each of the multiple linear sensor arrays of the line scan camera615. This is particularly useful when scanning faint fluorescence samples with low emitted light.

Accordingly, in a fluorescence scanner system embodiment, the line scan camera615is preferably a monochrome TDI line scan camera. Advantageously, monochrome images are ideal in fluorescence microscopy because they provide a more accurate representation of the actual signals from the various channels present on the sample. As will be understood by those skilled in the art, a fluorescence sample590can be labeled with multiple florescence dyes that emit light at different wavelengths, which are also referred to as “channels.”

Furthermore, because the low and high end signal levels of various fluorescence samples present a wide spectrum of wavelengths for the line scan camera615to sense, it is desirable for the low and high end signal levels that the line scan camera615can sense to be similarly wide. Accordingly, in a fluorescence scanner embodiment, a line scan camera615used in the fluorescence scanning system551is a monochrome 10 bit 64 linear array TDI line scan camera. It should be noted that a variety of bit depths for the line scan camera615can be employed for use with a fluorescence scanner embodiment of the scanning system551.

The movable stage580is configured for precise XY movement under control of the processor556or the motion controller571. The movable stage may also be configured for movement in Z under control of the processor556or the motion controller571. The moveable stage is configured to position the sample in a desired location during image data capture by the line scan camera615and/or the area scan camera. The moveable stage is also configured to accelerate the sample590in a scanning direction to a substantially constant velocity and then maintain the substantially constant velocity during image data capture by the line scan camera615. In one embodiment, the scanner system551may employ a high precision and tightly coordinated XY grid to aid in the location of the sample590on the movable stage580. In one embodiment, the movable stage580is a linear motor based XY stage with high precision encoders employed on both the X and the Y axis. For example, very precise nanometer encoders can be used on the axis in the scanning direction and on the axis that is in the direction perpendicular to the scanning direction and on the same plane as the scanning direction. The stage is also configured to support the glass slide585upon which the sample590is disposed.

The sample590can be anything that may be interrogated by optical microscopy. For example, a glass microscope slide585is frequently used as a viewing substrate for specimens that include tissues and cells, chromosomes, DNA, protein, blood, bone marrow, urine, bacteria, beads, biopsy materials, or any other type of biological material or substance that is either dead or alive, stained or unstained, labeled or unlabeled. The sample590may also be an array of any type of DNA or DNA-related material such as cDNA or RNA or protein that is deposited on any type of slide or other substrate, including any and all samples commonly known as a microarrays. The sample590may be a microtiter plate, for example a 96-well plate. Other examples of the sample590include integrated circuit boards, electrophoresis records, petri dishes, film, semiconductor materials, forensic materials, or machined parts.

Objective lens600is mounted on the objective positioner630which, in one embodiment, may employ a very precise linear motor to move the objective lens600along the optical axis defined by the objective lens600. For example, the linear motor of the objective lens positioner630may include a 50 nanometer encoder. The relative positions of the stage580and the objective lens600in XYZ axes are coordinated and controlled in a closed loop manner using motion controller571under the control of the processor556that employs memory566for storing information and instructions, including the computer-executable programmed steps for overall scanning system551operation.

In one embodiment, the objective lens600is a plan apochromatic (“APO”) infinity corrected objective with a numerical aperture corresponding to the highest spatial resolution desirable, where the objective lens600is suitable for transmission mode illumination microscopy, reflection mode illumination microscopy, and/or epi-illumination mode fluorescence microscopy (e.g., an Olympus 40×, 0.75NA or 20×, 0.75 NA). Advantageously, objective lens600is capable of correcting for chromatic and spherical aberrations. Because objective lens600is infinity corrected, focusing optics610can be placed in the optical path605above the objective lens600where the light beam passing through the objective lens becomes a collimated light beam. The focusing optics610focus the optical signal captured by the objective lens600onto the light-responsive elements of the line scan camera615and/or the area scan camera620and may include optical components such as filters, magnification changer lenses, etc. The objective lens600combined with focusing optics610provides the total magnification for the scanning system551. In one embodiment, the focusing optics610may contain a tube lens and an optional 2× magnification changer. Advantageously, the 2× magnification changer allows a native20X objective lens600to scan the sample590at 40× magnification.

The line scan camera615comprises at least one linear array of picture elements (“pixels”). The line scan camera may be monochrome or color. Color line scan cameras typically have at least three linear arrays, while monochrome line scan cameras may have a single linear array or plural linear arrays. Any type of singular or plural linear array, whether packaged as part of a camera or custom-integrated into an imaging electronic module, can also be used. For example, 3 linear array (“red-green-blue” or “RGB”) color line scan camera or a 96 linear array monochrome TDI may also be used. TDI line scan cameras typically provide a substantially better signal-to-noise ratio (“SNR”) in the output signal by summing intensity data from previously imaged regions of a specimen, yielding an increase in the SNR that is in proportion to the square-root of the number of integration stages. TDI line scan cameras comprise multiple linear arrays, for example, TDI line scan cameras are available with 24, 32, 48, 64, 96, or even more linear arrays. The scanner system551also supports linear arrays that are manufactured in a variety of formats including some with 512 pixels, some with 1024 pixels, and others having as many as 4096 pixels. Similarly, linear arrays with a variety of pixel sizes can also be used in the scanner system551. The salient requirement for the selection of any type of line scan camera615is that the motion of the stage580can be synchronized with the line rate of the line scan camera615so that the stage580can be in motion with respect to the line scan camera615during the digital image capture of the sample590.

The image data generated by the line scan camera615is stored a portion of the memory566and processed by the processor556to generate a contiguous digital image of at least a portion of the sample590. The contiguous digital image can be further processed by the processor556and the revised contiguous digital image can also be stored in the memory566.

In an embodiment with two or more line scan cameras615, at least one of the line scan cameras615can be configured to function as a focusing sensor that operates in combination with at least one of the line scan cameras615that is configured to function as an imaging sensor. The focusing sensor can be logically positioned on the same optical axis as the imaging sensor or the focusing sensor may be logically positioned before or after the imaging sensor with respect to the scanning direction of the scanner system551. In such an embodiment with at least one line scan camera615functioning as a focusing sensor, the image data generated by the focusing sensor is stored in a portion of the memory566and processed by the one or more processors556to generate focus information to allow the scanner system551to adjust the relative distance between the sample590and the objective lens600to maintain focus on the sample during scanning. Additionally, in one embodiment the at least one line scan camera615functioning as a focusing sensor may be oriented such that each of a plurality of individual pixels of the focusing sensor is positioned at a different logical height along the optical path605.

In operation, the various components of the scanner system551and the programmed modules stored in memory566enable automatic scanning and digitizing of the sample590, which is disposed on a glass slide585. The glass slide585is securely placed on the movable stage580of the scanner system551for scanning the sample590. Under control of the processor556, the movable stage580accelerates the sample590to a substantially constant velocity for sensing by the line scan camera615, where the speed of the stage is synchronized with the line rate of the line scan camera615. After scanning a stripe of image data, the movable stage580decelerates and brings the sample590to a substantially complete stop. The movable stage580then moves orthogonal to the scanning direction to position the sample590for scanning of a subsequent stripe of image data, e.g., an adjacent stripe. Additional stripes are subsequently scanned until an entire portion of the sample590or the entire sample590is scanned.

For example, during digital scanning of the sample590, a contiguous digital image of the sample590is acquired as a plurality of contiguous fields of view that are combined together to form an image strip. A plurality of adjacent image strips are similarly combined together to form a contiguous digital image of a portion or the entire sample590. The scanning of the sample590may include acquiring vertical image strips or horizontal image strips. The scanning of the sample590may be either top-to-bottom, bottom-to-top, or both (bi-directional) and may start at any point on the sample. Alternatively, the scanning of the sample590may be either left-to-right, right-to-left, or both (bi-directional) and may start at any point on the sample. Additionally, it is not necessary that image strips be acquired in an adjacent or contiguous manner. Furthermore, the resulting image of the sample590may be an image of the entire sample590or only a portion of the sample590.

In one embodiment, computer-executable instructions (e.g., programmed modules and software) are stored in the memory566and, when executed, enable the scanning system551to perform the various functions described herein. In this description, the term “computer-readable storage medium” is used to refer to any media used to store and provide computer executable instructions to the scanning system551for execution by the processor556. Examples of these media include memory566and any removable or external storage medium (not shown) communicatively coupled with the scanning system551either directly or indirectly, for example via a network (not shown).

FIG.11Billustrates a line scan camera having a single linear array640, which may be implemented as a charge coupled device (“CCD”) array. The single linear array640comprises a plurality of individual pixels645. In the illustrated embodiment, the single linear array640has 4096 pixels. In alternative embodiments, linear array640may have more or fewer pixels. For example, common formats of linear arrays include 512, 1024, and 4096 pixels. The pixels645are arranged in a linear fashion to define a field of view625for the linear array640. The size of the field of view varies in accordance with the magnification of the scanner system551.

FIG.11Cillustrates a line scan camera having three linear arrays, each of which may be implemented as a CCD array. The three linear arrays combine to form a color array650. In one embodiment, each individual linear array in the color array650detects a different color intensity, for example red, green, or blue. The color image data from each individual linear array in the color array650is combined to form a single field of view625of color image data.

FIG.11Dillustrates a line scan camera having a plurality of linear arrays, each of which may be implemented as a CCD array. The plurality of linear arrays combine to form a TDI array655. Advantageously, a TDI line scan camera may provide a substantially better SNR in its output signal by summing intensity data from previously imaged regions of a specimen, yielding an increase in the SNR that is in proportion to the square-root of the number of linear arrays (also referred to as integration stages). A TDI line scan camera may comprise a larger variety of numbers of linear arrays, for example common formats of TDI line scan cameras include 24, 32, 48, 64, 96, 120 and even more linear arrays.

The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles described herein can be applied to other embodiments without departing from the spirit or scope of the invention. Thus, it is to be understood that the description and drawings presented herein represent a presently preferred embodiment of the invention and are therefore representative of the subject matter which is broadly contemplated by the present invention. It is further understood that the scope of the present invention fully encompasses other embodiments that may become obvious to those skilled in the art and that the scope of the present invention is accordingly not limited.