Patent Description:
Identifying tumor regions in digital images of cancer tissue is often a prerequisite to performing diagnostic and treatment measures, such as classifying cancers using standard grading schemes. In this regard, <CIT> relates to a Gleason grading method that is improved by segmenting and combining digital images of co-registered slides of differently stained tissue. On the other hand, <CIT> discloses an image processing method for automatic detection of biological structures in a multi-channel image obtained from a biological tissue sample being stained by multiple stains and a respective image processing system. As further, <NPL>) evaluate an approach based on antibody-guided annotation and deep learning to quantify immune cell-rich areas in hematoxylin and eosin (H&E) stained samples. The digital images of tissue slices used in histopathology are very large. Individual images require gigabytes to store. Manual annotation of the tumor regions in whole slides through the visual assessment of a pathologist is laborious considering the high volume of data. Therefore, "chemical annotation" has been used to substitute the marking of tumor regions by a pathologist with image recognition of regions stained by biomarkers that identify tissue that tends to be cancerous. Annotating tumor regions using specific antibody staining decreases the subjectivity of the pathologist's evaluation and accelerates the otherwise tedious process. Immunohistochemical (IHC) staining can be used to distinguish marker-positive cells that express a particular protein from marker-negative cells that do not express the protein. IHC staining typically involves multiple dyes, which includes one or more dyes connected to protein-specific antibodies and another dye that is a counterstain. A common counterstain is hematoxylin, which labels DNA and thus stains nuclei.

A protein specific stain or biomarker can be used to identify the regions of the tissue that are likely cancerous. For example, a biomarker that stains epithelial cells can identify the suspected tumor regions. Then other protein specific biomarkers are used to characterize the cancerous tissue. However, the biomarker that identifies the tumor regions can be incompatible with other protein specific biomarkers so that the biomarker that identifies the cancerous region of interest cannot be included on all slides of a tissue sample. In some situations, the region-identifying biomarker cannot be stained on top of certain other stains. The incompatible biomarkers must be stained on different tissue from adjacent slices, and then the resulting digital images of the adjacent slices are coregistered.

Thus, obtaining all of the information available from the incompatible biomarkers by using multiple adjacent slices requires different tissue of the multiple slices to be stained. As more slices are used, the differences in the tissue of farther apart slices increases. In addition, in some situations there is insufficient tissue to produce a large number of adjacent slices that would accommodate fewer biomarkers per slice. Thus, a method is sought for determining cancerous regions of interest in an image of a tissue slice that would be marked by a region-identifying biomarker without requiring the tissue slice to be stained by that biomarker. Further aspects are disclosed in the subclaims.

The disclosed method uses a convolutional neural network model to predict which regions of a tissue slice would have been stained by a first stain by training the model to identify those regions based only on tissue stained by a second stain. The method obviates the need to use the first stain to mark likely cancerous regions on other slices of the tissue that are stained with the second stain. Typically, a training slice of tissue from a cancer patient is stained with both a first immunohistochemical stain and a second counterstain. A first digital image of the training slice is then acquired. A target region of a first digital image is identified using image analysis based on the first stain. Then a set of parameters for associated mathematical operations of the model are optimized to train the model to classify individual pixels of the first digital image as belonging to the target region based on the second stain but not on the first stain. One of the mathematical operations is convolution filtering that applies a kernel to neighboring pixels. The optimized parameters and associated mathematical operations of the trained model are then stored in a database. A second digital image is acquired from cancerous tissue that is stained with the second stain but not with the first stain. The parameters of the trained model are then applied to the second digital image to indicate the probability that each pixel of the second digital image falls within a likely cancerous region that would have been stained with the first stain. A prediction image is generated in which particular pixel locations of the second digital image have an intensity associated with belonging to the target region.

Another embodiment of the method includes a training mode and a prediction mode. In the training mode, a first slice of tissue from a cancer patient is stained with a first stain that has a first color. For example, the first stain is pan cytokeratin (panCK), and the first color is yellowish brown. PanCK stains epithelial cells and tends to mark regions of the tissue that are likely cancerous. The first slice of tissue is also stained with a second stain that has a second color. For example, the second stain is hematoxylin, and the second color is blue. A first digital image is acquired of the first slice of tissue. Image analysis is used to determine whether each pixel of the first digital image falls within a region of interest based on the first color. A set of parameters of a model are generated that indicate the probability that each pixel of the first digital image falls within the region of interest using the second color but not the first color. The model is a convolutional neural network model, and the set of parameters is generated by training the convolutional neural network model on the first digital image in which the region of interest has been identified.

A second digital image is analyzed in the prediction mode. The second digital image is acquired from a second slice of tissue that is stained with the second stain but not with the first stain. For example, the second slice of tissue can be a slice of the tissue sample taken from the cancer patient that is adjacent to the first slice of tissue. The second slice might not be stained with the first stain because the second slice is stained with a different immunohistochemical stain that is incompatible with the first stain. The optimized set of parameters of the trained model is used to indicate the probability that each pixel of a second digital image falls within the region of interest using only the second color. Thus, the method can be used to determine the regions of the second slice that are likely cancerous without staining the second slice with the first stain. A prediction image is generated in which each pixel location of the second digital image has an intensity indicative of the probability that the associated pixel of the second digital image falls within the region of interest.

The method can be used to predict the regions that would be stained by biomarkers other than panCK by using fluorescent <NUM>, <NUM>-diamidino-<NUM>-phenylindole (DAPI) instead of hematoxylin. The other biomarkers whose staining regions can be predicted using counterstains such as DAPI or hematoxylin include cytokeratin <NUM>, α-methylacyl coenzyme A racemase (AMACR), cluster of differentiation <NUM> (CD3) antibody stain, cluster of differentiation <NUM> (CD4) antibody stain and cluster of differentiation <NUM> (CD68) antibody stain.

The invention further relates to the following aspects: According to one aspect of the present invention, a method is provided comprising staining a slice of cancerous tissue with a first stain; staining the slice with a second stain; acquiring a first digital image of the slice; identifying a likely cancerous region in the first digital image based on the first stain; optimizing a plurality of parameters applied to associated mathematical operations to train a model distinguishing characteristics of the nuclei within a region that is stained by the first stain from nuclei that are outside that region based on the second stain but not on the first stain to classify individual pixels of the first digital image as belonging to the likely cancerous region, wherein the model is a convolutional neural network model, and wherein the optimizing the plurality of parameters involves training the convolutional neural network mode; storing the plurality of parameters and associated mathematical operations of the model in a database; and applying the stored parameters and operations of the trained model to a second digital image to indicate a probability that each pixel of the second digital image falls within the likely cancerous region, wherein the second digital image is acquired from cancerous tissue that is stained with the second stain but not with the first stain.

Preferably within the present invention, a step of generating a prediction image is comprised in which each pixel location of the second digital image is marked to indicate the probability that the associated pixel of the second digital image falls within the likely cancerous region.

Within the present invention, the model is a convolutional neural network model, and optimizing the plurality of parameters involves training the convolutional neural network model.

Within the present invention, the first stain stains epithelial cells, and the second stain stains nuclei.

Preferably, the likely cancerous region is predominantly epithelial cells that have been stained by a cytokeratin stain.

Within the present invention, the first stain is taken from the group consisting of: pan cytokeratin, cytokeratin <NUM>, α-methylacyl coenzyme A racemase (AMACR), cluster of differentiation <NUM> (CD3) antibody stain, cluster of differentiation <NUM> (CD4) antibody stain and cluster of differentiation <NUM> (CD68) antibody stain.

Within the present invention, the second stain is taken from the group consisting of: fluorescent <NUM>, <NUM>-diamidino-<NUM>-phenylindole (DAPI) and hematoxylin.

In a preferred embodiment of the method of the present invention, the first stain is an immunohistochemical stain.

Preferably within the present invention, the first stain is an immunofluorescence stain. Preferably, the second stain is a counterstain.

Also preferably within the present invention, one of the mathematical operations is convolution filtering that applies a kernel to pixels surrounding a central pixel. Preferably, one of the mathematical operations is a Gaussian filter applied to pixel values through a convolution matrix.

According to another aspect of the present invention, a method is provided comprising staining a slice of tissue from a cancer patient with a first stain; staining the slice of tissue with a second stain; acquiring a digital image of the slice of tissue; identifying a target region in the digital image using image analysis based on the first stain; optimizing a set of parameters for associated mathematical operations to train a model to classify individual pixels of the digital image as belonging to the target region by distinguishing characteristics of the nuclei within the target region that is stained by the first stain from nuclei that are outside that region based on the second stain but not on the first stain, wherein the model is a convolutional neural network model, and wherein optimizing a set of parameters involves training the convolutional neural network model; and generating a prediction image, based on the parameters and operations of the trained model, in which particular pixel locations of the digital image have an intensity associated with belonging to the target region.

Preferably the method comprises storing the set of parameters and associated mathematical operations of the model in a database.

Preferably within the present invention, one of the mathematical operations is convolution filtering that applies a kernel to neighboring pixels.

Also preferably within the present invention, one of the mathematical operations is a Gaussian filter applied to pixel values through a convolution matrix.

Preferably within the present invention, the intensity of each of the particular pixel locations is proportional to a probability that each of the particular pixel locations belongs to the target region.

Also preferably within the present invention, the intensity of each of the particular pixel locations indicates whether it is either more likely or less likely that each of the particular pixel locations belongs to the target region.

Also preferably within the present invention, the first stain is an immunohistochemical stain.

Also preferably within the present invention, the second stain is a counterstain.

Within the present invention, the second stain is taken from the group consisting of: hematoxylin and fluorescent <NUM>, <NUM>-diamidino-<NUM>-phenylindole (DAPI).

According to another aspect of the present invention, a method with a training mode and a prediction mode is provided, comprising in the training mode, staining a first slice of tissue from a cancer patient with a first stain, wherein the first stain has a first color; staining the first slice of tissue with a second stain, wherein the second stain has a second color; acquiring a first digital image of the first slice of tissue; determining whether each pixel of the first digital image falls within a region of interest using image analysis based on the first color; generating a set of parameters of a model that indicate a probability that each pixel of the first digital image falls within the region of interest based on the second color but not the first color by distinguishing characteristics of the nuclei within the region of interest that is stained by the first color from nuclei that are outside that region,
wherein the model is a convolutional neural network model, and wherein the generating the set of parameters involves training the convolutional neural network model; and in the prediction mode, applying the set of parameters of the trained model to indicate the probability that each pixel of a second digital image falls within the region of interest using the second color, wherein the second digital image is acquired from a second slice of tissue that is stained with the second stain but not with the first stain.

Also preferably within the present invention, generating a prediction image is comprised in which each pixel location of the second digital image has an intensity indicative of the probability that the associated pixel of the second digital image falls within the region of interest.

Other embodiments and advantages are described in the detailed description below. This summary does not purport to define the invention.

The accompanying drawings, where like numerals indicate like components, illustrate embodiments of the invention.

<FIG> shows a system <NUM> for predicting which regions of tissue would be stained by a first region-identifying biomarker by training a convolutional neural network model to identify those regions based solely on tissue stained with a second and different biomarker or stain. In one embodiment, the first region-identifying biomarker is an immunohistochemical (IHC) stain that distinguishes marker-positive cells expressing a particular protein from marker-negative cells that do not express the protein. The IHC stain is a dye that is connected to a protein-specific antibody. The second stain is the counterstain of the IHC biomarker, which is not protein specific. Examples of the first biomarker or stain include pan cytokeratin, cytokeratin <NUM>, α-methylacyl coenzyme A racemase (AMACR), cluster of differentiation <NUM> (CD3) antibody stain, cluster of differentiation <NUM> (CD4) antibody stain and cluster of differentiation <NUM> (CD68) antibody stain. Examples of the second biomarker or counterstain include hematoxylin and fluorescent <NUM>, <NUM>-diamidino-<NUM>-phenylindole (DAPI). Hematoxylin labels DNA and thus stains nuclei. Hematoxylin has the ability to stain tissue without the addition of a dye.

Thus, in one example of the first embodiment, system <NUM> is trained to predict which regions of tissue of a cancer patient would be stained by pan cytokeratin (panCK) by training a convolutional neural network model to identify those regions based solely on tissue stained with the counterstain hematoxylin. PanCK is a group of protein-specific monoclonal antibodies (a biomarker) that in humans are encoded by a family of genes including about twenty epithelial genes. PanCK is used together with an attached dye to form a stain.

System <NUM> operates in a training mode and in a prediction mode. In the training mode, the convolutional neural network model is optimized by determining the sets of parameters <NUM> for associated mathematical operations <NUM> that best predict based only on the staining of nuclei by hematoxylin those regions on training slices that are actually stained with panCK. The training calculations are performed by data processor <NUM>. In one embodiment, data processor <NUM> is a specialized processor that can simultaneously perform multiple convolution operations between a plurality of pixel matrices and corresponding kernels. The logical operations of the model are implemented on data processor <NUM> as hardware, firmware, software, and/or a combination thereof to provide a means for characterizing regions of tissue in the digital images. Each trained model comprising the optimized sets of parameters <NUM> and associated mathematical operations <NUM> is then stored in the database <NUM>.

Once trained, system <NUM> obviates the need in the prediction mode to use panCK to stain epithelial cells to determine a likely cancerous region of tissue samples of the cancer patient. The likely cancerous epithelial regions can be predicted on slices that are not stained with panCK by applying a trained model to analyze how the counterstain hematoxylin has stained the nuclei of the tissue. For example, the likely cancerous regions that would be stained by panCK can be identified in a non-training image of a tissue slice that has been stained with only the counterstain hematoxylin and a biomarker that is incompatible with panCK. In the prediction mode, data processor <NUM> executes software to analyze digital images of stained tissue by applying the trained parameters <NUM> and associated mathematical operations <NUM> to predict whether pixels in the images that were not stained by an antibody stain would have been stained by that antibody stain based only on the staining of DNA in nuclei. In one embodiment, a prediction image is generated in which particular pixel locations of the non-training image have an intensity associated with belonging to the likely cancerous region that would have been stained by panCK. The prediction image is displayed on the graphical user interface <NUM> of a user work station <NUM>.

<FIG> is a flowchart of steps <NUM>-<NUM> of a method <NUM> for predicting which regions of tissue would be stained by a first biomarker or stain by training a convolutional neural network model to identify those regions based solely on tissue stained with a second stain. In a first step <NUM>, a tissue sample <NUM> that is to be stained with the first protein and receptor biomarker is taken from a cancer patient <NUM> in the form of a biopsy. For example, the tissue sample <NUM> includes epithelial cells of glandular tissue, such as human prostate, breast, colon or lung tissue in which tumor regions are to be identified. In step <NUM>, the tissue sample <NUM> is then sliced into multiple parallel, adjacent slices. For example, the slices are consecutive sections of a formalin-fixed paraffin-embedded sample obtained from a tumor of the cancer patient <NUM>. In order to perform the novel method for predicting tumor regions described herein, one of the slices is immunohistochemically stained with a first stain and a second stain. In step <NUM>, one of the slices is stained with a first stain (the antibody stain panCK) as well as a second stain (the counterstain hematoxylin).

<FIG> illustrates the tissue sample <NUM> being taken from cancer patient <NUM> and being cut into multiple parallel, adjacent slices. The tissue slice <NUM> that has been stained is placed on a slide <NUM>. In step <NUM>, a high-resolution digital image <NUM> is acquired from slice <NUM> of the tissue from cancer patient <NUM> that has been stained with the first and second stains. A high-resolution digital image of a tissue slice typically has a resolution of <NUM>,<NUM> x <NUM>,<NUM> pixels, or <NUM> billion pixels. The image <NUM> is acquired by digital scanning of slide <NUM> in red, green and blue channels.

<FIG> shows a portion of the first digital image <NUM> acquired in step <NUM> displayed on the graphical user interface <NUM> of system <NUM>. On tissue slice <NUM>, the first stain PanCK has stained the epithelial cells <NUM> yellowish brown, and the second stain hematoxylin has stained the nuclei <NUM> blue.

In step <NUM>, thresholding is used to distinguish the tissue from the background of the first digital image <NUM>. The intensity of each pixel of image <NUM> is analyzed, a histogram is generated to determine a dividing intensity threshold, and the pixels are divided into tissue pixels and background pixels using the intensity threshold. Each pixel location is classified as belonging either to the class "tissue" or to the class "background" based on the intensity value of the pixel. In step <NUM>, image analysis and a closing operation are used to define contiguous tissue regions on image <NUM>. In an optional step <NUM>, an expert human pathologist marks larger regions of interest within the tissue regions. This manual annotation is optional and used to mark a limited number, for example no more than a dozen, of regions of interest that likely contain cancerous tissue.

In step <NUM>, image analysis is used to identify the region <NUM> stained by the first stain. System <NUM> recognizes the regions stained yellowish brown by panCK within the tissue regions. Where the optional step <NUM> is performed, the regions stained by panCK are detected in step <NUM> only within the regions of interest manually annotated in step <NUM>. The regions stained by panCK are determined by identifying whether each pixel of the tissue region has a color that falls within the ranges of the red, green and blue color components that make up the yellowish brown imparted by panCK. In step <NUM>, the detected region <NUM> comprises the regions of epithelial cells. Within a sample of glandular tissue <NUM> from cancer patient <NUM>, the regions of epithelial cells stained by panCK identify likely cancerous regions within the digital image <NUM>. In step <NUM>, system <NUM> uses image analysis to identify the tissue region <NUM> that has not been stained by panCK. The non-panCK region contains nuclei stained blue by hematoxylin but does not contain any tissue stained yellowish brown by panCK.

<FIG> shows the portion of digital image <NUM> of <FIG> with the region <NUM> marked with cross hatching that is stained by the first stain. The region <NUM> of tissue that has been stained with the second stain hematoxylin but not with the first stain panCK is the remaining area not covered by cross hatching.

In step <NUM>, system <NUM> optimizes a plurality of parameters applied to associated mathematical operations to train a model based on the second stain hematoxylin but not on the first stain panCK to classify individual pixels of the digital image <NUM> as belonging to a likely cancerous region that is stained by the second stain.

<FIG> illustrates the portion of digital image <NUM> of <FIG> showing only the nuclei stained by the second stain hematoxylin but not any epithelial cells stained by the first stain panCK. An outline of the region <NUM> that was stained by the first stain in <FIG> is superimposed over the image of <FIG>. In step <NUM>, the convolutional neural network model is trained to distinguish characteristics of the nuclei within region <NUM> from nuclei that are outside region <NUM>. No expert human pathologist can perceive the characteristics that distinguish the stained nuclei inside region <NUM> from the stained nuclei outside region <NUM>. All of the nuclei stained by the second stain in <FIG> appear to a human observer to have approximately the same shapes, sizes, geometric features and topologies. However, deep learning using a convolutional neural network is able to identify the subtle differences between the nuclei inside and outside of region <NUM>.

System <NUM> distinguishes the nuclei within region <NUM> from the nuclei outside region <NUM> by optimizing a plurality of parameters applied to associated mathematical operations that comprise the convolutional neural network model. The model is trained to optimize the parameters and operations based on the second stain hematoxylin but not on the first stain panCK so as to classify individual pixels of the digital image <NUM> as belonging to the likely cancerous region.

In step <NUM>, the optimized plurality of parameters and associated mathematical operations of the trained model are stored in the database <NUM>. The stored parameters and operations of the trained model are then applied to other digital images of tissue samples in step <NUM>. For example, the trained model is run on a second digital image <NUM> of a tissue slice adjacent to tissue slice <NUM> that is stained with the second stain hematoxylin but not with the first stain panCK. For example, the adjacent tissue slice could be stained by a biomarker other than panCK because it might be undesirable to stain the adjacent slice with both panCK and the other biomarker. Applying the trained module to the second digital image <NUM> indicates the probability that each pixel of the second digital image <NUM> falls within the likely cancerous region that would have been stained by the first stain panCK.

The trained module is software that executes on the data processor <NUM> and performs intelligent image processing. Thus, data processor <NUM> includes a computer-readable storage medium having program instructions thereon for performing a method of using a second stain to predict the region that would be stained by a first stain. Such a computer-readable storage medium can include instructions for characterizing pixels in digital images based on the surrounding pixels. The model is a computer program product tangibly embodied on the computer-readable storage medium in data processor <NUM> and comprises computer readable and executable program instructions that when executed by the processor provide a visual display on the graphical user interface <NUM> of the interconnected display device <NUM>, such as a personal computer.

<FIG> shows a portion of the second digital image <NUM> that corresponds to the coregistered portion of digital image <NUM> of <FIG>. First digital image <NUM> and second digital image <NUM> are images of adjacent slices of tissue of the sample <NUM> taken from cancer patient <NUM>. The trained model has been run on the second digital image <NUM> and has used only the second stain hematoxylin to predict those regions that would have been stained by the first stain panCK even though the slice adjacent to slice <NUM> was never stained with panCK. In <FIG>, system <NUM> has assigned a white color to each pixel of the second digital image <NUM> that has a greater than <NUM>% probability of falling within the region <NUM> predicted to have been stained by panCK.

In step <NUM>, system <NUM> generates a prediction image <NUM> in which each pixel location of the second digital image <NUM> has an intensity, color or marking indicative of the probability that the associated pixel of the second digital image <NUM> would fall within the likely cancerous region that would have been stained by the first stain panCK.

<FIG> shows a portion of a prediction image <NUM> that corresponds to the portion of second digital image <NUM> that is coregistered with the image portion of <FIG>. The pixel locations of prediction image <NUM> are marked with diagonal hatching for corresponding pixel locations of the coregistered second digital image <NUM> that have a high predicted probability of falling within the likely cancerous region that would have been stained by the first stain panCK. All or just a portion of the prediction image <NUM> is displayed on the graphical user interface <NUM> of the user work station <NUM>.

<FIG> is a flowchart of substeps <NUM>-<NUM> of the step <NUM> in <FIG> for optimizing the parameters of the model based on the second stain but not on the first stain to classify pixels of the first digital image <NUM> as belonging to the region <NUM> predicted to have been stained by the first stain.

In substep <NUM>, a predetermined number of sample pixels are selected from the detected region <NUM> that was stained by the first stain, as shown in <FIG>. The region <NUM> detected by image analysis in step <NUM> to have been stained by the first stain can extend over the entire first digital image <NUM> and not merely over the portion of image <NUM> shown in <FIG>. The sample pixels are selected at random pixels positions throughout region <NUM>. In one embodiment, the predetermined number of sample pixels is <NUM>,<NUM>; in another embodiment the predetermined number is <NUM>,<NUM>. The sample pixels are selected in order to reduce the computing resources required to predict the stained region. The resolution of the prediction of stained pixels is lower for a smaller number of sample pixels selected. Thus, in yet another embodiment where computing resources need not be conserved, the predetermined number of sample pixels is all of the pixels in the stained region <NUM> of the first digital image <NUM>. Still in substep <NUM>, a second set of the predetermined number of sample pixels are also selected from random pixel positions throughout region <NUM> of the first digital image <NUM> that was not stained by the first stain.

In substep <NUM>, a patch is defined around each of the sample pixels in the stained region <NUM> and in the unstained region <NUM>. Thus, each patch is classified as belonging to the stained region or to the unstained region. In one embodiment, each patch is a square of 142x142 pixels that is approximately centered (off center by <NUM>/<NUM> pixel) at the position of each sample pixel.

<FIG> illustrates the outlines of various patches centered on sample pixels on the image portion of <FIG> that shows only nuclei stained blue by the second stain and does not show any staining by the first stain. Where the random pixel positions of the sample pixels are closer together, the patches can overlap. The patches centered on sample pixels in the stained region <NUM> are outlined with dashed lines, whereas the patches centered on sample pixels in the unstained region <NUM> are outlined with dotted lines. Arrows point to the sample pixels at the centers of the patches.

In substep <NUM>, each patch is filtered by applying a mathematical operation to each pixel of each patch. The mathematical operation is applied by multiplying the convolution factors of a kernel with the pixel values of the pixels surrounding the pixel of the patch that is being operated upon. The result of the mathematical operation for each pixel is the sum of the products obtained by multiplying by the convolution factors of the kernel. In one embodiment, the kernel is a 3x3 matrix of convolution factors that is multiplied by the pixel values of a 3x3 matrix of pixels centered on the pixel being operated upon.

The result of the filtering is a filtered patch for each mathematical operation or filter applied to the patch. Multiple filters are applied to each patch resulting in multiple filtered images. Each mathematical operation or filter has an associated kernel with convolution factors. Each filter is used to recognize a basic geometric feature, such as a vertical line, a horizontal line or a curve. In one embodiment, sixteen filters are applied to each patch, which results in a 142x142-pixel patch with sixteen filtered layers.

In substep <NUM>, the resolution of the filtered patches with multiple filtered layers is reduced to generate maximum pooled images. The filtered 142x142-pixel patches are downsampled by selecting the maximum intensity pixel from each 2x2 box of the filtered patch, which generates a lower resolution patch of 71x71 pixels, called a "max pooled image. " The lower resolution patch still has sixteen filtered layers.

In substep <NUM>, each of the lower resolution patches is filtered by applying more mathematical operations to each pixel of the lower resolution patches. In one embodiment, sixteen additional filters are applied to each of the lower resolution patches, which results in 71x71-pixel patches each with sixteen times sixteen filtered layers. The filtering and downsampling steps are repeated until 7x7-pixel "max pooled images" with <NUM> filtered layers are obtained.

In substep <NUM>, the predicted probability of a center patch pixel belonging to the stained region is determined based on the lowest resolution patch with many filtered layers. A weighted sum of each 7x7-pixel patch is calculated, which results in a probability that classifies the central pixel of the patch as belonging either to the stained region <NUM> or to the unstained region <NUM>. The predicted classification of the central pixel is then compared to the actual class of the patch as illustrated in <FIG>.

In substep <NUM>, the mathematical operations of the filters as well as the parameters of those filters are optimized to more accurately predict the class of the central pixel of each patch. Substeps <NUM>-<NUM> are repeated using slightly varied mathematical operations of the filters and parameters of those filters. The filters and filter parameters are changed so as to minimize the difference between the predicted probability that a center pixel belongs to the stained class or the unstained class and the actual class of the center pixel of each patch. The optimized filters and filter parameters that most accurately classify the center pixels of all of the patches comprise the trained convolutional neural network model that predicts which regions of a tissue slice would have been stained by the first stain based only on how the tissue was stained by the second stain. The optimized mathematical operations (filters) and filter parameters are then stored in database <NUM> in step <NUM>. Then the optimized filters and filter parameters that were trained on the sample pixels of the first digital image <NUM> are applied to all of the pixels of the second digital image <NUM> to determine the predicted region <NUM> of staining by the first stain.

Example of optimizing filter operations and parameters.

An example is now provided of how substeps <NUM>-<NUM> of step <NUM> in the flowchart of <FIG> are performed. Substeps <NUM>-<NUM> are performed to optimize the parameters of the convolutional neural network model based on the second stain and without any first stain in order to classify pixels of the first digital image <NUM> as belonging to the region <NUM>, which is predicted to have been stained by the first stain. In this example, the image was captured as pixel values corresponding to red-green-blue values and subsequently converted to pixel values corresponding to gray scale values ranging from <NUM> (dark) to <NUM> (light).

In substep <NUM>, a predetermined number of sample pixels are selected from the detected region <NUM> that was stained by the first stain, as shown in <FIG>. In this example, the predetermined number is <NUM>,<NUM> pixels selected from an image having a total of <NUM>,<NUM> x <NUM>,<NUM> pixels. The predetermined number is arbitrary. Still in substep <NUM>, a second set of <NUM>,<NUM> sample pixels is selected from random pixel positions throughout region <NUM> of the first digital image <NUM> that was not stained by the first stain.

In substep <NUM>, a patch is defined around each of the sample pixels in the stained region <NUM> and in the unstained region <NUM>. In this example, each patch is a square of 142x142 pixels that is approximately centered (off center by <NUM>/<NUM> pixel) at the position of each sample pixel of the <NUM>,<NUM> sample pixels. <FIG> illustrates a patch <NUM> centered on a sample pixel <NUM> in the stained region <NUM>.

In substep <NUM>, each patch is filtered by applying a mathematical operation to each pixel of each patch. The mathematical operation is applied by multiplying the convolution factors of a convolution kernel with the pixel values of the pixels surrounding the pixel <NUM> of the patch that is being operated upon. For example, the filter can be a Laplacian operator, a Gaussian operator or a Sobel operator. In this example, the convolution kernel is a 3x3 matrix containing the nine convolution factors denoted as x11, x12, x13; x21, x22, x23; x31, x32, x33.

<FIG> illustrates a filter or convolution operation being performed on the exemplary pixel <NUM> of the patch <NUM>. Pixel <NUM> has the pixel value "<NUM>". The convolution kernel <NUM> is applied to a square of 3x3 pixels <NUM> centered on pixel <NUM> and having pixel values denoted as w11, w12, w13; w21, w22, w23; w31, w32, w33. In the example operation illustrated in <FIG>, the pixel values are <NUM>, <NUM>, <NUM>; <NUM>, <NUM>, <NUM>; <NUM>, <NUM>, <NUM>, and the convolution factors are <NUM>, <NUM>, -<NUM>; <NUM>, <NUM>, <NUM>; <NUM>, -<NUM>, -<NUM>. Each pixel value represents a gray-scale value of the pixel. The convolution operation is determined by calculating the sum of the products of the convolution factors of kernel <NUM> times the pixel values of the corresponding matrix position of pixel square <NUM>: x11*w11 + x12*w12 + x13*w13 + x21*w21 + x22*w22 + x23*w23 + x31*w31 + x32*w32 + x33*w33. In this example, the convolution operation is calculated as follows: <NUM>*<NUM> + <NUM>*<NUM> +(-<NUM>)*<NUM> + <NUM>*<NUM> + <NUM>*<NUM> + <NUM>*<NUM> + <NUM>*<NUM> + <NUM>*(-<NUM>)+ <NUM>*(-<NUM>)= <NUM>. Following the convolution operation on pixel <NUM>, the pixel value of "<NUM>" is replaced with the pixel value "<NUM>". In this example, the step by which the convolution kernel <NUM> is moved from one pixel of patch <NUM> to the next pixel that is being operated upon is <NUM>. Thus, the sum-of-the-products convolution operation using the convolution factors of kernel <NUM> is performed on each pixel of patch <NUM> to generate a filtered patch of 142x142 pixels.

The convolution operation shown in <FIG> is just one of several filters that are applied to each patch. Because multiple filters are applied to each patch, multiple filtered images are generated. Each mathematical operation or filter has an associated kernel with associated convolution factors. The first convolution is executed in two dimensions, and any subsequent convolutions are carried out in three dimensions. In this example, sixteen filters are applied to each patch, which results in a 142x142-pixel patch with sixteen filtered layers. Still in substep <NUM>, a rectified linear unit (ReLU) is applied to the pixels of each patch. The rectified linear unit is a function that returns zero for any pixel with a negative pixel value; for any pixel with a positive pixel value, that value is returned.

In substep <NUM>, the resolution of the filtered patches with multiple filtered layers (such as sixteen) is reduced by generating maximum pooled images. The filtered 142x142-pixel patches are downsampled by selecting the maximum intensity pixel from each 2x2 square of pixels of the filtered patch of 142x142 pixels, which generates a lower resolution patch called a "max pooled image. " The lower resolution patch still has sixteen filtered layers. In this example, because the convolution is carried out without a padding step, two pixels are lost in the convolution, so the convolution of the patch of 142x142 pixels results in a patch 140x140 pixels. After downsampling, each patch of 142x142 pixels is downsampled to a patch of 70x70 pixels ((<NUM>-<NUM>)/<NUM> pixels).

In substep <NUM>, each of the lower resolution patches is filtered by applying more mathematical operations to each pixel of the lower resolution patches. In this example, sixteen additional filters are applied to each of the lower resolution patches, which results in 70x70-pixel patches each with thirty-two (<NUM>*<NUM>) filtered layers. In the next step, filtering and downsampling results in 34x34-pixel patches ((<NUM>-<NUM>)/<NUM>), each containing sixty-four (<NUM>*<NUM>*<NUM>) layers. In the next step, filtering and downsampling results in 16x16-pixel patches ((<NUM>-<NUM>)/<NUM>), each containing one hundred twenty-eight layers (<NUM>*<NUM>*<NUM>*<NUM>). In the next step, filtering and downsampling results in 7x7-pixel patches ((<NUM>-<NUM>)/<NUM>), each containing two hundred fifty-six layers (<NUM>*<NUM>*<NUM>*<NUM>*<NUM>). Thus, the filtering and downsampling steps are repeated until 7x7-pixel "max pooled images", each with <NUM> filtered layers, are obtained.

In substep <NUM>, the predicted probability of a center patch pixel belonging to the stained region is determined based on the lowest resolution patch with many filtered layers. In this substep, filtering with a 7x7 convolution kernel results in 1x1 pixel patches each containing <NUM> layers (<NUM>*<NUM>*<NUM>*<NUM>*<NUM>*<NUM>). Each 7x7 pixel patch containing <NUM> layers results in single pixel values after convolution with the 7x7 convolution kernel. After downsampling the <NUM> single pixel values, <NUM>1x1-pixel patches are obtained that are reduced to two 1x1 pixel patches in an additional convolution step. Thus, two pixel values are obtained.

Finally, a SoftMax function is applied to the two pixel values to perform a prediction and classification operation that indicates the specific category to which these two pixel values belong. For example, a multinomial logistic loss function can be applied. A weighted sum of each 7x7-pixel patch is calculated, which results in a probability that classifies the central pixel of the patch as belonging either to the stained region <NUM> or to the unstained region <NUM>. In this example, the pixel values are <NUM> and <NUM>, and the predicted probability scores of the two pixels belonging to the stained region are <NUM> and <NUM>. This is a weighted sum because <NUM> + <NUM> = <NUM>. The predicted classification of the central pixel <NUM> of the patch <NUM> is transformed into a vector. In this example, the values <NUM> and <NUM> result in the vector (<NUM>,<NUM>). Then the vector is compared to the actual classification of the patch that has been determined by a human pathologist. In this example, the classification corresponding to the vector (<NUM>,<NUM>) and is PanCK-positive. The values determined by the pathologist are <NUM> and <NUM>, which can be expressed as the vector (<NUM>,<NUM>) corresponding to a classification as PanCK-negative. In this example, the comparison between the predicted scores and the actual scores is carried out by calculating the square error. The error for each value is <NUM> (<NUM>-<NUM> = <NUM> and <NUM>-<NUM> = - <NUM>), which leads to square errors of (<NUM> and <NUM>) = <NUM>/<NUM>√(<NUM>+<NUM>).

In substep <NUM>, the mathematical operations of the filters as well as the parameters (convolution factors) of those filters are optimized to more accurately predict the class of the central pixel of each patch. Substeps <NUM>-<NUM> are repeated using slightly varied mathematical operations of the filters and parameters of those filters. Back propagation or another method of supervised learning is used to vary and improve the mathematical operations of the filters and parameters of those filters. The optimized filters and filter parameters that most accurately classify the center pixels of all of the patches comprise the trained convolutional neural network model that predicts which regions of a tissue slice would have been stained by the first stain based only on how the tissue was stained by the second stain.

Claim 1:
A method comprising:
staining a slice of cancerous tissue with a first stain;
staining the slice with a second stain;
wherein the first stain stains epithelial cells, wherein the first stain is taken from the group consisting of: pan cytokeratin, cytokeratin <NUM>, α-methylacyl coenzyme A racemase (AMACR), cluster of differentiation <NUM> (CD3) antibody stain, cluster of differentiation <NUM> (CD4) antibody stain and cluster of differentiation <NUM> (CD68) antibody stain, and the second stain stains nuclei, wherein the second stain is taken from the group consisting of: fluorescent <NUM>, <NUM>-diamidino-<NUM>-phenylindole (DAPI) and hematoxylin,
acquiring a first digital image of the slice;
identifying a likely cancerous region in the first digital image based on the first stain;
optimizing a plurality of parameters applied to associated mathematical operations to train a model distinguishing characteristics of the nuclei within a region that is stained by the first stain from nuclei that are outside that region based on the second stain but not on the first stain to classify individual pixels of the first digital image as belonging to the likely cancerous region,
wherein the model is a convolutional neural network model, and wherein the optimizing the plurality of parameters involves training the convolutional neural network model;
storing the plurality of parameters and associated mathematical operations of the model in a database; and
applying the stored parameters and operations of the trained model to a second digital image to indicate a probability that each pixel of the second digital image falls within the likely cancerous region, wherein the second digital image is acquired from cancerous tissue that is stained with the second stain but not with the first stain.