MODEL FOR DETERMINING IHC POSITIVITY

Methods and systems for diagnosing and treating cancer include performing color deconvolution on an input image, stained according to a second staining process, to generate channels that correspond to dyes used in a first staining process and dyes using in the second staining process. Channels that correlate with a channel used to train a machine learning model are combined to produce a single combined channel. The combined channel is processed using the machine learning model to identify tumor cells. A positivity index is determined based on an output of the machine learning model to aid in medical decision making. A patient's treatment is automatically adjusted based on an output of the machine learning model.

BACKGROUND

Technical Field

The present invention relates to machine learning models and, more particularly, to models used for processing histological images.

Description of the Related Art

Staining is a technique used in histopathology, where dyes are applied to a tissue to identify different structures. Images of tissue stained with such dyes can then be analyzed by a trained pathologist to identify potentially dangerous structures, such as cancerous tumors. Such diagnosis is the first step to providing life-saving treatments to patients.

SUMMARY

A method for diagnosing and treating cancer includes performing color deconvolution on an input image of tissue, stained according to a second staining process, to generate image color channels that correspond to dyes used in a first staining process and dyes using in the second staining process. Channels that correlate with a channel used to train a machine learning model are combined to produce a single combined channel. The combined channel is processed using the machine learning model to identify tumor cells and non-tumor cells. A positivity index is determined based on an output of the machine learning model to aid in medical decision making. A patient's treatment is automatically adjusted based on an output of the machine learning model.

A system for diagnosing and treating cancer includes a hardware processor and a memory that stores a computer program. When executed by the hardware processor, the computer program causes the hardware processor to perform color deconvolution on an input image of tissue, stained according to a second staining process, to generate image color channels that correspond to dyes used in a first staining process and dyes using in the second staining process, to combine channels that correlate with a channel used to train a machine learning model to produce a single combined channel, to process the combined channel using the machine learning model to identify tumor cells and non-tumor cells, to determine a positivity index based on an output of the machine learning model to aid in medical decision making, and to automatically adjust a patient's treatment based on an output of the machine learning model.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Histological staining can be used to diagnose a variety of health conditions, by making it possible to analyze the cellular structure of different kinds of tissue. While histological images are often analyzed by an expert, they can also be used as the input to a machine learning model. Such a model can be trained on a set of previously analyzed and labeled histological images to automatically identify health conditions indicated by the images.

There are various types of staining that are used in histopathology to visualize and differentiate tissues and cells under a microscope. One such type is hematoxylin and eosin (H&E) staining, where hematoxylin stains the nuclei of cells a blue-purple color, while eosin stains the cytoplasm and extracellular matrix of the tissue pink. H&E staining may be used to visualize the general tissue architecture, cell morphology, and abnormalities such as inflammation, necrosis, and cancer. H&E images thus include a mix of two colors: blue-purple and pink.

In clinical practice, a pathologist may use H&E slides identify tumor regions and may order further stains to assess the aggressiveness of the tumor or to discover whether it includes particular mutations.

Another type of staining is immunohistochemistry (IHC) staining, which involves the use of antibodies that are labeled with a detectable marker, such as a fluorescent dye or an enzyme. When the antibody binds to its target antigen in the tissue section, the labeled marker produces a visible signal that can be visualized under a microscope. IHC staining is used to identify the presence of specific proteins in tissues, such as tumor markers or markers of inflammation. An example of such a marker is DAB (3,3′-diaminobenzidine), which exhibits a brown color. In addition, IHC samples may be counter-stained with hematoxylin. Thus, IHC may include two colors, such as blue-purple and brown.

For many IHC markers, a positivity index is calculated to quantitatively assess the level of expression in a tissue sample. The positivity index may be expressed as the percentage of positively staining cells relative to the total number of cells within the sample. However, the index may be calculated on only the tumor cells. This is because some tissues may have many normal cells, which would affect the accuracy of the index.

There are many immunohistochemical (IHC) markers that are used in clinical and research settings to diagnose and characterize different types of cancers and other diseases. An example of an IHC marker is the Ki-67 protein, which is a marker of cellular proliferation. The Ki-67 positivity index is a measure of the proportion of actively dividing cells in a given tissue sample. Ki-67 may be used as a prognostic marker in cancer diagnosis and treatment. A further example of an IHC marker is ER/PR (estrogen receptor/progesterone receptor). These markers are used to determine the hormonal status of breast cancer and guide treatment decisions. A further example of an IHC marker is HER2 (human epidermal growth factor receptor 2). This marker is used to identify breast cancer patients who may benefit from targeted HER2 therapy.

Stained histological images may be used as part of a training dataset for a machine learning model. For example, such a model may have the function of classifying an input image to indicate whether the image shows cancerous tissue. In some cases, a neural network model can calculate the IHC positivity index of tumor cells directly from an IHC-stained image. Such a neural network model can be trained using supervision from labeled H&E-stained images and can be applied directly to IHC images.

Once a given tissue sample is classified as being normal or tumorous, tumor cells can be classified as IHC-positive or IHC-negative directly by sampling a channel of an IHC color deconvolution at the location of detected tumor cells.

A model that was trained using H&E-stained data and labels may be re-trained for use with IHC-stained images. Thus, a model which was originally trained to detect cell nuclei on H&E-stained images may be retrained so that it can detect cell nuclei on IHC-stained images. The training reuses the image and labels of the original model, thereby avoiding the cost of manual relabeling.

Referring now toFIG.1, an exemplary stained slide image100is shown. The slide image100includes the scanned tissue image102. A number of tiles104have been identified within the image, for example dividing the image100into regions that can be processed separately, for example by applying a grid over the scanned tissue image102, with pixels falling within each respective grid making up a respective tile. A set of markers106may also be used, having been provided by a human operator, to limit the extent of the tiles to a region of interest, such that tiles104are only determined within a boundary that is established by the markers106.

Each tile104may be separately processed, for example using parallel processing across multiple processors or processor cores. The total number of tiles may be divided by the number of available processing threads to form sets of tiles. Each set may be processed in serial by a respective thread.

The image100will show different features depending on the type of staining that is used. As noted above, H&E staining will produce an image of blue-purple and pink structures, while IHC staining may produce blue-purple and brown (or some other appropriate marker color). An H&E image may include three channels of information, for example using a color camera to generate red, green, and blue channels. As will be described in greater detail below, color deconvolution can be used to convert this three-channel information to a single channel representing the hematoxylin stain alone. Similarly, a three-channel IHC image may be deconvoluted to produce two channels that represent the hematoxylin stain and the DAB stain. Since both hematoxylin and DAB may stain the nuclei of cells, those channels can be combined into one channel that corresponds semantically to the hematoxylin channel in the H&E image. A model trained on H&E-stained images may therefore function correctly when using IHC-stained images in this manner, without retraining the model for IHC-stained images.

Referring now toFIG.2, a method for training and using a machine learning model is shown. Three stages are shown, including model training200, image processing210, and diagnosis and treatment220. Each of these stages may be performed in the same location/system, or they may be performed separately at different locations and by different entities.

Training the model is performed based on training data with a first type of staining (e.g., H&E staining). Block202labels the first stained images in accordance with a task. The task may include, for example, tumor cell counting, tissue classification, or any other appropriate task that may be performed using histological images. As noted above, the image may be broken up into tiles, which may be processed independently from one another, or they may be whole images.

Block204then performs color deconvolution on the first stained images. Color deconvolution converts red-green-blue (RGB) image information from the first stained images into a set of three channels, with each channel representing hematoxylin (H), eosin (E), and DAB (D) respectively. Any appropriate color deconvolution technique may be used by block204.

Thus, the three-channel RGB first stained images are converted into three-channel HED images. In an HED image, each pixel of the H channel indicates the amount of hematoxylin dye at that pixel, each pixel of the E channel represents the amount of eosin dye at that pixel, and each pixel of the D channel represents an mount of the DAB marker at that pixel.

Since the H&E-stained images do not include the DAB staining, the D channel is empty in the output of this deconvolution. Further, since IHC staining does not use eosin, the E channel of the deconvolution may be ignored. Thus block206selects the H channel for use in training, producing respective first single-channel images that correspond to the first stained images. Block208trains a machine learning model to perform the task based on the first single-channel images.

The machine learning model may be any appropriate model, such as a neural network model that makes use of convolutional neural networks. The model may accept an image as input and produce an output. Training may be performed in a supervised fashion, using the labels of the first stained images corresponding to respective first single-channel images.

Once the model is trained, and has been deployed, it may be used to perform inferences during image processing210. Block212obtains a second image that is stained with a different staining technique, for example using IHC staining. Block214performs color deconvolution on the three-channel IHC-stained image to produce a three-channel HED image.

In this case, the IHC image includes the H channel and the D channel, but the E channel is empty because IHC staining does not use eosin. Because the model is trained with only one channel, the H and D channels are merged into a single channel in channel recombination216. Any appropriate technique for recombination can be used. For example, P=max (H, D) can be used where a recombined pixel's value is taken as the maximum value between the H and D channels for the pixel. Another approach is to use a linear combination P=kH+ (1−k) D to mix the values of the H and D channels, with k being a weighting parameter between zero and one.

Block218processes the image using the trained neural network, for example using the recombined channel. For example, the model may output two sets of cells, normal cells and tumor cells. For each identified cell, the model may output a location of the cell within the image. One exemplary architecture for the model is a fully convolutional U-net model, which detects and classifies the cells as described in greater detail below.

The recombined image may then be used as a single-channel input to the trained model, which performs the designated task and generates a corresponding output. For example, in a tumor cell classification task, the model may output a list of tumor and non-tumor cells present in the second stained image. In a cancer detection task, the model may output a determination of whether the tissue shown in the second stained image is cancerous.

Based on the output of the model, diagnosis and treatment220are performed. Diagnosis may include post-processing of the raw output of the model, for example using peak detection to identify and classify objects such as cell nuclei within the model's output maps. Diagnosis may include identifying a type of cancer or other illness, while treatment may include the administration of one or more drugs, surgeries, or therapies to treat the illness. The diagnosis may be performed automatically, for example in the case of a cancer detection task that indicates that a patient has cancer. The treatment may also be performed automatically, for example in the administration of an appropriate dosage of an anti-cancer medication.

Referring now toFIG.3, additional detail on processing the image with a pre-trained model in block218is shown. Block302uses the model to infer two maps from the input map, for example using the combined HD channels. A first map represents the probability that each pixel represents the center of a tumor cell nucleus, while a second map represents the probability that each pixel represents the center of a non-tumor cell nucleus. Block304transforms the maps into a list of detections using a peak detector, marking the peaks on the maps. Block306determines a score representing a probability of each detection being a tumor cell, for example using a softmax function that is applied to the two map values at each peak.

Using only tumor cells, the D channel can be sampled at each cell location to provide an IHC score based on the intensity of the D channel at the sampling location. A threshold may be obtained on a validation set. Tumor cells for which the IHC score is above the threshold may be identified as tumor-positive cells in diagnosis220, while tumor cells for which the IHC score is below the threshold may be identified as tumor_negative. Non-tumor cells for which the IHC score is above the threshold may be identified as non-tumor_positive and non-tumor cells for which the IHC score is below the threshold may be identified as non-tumor_negative. The IHC positivity index is obtained from tumor cells only and can therefore be computed in block308as:

Any appropriate process may be used to determine the threshold value. One exemplary approach is to obtain a validation set of IHC images where all nuclei have been labeled as either non-tumor, tumor-negative, or tumor-positive. Then a threshold value may be determined that minimizes the error between the labels and the model-classified cells.

Only the D channel of the IHC image carries information about the relevant immune marker (e.g., Ki-67), while the H channel carries information about nuclei morphology. As a result, classification to distinguish between tumor cells and non-tumor cells may be performed on the H channel alone, while the D channel may be sampled to calculate the IHC score.

The IHC positivity index may be used to help healthcare personnel in diagnosing specific aspects of a tumor. For example, a Ki-67 positivity index indicates the rate at which tumor cells are growing. In such embodiments the dye binds to the Ki-67 protein, which is involved in cell proliferation. This can help to identify fast-growing tumors that need to be treated aggressively, for example with early surgical and chemotherapeutic treatments being applied.

Referring now toFIG.4, a diagram of tissue staining and analysis is shown in the context of a healthcare facility400. Tissue staining and analysis408may be used to analyze tissue samples that have been stained according to different staining processes. For example, an analysis model may be trained on H&E training data, and so may accept H&E-stained tissue sample images, but may also accept IHC-stained tissue sample images after appropriate processing.

The healthcare facility may include one or more medical professionals402who review information from a patient's medical records406to determine their healthcare and treatment needs. These medical records406may include tissue samples collected from the patient and stained according to a staining process that the model can accept. Treatment systems404may furthermore monitor patient status to generate medical records406and may be designed to automatically administer and adjust treatments as needed.

Based on information drawn from the tissue staining and analysis408, the medical professionals402may then make medical decisions about patient healthcare suited to the patient's needs. For example, the medical professionals402may make a diagnosis of the patient's health condition and may prescribe particular medications, surgeries, and/or therapies.

The different elements of the healthcare facility400may communicate with one another via a network410, for example using any appropriate wired or wireless communications protocol and medium. Thus tissue staining and analysis408receives tissue samples from the medical records406, updates the medical records406with the output of the trained model, and may coordinate with treatment systems404in some cases to automatically administer a treatment.

Referring now toFIG.5, an exemplary computing device500is shown, in accordance with an embodiment of the present invention. The computing device500is configured to perform tissue analysis.

The computing device500may be embodied as any type of computation or computer device capable of performing the functions described herein, including, without limitation, a computer, a server, a rack based server, a blade server, a workstation, a desktop computer, a laptop computer, a notebook computer, a tablet computer, a mobile computing device, a wearable computing device, a network appliance, a web appliance, a distributed computing system, a processor-based system, and/or a consumer electronic device. Additionally or alternatively, the computing device500may be embodied as one or more compute sleds, memory sleds, or other racks, sleds, computing chassis, or other components of a physically disaggregated computing device.

As shown inFIG.5, the computing device500illustratively includes the processor510, an input/output subsystem520, a memory530, a data storage device540, and a communication subsystem550, and/or other components and devices commonly found in a server or similar computing device. The computing device500may include other or additional components, such as those commonly found in a server computer (e.g., various input/output devices), in other embodiments. Additionally, in some embodiments, one or more of the illustrative components may be incorporated in, or otherwise form a portion of, another component. For example, the memory530, or portions thereof, may be incorporated in the processor510in some embodiments.

The processor510may be embodied as any type of processor capable of performing the functions described herein. The processor510may be embodied as a single processor, multiple processors, a Central Processing Unit(s) (CPU(s)), a Graphics Processing Unit(s) (GPU(s)), a single or multi-core processor(s), a digital signal processor(s), a microcontroller(s), or other processor(s) or processing/controlling circuit(s).

The memory530may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, the memory530may store various data and software used during operation of the computing device500, such as operating systems, applications, programs, libraries, and drivers. The memory530is communicatively coupled to the processor510via the I/O subsystem520, which may be embodied as circuitry and/or components to facilitate input/output operations with the processor510, the memory530, and other components of the computing device500. For example, the I/O subsystem520may be embodied as, or otherwise include, memory controller hubs, input/output control hubs, platform controller hubs, integrated control circuitry, firmware devices, communication links (e.g., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.), and/or other components and subsystems to facilitate the input/output operations. In some embodiments, the I/O subsystem520may form a portion of a system-on-a-chip (SOC) and be incorporated, along with the processor510, the memory530, and other components of the computing device500, on a single integrated circuit chip.

The data storage device540may be embodied as any type of device or devices configured for short-term or long-term storage of data such as, for example, memory devices and circuits, memory cards, hard disk drives, solid state drives, or other data storage devices. The data storage device540can store program code540A for training a model,540B for performing image processing, and/or540C for performing diagnosis and treatment. Any or all of these program code blocks may be included in a given computing system. The communication subsystem550of the computing device500may be embodied as any network interface controller or other communication circuit, device, or collection thereof, capable of enabling communications between the computing device500and other remote devices over a network. The communication subsystem550may be configured to use any one or more communication technology (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, InfiniBand®, Bluetooth®, Wi-Fi®, WiMAX, etc.) to effect such communication.

As shown, the computing device500may also include one or more peripheral devices560. The peripheral devices560may include any number of additional input/output devices, interface devices, and/or other peripheral devices. For example, in some embodiments, the peripheral devices560may include a display, touch screen, graphics circuitry, keyboard, mouse, speaker system, microphone, network interface, and/or other input/output devices, interface devices, and/or peripheral devices.

Referring now toFIGS.6and7, exemplary neural network architectures are shown, which may be used to implement parts of the present models, such as the tissue analysis model600/700. A neural network is a generalized system that improves its functioning and accuracy through exposure to additional empirical data. The neural network becomes trained by exposure to the empirical data. During training, the neural network stores and adjusts a plurality of weights that are applied to the incoming empirical data. By applying the adjusted weights to the data, the data can be identified as belonging to a particular predefined class from a set of classes or a probability that the input data belongs to each of the classes can be output.

The empirical data, also known as training data, from a set of examples can be formatted as a string of values and fed into the input of the neural network. Each example may be associated with a known result or output. Each example can be represented as a pair, (x, y), where x represents the input data and y represents the known output. The input data may include a variety of different data types, and may include multiple distinct values. The network can have one input node for each value making up the example's input data, and a separate weight can be applied to each input value. The input data can, for example, be formatted as a vector, an array, or a string depending on the architecture of the neural network being constructed and trained.

The neural network “learns” by comparing the neural network output generated from the input data to the known values of the examples, and adjusting the stored weights to minimize the differences between the output values and the known values. The adjustments may be made to the stored weights through back propagation, where the effect of the weights on the output values may be determined by calculating the mathematical gradient and adjusting the weights in a manner that shifts the output towards a minimum difference. This optimization, referred to as a gradient descent approach, is a non-limiting example of how training may be performed. A subset of examples with known values that were not used for training can be used to test and validate the accuracy of the neural network.

During operation, the trained neural network can be used on new data that was not previously used in training or validation through generalization. The adjusted weights of the neural network can be applied to the new data, where the weights estimate a function developed from the training examples. The parameters of the estimated function which are captured by the weights are based on statistical inference.

In layered neural networks, nodes are arranged in the form of layers. An exemplary simple neural network has an input layer620of source nodes622, and a single computation layer630having one or more computation nodes632that also act as output nodes, where there is a single computation node632for each possible category into which the input example could be classified. An input layer620can have a number of source nodes622equal to the number of data values612in the input data610. The data values612in the input data610can be represented as a column vector. Each computation node632in the computation layer630generates a linear combination of weighted values from the input data610fed into input nodes620, and applies a non-linear activation function that is differentiable to the sum. The exemplary simple neural network can perform classification on linearly separable examples (e.g., patterns).

A deep neural network, such as a multilayer perceptron, can have an input layer620of source nodes622, one or more computation layer(s)630having one or more computation nodes632, and an output layer640, where there is a single output node642for each possible category into which the input example could be classified. An input layer620can have a number of source nodes622equal to the number of data values612in the input data610. The computation nodes632in the computation layer(s)630can also be referred to as hidden layers, because they are between the source nodes622and output node(s)642and are not directly observed. Each node632,642in a computation layer generates a linear combination of weighted values from the values output from the nodes in a previous layer, and applies a non-linear activation function that is differentiable over the range of the linear combination. The weights applied to the value from each previous node can be denoted, for example, by w1, w2, . . . . wn−1, wn. The output layer provides the overall response of the network to the input data. A deep neural network can be fully connected, where each node in a computational layer is connected to all other nodes in the previous layer, or may have other configurations of connections between layers. If links between nodes are missing, the network is referred to as partially connected.

Training a deep neural network can involve two phases, a forward phase where the weights of each node are fixed and the input propagates through the network, and a backwards phase where an error value is propagated backwards through the network and weight values are updated.

The computation nodes632in the one or more computation (hidden) layer(s)630perform a nonlinear transformation on the input data612that generates a feature space. The classes or categories may be more easily separated in the feature space than in the original data space.