DEEP LEARNING MODEL-BASED TUMOR-STROMA RATIO PREDICTION METHOD AND ANALYSIS DEVICE

A deep learning model-based tumor-stroma ratio prediction method comprises the steps in which an analysis device: receives an input of a first staining image of target tissue; generates a second staining image by inputting the first staining image into a trained deep learning model; generates a first binary image of the first staining image; generates a second binary image of the second staining image; and calculates the tumor-stroma ratio of the target tissue on the basis of the first binary image and the second binary image.

TECHNICAL FIELD

The following description relates to a technique for predicting a tumor-stroma ratio (TSR) of cancer tissue using a deep learning model.

BACKGROUND ART

A tumor-stroma ratio (TSR) is a ratio of tumor-related stroma in cancer tissue. It has proven that a high TSR is a predictor of bad prognosis in a plurality of tumors. In other words, accurate evaluation of a TSR is important for tumor diagnosis, prognosis prediction, and treatment.

DISCLOSURE

Technical Problem

Conventionally, pathologists analyze tissue samples of patients to estimate tumor-stroma ratios (TSRs). This method depends on subjective judgement and thus does not ensure accuracy. Further, a method of quantifying stroma using a computer has appeared. However, this method only helps medical workers with their judgement and is not a technique for accurately assessing a TSR.

The following description provides a technique for accurately calculating a TSR by analyzing a patient's tissue using a deep learning model.

Technical Solution

In one general aspect, there is provided a method of predicting a tumor-stroma ratio (TSR) on the basis of a deep learning model, the method including receiving, by an analysis device, a first stained image of target tissue, generating, by the analysis device, a second stained image by inputting the first stained image into a trained deep learning model, generating, by the analysis device, a first binary image of the first stained image, generating, by the analysis device, a second binary image of the second stained image, and calculating, by the analysis device, a TSR of the target tissue on the basis of the first binary image and the second binary image.

In another aspect, there is provided an analysis device for predicting a TSR, the analysis device including an input device configured to receive a first stained image of target tissue, a storage device configured to store a generative adversarial model for generating a second stained image on the basis of the first stained image, and a calculation device configured to generate a first binary image by binzarizing the received first stained image, generate a virtual second binary image by inputting the received first stained image to the generative adversarial model, generate a second binary image by binarizing the virtual second stained image, and calculate a TSR of the target tissue on the basis of the first binary image and the second binary image.

Advantageous Effects

According to the following description a tumor-stroma ratio (TSR) is automatically calculated based on only one stained image of a patient's tissue. The following description provides a TSR which is objectively and highly accurate in a short time, thereby contributing to tumor diagnosis and treatment.

MODES OF THE INVENTION

Since the following description may be diversely modified and have various embodiments, specific embodiments will be shown in the drawings and described in detail. However, it should be understood that this is not intended to limit the following description to specific embodiments, but to encompass all modifications, equivalents, and substitutions included in the spirit and scope of the following description.

Terms such as first, second, A, B, and the like may be used to describe various components, but the components are not limited by the above terms. The terms are used only to distinguish one component from other components. For example, without departing from the spirit of the following description, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component. The term “and/or” includes a combination of a plurality of associated listed items or any one of the plurality of associated listed items.

Among terms used herein, the singular expressions may include plural expressions, unless the context clearly dictates otherwise. It is to be understood that the term such as “include” and the like indicates the presence of described features, numerals, steps, operations, components, parts, or a combination thereof and does not preclude the presence or addition of one or more other features, numerals, steps, operations, components, parts, or a combination thereof.

Prior to a detailed description of drawings, it is to be noted that components herein are just distinguished by the main function of each component. In other words, two or more components to be described below may be combined into one component, or one component may be divided into two or more by subdivided function and provided. Also, each of the components to be described below may additionally perform some or all functions that are handled by other components in addition to main functions that the component is responsible for, and some main functions handled by each component may be exclusively performed by other components.

Further, in performing a method or an operating method, steps constituting the method may be performed in a different order from a described order unless a specific order is clearly described in the context. In other words, the steps may be performed in a described order, substantially simultaneously, or in the reverse order.

The following description is a technology for predicting a tumor-stroma ratio (TSR) of a specific sample. Conventionally, stained tissue images are used for TSR analysis.

The following description is an example of using two different types of stained images. As a first stained image, a haematoxylin and eosin (H&E) image is used. As a second stained image, cytokeratin (CK) image is used. The following description is not limited to a specific staining method, and any staining method in which tissue (stroma) is identified by one type of staining and tumors are identified by another type of staining may be used. The following description may be a method of predicting a TSR by automatically generating a second stained image on the basis of a specific first stained image. However, for convenience of description, it is assumed that a first stained image is an H&E image and a second stained image is a CK image.

The following description is related with generating a second stained image from a first stained image using a learning model. The learning model is a machine learning model. The learning model is one of various types of models. For example, the learning model is a decision tree model, a random forest model, a k-nearest neighbor (KNN) model, a naïve Bayes model, a support vector machine (SVM) model, an artificial neural network model, or the like.

An ANN is a statistical learning algorithm modeled from a neural network of a creature. Various neural network models are being researched. Lately, deep neural networks (DNNs) are attracting attention. A DNN is an artificial neural network model including an input layer, an output layer, and several hidden layers between the input layer and the output layer. Like general artificial neural networks, DNNs may model complex nonlinear relationships. Various types of models have been researched for DNNs. For example, there are a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a generative adversarial network (GAN), a relation network (RN), and the like.

The following description employs a deep learning model that generates a second stained image from an input first stained image. Various types of generative models may be used. However, for convenience of description, the following description will be described on the basis of a GAN which is a generative adversarial model.

A device for analyzing a TSR using a stained image of tissue is referred to as an “analysis device” below. The analysis device is a device that can process data and have a form of a personal computer (PC), a smart device, a server, or the like.

FIG.1illustrates an example of a system100,200for predicting a TSR.FIG.1shows examples in which the analysis device is a computer terminal150and a server250.

FIG.1Ashows the system100for a user to predict a TSR using the computer terminal150. A stained image generation device110is a device that stains a tissue slide and scans the stained tissue slide to generate a stained image. The stained image generation device110generates a first stained image (H&E stained image).

The stained image generation device110transmits the H&E stained image to the computer terminal150through a wired or wireless network. The computer terminal150generates a second stained image (CK stained image) on the basis of the H&E stained image. The computer terminal150may calculate a TSR using the H&E stained image and the CK stained image of the same tissue. A detailed TSR calculation process will be described below. The computer terminal150provides an analysis result to a user R.

InFIG.1A, the stained image generation device110and the computer terminal150are shown as separate objects. However, the stained image generation device110and the computer terminal150may be physically implemented as one device or connected devices.

FIG.1Bshows a system200in which a user accesses the analysis server250via a user terminal270to predict a TSR. A stained image generation device210is a device that stains a tissue slide and scans the stained tissue slide to generate a stained image. The stained image generation device210generates a first stained image (H&E stained image).

The stained image generation device110transmits the H&E stained image to the analysis server250through a wired or wireless network.

The analysis server250generates a second stained image (CK stained image) on the basis of the H&E stained image. The analysis server250may calculate a TSR using the H&E stained image and the CK stained image of the same tissue. A detailed TSR calculation process will be described below.

The analysis server250provides an analysis result to the user terminal270. Also, the analysis server250may transmit the analysis result to an electronic medical records (EMR) system260of a hospital.

FIG.2illustrates an example of a process300of predicting a TSR.FIG.2shows a schematic example of a TSR prediction process.FIG.2shows a process performed by the stained image generation device110/210and the analysis device150/250.

The stained image generation device110/210generates an H&E image of specific sample tissue (310).

The analysis device150/250receives the H&E image and generates a binary H&E image (320). The analysis device150/250generates a CK image on the basis of the received H&E image using a generator G trained in advance (330). The analysis device150/250binarizes the CK image (340).

The analysis device150/250determines a TSR on the basis of the binary H&E image and the binary CK image. The analysis device150/250may subtract the binary CK image from the binary H&E image to calculate a ratio of stroma to the entire area (350).

FIG.3illustrates an example of a generative adversarial model400.FIG.3is an example of the GAN400. The GAN corresponds to a non-supervised learning model. The GAN includes a generator G (410) and a discriminator D (420). The GAN is based on a concept that the generator G (410) for generating data and the discriminator D (420) for assessing the generated data learn in opposition to each other to gradually improve performance. Each of the generator G (410) and the discriminator D (420) may be generated using one of various machine learning models. For example, the generator G (410) may be implemented via a model such as U-net, an autoencoder, or the like.

The discriminator D (420) discriminates the generated data as real data or fake data. The generator G is trained to receive a latent code z and generate data, that is, information for deceiving the discriminator D. The generator G generates data G(z), and the discriminator D generates a distinction result of G(z). The generator G has an objective function of minimizing 1-D(G(z)). As a result, when D(G(z)) is 1, the value is minimized, and the generator G is trained so that the discriminator D (420) may mistake G(Z) for the original data.

In a conditional GAN (cGAN), certain condition information c is additionally input to each of the generator G (410) and the discriminator D (420).

FIG.4illustrates an example of a schematic process500of training a generative adversarial model.

A developer may carry out a learning process using the above-described analysis device. Alternatively, the developer may carry out a learning process using an additional computer device. The learning process is assumed to be performed by a computer device.

The learning process is coarsely divided into a process of preparing training data and a process of training a generative adversarial model with the training data.

The process of generating training data will be described. Training data includes an H&E image and a CK image of specific tissue. Accordingly, a process is necessary to generate an H&E image and a CK image first (510). The H&E image and the CK image are a pair of images of the same tissue. Training data includes pairs of an H&E image and a CK image of a plurality of samples (i.e., a plurality of pairs of images).

A researcher explains a process of preparing the training data.

369 gastric adenocarcinoma patients were selected for a cohort. The cohort included the patients in clinical stages II to IV. The cohort included the patients who underwent curative surgery at Samsung Seoul Hospital in 2014 and 2015. The patients belonging to the cohort did not undergo neoadjuvant chemotherapy and did not die in 30 days after the surgery. The patients belonging to the cohort had no confirmed distant metastases or other tumors at the time of surgery. Table 1 to be described below includes clinical information of the cohort.

As a slide for staining, a paraffin tissue section prepared after fixation in formalin was cut into 3 μm. Each of formalin-fixed paraffin-embedded (FFPE) samples was stained using an automatic H&E staining device. The 369 samples represent the 369 patients. As the samples representing the patients, slides showing the deepest invasion in the tumor were selected, clinical stages of the patients were determined from the slides.

Each of the FFPE samples underwent immunohistochemistry (IHC) staining. The researcher used CK staining. The researcher stained the slides using a Bond-MAX automated IHC stainer (a product of Leica Biosystem corp.) and a commercial reagent (a product of Novocastra corp.). The researcher generated 200-fold magnified images (tissue 0.032 μm/pixel) from the CK stained slides using a digital scanner (a product of Aperio Technologies corp.). The CK stained images are referred to as “CK whole slide images (WSIs).”

To perform H&E staining on the same samples again, the researcher washed the CK stained slides with distilled water and then performed H&E staining thereon. The H&E staining process includes a process of incubating the slides in a certain concentration of ethanol and KMnO4and staining the slides with a staining reagent. Since H&E staining is a general process, detailed description thereof will be omitted. Also, images were generated from the H&E stained slides by the digital scanner. The H&E stained images are referred to as “H&E WSIs.”

As described above, the researcher prepared an H&E WSI and a CK WSI of exactly the same tissue section. The researcher confirmed that an H&E WSI and a CK WSI of the same tissue section do not have nonlinear deformations but certain vertical and horizontal shifts occur between the H&E WSI and the CK WSI. Therefore, the researcher aligned a pair of an H&E WSI and a CK WSI of the same section through a certain preprocessing process so that the pair of the H&E WSI and the CK WSI shows the same area as much as possible at a pixel level (520).

The computer device trains a generative adversarial model using aligned pairs of an H&E WSI and a CK WSI (530). In brief, a cGAN receives a latent code and generates a CK WSI. A generative model is trained to receive an H&E WSI and generate a CK WSI. A condition c input to the generative model is information for generating a fine CK WSI. A discrimination model receives a generated (fake) CK WSI and a real CK WSI and distinguish the generated CK WSI. The discrimination model also receives the condition c. When this process is repeated, parameters of the generative model and the discrimination model are optimized, and thus the generative adversarial model is trained.

FIG.5illustrates an example of a process600of aligning an H&E image and a CK image.

An image alignment process includes a global process and a local process. The global process is a process of coarsely aligning a pair of WSIs of the same section by calculating a global shift vector. The local process is a process of precisely aligning the pair of WSIs by calculating a local shift vector for each local area.

A global shift vector calculation process610will be described below.

A computer device may down-sample an H&E WSI and a CK WSI at a certain ratio (611). The researcher performed 32-times down-sampling on the H&E WSI and the CK WSI of the same tissue section.

The computer device performs color deconvolution on each of the down-sampled images (612). Here, the color deconvolution converts each of the H&E WSI and the CK WSI from a RGB color space to a hematoxylin-eosin-diaminobenzidine (DAB) (HED) color space. Since CK antibodies mainly stain cytoplasm, an eosin channel may be extracted from the H&E image, and a DAB channel may be extracted from the CK image.

The computer device binarizes the two channel images whose color space has been converted (613). Here, the computer device uses a threshold value which is a criterion for binarization. As the threshold value, an experimentally appropriate value may be used. The threshold value may be set for each small area of an entire image rather than for the entire image. For example, as the threshold value, a value obtained by subtracting a specific offset value from a weighted pixel value average of a neighbor area (block) may be used.

Meanwhile, the computer device may remove ignorable small objects after binarizing the images.

The computer device may generate a two-dimensional (2D) cross-correlation heatmap of the two binary images (614). The cross-correlation may be calculated using a fast Fourier transform (FFT) algorithm. The computer device may determine a global shift vector on the basis of the position of a maximum value in the cross-correlation heatmap (615). In other words, the computer device can acquire global information for aligning one image with another image.

The computer device may align (global alignment) the H&E WSI and the CK WSI on the basis of the global shift vector. For more precise alignment, the computer device may perform local alignment after the global alignment. In some cases, the computer device may determine a local shift vector for each patch and only perform local alignment on each patch without performing global alignment.

A local shift vector calculation process620will be described below.

An entire image may be divided into patches having a certain size. One patch may have a certain horizontal×vertical size (pixels). Local alignment is a process of aligning each patch. For example, when an entire image has a size of 10240×10240 pixels, the computer device may perform local alignment on the basis of patches having a size of 1024× 1024 pixels. The computer device may align each patch in the same way as in global alignment except for patches having a white background. In other words, the computer device may down-sample an H&E WSI and a CK WSI from each patch (621), perform color deconvolution on the WSIs (622), binarize the WSIs (623), and then calculate local shift vectors on the basis of a maximum value in the cross-correlation heatmap (624).FIG.5shows no cross-correlation heatmap of patches.

The computer device may determine a local shift vector for each patch (624). The computer device may calculate a mean value of local shift vectors for a plurality of patches. In this process, the computer device may remove a local shift vector having an outlier and then calculate the mean value of the local shift vectors. The computer device may determine a value obtained by adding the mean value of local shift vectors and the global shift vector as a final local shift vector (625).

Meanwhile, a variety of methods may be used to determine a local shift vector. For example, (i) the computer device may calculate a local shift vector for each patch and perform local alignment on the patch. (ii) The computer device may calculate a local shift vector for each patch and calculate a final local shift vector for the patch by adding the global shift vector to the local shift vector. (iii) As described above, the computer device may calculate a local shift vector for each patch, calculate a mean value of the local shift vectors, and then calculate a final local shift vector by adding the mean value to the global shift vector. In this case, the computer device aligns each patch on the basis of the final local shift vector.

FIG.6illustrates an example of a process700of training a generative adversarial model using an H&E image and a CK image.FIG.6may be a training process after the H&E WSI and the CK WSI are aligned in patch units inFIG.5. The researcher used a Pix2Pix cGAN among cGAN models. Pix2Pix inputs an edge image of an object to the generator G to generate a target image. This is just an example, and various generative models may be used in generating a CK WSI from an H&E WSI.

Global alignment may be performed on the whole CK WSI on the basis of the whole H&E WSI paired with the CK WSI (711). For each patch of the CK WSI, a final local shift vector may be determined. Each patch of the CK WSI is locally aligned (712). This corresponds to the preprocessing process described inFIG.5.

The generative adversarial model includes the generator G and the discriminator D. The generator G may use a network having the U-net architecture. The discriminator D may use a PatchGAN for discriminating whether the image generated is real or fake by each patch unit.

Each pair of an H&E WSI and a CK WSI of the same tissue section may be identified as the same patch. The computer device extracts an H&E WSI from each patch (720). The computer device generates a virtual CK patch by inputting each patch of the H&E WSI to the generator G (730). The computer device generates a virtual CK image G(x) by inputting an H&E patch image x to the generator G.

The computer device extracts each patch from the aligned CK WSI (740). The computer device inputs the virtual CK patch and a real CK patch corresponding thereto to the discriminator D (750). The computer device may discriminate by each patch unit with the discriminator (760). A real CK image is assumed to be y. The discriminator D is trained to discriminate between a pair {x, y} and a pair {x, G(x)}. The generator G is trained so that the generated virtual CK image is not distinguished from the real CK image.

The Pix2Pix architecture basically includes a dropout layer in the generator G. The dropout layer causes input noise to be generated as a highly stochastic result. However, a model that generates a CK image requires a predictable and deterministic result. Accordingly, the generator G may not include a dropout layer.

An objective function of the cGAN may be represented as shown in Equation 1 below.

The generator G is trained to minimize the objective function, and the discriminator D is inversely trained to maximize to maximize the objective function. The generator G does not only reduce loss of the discriminator but also approximates virtual distribution to real distribution using an L1 loss. Here, it is necessary to consider an L1 loss of the HED color space as well as an L1 loss of the RGB color space. Equation 2 below corresponds to an L1 loss function of the RGB color space, and Equation 3 below corresponds to an L1 loss function of the HED color space.

Therefore, a final loss function of the generative adversarial model may be represented as shown in Equation 4 below.

FIG.7illustrates an example of an analysis device800. The analysis device800corresponds to the above-described analysis device (150and250ofFIG.1). The analysis device800may be physically implemented in various forms. For example, the analysis device800may have the form of a computer device, such as a PC, a network server, a data processing chipset, or the like.

The analysis device800may include a storage device810, a memory820, a calculation device830, an interface device840, a communication device850, and an output device860.

The storage device810may store a first stained image of a specific sample.

The storage device810may store a program or code for binarizing an image.

The storage device810may store a deep learning model for generating a second stained image on the basis of the first stained image. The deep learning model may be the above-described generative adversarial model.

The storage device810may store the second stained image calculated by the deep learning model.

The storage device810may store another program or code for image processing.

Also, the storage device810may store a command or program code for a process of calculating a TSR as described above.

The storage device810may store a TSR which is an analysis result.

The memory820may store data, information, and the like generated in a process in which the analysis device800calculates the TSR.

The interface device840is a device to which a certain command and data are externally input. The interface device840may receive the first stained image from an input device or external storage device which is physically connected thereto. The first stained image may be an H&E stained image.

The communication device850is a component that receives and transmits certain information through a wired or wireless network. The communication device850may receive the first stained image from an external object. Alternatively, the communication device850may transmit the analysis result to an external object such as a user terminal.

The interface device840and the communication device850are components that receive certain information and images from a user or another physical object, and thus may be collectively referred to as an input device. Alternatively, an input device may be an interface in a route for transmitting the first stained image or a request received by the communication device850to the inside of the analysis device800.

The output device860is a device that outputs certain information. The output device860may output an interface required for a data processing process, the analysis result, and the like.

The calculation device830may predict a TSR using the command or program code stored in the storage device810.

The calculation device830may generate the virtual second stained image on the basis of the first stained image using the deep learning model stored in the storage. The virtual second stained image may be a CK stained image.

The calculation device830may generate a first binary image by binarizing the first stained image.

The calculation device830may generate a second binary image by binarizing the virtual second stained image.

The calculation device830may calculate a TSR on the basis of the first binary image and the second binary image.

The calculation device830may identify a stroma area in the entire tissue on the basis of a result of removing a white area in the second binary image from a white area in the first binary image. Subsequently, the calculation device830may calculate the TSR by calculating a ratio of the stroma area to the entire tissue area.

The calculation device830may convert an H&E WSI into a virtual CK WSI using a trained generative model. The calculation device830may separately convert patches of the H&E WSI into patches of the CK WSI. The calculation device830may determine a tumor area and a stroma area in each patch and finally calculate the TSR for the entire image. Alternatively, the calculation device830may generate one H&E WSI and one CK WSI by combining the patches and calculate the TSR on the basis of the H&E WSI and the CK WSI.

The calculation device830binarizes the H&E WSI (or the individual patches) and the CK WSI (or the individual patches). The calculation device830subtracts (removes) a white area (tumor area) in the binary CK WSI from a white area (stroma area) in the binary H&E WSI. The calculation device830may calculate the TSR by determining a ratio of an area from which tumor tissue has been removed to the entire stroma area. In other words, the calculation device830may calculate a TSR as shown in Equation 5 below.

The calculation device830may be a device such as a processor which processes data and certain calculation, an application processor (AP), or a chip on which a program is embedded.

FIG.8is an example of a virtual CK image generated by a generative adversarial model.FIG.8shows a result of generating a virtual CK image C on the basis of a real H&E image A. Comparing a real CK image B with the virtual CK image C, tumor cells are stained with high accuracy in the virtual CK image.

FIG.9is an example of evaluating accuracy of TSR prediction results based on a deep learning model. The researcher calculated TSRs of the above-described cohort (the 369 patient samples) using the proposed TSR prediction method. Also, an expert pathologist evaluated the samples. Results based on a generative adversarial model were compared with actual evaluation results of the pathologist. TSRs are measured as ratio values. However, it is difficult to evaluate accuracy of each measured value, and thus evaluations are made as low TSRs and high TSRs. Data used inFIG.9corresponds to stomach data, colon data, and stomach+colon data.

FIG.9Ashows results of a model that does not use the loss function (Equation 3) of the HED color space, andFIG.9Bshows results of a model that uses the loss function of the HED color space. In the case of considering the loss function of the HED color space, areas under the curves (AUCs) were high (all were 0.9 or above). Even in the case of not using the loss function of the HED color space, evaluation results were considerably high.

The virtual CK image generation method and TSR calculation method described above may be implemented as a program (or application) including an executable algorithm that may be executed in a computer. The program may be stored and provided in a transitory or non-transitory computer-readable medium.

The non-transitory computer-readable medium is a medium that semi-permanently stores data and is readable by a device, rather than a medium that stores data for a short time period such as a register, a cache, a memory, or the like. Specifically, the various applications or programs described above may be stored and provided in a non-transitory computer-readable medium such as a compact disc (CD), a digital versatile disc (DVD), a hard disk, a Blu-ray disc, a universal serial bus (USB) memory, a memory card, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, or the like.

The transitory computer-readable medium is one of various random-access memories (RAMs) such as a static RAM (SRAM), a dynamic RAM (DRAM), a synchronous DRAM (SDRAM), a double data rate (DDR) SDRAM, an enhanced SDRAM (ESDRAM), a synclink DRAM (SLDRAM), and a direct Rambus RAM (DRRAM).

The embodiments and the drawings appended to the present specification clearly describe only a part of the technical spirit of the examples described herein, and it is obvious that all modified examples and specific embodiments that can be easily inferred by those of ordinary skill in the art within the technical spirit included in the specification and drawings of the examples described herein fall in the scope of the present embodiment.