Systems and methods for detecting cancer metastasis using a neural network

Embodiments of the disclosure provide systems and methods for detecting cancer metastasis in a whole-slide image. The system may include a communication interface configured to receive the whole-slide image and a learning model. The whole-slide image is acquired by an image acquisition device. The system may also include a memory configured to store a plurality of tiles derived from the whole-slide image in a queue. The system may further include at least one processor, configured to apply the learning model to at least two tiles stored in the queue in parallel to obtain detection maps each corresponding to a tile, and detect the cancer metastasis based on the detection maps.

TECHNICAL FIELD

The present disclosure relates to systems and methods for detecting cancer metastasis in a whole-slide image, and more particularly to, systems and methods for detecting cancer metastasis by processing tiles of a whole-slide image according to an asynchronous queue.

BACKGROUND

Early diagnosis can significantly reduce the death rate of breast cancer. Studies have shown that the five-year survival rate can be boosted from 24% to 99% when the breast cancer is diagnosed at an early stage. Clinically, cancer metastasis in lymph nodes is a commonly adopted criterion for early diagnosis, considering they are usually the first places the breast cancer metastasizes to. Gigapixel Whole-slide Images (WSIs) are useful, but they suffer from two major limitations. First, while the pathologists informative image modality has been widely used for this purpose, current examination process needs to exhaustively examine the extremely large WSIs (e.g., 200,000×100,000) to find the potential metastases. This procedure is time-consuming and error-prone. Additionally, because of large image and structure complexity, the inter- and intra-observer variabilities among pathologists are high, especially for those who lack of clinical experience.

Therefore, accurately and efficiently detecting cancer metastasis in lymph nodes imaged using WSIs is critical for early breast cancer diagnosis. Learning networks, such as Convolutional Neural Network (CNN) with powerful hierarchical architectures, have been applied to cancer metastasis detection to improve accuracy. For example, automatic detections using CNN could significantly outperform the pathologists. However, these methods face computational challenges due to the large size of WSIs and the large variances. For example, to achieve high accuracy, current methods usually divide the WSIs into heavily overlapping patches and analyze each one with powerful but computationally intensive models (e.g., Inception V3), which is extremely time-consuming.

One approach is to group the pixels into superpixels, so the computation is no longer conducted on the pixel level, thereby significantly reducing the computation. However, this approach typically uses “hand-crafted” features, which are not specifically designed for the WSI detection problem. Another approach is to leverage the state-of-the-art deep CNNs to tackle this problem. However, because of CNN's structure limitation, the input image should have a fixed size. Therefore, WSIs are often divided into small patches (e.g., 256×256) with each of them being processed independently. These attempts are computationally expensive and are not readily applicable to real clinical setups.

From clinical point of view, the detection method's computation efficiency determines whether it can be adopted as a pathologist's daily tool. Therefore, there is a need for Computer-Aided-Diagnosis (CAD) based cancer metastasis detection systems and methods to aid real clinical applications.

Embodiments of the disclosure address the above problems by systems and methods for accurately and efficiently detecting cancer metastasis in a whole-slide image based on a learning model.

SUMMARY

Embodiments of the disclosure provide a system for detecting cancer metastasis in a whole-slide image. The system may include a communication interface configured to receive the whole-slide image and a learning model. The whole-slide image is acquired by an image acquisition device. The system may also include a memory configured to store a plurality of tiles derived from the whole-slide image in a queue. The system may further include at least one processor, configured to apply the learning model to at least two tiles stored in the queue in parallel to obtain detection maps each corresponding to a tile, and detect the cancer metastasis based on the detection maps.

Embodiments of the disclosure also provide a method for detecting cancer metastasis in a whole-slide image. The method may include receiving the whole-slide image and a learning model. The whole-slide image is acquired by an image acquisition device. The method may further include reading a plurality of tiles derived from the whole-slide image into a queue, and reading at least two tiles from the queue. The method may also include applying, by a processor, the learning model to the at least two tiles stored in the queue in parallel to obtain detection maps each corresponding to a tile; and detecting, by the processor, the caner metastasis based on the detection maps.

Embodiments of the disclosure further provide a non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, causes the one or more processors to perform a method for detecting cancer metastasis in a whole-slide image. The method may include receiving the whole-slide image and a learning model. The whole-slide image is acquired by an image acquisition device. The method may further include reading a plurality of tiles derived from the whole-slide image into a queue, and reading at least two tiles from the queue. The method may also include applying the learning model to the at least two tiles stored in the queue in parallel to obtain detection maps each corresponding to a tile; and detecting the caner metastasis based on the detection maps.

DETAILED DESCRIPTION

FIG. 1illustrates an exemplary cancer metastasis detection system100, according to some embodiments of the present disclosure. Consistent with the present disclosure, cancer metastasis detection system100is configured to detect cancer metastasis in a WSI acquired by an image acquisition device105. WSI is a digital imaging technique used in pathology. WSI, also commonly referred to “virtual microscopy,” emulates conventional light microscopy in a computer-generated manner. Image acquisition device105may generate WSIs in two steps: first step utilizes specialized hardware (scanner) to digitize glass slides, which generates a large representative digital image (so-called “digital slide”), and the second step employs specialized software (e.g., virtual slide viewer) to view and/analyze these digital files.

Although the descriptions are made using WSIs as an example, it is contemplated that the disclosed systems and methods may also be applied or adapted to process images acquired using various other imaging modalities, including MRI, functional MRI (e.g., fMRI, DCE-MRI and diffusion MRI), CT, CBCT, Spiral CT, PET, SPECT, X-ray, optical tomography, fluorescence imaging, ultrasound imaging, and radiotherapy portal imaging, etc.

As shown inFIG. 1, cancer metastasis detection system100may include components for performing two stages, a training stage and a detection stage. To perform the training stage, cancer metastasis detection system100may include a training database101and a model training device102. To perform the detection stage, cancer metastasis detection system100may include an image processing device103and a WSI database104. In some embodiments, cancer metastasis detection system100may include more or less of the components shown inFIG. 1. For example, when a learning model for cancer metastasis detection is pre-trained and provided, cancer metastasis detection system100may only include image processing device103and WSI database104.

Cancer metastasis detection system100may optionally include a network106to facilitate the communication among the various components of cancer metastasis detection system100, such as databases101and104, devices102,103, and105. For example, network106may be a local area network (LAN), a wireless network, a cloud computing environment (e.g., software as a service, platform as a service, infrastructure as a service), a client-server, a wide area network (WAN), etc. In some embodiments, network106may be replaced by wired data communication systems or devices.

In some embodiments, the various components of cancer metastasis detection system100may be remote from each other or in different locations, and be connected through network106as shown inFIG. 1. In some alternative embodiments, certain components of cancer metastasis detection system100may be located on the same site or inside one device. For example, training database101may be located on-site with or be part of model training device102. As another example, model training device102and image processing device103may be inside the same computer or processing device.

As shown inFIG. 1, model training device102may communicate with training database101to receive one or more sets of training data. Each set of training data may include a WSI image and its corresponding ground truth detection map that provides the detection result to each of the pixel of the WSI image. Training images stored in training database101may be obtained from a medical image database containing previously acquired medical images for pathology use.

Model training device102may use the training data received from training database101to train a learning model for detecting cancer metastasis in a WSI. Model training device102may be implemented with hardware specially programmed by software that performs the training process. For example, model training device102may include a processor and a non-transitory computer-readable medium. The processor may conduct the training by performing instructions of a training process stored in the computer-readable medium. Model training device102may additionally include input and output interfaces to communicate with training database101, network106, and/or a user interface (not shown). The user interface may be used for selecting sets of training data, adjusting one or more parameters of the training process, selecting or modifying a framework of the learning model, and/or manually or semi-automatically providing detection results associated with an image for training.

Consistent with some embodiments, the learning model may be a CNN model or an FCN model. The CNN/FCN model may be trained using supervised learning. The architecture of a CNN/FCN model includes a stack of distinct layers that transform the input into the output. Examples of the different layers may include one or more convolution layers or fully-convolutional layers, non-linear operator layers, pooling or subsampling layers, fully connected layers, and/or final loss layers. Each layer may connect one upstream layer and one downstream layer.

As used herein, “training” a learning model refers to determining one or more parameters of at least one layer in the learning model. For example, a convolutional layer of a CNN model may include at least one filter or kernel. One or more parameters, such as kernel weights, size, shape, and structure, of the at least one filter may be determined by e.g., a backpropagation-based training process.

Consistent with the present disclosure, the FCN model may be trained directly by model training device102using the training data or transformed from a trained CNN model. In some embodiments, the convolutional layers in a CNN model may be transformed to fully-convolutional layers in an FCN model. Unlike a CNN model that can only process an input of a set size, an FCN model can process input of an arbitrary size (e.g., the entire WSI). Therefore, the FCN model may offer more flexibility than the CNN model for image processing device103. Using an FCN transformed from a CNN (e.g., Inception V3), the resulting probability map will be the same as using the CNN architecture itself except that the latter uses a sufficiently small stride.

Image processing device103may receive at least one FCN model from model training device102. Image processing device103may include a processor and a non-transitory computer-readable medium (discussed in detail in connection withFIG. 2). The processor may perform instructions of a cancer metastasis detection process stored in the medium. Image processing device103may additionally include input and output interfaces (discussed in detail in connection withFIG. 2) to communicate with WSI database104, network106, and/or a user interface (not shown). The user interface may be used for selecting a WSI for diagnosis, initiating the detection process, displaying the WSI image and/or the detection results.

Image processing device103may communicate with WSI database104to receive one or more WSIs. In some embodiments, the WSIs stored in WSI database104may be obtained from a medical image database maintained by a pathology department. The WSIs may be acquired by image acquisition device105. Image processing device103may use the trained model received from model training device102to predict whether each pixel or superpixel of the WSI corresponds to cancer metastasis, and output a detection map. Consistent with some embodiments, the WSI may be divided into tiles (e.g., patches) and image processing device103may perform cancer metastasis detection in each tile. In some embodiments, image processing device103may read the tiles into memory and store them in a queue at a different speed from the speed at which the tiles are read out from the memory and processed. In some embodiments, image processing device103may include multiple processors for processing tiles in parallel. The queue is therefore an asynchronous queue with different in and out speeds. The detection maps obtained for the tiles may be merged to obtain the detection results for the entire WSI. In some embodiments, the WSI and its corresponding detection map may be automatically stored in training database101and become an additional training sample.

FIG. 2illustrates an exemplary image processing device103, according to some embodiments of the present disclosure. In some embodiments, image processing device103may be a special-purpose computer, or a general-purpose computer. For example, image processing device103may be a computer custom-built for hospitals to perform image acquisition and image processing tasks. As shown inFIG. 2, image processing device103may include a communication interface202, a storage204, a memory206, and a processor208.

Communication interface202may include a network adaptor, a cable connector, a serial connector, a USB connector, a parallel connector, a high-speed data transmission adaptor, such as fiber, USB 3.0, thunderbolt, and the like, a wireless network adaptor, such as a WiFi adaptor, a telecommunication (3G, 4G/LTE and the like) adaptor, etc. Image processing device103may be connected to other components of system100and network106through communication interface202. In some embodiments, communication interface202may receive WSI220from WSI database104, and FCN model230from modeling training device102.

Storage204/memory206may be a non-transitory computer-readable medium, such as a read-only memory (ROM), a random access memory (RAM), a phase-change random access memory (PRAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), an electrically erasable programmable read-only memory (EEPROM), other types of random access memories (RAMs), a flash disk or other forms of flash memory, a cache, a register, a static memory, a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD) or other optical storage, a cassette tape or other magnetic storage devices, or any other non-transitory medium that may be used to store information or instructions capable of being accessed by a computer device, etc.

In some embodiments, storage204may store the trained model(s), e.g., FCN model230or a CNN model, and data used or generated while executing the computer programs, such as WSI220, etc. In some embodiments, memory206may store computer-executable instructions, such as one or more image processing programs. Consistent with the present disclosure, memory206may additionally store a tile queue262. In some embodiments, a WSI stored in storage204may be divided into multiple tiles. Each tile is an image patch, containing a subset of the WSI. In some embodiments, the tiles may be partially overlapping with other. The tiles may be read from storage204one by one and stored in memory206to form tile queue262. In some embodiments, the tiles may be read and stored in tile queue262at a first speed.

Processor208may be a processing device that includes one or more general processing devices, such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), and the like. More specifically, the processor may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor running other instruction sets, or a processor that runs a combination of instruction sets. The processor may also be one or more dedicated processing devices such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), system-on-chip (SoCs), and the like. Processor208may be communicatively coupled to memory206and configured to execute the computer-executable instructions stored thereon.

In some embodiments, processor208may include multiple tile processors282-286. Processor208may read image tiles from tile queue262and distribute them among tile processors282-286. Consistent with the present disclosure, tile processors282-286may analyze tiles in parallel to determine detection maps for the respective tiles. In some embodiments, because tile analysis is relatively time-consuming, the tiles may be read out of tile queue262at a second speed lower than the first speed at which the tiles are read into tile queue262. That is, tile queue262may be an asynchronous queue. In some embodiments, the tiles may be analyzed using the FCN model to determine a cancer metastasis detection map for each tile. In some embodiments, the detection map may be a probability map indicating the probability of each pixel in the tile being corresponding to cancer metastasis. Although the description focuses on embodiments where the second speed (speed reading tiles out of queue262) is lower than the first speed (speed reading tiles into queue262), it is contemplated that in some embodiments, the second speed can be higher than the first speed. In that case, processor208can include multiple tile readers to read the tiles into queue262in parallel.

Processor208may additionally include a probability map processor288. Probability map processor288may be configured to post-process the detection maps (e.g., probability map) for the respective tiles to obtain the cancer metastasis detection results for the entire WSI.

FIG. 3illustrates an exemplary WSI310and an asynchronous queue262to process the same, according to embodiments of the disclosure. WSI may be extremely large (e.g., 200,000×100,000) and it is therefore computationally expensive to analyze the entire WSI. In some embodiments, WSI310is preprocessed to segment the tissue regions and non-tissue region. By only analyzing the tissue regions, the computation task can be reduced in size. In some embodiments, image processing device103may further divide the tissue regions into small tiles, e.g., tiles312-316. The tiles are read into memory206by tile readers242and stored in tile queue262. In some embodiments, the tiles (e.g., tiles A-O) are stored in tile queue262in the order they are received by memory206.

Consistent with the present disclosure, the tiles stored in tile queue262may be read out by processor208and assigned to different threads. For example, the tiles may be assigned to multiple tile processors282-286for parallel processing. Each tile processor282-286determines a detection map (e.g., probability map) for the tile being analyzed. The detection maps may be merged to obtain an overall detection map320for the entire WSI. For example, the detection maps may be placed in a position of overall detection map320corresponding to the position of the respective tiles in WSI310. In some embodiments, overall detection map320may be post-processed to obtain the prediction results.

In some embodiments, tile processors282-286may become the computational bottleneck given the CNN models only take a set size of tiles. Because fully-connected layers of a CNN model are equivalent to fully convolutional layers of an FCN model, in some embodiments, modeling training device102may transform a trained CNN into an FCN model by transforming the last fully-connected layers of the CNN model into equivalent convolutional layers and make it fully convolutional. After the transformation, the FCN can take input of arbitrary size (e.g., the whole WSI310), and output a probability map. The size of the probability map depends on the receptive field k of the original CNN and the down-sample rate d. Regardless of the boundary, the size of the probability map is the original WSI size multiplied by a factor of 1/d.

In some embodiments, each pixel in the probability map xprob(xprob=0,1,2, . . . ) for a tile can be mapped back to the location xWSI(xprob=b 0,1,2, . . . ) of the original WSI, based on the following equation:

xWSI=d·xprob+⌊k-12⌋.(1)
In other words, each prediction value xprobin the probability map depends on the pixel xWSIand its left and right

⌊k-12⌋
pixels of the original WSI. Accordingly, the resulting probability produced by the FCN is equivalent to that produced by the original CNN on the WSI with stride c. The size of the probability map sizeprobcan be determined (potentially with padding for the boundary pixels) based on the WSI size sizeWSIaccording to the following equation:

In one specific example, suppose a WSI to be analyzed has an image size of 163, 840×93,184 and an Inception V3 model is used for the analysis. The original Inception V3 model may take input of size 299×299 and produce a single prediction value. In other words, the model has an effective receptive field size of 299. As the Inception V3 model down-samples the original input for 5 times (either with pooling or convolution with stride 2), it down-samples the input image by a factor of 25=32. In some embodiments, model training device102may transform the last fully-connected layers of this Inception V3 model into equivalent convolutional layers and make it fully convolutional. The resulted FCN model will be able to take variable sized input. In this specific example, according to Equation (2), it will produce a probability map of size

In some embodiments, such as in the specific example above, the whole gigapixel WSI may not be fed into a single GPU memory due to a limited memory size. Accordingly, the WSI may be divided into tiles for parallel processing using multiple threads. Consistent with the present disclosure, the tiles may be partially overlapping with other, in order for the FCN model to yield the equivalent results as if a CNN model is used. For example, for a CNN model with receptive field of k and a down-sampling rate of d, the overlapping size may be determined as k-d. For example,FIG. 4Aillustrates exemplary overlapping tiles412and414, according to embodiments of the disclosure. As shown, tiles412and414share at least some common pixels in each dimension. NoteFIG. 4Ais not necessarily drawn in scale to show the extent of overlapping. For example, in the embodiment described above, the overlapping size is only 267 when the tile size is 2091.

FIG. 4Billustrates an exemplary transformation from a CNN model to an FCN model, according to embodiments of the disclosure, and explains the concept behind using overlapping tiles. For illustration purpose, a 1D CNN with a down-sample factor d=2 and an effective receptive field k=4 is used, although practically 2D or higher-dimensional CNNs are used for processing the WSIs. The 1D CNN therefore takes an input of size 4 and provides an output of size 1. After transforming the CNN into an FCN, the FCN can take input of an arbitrary size. For example, if the input size is 8, there will be 3 output units (top ofFIG. 4B). The result is equivalent to the evaluation of the original CNN on the input with stride 2 (bottom ofFIG. 4B). That is, in order for the FCN to produce the equivalent prediction results, the input parts should overlap by k-d=2 between each other.

FIG. 5illustrates a flow chart of an exemplary process500for detecting cancer metastasis in a WSI, according to embodiments of the disclosure. For example, method500may be implemented by cancer metastasis detection system100, or more specifically, image processing device103, inFIG. 1. However, method500is not limited to that exemplary embodiment. Method500may include steps S502-S514as described below. It is to be appreciated that some of the steps may be optional to perform the disclosure provided herein. Further, some of the steps may be performed simultaneously, or in a different order than shown inFIG. 5.

In step S502, image processing device103may receive a WSI and a learning model. For example, image processing device103may receive WSI220from WSI database104. The learning model may be trained in advance or on the fly by model training device102. In some embodiments, the learning model may be a CNN model such as an Inception V3 model, or FCN model230. FCN model230may be trained by model training device102directly using training data obtained from training database101, or transformed from a trained CNN model. The transformation may be performed by model training device102or by image processing device103. In some embodiments, fully-connected layers of the CNN model may be transformed to convolutionally layers and made fully convolutional.

In step S504, WSI220may be divided into multiple tiles. Each tile may include a subset of pixels of WSI220. In some embodiments, when an FCN transformed from a CNN is used for image analysis, the tiles may be divided such that they overlap with each other as illustrated inFIG. 4A. For example, for a receptive field size of k and a down-sample factor of d, the overlapping size may be set as k-d in each image dimension.

In step S506, the image tiles (e.g., tiles312-316or412-414) may be read into storage204by tile readers242. In step S508, the tiles may be added to a tile queue262in memory206. In some embodiments, tile queue262may be asynchronous. In other words, tile queue262may be a buffer queue, such that the tiles are read into the queue at a speed different from that of the tiles being read out of the queue. For example, five tiles may be read into tile queue262per second, while only three tiles are read out from the queue. Tile queue262therefore helps to hide the input/output latency.

In step S510, the tiles may be read by processor208from tile queue262into multiple threads. For example, process208may read one tile into each of tile processor282-286. In some embodiments, the tiles in the multiple threads may be processed in parallel, e.g., simultaneously. In step S512, the learning model may be applied to analyze the tile in each thread, e.g., by one of tile processors282-286, and obtain a probability map. The probability map indicates the likelihood of each tile pixel being corresponding to cancer metastasis.

In step S514, the multiple probability maps obtained in step S512may be merged to determine the detection/predication results for the entire WSI. In some embodiments, the predication result (e.g., pixel values) of each probability map may be mapped, one-to-one, to a pixel of the WSI, e.g., according to Equation (1). Because the tiles may overlap with each other, some WSI pixels therefore might have multiple predication results mapped from different probability maps. Various methods may be used to merge such predication results. For example, the maximum probability value may be assigned to as the predication result for the WSI pixel. As another example, an average or median value might be selected. The overall detection map determined for the entire WSI may be provided to a pathologist for review.

The disclosed systems and methods have been validated using WSIs of large size. A baseline is created by using a CNN architecture (an Inception V3 model). In the baseline experiment, the WSIs are divided into tiles of size 299×299 with stride of 128. The tiles are evaluated sequentially without implementing the asynchronous queue disclosed in this disclosure. One batch of tiles (27 tiles) are read into the GPU memory at a time and processed with the trained Inception V3 model. In the second experiment, the asynchronous queue is implemented to hide the I/O latency and 3 threads are used to read tiles into 3 GPUs for processing by the Inception V3 model in parallel. In the third experiment, the trained Inception V3 is transformed into an FCN model. In this experiment, the WSIs are divided into overlapping tiles of size 2091×2091 with stride 1824. That means the neighboring patches have an overlapping of 267. The validation shows that the second experiment (with the queue implementation only) can achieve a more than 50% reduction in time compared to the baseline experiment, and the third experiment (with the queue implementation and model transformation) can achieve more than 85% reduction in processing time.

It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed system and related methods. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice of the disclosed system and related methods.

It is intended that the specification and examples be considered as exemplary only, with a true scope being indicated by the following claims and their equivalents.