Uncertainty Estimation in Medical Imaging

For uncertainty estimation for a machine-learned model prediction in medical imaging, a distribution in latent space is sampled to determine uncertainty in machine-learned model prediction. For example, a segmentation is output by a machine-learned model in response to input of multi-parametric data. A latent space generated by the machine-learned model is used to estimate the uncertainty of the segmentation, such as a segmentation of a prostate lesion. The model of Kohl, et al., may be used as the variational auto encoder generates a latent space representing the distribution of the training data, which latent space may be used to determine uncertainty.

BACKGROUND

The present embodiments relate to machine-learned-based prediction in medical imaging. It is cumbersome and time-consuming for the radiologist to detect and label lesions manually. Artificial intelligence (AI)-driven automated detection frameworks may detect lesions, such as prostate cancer based on multi-parametric magnetic resonance imaging (MRI) (mp-MRI).

As deep learning-based methods are data-dependent, the AI lesion detector may predict overconfident results on the cases with different data distributions from the training data. As the MRIs are considered a standard image modality for PCa diagnosis, the scans can be acquired by various MRI scanners from different vendors, such as Siemens, GE, or Philips, with different machine settings and machine noise and can be processed by different protocols. Additionally, the different b-value settings may lead to intensity various of Apparent Diffusion Coefficient (ADC) images and Diffusion-Weighted Images (DWI). These artifacts of collected MRIs may cause systematical differences of the Al model trained with cases from specific settings. The lack of ability to distinguish the testing cases with different data distribution from training cases may lead to inaccurate prediction for the lesion location and the predicted lesion boundary in actual clinical diagnosis.

Out-of-distribution (OOD) detection algorithms learn data distributions of the training data directly. The out-of-distribution classes that do not belong to the existing training classes may be distinguished, such as distinguishing the scene understanding dataset for street view houses (SVHN) from CIFAR-100, which consists of objects (such as fish, flowers) and acquiring the semantical distribution of the input data by a discriminative network. Some other approaches aim at detecting the abnormal region on image or video, such as lesion detection on X-ray and disease screening on retinal images. However, these methods address the data with apparent abnormal regions or learnable data distribution. There are many cases where the data has inherent ambiguities, such as medical imaging applications. The lesions in medical images may have blurry lesion boundaries or insufficient information for scoring the level of suspicion that may cause various diagnoses. Independent pixel-wise prediction may be learned, and the uncertainty predicted by an ensemble of models via Monte Carlo dropout. However, this group of work learns the data independently and cannot guarantee that the variance is sufficiently learned.

Kohl, et al., in “A Probabilistic U-Net for Segmentation of Ambiguous Images,” provide multiple segmentation hypotheses for ambiguous images via a Variational Auto-Encoder (VAE) conditional on the image. Instead of directly predicting pixel-wise probability via multiple models, the probabilistic U-Net learns the co-variance between pixels and is flexible enough to predict the joint probability of all pixels in the segmentation. This effectively learns the training data distributions, samples diverse possible lesion regions, and works well for predicting images that might lead to different expertise evaluations.

SUMMARY

Systems, methods, and instructions on computer readable media are provided for uncertainty estimation for a machine-learned model prediction in medical imaging. A distribution in latent space is sampled to determine uncertainty in machine-learned model prediction. For example, a segmentation is output by a machine-learned model in response to input of multi-parametric data. A latent space generated by the machine-learned model is used to estimate the uncertainty of the segmentation, such as a segmentation of a prostate lesion. The model of Kohl, et al., may be used as the variational auto encoder generates a latent space representing the distribution of the training data, which latent space may be used to determine uncertainty.

In a first aspect, a method is provided for lesion segmentation for a medical imager. The medical imager acquires multi-parametric data representing a patient. A machine-learned network generates a segmentation of a lesion represented in the multi-parametric data. The segmentation is generated in response to input of the multi-parametric data to the machine-learned network. An uncertainty of the segmentation is determined using the machine-learned network. The segmentation and the uncertainty of the segmentation are displayed.

In one embodiment, the medical imager is a magnetic resonance (MR) imager. The multi-parametric data is, for example, T2-weighted data, apparent diffusion coefficient data, and diffusion-weighted data. The multi-parametric data represents any of various regions of the patient, such as a prostate region. In a further approach, the segmentation of the lesion is a segmentation of a prostate lesion, which is generated in response to input of the T2-weighted data, apparent diffusion coefficient data, diffusion-weighted data, and a prostate mask.

In another embodiment, the machine-learned network is a U-Net with convolutional layers and an encoder. The multi-parametric data is input to both the U-Net and the encoder. A sample from a latent space at an output of the encoder is fed to one of the convolutional layers of the U-Net. The U-Net generates the segmentation based in part on the input of the multi-parametric data and in part on the sample from the latent space. As a further refinement, the U-Net is a residual U-Net (multi-scale), and the encoder is a conditional variational auto encoder. The sample from the latent space is concatenated with a last activation map of the residual U-Net. As another refinement, the segmentation and additional segmentations are generated. The additional segmentations are generated with different samples from the latent space. For example, the segmentation and additional segmentations are generated with samples being randomly sampled from the latent space. As a further example, a variance of the segmentation and additional segmentations is calculated as a heat map representing the uncertainty. In another refinement, a mean in the latent space is determined. The segmentation is generated based in part on the mean in the latent space, and the uncertainty is determined as a variation in the latent space. A heat map is output by the U-Net based in part on the variation.

In yet another embodiment, the machine-learned network generates the segmentation and additional segmentations. The uncertainty is determined as a variance of the segmentation and additional segmentations.

As another embodiment, the segmentation is displayed as a mean of multiple predictions by the machine-learned network, and the uncertainty is displayed as a heatmap by spatial location of the segmentation.

In a second aspect, a medical system is provided for uncertainty estimation. An imager is configured to capture scan data representing a patient. An image processor is configured to apply the scan data to a machine-learned network. The machine-learned network is configured to output a classification, detection, and/or segmentation in response to the application. The image processor is further configured to determine an uncertainty of the classification, detection, and/or segmentation with random samples from a latent space. A display is configured to display the classification, detection, and/or segmentation and to display the uncertainty.

In one embodiment, the imager is a magnetic resonance (MR) imager, and the scan data is multi-parametric MR data. The machine-learned network is configured to output the segmentation as a segmentation of a lesion, and the uncertainty is a heat map with greater uncertainty for some locations than others represented in the heat map.

As another embodiment, the machine-learned network is a U-Net and an encoder separate from the U-Net. The encoder is configured to generate the latent space. For example, the U-Net is configured to output a plurality of the classification, detection, and/or segmentation in response to the application of the scan data representing the patient using the random samples. The uncertainty is a variance of the plurality of the classification, detection, and/or segmentation. As a further refinement, the display of the classification, detection, and/or segmentation is display of a mean of the plurality. As another example, the uncertainty is the output based on a variance of the random samples.

In a third aspect, a method is provided for uncertainty determination in machine-learned prediction for a medical scanner. A machine-learned model generates multiple first predications for a patient based on input of scan data from the medical scanner to the machine-learned model. An uncertainty of the multiple first predictions is determined from random sampling of a latent space. A second prediction based on the multiple first predictions (e.g., a mean of the first predictions) and the uncertainty are displayed.

In one embodiment, the multiple first predictions are generated from the random sampling of the latent space. A variance of the multiple first predictions is determined as the uncertainty.

In another embodiment, a variance in the latent space is determined, and the uncertainty is generated based on the variance in the latent space.

Any one or more of the aspects described above may be used alone or in combination. Any aspects of one of method, system, or computer readable media may be used in the others of method, system, or computer readable media. These and other aspects, features and advantages will become apparent from the following detailed description of preferred embodiments, which is to be read in connection with the accompanying drawings. The present invention is defined by the following claims, and nothing in this section should be taken as a limitation on those claims. Further aspects and advantages of the invention are discussed below in conjunction with the preferred embodiments and may be later claimed independently or in combination.

DETAILED DESCRIPTION OF EMBODIMENTS

To assist in diagnosis, uncertainty in a prediction (e.g., segmentation, classification, or detection) is estimated. The training data distribution is learned, and the learned distribution is used to predict the uncertainty in prediction for a given patient. For example, uncertainty of an existing lesion detector for prostate lesion segmentation using multiparametric MR scan is estimated.

Inspired by Kohl, et al., and the ability of the predictor to predict multiple hypotheses, the machine-learned model may be used to estimate uncertainties. For example, the probabilistic U-Net of Kohl, et al., learns the co-variance pixel-level uncertainties on PCa lesion segmentation under testing cases with different distributions from the training case.

The uncertainty of the unknown dataset is explored. A given patient image may have multiple segmentation hypotheses for lesion detection due to the ambiguity of different data distributions. The proposed framework can detect abnormal areas and suspicious areas predicted as lesions. The uncertainty identifies the abnormal or suspicious areas. For example, U-Net or other machine-learned model tumor prediction and uncertainty prediction joint training provides an end-to-end training network that can directly and instantly obtain uncertainty through a set of sampled images. The latent space with large variance reflects the high degree of uncertainty in the prediction area.

The framework can be generalized to different tasks, such as classification, detection, and/or segmentation. The random sampling from the learned latent space provides a distribution of underlying factors that may impact the uncertainty of the prediction (e.g., lesion segmentation). The uncertainty is then an additional function of a computer-aided diagnosis (CAD) system (e.g., prostate MR) to provide radiologists highly suspicious regions where lesion borders are difficult to determine. This would also be useful for clinicians to identify the optimal location for biopsy.

FIG.1is a flow chart diagram of one embodiment of a method for uncertainty determination in machine-learned prediction for a medical scanner. For example, the method is for lesion segmentation (e.g., PCa or prostate lesion). The AI or machine-learned model for prediction is used to determine uncertainty. Rather than a Monte-Carlo drop-out approach by varying the input data, the learned distribution is used to estimate uncertainty.

The method is performed by a medical imager, such as a magnetic resonance or computed tomography system. An image processor may acquire scan data, generate a prediction using a machine-learned model, determine uncertainty in the prediction (e.g., spatial variation in uncertainty), and/or generate an image for display on a display screen of the prediction and uncertainty. Other devices may be used, such as using a medical imager or memory to acquire the scan data. In other embodiments, a computer, server, or workstation with access to a memory or medical imager, performs the method.

The acts are performed in the order shown (numerical or top-to-bottom) or other orders. For example, act130is performed prior to or simultaneously with act120. Additional, different, or fewer acts may be provided. For example, act140is not performed, such as where the uncertainty and prediction are stored in a patent electronic medical record. As another example, act120is not performed. Instead, the uncertainty is determined from variance in the latent space.

In act110, a medical imager acquires scan data. The image processor acquires scan data by transfer from memory or over a computer network. In other embodiments, the scan data is acquired by scanning a patient. The medical imager may include an image processor that acquires the data.

In one embodiment, the scan data is imaging or image data representing the patient. The scan data may be at any stage of processing, such as acquired from an antenna or detector or after image processing (e.g., after filtering, detecting, reconstruction, and/or scan conversion). For example, the scan data is a reconstructed representation of an area or volume of the patient. As another example, the scan data is formatted as an image for display on a display screen.

Any type of medical imaging may be used. In one embodiment, the scan data is magnetic resonance (MR) data acquired by an MR imager or system. In the examples used herein, MR data for imaging a prostate is used. In other examples, MR of other regions of the patient, computed tomography (CT), ultrasound, optical, positron emission tomography, single photon emission computed tomography, x-ray, or other types of scanning and resulting scan data may be used.

In an example MR embodiment for detection of lesions or cancer in the prostate, the medical imager acquires multi-parametric data representing a patient. For example, T2-weighted data, apparent diffusion coefficient (ADC) data, and diffusion-weighted imaging (DWI) data are acquired. Additional, different, or fewer MR parameters may be acquired. In other embodiments, multi-parametric data for CT, ultrasound, or other type of imaging is used. In yet other embodiments, the multi-parametric data is data from different modalities (e.g., ultrasound and MR).

In act120, an image processor generates one or more predictions. The prediction is a classification, segmentation, and/or detection. For example, the prediction is a segmentation as a boundary and/or pixel or voxel labels for locations belonging to an object, such as a prostate lesion.

The prediction is made by a machine-learned model. Any machine-learned model may be used, such as a neural network (e.g., fully connected or convolutional neural network). Other models may include support vector machine (SVM) or Bayesian models. The machine-learned model is of any of various architectures, such as U-Net, image-to-image, and/or encoder-decoder networks.

The machine-learned model generates one or more predictions for a patient based on input of the scan data with or without input of other data (e.g., clinical data for the patient). The scan data is input to an input layer of the machine-learned model, which generates the prediction in response to input.

The machine-learned model was previously trained, such as having learned values for learnable parameters of a defined architecture based on optimization minimizing a loss over a large number (e.g., hundreds, thousands, or more) samples of training data (i.e., example input scan data and corresponding output ground truth). The optimization of the values of the learnable parameters minimizing the loss between the outputs of the model and the ground truths given the input examples identifies or sets the values of the learnable parameters to be used in the trained model.

For training the machine-learned model, the machine learning model arrangement is defined. The definition is by configuration or programming of the learning. The number of layers or units, type of learning, and other characteristics of the model are controlled by the programmer or user. In other embodiments, one or more aspects (e.g., number of nodes, number of layers or units, or type of learning) are defined and selected by the machine during the learning. Training data, including many samples of the input scan data and the corresponding ground truths (i.e., values of the characteristics), is used to train. The relationship of the input to the output is machine learned based on the training data. Once trained, the machine-learned model (machine-learned network) may be applied to generate the prediction from input scan data for a patient.

FIG.2shows an example model for machine training. The model is a neural network formed from a prior network210, a posterior network230and a U-Net250. This architecture uses convolutional layers and is the same or like the network used in Kohl, et al. Additional, different, or fewer components (e.g., networks) may be provided.

As taught in Kohl, et al., the U-Net250is a probabilistic U-Net. In another embodiment, the U-Net is a multi-scale U-Net, such as a residual U-Net (Res U-Net) or other network taught by Yu, et al. in “False Positive Reduction using Multi-scale Contextual Features for Prostate Cancer Detection in Multi-Parametric MRI Scans.” In alternative embodiments, an image-to-image or another model or network is used.

The prior network210and/or posterior network230are the same or different types of networks. In one embodiment, the prior and posterior networks210,230are separate encoders, such as conditional variational auto encoders. Other arrangements of convolutional or pooling layers with down-sampling may be used for the encoders. In alternative embodiments, other models (e.g., DenseNet) are used. Any regressor for generating values in a latent space from input scan data may be used.

For each encoder of the U-Net250, prior network210, and/or posterior network230, a series of down sampling and convolutional layers are used, such as an encoder from a U-Net or image-to-image network. Max or other pooling layers may be included. Dropout layers may be used. The encoder increases abstraction and decreases resolution. The final layer outputs a latent representation in latent space (e.g., latent space220for the prior network210, latent space240for the posterior network230, and bottleneck for the U-Net250). Values for one or more features are output by the encoder in response to input data, such as the MRI multi-parametric scan data200. In the example ofFIG.2, the multi-parametric scan data includes T2-weighted scan data, DWI scan data, ADC scan data, and a prostate mask. The latent representations are values for features at an abstracted level relative to the input scan data200. This latent representation is a fingerprint for the patient. The decoder of the U-Net250is a neural network, but other models may be used.

The neural networks are fully connected or convolutional networks. Any number of layers and nodes in layers may be used. Various layer arrangements may be used, such as sequential layers or densely connected layers. In one embodiment, some of the layers are convolutional layers. Pooling, up-sampling, down-sampling, dropout, or other layers may be used. Other architectures may be used. Different numbers of stages, layers, parameters, and/or convolutions in a layer may be used. Various kernel sizes, combinations of layers, and/or types of layers may be used.

InFIG.2, the prior network210, posterior network230, and U-Net250are to be trained using training data of example scan data200and corresponding ground truths270. The training data may be a database created for study or analysis, such as part of a clinical trial. In other approaches, the training data is created by an expert from medical records of patients. Some of the creation of the training data may be automated, such as using different segmentation algorithms to create the ground truth and/or using modeling to create synthetic samples.

FIG.2represents an arrangement to train the model. The image processor machine trains the model. The training learns values for weights, connections, filter kernels, and/or other learnable parameters of the defined architecture. Deep or another machine learning may be used. The weights, connections, filter kernels, and/or other parameters are the features being learned. For example, convolution kernels are features being trained. Using the training data, the values of the learnable parameters of the model are adjusted and tested to determine the values leading to an optimum estimation of the output given an input. Adam or another optimization is used to train.

The prior network210learns the variance of the input image or scan data200. The posterior network230learns the prediction distribution of the input image or scan data200over ground-truth segmentation maps270. The input scan data200is provided to each of the prior network210, posterior network230, and the U-Net250. The posterior network230also receives the ground truth segmentation270as an input. The U-Net250predicts the segmentation map260based on the learned segmentation distribution.

A low-dimensional latent space240is introduced withNto encode the learned N position segmentation variances. The posterior net230aims to find the valuable embeddings of the distribution for predicting segmentation maps260. The training images X (e.g., T2-weighted image, ADC, DWI, and prostate mask of scan data200) and ground-truth segmentation map Y270are encoded via neural networks. The posterior net230learns the mapping from the segmentation variances to a Gaussian distributed latent space with the mean μposterior(X, Y; v)∈Nand variance σposterior(X, Y; v)∈N, where v is the weight parameter (i.e., learnable parameter) of the posterior net230. The prior net210learns the embeddings of the training images X (scan data200) to estimate these segmentation variances in another latent space220with the probability distribution P. The distribution is modeled as axis-aligned Gaussian with mean μprior(X; ω)∈Nand variance σprior(X; ω)∈N, where ω is the weight parameter (i.e., learnable parameter) of the prior net210.

During training, the output of the posterior distribution Q (i.e., latent space240) is randomly sampled from the distribution donated as z˜Q(·|X, Y)=N(μposterior(X, Y; v)∈N, diag(σposterior(X, Y; v)∈N)). The last feature map (e.g., values of the features in the last hidden layer or another layer) from Res U-Net250is concatenated with the sample z from the continuous sampling of the latent space240and predicts the variance segmentation map S. The last layer of the feature map of the U-Net250is concatenated with the z sampled from the latent space240and predicts the semantic lesion candidate regions260.

In training, two losses are used as the objective function for minimization. The loss between the values of features in the latent space220from the prior network210and the values of features in the latent space240from the posterior network230is one loss, and the loss between the predicted segmentation260and the ground truth270is another loss. Additional, different, or fewer losses may be used.

The loss between latent spaces220,240is a Kullback-Leibler Divergence (KL loss). The KL loss is computed to pull the prior distribution and the posterior distribution to be close as possible. Thus, with input images or scan data200only, the prior network210is able to map to the distribution that is closer to the posterior distribution. The KL loss is donated as LKL=DKL(Q∥P). Other losses between values of features in latent space may be used.

The loss between the prediction (e.g., segmentation260) and the ground truth (e.g., ground truth segmentation270) is a cross-entropy loss (CE loss). Considering the output of the segmentation map S as the probability of pixel-level classification, the CE loss is conducted to compare S with ground-truth Y. The CE loss is denoted asz˜Q(·|X,Y)[−log Pc(Y|S(X,z))]. Other losses between a prediction and ground truth may be used, such as L1or L2losses.

The training uses the combination of losses for joint learning. The final objective function is an average or weighted average of losses, such as a joint loss L:L=LLK+λLCEwhere the λ is a relative weight factor for the losses. In alternative embodiments, the U-Net25is trained separately from the prior and posterior networks210,230, such as sequentially training in an iterative manner. Other training of other networks learning the distribution of the training data may be used.

Once trained, the machine-learned model (e.g., machine-learned network) generates a prediction (e.g., segmentation of a lesion) in response to input for a particular patient. The machine-learned model operates based on how the model was trained. Different training results in different values of learnable parameters, so results in different operation. The values of the learnable parameters are fixed or stay the same for application or testing. Given a previously unseen input, the machine-learned model generates an output (e.g., predicated segmentation260).

FIG.3shows an example. The posterior network230is not used in testing or application. Instead, the U-Net250generates one or more segmentations260in response to input of the multi-parametric data (e.g., in response to input of the T2-weighted data, apparent diffusion coefficient data, diffusion-weighted data, and a prostate mask). The segmentation(s)260are generated in response to input of the multi-parametric data200to the U-Net250and the prior network210of the machine-learned network. The latent space220generated by the prior network210in response to input is sampled (i.e., the samples z), which samples z are concatenated with the last or other layer of the U-Net250to form the output segmentation or segmentations260. The multi-parametric data200is input to both the U-Net250and the encoder of the prior network210. The prior network210generates values in the latent space220, where one or more samples of the values in the latent space220are provided to one of the convolutional layers of the U-Net250. The U-Net250generates the segmentation260based in part on the input of the multi-parametric data200and in part on the sample z from the latent space220. The sample z from the latent space220is concatenated with a last activation map of the residual U-Net250.

Where different samples z are selected from the latent space220, then different corresponding segmentations260are generated for the same scan data200. The machine-learned model (e.g., network210and network250) generates the segmentations260with different samples z. Any number of samples z may be selected from the latent space220to generate a corresponding number of segmentations260.

In the example ofFIG.3, M predictions of segmentations260are generated from M different samples z from the latent space220. The sampling from the latent space220may be systematic. In one embodiment, the sampling is random. The latent space220is randomly sampled for the M different samples z to generate the M different segmentations260.

In another embodiment, samples z are combined. For example, a mean of selected or all samples is calculated. The mean is provided as the sample z to generate a segmentation260. As another example, a variance of the selected or all samples is calculated. The variance is provided as the sample z to generate a segmentation260.

In act130ofFIG.1, the image processor determines the uncertainty of the prediction by the machine-learned model. The quantitative uncertainty is predicted, such as in the PCa lesion detection task, to provide information to the user regarding certainty of the prediction and/or locations of the prediction with greatest or different uncertainty.

The uncertainty is determined using the machine-learned network. Rather than testing different inputs to determine uncertainty, the latent space is sampled to determine the uncertainty. Since the latent space represents a variance distribution, the latent space may be used to determine uncertainty.

In one embodiment shown inFIG.3, the uncertainty is a variance in the segmentations260. Multiple predictions are generated using sampling of the latent space220. A variance of the multiple predictions is the uncertainty. Greater variance indicates greater uncertainty, and lesser variance indicates lesser uncertainty. The uncertainty may be determined spatially, such as variance for each pixel or voxel of the predicted segmentation260. For example, the uncertainty is a spatial heat map of the variance.

The uncertainty is determined via a group of samples. As shown inFIG.3, for the testing, a group of samples ZMare randomly selected from the latent space220and further concatenated with the last feature map of Res U-Net250to predict lesion segmentation maps SM. The final prediction of the segmentation map300is donated as the mean, average, or other combination of the samples. For the mean, the mean segmentation S′ is given as:

and the uncertainty is defined as the variance of samples

Other measures of variation may be used to reflect the uncertainty. Where the prediction is not spatial, the uncertainty may be a variance calculated as a statistic.

If the testing or application images have ambiguous regions, the samples selected from latent space might have a significant variance due to the uncertainty of the encoded input images. Vice versa, if the network has high confidence to encode the segmentation variance, the predicted segmentation maps may have high correlation but vary in segmentation size or shape, especially in the boundary. The spatial representation of the uncertainty calculated from the segmentations260indicates these different types of uncertainty. Visual examination may be used to determine where and/or what type of uncertainty exists in the predicted segmentation or other prediction.

In one embodiment, a given number of segmentations260is selected and used. For example, the M different segmentations260are randomly selected (or M samples are randomly selected to form the M segmentations) and used to form the mean300and variance310. M is any positive integer, such as 8, 16, or 32. The uncertainty is determined based on variance in the output space260.

In another embodiment, the predicted segmentation300is formed by a sample z that is the mean of the values in the latent space220and/or by random sampling and forming a mean of the resulting segmentations260.

In act140ofFIG.1, the image processor generates an image, which is displayed on a display. The display includes the prediction and/or the uncertainty. For example, an MR image is displayed. A boundary and/or region for the segmentation is overlaid or added to the image of tissue of the patient. Similarly, the uncertainty is displayed adjacent to the MR image or overlaid on a repetition of the same MR image displayed at a same time. Other display arrangements showing the prediction and uncertainty in the prediction may be used, such as one medical image of the patient with a boundary overlay showing the segmentation and a color overlay showing uncertainty by location.

In one embodiment, the prediction (e.g., mean300of multiple predicted segmentations260) and the uncertainty for the prediction (e.g., variance310) are displayed as heat maps. Color and/or intensity modulation of an MR image provide the heat map. Different colors and/or different simultaneously displayed images may be used for the segmentation and the uncertainty. In alternative embodiments, the uncertainty and/or prediction are displayed as text or a graphic, such as where the prediction is detection or classification.

Using the arrangement ofFIG.2, the networks210,230,250are trained. In the training and validation, 14 datasets are collected by a Siemens's MR scanner. Training and validation cases are conducted to train the Probabilistic U-Net250and the prior network210. Seven datasets are evaluated for testing. There are 664 (47% of total testing data) in-distribution cases (i.e., cases using a same scanning or imaging configuration and same type of scanner). Additionally, 768 out-of-distribution cases (53% of total testing data) are collected from different manufactories, such as GE, with high b-values larger than 1200, from different protocols, and/or containing artifacts. Some of the out-of-distribution cases show poor performance on lesion detection results.

After training (i.e., for testing), the uncertainty is determined. Examples of the visualization of the uncertainty prediction is shown inFIGS.4and5. The input images are T2, ADC, DWI, and prostate segmentation (first four columns). The ground truth is the ground truth of the lesion segmentation mask (fifth column). The predicted mean (sixth column) is a predicted lesion heatmap formed as a mean of the learned distribution from the latent space. The predicted average (seventh column) is the mean of 32 predicted lesion heatmaps via sampled vectors within the prostate region. The uncertainty (overly of eight column and mask of ninth column) is computed by obtaining the variance of the 32 predicted lesion heatmaps within the prostate region.

FIG.4illustrates three testing cases (i.e., each row is one testing case) with high uncertainty on the boundary of the predicted segmentation. The inputs, ground truth, mean, average, uncertainty as an overlay, and uncertainty as a mask are shown (rows) for each case.FIG.5shows the high uncertainty area over the suspicious region in the high confident segmentation prediction. Color overlay is used to show the segmentation and uncertainty. The results demonstrate the effectiveness of detecting the uncertain pixel-level lesion prediction of shape and boundaries furthermore the suspicious predicted regions. InFIG.4, the uncertainty is greatest along the boundaries. InFIG.5, the uncertainty is of the lesion.

FIG.6shows different predicted segmentations for two cases with high uncertainty (i.e., false positive). Two cases each with 16 different predicted segmentations are shown as color overlays on MR images of the prostate. The 16 different cases are for 16 randomly sampled samples z of the latent space. Two dimensions are involved in visualizing the prior sample zNin the probabilistic U-Net250. The two cases map to a variance of 2 in z0-z1latent space. The green area indicates the predicted region under different z samples. The learned variance on the space and size of the predicted region is clearly shown.

The probabilistic U-Net250is applied to PCa detection for uncertainty estimation. The results show the effectiveness of the model for lesion segmentation uncertainty prediction and the suspicious predicted region detection. This end-to-end training strategy can predict the lesion region, learn the segmentation variance, and detect the uncertainty region simultaneously. Further, the uncertainty map improves the PCa detection accuracy by delivery of more information to the physician.

FIG.7shows one embodiment of a medical system for uncertainty estimation. The medical system implements the method ofFIG.1or another method.

The medical system includes the display700, memory720, and image processor710. The display700, image processor710, and memory720may be part of the MR imager730, a computer, server, workstation, or other system for image processing medical images from a scan of a patient. A workstation or computer without the MR imager730may be used as the medical system.

Additional, different, or fewer components may be provided. For example, a computer network is included for remote image generation of locally captured scan data or for local estimation of patient characteristics from remotely captured scan data. The machine-learned model715is applied as a standalone application on the workstation or a local device or as a service deployed on network (cloud) architecture. As another example, a user input device (e.g., keyboard, buttons, sliders, dials, trackball, mouse, or other device) is provided for user alteration or placement of one or more markers (e.g., landmarks) or segmentation selection or adjustment. In yet another example, the medical imager730is not provided.

The scan data (e.g., multi-parameter data), network definition, values of learned parameters, machine-learned model715, feature values, values of characteristics, predictions (e.g., segmentations), uncertainty, other outputs, display image, and/or other information are stored in a non-transitory computer readable memory, such as the memory720. The memory720is an external storage device, RAM, ROM, database, and/or a local memory (e.g., solid state drive or hard drive). The same or different non-transitory computer readable media may be used for the instructions and other data. The memory720may be implemented using a database management system (DBMS) and residing on a memory, such as a hard disk, RAM, or removable media. Alternatively, the memory720is internal to the processor710(e.g., cache).

The instructions for implementing the training or application processes, the methods, and/or the techniques discussed herein are provided on non-transitory computer-readable storage media or memories, such as a cache, buffer, RAM, removable media, hard drive, or other computer readable storage media (e.g., the memory720). Computer readable storage media include various types of volatile and nonvolatile storage media. The functions, acts or tasks illustrated in the figures or described herein are executed by the image processor710in response to one or more sets of instructions stored in or on computer readable storage media. The functions, acts or tasks are independent of the particular type of instructions set, storage media, processor or processing strategy and may be performed by software, hardware, integrated circuits, firmware, micro code, and the like, operating alone or in combination.

In one embodiment, the instructions are stored on a removable media device for reading by local or remote systems. In other embodiments, the instructions are stored in a remote location for transfer through a computer network. In yet other embodiments, the instructions are stored within a given computer, CPU, GPU, or system. Because some of the constituent system components and method steps depicted in the accompanying figures may be implemented in software, the actual connections between the system components (or the process steps) may differ depending upon the way the present embodiments are programmed.

The imager730is a diagnostic scanner, such as an MR scanner or system. The imager730operates pursuant to one or more settings to scan a patient740resting on a bed or table750. The settings control scanning including transmission, reception, reconstruction, and image processing. A scanning protocol is followed to generate data representing the patient740, such as generating multi-parametric scan data representing the prostate of the patient. The patient740is imaged by the imager730using the settings. Other imagers, such as CT, PET, SPECT, or ultrasound imagers, may alternatively, or additionally, be used.

The image processor710is a controller, control processor, general processor, digital signal processor, three-dimensional data processor, graphics processing unit, application specific integrated circuit, field programmable gate array, artificial intelligence processor, tensor processor, digital circuit, analog circuit, combinations thereof, or another now known or later developed device for processing scan data. The image processor710is a single device, a plurality of devices, or a network of devices. For more than one device, parallel or sequential division of processing may be used. Different devices making up the image processor710may perform different functions. In one embodiment, the image processor710is a control processor or other processor of the medical imager730. The image processor710operates pursuant to and is configured by stored instructions, hardware, and/or firmware to perform various acts described herein.

The image processor710or another remote processor is configured to train a machine learning architecture. Based on a user provided or other source of the network architecture and training data, the image processor710learns to relate one or more input variables (e.g., MR scan data) to output predictions (e.g., segmentation). The result of the training is a machine-learned network for patient modeling.

Alternatively, or additionally, the image processor710is configured to apply the scan data to a machine-learned network. The machine-learned network, based on past training, is configured to output a classification, detection, and/or segmentation in response to the application. For example, the machine-learned network is configured to output the segmentation as a segmentation of a lesion, such as a prostate cancer. The machine-learned network is configured to output one or a plurality of the classification, detection, and/or segmentation in response to the application of the scan data representing the patient. The outputs are generated using, in part, the random samples. The prediction (classification, detection, and/or segmentation) for display or output may be a mean, average, or other combination of multiple predictions.

The image processor is further configured to determine an uncertainty of the classification, detection, and/or segmentation with random samples from a latent space. The uncertainty is a heat map with greater uncertainty for some locations than others represented in the heat map. Other uncertainty representations may be used. The uncertainty may be a variance in output segmentations and/or a variance in the latent space. In one embodiment, the uncertainty is a variance of random samples or segmentations generated from random samples.

The machine-learned network is the U-Net and an encoder (prior network) separate from the U-Net ofFIG.3but may be other networks. The machine-learned network is configured to generate the latent space from which latent samples are sampled for determining uncertainty.

The display700is a CRT, LCD, projector, plasma, printer, tablet, smart phone, or other now known or later developed display device for displaying the output, such as an image of classification, detection, and/or segmentation. In one embodiment, the display700displays a MR image of the prostate with a color, boundary graphic, or other designation of detection of a segment representing a lesion. The display700also displays the uncertainty, such as a different heatmap of color or graphic overlaying the MR image.