Patent Description:
Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output. <NPL> relates to a generative segmentation model based on a combination of a U-Net with a conditional variational autoencoder that is capable of efficiently producing an unlimited number of plausible hypotheses.

This specification describes a segmentation system implemented as computer programs on one or more computers in one or more locations that is configured to process an image to generate multiple possible segmentations of the image. Each segmentation may represent a different hypothesis for the ground truth (i.e., actual) segmentation of the image.

A method, a system and one or more computer storage media according to the present invention are set out in the appended claims.

The segmentation system described in this specification can generate a large number of plausible segmentations of an image. By generating multiple possible segmentations of an image, the segmentation system models the inherent ambiguities present in segmenting the image. In one example, in medical imaging applications, a lesion might be visible in an image, but whether or not the lesion is malignant and the precise boundaries of the lesion may be ambiguous. In this example, the segmentation system may generate multiple different segmentations that each define respective boundaries of the lesion and respective predictions for whether the lesion is malignant. Diagnosis and treatment of patients may depend on segmentations of medical images (e.g., CT images). Providing only the most likely segmentation of a medical image may lead to misdiagnosis and sub-optimal treatment. The segmentation system described in this specification can provide multiple possible segmentations of a medical image that can be directly propagated into the next step in a diagnosis pipeline, used to suggest further diagnostic tests to resolve ambiguities, or provided to an expert for analysis, thereby potentially improving patient outcomes.

The segmentation system described in this specification may enable more efficient use of resources. For example, the segmentation system can facilitate more efficient resource usage by obviating the need for multiple people (e.g., physicians) to generate different manual segmentations of an image. As another example, rather than using different systems to generate multiple segmentations of an image from scratch (e.g., different neural network models), the system described in this specification can re-use the results of certain previous computations each time it generates a new segmentation of an image.

To generate a possible segmentation of an image, the segmentation system samples a latent variable from a latent space, and processes the latent variable and the image using a segmentation neural network. The system described in this specification can model the diversity and variations of possible image segmentations at various scales (e.g., spanning from the pixel level to the image level) by sampling latent variables from a "hierarchical" latent space that includes a hierarchy (sequence) of latent sub-spaces. The system samples a respective latent sub-variable from each latent sub-space in accordance with a probability distribution over the latent sub-space that is generated based on: (i) the image, and (ii) latent sub-variables sampled from any preceding latent sub-spaces in the hierarchy. By sampling from a hierarchical latent space, the system can generate segmentations of an image that more accurately model complex variations in segmentation structures across scales.

<FIG> shows an example segmentation system <NUM>. The segmentation system <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The segmentation system <NUM> is configured to process an image <NUM> to generate multiple possible segmentations <NUM> of the image <NUM>, where each segmentation <NUM> represents a different hypothesis for the ground truth (i.e., actual) segmentation of the image <NUM>.

The image <NUM> may be represented as an ordered collection of numerical values, e.g., as a one-dimensional (<NUM>-D), two-dimensional (<NUM>-D), three-dimensional (<NUM>-D), or N-dimensional (N-D) array of numerical values (where A is a positive integer value). As used throughout this specification, the components of an array of numerical values representing the image may be referred to as "pixels" or "voxels".

The image <NUM> may be captured using any appropriate imaging modality, e.g., a magnetic resonance imaging (MRI) modality, a computed tomography (CT) imaging modality, a microscope imaging modality (e.g., a light microscope or electron microscope imaging modality), a photographic (camera) imaging modality, an optical coherence tomography (OCT) imaging modality, an ultrasound (US) imaging modality, or an X-ray imaging modality.

In some implementations, the image <NUM> may include multiple images that are each captured using a respective imaging modality. For example, the image <NUM> may include a concatenation of an MRI image and a CT image (e.g., of the brain of a patient). In some implementations, the image <NUM> may include multiple images that are each captured at respective time points. For example, the image <NUM> may include a concatenation of MRI images (e.g., of the brain of a patient) that are each captured at a respective time point (e.g., such that the image <NUM> characterizes the progression of a disease in a patient).

Each segmentation <NUM> of the image <NUM> may be a "soft" segmentation that defines, for each pixel of the image <NUM>, a respective score for each category in a predefined set of categories, where the score for a category characterizes a probability that the pixel is included in the category. Alternatively, each segmentation <NUM> of the image <NUM> may be a "hard" segmentation that specifies, for each pixel of the image <NUM>, a corresponding category from the predefined set of categories. Generally, each segmentation <NUM> may be represented as an ordered collection of numerical values, e.g., an array of numerical values.

The segmentations <NUM> may segment the image <NUM> into any appropriate set of categories. For example, the image <NUM> may be a medical image (e.g., an MRI or CT image of a region of the body of a patient, or a microscope image of a fragment of tissue from a patient) and each category may represent a possible tissue type. In one example, the tissue types may include: "normal" tissue and "abnormal" (e.g., cancerous) tissue. In another example, each tissue type may specify a respective region of the body (e.g., bone tissue, heart tissue, liver tissue, lung tissue, etc.).

To generate a segmentation <NUM> of the image <NUM>, the system <NUM> processes the image <NUM> to determine a probability distribution <NUM> over a latent space, and uses a sampling engine <NUM> to sample a latent variable <NUM> from the latent space in accordance with the probability distribution <NUM>. The system <NUM> processes: (i) the image <NUM>, and (ii) the sampled latent variable <NUM>, using a segmentation neural network <NUM> to generate a segmentation <NUM> of the image <NUM>.

Generally, the "latent space" may represent a collection of possible latent variables, and each latent variable may be represented as an ordered collection of one or more numerical values (e.g., a vector or matrix of numerical values). Each latent variable may specify a corresponding segmentation <NUM> of the image <NUM>, and the system <NUM> may generate a probability distribution <NUM> over the latent space that assigns a higher probability to latent variables corresponding to more accurate segmentations of the image. To generate multiple segmentations <NUM> of the image <NUM>, the system <NUM> repeatedly samples latent variables <NUM> from the latent space, and generates a respective segmentation <NUM> corresponding to each sampled latent variable <NUM>.

The system <NUM> can generate probability distributions <NUM> over the latent space and sample latent variables <NUM> in a variety of ways; a few examples are described in more detail next.

The system <NUM> may generate the latent variable by sampling from the latent space in accordance with a probability distribution over the latent space that depends on the image <NUM>, but not on any portion of the latent variable itself. In this example, which does not form part of the invention claimed herein, the latent variable may be referred to as a "global" latent variable obtained using a "global" sampling technique. An example implementation of the segmentation system <NUM> that samples global latent variables and does not form part of the invention claimed herein is described in more detail with reference to <FIG>.

The system generates the latent variable by sequentially sampling a respective latent sub-variable from each latent sub-space in a sequence of latent sub-spaces in accordance with a respective probability distribution over each latent sub-space. The generated sequence of latent sub-variables collectively define the latent variable <NUM>. As used throughout this specification, a "latent sub-space" may represent a collection of possible latent sub-variables, and each latent sub-variable may be represented as an ordered collection of one or more numerical values (e.g., a vector or matrix of numerical values). In this example, the latent variable may be referred to as a "hierarchical" latent variable obtained using a "hierarchical" sampling technique. An example implementation of the segmentation system <NUM> that samples hierarchical latent variables is described in more detail with reference to <FIG>.

The system <NUM> generates the probability distribution over each latent sub-space based on: (i) the image <NUM>, and (ii) latent sub-variables sampled from preceding latent sub-spaces (i.e., that precede the latent sub-space in the sequence of latent sub-spaces). For example, the system <NUM> may generate a probability distribution over the first latent sub-space (i.e., in the sequence of latent sub-spaces) as an output of one or more neural network layers that process the image <NUM> (or feature maps corresponding to the image <NUM>). For each subsequent latent sub-space, the system <NUM> may generate a probability distribution over the latent sub-space as an output of one or more neural network layers that process: (i) the image <NUM> (or feature maps corresponding to the image <NUM>), and (ii) respective latent sub-variables sampled from preceding latent sub-spaces.

As used throughout this specification, a feature map corresponding to the image <NUM> may refer to an output generated by processing the image <NUM> using one or more neural network layers. For example, a feature map corresponding to the image <NUM> may be generated as an intermediate output of the segmentation neural network by processing the image <NUM>. An intermediate output of the segmentation neural network <NUM> may refer to an output generated by one or more intermediate (hidden) layers of the segmentation neural network.

Optionally, the sequence of latent sub-spaces may be a sequence of increasing dimensionality, i.e., such that the dimensionality of each latent sub-space is greater than the dimensionality of the preceding latent sub-space. Latent variables sampled from latent sub-spaces with lower dimensionalities may implicitly characterize global features relevant to the segmentation as a whole, e.g., the predisposition of a patient for a certain disease. Latent variables sampled from latent sub-spaces with higher dimensionalities may implicitly characterize local features relevant to specific parts of the segmentation, e.g., whether a particular nodule is cancerous, or the position of the boundary of a particular nodule.

The segmentation neural network <NUM> can have any appropriate neural network architecture that enables it to perform its described functions, in particular, processing an image <NUM> and a latent variable <NUM> to generate a corresponding segmentation <NUM> of the image <NUM>. For example, the segmentation neural network <NUM> may have a convolutional neural network architecture that includes one or more convolutional neural network layers. Example architectures of the segmentation neural network <NUM> are described in more detail below with reference to <FIG> and <FIG>.

<FIG> shows an example segmentation system <NUM>, which is an example implementation of the segmentation system <NUM> (described with reference to <FIG>) that uses a global sampling technique for sampling latent variables and does not form part of the invention claimed herein. The segmentation system <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The segmentation system <NUM> is configured to process an image <NUM> to generate multiple segmentations <NUM> of the image <NUM>, where each segmentation is specified by a latent variable sampled from a latent space in accordance with a global sampling technique.

To generate a segmentation <NUM>, the system <NUM> may process the image <NUM> using a prior neural network <NUM> to generate an output that defines the parameters of a probability distribution <NUM> over the latent space. In one example, the latent space may be an N-dimensional Euclidean space (i.e., <MAT>) and the probability distribution over the latent space may be a Normal probability distribution specified by a mean vector µprior and a covariance matrix (or covariance vector) σprior.

The prior neural network <NUM> may have any appropriate neural network architecture that enables it to perform its described function. For example, the prior neural network <NUM> may have a convolutional neural network architecture that includes an input layer, one or more intermediate (i.e., hidden) convolutional neural network layers, and a fully-connected output layer.

The system <NUM> samples a latent variable <NUM> from the latent space in accordance with the probability distribution <NUM> over the latent space, and provides an input including: (i) the image <NUM>, and (ii) the sampled latent variable <NUM>, to the segmentation neural network <NUM>. The segmentation neural network <NUM> is configured to process the input to generate a corresponding segmentation <NUM> of the image <NUM>.

The segmentation neural network <NUM> may have any appropriate neural network architecture that enables it to perform its described function. For example, the segmentation neural network may include: (i) a feature extraction sub-network, and (ii) a prediction sub-network. (As used throughout this specification, a "sub-network" of a neural network refers to a group of one or more layers of the neural network).

The feature extraction sub-network of the segmentation neural network <NUM> may be configured to process the image <NUM> to generate a feature map characterizing the image <NUM>. The feature map may be represented, e.g., as an array of numerical values having one or more "spatial" dimensions and a "channel" dimension. The feature extraction sub-network may have any appropriate neural network architecture, e.g., a convolutional neural network architecture including one or more convolutional neural network layers. In a particular example, the feature extraction sub-network may have the U-Net neural network architecture described with reference to: <NPL>.

The prediction sub-network may be configured to combine the feature map (i.e., generated by the feature extraction sub-network) and the latent variable <NUM>, and to process the combination of the feature map and the latent variable <NUM> to generate the segmentation <NUM>. The prediction sub-network may combine the feature map and the latent variable, e.g., by "broadcasting" (i.e., concatenating) the latent variable along the spatial dimensions of the feature map. The prediction sub-network may have a neural network architecture that includes one or more <NUM> × <NUM> convolutional neural network layers, i.e., convolutional layers with <NUM> × <NUM>-dimensional kernels.

To generate multiple segmentations <NUM> of the image <NUM>, the system <NUM> can sample multiple latent variables from the latent space, i.e., in accordance with the probability distribution over the latent space generated by the prior neural network <NUM>. The system <NUM> may avoid regenerating the parameters of the probability distribution over the latent space each time a latent variable is sampled, e.g., by generating the probability distribution parameters only once, and thereafter storing and reusing them. Similarly, the system <NUM> may avoid regenerating the feature map output by the feature extraction sub-network of the segmentation neural network <NUM> each time a latent variable is sampled, e.g., by generating the feature map only once, and thereafter storing and reusing it.

<FIG> shows an example training system <NUM> for training a segmentation system that uses a global sampling technique for sampling latent variables, e.g., the segmentation system <NUM> described with reference to <FIG>, and does not form part of the invention claimed herein. The training system <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The training system <NUM> is configured to train the segmentation system by jointly training the prior neural network <NUM> and the segmentation neural network <NUM> (i.e., described with reference to <FIG>) along with a posterior neural network <NUM> on a set of training data. The posterior neural network <NUM> (which will be described in more detail below) facilitates training the segmentation system to generate probability distributions over the latent space that assign a higher probability to latent variables corresponding to more accurate segmentations of input images.

The training data may include multiple training examples, where each training example includes: (i) a training image, and (ii) a target segmentation of the training image. Each target segmentation may be manually generated, e.g., by a human expert. To enable the segmentation system to learn a range of variations among plausible segmentations of a training image, the training data may include multiple training examples having the same training image but different target segmentations, e.g., that are generated by different human experts.

The posterior neural network <NUM> is configured to process: (i) a training image <NUM>, and (ii) a target segmentation <NUM> of the training image <NUM>, to generate an output that defines the parameters of a "posterior" probability distribution <NUM> over the latent space. In one example, the latent space may be an N-dimensional Euclidean space (i.e., <MAT>) and the posterior probability distribution may be a Normal (i.e., Gaussian) probability distribution specified by a mean vector µpost and a covariance matrix (or covariance vector) σpost.

The posterior neural network <NUM> may have any appropriate neural network architecture that enables it to perform its described function. For example, the posterior neural network <NUM> may have a convolutional neural network architecture that includes an input layer, one or more intermediate (i.e., hidden) convolutional neural network layers, and a fully-connected output layer.

At each of multiple training iterations, the training system <NUM> obtains a batch of training examples, e.g., by sampling one or more training examples from the training data. For each training example, the training system <NUM> processes the training image <NUM> and target segmentation <NUM> specified by the training example using the posterior neural network <NUM> to generate a corresponding posterior probability distribution <NUM> over the latent space. The training system <NUM> samples a latent variable <NUM> from the latent space in accordance with the posterior probability distribution <NUM> generated for the training example, and processes the training image <NUM> and the sampled latent variable <NUM> using the segmentation neural network <NUM> to generate a predicted segmentation <NUM>. The training system <NUM> also processes the training image <NUM> specified by the training example using the prior neural network <NUM> to generate a "prior" probability distribution <NUM> over the latent space.

The training system <NUM> then jointly optimizes the parameters of the prior neural network <NUM>, the segmentation neural network <NUM>, and the posterior neural network <NUM> to optimize an objective function, e.g., using gradient descent. The objective function<IMG> (evaluated for a given training example) may be given by:<MAT> where Err(Y, Ŷ) denotes an error (e.g., a cross entropy error) between the target segmentation Y <NUM> and the predicted segmentation Ŷ <NUM> (i.e., that is generated by processing a latent variable sampled in accordance with the posterior probability distribution), β is a constant factor, and D(P, Q) represents an error (e.g., a Kullback-Leibler divergence) between the prior probability distribution P <NUM> and the posterior probability distribution Q <NUM> for the training example. Optimizing the objective function in equation (<NUM>) encourages the prior probability distribution <NUM> to match the posterior probability distribution <NUM>, and encourages the predicted segmentation <NUM> corresponding to a latent variable sampled in accordance with the posterior distribution <NUM> to match the target segmentation <NUM>. The objective function given by equation (<NUM>) is provided for illustrative purposes only; other objective functions are possible.

<FIG> shows an example segmentation system <NUM>, which is an example implementation of a segmentation system <NUM> (described with reference to <FIG>) that uses a hierarchical sampling technique for sampling latent variables. The segmentation system <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The segmentation system <NUM> is configured to process an image <NUM> using a segmentation neural network <NUM> to generate multiple segmentations <NUM> of the image <NUM>, where each segmentation is specified by a latent variable sampled from a latent space in accordance with a hierarchical sampling technique. To sample a latent variable, the system <NUM> samples a respective latent sub-variable from each latent sub-space in a sequence of latent sub-spaces in accordance with a probability distribution over each latent sub-space, and the generated sequence of latent sub-variables collectively defines the latent variable.

The segmentation neural network <NUM> includes a sequence of one or more "encoder" blocks, e.g., <NUM>-A-C, a sequence of one or more "decoder" blocks, e.g., <NUM><NUM>-A-E, and one or more "prior" blocks, e.g., <NUM>-A-C. A "block" refers to a group of one or more neural network layers. Generally, the input to a block and the output of a block may be represented as respective arrays of numerical values that are indexed along one or more "spatial" dimensions (e.g., x-y dimensions, or x-y-z dimensions) and a "channel" dimension. The "resolution" of a block input/output along a dimension refers to the number of index values along that dimension.

Each encoder block is configured to process an encoder block input to generate an encoder block output having a lower spatial resolution than the encoder block input, i.e., such that the resolution of the encoder block output is less than the resolution of the encoder block input along at least one spatial dimension. The first encoder block may process the image <NUM>, and each subsequent encoder block may process a feature map output by the preceding encoder block.

Each decoder block is configured to process a decoder block input to generate a decoder block output having a higher spatial resolution than the decoder block input, i.e., such that the resolution of the decoder block output is greater than the resolution of the decoder block input along at least one spatial dimension. The first decoder block in the segmentation neural network <NUM> may process an input including the output of a corresponding encoder block, and each subsequent decoder block may process an input including: (i) the output of a corresponding encoder block, and (ii) the output of the preceding decoder block.

Each prior block corresponds to a respective decoder block, and is configured to process a feature map generated by the corresponding decoder block to generate an output that defines the parameters of a probability distribution over a corresponding latent sub-space. The latent sub-space may be a space of arrays of numerical values having the same spatial resolution as the feature map generated by the corresponding decoder block. For example, as illustrated by <NUM>, the prior block <NUM>-B may process a feature map <MAT> (i.e., having spatial resolution <NUM> × <NUM> and channel resolution C) by one or more <NUM> × <NUM> convolutional neural network layers to generate the parameters of a Normal probability distribution over <MAT>, where µ ∈ <MAT> is an array of mean parameters and <MAT> is an array of standard deviation parameters. The system <NUM> may then sample a latent sub-variable <MAT> in accordance with the probability distribution parameters (µ, σ).

After generating the parameters of a probability distribution over a latent sub-space corresponding to a decoder block, the system <NUM> may sample a latent sub-variable from the latent sub-space and provide the latent sub-variable as an input to the next decoder block (i.e., in the sequence of decoder blocks). For example, the system <NUM> may concatenate the latent sub-variable to the other inputs to the next decoder block along their respective spatial dimensions. In some cases, the latent sub-variable may have a lower spatial resolution than the decoder block output, and the system <NUM> may up-scale the latent sub-variable to have the same spatial resolution as the other inputs to the next decoder block. Up-scaling the latent sub-variable may refer to mapping the array of numerical values representing the latent sub-variable to another array of numerical values having a higher spatial resolution, e.g., where some data values in the up-scaled array may be interpolated between data values in the original array. It can be appreciated that the sampled sequence of latent sub-variables may be a sequence of arrays of numerical values of increasing spatial resolution, i.e., such that the spatial resolution of each latent sub-variable is greater than the spatial resolution of the preceding latent sub-variable.

The final decoder block of the segmentation neural network <NUM> may generate an output that includes the segmentation <NUM> of the input image <NUM>. Each time the system <NUM> processes an image <NUM> using the segmentation neural network <NUM>, a different sequence of latent sub-variables may be sampled from the respective latent sub-spaces and a different segmentation <NUM> of the image <NUM> may be generated. The system <NUM> may avoid regenerating the feature maps generated by the encoder blocks each time the same image <NUM> is processed, e.g., by generating these feature maps only once, and thereafter storing and reusing them each time the same image is processed.

Generating latent sub-variables represented as arrays of numerical values having the same spatial resolution as the feature maps generated by corresponding decoder blocks (as described above) endows the latent sub-variables with spatial structure. The spatial structure of the latent sub-variables can improve the capacity of the system <NUM> to model independent local variations, e.g., the presence of multiple lesions.

In some implementations, the segmentation neural network <NUM> includes a respective prior block corresponding to each decoder block, while in other implementations, the segmentation neural network <NUM> includes a respective prior block corresponding to only a proper subset of the decoder blocks.

In some implementations, the segmentation neural network <NUM> may be implemented as a fully-convolutional neural network, i.e., where each layer is a convolutional layer. In these implementations, the segmentation neural network <NUM> may process images having arbitrary spatial resolutions, and the spatial resolutions of the inputs/outputs of the neural network blocks may vary (e.g., proportionally) based on the spatial resolution of the image being processed. In particular, the spatial resolutions of the latent sub-variables (which may be directly linked to the spatial resolutions of the feature maps generated by the decoder blocks) may vary based on the spatial resolution of the image being processed. For example, the system <NUM> may sample latent sub-variables with higher spatial resolutions when processing an image with a higher spatial resolution, which can enable the system <NUM> to more effectively capture the greater range of possible variations in segmentations of images with higher spatial resolutions.

<FIG> shows an example training system <NUM> for training a segmentation system that uses a hierarchical sampling technique for sampling latent variables, e.g., the segmentation system <NUM> described with reference to <FIG>. The training system <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The training system <NUM> is configured to train the segmentation system by jointly training the segmentation neural network <NUM> (i.e., described with reference to <FIG>) and a posterior neural network <NUM> on a set of training data. The posterior neural network <NUM> (which will be described in more detail below) facilitates training the segmentation neural network <NUM> to generate probability distributions over the latent sub-spaces that assign higher probabilities to sub-latent variables corresponding to more accurate segmentations of input images.

The posterior neural network <NUM> is configured to process: (i) a training image <NUM>, and (ii) a target segmentation <NUM> of the training image <NUM>, to generate an output that defines the parameters of a respective "posterior" probability distribution <NUM> over each latent sub-space in the sequence of latent sub-spaces. In one example, each latent sub-space may be a space of arrays of numerical values having a respective spatial resolution and the posterior probability distribution may be a Normal probability distribution specified by an array of mean parameters and an array of standard deviation parameters.

The posterior neural network <NUM> may have any appropriate neural network architecture that enables it to perform its described function. For example, the posterior neural network <NUM> may have an architecture that substantially matches the architecture of the segmentation neural network <NUM>, e.g., with a sequence of encoder blocks, a sequence of decoder blocks, and one or more "posterior" blocks (i.e., corresponding to the prior blocks of the prior neural network). Each posterior block may be configured to process a feature map generated by a corresponding decoder block of the posterior neural network to generate an output that defines the parameters of a posterior probability distribution over a corresponding latent sub-space. In contrast to the segmentation neural network <NUM>, the posterior neural network <NUM> may exclude the final decoder block of the segmentation neural network <NUM>, i.e., that generates the predicted segmentation. The dimensionalities of the inputs/outputs of the blocks of the posterior neural network <NUM> may also differ from the segmentation neural network <NUM>, e.g., because the posterior neural network <NUM> also processes the target segmentation <NUM> in addition to the training image <NUM>.

At each of multiple training iterations, the training system <NUM> obtains a batch of one or more training examples, and provides the training image <NUM> and target segmentation <NUM> specified by each training example as inputs to the posterior neural network <NUM>. For each training example, the training system <NUM> processes the training image <NUM> and the target segmentation <NUM> using the posterior neural network <NUM> to generate a respective posterior probability distribution <NUM> over each latent sub-space. The training system <NUM> also processes the training image <NUM> using the segmentation neural network <NUM> to generate a respective prior probability distribution <NUM> over each latent sub-space and a predicted segmentation <NUM> of the training image <NUM>. To generate the predicted segmentation <NUM>, the segmentation neural network <NUM> processes a respective latent variable sampled from each latent sub-space in accordance with the posterior probability distribution <NUM> over the sub-space (i.e., rather than latent sub-variables sampled in accordance with the prior probability distributions <NUM>).

The training system <NUM> then jointly optimizes the parameters of the segmentation neural network <NUM> and the posterior neural network <NUM> to optimize an objective function, e.g., using gradient descent. The objective function <IMG> (evaluated for a given training example) may be given by: <MAT> where Err(Y, Ŷ) denotes an error (e.g., a cross entropy error) between the target segmentation Y and the predicted segmentation Ŷ (i.e., that is generated by processing latent sub-variables sampled in accordance with the posterior probability distributions), β is a constant factor, i indexes the latent sub-spaces, L is the number of latent sub-spaces, and D(Pi, Qi) represents an error (e.g., a Kullback-Leibler divergence) between the prior probability distribution Pi and the posterior probability distribution Qi over the i-th latent sub-space. Optimizing the objective function in equation (<NUM>) encourages the prior probability distributions to match the posterior probability distributions, and encourages the predicted segmentation corresponding to latent sub-variables sampled in accordance with the posterior distributions to match the target segmentation. The objective function in equation (<NUM>) is provided for illustrative purposes only, and other objective functions are possible.

<FIG> shows a CT image <NUM> of a portion of a lung and an illustration of a (<NUM>-D) latent space <NUM>. For each of multiple positions in the latent space <NUM>, a segmentation (e.g., <NUM>) of a region of abnormal tissue in the CT image <NUM> that is generated by a segmentation neural network by processing the latent variable at the position in the latent space is overlaid on the latent space <NUM>. For ease of presentation, the latent space <NUM> has been rescaled so that the prior probability distribution over the latent space is a unit Normal distribution, and the circles <NUM>-A-D denote deviations from the mean of the prior probability distribution in integer multiples of the standard deviation σ. The white Xs (e.g., <NUM>) overlaid on the latent space <NUM> denote the positions in the latent space <NUM> corresponding to target segmentations of the abnormal tissue in the CT image <NUM> that were manually generated by different human experts.

<FIG> is a flow diagram of an example process <NUM> for generating multiple possible segmentations of an image. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a segmentation system, e.g., the segmentation system <NUM> of <FIG>, appropriately programmed in accordance with this specification, can perform the process <NUM>.

The system receives a request to generate multiple segmentations of an image (<NUM>).

The system samples multiple latent variables from a latent space, where each latent variable is sampled from the latent space in accordance with a respective probability distribution over the latent space that is determined based on (at least) the image (<NUM>).

In implementations, which do not form part of the invention claimed herein, the system samples each latent variable using a global sampling technique. In these implementations, the system samples each latent variable in accordance with a respective probability distribution over the latent space that depends on the image, but not on any portion of the latent variable itself.

The system samples each latent variable using a hierarchical sampling technique. In these implementations, the latent space includes a hierarchy (sequence) of latent subspaces, and sampling a latent variable from the latent space includes sequentially sampling a respective latent sub-variable from each latent sub-space of the latent space. To sample a latent sub-variable from a given latent sub-space, the system generates a probability distribution over the given latent sub-space based on: (i) the image, and (ii) latent sub-variables sampled from any latent sub-spaces that precede the given latent sub-space in the hierarchy of latent sub-spaces. The system then samples the latent sub-variable from the given latent sub-space in accordance with the probability distribution over the given latent sub-space.

The system generates multiple possible segmentations of the image (<NUM>). In particular, for each latent variable, the system processes the image and the latent variable using a segmentation neural network having a set of segmentation neural network parameters to generate the possible segmentation of the image.

The system provides the multiple segmentations of the image in response to the request (<NUM>).

Claim 1:
A method (<NUM>) performed by one or more data processing apparatus, the method comprising:
receiving (<NUM>) a request to generate a plurality of possible segmentations (<NUM>; <NUM>; <NUM>) of an image (<NUM>; <NUM>; <NUM>);
sampling (<NUM>) a plurality of latent variables (<NUM>) from a latent space (<NUM>), wherein each latent variable (<NUM>) is sampled from the latent space (<NUM>) in accordance with a respective probability distribution (<NUM>) over the latent space (<NUM>) that is determined based at least in part on the image (<NUM>; <NUM>; <NUM>), wherein:
the latent space (<NUM>) comprises a hierarchy of latent sub-spaces; and
sampling a latent variable (<NUM>) from the latent space (<NUM>) comprises sampling a respective latent sub-variable from each latent sub-space of the latent space (<NUM>), wherein sampling a latent sub-variable from a given latent sub-space comprises:
generating a probability distribution (<NUM>) over the given latent sub-space based on: (i) the image (<NUM>; <NUM>; <NUM>), and (ii) latent sub-variables sampled from any latent sub-spaces that precede the given latent sub-space in the hierarchy of latent sub-spaces; and
sampling the latent sub-variable from the given latent sub-space in accordance with the probability distribution (<NUM>) over the given latent sub-space;
generating (<NUM>) a plurality of possible segmentations (<NUM>; <NUM>; <NUM>) of the image (<NUM>; <NUM>; <NUM>), comprising, for each latent variable (<NUM>), processing the image (<NUM>; <NUM>; <NUM>) and the latent variable (<NUM>) using a segmentation neural network (<NUM>; <NUM>) having a plurality of segmentation neural network parameters to generate the possible segmentation of the image (<NUM>; <NUM>; <NUM>); and
providing (<NUM>) the plurality of possible segmentations (<NUM>; <NUM>; <NUM>) of the image (<NUM>; <NUM>; <NUM>) in response to the request.