Patent Publication Number: US-2022215646-A1

Title: Abdominal multi-organ segmentation with organ-attention networks

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application claims the benefit of U.S. Provisional Patent Application No. 62/837,679 filed Apr. 23, 2019, the entirety of which is incorporated herein by reference. 
    
    
     TECHNICAL FIELD 
     The present disclosure is directed to segmentation of internal structures in images using a multi-stage deep convolutional network that includes a reverse connected first stage. 
     SUMMARY 
     According to one innovative aspect of the present disclosure, a method for segmenting internal structures of an internal image is disclosed. In one aspect, the method can include actions of receiving, by a data processing apparatus, data representing image data that depicts internal structures of a subject, providing, by the data processing apparatus, an input data structure to a machine learning model, wherein the input data structure comprises fields structuring data that represents the received data representing the image data that depicts internal structures of the subject, wherein the machine learning model is a multi-stage deep convolutional network that has been trained to segment internal structures depicted by one or more images, receiving, by the data processing apparatus, output data generated by the machine learning model based on the machine learning model&#39;s processing of the input data structure, and processing, by the data processing apparatus, the output data to generate rendering data that, when rendered, by one or more computers, causes the one or more computers to output, for display, data that visually distinguishes between different internal structures depicted by the image data. 
     Other versions include corresponding systems, apparatus, and computer programs to perform the actions of methods defined by instructions encoded on computer readable storage devices. 
     These and other versions may optionally include one or more of the following features. For instance, in some implementations, the multi-stage deep convolutional network can include a first stage and a second stage. 
     In some implementations, the first stage of the machine learning model can be configured to: receive the input data structure and generate a first probability map that includes a probability, for each pixel of the image data, that the pixel corresponds to a particular target organ, generate an organ-attention map based on the first probability map and one or more convolutional filters, and generate a second input data structure for input to the second stage, wherein the second input data structure is generated based on the organ-attention map and the image data structured by the fields of the input data structure. 
     In some implementations, the first stage of the two-stage deep convolutional network can include multiple convolutional layers, wherein each subsequent convolutional layer includes a reverse connection to a prior convolutional layer to feed coarse-scale layer information from the subsequent convolutional layer to the prior fine-scale convolutional layer. 
     In some implementations, the second stage can be configured to: receive the generated second input data structure, and generate, based on the processing of the generated second input data structure, an adjusted probability map that includes a probability, for each pixel of the image data, that the pixel corresponds to a particular target organ. 
     In some implementations, the image that depicts internal structures of the object can include a 2D slice of an image from at least one of an axial direction, sagittal direction, or a coronal direction. 
     In some implementations, the internal structures can include multiple organs of the subject. 
     In some implementations, the internal structures can include an aorta, a duodenum, an IVC, a liver, one or more kidneys, a pancreas, a spleen, or a stomach. 
     According to another innovative aspect of the present disclosure, a run-time method for using a trained multi-stage machine learning model for performing multi-organ segmentation of an image depicting internal structures is disclosed. In one aspect, the run-time method can include actions of methods that include obtaining an internal image of an abdomen of a subject, providing the obtained internal image as an input to a first stage of machine learning model that is trained to generate a first probability map that includes, for each pixel, data indicating a likelihood that the pixel corresponds to a particular internal structure based on processing of the obtained internal image, obtaining the first probability map generated by the first stage of the machine learning model, generating input data for a second stage of a machine learning model based on (i) the obtained internal image and (ii) the obtained first probability map, obtaining the generated input data, providing the generated input data as an input to the second stage of the machine learning model that is trained to generate a second probability map that includes, for each pixel, data indicating a likelihood that the pixel corresponds to a particular internal structure based on the processing of the generated input data, obtaining the second probability map generated by the second stage of the machine learning model, and generating based on the second probability map generated by the second machine learning model, output image data, the output image data, when rendered by a user device, causes the user device to display a graphical representation of the obtained internal image that visually distinguishes between different internal structures in the internal image. 
     Other versions include corresponding systems, apparatus, and computer programs to perform the actions of methods defined by instructions encoded on computer readable storage devices. 
     According to another innovative aspect of the present disclosure, a method for training a multi-stage machine learning model to perform multi-organ segmentation of an image depicting internal structures is disclosed. In one aspect, training method can include actions of obtaining an training image of an abdomen of a subject, providing the obtained training image as an input to a first stage of machine learning model that is trained to generate a first probability map that includes, for each pixel, data indicating a likelihood that the pixel corresponds to a particular internal structure based on processing of the obtained training image, obtaining the first probability map generated by the first stage of the machine learning model, determining a difference between (i) the first probability map and (ii) a ground truth image having an internal structure label for each pixel of the ground truth image, adjusting one or more parameters of the first stage of the machine learning model based on the difference between (i) the first probability map and (ii) the ground truth image, generating input data for a second stage of a machine learning model based on (i) the obtained training image and (ii) the obtained first probability map, obtaining the generated input data, providing the generated input data as an input to the second stage of the machine learning model that is trained to generate a second probability map that includes, for each pixel, data indicating a likelihood that the pixel corresponds to a particular internal structure based on the processing of the generated input data, obtaining the second probability map generated by the second stage of the machine learning model, determining a difference between (i) the second probability map and (ii) the ground truth image having an internal structure label for each pixel of the ground truth image, and adjusting one or more parameters of the first stage of the machine learning model based on the difference between (i) the second probability map and (ii) the ground truth image. 
     Other versions include corresponding systems, apparatus, and computer programs to perform the actions of methods defined by instructions encoded on computer readable storage devices. 
     In accordance with another aspect of the present disclosure, a run-time method for using a plurality of trained multi-stage machine learning models for performing multi-organ segmentation of an image depicting internal structures is disclosed. In one aspect, the method can include obtaining a three-dimensional image depicting internal structures, generating one or more two-dimensional cross-sectional images from the obtained three-dimensional image, providing, each of the one or more two-dimensional cross-sectional images to a respective multi-stage machine learning model that has been trained to generate a second probability map based on processing of an input two-dimensional image, obtaining each second probability map generated by each of the multi-stage machine learning models, and generating a fused image based on the plurality of second probability maps obtained, wherein the fused image is an image that visually distinguishes each internal structure of the internal structures depicted by the obtained three-dimensional image. 
     Other versions include corresponding systems, apparatus, and computer programs to perform the actions of methods defined by instructions encoded on computer readable storage devices. 
     These and other aspects of the present disclosure are discussed in more detail in the detailed description below with reference to the accompanying drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block diagram of an example of a run-time system that uses a trained multi-stage machine learning model for performing multi-organ segmentation of CT images. 
         FIG. 2  is a flowchart of an example of a run-time process for using a trained multi-stage machine learning model for performing multi-organ segmentation of CT images. 
         FIG. 3  is a diagram of an example of a system for training a multi-stage machine learning model to perform multi-organ segmentation of CT images. 
         FIG. 4  is a flowchart of an example of a process for training a multi-stage machine learning model to perform multi-organ segmentation of CT images. 
         FIG. 5 . is a diagram of an example of a system that uses a plurality of trained multi-stage machine learning models to perform multi-organ segmentation of CT images. 
         FIG. 6  is a block diagram of an example of system components that can be used to implement the multi-stage learning model for performing multi-organ segmentation of CT images. 
     
    
    
     DETAILED DESCRIPTION 
     The present disclosure relates to a method, system, machine learning model, and computer program, for a multi-stage machine learning model that can be used to perform multi-organ segmentation of images such as CT images. Multi-stage machine learning model is configured to receive an internal image depicting an image of an abdomen of a living organism such as a human. The multi-stage machine learning model can process input data that includes the received internal image and generate, based on processing of the internal image, output data that annotates each pixel of the received input image with a label that indicates an organ to which the pixel likely corresponds. 
       FIG. 1  is a block diagram of an example of a run-time system  100  that uses a trained multi-stage machine learning model for performing multi-organ segmentation of CT images. For purposes of this disclosure, multi-organ segmentation of CT images can include segmentation of any internal structure depicted by a CT image and not only organs. Such internal structures can include, for example, an aorta, a duodenum, an inferior vena cava (IVC), a liver, one or more kidneys, a pancreas, a spleen, a stomach, or any other internal structure depicted by an internal image such as a CT image. The system  100  can include a first stage of the machine learning model  110 , an organ attenuation module  120 , and a second stage of the machine learning model  130 . In some implementations, the first stage of the machine learning model  110  and the second stage of the machine learning model can be part of the same machine learning model. In other implementations, the first stage of the machine learning model  110  and the second stage of the machine learning model each be part of separate machine learning models. In some implementations, the machine learning model(s) can include one or more neural networks. The first stage and the second stage of machine learning model  110 ,  130  can be trained using the system and process described by  FIGS. 3 and 4 . 
     The first stage of the machine learning  110  can be trained receive and process input data representing an internal image of an abdomen of a subject such as a human. In some implementations, the internal image  105  can include a CT image. The internal image can depict one or more internal structures of the subject. Internal structures can include, for example, an aorta, a duodenum, an inferior vena cava (IVC), a liver, one or more kidneys, a pancreas, a spleen, a stomach, or any other internal structure depicted by an internal image such as a CT image. In some implementations, the internal image  105  can include a two-dimensional slice of a three-dimensional image from one of the axial, sagittal, and coronal directions. In some implementations, the internal image  105  can be a representation of the internal image  105  such as a vector representation of the internal image  105 . 
     In some implementations, the first stage of the machine learning model  110  can include one or more deep neural networks having one or more convolutional layers. Each convolutional layer can be trained to assign a probability to one or more features of an image processed by the convolutional layer and distinguish the feature from other features of the internal image. For example, a convolutional layer can be configured to identify pixel(s) corresponding to a first internal structure such as a first organ and distinguish the identified pixel(s) from other internal structures such as other organs. In some implementations, for example, the convolutional layer can also assign a probability to a feature such as a pixel or group of pixels indicating a likelihood that the pixel or set of pixels corresponds to a particular internal structure. The example of a first stage of a machine learning model  110  shown in  FIG. 1  depicts three convolutional layers  112   a ,  112   b ,  112   c . However, the present disclosure need not be so limited and the second stage of the machine learning model can include less convolutional layers or more convolutional layers than those depicted in  FIG. 1 . For example, in some implementations, the first stage of the machine learning model  110  can include many convolutional layers that each correspond to particular pixels, regions of pixels, pixels corresponding to a particular internal structure, or other features of an internal image  105 . 
     The first stage of the machine learning model  110  can process the internal image through each of the convolutional layers  112   a ,  112   b ,  112   c  of the strained first stage  110 . In some implementations, the final layer  113  of the first stage of the machine learning model  110  can be used to receive a set of activations from the last convolutional layer  112   c  and generate a first probability map  114  based on the set of received activations from the final convolutional layer  112   c . The first probability map can include, for example, one or more numerical values, for each pixel of the internal image, that each indicate a likelihood that the pixel corresponds to a particular internal structure. In some implementations, each pixel may have up to 1 values, with  1  being the number of target internal structures. Though the example system  100  of  FIG. 1  is described as including a first stage of the machine learning model  110  that has a final layer  113  that generates the probability map  114 , the present disclosure need not be so limited. Instead, in some implementations, the layer  113  can be separate computing module that is separate from first stage of the machine learning model  110 . 
     In some implementations, the probability map output by the first machine learning model  110  can include a probability map P (1) =ƒ(I; Θ (1) )⊂R H×W×|L|  for each label at every pixel, where the probability density function f(⋅; □(1)Θ (1) ) is a segmentation network parameterized by Θ (1) . Each element p (i,l)   (1) ∈P (1)  is the probability that the ith pixel in the input internal image  105  that belongs to label l, where l=0 is the background and l=1, . . . , |L| are target internal structures. We define 
     
       
         
           
             
               
                 p 
                 
                   ( 
                   
                     i 
                     , 
                     l 
                   
                   ) 
                 
                 
                   ( 
                   1 
                   ) 
                 
               
               = 
               
                 
                   σ 
                   ⁡ 
                   
                     ( 
                     
                       a 
                       
                         ( 
                         
                           i 
                           , 
                           l 
                         
                         ) 
                       
                       
                         ( 
                         1 
                         ) 
                       
                     
                     ) 
                   
                 
                 = 
                 
                   
                     
                       exp 
                       ⁡ 
                       
                         ( 
                         
                           a 
                           
                             ( 
                             
                               i 
                               , 
                               l 
                             
                             ) 
                           
                           
                             ( 
                             1 
                             ) 
                           
                         
                         ) 
                       
                     
                     
                       
                         ∑ 
                         
                           t 
                           = 
                           0 
                         
                         
                            
                           L 
                            
                         
                       
                       ⁢ 
                       
                         exp 
                         ⁡ 
                         
                           ( 
                           
                             a 
                             
                               ( 
                               
                                 i 
                                 , 
                                 l 
                               
                               ) 
                             
                             
                               ( 
                               1 
                               ) 
                             
                           
                           ) 
                         
                       
                     
                   
                   ⁢ 
                   
                      
                     L 
                      
                   
                 
               
             
             , 
           
         
       
     
     where a (i,l)   (1)  is the activation value of the ith pixel on the lth channel dimension. Let A (1) ={a (i,l)   (1) } i=1, . . . , H×W,l=0 , . . . , |L| be the activation map. The objective function to minimize for Θ (1)  is given by 
     
       
         
           
             
               
                 
                   
                     
                       
                         J 
                         
                           ( 
                           1 
                           ) 
                         
                       
                       ⁡ 
                       
                         ( 
                         
                           Θ 
                           
                             ( 
                             1 
                             ) 
                           
                         
                         ) 
                       
                     
                     - 
                     
                       
                         1 
                         
                           H 
                           × 
                           W 
                         
                       
                       ⁡ 
                       
                         [ 
                         
                           
                             ∑ 
                             
                               i 
                               = 
                               1 
                             
                             
                               H 
                               × 
                               W 
                             
                           
                           ⁢ 
                           
                             
                               ∑ 
                               
                                 l 
                                 = 
                                 0 
                               
                               
                                  
                                 L 
                                  
                               
                             
                             ⁢ 
                             
                               1 
                               ⁢ 
                               
                                 ( 
                                 
                                   
                                     t 
                                     i 
                                   
                                   = 
                                   l 
                                 
                                 ) 
                               
                               ⁢ 
                               log 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               
                                 p 
                                 
                                   ( 
                                   
                                     i 
                                     , 
                                     l 
                                   
                                   ) 
                                 
                                 
                                   ( 
                                   1 
                                   ) 
                                 
                               
                             
                           
                         
                         ] 
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
     where l(⋅) is an indicator function. 
     The first stage  110  of multi-stage machine learning model can thus perform a preliminary a segmentation of the internal image  105 . The initial probability map  114  generated by the layer  113  can be used to generate second input data  124 , to the second stage  130 , that provides spatial attention to the internal structure locations in the internal image  105  to the second stage  130  of the multi-stage machine learning model. This enables the second stage  130  of the multi-stage machine learning model to be more discriminative for segmenting internal structures and the second stage  130  can primarily deal with local context of image data input into the second stage  130  for processing. This leads to improved results over conventional methods. This multi-stage machine learning model can be referred to as an organ-attention model. 
     In some implementations, one or more convolutional layers of the first stage of the machine learning model  110  can have a reverse connection  116 ,  118  to a previous, or lower level, convolutional layer. For example, the convolutional layer  112   c  can have a reverse connection  116  to the previous convolutional layer  112   b . By way of another example, the convolutional layer  112   b  can have a reverse connection  118  to the previous convolutional layer  112   a . These reverse connections  116 ,  118  enable learned parameters of abstract high-level semantic information to be passed back to the more detailed lower layers so that target internal structures have similar levels of details and abstract information in the same convolutional layer. Those these reverse connections  116 ,  118  are only shown in the first stage  110 . However, other convolutional layers of other stages such as the convolutional layers  132   a ,  132   b ,  132   c  may also have similar types of reverse connections. Such reverse connections are described in more detail in “Abdominal Multi-organ Segmentation with Organ-Attention Networks and Statistical Fusion,” to Alan L. Yuille, et al., published Apr. 23, 2018, which is hereby incorporated by reference in its entirety. 
     The first output data, which is the first probability map  114 , generated by the first stage  113  can be provided to an organ-attention module  120 . The organ-attention module  120  can be used to generate a second input data  124  for input to the second stage of the machine learning model. In some implementations, the organ-attention module  120  can generate second input data  124  as a function of the original internal image  105  and the first probability map. For example, in some implementations, organ-attention module  120  can generate the second input data to the second stage  130  by using an element-wise product operation to multiply the first probability map  114  by the original representation of the internal image  105  that was input into the first stage  110 . This original representation of the internal image  105  may include, for example, a vector representation of the internal image  105 . Though referred to as an organ-attention module  120 , the images processed by the organ-attention module can also depict any internal structure including, for example, an aorta, a duodenum, an inferior vena cava (IVC), a liver, one or more kidneys, a pancreas, a spleen, a stomach, or the like. 
     For purposes of this specification, the term module can include one or more software components, one or more hardware components, or any combination thereof, that can be used to realize the functionality attributed to a respective module by this specification. A software component can include, for example, one or more software instructions that, when executed, cause a computer to realize the functionality attributed to a respective module by this specification. A hardware component can include, for example, one or more processors such as a central processing unit (CPU) or graphical processing unit (CPU) that is configured to execute the software instructions to cause the one or more processors to realize the functionality attributed to a module by this specification, a memory device configured to store the software instructions, or a combination thereof. Alternatively, a hardware component can include one or more circuits such as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like, that has been configured to perform operations using hardwired logic to realize the functionality attributed to a module by this specification. 
     The second stage of the machine learning model  130  is trained to be more discriminative for segmenting internal structures such as organs. The second stage  130  is configured to receive, an input, the second input data  124  that is generated by the organ-attention module  120 . The second input data  140  is a combination of the original internal image  105  and the first probability map  114 . 
     The second stage  140  can include a deep neural network that includes a plurality of convolutional layers  132   a ,  132   b ,  132   c . In some implementations, the second stage of the machine learning model  140  can include one or more deep neural networks having one or more convolutional layers. Each convolutional layer can be trained to assign a probability to one or more features of an image processed by the convolutional layer and distinguish the feature from other features of the internal image. For example, a convolutional layer can be configured to identify pixel(s) corresponding to a first internal structure such as an aorta and distinguish the identified pixel(s) from other internal structures such as one or more organs. In some implementations, for example, the convolutional layer can also assign a probability to a feature such as a pixel or group of pixels indicating a likelihood that the pixel or set of pixels corresponds to a particular internal structure. The example of a second stage of a machine learning model  130  shown in  FIG. 1  depicts three convolutional layers  113   a ,  132   b ,  132   c . However, the present disclosure need not be so limited and the second stage of the machine learning model can include less convolutional layers or more convolutional layers than those depicted in  FIG. 1 . For example, in some implementations, the second stage of the machine learning model  130  can include many convolutional layers that each correspond to particular pixels, regions of pixels, pixels corresponding to a particular internal structure, or other features of the internal image  105 . 
     The second stage of the machine learning model  130  can process the second input data  124  through each of the convolutional layers  132   a ,  132   b ,  132   c  of the trained second stage  130 . In some implementations, the final layer  133  of the first stage of the machine learning model  130  can be used to receive a set of activations from the last convolutional layer  112   c  and generate a second probability map  140  based on the set of received activations from the final convolutional layer  132   c . The second probability map can include, for example, one or more numerical values, for each pixel of the internal image, that each indicate a likelihood that the pixel corresponds to a particular internal structure. In some implementations, each pixel may have up to l values, with l being the number of target internal structures. The second probability map  134  can be generated in the same manner as described with respect to the first probability map  114  above. However, this second stage  130  generates second probability map that is more focused on the segmentation between internal structures as a result of the second input data that is generated and provided as an input to the second stage  130 . Though the example system  100  of  FIG. 1  is described as including a second stage of the machine learning model  130  that has a final layer  133  that generates the second probability map  134 , the present disclosure need not be so limited. Instead, in some implementations, the layer  133  can be separate computing module that is separate from second stage of the machine learning model  130 . The system  100  can provide the second probability map  134  to the visualization module  140 . 
     The visualization module  140  can be configured to receive the second probability map  134  and generate an output image  144 . The output image  144  can be an image, when output on the display of a user device, displays each internal structure in the image  144  in a manner that visually distinguishes each respective internal structure from one another. In some implementations, the internal structures may be visually distinguished using different colors, different shades of colors, different fill patterns (e.g., dotted, diagonal, horizontal lines, vertical lines), or the like. The visualization module  140  can generate the visually distinguished internal structures based on the probabilities and labels in the second probability map  134 . In some implementations, the output image  144  may be rendering data that, when rendered by a user device, causes the user device to display an image of the internal structures depicted by internal image  105  such that the internal structures depicted by internal image  105  are visually distinguishable from each respective internal structure. 
       FIG. 2  is a flowchart of an example of a run-time process  200  or using a trained multi-stage machine learning model for performing multi-organ segmentation of CT images. In general, the process  200  can include obtaining an internal image of an abdomen of a subject ( 210 ), providing the obtained internal image as an input to a first stage of machine learning model that is trained to generate a first probability map that includes, for each pixel, data indicating a likelihood that the pixel corresponds to a particular internal structure based on processing of the obtained internal image ( 220 ), generating input data for a second stage of a machine learning model based on (i) the obtained internal image and (ii) the first probability map ( 230 ), providing the generated input data as an input to the second stage of the machine learning model that is trained to generate a second probability map that includes, for each pixel, data indicating a likelihood that the pixel corresponds to a particular internal structure based on the processing of the input generated at stage  230  ( 240 ), and generating based on the second probability map generated by the second machine learning model, output image data, the output image data, when rendered by a user device, causes the user device to display a graphical representation of the obtained internal image that visually distinguishes between different internal structures in the internal image  250 . 
       FIG. 3  is a diagram of an example of a system  300  for training a multi-stage machine learning model to perform multi-organ segmentation of CT images. The system  300  can include an input image  105 , a first stage  110  of a machine learning model that includes one or more convolutional layers  112   a ,  112   b ,  112   c , a module  113  for generate a first probability map  114 , an organ-attention module  120  that operates on the first probability map  114  and the input image  105 , a second stage  130  of the machine learning model that includes one or more convolutional layers  132   a ,  132   b ,  132   c , and a module  114  for generate a second probability map  133 . Elements of  FIG. 3  that share like numbers to a corresponding portion of  FIG. 1  represent the same element as shown in  FIG. 1 , but during training. 
     The system  300  can begin training a multi-stage machine learning model to perform the operations described with reference to  FIGS. 1 and 2  by obtaining an input image  105 . The obtained image can include an image that depicts internal structures in an abdomen of a subject such as human. In some implementations, the obtained image can be an actual CT image obtained from a library of training images. In other implementations, the obtained image can include a CT image that was generated by a simulator for training the multi-stage machine learning model. In yet other implementations, the image can be a different type of image shown internal structures of an abdomen such as an MRI image, an X-Ray image, an Ultrasound image, or the like. 
     The system  300  can process the obtained image  105  through each of the convolutional layers  112   a ,  112   b ,  112   c . The set of activation data output by the last convolutional layer  112   c  can be provided as an input to the module  113 . The module  113  can process the set of activation data and generate a first probability map as described above with reference to  FIG. 1 . In some implementations, the module  113  may be a final layer of the first stage  110 . In other implementations, the module  113  may be a separate stand along module that is a different computing module than the first stage  110 . 
     The system  300  can obtain the first probability map  114  generated by the module  113  and compare the first probability map to a labeled input image  108 . The labeled input image can include the image  105  that has had each pixel labeled as corresponding to a particular internal structure. The system  300  can determine the difference between the first probability map  114  generated by the module  113  and the image  108  using a loss function. Then, the system  300 , can update the parameters of one or more of the convolutional layers  112   a ,  112   b ,  112   c  of the first stage  110  based on the difference between the first probability map  114  produced by the module  113  and the labeled input image  108 , which represents the ground truth. The system  300  can iteratively update the parameters of the one or more convolutional layers  112   a ,  112   b ,  112   c  for different training inputs until the loss function is optimized. An example loss function is described later below. 
     The system  300  can continue training the multi-stage machine learning model by using the organ-attention module to generate second input data for input to the second stage of the multi-stage machine learning model  130 . In some implementations, for example, the organ-attention module  120  can generate second input data  124  as a function of the original internal image  105  and the first probability map  114 . In some implementations, for example, the organ-attention module  120  can generate the second input data to the second stage  130  by using an element-wise product operation to multiply the first probability map  114  by the original representation of the internal image  105  that was input into the first stage  110 . In this example, the result of the multiplication can serve as the input  124  to the second stage  140 . This original representation of the internal image  105  may include, for example, a vector representation of the internal image  105 . 
     The system  300  can process the generated second input data through each of the convolutional layers  132   a ,  132   b ,  132   c . The set of activation data output by the last convolutional layer  132   c  can be provided as an input to the module  133 . The module  133  can process the set of activation data and generate a second probability map as described above with reference to  FIG. 1 . In some implementations, the module  133  may be a final layer of the second stage  140 . In other implementations, the module  133  may be a separate stand along module that is a different computing module than the second stage  140 . 
     The system  300  can obtain the second probability map generated by the module  113  (not explicitly shown in  FIG. 3 ) and compare the second probability map to a labeled input image  108 . The labeled input image can include the image  105  that has had each pixel labeled as corresponding to a particular internal structure. The system  300  can determine the difference between the second probability map generated by the module  133  and the image  108  using a loss function. Then, the system  300 , can update the parameters of one or more of the convolutional layers  132   a ,  132   b ,  132   c  of the second stage  140  based on the difference between the second probability map by the module  133  and the labeled input image  108 , which represents the ground truth. The system  300  can iteratively update the parameters of the one or more convolutional layers  132   a ,  132   b ,  132   c  for different training inputs until the loss function is optimized. An example loss function is described later below. 
     In one implementation, to jointly optimize first stage  110  and stage  140 , a loss function that estimates parameters aiming at estimating parameters Om for the first probability map and Θ (2)  for the second probability map, W, and b by optimizing the function: 
         J=h   (1)   J   (1) (Θ (1) )+ h   (2)   J   (2) (Θ (2)   ,W,b ),
 
     where, W indicates convolutional filters, for example, whose dimension is (5×5 |L|), b is bias, Θ (1)  includes one or more performance parameters of the first stage of the machine learning model, → (2)  one or more performance parameters of the second stage of the machine learning model, h 1  and h 2  are fusion weights, and J is the total loss function that is the weighted average of loss functions J 1  for the first stage of the machine learning mode and J 2  for the second stage of the machine learning model. J 1  is dependent on Θ (1)  and J 2  is dependent on Θ (2) . In some implementations, using a stronger fusion weight, h 2 , for the second stage of the machine learning model than the fusion weight, h 1 , for the first stage of the machine learning model such that h 1 &lt;h 2  can result in better performance relative to implementations where h 1 ≥h 2 . In at least one implementation, values of h 1 =0.5 and h 2 =1.5 can be set and fixed at these values during training. 
       FIG. 4  is a flowchart of an example of a process  400  for training a multi-stage machine learning model to perform multi-organ segmentation of CT images. In general, the process  400  can include obtaining an training image of an abdomen of a subject ( 410 ), providing the obtained training image as an input to a first stage of machine learning model that is trained to generate a first probability map that includes, for each pixel, data indicating a likelihood that the pixel corresponds to a particular internal structure based on processing of the obtained training image ( 420 ), determining a difference between (i) the first probability map and (ii) a ground truth image having an internal structure label for each pixel of the ground truth image ( 430 ), adjusting one or more parameters of the first stage of the machine learning model based on the difference between (i) the first probability map and (ii) the ground truth image ( 440 ), generating input data for a second stage of a machine learning model based on (i) the obtained training image and (ii) the first probability map ( 450 ), providing the generated input data as an input to the second stage of the machine learning model that is trained to generate a second probability map that includes, for each pixel, data indicating a likelihood that the pixel corresponds to a particular internal structure based on the processing of the input generated at stage  450  ( 460 ), determining a difference between (i) the second probability map and (ii) the ground truth image having an internal structure label for each pixel of the ground truth image ( 470 ), and adjusting one or more parameters of the first stage of the machine learning model based on the difference between (i) the second probability map and (ii) the ground truth image ( 480 ). 
     The process  400  may be iteratively performed for multiple different training images until a loss function that measures the difference between the respective probability maps and the ground truth image is optimized. 
       FIG. 5 . is a diagram of an example of a system  500  that uses a plurality of trained multi-stage machine learning models to perform multi-organ segmentation of CT images. The system  500  can use multiple instances of the system  100  to analyze different cross sections of a CT image of a subject&#39;s abdomen. 
     The system  500  can be configured to receive input image  510  such as a three-dimensional CT image. In some implementations, the system  500  can use a computing module such as an extraction module to obtain multiple different cross-sections of the input image  510 . By way of example, the system  500  can obtain an axial image cross-section  510 A, a sagittal image cross-section  510 B, and a coronal image cross-section  510 C. Note that  FIG. 5  generally shows the process of dividing an input image  510  into multiple cross-sections  510 A,  510 B,  510 C. However, the actual image use in  FIG. 5  may not actually depict cross-section  510 A, a sagittal image cross-section  510 B, and a coronal image cross-section  510 C. 
     The sub-system  500 A is the same system as that set forth with respect to  FIG. 1, 100 . The system  500  can use the sub-system  500 A to process the axial image cross-section  510 A and generate first output data, which can be provided as an input to the visualization module  540 . The first output generated by the sub-system  500 A can include a second probability map, as described with reference to the system  100  of  FIG. 1 . The sub-system  500 A can generate the first output which includes the second probability map in the same manner as the system  100  of  FIG. 1  processes the input internal image  105  to generate a second probability map. 
     Likewise, the sub-system  500 B is the same system as that set forth with respect to  FIG. 1, 100 . The system  500  can use the sub-system  500 B to process the sagittal image cross-section  510 B and generate second output data, which can be provided as an input to the visualization module  540 . The second output generated by the sub-system  500 B can include another second probability map, as described with reference to the system  100  of  FIG. 1 . The sub-system  500 B can generate the second output which includes the other second probability map in the same manner as the system  100  of  FIG. 1  processes the input internal image  105  to generate a second probability map. 
     Similarly, the system  500  can use the sub-system  500 C to process the coronal image cross-section  510 C and generate third output data, which can be provided as an input to the visualization module  540 . The third output generated by the sub-system  500 C can include another second probability map, as described with reference to the system  100  of  FIG. 1 . The sub-system  500 C can generate the third output which includes other second probability map by processing the coronal cross-section image  510 C in the same manner as the system  100  of  FIG. 1  processes the input internal image  105  to generate a second probability map. The first output of the sub-system  500 A, the second output of sub-system  500 B, and the third output of sub-system  500 C each include a second probability map, as described with reference to  FIGS. 1 and 2 . 
     The system  500  can then use a visualization module  540  use a fusion module  520  to fuse the first output data generated by sub-system  500 A based on  500 A processing image  510 A, the second output data generated by sub-system  500 B based on  500 B processing image  510 B, and the third output data generated by sub-system  500 C based on  500 C processing image  510 C a single output image. The fusion module  520  can generate the single output image based on the output data generated by subsystems  500 A,  500 B,  500 C in a number of different ways by processing input data the fusion module  520  receives. In some implementations, for example, the fusion module  520  can execute Boolean operations such as a union, intersection, or a combination thereof, on the output data generated by the subsystems  500 A,  500 B,  500 C to generate a single probability map. Then, the single probability map can be used to generate the fused image  544  by generating rendering data, that when rendered by a user device, causes the user device to generate an image that visually distinguishes portions of an image corresponding to the image  510  using probabilities of the single probability map, internal structure labels of the single probability map, or a combination thereof, to visually differentiate between internal structures of an image that corresponds to the image  510 . 
     In other implementations, the fusion module  520  can use majority voting as a method for generating the fused image  544  from the multiple sets of output data generated by the subsystems  500 A,  500 B,  500 C. In other implementations, the fusion module  520  can use label fusion algorithms that use an expectation-maximization (EM) framework such as simultaneous truth and performance level estimation (STAPLE). In other implementations, the system  500  can generate local similarity maps associated with each cross-sectional image, the output data generated by a sub-system for each respective cross-sectional image, or a combination thereof, and generate the fused image  520  using the generated local similarity maps. In other implementations, the fusion module  520  can generate a fused image  544  from the output data generated by the subsystems  500 A,  500 B,  500 C using an image fusion process described by “Abdominal Multi-organ Segmentation with Organ-Attention Networks and Statistical Fusion,” to Alan L. Yuille, et al., published Apr. 23, 2018, which is hereby incorporated by reference in its entirety. 
       FIG. 6  is a block diagram of an example of system components that can be used to implement the multi-stage learning model for performing multi-organ segmentation of CT images. 
     Computing device  600  is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device  650  is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, and other similar computing devices. Additionally, computing device  600  or  650  can include Universal Serial Bus (USB) flash drives. The USB flash drives can store operating systems and other applications. The USB flash drives can include input/output components, such as a wireless transmitter or USB connector that can be inserted into a USB port of another computing device. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the inventions described and/or claimed in this document. 
     Computing device  600  includes a processor  602 , memory  604 , a storage device  608 , a high-speed interface  608  connecting to memory  604  and high-speed expansion ports  610 , and a low speed interface  612  connecting to low speed bus  614  and storage device  608 . Each of the components  602 ,  604 ,  608 ,  608 ,  610 , and  612 , are interconnected using various busses, and can be mounted on a common motherboard or in other manners as appropriate. The processor  602  can process instructions for execution within the computing device  600 , including instructions stored in the memory  604  or on the storage device  608  to display graphical information for a GUI on an external input/output device, such as display  616  coupled to high speed interface  608 . In other implementations, multiple processors and/or multiple buses can be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices  600  can be connected, with each device providing portions of the necessary operations, e.g., as a server bank, a group of blade servers, or a multi-processor system. 
     The memory  604  stores information within the computing device  600 . In one implementation, the memory  604  is a volatile memory unit or units. In another implementation, the memory  604  is a non-volatile memory unit or units. The memory  604  can also be another form of computer-readable medium, such as a magnetic or optical disk. 
     The storage device  608  is capable of providing mass storage for the computing device  600 . In one implementation, the storage device  608  can be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configurations. A computer program product can be tangibly embodied in an information carrier. The computer program product can also contain instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory  604 , the storage device  608 , or memory on processor  602 . 
     The high-speed controller  608  manages bandwidth-intensive operations for the computing device  600 , while the low speed controller  612  manages lower bandwidth intensive operations. Such allocation of functions is only an example. In one implementation, the high-speed controller  608  is coupled to memory  604 , display  616 , e.g., through a graphics processor or accelerator, and to high-speed expansion ports  610 , which can accept various expansion cards (not shown). In the implementation, low-speed controller  612  is coupled to storage device  608  and low-speed expansion port  614 . The low-speed expansion port, which can include various communication ports, e.g., USB, Bluetooth, Ethernet, wireless Ethernet can be coupled to one or more input/output devices, such as a keyboard, a pointing device, microphone/speaker pair, a scanner, or a networking device such as a switch or router, e.g., through a network adapter. The computing device  600  can be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a standard server  620 , or multiple times in a group of such servers. It can also be implemented as part of a rack server system  624 . In addition, it can be implemented in a personal computer such as a laptop computer  622 . Alternatively, components from computing device  600  can be combined with other components in a mobile device (not shown), such as device  650 . Each of such devices can contain one or more of computing device  600 ,  650 , and an entire system can be made up of multiple computing devices  600 ,  650  communicating with each other. 
     The computing device  600  can be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a standard server  620 , or multiple times in a group of such servers. It can also be implemented as part of a rack server system  624 . In addition, it can be implemented in a personal computer such as a laptop computer  622 . Alternatively, components from computing device  600  can be combined with other components in a mobile device (not shown), such as device  650 . Each of such devices can contain one or more of computing device  600 ,  650 , and an entire system can be made up of multiple computing devices  600 ,  650  communicating with each other. 
     Computing device  650  includes a processor  652 , memory  664 , and an input/output device such as a display  654 , a communication interface  666 , and a transceiver  668 , among other components. The device  650  can also be provided with a storage device, such as a micro-drive or other device, to provide additional storage. Each of the components  650 ,  652 ,  664 ,  654 ,  666 , and  668 , are interconnected using various buses, and several of the components can be mounted on a common motherboard or in other manners as appropriate. 
     The processor  652  can execute instructions within the computing device  650 , including instructions stored in the memory  664 . The processor can be implemented as a chipset of chips that include separate and multiple analog and digital processors. Additionally, the processor can be implemented using any of a number of architectures. For example, the processor  610  can be a CISC (Complex Instruction Set Computers) processor, a RISC (Reduced Instruction Set Computer) processor, or a MISC (Minimal Instruction Set Computer) processor. The processor can provide, for example, for coordination of the other components of the device  650 , such as control of user interfaces, applications run by device  650 , and wireless communication by device  650 . 
     Processor  652  can communicate with a user through control interface  658  and display interface  656  coupled to a display  654 . The display  654  can be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface  656  can comprise appropriate circuitry for driving the display  654  to present graphical and other information to a user. The control interface  658  can receive commands from a user and convert them for submission to the processor  652 . In addition, an external interface  662  can be provided in communication with processor  652 , so as to enable near area communication of device  650  with other devices. External interface  662  can provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces can also be used. 
     The memory  664  stores information within the computing device  650 . The memory  664  can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Expansion memory  674  can also be provided and connected to device  650  through expansion interface  672 , which can include, for example, a SIMM (Single In Line Memory Module) card interface. Such expansion memory  674  can provide extra storage space for device  650 , or can also store applications or other information for device  650 . Specifically, expansion memory  674  can include instructions to carry out or supplement the processes described above, and can also include secure information. Thus, for example, expansion memory  674  can be provided as a security module for device  650 , and can be programmed with instructions that permit secure use of device  650 . In addition, secure applications can be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner. 
     The memory can include, for example, flash memory and/or NVRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory  664 , expansion memory  674 , or memory on processor  652  that can be received, for example, over transceiver  668  or external interface  662 . 
     Device  650  can communicate wirelessly through communication interface  666 , which can include digital signal processing circuitry where necessary. Communication interface  666  can provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication can occur, for example, through radio-frequency transceiver  668 . In addition, short-range communication can occur, such as using a Bluetooth, Wi-Fi, or other such transceiver (not shown). In addition, GPS (Global Positioning System) receiver module  670  can provide additional navigation- and location-related wireless data to device  650 , which can be used as appropriate by applications running on device  650 . 
     Device  650  can also communicate audibly using audio codec  660 , which can receive spoken information from a user and convert it to usable digital information. Audio codec  660  can likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device  650 . Such sound can include sound from voice telephone calls, can include recorded sound, e.g., voice messages, music files, etc. and can also include sound generated by applications operating on device  650 . 
     The computing device  650  can be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a cellular telephone  680 . It can also be implemented as part of a smartphone  682 , personal digital assistant, or other similar mobile device. 
     Various implementations of the systems and methods described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations of such implementations. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device. 
     These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” “computer-readable medium” refers to any computer program product, apparatus and/or device, e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs), used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor. 
     To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. 
     The systems and techniques described here can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), and the Internet. 
     The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. 
     Other Embodiments 
     A number of embodiments have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the invention. In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps can be provided, or steps can be eliminated, from the described flows, and other components can be added to, or removed from, the described systems. Accordingly, other embodiments are within the scope of the following claims.