Progressively-trained scale-invariant and boundary-aware deep neural network for the automatic 3D segmentation of lung lesions

A system and method are disclosed for segmenting a set of two-dimensional CT slices corresponding to a lesion. In an embodiment, for each of at least a subset of the set of CT slices, the system inputs the CT slice into a plurality of branches of a trained segmentation block. Each branch of the segmentation block includes a convolutional neural network (CNN) with filters at a different scale, and produces one or more levels of output. The system generates, for each CT slice in the subset, feature maps for each level of output. The system generates a segmentation of each CT slice in the subset based on the feature maps of each level of output. The system aggregates the segmentations of each slice in the subset to generate a three-dimensional segmentation of the lesion. The system transmits data representing the three-dimensional segmentation to a user interface for display.

TECHNICAL FIELD

The disclosure generally relates to the field of image processing, and in particular to the automatic segmentation of Computed Tomography (CT) images.

BACKGROUND

In various medical fields, computed tomography (CT) and magnetic resonance (MR) images are used in clinical care and clinical trials to assess the effects of treatment on lesion size over time. Currently, lesion size is often characterized using a single diameter measurement. Current methods of obtaining a single diameter measurement often involve medical professionals manually delineating the boundary of a lesion on a single slice where the lesion looks largest. However, this method is a time-consuming, resource-intensive, and labor-intensive task, which prohibits this method from being applied to three-dimensional images. Further, because current automated three-dimensional segmentation methods are often highly sensitive to initial conditions and model parameters, they are unable to adapt to lesions of varying sizes and shapes.

SUMMARY

Embodiments relate to a model for the automatic segmentation of computed tomography (CT) images. The model is progressively trained such that it can more accurately and consistently generate three-dimensional segmentations of lesions of varying sizes and shapes than is possible using conventional approaches.

In various embodiments, a computer-implemented method of segmenting a set of two-dimensional CT slices corresponding to a lesion includes, for each of at least a subset of the set of CT slices, inputting the CT slice into a plurality of branches of a trained segmentation block. Each branch of the segmentation block includes a convolutional neural network (CNN) with convolutional filters at a different scale, and each branch produces one or more levels of output. In one embodiment, the segmentation block includes three branches, and each branch has three levels of output. The first branch may be a primary branch and each additional branch may adopt convolutional filters based on the filters of the primary branch. The method also includes, for each of the CT slices in the subset, generating feature maps for each of the levels of output. The feature maps are generated by combining same-level outputs across each of the branches. The same-level outputs may be combined using element-wise max-pooling.

The method further includes, for each of the CT slices in the subset, generating a segmentation of the CT slice based on the feature maps. The method further includes aggregating the segmentations of each slice in the subset of CT slices to generate a three-dimensional segmentation of the lesion. The method also includes transmitting data representing the three-dimensional segmentation of the lesion to a user interface for display of the three-dimensional segmentation.

The method may further include training the segmentation block by accessing a set of two-dimensional CT slices of lesions, the set including a first segment of CT slices and a plurality of sequential segments of CT slices. Each CT slice in the first segment represents a first layer of a lesion and includes a single diameter measurement. Each slice in a sequential segment represents a sequential layer of a corresponding lesion. The method of training the segmentation block further includes initializing weights of the segmentation block with initial values and applying the segmentation block to the first segment of CT slices to generate a prediction of segmentation for each CT slice in the first segment. The method of training the segmentation block further includes updating the weights of the segmentation block based on the prediction and the single diameter measurement of each CT slice in the first segment. The method of training the segmentation block further includes sequentially updating the weights of the segmentation block based on each additional segment of the plurality of sequential segments. The weights are sequentially updated by applying the segmentation block to the additional segment to generate a prediction of a segmentation for each CT slice in the additional segment, refining the prediction of the segmentations to produce a refined segmentation of each CT slice in the additional segment, updating the weights of the segmentation block based on the refined segmentations and the CT slices in the additional segment, and adding the CT slices in the additional segment to a sequential segment. The method of training the segmentation block may further include refining the prediction of the segmentations by applying the predictions and each CT slice in the additional segment to a fully-connected conditional random field.

Each two-dimensional CT slice may include an object of interest, such that, during training, a top level of the one or more levels of output is supervised by a segmentation mask of the object and a bottom level of the one or more levels of output is supervised by a boundary mask of the object. For example, same level outputs across each of the branches are pooled and convolved to generate a top level of output, which may be a probability map. The top level of output is supervised with the segmentation mask of the object. Similarly, the same level outputs across each of the branches may be pooled and convolved to generate a bottom level of output. The bottom level of output is supervised by the boundary mask of the object.

In other embodiments, the segmentation system is stored on a non-transitory machine-readable storage medium. The system is manufactured by a process including, for each of at least a subset of a set of CT slices of a lesion, inputting the slice into a plurality of branches of a trained segmentation block, each branch including a CNN with convolution filters at a different scale, and each branch produces one or more levels of output. The process further includes generating features maps for each of the levels of output based on a concatenation of same-level outputs across each of the branches, and generating a segmentation of the CT slice based on the feature maps of each of the levels of output. The process further includes aggregating the segmentations of each slice in the subset to generate a three-dimensional segmentation of the lesion, and transmitting data representing the three-dimensional segmentation of the lesion to a user interface for display of the three-dimensional segmentation.

In further embodiments, the image segmentation system includes a hardware processor and a non-transitory computer-readable medium that contains instructions that cause the hardware processor to, for at least a subset of a set of CT slices of a lesion, input the slice input a plurality of branches of trained segmentation block, where each branch includes a CNN with convolutional filters at a different scale and produces one or more levels of output; generate feature maps for each level of output based on a concatenation of same-level outputs across each of the branches; and generate a segmentation of the slice based on the feature maps. The instruction further causes the hardware processor to aggregate segmentations of each slice in the subset of CT slices to generate a three-dimensional segmentation of the lesion, and transmit data representing the three-dimensional segmentation of the lesion to a user interface for display of the three-dimensional segmentation.

DETAILED DESCRIPTION

The Figures (FIGS.) and the following description describe certain embodiments by way of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods may be employed without departing from the principles described. Reference will now be made to several embodiments, examples of which are illustrated in the accompanying figures. It is noted that wherever practicable similar or like reference numbers are used in the figures to indicate similar or like functionality.

Overview of Method

FIG. 1illustrates one embodiment of a system environment100suitable for providing image segmentation. In the embodiment shown, the environment100includes a user device105, a network110, a storage module115, and an image segmentation system125. In other embodiments, the system environment100includes different and/or additional elements. In addition, the functions may be distributed among the elements in a different manner than described.

The user device105is one or more computing devices capable of receiving user input as well as transmitting and/or receiving data via a network110. In one embodiment, a user device105is a computer system, such as a desktop or a laptop computer. Alternatively, a user device105may be a device having computer functionality, such as a personal digital assistant (PDA), a mobile telephone, a smartphone, or another suitable device. A user device105is configured to communicate via the network110. The user device105may execute an application allowing a user of the user device105to interact with the image segmentation system125via a user interface. For example, a web browser application may enable interaction between the user device105and the image segmentation system125via the network110or a graphical user interface may be provided as part of a software application published by the image segmentation system125and installed on the user device105. Alternatively, a user device105interacts with the image segmentation system125through an application programming interface (API) running on a native operating system of the user device105, such as IOS® or ANDROID™.

The storage module115is one or more machine-readable media that store images120. The storage module115may receive and store images120from a user device105, the image segmentation system125, third-party databases, and the like. In some embodiments, images are computed tomography (CT) images taken of one or more subjects. Subjects may include lesions, such as lung lesions, liver lesions, mediastinal lesions, subcutaneous lesions, and abdominal lesions. Subjects may also include enlarged lymph nodes and the like. Images taken consecutively may have spatial dependence. For example, consecutive images may correspond to a set of CT images of a single subject (e.g., CT slices captured along an axis of a lesion). Images may correspond to “center slices” and/or “edge slices.” Center slices are CT images that have been taken where the lesion looks largest along a longitudinal axis of the lesion. Edge slices are additional CT images of lesions taken at additional locations along the longitudinal axis of the lesions. The storage module115may store images used for training, validation, and testing of the image segmentation system125. A portion of these images may include manual delineations, such as center slices that have been manually delineated to include a single diameter measurement. An additional portion of these images may not include manual delineations, such as edge slices, that are segmented and refined during training. In one embodiment, a module with similar or identical functionality to the storage module115is integrated into the image segmentation system125.

The image segmentation system125segments CT images to generate three-dimensional segmentations from two-dimensional slices. In some embodiments, the image segmentation system125uses a progressively trained scale-invariant and boundary-aware deep convolutional network to automatically segment three-dimensional volumes of subjects, such as lesions. Through progressive training, the image segmentation system125can effectively cope with scale-, size-, and shape-variability across lesions. In some embodiments, the deep convolutional network is progressively trained by 1) iteratively generating two-dimensional segmentations from center slices to edge slices and 2) updating network weights after each iteration based on the generated segmentations. Various embodiments of the image segmentation system125are described in greater detail below, with reference toFIGS. 2 through 4.

The user device105, storage module115, and image segmentation system125are configured to communicate via a network110, which may include any combination of local area and/or wide area networks, using both wired and/or wireless communication systems. In one embodiment, a network110uses standard communications technologies and/or protocols. For example, a network110includes communication links using technologies such as Ethernet, 802.11, worldwide interoperability for microwave access (WiMAX), 3G, 4G, code division multiple access (CDMA), digital subscriber line (DSL), etc. Examples of networking protocols used for communicating via the network110include multiprotocol label switching (MPLS), transmission control protocol/Internet protocol (TCP/IP), hypertext transport protocol (HTTP), simple mail transfer protocol (SMTP), and file transfer protocol (FTP). Data exchanged over a network110may be represented using any suitable format, such as hypertext markup language (HTML) or extensible markup language (XML). In some embodiments, all or some of the communication links of a network110may be encrypted using any suitable technique or techniques.

FIG. 2shows one embodiment of the image segmentation system125ofFIG. 1. In the embodiment shown, the image segmentation system125includes an image store205, a feature store210, a segmentation block215, a segmentation refinement block220, and a user interface225. In other embodiments, the image segmentation system125includes different and/or additional elements. In addition, the functions may be distributed among the elements in a different manner than described.

The image segmentation system125maintains images and image data in the image store205. Images maintained by the image store205may include local copies of some or all of the images120stored in the storage module115. In one embodiment, the image store205stores the original, preprocessed, and segmented images. The image store205may also store images for training, validation, and testing of the image segmentation system125. Original images include one or more CT slices of lesions that have been captured along an axis of the lesion. The number of slices captured and stored for a single lesion and slice thickness may vary. For example, the image store205may include CT slices with a thickness of 1 mm or less, 5 mm or less, 10 mm or less, 15 mm or less, and the like. In some embodiments, images may be preprocessed before segmentation. Preprocessing may include cropping a region of interest on the image to a bounding box. The edges of the bounding box may correspond to a dimension of the subject. For example, each edge of the bounding box may be a multiple of the lesion's longest diameter (e.g., one, two, three, five times the longest diameter). Preprocessing may also include modifying the intensity of each image. For example, the intensity may be converted into Hounsfield units using various linear transformation parameters (e.g., linear transform parameters recorded in the DICOM header). The dynamic range of the intensity may then be shifted and/or cut-off at zero such that all intensity values are non-negative. Further, the intensity may be normalized such that the intensity values ranges from zero to unity. In this way, the relative intensity changes between voxels are preserved.

Training images may include CT images of lesions in which a single diameter measurement is used to represent each lesion. For example, training images may include images that have been assessed according to Response Evaluation Criteria in Solid Tumors (RECIST). These images may correspond to center slices of lesions that have been scanned longitudinally and have been manually delineated. Training images may also include additional CT images of lesions taken at additional locations along the longitudinal axis of the lesions (“edge slices”). Edge slices used for training may not include manual delineations. Instead, unlabeled edge slices are used to improve model performance and are refined during training using one or more refinement techniques, such as conditional random fields. A subset of the training images may be used for validation. These images include center slices and edge slices that have been manually delineated in a slice-by-slice manner to form a dataset of three-dimensional delineations for validation. Training images may be stored in groups with various configurations. For example, images may be grouped by subject, region of interest, relative position along an axis of a subject (e.g., CT images of transaxial planes along a longitudinal axis), and the like. In some embodiments, users may manually delineate training images using a user interface. This may be a part of the user interface of the image segmentation system125, or it may be a separate process performed on a system other than the image segmentation system125.

The image store205may also store metadata for the images. Metadata may include date and time of image capture, preprocessing, or segmentation; author; device identification; imager settings; subject identification label; and the like. Metadata may also include measurement data, such as two-dimensional segmentation dimensions, three-dimensional segmentation dimensions, location of slice along subject axis, subject type (e.g., lesion and/or tumor type), and the like.

The image segmentation system125maintains the results of image segmentations in the feature store210. Results may include two-dimensional segmentations of individual CT slices and aggregated three-dimensional segmentations of lesions from sets of CT slices. Results of segmentations may be in the form of probability maps and/or feature maps. Results may also include the location of image subjects (e.g., location coordinates of a lesion within a slice), lesion features, such as dimensions, volume, size progression, and the like. Results may also include evaluation metrics, such as the Dice similarity coefficient (DSC), volumetric similarity (VS), Hausdorff Distance (HD), etc., that are used to assess segmentation performance of the segmentation block215and/or segmentation refinement block220. Results may be displayed to users on a user interface225of the image segmentation system125.

The feature store210may also include parameters and features of the models generated by the segmentation block215and/or segmentation refinement block220. Examples may include optimizer type, learning rate, batch size, preprocessing techniques, kernel size, filter scales, number of scales, number of branches, block architecture, loss functions, and the like. Further, model parameters and features may include model parameters and features of baseline models used by the image segmentation system125to evaluate the performance of the segmentation block215and/or segmentation refinement block220. Baseline models may include mainstream convolution neural networks (CNN), such as classical CNNs used for biomedical image segmentation, alternative networks, such as holistically-nested networks (HNN) for edge detection and multi-scale/multi-level learning, and/or weakly supervised self-paced segmentation (WSSS) methods. In some embodiments, models maintained by the feature store210are subject-specific. For example, the feature store210may maintain separate tissue-specific models for segmenting lung, soft tissue, and liver lesions.

The segmentation block215is a progressively-trained scale-invariant and boundary-aware convolutional neural network for performing volumetric lesion segmentations from two-dimensional CT slices. In some embodiments, the segmentation block215includes two or more branches of CNN (e.g., two, three, four, five, or ten branches, etc.). Each branch of the segmentation block215includes filters at a different scale. Output from each of the branches are combined to generate scale-invariant responses. Filter weights may be regularized and tied to a primary branch such that the number of trainable parameters remain the same as the architecture of a CNN with a single branch. For example, a primary branch may adopt 3×3 filters in some or all of the convolutional layers, and the additional branches may adopt convolutional filters scaled up from the filters on the primary branch, such as 5×5 filters and 7×7 filters.

In some embodiments, each branch of the segmentation block produces outputs at multiple levels. In these embodiments, each branch of the segmentation block215includes two or more blocks that each produce a different level of output. Outputs from the different branches at the same level are combined (e.g., using element-wise max-pooling). An example architecture of a single branch of the segmentation block215is discussed in detail below with reference toFIG. 4.

The segmentation refinement block220refines segmentations generated by the segmentation block215during training of the segmentation block215. Refined segmentations may include refined two-dimensional segmentations that are used by the image segmentation system125to progressively train the segmentation block215. In these embodiments, two-dimensional segmentations are refined after each training iteration. Refined segmentations and their corresponding images are then added to the training dataset to update the segmentation block215in the next training iteration. Refined two-dimensional segmentations may be generated by applying a current segmentation prediction and corresponding image as input into a fully-connected conditional random field. This results in refined segmentations with tight boundaries.

Refined segmentations may also include refined three-dimensional segmentations that have been processed to eliminate potential irregularities between slices in the axial direction of a lesion. Various criteria may be used to determine if a two-dimensional segmentation is valid, and may be based on the segmentation mask, mask's centroid, area ratio, and the like. For example, two-dimensional slices may be considered valid if the segmentation mask or the mask's centroid lies inside the previous slices segmentation mask; the area ratio between the new and previous two-dimensional segmentations are within a predetermined range (e.g., within a range of [0.7, 1.3]), and the like. In some embodiments, when a two-dimensional slice is not considered valid, the previous two-dimensional segmentation may be copied onto the new image slice as an initialization for the refinement to obtain a new segmentation.

Users may provide images to the image segmentation system125via the user interface225. For example, the user may be prompted by an element of a user interface to upload images to the image segmentation system125. The user interface225may provide additional elements to the user to add grouping information to images. For example, the user interface225may include elements that allow the user to indicate which group an image is a member of, the number of slices in a group, location of slice relative to an axis of the subject, subject information (e.g., material captured and subject identification number), and the like. Users may also manually delineate images using interactive elements of the user interface225to produce additional training/validation data. The user interface225may also include additional elements that allow the user to select model parameters and/or training schema.

FIG. 3is a block diagram illustrating an architecture of a segmentation block215of the image segmentation system125, according to one embodiment. The segmentation block215is a progressively-trained scale-invariant and boundary-aware convolutional neural network for performing volumetric lesion segmentations from a set of two-dimensional CT slices305of a lesion. To produce a three-dimensional segmentation of the lesion from the set of two-dimensional CT slices305, each CT slice is processed through the segmentation block215to produce a two-dimensional segmentation that captures the patterns of the lesion in the slice. The segmentations of each slice in the set305are aggregated to produce a volumetric prediction of the lesion. Data representing the volumetric prediction may be transmitted for display on a user interface of the image segmentation system125. Further, features of the volumetric prediction and/or two-dimensional segmentations may be extracted. Features may be used to classify the lesion, analyze lesion progression, assess treatment efficacy, and the like.

The segmentation block215may include multiple branches of CNN. InFIG. 3, the segmentation block215includes three branches of CNN, and each branch of CNN is represented by a row. Other embodiments may include a segmentation block215with greater or fewer branches of CNN, branches with different architectures, and the like. A first branch310of the segmentation block215may be a primary branch, and a second branch315and third branch320of the segmentation block215may be secondary branches. Each branch of the segmentation block215includes convolutional layers that have adopted filters at different scales such that the segmentation block215can robustly respond to patterns of lesions at various scales. In some embodiments, the secondary branches adopt filters scales based on the filters of the primary branch. For example, if the convolutional layers in the first branch310adopt 3×3 filters, the convolutional layers in the second branch315may adopt 5×5 filters, and the convolutional layers in the third branch320may adopt 7×7 filters.

Each branch of the segmentation block215includes multiple blocks that each produce a different level of output. In the embodiment shown inFIG. 3, each branch includes three blocks, and each block produces output at a different level. An example architecture of each branch is discussed in detail below with reference toFIG. 4. Same-level outputs from each of the branches are combined to generate scale-invariant responses. In some embodiments, the same-level outputs from the branches are combined using element-wise max-pooling, as shown inFIG. 3. The outputs across the branches are combined to produce a final prediction325for each CT slice. In this way, the patterns of a single slice are captured by one or more branches of the segmentation block215, which brings robustness to the segmentation of lesion images at different scales. The final prediction325of each slice may take the form of a probability map. The final predictions for each slice in set of CT slices of a lesion are aggregated (e.g., stacked) to produce a volumetric prediction of the lesion. In some embodiments, to produce a volumetric prediction of the lesion, the predicted lesion masks of each slice in the set of CT slices of the lesion are placed adjacent to each other in three-dimensional space.

In some embodiments, volumetric predictions produced during training are further processed to eliminate potential irregularities between slices in the axial direction as a product of the slice-by-slice prediction mechanism. Post-processing may include analyzing the validity of each new two-dimensional segmentation along the axial direction of the lesion. Criteria may be based on the segmentation mask, the centroid of the mask, and the area ratio between successive slices. For example, a slice may be considered valid if 1) the segmentation mask or the mask's centroid lies inside the previous slice's segmentation mask and/or 2) the area ratio between the new and previous two-dimensional segmentations is within a predetermined range (e.g., [0.5, 1.5]). In some embodiments, when a two-dimensional segmentation is found to be invalid, the previous two-dimensional segmentation may be applied to the new image slice as an initialization for the CRF-based refinement to obtain the new segmentation on the new image slice.

In some embodiments, the segmentation block215is jointly optimized and deeply supervised by the lesions' segmentation masks and boundaries derived from the masks. In these embodiments, the joint loss function is defined by Equation 1.

In Equation 1, {circumflex over (M)}nis a regional prediction, Mnis a regional ground-truth, {circumflex over (B)}nis a boundary prediction, Bnis a boundary ground-truth. {circumflex over (L)} is a final prediction. L is a final ground truth. The regional prediction is the predicted region of the image that includes the lesion. The regional ground truth is the actual region of the image that includes the lesion, as determined by a sufficiently reliable method, such as manual labeling. The boundary prediction is the predicted boundary of the lesion in the image. The boundary ground truth is the actual boundary of the lesion in the image, as determined by a sufficiently reliable method, such as manual labeling. ωm, ωb, and ωfare weights determining a respective contribution of a respective type of loss in the joint loss function. In particular, ωmis the weight representing the contribution of the mask loss in the joint loss function, ωbis the weight representing the contribution of the boundary mask in the joint loss function, and ωfis the weight representing the contribution of the final segmentation prediction in the joint loss function. l1is a first level of output. l2is a second level of output. l3is a third level of output. lcomis a combination of the first level of output, the second level of output, and the third level of output from the deep supervisions.

The regional prediction, {circumflex over (M)}n, is generated from the outputs at the top (e.g., output330, output335, and output340), which are probability maps that have been deeply supervised by the segmentation masks of the lesion (e.g., mask345), and their combination360. In some embodiments, output330is generated from the first level of output, l1, of one or more of the branches of the segmentation block215, and is deeply supervised by the segmentation mask345of the lesion. Similarly, output335is generated from the second level of output, l2, of one or more of the branches of the segmentation block215and is supervised by the segmentation mask350of the lesion. Further, output340is generated from the third level of output, l3, of one or more of the branches of the segmentation block215and is supervised by the segmentation mask355of the lesion.

The boundary prediction, {circumflex over (B)}n, is generated from the outputs at the bottom (e.g., output365, output370, and output375), which are probability maps that have been deeply supervised by the boundaries derived from the segmentation masks of the lesions, and their combination380. For example, output365is generated from the first level of output, l1, of one or more of the branches of the segmentation block215, and is supervised by the boundary385of the lesion. Similarly, output370is generated from the second level of output, l2, of one or more of the branches of the segmentation block215, and is supervised by the boundary390of the lesion, and output375is generated from the third level of output, l3, of one or more of the branches of the segmentation block215, and is supervised by the boundary395of the lesion.

FIG. 4is a block diagram illustrating an architecture of a branch of the segmentation block215, according to one embodiment. The branch400includes three blocks that each produce output at a different level. In some embodiments, each branch may include a greater or smaller number of blocks and produce output accordingly. Block 1410produces level 1 output415, block 2420produces level 2 output, and block 3430produces level 3 output435. To produce the level 1 output415, level 2 output425, and level 3 output, the output from the corresponding blocks are up-sampled and/or convolved so that element-wise max-pooling may be performed in subsequent steps of the segmentation block215. Further, block outputs may be further pooled before being passed to a subsequent block. For example, the output from block 1410is max-pooled before being passed to block 2420. Similarly, the output from block 2 is max-pooled445before being passed to block 3430. In some embodiments, each block includes VGG16 architecture. Other embodiments of segmentation block215branches may include architectures different and/or additional layers than those shown inFIG. 4. For example, any suitable stackable CNN that introduces resolution change to generate feature maps through pooling mechanisms and/or changing convolution stride size may be used.

FIG. 5is a diagram illustrating a method500of training a segmentation block215, according to one embodiment. The segmentation block215is progressively trained using a self-paced algorithm. During training, training images are progressively added to the training dataset such that the segmentation block215can cope with scale-variability as the segmentation block215progresses from center slices to edge slices and/or is applied to segment lesions with a large range of size variability.

Training images include CT images of lesions. Each lesion is associated with a center slice, which is the slice corresponding to where the lesion looks largest along an axis of the lesion. Each lesion is also associated with one or more edge slices, representing sequential slices taken along the axis of the lesion. For example, the object volume505is associated with five slices: a center slice510, and four edge slices taken along an axis of the lesion in superior and inferior directions, namely slice515, slice520, slice525, and slice530. The number and thickness of lesion slices may vary based on a set of predetermined preferences chosen by the image segmentation system125and/or a user of the image segmentation system125. Further, the number of slices of each lesion used during training may be based on a set of predetermined rules and/or preferences selected by the image segmentation system125and/or user of the image segmentation system125. For example, lesions may be assumed to have a spheroidal or ellipsoidal shape in three-dimensions. Under this assumption, the range for evaluating successive CT slices on the axial direction may be approximated based on the diameter of the center slice, d. For example, the axial range may be defined for the evaluation to be [−0.8 d, 0.8d] from the center slice on the axial direction. In this example, CT slices falling within this range are included and evaluated in the volumetric lesion segmentation task.

A portion of training images may include slices that have been manually delineated, such as center slices that have been delineated in a pixel-wise manner. Center slices are associated with a single diameter measurement that represents the largest diameter of the lesion along an axial direction of the lesion. A portion of training images may also include slices that have not been manually delineated, such as edge slices. Model performance is improved by progressively training the segmentation block on unlabeled images. Images that have not manually delineated are refined during training using one or more refinement techniques, such as conditional random fields. Further, training images used for validation may include a set of center and edge slices that have been manually delineated to form a dataset of three-dimensional delineations.

In the method500shown, the training dataset initially contains training images of center slices. As the segmentation block215is progressively trained, additional training images are added to the training dataset. In some embodiments, neighboring edge slices are adding during each iteration of training. For example, in a first iteration, the segmentation block215may be trained with center slices only, such as the center slice510of the lesion volume505. In a second iteration, the segmentation block215may be trained with center slices and neighboring slices, such as slice515and slice520. In a third iteration, additional neighboring slices such as slice525and slice530, are added to the training dataset. The training dataset may be continuously augmented with successive slices until the training dataset includes a threshold number the training images.

As shown inFIG. 5, the segmentation block535is initially trained with a set of center slices540. Training of the segmentation block535is supervised using the two-dimensional delineations of the center slices545. After the segmentation block535converges, the converged segmentation block555is applied to a set of successive slices, such as edge slices550, to predict two-dimensional segmentations560of the set of successive slices. The segmentations560are refined with a tight boundary by applying a conditional random field to the segmentations560and the edge slices550. The refined segmentations and the corresponding images are added565to the training dataset to update the segmentation block555in the next iteration. For example, the refined predictions and corresponding images are added to the training dataset, which is then used to train the segmentation block555until it converges, producing an updated segmentation block570. The updated segmentation block570is then applied to an additional set of successive slices, e.g., slices575, to predict two-dimensional segmentations580of the additional set of successive slices. The predictions are refined with a tight boundary, and the refined predictions and corresponding images are added585to the training dataset. The training dataset is then used to update the segmentation block570. This process may be repeated until a threshold portion of the training images have been added to the training dataset and the segmentation block converges. To test the converged segmentation block, the converged segmentation block is applied to a testing dataset to produce segmentations for the CT slices in the testing dataset.

FIG. 6is a flowchart of a method600for segmenting images, according to one embodiment. In the method600shown, a set of two dimensional CT slices corresponding to a lesion are segmented to generate a three-dimensional segmentation of the lesion. To generate the three-dimensional segmentation of the lesion, each slice for at least a subset of the set of CT slices is input605into a plurality of branches of a trained segmentation block215. Each branch includes a CNN with convolution filters at a different scale, and each branch produces one or more levels of output. Feature maps are generated610for each of the levels of output based on a combination of same-level outputs across each of the branches. A two-dimensional segmentation of the slice is generated615based on the feature maps of each of the levels of output. The two-dimensional segmentations of each slice in the subset of CT slices are aggregated620to generate a three-dimensional segmentation of the lesion. Data representing the three-dimensional segmentation of the lesion is transmitted625to a user interface for display of the three-dimensional segmentation.

FIG. 7is a flowchart of a method700of training the segmentation block215, according to one embodiment. In the method700shown, a set of two-dimensional CT slices of lesions is accessed705. The set includes a first segment of CT slices and a plurality of sequential segments of CT slices. Each CT slice in the first segment represents a first layer of a lesion and includes a single diameter measurement (e.g., center slices). Each slice in sequential segment represent a sequential layer of a corresponding lesion (e.g., edge slices). The weights of the segmentation block215are initialized710with initial values. The segmentation block215is applied715to the first segment of CT slices to generate a prediction of a segmentation of each CT slice in the first segment. The weights of the segmentation block215are updated720based on the prediction and the single diameter measurement of each CT slice in the first segment. The weights of the segmentation block215are sequentially updated725based on each additional segment for the plurality of sequential segments.

The weights of the segmentation block215are sequentially updated725by applying the segmentation block215to the additional segment to generate a prediction of a segmentation for each CT slice in the additional segment, refining the prediction of the segmentations to produce a refined segmentation of each CT slice in the additional segment, updating the weights of the segmentation block215based on the refined segmentations and the CT slices in the additional segment, and adding the CT slices in the additional segment to a sequential segment. In some embodiments, the segmentations are refined by applying the predictions and each CT slice in the additional segment to a fully-connected conditional random field.

FIG. 8is a high-level block diagram illustrating an example computer800suitable for implementing the entity ofFIG. 1. The example computer800includes at least one processor802coupled to a chipset804. The chipset804includes a memory controller hub820and an input/output (I/O) controller hub822. A memory806and a graphics adapter812, which contains a graphics processing unit (GPU)813, are coupled to the memory controller hub820, and a display818is coupled to the graphics adapter812. A storage device808, keyboard810, pointing device814, and network adapter816are coupled to the I/O controller hub822. Other embodiments of the computer800have different architectures.

In the embodiment shown inFIG. 8, the storage device808is a non-transitory computer-readable storage medium such as a hard drive, compact disk read-only memory (CD-ROM), DVD, or a solid-state memory device. The memory806holds instructions and data used by the processor802. The pointing device814is a mouse, track ball, touch-screen, or other type of pointing device, and is used in combination with the keyboard810(which may be an on-screen keyboard) to input data into the computer800. The graphics adapter812displays images and other information on the display818. The network adapter816couples the computer800to one or more computer networks.

The GPU813in the graphics adapter812may be used for other high-performance computation as well as processing graphical data for presentation on the display818. In one embodiment, the GPU813is used to process data from the image segmentation system125, where it is used to accelerate model training, image processing, and image segmentation.

The types of computers used by the entities ofFIG. 8can vary depending upon the embodiment and the processing power required by the entity. For example, the image segmentation system125might include a desktop computer to provide the functionality described. Furthermore, the computers can lack some of the components described above, such as keyboards810, graphics adapters812, and displays818.