Patent Description:
Radiotherapy (radiation-based therapy) is used as a cancer treatment to emit high doses of radiation that can kill cells or shrink a tumor. The goal is to deliver enough radiation to a target region of the patient's anatomy to kill the cancerous cells during the radiotherapy treatment. Other organs or anatomical regions that are adjacent to, or surrounding, the target region can be in the way of radiation beams and can receive enough radiation to damage or harm such organs. A physician or a radiation oncologist identifies organs of interest, which would typically include both the target region and the organs at risk, prior to radiotherapy using various imaging modalities. Furthermore, simulation images of the patient's anatomy may be obtained.

For safe and effective radiotherapy treatment, it is crucial to accurately segment organs of interest. Manual delineation of target volumes and organs at risk is still standard routine for many clinics, even though it is time consuming and prone to intra-observer and inter-observer variations. Automated segmentation methods seek to reduce delineation workload and unify the organ boundary definition. In deploying automated segmentation to clinical applications, however, it is necessary to address the issue of quality control. State-of-the-art autosegmentation methods such as methods incorporating machine learning can still fail. Additionally, existing methods do not lend themselves to fully automated contouring. It is important to detect any critical inaccuracies, which can lead to misidentification.

<NPL>, discusses that Class Activation Mapping (CAM) can be used to obtain a visual understanding of the predictions made by Convolutional Neural Networks (CNNs), facilitating qualitative insight into these neural networks when they are, for instance, used for the purpose of medical image analysis.

<NPL>), discusses using a weakly supervised learning to address the problem of lack of data and annotated data in medical imaging. The paper discusses an approach to locate different type of lesions in positron emission tomography (PET) images using only a class label at the image-level.

Current clinical practice of segmentation quality control requires human visual inspection, e.g., by a radiation oncologist. Autosegmentation algorithms do not provide any guidance to the user as to which regions of the segmentation results should be evaluated closely. Therefore a clinician providing second check to the segmentation algorithm needs to evaluate every region of every contour. This is very resource intensive. This poses a need for automated techniques that can highlight regions of the contours, most likely to need correcting, for a human to evaluate.

According to one aspect of the invention, there is provided a method according to claim <NUM>.

According to a third aspect of the invention, there is provided a system according to claim <NUM>.

Optional features are defined in the dependent claims.

For the aforementioned reasons, there is a need for systems and methods for automatic localized evaluation of contours of organs of interest to highlight problematic regions of the contours for a human to evaluate. There is a need for automatic evaluation of segmentation of an organ of interest that provides a confidence level for the organ segmentation. Disclosed systems and methods for quality assurance identify regions of a contour that contribute to a low confidence level of an organ segmentation. Disclosed systems and methods automatically evaluate organ contours to provide visual indications of confidence level of localized areas of the contours.

As described in embodiments herein, a dual-input machine learning neural network classifier may be trained on image pairs including medical images of organs of interest and segmentation (mask) images. These image pairs are of two types: image pairs in which the mask image is the ground truth segmentation, and image pairs in which the mask differs from the ground truth. The machine learning classifier is trained to distinguish whether an image pair represents the ground truth, or does not represent the ground truth. Training data may incorporate two data sets. One data set includes medical images of organs of interest plus ground truth segmentation (mask) images. The other data set includes medical images of organs of interest plus segmentation (mask) images that differ from the ground truth. The neural network classifier is trained to distinguish between the two data sets.

After training, when presented with a medical image/mask pair, the neural network classifies the medical image/mask pair as representing the ground truth or not. If the image pair is classified as not representing ground truth, the network may determine localized regions of the medical image/mask pair that contribute to the classification by analyzing layer weights of the neural network. These localized regions may be highlighted and reported to a clinician for quality assurance review.

A localized evaluation network may incorporate a discriminative network, also referred to herein as discriminator, which acts as a classifier. The localized evaluation network may include a generative network, also referred to as a generator. In an embodiment, the discriminative and generative networks collectively form a Generative Adversarial Network (GAN). In an embodiment, the generative network may be a Convolutional Neural Network (CNN), such as U-Net, configured to generate segmentations.

The localized evaluation network may apply the GAN to determine weights of interior layers of the discriminator. In an embodiment, a Class Activation Map (CAM) may examine the interior layers and evaluate how much each localized image region contributes to the final classification. Disclosed embodiments provide visual indications of confidence level of localized regions of the contours, e.g., via a heat map.

According to the invention, a method is disclosed in claim <NUM>.

The overlay within the second image may be generated via a second machine learning model.

The set of fourth images may be previously generated via the second machine learning model.

The method may further comprise training the machine learning model based on a set of fifth images depicting a set of third organs having a set of third outlines and a set of sixth images comprising a set of fourth overlays correctly contouring the set of third outlines.

The first confidence score may indicate a likelihood of a plurality of localized regions of the overlay correctly contouring localized regions of the outline. The first confidence score may indicate likelihood that the relation between the overlay and the anatomical image is similar to those in the training data that are labelled as correct.

The machine learning model may be trained by applying a GAN to training data from the set of third images and the set of fourth images.

The machine learning model may be trained by applying the GAN to determine weights of interior layers of a discriminator.

The discriminator may generate a CAM to determine image regions that result in a reduced confidence score based on training data from the set of third images and the set of fourth images.

The contributions of the respective localized regions to the first confidence score may correspond to weights of layers of the machine learning model.

The visual attribute may correspond to color coding of a spatial heat map.

The machine learning model may be trained via a supervised training protocol.

The machine learning model may be a binary classifier with matched and unmatched classes.

The first image may be a computed tomography (CT) image, magnetic resonance imaging (MRI) image, or positron emission tomography (PET) scan image.

According to the invention, a system is disclosed in claim <NUM>.

The instructions when executed by the processor may cause the processor to train the machine learning model by applying a GAN to training data from the set of third images and the set of fourth images.

The CAM may comprise a Gradient-weighted Class Activation Map (Grad-CAM).

The machine learning model may be further trained based on a set of fifth images depicting a set of third organs having a set of third outlines and a set of sixth images comprising a set of fourth overlays correctly contouring the set of third outlines.

Non-limiting embodiments of the present disclosure are described by way of example with reference to the accompanying figures, which are schematic and are not intended to be drawn to scale. Unless indicated as representing the background art, the figures represent aspects of the disclosure.

Reference will now be made to the illustrative embodiments depicted in the drawings, and specific language will be used here to describe the same. It will nevertheless be understood that no limitation of the scope of the claims or this disclosure is thereby intended. Alterations and further modifications of the inventive features illustrated herein, and additional applications of the principles of the subject matter illustrated herein, which would occur to one skilled in the relevant art and having possession of this disclosure, are to be considered within the scope of the subject matter disclosed herein. Other embodiments may be used and/or other changes may be made without departing from the scope of the present disclosure. The illustrative embodiments described in the detailed description are not meant to be limiting of the subject matter presented.

When medical imaging is necessary to observe an internal organ or a set of internal organs, there are several systems that may be utilized such as X-ray, computed tomography (CT), cone beam CT images (CBCT), four-dimensional CT images (e.g., CT images over time), magnetic resonance imaging (MRI) images, positron emission tomography (PET) images, ultrasound images, and or a combination thereof. Medical images may be two-dimensional or three-dimensional (volumetric). A two-dimensional medical image may be a cross-sectional "slice" of a three-dimensional volume.

One purpose of three-dimensional reconstruction of the structure(s) of interest containing diseased or abnormal tissues or organs is the preparation of a three-dimensional radiation therapy treatment plan. To verify that a radiation treatment procedure is correctly applied, quality assurance protocols are implemented to verify that the developed treatment plan is accurate.

Radiation therapy treatment plans are used during medical procedures that selectively expose precise areas of the body, such as cancerous tumors, to specific doses of radiation to destroy the undesirable tissues. An initial treatment plan may be prepared that defines the area in the human body to be treated, such as cancerous tissue, abnormal cells, lesions, and organs, called the clinical target volume (CTV). Another volume called the planning target volume (PTV) allows for uncertainties in planning or treatment delivery to ensure that the radiotherapy dose is actually delivered to the CTV. Radiotherapy planning generally considers critical normal tissue structures near the CTV, known as organs at risk (OARs). The goal is to deliver enough radiation to the PTV to kill the cancerous cells during the radiotherapy treatment. OARs that are adjacent to, or surrounding, the PTV can be in the way of radiation beams and can receive enough radiation to damage or harm such organs or anatomical regions. Typically a physician or a radiation oncologist identifies both the PTV and the OARs prior to radiotherapy using a suitable imaging modality. Additionally, a physician or radiation oncologist may identify lesions, e.g., regions in an organ or tissue that have suffered damage through injury or disease. Furthermore, simulation images of the patient's anatomy may be obtained. PTV, OARs, lesions and other organs or anatomical regions of interest to a physician or a radiation oncologist in planning radiotherapy are referred to herein as organs of interest.

For safe and effective radiotherapy treatment, it may be crucial to accurately segment organs of interest. Manual segmentation methods require significant amounts of specialist time, while automated segmentation methods may be deployed to reduce delineation workload. In contouring organs of interest in clinical applications it is necessary to address the issue of quality control. State-of-the-art autosegmentation methods such as methods incorporating machine learning can still fail. Autosegmentation algorithms typically do not provide any guidance to the user as to which regions of the segmentation results should be evaluated closely. Clinicians providing second check to the segmentation algorithm need to evaluate every region of every contour which is very resource intensive.

Embodiments disclosed herein integrate automatic localized evaluation of contours of organs of interest to highlight problematic regions of the contours for a human to evaluate. Automatic evaluation of segmentation of an organ of interest may provide a confidence level for the organ segmentation. Systems and methods for quality assurance identify regions of a contour that contribute to a low confidence level of an organ segmentation. Embodiments disclosed herein automatically evaluate contours and provide visual indications of confidence level of localized regions of the contours.

<FIG> illustrates components of a system for automated localized evaluation of contours <NUM>, according to an embodiment. The system <NUM> may include an analytics server <NUM>, system database 110b, machine learning models 111a, 111b, electronic data sources 120a-d (collectively electronic data sources <NUM>), end-user devices 140a-c (collectively end-user devices <NUM>), an administrator computing device <NUM>, and a medical device <NUM> having a medical device computer <NUM>. Various components depicted in <FIG> may belong to a radiotherapy clinic at which patients may receive radiotherapy treatment, in some cases via one or more radiotherapy machines located within the clinic (e.g., medical device <NUM>). The above-mentioned components may be connected to each other through a network <NUM>. Examples of the network <NUM> may include, but are not limited to, private or public LAN, WLAN, MAN, WAN, and the Internet. The network <NUM> may include wired and/or wireless communications according to one or more standards and/or via one or more transport mediums.

The system <NUM> is not confined to the components described herein and may include additional or other components, not shown for brevity, which are to be considered within the scope of the embodiments described herein.

The communication over the network <NUM> may be performed in accordance with various communication protocols such as Transmission Control Protocol and Internet Protocol (TCP/IP), User Datagram Protocol (UDP), and IEEE communication protocols. In one example, the network <NUM> may include wireless communications according to Bluetooth specification sets or another standard or proprietary wireless communication protocol. In another example, the network <NUM> may also include communications over a cellular network, including, e.g., a GSM (Global System for Mobile Communications), CDMA (Code Division Multiple Access), EDGE (Enhanced Data for Global Evolution) network.

The analytics server <NUM> may generate and display an electronic platform configured to use various computer models <NUM> (including artificial intelligence and/or machine learning models such as Discriminator 111a and Generator 111b) for automated localized evaluation of contours. The electronic platform may display one or more medical images such as images of patient organs of interest, and images of contours of such organs. At an inference phase for a patient treatment plan, the electronic platform may display a confidence score indicative of an accuracy of medical images containing contours of a current patient's organs of interest or other target volumes. Depending on confidence score determined, the platform may display regions of medical images containing contours of organs of interest that contribute to a negative classification. These regions may be highlighted and reported for quality assurance review. Disclosed embodiments may provide visual indications of confidence level of localized regions of the contours, e.g., via heat maps.

The electronic platform may include a graphical user interface (GUI) displayed on each electronic data source <NUM>, the end-user devices <NUM>, and/or the administrator computing device <NUM>. An example of the electronic platform generated and hosted by the analytics server <NUM> may be a web-based application or a website configured to be displayed on different electronic devices, such as mobile devices, tablets, personal computers, and the like.

In a non-limiting example, a physician or radiation oncologist operating the medical professional device 120b, 140c may access the platform, review the confidence score and medical images, and in appropriate cases may initiate a manual quality assurance review. The physician or radiation oncologist may review results highlighting problematic regions of organs of interest that contribute to a low confidence score for an organ contour. The physician or radiation oncologist may visually inspect heat maps or other data visualizations displaying spatial patterns of confidence level of localized regions of the contours. Therefore, the medical professional devices (e.g., the medical professional device 140c) may be used as both a device to display results predicted by the analytics server <NUM> and in some cases may also be used as an electronic data source (e.g., electronic data source 120b) to train the machine learning models <NUM>.

The analytics server <NUM> may host a website accessible to users operating any of the electronic devices described herein (e.g., end users, medical professionals), where the content presented via the various webpages may be controlled based upon each particular user's role or viewing permissions. The analytics server <NUM> may be any computing device comprising a processor and non-transitory machine-readable storage capable of executing the various tasks and processes described herein. The analytics server <NUM> may employ various processors such as central processing units (CPU) and graphics processing units (GPU), among others. Non-limiting examples of such computing devices may include workstation computers, laptop computers, server computers, and the like. While the system <NUM> includes a single analytics server <NUM>, the analytics server <NUM> may include any number of computing devices operating in a distributed computing environment, such as a cloud environment.

The analytics server <NUM> may execute software applications configured to display the electronic platform (e.g., host a website), which may generate and serve various webpages to each electronic data source <NUM> and/or end-user devices <NUM>. Different users may use the website to view and/or interact with displayed content.

The analytics server <NUM> may be configured to require user authentication based upon a set of user authorization credentials (e.g., username, password, biometrics, cryptographic certificate, and the like). The analytics server <NUM> may access the system database 110b configured to store user credentials, which the analytics server <NUM> may be configured to reference in order to determine whether a set of entered credentials (purportedly authenticating the user) match an appropriate set of credentials that identify and authenticate the user.

The analytics server <NUM> may generate and host webpages based upon a particular user's role within the system <NUM>. In such implementations, the user's role may be defined by data fields and input fields in user records stored in the system database 110b. The analytics server <NUM> may authenticate the user and may identify the user's role by executing an access directory protocol (e.g., LDAP). The analytics server <NUM> may generate webpage content that is customized according to the user's role defined by the user record in the system database 110b.

The analytics server <NUM> may receive medical images from a user or retrieve such data from a data repository, analyze the data, and display the results on the electronic platform. For instance, in a non-limiting example, the analytics server <NUM> may query and retrieve medical images from the database 120d and combine the medical images with segment data received from a physician operating the medical professional device 120b. Additionally, or alternatively, the analytics server <NUM> may segment the medical image automatically or perform other pre-processing steps on the medical image captured from the medical device <NUM>.

The analytics server <NUM> may execute various machine learning models <NUM> (e.g., Discriminator 111a and Generator 111b) to analyze the retrieved data. The analytics server <NUM> may then display the results via the electronic platform on the administrator computing device <NUM> and/or the end-user devices <NUM>.

The electronic data sources <NUM> may represent various electronic data sources that contain, retrieve, and/or input data associated with a patient's treatment plan including patient data and treatment data. For instance, the analytics server <NUM> may use the clinic computer 120a, medical professional device 120b, server 120c (associated with a physician and/or clinic), and database 120d (associated with the physician and/or the clinic) to retrieve/receive data associated with the patient's treatment plan.

End-user devices <NUM> may be any computing device comprising a processor and a non-transitory machine-readable storage medium capable of performing the various tasks and processes described herein. Non-limiting examples of an end-user device <NUM> may be a workstation computer, laptop computer, tablet computer, and server computer. In operation, various users may use end-user devices <NUM> to access the GUI operationally managed by the analytics server <NUM>. Specifically, the end-user devices <NUM> may include clinic computer 140a, clinic server 140b, and a medical processional device 140c. Even though referred to herein as "end user" devices, these devices may not always be operated by end users. For instance, the clinic server 140b may not be directly used by an end user. However, the results stored onto the clinic server 140b may be used to populate various GUIs accessed by an end user via the medical professional device 140c.

The administrator computing device <NUM> may represent a computing device operated by a system administrator. The administrator computing device <NUM> may be configured to display radiation therapy treatment attributes generated by the analytics server <NUM> (e.g., various analytic metrics determined during training of one or more machine learning models and/or systems); monitor various models <NUM> utilized by the analytics server <NUM>, electronic data sources <NUM>, and/or end-user devices <NUM>; review feedback; and/or facilitate training or retraining (calibration) of the machine learning models <NUM> that are maintained by the analytics server <NUM>.

The medical device <NUM> may be a radiotherapy machine configured to implement a patient's radiotherapy treatment. The medical device <NUM> may also include an imaging device capable of emitting radiation such that the medical device <NUM> may perform imaging according to various methods to accurately image the internal structure of a patient. For instance, the medical device <NUM> may include a rotating system (e.g., a static or rotating multi-view system). A non-limiting example of a multi-view system may include a stereo systems (e.g., two systems may be arranged orthogonally). The medical device <NUM> may also be in communication with a medical device computer <NUM> that is configured to display various GUIs discussed herein. For instance, the analytics server <NUM> may display the results predicted by the machine learning models 111a, 111b onto the medical device computer <NUM>.

In operation, a physician or other medical professional may access an application executing on the medical professional device 120b and input patient data and the patient's treatment data (e.g., patient information, patient diagnosis, radiation therapy radiation requirements and thresholds). The analytics server <NUM> may then use a patient identifier to query patient data (e.g., patient anatomy and/or medical images) from the electronic data sources <NUM>. The analytics server may then identify a clinic associated with the patient (e.g., clinic performing the treatment) and retrieve one or more files associated with treatment templates and clinic rules. The analytics server <NUM> may then utilize the systems and methods described herein to generate data pertaining to automated localized evaluation of contours.

A medical professional at a radiotherapy clinic may access an end-user device <NUM> located at the clinic or access an account associated with the clinic. The medical professional may provide an input at a user interface that causes the end user device <NUM> to transmit a request to access machine learning models <NUM> that are associated with the clinic and/or the radiotherapy machines located within the clinic. The request may include an identifier associated with the machine learning models <NUM>, the clinic, a treatment plan generated by the one or more medical professionals, and/or the set of radiotherapy machines that the analytics server <NUM> may use as a key in a look-up table to identify the machine learning models <NUM>. The analytics server <NUM> may receive the request and, in some cases, after authenticating the user, identify the machine learning models <NUM> via the identifier. The analytics server <NUM> may transmit the identified machine learning models <NUM> to the end-user device <NUM> or send an alert indicating the end-user device is authorized to access the models <NUM>. Upon receipt or access to the machine learning model <NUM>, the end user device <NUM> may perform the systems and methods described herein to train or retrain the machine learning models <NUM> to predict automated localized evaluations of contours.

The analytics server <NUM> may store machine learning models <NUM> (e.g., neural networks, random forest, support vector machines, or other deep learning models including Discriminator 111a and Generator 111b, combined in GAN <NUM>), that are trained to predict the anatomical structure represented by various pixels or voxels of a medical image. Various machine learning techniques may involve "training" the machine learning models to predict (e.g., estimate the likelihood of) each pixel or voxel of a medical image being associated with or otherwise representing a particular anatomical structure.

Machine learning models <NUM> may be stored in the system database 110b and may correspond to individual radiotherapy clinics or otherwise different sets of radiotherapy machines (e.g., radiotherapy machines that are located at individual radiotherapy clinics, are located in different geographical regions, treat specific types of diseases such as different types of cancer, treat specific genders, etc.). For example, the machine learning model <NUM> may be associated with an identifier indicating the radiotherapy clinic, set of radiotherapy machines, or a specific disease.

In various embodiments, machine learning models <NUM> use one or more deep learning engines to perform automatic segmentation of image data for radiotherapy treatment planning. Although exemplified using deep convolutional neural networks, it should be understood that any alternative and/or additional deep learning model(s) may be used to implement deep learning engines. The deep learning engines include processing pathways that are trained during training phase. Once trained, deep learning engines may be used (e.g., by a clinician) to perform automatic segmentation for current patients during inference phase.

One type of deep learning engine is a convolutional neural network (CNN). A CNN is a branch of neural networks and consists of a stack of layers each performing a specific operation, e.g., convolution, pooling, loss calculation, etc. Each intermediate layer receives the output of the previous layer as its input. The beginning layer is an input layer, which is directly connected to an input image and may have a number of neurons equal to the number of pixels in the input image. The next set of layers are convolutional layers that present the results of convolving a certain number of filters with the input data and perform as a feature extractor. The filters, commonly known as kernels, are of arbitrary sizes defined by designers. Each neuron responds only to a specific area of the previous layer, called a receptive field. The output of each convolution layer is considered as an activation map, which highlights the effect of applying a specific filter on the input. Convolutional layers may be followed by activation layers to apply non-linearity to the outputs of each layer. The next layer can be a pooling layer that helps to reduce the dimensionality of the convolution's output. In various implementations, high-level abstractions are extracted by fully connected layers. The weights of neural connections and the kernels may be continuously optimized in the training phase.

In practice, training data may be user-generated through observations and experience to facilitate supervised learning. For example, training data may be extracted from past treatment plans developed for prior patients. Training data may be pre-processed via any suitable data augmentation approach (e.g., rotation, flipping, translation, scaling, noise addition, cropping, any combination thereof, etc.) to produce a new dataset with modified properties to improve model generalization using ground truth.

As shown in <FIG>, models <NUM> for automatic localized evaluation of contours of organs of interest incorporate a Discriminator 111a and a Generator 111b combined in GAN <NUM>. The generator 111b may be CNN such as U-NET configured to generate segmentations. GAN <NUM> is an adversarial network that serves as a classifier. U-NET 111b generates segmentations during model training, and GAN discriminator 111a evaluates the segmentations. Models <NUM> may apply GAN to determine weights of interior layers of the discriminator 111a. GAN <NUM> utilizes unsupervised machine learning based on indirect training through the discriminator, which itself may be updated dynamically. The training of the adversarial network <NUM> can be based on similar organs or based on image segmentations via different machine learning models.

Inputs to the models <NUM> include a set of image pairs each of which includes a medical image such as a CT scan and a segmentation or contour. Image pairs may embody a correct association of the CT scan with a mask or segmentation created from the CT scan. Alternatively, image pairs may embody an association of the CT scan with a fake or incorrect mask. GAN discriminator 111a evaluates image pairs to determine which image pairs are correctly matched (e.g., have sufficiently high confidence score), and which image pairs are incorrectly matched. Model outputs may include a confidence score indicating likelihood that the image pair is matched, and may include a binary classification (e.g., Matched or Unmatched).

The flow chart of <FIG> shows a localized evaluation network <NUM>. Localized evaluation network <NUM> includes a discriminator, which acts as a classifier. The localized evaluation network also includes a generative network, or generator. In an embodiment, the discriminator and generator collectively form a Generative Adversarial Network (GAN). Localized evaluation network <NUM> may apply the GAN to determine weights of interior layers of the discriminator. In an embodiment, the generative network may be a CNN such as U-Net configured to create segmentations.

Network <NUM> receives an image pair <NUM> as input. The image pair includes the medical image on the left (CT <NUM>) and the segmentation or contour on the right (Mask <NUM>). In an embodiment, a clinician has delineated the brain in the Mask <NUM>. The Mask <NUM> may be produced by autosegmentation, or may be produced by a human such as a radiation therapist or medical dosimetric. Image pair <NUM> may embody a correct association of CT <NUM> with a mask <NUM> created from the CT scan. This type of image pair may be labeled as CORRECT in training data of a supervised training protocol. Alternatively, image pair <NUM> may embody an association of CT <NUM> with a fake or incorrect mask <NUM>. This type of image pair may be labeled as INCORRECT in training data of a supervised training protocol. For example, an incorrect mask may be selected from a series of segmentations different than the mask correctly associated with CT <NUM>.

Input layer <NUM> concatenates the CT image input <NUM> and the Mask image input, also called target image, <NUM>. This results in an N × N × <NUM> spatially local pattern for each convolutional layer, including two values for each pixel. Each of a series of ResNet blocks <NUM>. <NUM> of the network corresponds to a given convolutional layer. Using residual blocks <NUM>. <NUM>, in addition to feeding the layer activation to the next layer, the activation map is fast-forwarded to a deeper layer in the convolutional neural network. Each layer may feed into the next layer and may directly feed into layers about <NUM>-<NUM> hops away. ResNets allow neural networks with a large number of layers to be trained more easily.

Each convolutional layer takes a small portion of the image and applies convolutional filters to it in order to abstract features of the image. ResNet block layers <NUM>. <NUM> apply convolutional filters to the image to abstract the image to a feature map, also called an activation map. Each filter may be replicated across the entire visual field. These replicated units share the same parameterization (weight vector and bias) and form an activation map. The convolutional filters identify features present in the image such as edges, vertical lines, horizontal lines, and/or bends, etc. Given convolutional filters may correspond to given spatial patterns, such as a filter that looks for horizontal lines or a filter that looks for vertical lines. Convolutional filters have learnable parameters. The weights of a layer are the values determined by applying the convolutional filters to the layer.

Just before the final output layer, the CNN performs global average pooling <NUM>. The features thus obtained are fed to a dense layer <NUM> that produces the desired output. Dense layer <NUM> is a fully connected layer in which all the neurons in the layer are connected to those in the next layer. In the localized evaluation network <NUM>, dense layer <NUM> outputs a single number representing a binary classification, either Matched <NUM> or Unmatched <NUM>. Dense <NUM> may output a confidence score S, e.g., a value between <NUM> and <NUM> that may classify the image pair <NUM> as Matched <NUM> if S > <NUM> and may classify the image pair <NUM> as Unmatched <NUM> if S ≤ <NUM>.

The confidence scores of respective localized regions may correspond to weights of layers of the machine learning model. The weights of model layers represent contributions to the final confidence score. For example, layer weights may indicate image regions that make a major contribution to an Unmatched <NUM> binary classification. These regions may represent the highest priority for quality assurance review of a low confidence score segmentation.

The network <NUM> applies Class Activation Maps (CAM) as a visualization technique. A CAM for a particular target category indicates the discriminative region used by CNN to identify the category. In an embodiment, a CAM looks at the intermediate layers before arriving at the final result. CAM examines each layer and evaluates how much each sub-image or image region contributes to the binary classification <NUM>, <NUM>. The discriminator may generate a CAM to determine image regions that result in a reduced confidence score based on training data from the set of third images and the set of fourth images.

Disclosed embodiments may apply various techniques to visualize local features of an image captured in activation maps. Visualization techniques may generate a heat map. In disclosed embodiments, a heat map, also referred to herein as a spatial heat map, employs variation in color such as by intensity or hue to provide obvious visual cues about how a target phenomenon varies over space. A spatial heat map may be created by a software program for visualizing spatial data.

In an embodiment, localized evaluation network <NUM> applies Gradient-weighted Class Activation Mapping (Grad-CAM) to provide visual explanations of local features of an image. Grad-CAM is described in <NPL>. Grad-CAM uses gradients of a target concept entering the final convolutional layer to produce a coarse localization map. Grad-CAM highlights important regions in the image for predicting the target concept. Grad-CAM creates a high-resolution class-discriminative visualization. Disclosed embodiments apply Grad-CAM to understand how local image features affect an overall confidence score for the image. In an embodiment, Grad-CAM computes a coarse Grad-CAM localization to generate a heat map. The heat map may show relative contributions of different local regions to a confidence score.

<FIG> illustrates results from running the trained adversarial network discriminator of <FIG> at inference time to generate a series of medical image/mask pairs <NUM>, <NUM>, <NUM>, and <NUM>. Training data for the adversarial network discriminator inputted four medical images from reference: <NPL>).

In each medical image/mask pair, a main black-and-white image shows a CT slice of the brain. Each of the green lines <NUM>, <NUM>, <NUM>, and <NUM> contours the inside of the skull and represents the ground truth segmentation. For example, the ground truth segmentation may be delineated by a human, or may be an autosegmentation evaluated to be correct by human review. Each of the black lines <NUM>, <NUM>, <NUM>, and <NUM> contours the inside of the skull and represents an incorrect segmentation.

The trained adversarial network generated heat maps <NUM>, <NUM>, <NUM>, and <NUM> providing CAM visualizations. Each color wash includes a heat map having a spatial pattern of hues representing weights of layers of the machine learning model captured in the CAM. The weights of layers of the machine learning model may correspond to contributions of localized regions of the segmentation to an overall confidence score. As shown in the color gradient bar <NUM>, higher values corresponding to yellow and green hues represent layer regions that more likely to be incorrect. Lower values corresponding to blue and purple hues represent layer regions that are more likely to be correct. The yellow and green regions provide the highest contributions to a negative overall confidence score.

<FIG> illustrates a flow diagram of a process executed by an analytic server. In step <NUM>, the analytic server executes a machine learning model that receives an input of a first image of an anatomical region of a patient depicting an organ. A second image includes an overlay contouring an outline of the anatomical region. The machine learning model is configured to predict a first confidence score indicating a likelihood of the overlay correctly contouring the outline. The machine learning model is trained based on a set of third images depicting a set of second organs having a set of second outlines and a set of fourth images including a set of second overlays incorrectly contouring the set of second outlines.

In various embodiments of step <NUM>, the machine learning model is further trained based on a set of fifth images depicting a set of third organs having a set of third outlines and a set of sixth images including a set of fourth overlays correctly contouring the set of third outlines.

In step <NUM>, the analytic server presents, for display on a graphical user interface, the first image and a third overlay. A plurality of localized regions within the third overlay have a visual attribute that represents contributions of respective localized regions of the plurality of localized regions to the first confidence score.

Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of this disclosure or the claims.

Embodiments implemented in computer software may be implemented in software, firmware, middleware, microcode, hardware description languages, or any combination thereof. A code segment or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc..

The actual software code or specialized control hardware used to implement these systems and methods is not limiting of the claimed features or this disclosure. Thus, the operation and behavior of the systems and methods were described without reference to the specific software code being understood that software and control hardware can be designed to implement the systems and methods based on the description herein.

When implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer-readable or processor-readable storage medium. The steps of a method or algorithm disclosed herein may be embodied in a processor-executable software module, which may reside on a computer-readable or processor-readable storage medium. A non-transitory computer-readable or processor-readable media includes both computer storage media and tangible storage media that facilitate transfer of a computer program from one place to another. A non-transitory processor-readable storage media may be any available media that may be accessed by a computer. By way of example, and not limitation, such non-transitory processor-readable media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible storage medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer or processor. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and/or instructions on a non-transitory processor-readable medium and/or computer-readable medium, which may be incorporated into a computer program product.

The preceding description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the embodiments described herein and variations thereof. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other embodiments without departing from the scope of the subject matter disclosed herein.

Claim 1:
A method comprising:
executing, by a processor, a machine learning model that receives an input of a first image of an anatomical region of a patient depicting an organ having an outline and a second image comprising an overlay contouring the outline and further configured to predict a first confidence score indicating a likelihood of the overlay correctly contouring the outline, wherein the overlay is a contour,
wherein the machine learning model is trained based on a set of third images depicting a set of second organs having a set of second outlines and a set of fourth images comprising a set of second overlays incorrectly contouring the set of second outlines, and
wherein the machine learning model is further trained based on a set of fifth images depicting a set of third organs having a set of third outlines and a set of sixth images comprising a set of fourth overlays correctly contouring the set of third outlines,
wherein the machine learning model comprises a discriminative network, and
wherein the discriminative network generates a Class Activation Map, CAM, to determine contributions of localized regions of the contour to the first confidence score; and
presenting, by the processor for display on a graphical user interface, the first image and a third overlay, and providing visual indications of the localized regions of the contour that contribute to the first confidence score