Patent Description:
Segmentation of anatomy and lesions is an integral part of treatment planning for cancer and other similar diseases. Manual segmentation of anatomical features takes a long time and can only be performed by qualified expert radiologists. Therefore, various automated methods of segmentation have been developed, some of which use deep learning to segment medical images into different segments or portions corresponding to different features in the respective images. Due to recent advancements in the field of AI, machine learning (ML) models for segmentation have begun to exceed (individual) human performance. ML models for segmentation are described in the paper by <NPL>.

These tools work generally well on data similar to that on which the model has been trained, however with the ever-changing quality of the imaging and other image characteristics, models need to be frequently updated and re-trained as part of ongoing maintenance. Further training requires new annotated data e.g. new radiological images that are manually segmented in order to provide "ground truth" or correct annotations with which to train the model. Since obtaining the annotations for segmentation problems takes significantly more time and skill than classification problems involving simple descriptive labels, it can be difficult to obtain a large enough dataset with which to train an AI model to segment medical images.

<NPL>, discloses a segmentation framework combined with the quality-driven active learning (QDAL) module for suggestive annotation, whereby the proposed Active Learning module, attention mechanism and deep supervision mode are used to improve the segmentation accuracy and feedback segmentation quality information.

It is thus an object of embodiments herein to reduce the annotation burden on expert human annotators when producing training data for segmentation models, without degrading the quality of the resulting models.

Active Learning / in-product learning is a supervised machine learning method whereby particular unlabeled data examples are strategically selected for annotation, and for use in further training of the model, according to which data examples are most likely to provide the most benefit (e.g. lead to the biggest improvement) to the training of the model. Active learning is a special case of supervised learning in which the active learning process itself is able to interactively query or request labels for data examples that are the most confusing or that the model is able to label with lowest confidence. For example, the model may select data examples for which the model has low confidence in its output, edge cases, or examples lying in a region of the input feature space that has previously been under-represented. In this way, the machine learning process prioritizes the annotation work i.e. instead of training data being provided in a more random manner, training data for annotation is selected strategically based on which data is most uncertain/complex and/or informative with which to seed the model. The selected training examples are then sent to an annotator for annotation and the model is then trained on the selected annotated data. The process is repeated until there is improvement in a scoring metric for the model. In this way, the model is only trained on the most important data, and full annotations only need to be provided on a selected subset of data examples, rather than the full dataset that would be required if the training data were selected in a traditional manner. By learning from the most informative cases first, active learning tends to lead to similar or generally better outcomes than traditional fully supervised methods.

Active Learning is often used for classification problems where it can be comparatively straight forward to select samples from an unlabeled pool of data examples that are confusing to the model. For example, in a classification problem, the output probability can be used as a criterion for selecting samples from an unlabeled pool of data, based on output probability values (e.g. by selecting data examples from the pool that the model classified with low confidence).

In the medical field, radiologists spend a lot of time segmenting (e.g. outlining structures in) images of lesions (such as tumors), organs at risk and organs for treatment planning. Deep-learning based segmentation models can assist radiologists to optimize their workflows. In theory, active learning provides an efficient mechanism with which to choose appropriate training data examples for labelling from a pool of unlabeled training examples, thus minimizing the annotation effort needed from radiologists whilst maximizing the improvement in the model for each annotation received. However, in practice, the criterion for selecting data examples from an unlabeled pool of samples is not as clearly defined for segmentation problems as it is for more simple classification problems, and is generally more complex.

Thus, it is an object of embodiments herein to provide improved methods of selecting unlabeled medical images for annotation and subsequent training of a model to segment medical images.

Thus, according to a first aspect, there is provided a computer implemented method for use in selecting training data for annotation by an annotator as part of an active-learning machine learning process to train a model to take as input a medical image and output a segmentation of the medical image. The method comprises: providing an unlabeled medical image as input to the model and obtaining a segmentation of the unlabeled medical image as output from the model; (a) determining whether a geometric property of a segment of the segmentation satisfies one or more geometric criteria of a radiological feature represented by the segment; and/or (b) determining whether an edge region of the segment satisfies one or more edge criteria; and selecting the unlabeled medical image as training data that is to be annotated by the annotator as part of the active-learning machine learning process, if the segment does not satisfy the one or more geometric criteria and/or the one or more edge criteria.

According to a second aspect there is a computer program product comprising a computer readable medium, the computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform the method of the first aspect.

According to a third aspect there is a system for use in selecting training data for annotation by an annotator as part of an active-learning machine learning process to train a model to take as input a medical image and output a segmentation of the medical image. The system comprises a memory comprising instruction data representing a set of instructions, and a processor configured to communicate with the memory and to execute the set of instructions. The set of instructions, when executed by the processor, cause the processor to: provide an unlabeled medical image as input to the model and obtain a segmentation of the unlabeled medical image as output from the model. The processor is further caused to (a) determine whether a geometric property of a segment of the segmentation satisfies one or more geometric criteria of a radiological feature represented by the segment; and/or (b) determine whether an edge region of the segment satisfies one or more edge criteria. The processor is further caused to select the unlabeled medical image as training data that is to be annotated by the annotator as part of the active-learning machine learning process, if the segment does not satisfy the one or more geometric criteria and/or the one or more edge criteria.

Example embodiments will now be described, by way of example only, with reference to the following drawings, in which:.

As described above, it is an object of embodiments herein to provide more efficient training of machine learning models for use in segmenting medical images. This object is achieved herein, through the use of active learning and through improved methods of selecting unlabeled medical images from a pool of unlabeled medical images, for labelling by a human expert or Oracle. Through improved selection of unlabeled medical images for labelling, the systems and methods herein save time and resources of human experts, whilst providing models of the same or higher quality.

Turning now to <FIG> in some embodiments there is an apparatus <NUM> for use in selecting training data for annotation by an annotator as part of an active-learning machine learning process, according to some embodiments herein. Generally, the apparatus may form part of a computer apparatus or system e.g. such as a laptop, desktop computer or other computing device. In some embodiments, the apparatus <NUM> may form part of a distributed computing arrangement or the cloud.

The apparatus comprises a memory <NUM> comprising instruction data representing a set of instructions <NUM> and a processor <NUM> (e.g. processing circuitry or logic) configured to communicate with the memory and to execute the set of instructions. Generally, the set of instructions, when executed by the processor, may cause the processor to perform any of the embodiments of the method <NUM> as described below.

Embodiments of the apparatus <NUM> may be for use in selecting training data for annotation by an annotator as part of an active-learning machine learning process wherein the selected training data is for use in training a model to take as input a medical image and output a segmentation of the medical image. More specifically, the set of instructions, when executed by the processor, cause the processor to: provide an unlabeled medical image as input to the model and obtain a segmentation of the unlabeled medical image as output from the model. The processor is further caused to a) determine whether a geometric property of a segment of the segmentation satisfies a geometric criteria of a radiological feature represented by the segment; and/or b) determine whether an edge region of the segment satisfies an edge criteria; and select the unlabeled medical image as training data that is to be annotated by the annotator as part of the active-learning machine learning process, if the segment does not satisfy the geometric criteria and/or the edge criteria.

The processor <NUM> can comprise one or more processors, processing units, multi-core processors or modules that are configured or programmed to control the apparatus <NUM> in the manner described herein. In particular implementations, the processor <NUM> can comprise a plurality of software and/or hardware modules that are each configured to perform, or are for performing, individual or multiple steps of the method described herein. The processor <NUM> can comprise one or more processors, processing units, multi-core processors and/or modules that are configured or programmed to control the apparatus <NUM> in the manner described herein. In some implementations, for example, the processor <NUM> may comprise a plurality of (for example, interoperated) processors, processing units, multi-core processors and/or modules configured for distributed processing. It will be appreciated by a person skilled in the art that such processors, processing units, multi-core processors and/or modules may be located in different locations and may perform different steps and/or different parts of a single step of the method described herein.

The memory <NUM> is configured to store program code that can be executed by the processor <NUM> to perform the method described herein. Alternatively or in addition, one or more memories <NUM> may be external to (i.e. separate to or remote from) the apparatus <NUM>. For example, one or more memories <NUM> may be part of another device. Memory <NUM> can be used to store the medical image, the segmentation, the model and/or any other information or data received, calculated or determined by the processor <NUM> of the apparatus <NUM> or from any interfaces, memories or devices that are external to the apparatus <NUM>. The processor <NUM> may be configured to control the memory <NUM> to store the medical image, the segmentation, the model and/or the any other information.

In some embodiments, the memory <NUM> may comprise a plurality of sub-memories, each sub-memory being capable of storing a piece of instruction data. For example, at least one sub-memory may store instruction data representing at least one instruction of the set of instructions, while at least one other sub-memory may store instruction data representing at least one other instruction of the set of instructions.

It will be appreciated that <FIG> only shows the components required to illustrate this aspect of the disclosure and, in a practical implementation, the apparatus <NUM> may comprise additional components to those shown. For example, the apparatus <NUM> may further comprise a display. A display may comprise, for example, a computer screen, and/or a screen on a mobile phone or tablet. The apparatus may further comprise a user input device, such as a keyboard, mouse or other input device that enables a user to interact with the apparatus, for example, to provide initial input parameters to be used in the method described herein. The apparatus <NUM> may comprise a battery or other power supply for powering the apparatus <NUM> or means for connecting the apparatus <NUM> to a mains power supply.

The medical images referred to herein may be two-dimensional (2D) images or three-dimensional (3D) images. The medical images can be of any modality, such as computed tomography (CT) images, magnetic resonance (MR) images, ultrasound (US) images, X-ray images, positron emission tomography (PET) images, single photon emission computed tomography (SPECT) images, nuclear medicine images, or any other medical images.

The medical images herein may be made up of (e.g. comprise) image components which, as used herein, refer to the pixels in 2D images or the voxels of 3D images. The medical images herein may be in any suitable format, such as for example, the Digital Imaging and Communications in Medicine (DICOM) image format.

The model is for use in segmenting medical images. The skilled person will be familiar with segmentation (e.g. image segmentation), but in brief, segmentation involves extracting shape/form information about the objects or shapes captured in an image. This may be achieved by converting the image into constituent blocks or "segments", where the pixels or voxels assigned to each segment have a common attribute. For example, segmentation may involve outlining (creating contours of) the structures in the medical image, labelling image components according to the structure to which they correspond and/or adding color or texture to different structures in the medical image.

The systems and methods herein relate to Machine Learning (ML) segmentation whereby a ML model is used to convert an image into a plurality of constituent shapes (e.g. block shapes or block volumes), based on similar pixel/voxel values and image gradients. A recent review of the use of deep-learning in image segmentation is given in the paper by <NPL>.

The model may be any model trained using a machine learning process to take as input a medical image and output a segmentation of one or more structures in the medical image. For example, the model may be trained to segment an anatomical feature (e.g. part of the anatomy) or lesions (single or multiple) present in a particular anatomical feature. As an example, the model may be trained to segment the liver and/or lesions in the liver. This is merely an example however, and the model may be trained to segment other anatomical features and/or lesions in other anatomical features.

Examples of models that may be trained to perform image segmentation include but are not limited to convolutional neural networks, U-Nets and Encoder-decoder architectures (see paper on "SegNet" by Badrinarayanan et. These are merely examples and the skilled person will appreciate that the disclosure herein applies equally to any type of model that can be trained to perform a segmentation task on medical images that can be trained using an active-learning process.

The model herein may have had initial training. For example, the model may have been trained on an initial training data set comprising training examples, each training example comprising i) an example medical image and ii) a segmentation of said example medical image. The segmentation for each training example may have been provided by a radiologist. As such, the segmentation for each training example represents the "ground truth" segmentation for that training example. The initial training data set may have been selected in any manner (e.g. randomly or based on the available annotated medical images etc). The model may have been initially trained on the initial training data set according to a supervised machine learning process (e.g. using techniques such as back propagation, gradient descent etc), in a known manner. For example, initial training may be performed using an opensource library such as Scikit-learn which is described in the paper: "<NPL>.

Following the initial training, the model is subsequently trained using an active learning process. Such subsequent or further training may be performed e.g. in order to update or refine the model. Updating may be for the purposes of re-training or refinement of the model. Further training may be performed to enable the model to be able to segment a wider range of input images (e.g. to extend the model), to enable the model to keep up-to-date with changes to the input images (e.g. due to improved quality, different manufacturer of the imaging equipment etc, new demographic of patient data, use of the model at a different hospital), or for any other reason.

Generally, some initial training (as described above) is performed before active learning takes place, however it will be appreciated that active learning could equally be used directly on a new model (e.g. with random weights and/or biases).

The skilled person will be familiar with active learning, otherwise known as query learning or in-product learning. Assuming that there is a huge amount of unlabeled data available free of cost, active learning leverages the help of the model to determine which unlabeled data needs to be labelled and fed to the model for retraining. which examples from a pool of unlabeled data samples should be labelled and used as training data in order to best improve the model. In this way, training data can be labelled selectively, better utilizing the expertise and limited resources of the (human) labeller. Active Learning techniques are reviewed in the paper by <NPL>.

As used herein, the human expert may be referred to as the labeller, "Oracle" or annotator of the selected unlabeled data examples in the Active Learning process. The annotator is any expert qualified to segment the data. generally a radiographer who is able to provide a correct ("ground truth") segmentation for the selected training data from the pool of unlabeled data samples.

Turning to <FIG>, there is a computer implemented method <NUM> for use in selecting training data for annotation by an annotator as part of an active-learning machine learning process, wherein the selected training data is for use in training a model to take as input a medical image and output a segmentation of the medical image. For example, the model may output a segmentation of an anatomical feature and/or lesion in the medical image. Embodiments of the method <NUM> may be performed, for example by an apparatus such as the apparatus <NUM> described above.

Briefly, in a first step <NUM>, the method <NUM> comprises: providing an unlabeled medical image as input to the model and obtaining a segmentation of the unlabeled medical image as output from the model. In step <NUM> the method comprises: a) determining whether a geometric property of a segment of the segmentation satisfies a geometric criteria of a radiological feature represented by the segment; and/or
determining whether an edge region of the segment satisfies an edge criteria. In step <NUM> the method comprises: selecting the unlabeled medical image as training data that is to be annotated by the annotator as part of the active-learning machine learning process, if the segment does not satisfy the geometric criteria and/or the edge criteria.

The unlabeled medical image may be retrieved from a pool of unlabeled medical images, e.g. from a hospital or hospital(s) or any other database of relevant data. For example, it may be retrieved from a picture archiving and communication system (PACS). "Unlabeled" in this sense means that it has not been sent for labelling to an oracle, e.g. radiologist in order to obtain a ground-truth segmentation.

The unlabeled medical image may have been selected from the pool of unlabeled medical images according to the input parameters of the model. it will have the same modality and be an image of the same anatomical feature(s) as that taken as input by the model. Furthermore, the subject (e.g. patient) in the image will satisfy any demographic constraints applicable to valid inputs to the model (e.g. the model may be trained to only take inputs for particular patient demographics).

In step <NUM>, the unlabeled medical image is provided (or fed) as input to the model (in the normal way). The model processes the unlabeled medical image and provides a segmentation of the medical image as output.

In option a) of step <NUM>, as noted above, it is determined whether a geometric property of a segment of the segmentation satisfies a geometric criteria of a radiological feature represented by the segment. In option b) (which may be performed alternatively or in addition to option a), it is determined whether an edge region of the segment satisfies an edge criteria. Thus, it is automatically identified through the use of a geometric criteria and/or an edge criteria whether the segment is likely to be correct, or incorrect. If the segment is likely to be incorrect, then the unlabeled image is selected for annotation by an annotator (radiologist) in step <NUM>, so that this image along with its annotation (which is taken as ground truth) can be used as training data to train the model.

As noted above, the model outputs a segmentation of the unlabeled medical image. The segmentation comprises a segment (which may otherwise be known as a mask) corresponding to a particular radiological feature in the unlabeled medical image.

The radiological feature may be an anatomical feature (e.g. organ, liver, heart, part of the vascular structure, or any other anatomical feature), a lesion (e.g. tumor) or any other feature imaged in the unlabeled medical image.

The geometric criteria may be a criteria associated with the particular shape of the radiological feature to which the segment corresponds. For example, if the segment corresponds to a lesion, then the geometric criteria may relate to typical (or expected) geometric shapes associated with lesions. If the segment corresponds to an anatomical feature, then the geometric criteria may relate to typical (or expected) geometric shapes associated with the particular anatomical feature.

As described in more detail below, the geometric feature may be related to the fractal dimension of the surface of the segment, the sphericity, or any other shape feature associated with the radiological feature to which the segment corresponds.

Generally, lesions of different organs have different geometric structures and limitations. For example, liver lesions are generally spherical in shape. Thus, the inventors herein have appreciated that geometry-based estimations can be used to select informative cases for annotation in an active learning framework.

In some embodiments, where the unlabeled medical image comprises an image of a lesion (e.g. tumor or cancer), a segment representing the lesion can be verified (e.g. as likely to correspond to a real tumor) according to the fractal dimension (FD) of its surface. Generally, FD increases with complexity and roughness of a structure. Particularly for cancer, it has been proven that FD increases with aggressiveness. See paper entitled: "<NPL>). Malignant tumors generally have higher FD because they are more ragged and spiculated than benign masses. So, if a lower FD is obtained from a segment for a predicted lesion, it indicates that either the lesion is benign or the model has performed under segmentation. Either way, the unlabeled medical image may be selected and provided to the annotator, for annotation to improve model performance. So, a threshold fractal dimension can be put on the FD of a segment of a predicted lesion structure. If the FD is less than the threshold fractal dimension, then it is more likely not a cancer region and thus it is an incorrect segmentation of the unlabeled medical image that should be selected in step <NUM> in order to reduce false positives. The value of the threshold fractal dimension will change dependent on the segmentation task. For example, it may be set according to the anatomy and kind of lesion that is to be detected. An appropriate value may thus be set by subject matter experts, e.g. human engineers. The appropriate value of the threshold fractal dimension may also be set dependent on the training requirements of the model, for example a wider range of fractal dimension values might be selected in more training examples are needed.

Fractal dimensions for complex structures like lesions can be calculated in many ways. As an example, a box counting method may be used. In a box counting method, the surface is covered with small grids. To calculate the fractal dimension, it is determined how the count of grids covering the lesion changes as we change the grid size. If N(ε) boxes are required to cover surface contour, S with grids of side length ε, the Fractal dimension(D) of contour S can be calculated as: <MAT>.

In other words, in some embodiments of the method <NUM>, the segment represents a lesion. In some embodiments, the lesion is a tumor. As noted above, a lesion is an abnormal region such as a cut, injury, infected region or a tumor. For example, the model may label a segment of the segmentation as a lesion. In such embodiments, the geometric property is a fractal dimension of the surface of the segment, and the geometric criteria is based on fractal dimensions observed on the surfaces of (real) lesions. For example, the step of determining <NUM> whether a geometric property of a segment of the segmentation satisfies a geometric criteria of a radiological feature represented by the segment may comprise determining the fractal dimension of the surface of the segment, comparing the determined fractal dimension to a threshold fractal dimension, and selecting the unlabeled medical image as training data if the determined fractal dimension is less than the threshold fractal dimension.

An appropriate threshold fractal dimension may be determined empirically, e.g. from the fractal dimensions of real lesions. The threshold fractal dimension may be based on an average (e.g. mode, mean, or median) fractal dimension observed in lesions. The threshold fractal dimension may be set according to the number of samples that are required for annotation and further training of the model (e.g. a higher threshold will result in fewer selections in step <NUM> than a lower threshold). It will be appreciated that this applies to embodiments where the medical image is 2D or 3D.

In this way, segments of lesions can be verified as correctly corresponding to a real lesion, or as corresponding to a false positive, based on the fractal dimension of the surface of the segment.

In other embodiments, a segment of a lesion can also be verified and/or determined as being a false-positive based on its sphericity in a 3D image (or its roundness in a 2D image). Sphericity values range between <NUM> and <NUM>. It is expected that, as lesions don't grow in a specific direction, they will tend to have higher sphericity values. Thus, in some embodiments, unlabeled medical images for which the model has output a segment of a lesion with a sphericity below a threshold sphericity, can be selected in step <NUM> for annotation by an annotator according to the active learning method.

The sphericity coefficient, S can be calculated as: <MAT> where Vsegment is the volume of the segment and Asegment is the area of the segment.

Put another way, in some embodiments, the segment represents a lesion, the geometric property is based on the sphericity of the segment, and the geometric criteria is based on sphericities observed in (real) lesions. For example, the step of determining <NUM> whether a geometric property of a segment of the segmentation satisfies a geometric criteria of a radiological feature represented by the segment may comprise determining a measure of sphericity of the segment, such as the measure S above, comparing the determined measure of sphericity to a threshold sphericity, and selecting the unlabeled medical image as training data if the determined measure of sphericity is less than the threshold sphericity.

An appropriate threshold sphericity may be determined empirically, e.g. from the sphericities of real lesions. The threshold sphericity may be based on an average (e.g. mode, mean, or median) sphericity observed in lesions. Furthermore, the threshold sphericity may be set according to the number of samples that are required for annotation and further training of the model (e.g. a higher threshold will result in fewer selections in step <NUM> than a lower threshold).

It will be appreciated that the sphericity is only an example and any other measure that can be related to sphericity may equally be used. For example, a measure of how extended the lesion is could equally be used. For example, a ratio of the minor to major axes. The criteria in such an example being that lesions that have a major axis (minor/major < threshold) are likely to be incorrectly labelled and should thus be sent for annotation in step <NUM>.

In this way, segments of lesions can be verified as correctly corresponding to a real lesion, or as corresponding to a false positive, based on the sphericity of the segment.

Turning now to other embodiments, in some embodiments, an atlas of shapes is used to determine whether the segment is consistent with the radiological feature. For example, if a segment corresponds to the liver of a subject, then the segment may be compared to an atlas of liver shapes.

Put another way, in some embodiments, the step of: determining <NUM> whether a geometric property of a segment of the segmentation satisfies a geometric criteria of a radiological feature represented by the segment comprises fitting shapes from an atlas of shapes of the radiological feature to the segment and selecting the unlabeled medical image as training data if the overlap between the best fitting shape from the atlas of shapes is less than a threshold overlap.

The threshold overlap may vary dependent on the particular training requirements of the model and/or the number of training examples required. For example, the threshold overlap may be increased if too many unlabeled medical images are flagged for annotation, or alternatively lowered if more unlabeled images are needed.

In more detail, imaging of a human body usually involves segmenting an organ within a body and most organs tend to have similar shapes. Badly mis-segmented samples can be identified by mapping the atlas on to every segmentation output. Atlas mapping is performed by registering the atlas on to the mask/segment. The registration method used can be deformable to get a better estimate. One way of measuring the quality of the segmented output is using the following formula: <MAT> The above equation helps compute the amount of overlap (Overlap Coefficient), the intersection of the atlas area and segmented area is taken then divided by the atlas area. This gives a good estimate if there is a lot of under segmentation observed following registration. <MAT> The Exclusivity Coefficient may also be calculated, this provides information on how much of the segmentation from the model is different from the atlas. The weighted summation of the Overlap coefficient and the Exclusivity Coefficient can be used to provide an Atlas Coefficient as follows: <MAT> w1 and w2 are weights, the values of which can be set or changed depending on the problem. The Atlas Coefficient works very well for segmentation problems where the shape of the segmentation is already known, such as organ segmentation. In cases such as lesion segmentation, this technique can be used to verify if the lesion falls in the correct organ. This can be used for detecting false positive lesions and also used to check if the lesion segmentation falls within or outside of an organ.

Thus, in embodiments where the radiological feature is a lesion or tumor, associated with an organ, the method <NUM> may comprise fitting shapes from an atlas of shapes to a second segment (in the segmentation) representing the organ. The best fitting shape from the atlas may then be used to determine which organ the lesion is associated with or forms part of. The relative locations of the lesion and organ may be established. Furthermore, false-positive labels of lesions associated with particular organs may be identified if a segment corresponding to the lesion is found not to be co-located with another segment corresponding to said particular organ. Co-located in this sense may mean that the segments overlap, at least partially.

Generally, to compute the atlas coefficient described above, an appropriate atlas needs to be selected. The altas selection process depends on patient demographic information such as Age, gender, race and country of origin. For a particular age group of a particular gender, the atlas may be selected to make the atlas coefficient more "context aware" and accurate. There may also be separate atlases for each country or place, as it has been observed that there are vast physiological changes across patients of various race and in different countries.

A "context aware" rule engine may be used to link patient specific metadata with performance of model. Here, patient specific information may be extracted such as gender, age group, geographical location, and/or organ in order to create an organ specific atlas. Variation from such atlas to the predicted segment of the model (as output by the model) can be computed using a distance matrix. The distance matrix (such as the KL divergence) may be used to select or reject the image for active learning.

<FIG> (top left hand) shows an example CT slice through a liver <NUM>. The middle image in <FIG> shows an accurate segmentation <NUM> of the liver, and the right hand image shows an inaccurate segmentation <NUM> of the liver. Histograms are also shown for the distribution of intensity values for the whole image slice <NUM>, the accurate liver segmentation <NUM> and the inaccurate liver segmentation <NUM>.

From the segmented regions, statistical features such as Radiomics as well as histogram-based features may be extracted. For example, in the case of a CT scan, the performance of (i.e., liver or lung) segmentation model can be determined from the data distribution using histogram analysis as shown in <FIG>. This allows determination of whether the model is not performing well. Organ specific Volumetric information helps to determine over or under segmentation behavior of a model. For example, in the example of <FIG>, the circled region of the histogram corresponding to the inaccurate segmentation <NUM> of the liver, indicates over-segmentation whereby image components not consistent with the expected distribution liver are included, creating a second peak (as circled) in the distribution of the segment, that is not present in the histogram of the correct segment <NUM>.

The Kullback-Leibler (KL) divergence, DKL, allows computation/quantification of the divergence between two distributions. The KL divergence can be used to quantify the divergence between a Histogram of the expected liver segmentation, "Q", e.g. from atlas, from a Histogram of the predicted liver segmentation, as output from the model, "P" as given in the equation below.

Also, the nth percentiles (e.g., <NUM>th, <NUM>th, <NUM>th, <NUM>th) may be calculated for the expected distribution (from the atlas) and compared with those of the distribution of the segment output from the model. <MAT> <MAT> Where, R is a rank, P is percentile and S is sorted array of N number of elements (items) in each histogram. RI and RF are integer and fractional part of R respectively.

As an example, for illustration purposes: Condition <NUM> :: Assume N=<NUM> and p=<NUM> This gives R=<NUM>, and thus the 100th element is selected from sorted array S. Condition <NUM> :: Assume N=<NUM> and p=<NUM> This gives R=<NUM>, so, integer part R_I=<NUM> and fractional part R_F=<NUM>.

In this way, volumetric information of the segment and atlases may also be compared. As such, the method <NUM> may comprise in step <NUM>, matching the radiological feature to an atlas of shapes of the radiological feature, according to demographic characteristics of the subject of the medical image. The method may then comprise obtaining expected volumetric information for the radiological feature from the matched shape from the atlas of shapes. CT and MR scans are generally 3D scans, and volumetric information in this sense means any information (features) extracted across such 3D scans. The expected volumetric information for the radiological feature may then be compared to volumetric information of the segment, e.g. using measures such as DKL and the percentiles described above, in order to determine whether there is under-segmentation or over-segmentation of the segment compared to the expected volumetric information for the radiological feature. For example, a histogram of the expected volumetric information from the atlas may be compared to a histogram of the volumetric information of the segment.

Turning now to other embodiments, in step <NUM>, alternatively or additionally to the geometric-based criteria herein, an edge criteria may be used to determine whether an edge region of the segment satisfies an edge criteria. The edge criteria may be a criteria defining well-defined edges of a segment. Thus, edges or edge regions may further be used to determine whether a segment is likely to be over-or under segmented and thus whether the unlabeled medical image should be sent for annotation in step <NUM>.

An edge region may be any region encompassing part of the edge of the segment. In some examples, an edge region may follow the outline of the surface of the segment, e.g. the edge region may be a band encompassing the edge region and following the outline of the edge region (e.g. with outer and inner surfaces parallel to the surface of the segment). An edge region may alternatively be defined relative to the center of gravity, or centroid of the segment, for example, e. g an edge region may be bounded by two concentric spherical surfaces centered on the centroid of the segment.

In some embodiments, the edge criteria is a threshold gradient between values of image components (e.g. pixels or voxels) within the segment and values of adjacent image components outside the segment, in order for the segment to be classed as having a well-defined edge.

As an example, if the sum of gradients in the original image along the edge/surface of each segment (in other words at the predicted mask contour), is less than a threshold gradient, then the segment can be labelled as being an uncertain segmentation, which should be sent for annotation in step <NUM>. For example, a gradient coefficient, G, can be calculated for a contour outlining the segment. Nx, Ny Nz are co-ordinates of the set of points on the periphery of the predicted contour/segment S. dl is the derivation of the segmented volume in the x, y and z directions and ∇(x,y,z) is gradient value, e.g. the change of Intensity at each point on the periphery of the segment: <MAT> <MAT>.

G may then be compared to a threshold G value in step <NUM>. A threshold value of the gradient coefficient, G value may be set according to the requirements of the training. Generally, lower G values indicate that the predicted segmentation is closer to the expected segmentation, and higher G values indicate the model is failing to predict the expected segmentation. Samples are therefore selected for active learning, if they are greater than a threshold G value. The value of G may be set by a human engineer, Higher values of G would result in fewer unlabeled images being selected compared to lower values of G; thus lowering the threshold, will increase the number of unlabeled medical images for annotation.

Turning now other embodiments, the edge criteria may be a criteria based on statistical features of clusters of image components in a border region (otherwise referred to herein as a penumbra region) encompassing the outer edge of the segment.

A border region may be defined in relation to the centroid or center of mass of the segment. For example, the center of mass of the segment may be determined. Then an outer surface of the border region may be defined, as a spherical surface centered on the center of mass and lying beyond the outer edge of the segment. An inner surface of the border region may be defined as a spherical surface centered on the center of mass and lying closer to the center of mass than the outer edge of the segment. The border region may then be defined as the volume between the outer surface and the inner surface.

The outer surface, may for example, be <NUM> percent further out than the outer edge of the segment and the inner surface may be <NUM> percent closer in than the outer edge of the segment. This is merely an example, however and the outer and inner surfaces may be at other distances, depending on factors such as the shape of the segment and/or the number of images required for the training.

The image components (voxels/pixels) may be clustered according to their values in the border region, e.g. using known clustering techniques. Each cluster may then be compared to the values at, or near to a central region of the segment to determine whether the cluster should be added to the segment or not. A central region may, for example, be a sphere centered on the centroid of the segment. The central region may encompass the center of gravity of the segment. The central region is closer to the center of the segment than the border region. Generally, the central region does not overlap the border region. A central region may also be referred to as a core region of the segment.

Thus, the method <NUM> may further comprise steps of: clustering image components in the border region into clusters; comparing the features of each cluster to image components in a central region of the segment, and selecting (<NUM>) the unlabeled medical image as training data that is to be annotated by the annotator as part of the active-learning machine learning process, based on the comparison between the clusters and the central region.

In other words step <NUM> may comprise selecting the unlabeled medical image as training data if the values of clusters of image components in the border region are statistically similar (comparable with) to the values of clusters of image components in a central region of the segment. The edge criteria may thus be whether image components in the border region are statistically similar to those in the central region (and should thus be added to the segment).

For example, for each cluster, a comparison may be made between the values of the image components in the cluster and the values of image components in the central region. A comparison may be made between statistical properties of values of the image components in the cluster and corresponding statistical properties of the values of image components in the central region. The comparison may be made based on radiomics features of each cluster. For example, the (Euclidean) distance between feature values in the central region and feature values in the cluster; intensity variations in the cluster, which may be measured, e.g. via the mean squared error (MSE); entropy of the cluster and/or a likelihood measure of the cluster compared to the central region may be determined. A score for each cluster may be determined based on one or more of the radiomics measures described above. The scores for each cluster can be combined, e.g. as a summation or multiplication, so as to determine an overall score describing whether the clusters in the border region are likely to be over/under segmented regions. Thus, the score may be used as an indication of whether the segmentation is a good segmentation or a poor segmentation and thus, whether the unlabeled image should be sent for annotation in step <NUM>.

This is described in more detail with respect to <FIG> which illustrates an example application of the method <NUM> to the segmentation of a 3D image of a liver <NUM> according to an embodiment herein. In this example, during the initial pass of the model, in step <NUM>, all image component (e.g. pixel/voxel) values in the image are scored to assess whether the given image component is part of the liver or not. The liver in this example may be referred to as the "intended segment" or Region of Interest. In step <NUM>, a border region <NUM> is defined as described above. A border region may be thought of as a "probability error zone". The border region is determined according to the following steps:.

Heat map zones can then be created within the border region to ascertain the probability of pixel values that are close to decision boundary. Heat maps are created by analyzing the pixel intensity variation in the penumbra region (e.g. similar pixel intensities get similar heat values in the heat map).

The heat maps are further analyzed to assess if there is over or under segmentation quantitatively. The scoring function is derived based on the heatmap that helps to arrive at a score for the border region. As described above, the idea here is to compare extracted features within the core of the segment (where the segmentation is most certain) to the penumbra region (region of the segment with high uncertainty). This comparison is done using distance metrics (i.e., Euclidean distance) describing the "distance" between each feature.

Where, the 'SN', 'SA' are the Gaussian cluster models of segmented (within) and cluster under questions, cases derived from Step. <NUM> and 'FV' is the feature vector.

The above described measures can be combined into a single score or probability that the cluster is part of the segment: <MAT>.

Further, the value of P(Border Region) can be used as contributing factor to decide if the region is segmented correctly, or whether the image should be flagged for annotation in step <NUM>. Higher values of P(Border Region) indicate good segmentation of the unlabeled medical image. To select cases where model is failing, a threshold value of P(Border Region) may be set. The method may comprise selecting unlabeled medical images for annotation that have a value of P(Border Region) below threshold value of P(Border Region).

Turning now to other embodiments, the skilled person will appreciate that a combination of the measures above may also be used. For example, a weighted combination of the Fractal Dimension, D, the gradient coefficient, G and the Sphericity, S as described above may be combined into a coefficient Gs ("Geometry estimation coefficient"). For example, by taking a weighted sum: <MAT>.

A threshold value of the Gs may be set. If a segment has a value of Gs below this value, then the medical image may be sent for annotation in step <NUM>.

The values of the weights w<NUM>-w<NUM> can be set dependent on type of segmentation. For example, w3 may be set to a higher level for lesion segmentation compared to organ segmentation (as the sphericity is more relevant to lesion segmentation than organ segmentation). w<NUM> may be set to a higher level for organ segmentation, as sharp edges may be more important for organ segmentation compared to FD or Sphericity. Another example score is given by the "retraining Confidence" measure: <MAT>.

Again, this is a weighted combination of some of the estimations described above, whereby the weights are set based on the particular segmentation problem. For e.g. Lesions, the Atlas weights may be set very low (even to <NUM>), whereas for organs the atlas weights may be set to larger values.

If the retraining confidence score is obtained for a plurality of unlabeled medical images (e.g. by repeating the method <NUM> on each), then the top n number of unlabeled medical images may be selected for re-training. The number n can be decided for example, based on the labeller's time availability and/or the cost of obtaining the ground truth annotations.

Turning back to the method <NUM>, in step <NUM>, the method comprises selecting the unlabeled medical image as training data that is to be annotated by the annotator as part of the active-learning machine learning process, if the segment does not satisfy the geometric criteria and/or the edge criteria.

For example, the unlabeled medical image may be selected as training data that is to be annotated and used as training data if the segment does not satisfy one or more of the criteria described above.

Then in step <NUM>, the method comprises sending the unlabeled medical image to the annotator for annotation. The skilled person will appreciate that these are examples, and that other geometric and/or edge criteria may be defined that are different to those listed above or described elsewhere in this disclosure. Furthermore, combinations of the criteria above may also be used.

Following step <NUM>, the selected unlabeled medical images are sent to the annotator (e.g. radiologist or other medical professional) in order to obtain a ground truth or "correct" segmentation for the unlabeled medical image. The medical image and the segmentation from the annotator are then used as training data with which to (re)train the model.

In embodiments where the probability error zone (border region) is determined, the annotator may be prompted to annotate only those regions containing clusters for which the system has low confidence, e.g. only in an error zone (they don't need to annotate completely, only a part of the mask has to be annotated and other part has to be verified). This is more efficient for the annotator.

The annotator checks and/or corrects mistake in the segmentation (or re-segments the unlabeled medical image entirely anew) and the resulting annotated medical image is added to the training data. The model can be retrained with both old training data and newly updated training data for better performance. The process of active learning can be performed on a timely basis, for eg. Once a week or once a month. This process helps keeps the model healthy in a product environment.

This is illustrated in <FIG> which shows an example according to some embodiments herein. <FIG> illustrates an embodiment of the method <NUM> described above. The embodiment in <FIG> may be performed, for example, by the apparatus <NUM> described above.

In <NUM> an unlabeled medical image is selected and in <NUM> this is provided as input to a model. At <NUM>, the model outputs (a prediction of) a segmentation of the unlabeled medical image. Steps <NUM>, <NUM> and <NUM> are performed as part of step <NUM> of the method <NUM> as described above, and it will be appreciated that the detail therein applies equally to steps <NUM>, <NUM> and <NUM>.

A segment of the segmentation is then assessed by one or more of a geometry estimation module <NUM>, a probability error zone module <NUM>, an atlas mapping module <NUM> and a context aware rule engine module <NUM>.

The geometry estimation module determines whether a geometric property of a segment of the segmentation satisfies a geometric criteria of a radiological feature represented by the segment. Various example geometric criteria were described in detail above (e.g. relating to sphericity, FD, etc) with respect to step <NUM> and the detail therein will be understood to apply equally to the Geometry Estimation module <NUM>.

The probability error zone module <NUM> determines whether an edge region of the segment satisfies an edge criteria by comparing statistical features of clusters of image components in a border region encompassing the outer edge of the segment to those in a region close to the center of the segment, in order to determine over- or under-segmentation. This as described above with respect to <FIG> and step <NUM> of the method <NUM> and the detail therein will be understood to apply equally to the probability error zone module <NUM>.

The atlas mapping module <NUM> fits shapes from an atlas of shapes of the anatomical feature to the segment and selects the unlabeled medical image as training data if the overlap between the best fitting shape from the atlas of shapes is less than a threshold overlap. The threshold overlap was described above with respect to step <NUM> and the detail therein will be understood to apply equally to the atlas mapping module <NUM>.

The context aware rule engine <NUM> uses patient specific metadata with performance of model. Here, patient specific information may be extracted such as gender, age group, geographical location and/or organ in order to create a specific atlas. Variation from such atlas to the predicted segment of the model (as output by the model) may be used as a distance matrix, as described above with respect to step <NUM>. The distance matrix was described above with respect to step <NUM> and the detail therein will be understood to apply equally to the context aware rule engine <NUM>.

If the segment does not satisfy the geometric criteria and/or the edge criteria then the unlabeled medical data is selected for annotation by an annotator <NUM> and subsequently used in re-training the model. However, if the segment satisfies the aforementioned criteria then there is no need for annotation or re-training based on the unlabeled medical image and the method ends at <NUM>.

In this way, there is a method and apparatus for selecting which unlabeled medical data should be send for annotation and used for further training of a model to segment medical images.

<FIG> shows an example graphical user interface (GUI) that can be used to train a model according to the method <NUM> herein. The GUI <NUM> may be displayed on a display of the apparatus <NUM> described above. A user interface as shown in <FIG> can be built to perform active learning using the method <NUM>. In this, a range of proposed retraining confidences/thresholds may be given <NUM>, specifying the values for the thresholds described herein against which medical images should be assessed and be selected for annotation. A mode of retraining can also be given that specifies for example, a minimum number of selected samples above which retraining should be started, or when the training should be performed (e.g. it can be overnight training). In the example of <FIG>, the GUI shows three windows, two for showing graphs <NUM>, <NUM>, <NUM> of metrics of model performance on the test set (which may be a validation set). An area <NUM> may be provided for displaying the selected unlabeled medical images for annotation (output in step <NUM> above). The dicom series may thus be openable in an inbuilt dicom viewer in <NUM>. After annotation, retraining can be started using UI only and same steps will be repeated for further iterations of active learning. Window <NUM> may display various statistical properties of the unlabeled medical images.

Thus, a GUI as illustrated in <FIG> may be used to perform and customize the method <NUM> as described above.

In another embodiment, there is provided a computer program product comprising a computer readable medium, the computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform the method or methods described herein.

Thus, it will be appreciated that the disclosure also applies to computer programs, particularly computer programs on or in a carrier, adapted to put embodiments into practice. The program may be in the form of a source code, an object code, a code intermediate source and an object code such as in a partially compiled form, or in any other form suitable for use in the implementation of the method according to the embodiments described herein.

Claim 1:
A computer implemented method for use in selecting training data for annotation by an annotator as part of an active-learning machine learning process, wherein the selected training data is for use in training a model to take as input a medical image and output a segmentation of the medical image, the method comprising:
providing (<NUM>) an unlabeled medical image as input to the model and obtaining a segmentation of the unlabeled medical image as output from the model;
the method characterized by:
determining (<NUM>) whether a geometric property of a segment of the segmentation satisfies one or more geometric criteria of a radiological feature represented by the segment; and/or
determining whether an edge region of the segment satisfies one or more edge criteria; and
selecting (<NUM>) the unlabeled medical image as training data that is to be annotated by the annotator as part of the active-learning machine learning process, if the segment does not satisfy the one or more geometric criteria and/or the one or more edge criteria.