Patent Description:
Medical images, for example, Chest X-rays (CXR), are one of the first choices for diagnosis and are fundamental to many patient pathways, revealing some unsuspected pathologic alterations. Generally, the medical images are reviewed by trained radiologists and the reviewing heavily depends on the experience and expertise of the radiologists. Presently, the radiologists face various challenges with the complexity of interpretation of the medical images.

Some of the challenges faced by the radiologists may be attributed to presence of anatomical noise in the images caused by superimposition of thoracic structures, similarity in radiographic appearances of some diseases, and subtlety of some pathologies which are indistinguishable. Other challenges may be in terms of time, where each trained radiologist takes several minutes to review a single medical image and write a report. Also, many radiologists have to work over-time, which in turn may increase the chances of misdiagnosis due to exhaustion. Additionally, in a comparative study of casualty plain radiographs, there may be a significant disagreement of diagnosis, which constitutes <NUM>% - <NUM>% of the total reviews. Due to this, the presence of an abnormality may, more likely, give rise to observer variations. Such variations may be classified as perceptual or cognitive. Perceptual errors occur when the image features, though recorded, were not appreciated by a reviewer. Cognitive errors may lead to either false positive or false negative reporting. Thus, there is a necessity to automate the image analysis process.

However, the existing automated systems face numerous challenges in terms of learning and analysis. For example, to train an automated computer aided diagnosis system, a high quality and larger magnitude dataset may be needed. On the larger data sets, the labelling of images may be normal or abnormal and may not be very accurate since the labelling process is automated by reading of radiology reports by a Natural Language Processing (NLP) based algorithm. In doing so, an additional layer of error may be introduced into the ground truth. Also, most of the existing automated systems focus on extracting and generalizing global features like boundary, shape, edge properties of region of interests and does not focus on classifying the normal and abnormal images.

Thus, presently, there is a need for an automated image analysis system which addresses the issues discussed above and results in optimal assessment of the medical images.

The paper "<NPL>, proposes a system for lung nodule diagnosis using features extracted from a CT scan. The system integrates global and local features. The global features are a 3D adjustable local binary pattern and several basic geometric features. Local features are extracted using a 3D-CNN that receives the image of the long nodule as an input. The features are integrated using an autoencoder to give a final diagnosis.

The information disclosed in this background of the disclosure section is only for enhancement of understanding of the general background of the invention and should not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person skilled in the art.

Disclosed herein is a method for identifying abnormal images in a set of medical images for optimal assessment of the medical images. The term "abnormal images" refers to medical images that show any condition that is clinically diagnosed as not normal. For example, when an image shows presence of pneumonia, nodules etc. in a medical image, such image is an abnormal image and shows abnormality in one or more organs of a patient. The method comprises extracting a plurality of global features from each medical image of the set of medical images based on pretrained weights associated with each of the plurality of global features. The medical image is characterized by a baseline resolution. Further, the method comprises extracting a plurality of local features from each medical image by analyzing a predefined number of image patches corresponding to each medical image. The local features are extracted using a pre-determined weights that are determined using CNN. The predefined number of image patches are generated by obtaining a higher resolution image corresponding to each medical image and splitting each higher resolution image. Higher resolution refers to a resolution that is greater/higher than the baseline resolution of the medical image. High resolution image is generated by up-sampling the medical image or a part of the medical image. Upon extracting the plurality of local features, the method comprises determining an abnormality score for each medical image based on weights associated with a combined feature set obtained by concatenating the plurality of global features and the plurality of local features. The abnormality score is determined using a pretrained feature classifier. Finally, the method comprises identifying the medical image as an abnormal image when the abnormality score of the medical image is higher than a predefined first threshold score.

Further, the present disclosure relates to an automated assessment system for identifying abnormal images in a set of medical images for optimal assessment of the medical images. The automated system comprises a processor and a memory. The memory is communicatively coupled to the processor and stores processor-executable instructions, which on execution, cause the processor to extract a plurality of global features from each medical image of the set of medical images based on pretrained weights associated with each of the plurality of global features. Further, the instructions cause the processor to extract a plurality of local features from each medical image by analyzing a predefined number of image patches corresponding to each medical image. The predefined number of image patches are generated by obtaining a higher resolution image corresponding to each medical image and splitting each higher resolution image. Thereafter, the instructions cause the processor to determine an abnormality score for each medical image based on weights associated with a combined feature set obtained by concatenating the plurality of global features and the plurality of local features. The abnormality score is determined using a pretrained feature classifier. Finally, the instructions cause the processor to identify the medical image as an abnormal image when the abnormality score of the medical image is higher than a predefined first threshold score.

The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and, together with the description, explain the disclosed principles. The same numbers are used throughout the figures to reference like features and components. Some embodiments of system and/or methods in accordance with embodiments of the present subject matter are now described, by way of example only, and regarding the accompanying figures, in which:.

It should be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative systems embodying the principles of the present subject matter. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and executed by a computer or processor, whether such computer or processor is explicitly shown.

While the disclosure is susceptible to various modifications and alternative forms, specific embodiment thereof has been shown by way of example in the drawings and will be described in detail below.

The terms "comprises", "comprising", "includes", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a setup, device, or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus proceeded by "comprises. a" does not, without more constraints, preclude the existence of other elements or additional elements in the system or method.

In an embodiment, the method and automated assessment system of the present disclosure may be used in hospitals and medical institutions for more accurate diagnosis of patients using medical images such as, Chest X-Ray (CXR) of the patients. Additionally, the method and the automated assessment system of the present disclosure may be used for generating an optimized worklist assignment for expert reviewers/radiologists for aiding and improving diagnosis, and for optimally utilizing time of the expert reviewers/radiologists.

In an embodiment, the present disclosure focuses on improving binary classification performance during automated image analysis. The classification performance may be improved by considering local fine aspects in an image and with improved label consistency. Accordingly, the present disclosure discloses extracting image biomarkers from the medical images and then performing binary classification on the input medical images to classify them as 'normal' or an 'abnormal' image. The term "normal image" refers to images that show absence of a clinical medical condition. The term "abnormal image" refers to a medical image that show any condition that is clinically diagnosed as not normal. For example, when an image shows presence of pneumonia, nodules etc. in a medical image, such image is an abnormal image and shows abnormality in one or more organs of a patient. Additionally, the present disclosure uses a selective sampler for cleaning of noisy labels in the input medical images during training of an image classification unit in the automated assessment system. This helps in further enhancing the classification accuracy and performance.

In an embodiment, the present disclosure proposes a principled method for worklist assignment, where only the difficult and/or high priority diagnosis cases may be assigned to the expert/experienced radiologists. This ensures optimal utilization of the highly experienced and skilled radiologists' time.

In an embodiment, the worklist assignment for ensuring the optimal utilization of the expert time may be dependent on the accuracy of the automated assessment system itself. The present disclosure addresses the above issue by extracting local features from a patch of a high-resolution image and combining it with the global features of the image extracted from a low-resolution image. By doing so, both the high-level edge information, as well as the minute details are captured and used for analysis, which improves the performance and reduces false diagnosis and reporting of the medical images. That is, the accurate classification of the images also helps in reducing number of false negatives (that is, an abnormal image getting classified as a normal image) during the automated assessment of the medical images.

Moreover, the existing deep-learning analysis models, which analyze solely based on the high-resolution image samples, may have memory constraints and experience challenges due to higher image size of the high-resolution images used for the analysis. Also, each feature that needs to be reviewed from the images must be exhaustively framed and labelled by an expert at the training stage, which is a very challenging task for the expert. On the other hand, even with an exhaustive feature set, using only the lower-resolution images may provide inaccurate results since the finest details on the images may be missed out during the analysis. The present disclosure aims to address the above issues by using a combination of the global high-resolution images and the corresponding down-scaled lower resolution images. The lower resolution refers to resolution that is lower than the baseline resolution. Lower resolution images are generated by down-sampling the medical images.

In an embodiment, the present disclosure additionally uses a two-step sampling mechanism for cleaning a subset of the noisy labels, in order to eliminate inaccuracy in the labelling, thereby improving consistency in the labelling of the images as normal or abnormal and improving the performance of the image classifier.

In an embodiment, the present disclosure also provides a provision for junior and/or inexperienced radiologists to view additional review areas and review information from the automated assessment system. Therefore, when in doubt, the inexperienced radiologist may request and use the additional review information before final reporting and diagnosis.

In the following detailed description of the embodiments of the disclosure, reference is made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration specific embodiments in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

<FIG> illustrates an exemplary architecture for identifying abnormal images <NUM> in a set of medical images <NUM> for an optimal assessment of the medical images <NUM> in accordance with some embodiments of the present disclosure.

The environment <NUM> may include, without limitation, an automated assessment system <NUM>. Further, the automated assessment system <NUM> may be associated with a pre-trained feature extraction model <NUM> and a pre-trained feature classifier <NUM>. In an embodiment, the automated assessment system <NUM> may be a computing device including, without limitation, a desktop computer, a laptop, a Personal Digital Assistant, smartphone or any other device capable of performing the functions in accordance with embodiments of the present disclosure for identifying abnormal images <NUM> from a set of medical images <NUM> (alternatively referred as medical images <NUM>). In an implementation, the automated assessment system <NUM> may be deployed in hospitals and/or medical institutions for aiding in diagnosis and reporting of the medical images <NUM>. In an implementation, the pre-trained feature extraction model <NUM> and the pre-trained feature classifier <NUM> may be configured and stored within the automated assessment system <NUM>.

In an embodiment, the automated assessment system <NUM> may be configured to receive the medical images <NUM> from an image source, which need to be analysed for identifying abnormal images <NUM>. In one embodiment, the image source may be a physical storage/repository or a storage server that stores a plurality of medical images to be analysed. In another embodiment, the image source may be an image capturing unit such as, without limiting to, an X-Ray machine, a Computed Tomography (CT) scanner and the like, from which a plurality of medical <NUM> images may be received in real-time. In an embodiment, the medical images <NUM> may be visual representations of an internal organ/part of a person/patient, used for clinical analysis and diagnosis of the person/patient. As an example, the medical images <NUM> may include, without limiting to, X-Ray images, CT scans, ultrasound images and the like. In an implementation, the image source, as disclosed above, may be configured as a part of the automated assessment system <NUM>. The resolution of the medical images <NUM> as received from the imaging devices or from storage devices is referred to as baseline resolution of the medical images <NUM>.

In an embodiment, upon receiving the set of medical images <NUM>, the automated assessment system <NUM> extracts a plurality of global features <NUM> from each medical image of the set of medical images <NUM> based on pretrained weights associated with each of the plurality of global features <NUM>. As an example, the global features <NUM> of the medical images <NUM> may include high-level morphological features such as, without limiting to, boundary of a Region of Interest (RoI) within the medical image, shape of the RoI and edges of the RoI or the medical image. In an embodiment, the plurality of global features <NUM> may be extracted from lower resolution images of the medical images <NUM>. The lower resolution images have a resolution lower than the baseline resolution of the medical images <NUM>.

In an embodiment, once the plurality of global features <NUM> are extracted, the automated assessment system <NUM> extracts a plurality of local features <NUM> from each medical image of the set of medical images <NUM>. The plurality of local features <NUM> are extracted by analysing a predefined number of image patches corresponding to the medical images <NUM>. In an embodiment, the predefined number of image patches may be generated by obtaining a higher resolution image corresponding to each medical image and then splitting the higher resolution images into the predefined number of smaller patches. The higher resolution images have a resolution higher than the baseline resolution of the medical images <NUM>. The predefined number, for example, may be <NUM>, which means that each medical image is split into <NUM> image patches and each of the <NUM> image patches are analysed for extracting the plurality of local features <NUM>. As an example, the plurality of local features <NUM> may include finer details of the medical images <NUM> and/or the RoI within the medical images <NUM> and may include, without limiting to, texture of the RoI, pattern of the RoI and the like.

In an embodiment, after extracting the plurality of global features <NUM> and the plurality of local features <NUM> from each of the medical images <NUM>, the automated assessment system <NUM> forms a combined feature set of the features by concatenating the plurality of global features <NUM> and the plurality of local features <NUM>. Further, the automated assessment system <NUM> determines an abnormality score <NUM> for each of the medical images <NUM> based on weights associated with the combined feature set. Here, the weights for the combined feature set are assigned using the pretrained feature classifier <NUM> associated with the automated assessment system <NUM>.

In an embodiment, the plurality of global features <NUM> and the plurality of local features <NUM> may be extracted using the pre-trained feature extraction model <NUM> associated with the automated assessment system <NUM>. The pretrained feature extraction model <NUM> may be trained with a plurality of train medical images and corresponding ground truth labels prior to real-time analysis of the medical images <NUM> by the automated assessment system <NUM>. The train medical images are the medical images used for training the automated assessment system <NUM>. The training, testing and implementation of the pre-trained feature extraction model <NUM> is described in further section of the description.

In an embodiment, after determining the abnormality score <NUM>, the automated assessment system <NUM> identifies one or more of the set of medical images <NUM> as an abnormal image <NUM> when the abnormality score <NUM> of the medical image is higher than a predefined first threshold score. Here, the predefined first threshold score may be set by an expert and/or a radiologist operating the automated assessment system <NUM>. As an example, the predefined first threshold score may be a value between <NUM> and <NUM>, such as <NUM>. In one embodiment, each medical image <NUM>, whose abnormality score <NUM> is higher than <NUM> may be identified and categorized as the abnormal image <NUM>. In an embodiment, the abnormal image <NUM> may be a medical image that requires an expert analysis and/or that needs to be analysed by an experienced radiologist for a correct diagnosis and reporting of the medical image. In other words, the abnormal image <NUM> may be a complex/difficult case for the automated assessment system <NUM>, such that the automated assessment system <NUM> finds it difficult to make an accurate diagnosis of the medical image.

In an embodiment, each medical image in the set of medical images <NUM> may be classified into two classes/categories of the abnormal images <NUM> as per the following arrangement. The first category of the abnormal images <NUM> may be a 'highly likely to be abnormal and hence critical' category. As an example, the medical images <NUM> whose abnormality score <NUM> is higher than <NUM> may be assigned to this category and flagged as a critical abnormal image. The second category of the abnormal images <NUM> may be the 'questionable images' and/or 'difficult cases', where it may be challenging for a radiologist to determine whether or not the case is normal, and hence are error prone, especially when being assessed by the junior radiologists. As an example, the medical images <NUM> having very small regions of abnormality may be grouped under the second category of the abnormal images <NUM> and tagged as a challenging abnormal image.

In an embodiment, by identifying and classifying the abnormal images <NUM> into the first and the second category of the abnormal images <NUM>, the automated analysis system <NUM> ensures that a most appropriate assessment strategy is followed during analysis of the abnormal images <NUM>. That is, the automated assessment system <NUM> intelligently decides when an expert involvement is required for correct analysis of the medical image. As a result, the automated assessment system <NUM> ensures optimal usage of time of the expert analysts and improves accuracy of the analysis as well.

<FIG> shows a detailed block diagram illustrating an automated assessment system <NUM> for identifying abnormal images <NUM> in a set of medical images <NUM> in accordance with some embodiments of the present disclosure.

In an implementation, the automated assessment system <NUM> may include an I/O interface <NUM>, a processor <NUM>, and a memory <NUM>. The I/O interface <NUM> may be communicatively interfaced with an image repository and/or an image source for receiving a set of medical images <NUM> to be analyzed by the automated assessment system <NUM>. The processor <NUM> may be configured to perform each of one or more functions of the automated assessment system <NUM>. In an embodiment, the memory <NUM> may be communicatively coupled to the processor <NUM>.

In some implementations, the image segmentation system <NUM> may include data <NUM> and modules <NUM> for performing various operations in accordance with the embodiments of the present disclosure. In an embodiment, the data <NUM> may be stored within the memory <NUM> and may include, without limiting to, global features <NUM>, local features <NUM>, abnormality score <NUM> and other data <NUM>.

In some embodiments, the data <NUM> may be stored within the memory <NUM> in the form of various data structures. Additionally, the data <NUM> may be organized using data models, such as relational or hierarchical data models. The other data <NUM> may store temporary data and temporary files, generated by the modules <NUM> while performing various functions of the automated assessment system <NUM>. As an example, the other data <NUM> may include, without limiting to, pretrained weights associated with the global features <NUM>, image patches, a combined feature set of the features and values of first threshold score and second threshold score.

In an embodiment, the data <NUM> may be processed by one or more modules <NUM> of the automated assessment system <NUM>. As used herein, the term module refers to an Application Specific Integrated Circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and memory that execute one or more software or firmware programs, a combinational logic circuit, a specialized hardware unit and/or other suitable components that provide the described functionality. In an embodiment, the other modules <NUM> may be used to perform various miscellaneous functionalities of the automated assessment system <NUM>. It will be appreciated that such modules <NUM> may be represented as a single module or a combination of different modules.

In one implementation, the one or more modules <NUM> may be stored as instructions executable by the processor <NUM>. In another implementation, each of the one or more modules <NUM> may be separate hardware units, communicatively coupled to the processor <NUM> for performing one or more functions of the automated assessment system <NUM>. The one or more modules <NUM> may include, without limiting to, an image acquisition module <NUM>, an extraction module <NUM>, a determining module <NUM>, an identification module <NUM>, a worklist assignment module <NUM> and other modules <NUM>.

In an embodiment, the image acquisition module <NUM> may be configured for acquiring/receiving the set of medical images <NUM> from the image source associated with the automated assessment system <NUM>.

In an embodiment, the extraction module <NUM> may be configured for extracting the plurality of global features <NUM> and the plurality of local features <NUM> from the set of medical images <NUM>. In an embodiment, the extraction module <NUM> may be interfaced with the pre-trained feature extraction model <NUM>, which aids in extracting the plurality of the global features <NUM> and the local features <NUM> from the medical images <NUM>.

In an embodiment, the determining module <NUM> may be configured for determining the abnormality score <NUM> for each of the medical images <NUM>. The determining module <NUM> may be interfaced with the pretrained feature classifier <NUM>, which determines weights for each feature in a combined feature set of the plurality of global features <NUM> and the plurality of local features <NUM>.

In an embodiment, the identification module <NUM> may be configured for identifying the abnormal images <NUM> among the set of medical images <NUM>. In an embodiment, the identification module <NUM> may compare the abnormality score <NUM> of each of the medical images <NUM> with the predefined first threshold score and may categorize the medical images <NUM> whose abnormality score <NUM> is higher than the predefined first threshold score as the abnormal image <NUM>.

In an embodiment, the worklist assignment module <NUM> may be configured for dynamically generating a worklist to be assigned for an expert radiologist for expert analysis of the abnormal images <NUM>. The worklist may comprise one or more medical images <NUM> identified as abnormal images <NUM>, along with the information related to the category of the abnormal images <NUM> to which each of the abnormal images <NUM> belong (that is, either critical abnormal image or a challenging abnormal image). In an embodiment the worklist may be continuously updated with the newly identified abnormal images <NUM>.

<FIG> shows a block diagram illustrating reporting of abnormal images <NUM> in accordance with an embodiment of the present disclosure.

In an embodiment, once an updated worklist <NUM> is generated at the automated assessment system <NUM>, the updated worklist <NUM> may be provided to a viewing and reporting unit <NUM>, associated with the automated assessment system <NUM>, for proper reporting of the abnormal images <NUM> comprised in the updated worklist <NUM>. In other words, an expert and/or an experienced reviewer may retrieve the updated worklist <NUM> from the automated assessment system <NUM> through the viewing and reporting unit <NUM>. In an embodiment, the viewing and reporting unit <NUM> may comprise, without limiting to, an intelligent viewing module <NUM> and a reporting module <NUM>. Using the intelligent viewing module <NUM>, the expert reviewer may get additional information related to the abnormality score <NUM> of the abnormal images <NUM> and specially highlighted review areas within the abnormal images <NUM>. In an embodiment, the additional information may be displayed on a display screen/user interface associated with the automated assessment system <NUM>. Here, the expert reviewer may perform various operations such as, without limitation, zooming in/out of the highlighted review areas, changing views/angle of the review areas and the like, for an easy review of the abnormal image <NUM>.

With the additional information, the expert reviewer may instantly focus on the highlighted review areas for analysing the abnormal images <NUM>. Once the analysis is complete, the expert reviewer may generate a report for each of the abnormal images <NUM> using the reporting module <NUM>. Further, the generated reports <NUM> may be stored on a storage server <NUM>.

In an implementation, the storage server may be communicatively interfaced with the automated assessment system <NUM>. The automated assessment system <NUM> may refer to the expert reports <NUM> stored on the storage server <NUM> for learning and improving decisions on identifying the abnormal images <NUM> from the set of medical images <NUM>. Thus, the expert reports <NUM> stored on the storage server <NUM> may form a part of the training data set used for training the automated assessment system <NUM>.

In an embodiment, the automated assessment system <NUM> may be trained in two stages. The first stage training may be before starting analysis of the medical images <NUM>, which is performed with the help of train medical images and associated ground truth labels. The second stage training may be corrective in nature and may be performed after the automated assessment system <NUM> has analysed a number of train medical images. For example, the second stage training may be performed based on expert comments and inputs on the analysis reports <NUM> of the automated assessment system <NUM>, received from the expert reviewers.

<FIG> shows a flowchart illustrating a method of preparing a clean training data set of the train medical images, for training the automated assessment system <NUM> in accordance with some embodiments of the present disclosure.

At block <NUM>, the automated assessment system <NUM> may receive a set of train medical images (I) <NUM> and the corresponding ground truth labels (G) from an image source associated with the automated assessment system <NUM>. The ground truth labels for the train medical images (I) may be assigned by the expert reviewers, before the train medical images (I) are sent to the automated assessment system <NUM>. In an embodiment, in order to enhance quality of the training data set and to increase accuracy of the automated assessment system <NUM>, a sampling strategy is applied on the set of medical images (I) for eliminating noisy labels in the set of train medical images (I). In other words, the sampling strategy helps in determining goodness of the ground truth labels associated with the set of medical images (I).

In an embodiment, as shown in block <NUM>, the sampling strategy may be applied on a selected subset (IM) of the set of train medical images (I). Further, at block <NUM>, the ground truth labels (GM) for the images in the subset (IM) may be determined, for example, by using a Natural Language Processing (NLP) technique configured in the automated assessment system <NUM>. Further, at block <NUM>, the ground truth labels (RM) for the subset of images (IM) may be obtained by manual analysis of the subset of images (IM) by expert radiologists. That is, for the same subset of images (IM), two different sets of ground truth labels (GM) and (RM) may be obtained. Thereafter, as shown in block <NUM>, the ground truth labels (GM) may be compared with the corresponding ground truth labels (RM) to determine a match between the ground truth labels (GM) and (RM). That is, a one-to-one comparison between the ground truth labels (GM) and (IM) for the corresponding images may be drawn to determine a consensus between the ground labels obtained from the automated NLP analysis and the manual expert analysis.

In an embodiment, the consensus may be defined as the total number of matches in the labels between (GM) and (RM) to the total number of images in the subset of images (IM). If the consensus ratio is greater than a threshold 'T', which may be defined by the user of the automated assessment system <NUM>, for a given class 'c', then set of all images (from I) of the class 'c' may be said to be in consensus. Subsequently, at block <NUM>, all the images (INS), from all the classes having consensus, and the corresponding ground truth labels (G NS), whose consensus match, may be extracted and stored in the clean training dataset (D').

On the other hand, for the remaining images (IS), from all the classes of 'I' that do not have consensus, and the corresponding ground truth labels (Gs), predictive ground truth labels (Ps) may be determined using a sampling classifier, as shown in block <NUM>. Thereafter, at block <NUM>, the predictions (Ps) may be compared with the corresponding ground truth labels (Gs) for determining a consensus between the labels (Ps) and (Gs). Further, the images whose predicted labels (Ps) match with the corresponding ground truth labels (Gs), may be selected and stored as a part of the training dataset (D'), as shown in block <NUM>. Alternatively, the images whose ground truth labels do not match and/or not in consensus, may be rejected/discarded and not used for training the automated assessment system <NUM>, as shown in block <NUM>. Finally, at block <NUM>, the clean training dataset (D') may be prepared using the images and the ground truth labels selected at blocks <NUM> and <NUM>.

<FIG> shows a flowchart illustrating a method of extracting the plurality of local features <NUM> and the plurality of global features <NUM> for training the pre-trained feature extraction model <NUM> in accordance with some embodiments of the present disclosure.

In an embodiment, during training of the feature extraction model <NUM>, the images and the corresponding ground truth labels stored in the clean training dataset (D') may be obtained, as shown in block <NUM> of <FIG>. Subsequently, the plurality of local features <NUM> and the plurality of global features <NUM> are extracted by performing the feature extraction methods shown in blocks <NUM> and <NUM> respectively. During extraction of the local features <NUM>, high resolution images corresponding to each image in the clean training dataset (D') may be obtained as shown in block 503A. Thereafter, at block 503B, each high-resolution image may be sub-divided into a predetermined number of image patches. The predetermined number may be defined by a user of the automated assessment system <NUM> based on application and complexity of the training dataset (D'). As an example, the predetermined number may be <NUM>, which means that each high-resolution image is divided into <NUM> image patches. In one embodiment, the patches are of equal size and/or resolution. In an embodiment, using the higher resolution image patches may be helpful in analysing finest details of the images and better training of the feature extraction model <NUM>. After obtaining the image patches, each image patch is analysed for extracting the plurality of local features <NUM> from each of the image patches, as shown in block 503C. Finally, the local features <NUM> from each image patch may be concatenated and/or combined to collectively obtain the local features <NUM> corresponding to each image in the training dataset (D').

In an embodiment, for extraction of the global features <NUM>, lower resolution images corresponding to each image in the training dataset (D') are obtained, as shown in block 505A. Further, at block 505B, a global feature network may be trained for assigning weights for each global feature extractor identified in each of the lower resolution images. Thereafter, the global feature extractor weights may be loaded onto the feature extractor model, as shown in block 505C. Finally, at block 505D, the plurality of global features <NUM> are extracted from each lower resolution image by running the images through the feature extraction model <NUM>.

Once the plurality of local features <NUM> and the plurality of global features <NUM> are extracted from each of the images in the training dataset (D'), each of the global features <NUM> and the local features <NUM> are concatenated to form a combined feature set of all the features extracted from the images, as shown in block <NUM>. Subsequently, a feature classifier <NUM> associated with the automated assessment system <NUM> may be trained with the combined feature set for determining abnormality scores <NUM> for the images, as shown in block <NUM>. Finally, the feature extractor weights, and the combined feature set may be stored in the automated assessment system <NUM> for future training purposes.

In an embodiment, the global feature network used for assigning the weights may consist of a Deep Convolutional Neural Network (DCNN), such as DenseNet. The output of the last global average-pooling layer of the DCNN may be flattened into a <NUM>-Dimensional vector represented by a GF<NUM>. The global features are determined using a CNN network that is different from a CNN network used for determining local features. For example, a first CNN network can be used for determining the local features and a second CNN network can be used for determining the global features, wherein the first CNN network is different from the second CNN network. This allows for a freedom to choose usage of different resolutions for determining the global and local features. For example, for global features, low resolution medical images are used that allows to focus on broader or higher-level features such as texture of the ROI, pattern of the ROI, and the like. Similarly, for local features higher resolution medical images are used that allows to focus on finer details of the medical images <NUM>.

In an embodiment, the global network may be trained with the training dataset (D'), to optimize the classification of an image, given the global features <NUM> only. The parameters 'Wg' of the global branch may be optimized using a classification loss function, for example, the Binary Cross-Entropy (BCE) loss function. Further, the weights 'Wg' of the global feature extractor alone may be then stored as pretrained weights. The pretrained weights may be loaded to extract the global features <NUM>. The global features <NUM> may then be extracted for all the images in the training dataset (D') from the pretrained weights 'Wg'.

In an embodiment, the pre-trained feature classifier <NUM> may consist of a series of FC layers. The output of the final FC classifier layer may be followed by a sigmoid activation for classification. The probability score for a given image I, for a class 'a' (i.e. normal or abnormal) may be represented as 'pf(a~| [GFI, LFI])'. Here, 'LFI' represents the <NUM>-dimensional vector obtained by flattening the features, for example, the radiomic features corresponding to the plurality of local features <NUM>. Further, the 'Wf' parameters of the fusion branch may be optimized using a classification loss function (L(Wf)), similar to the global feature network. Here, L(Wf) may represent the Binary Cross-Entropy (BCE) loss of fusion classifier, following which the weights 'Wf' may be saved.

<FIG> shows a flowchart illustrating a method of preparing and assigning a worklist for expert analysis of the abnormal images <NUM> in accordance with some embodiments of the present disclosure.

At block <NUM>, the abnormality scores <NUM> assigned to each of the medical images <NUM> may be obtained. Further, at block <NUM>, the abnormality score 'S' is compared with a predefined first threshold score T1. The value of the predefined first threshold score T1 may be defined by the user of the automated assessment system <NUM> and may be dynamically changed based on application and complexity of the medical images <NUM>. In an embodiment, if the abnormality score 'S' of the medical image is less than the predefined first threshold score T1, then the medical image may be analysed using a regular analysis strategy using the automated assessment system <NUM>, as shown in block <NUM>. On the other hand, if the abnormality score 'S' is more than the predefined first threshold score T1, then the image may be exempted from the regular analysis process and the abnormality score 'S' may be compared against a predefined second threshold score T2, as shown in block <NUM>. In an embodiment, the predefined second threshold score T2 may also be user defined and may be a value higher than the predefined first threshold score T1. Thus, at block <NUM>, it is determined whether the abnormality score 'S' is between the two borderline scores T1 and T2. In an embodiment, values of the predefined first threshold score and the predefined second threshold score may be stored as a part of the worklist assignment configurations, as shown in block <NUM>.

In an embodiment, if the abnormality score 'S' is between the threshold scores T1 and T2, then the medical image may be assigned to a highly skilled expert for analysis, as shown in block <NUM>. Alternatively, if the abnormality score 'S' is outside the range T1 and T2 and/or beyond the value of T2, then the corresponding medical image may be marked with a higher priority and assigned to the highly skilled expert for review, as shown in block <NUM>. In an embodiment, all the images selected for assigning to the skilled experts may be updated in the worklist.

<FIG> shows a flowchart illustrating a method of creating an analysis report based on expert analysis of the medical images <NUM> in accordance with some embodiments of the present disclosure.

Once the abnormal images <NUM> are identified and updated in the worklist, the updated worklist <NUM> may be provided to the expert reviewers for their review. Upon receiving the updated worklist <NUM>, the expert analyst may obtain the medical images <NUM> corresponding to the entries in the worklist for manually reviewing the medical images <NUM>, as shown in block <NUM>. Further, the expert analyst may manually review the medical images <NUM> for identifying one or more pathological abnormalities from the medical images <NUM>. In an embodiment, if the image is too complex and/or not easily analyzable, then the expert analyst may request for additional information related to the image, as shown in block <NUM>. As an example, the additional information may include, without limiting to, abnormality score <NUM> of the image, highlighted review areas in the image and the like, as indicated in block <NUM>. Alternatively, if the image is easily analyzable and/or the expert analyst does not require any additional information, the expert analyst may review and report the image without seeking the additional information, as shown in block <NUM>.

In an embodiment, when the expert analyst requests the additional information, the requested additional information may be retrieved from the memory of the automated assessment system <NUM>, through a query <NUM> sent to the memory. In response to the query <NUM>, the requested additional information comprising the abnormality score <NUM> and the highlighted review areas <NUM> may be provided to the expert analyst, as shown in block <NUM>. Thereafter, at block <NUM>, the expert analyst may manually assess the image using the additional information. Finally, a report comprising review results from the expert analyst may be prepared in the form an expert report.

In an embodiment, in order to highlight the review areas causing the abnormality score <NUM>, a guided grad-CAM may be applied on the last layer of the global feature extractor. A threshold of Tv may be used to generate a binary masked image of the guided grad-CAM output. From the binary mask image, a bounding box may be created for each closed region, the coordinates of which, along with the score of abnormality, may be stored back in the memory of the automated assessment system <NUM>, for future retrieval and viewing. As an example, value of the masking threshold Tv may be set to <NUM>.

<FIG> shows a flowchart illustrating a method of identifying abnormal images <NUM> in a set of medical images <NUM> for optimal assessment of the medical images <NUM> in accordance with some embodiments of the present disclosure.

As illustrated in <FIG>, the method <NUM> includes one or more blocks illustrating a method for identifying abnormal images <NUM> in a set of medical images <NUM> for optimal assessment of the medical images <NUM> using an automated assessment system <NUM>, illustrated in <FIG>. The method <NUM> may be described in the general context of computer executable instructions. Generally, computer executable instructions can include routines, programs, objects, components, data structures, procedures, modules, and functions, which perform specific functions or implement specific abstract data types.

The order in which the method <NUM> is described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the method. Furthermore, the method can be implemented in any suitable hardware, software, firmware, or combination thereof.

At block <NUM>, the method <NUM> comprises extracting a plurality of global features <NUM> from each medical image of the set of medical images <NUM> based on pretrained weights associated with each of the plurality of global features <NUM>. In an embodiment, the plurality of global features <NUM> and the plurality of local features <NUM> may be extracted using a pre-trained feature extraction model <NUM>. In an embodiment, the pre-trained feature extraction model <NUM> may be trained by obtaining a plurality of train medical images and extracting ground truth labels corresponding to each of the plurality of the train medical images. Here, the ground truth labels may comprise one or more system-assigned labels and corresponding one or more expert-assigned labels. Once the ground truth labels are extracted the one or more system-assigned labels may be compared with the corresponding one or more expert-assigned labels to determine one or more matching ground truth labels. Thereafter, the one or more matching ground truth labels and corresponding train medical images may be used for training the feature extraction model <NUM>. In an embodiment, the plurality of global features <NUM> is extracted from a lower resolution image corresponding to the medical image.

At block <NUM>, the method <NUM> comprises extracting a plurality of local features <NUM> from each medical image by analysing a predefined number of image patches corresponding to each medical image. The predefined number of image patches is generated by obtaining a higher resolution image corresponding to each medical image and splitting each higher resolution image. In an embodiment, the resolution of the medical images used for extracting the plurality of local features <NUM> is more than the resolution of the medical images used for extracting the plurality of global features <NUM>.

At block <NUM>, the method <NUM> comprises determining an abnormality score <NUM> for each medical image based on weights associated with a combined feature set obtained by concatenating the plurality of global features <NUM> and the plurality of local features <NUM>. The abnormality score <NUM> is determined using a pretrained feature classifier <NUM>. In an embodiment, determining the abnormality score <NUM> for each medical image may include comparing the weights associated with the combined feature set with corresponding stored weights associated with the pretrained feature classifier <NUM>.

In an embodiment, upon identifying the abnormality scores <NUM> of the medical images <NUM>, the method <NUM>, at block <NUM>, comprises classifying the medical images <NUM> as normal images or abnormal images <NUM>. In an embodiment, a medical image <NUM> is classified as the normal image when the abnormality score <NUM> of the medical image <NUM> is less than the predefined first threshold score (T1). In such a case, the normal image may be taken through the default analysis strategy used in the hospital/organization. In an embodiment, a medical image <NUM> is classified as an abnormal image <NUM> when the abnormality score <NUM> of the medical image <NUM> is greater than the predefined first threshold score (T1).

In an embodiment, the abnormal images <NUM> may be further classified as a critical abnormal image and a challenging abnormal image. As an example, an abnormal image <NUM> may be tagged and/or marked as a critical abnormal image when the abnormality score <NUM> is more than a predefined second threshold score (T2) and may be sent to an expert analyst for immediate review and/or assessment of the image. However, if the abnormality score <NUM> is more than the predefined first threshold score (T1) but less than the predefined second threshold score (T2), then the abnormal image <NUM> may be marked as a challenging abnormal image and routed to the expert analyst for assessment. In other words, an abnormal image <NUM> classified as the critical abnormal image may be given a higher priority than the abnormal image <NUM> classified as the challenging abnormal image and sent to an immediate review by the expert analyst. In an embodiment, the predefined first and the second threshold scores may be set by the expert analyst and/or a radiologist.

<FIG> illustrates a block diagram of an exemplary computer system <NUM> for implementing embodiments consistent with the present disclosure. In an embodiment, the computer system <NUM> may be an automated assessment system <NUM>, which is used for identifying abnormal images <NUM> in a set of medical images <NUM>. The computer system <NUM> may include a central processing unit (Central Processing Unit (CPU), Graphics Processing Unit (GPU) or "processor") <NUM>. The processor <NUM> may comprise at least one data processor for executing program components for executing user- or system-generated business processes. A user may include a person, a radiologist, a medical practitioner and/or a technologist, a person reviewing the set of medical images <NUM> and the like. The processor <NUM> may include specialized processing units such as integrated system (bus) controllers, memory management control units, floating point units, graphics processing units, digital signal processing units, etc..

The processor <NUM> may be disposed in communication with one or more input/output (I/O) devices (<NUM> and <NUM>) via I/O interface <NUM>. The I/O interface <NUM> may employ communication protocols/methods such as, without limitation, audio, analog, digital, stereo, IEEE-<NUM>, serial bus, Universal Serial Bus (USB), infrared, PS/<NUM>, BNC, coaxial, component, composite, Digital Visual Interface (DVI), high-definition multimedia interface (HDMI), Radio Frequency (RF) antennas, S-Video, Video Graphics Array (VGA), IEEE <NUM>. n /b/g/n/x, Bluetooth, cellular (e.g., Code-Division Multiple Access (CDMA), High-Speed Packet Access (HSPA+), Global System For Mobile Communications (GSM), Long-Term Evolution (LTE) or the like), etc. Using the I/O interface <NUM>, the computer system <NUM> may communicate with one or more I/O devices <NUM> and <NUM>.

In some embodiments, the processor <NUM> may be disposed in communication with a communication network <NUM> via a network interface <NUM>. The network interface <NUM> may communicate with the communication network <NUM>. The network interface <NUM> may employ connection protocols including, without limitation, direct connect, Ethernet (e.g., twisted pair <NUM>/<NUM>/<NUM> Base T), Transmission Control Protocol/Internet Protocol (TCP/IP), token ring, IEEE <NUM>. 11a/b/g/n/x, etc. Using the network interface <NUM> and the communication network <NUM>, the computer system <NUM> may communicate with an image source for receiving set of medical images <NUM>. Further, the communication network <NUM> may be used to communicate with a pre-trained feature extraction model <NUM> and a pre-trained feature classifier <NUM>.

The communication network <NUM> can be implemented as one of the several types of networks, such as intranet or Local Area Network (LAN) and such within the organization. The communication network <NUM> may either be a dedicated network or a shared network, which represents an association of several types of networks that use a variety of protocols, for example, Hypertext Transfer Protocol (HTTP), Transmission Control Protocol/Internet Protocol (TCP/IP), Wireless Application Protocol (WAP), etc., to communicate with each other. Further, the communication network <NUM> may include a variety of network devices, including routers, bridges, servers, computing devices, storage devices, etc..

In some embodiments, the processor <NUM> may be disposed in communication with a memory <NUM> (for example, RAM <NUM> and ROM <NUM> as shown in <FIG>) via a storage interface <NUM>. The storage interface <NUM> may connect to memory <NUM> including, without limitation, memory drives, removable disc drives, etc., employing connection protocols such as Serial Advanced Technology Attachment (SATA), Integrated Drive Electronics (IDE), IEEE-<NUM>, Universal Serial Bus (USB), fiber channel, Small Computer Systems Interface (SCSI), etc. The memory drives may further include a drum, magnetic disc drive, magneto-optical drive, optical drive, Redundant Array of Independent Discs (RAID), solid-state memory devices, solid-state drives, etc..

The memory <NUM> may store a collection of program or database components, including, without limitation, user/application <NUM>, an operating system <NUM>, a web browser <NUM>, and the like. In some embodiments, computer system <NUM> may store user/application data <NUM>, such as the data, variables, records, and the like as described in this disclosure. Such databases may be implemented as fault-tolerant, relational, scalable, secure databases such as Oracle or Sybase.

The operating system <NUM> may facilitate resource management and operation of the computer system <NUM>. Examples of operating systems include, without limitation, Apple Macintosh OS X, UNIX, Unix-like system distributions (e.g., Berkeley Software Distribution (BSD), FreeBSD, Net BSD, Open BSD, etc.), Linux distributions (e.g., Red Hat, Ubuntu, K-Ubuntu, etc.), International Business Machines (IBM) OS/<NUM>, Microsoft Windows (XP, Vista/<NUM>/<NUM>, etc.), Apple iOS, Google Android, Blackberry Operating System (OS), or the like.

A user interface may facilitate display, execution, interaction, manipulation, or operation of program components through textual or graphical facilities. For example, user interfaces may provide computer interaction interface elements on a display system operatively connected to the computer system <NUM>, such as cursors, icons, check boxes, menus, windows, widgets, etc. Graphical User Interfaces (GUIs) may be employed, including, without limitation, Apple Macintosh operating systems' Aqua, IBM OS/<NUM>, Microsoft Windows (e.g., Aero, Metro, etc.), Unix X-Windows, web interface libraries (e.g., ActiveX, Java, JavaScript, AJAX, HTML, Adobe Flash, etc.), or the like.

Furthermore, one or more computer-readable storage media may be utilized in implementing embodiments consistent with the present invention. The term "computer-readable medium" should be understood to include tangible items and exclude carrier waves and transient signals, i.e., non-transitory. Examples include Random Access Memory (RAM), Read-Only Memory (ROM), volatile memory, nonvolatile memory, hard drives, Compact Disc (CD) ROMs, Digital Video Disc (DVDs), flash drives, disks, and any other known physical storage media.

The terms "an embodiment", "embodiment", "embodiments", "the embodiment", "the embodiments", "one or more embodiments", "some embodiments", and "one embodiment" mean "one or more (but not all) embodiments of the invention(s)" unless expressly specified otherwise.

The terms "including", "comprising", "having" and variations thereof mean "including but not limited to", unless expressly specified otherwise. The enumerated listing of items does not imply that any or all the items are mutually exclusive, unless expressly specified otherwise.

The terms "a", "an" and "the" mean "one or more", unless expressly specified otherwise. A description of an embodiment with several components in communication with each other does not imply that all such components are required. On the contrary, a variety of optional components are described to illustrate the wide variety of possible embodiments of the invention.

When a single device or article is described herein, it will be clear that more than one device/article (whether they cooperate) may be used in place of a single device/article. Similarly, where more than one device or article is described herein (whether they cooperate), it will be clear that a single device/article may be used in place of the more than one device or article or a different number of devices/articles may be used instead of the shown number of devices or programs. The functionality and/or the features of a device may be alternatively embodied by one or more other devices which are not explicitly described as having such functionality/features. Thus, other embodiments of the invention need not include the device itself.

Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope of the invention be limited not by this detailed description, but rather by any claims that issue on an application based here on. Accordingly, the embodiments of the present invention are intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.

While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the scope being indicated by the following claims.

Claim 1:
A computer-implemented method (<NUM>) of identifying abnormal images in a set of medical images for optimal assessment of the medical images, the method comprising:
extracting (<NUM>) a plurality of global features from each medical image of the set of medical images based on pretrained weights associated with each of the plurality of global features, wherein the global features are assigned the pretrained weights using a first Convolutional Neural Network;
extracting (<NUM>) a plurality of local features from each medical image by analysing a predefined number of image patches corresponding to each medical image, wherein the predefined number of image patches are generated by obtaining a higher resolution image corresponding to each medical image and splitting each higher resolution image into the predefined number of image patches, wherein the respective image patches are analysed to extract local features from the respective image patches, and the local features for the respective image patches are combined to obtain the plurality of local features, wherein a resolution of the medical image is less than the resolution of the higher resolution image;
determining (<NUM>) an abnormality score for each medical image based on weights associated with a combined feature set obtained by concatenating the plurality of global features and the plurality of local features, wherein the abnormality score is determined using a pretrained feature classifier; and
identifying the medical image as an abnormal image when the abnormality score of the medical image is higher than a predefined first threshold score,
wherein the first Convolutional Neural Network is different from a second Convolutional Neural Network used to extract the local features.