Patent Description:
Unstructured medical notes pose challenges in many situations:.

In use case (<NUM>), a clinician or researcher seeks to identify from a large hospital archive the records of patients afflicted with a particular condition, say, or patients who received a particular form of treatment. This might be for recruiting subjects to participate in a clinical trial based on eligibility criteria, or for educational/training purposes. This task is incredibly painstaking without a database of structured clinical variables against which to query. Simple string matching of the text notes can have very poor specificity. And later, medical personnel must comb through each record to determine if a particular condition was affirmed.

In use case (<NUM>), a healthcare provider seeks to absorb a patient's history at first encounter. A patient may have been in the healthcare system for quite a while; to effectively treat the patient they must gain an understanding of prior encounters based on scattered notes written by numerous providers (hospitalists, radiologists and other specialists. ) This can be time-consuming and error-prone when there are many documents to review. An automated system that surfaces concrete clinical variables based on prior documents and draws the reader's attention to relevant portions of a longer note could save valuable time in the clinic.

Research in medical informatics often seeks to replicate the judgments of a human expert through automated systems. Such efforts often rely on retrospective data comprising many instances of patient information (e.g. lab tests, medical images) paired with diagnoses, treatment decisions, or survival data. In the paradigm of supervised learning, the patient data are the "inputs", while the diagnoses, outcomes, or treatment recommendations are the "outputs". The modeling task is then to learn to predict this information given access to the patient record from an earlier point in time. For instance, an application of computer vision might seek to automatically determine what is causing a patient's chest pain by inspecting an X-ray image of the patient's chest. Such a system would be trained by leveraging previously captured images and the corresponding opinions of their radiologists.

For training and evaluating these models, it is important to have highly structured outputs on which clinically significant metrics can be defined. An example of a highly structured output would be the code of the medication that was subsequently prescribed. In many areas of medicine (e.g. radiology, pathology) the "inputs" to a diagnostic system are consistently archived as digital images, and are thus amenable to modern machine learning methods. Lamentably, historical diagnoses (our "outputs") are almost always sequestered among natural language rendered by the physician. This text is often unstructured and poorly standardized, and thus difficult to use in the context of machine learning. Recasting this information using a standardized schema or rubric requires a great deal of effort from medical personnel, and is thus a time-consuming and costly effort. <CIT> describes a deep learning model to detect disease from an image and annotate its contexts.

Aspects of the present disclosure are defined by the independent claims. Further aspects and preferred features are set out in the dependent claims.

This disclosure relates to a method of generating structured labels for free-text medical reports, such as doctor notes, and associating such labels with medical images, such as chest X-rays which are adjunct to the medical reports. This disclosure also relates to a computer vision system which classifies or generates structured labels from input image pixels alone. This computer vision model can assist in diagnosis or evaluation of medical images, for example chest X-rays.

In this document, the term "structured labels" refers to data according to a predetermined schema, or in other words a standardized format. The data is for conveying diagnostic or medical condition information for a particular sample (e.g., free-text report, associated image or image alone). One example of a structured label is a set of one or more binary values, for example positive (<NUM>) or negative (<NUM>) for a medical condition or diagnosis, such as pneumothorax (collapsed lung), pulmonary embolism, misplaced gastric feeding tube, etc. in the context of a chest X-ray or associated medical report. Alternatively, the structured labels could be a schema or format in the form of a set of one or more binary values for the assignment of a diagnostic billing code, e.g., "<NUM>" meaning that a particular diagnostic billing code for pulmonary embolism should be assigned, "<NUM>" meaning that such code should not be assigned. The particular schema will depend on the application, but in the example of chest X-rays and associated medical reports, it can consist of a set of binary values to indicate presence or absence of one or more of the following conditions: airspace opacity (including atelactasis and consolidation), pulmonary edema, pleural effusion, pneumothorax, cardiomegaly, nodule or mass, misplaced nasogastric tube, misplaced endotracheal tube, misplaced central venous catheter, and presence of chest tube. In addition, each of these conditions might have binary modifiers such as laterality (e.g., <NUM> for left, <NUM> one for right) or severity. The structured label may also include a severity term, in one possible example it could a binary modifier that may be <NUM>, <NUM> or <NUM> bits of information in the structured label in order to encode different values of severity. Or, as another example, the structured label could encode some rubric or schema for severity such some integer value, e.g., absent = <NUM>, mild = <NUM>, moderate = <NUM>, severe = <NUM>, or a severity scale from <NUM> to <NUM>.

For example, a structured label for a given medical report, or for a given medical image, could take the form of [<NUM>] where each bit in the label is associated with a positive or negative finding of a particular medical condition, billing code, or diagnosis, and in this example there are six different possible findings or billing codes for this example. The structured labels can be categorical as well as binary. For example, a model can produce a label in the set {absent, mild, moderate, severe} or some numerical equivalent thereof.

The method uses a natural language processor (NLP) in the form of a one-dimensional deep convolutional neural network trained on a curated corpus of medical reports with structured labels assigned by medical specialists. This NLP learns to read free-text medical reports and produce predicted structured labels for such reports. The NLP is validated on a set of reports and associated structured labels to which the model was not previously exposed.

The NLP is then applied to a corpus of medical reports (free-text) for which there are associated medical images such as chest X-rays but no clinical variables or structured labels of interest which are available. The output of the NLP is the structured label associated with each medical image.

The medical images with the structured labels assigned by the NLP are then used to train a computer vision model (for example, a deep convolutional neural network pattern recognizer) to assign or replicate the structured labels to medical images based on image pixels alone. The computer vision model thus functions essentially as a radiologist (or, more typically, as an intelligent assistant to a radiologist) producing a structured output or label for a medical image such as a chest X-ray, instead of a natural language or free-text report.

In one possible embodiment, the methodology includes a technique or algorithm, known as Integrated Gradients, implemented as a software module which assigns attribution to the words or phrases in the medical reports which contributed to the assignment of the structured label to the associated images(s). The methodology allows such attributions to be represented to a user for example by showing excerpts from the medical report with pertinent terms highlighted. This has tremendous applicability in the healthcare context where providers typically sift through long patient records to find information of interest. For example, if the NLP identifies that the medial image shows signs of a cancerous lesion, the relevant text in the associated report can be highlighted.

Further described herein is a system for processing medical text and associated medical images. The system includes (a) a computer memory storing a first corpus of curated free-text medical reports each of which having one or more structured labels assigned by a medical expert; (b) a natural language processor (NLP) configured as a deep convolutional neural network trained on the first corpus of curated free-text medical reports to learn to read additional free-text medical reports and produce predicted structured labels for such additional free-text medical reports; and (c) a computer memory storing a second corpus of free-text medical reports (and which are typically without structured labels) that are associated with medical images. The NLP is applied to such second corpus of free-text medical reports and responsively generates structured labels for the associated medical images. The system further includes (d) a computer vision model which is trained on the medical images and the structured labels generated by the NLP. The computer vision model operates (i.e., performs inference) to assign a structured label to a further input medical image, e.g. a chest X-ray for which there is no associated free-text medical report. In this manner, the computer vision model functions as an expert system to read medical images and generate structured labels, like a radiologist or as an assistant to a radiologist. The NLP and the computer vision model are typically implemented in special purpose computers configured with hardware and software for implementing deep learning models as is customary in the art.

In one embodiment the computer vision model and NLP are trained in an ensemble manner as will be apparent from the following description.

In one possible configuration, the system includes a module implementing an Integrated Gradients algorithm assigning attribution to input words in the free-text medical reports to the structured label generated by the NLP. The system may further include a workstation having a display for displaying both the medical images, free-text reports, and the attribution to elements in the report calculated by the Integrated Gradients algorithm.

As will be explained below, in one embodiment the medical images are chest X-rays. The computer vision model is trained to generate structured labels for a chest X-ray based on the image pixels alone, and without requiring an associated free-text medical report. The structured labels can for example be a series binary values indicating positive or negative for particular findings or diagnostic billing codes, optionally including laterality or severity, as explained above.

There is also provided herein a method for processing medical text and associated medical images comprising:.

Also described herein is a machine learning system comprising, in combination,.

The term "curated" is used to mean that a corpus of data includes at least some elements generated by human intelligence, such as the structured labels assigned by the medical expert during training.

In one aspect, the work described in this document relates to a methodology of developing a computer vision model, typically embodied in a computer system including memory and processor functions, and consisting of certain algorithms, which is capable of emulating a human expert's interpretation of medical images. To build such a model we follow the highly successful paradigm of supervised learning: use many examples of radiology images (the input) paired with a corresponding set of findings generated by a radiologist (the outputs) to train a deep learning computer vision model to replicate the specialist's diagnostic process. The goal is that if we train on enough data, from enough doctors, our model can perhaps surpass human performance in some domains.

<FIG> is an illustration of a machine learning process for generating structured labels for sets of medical images and associated free-text medical reports which will explain the motivation of this disclosure. The goal in this context is to generate a machine learning model <NUM> that can use as input (X) one or more medical images <NUM> and a free text report <NUM> associated with the medical images and produce an output (Y) in the form of a structured label <NUM>. For each input X<NUM>. XN the model <NUM> generates an output Y<NUM>. The model <NUM> is shown as having two components, natural language processing (NLP) applied to the text component and inference (pattern recognizer) which is applied to the medical images.

Lamentably, in most medical clinics, radiologists transmit their opinions back to the referring physician in the form of unstructured, natural language text reports of a few hundred words. Thus, while the inputs to our computer vision model-radiological images- are consistently archived in digital format and require little preprocessing, the outputs-i.e. the medical diagnoses, or findings, or equivalently structured labels which the computer vision model should replicate-are sequestered among unstructured, poorly standardized natural language.

Although frameworks exist for translating images directly into strings of text, our application demands a more structured output, i.e., the labels <NUM> of <FIG>, on which clinically significant metrics can be defined. One approach to this problem might be to recruit radiologists to revisit historical scans and annotate them using a structured form. However, to attempt to meet the data requirements of a state-of-the-art computer vision model in this fashion would be slow, costly and wasteful-considering all of these scans have already been interpreted. Hence, this disclosure provides for a more efficient and cost effective approach to generating the structured labels and in turn training a computer vision model.

An automated system that can extract structured data (labels <NUM> of <FIG>) from a free-text radiology report, as described in this document, unlocks troves of training data for our computer vision model and associated algorithms. Investing in such a system by collecting human-labeled data expressly for this purpose, rather than collecting fresh image-based labels for training, would be extremely high-leverage. The annotation task is several times faster per case and can be performed by less specialized people. Plus, once a natural language processing model is trained, it can be applied to an arbitrary number of cases, yielding practically endless training data for a computer vision model.

Accordingly, described herein is a method of generating structured labels for free-text medical reports, such as doctor notes, and associating such labels with medical images, such as chest X-rays which are adjunct to the medical reports. Referring now to <FIG>, the method uses a natural language processor (NLP) <NUM> in the form of a one-dimensional deep convolutional neural network which is trained on a curated corpus of medical reports <NUM> consisting of a multitude of individual free text reports <NUM> each of which is associated with respective structured labels <NUM> which were assigned by a medical specialists (i.e., human experts). This input by experts is shown at <NUM> and could take for example one or more medical experts using a workstation to review the reports <NUM> and using an interface of the workstation assigning one or more binary values to various diagnostic or medical conditions which in their judgment are present in the patient associated with the report in a form-type document, for example a label such as [<NUM>] where each bit is associating with positive or negative for a particular medical condition, diagnostic billing code, etc., as explained earlier. Additionally, the label may include bits or alphanumeric data to reflect laterality or severity as explained earlier.

This NLP <NUM> learns from the training corpus <NUM> to read free-text medical reports and produce predicted structured labels for such reports. For example, after this network has been trained it can be applied to a set <NUM> of free text reports and for each report 112A, 112B and 112C it generates a structured label 114A, 114B and 114C respectively. Additionally, this NLP <NUM> is validated on a set of reports and associated structured labels (<FIG>, <NUM>) to which the model was not previously exposed. Background information on machine learning methods for generating labels from text reports is described in <NPL>. The scientific literature includes various descriptions of convolutional neural networks for text classification which may be suitable for use in this context, for example <NPL>, and <NPL>, and <NPL>) and therefore a more detailed description of the NLP <NUM> is omitted for the sake of brevity.

Referring now to <FIG>, the NLP <NUM> trained in accordance with <FIG> is then applied to a corpus <NUM> of medical reports (free-text) <NUM>, each of which is associated with one or more medical images <NUM>, such as for example chest X-rays but for which no structured labels encoding clinical variables of interest are available. The NLP's output is a structured label <NUM> which is associated with each image.

For example, the NLP <NUM> may have as input a free text report <NUM> indicating the presence of a misplaced gastric feeding tube, and an associated chest X-ray showing such a condition. The network has been trained from <FIG> to generate a label for misplaced gastric feeding tube (e.g., a label such as [<NUM>] where the <NUM> indicates misplaced gastric feeding tube and the <NUM> indicate absence of other conditions), and this label [<NUM>] is assigned to the associated chest X-ray.

The generation of structured labels from free text reports and assigning them to associated medical images allows for the generation of a large body of medical images which have assigned structured labels automatically. We previously noted it would in theory be possible to recruit radiologists to revisit historical scans and annotate them using a structured form. However, to attempt to meet the data requirements of a state-of-the-art computer vision model in this fashion would be slow, costly and wasteful-considering all of these scans have already been interpreted. Instead, using the training procedure of <FIG> and the labeling procedure of <FIG>, we generate the structured labels for medical images automatically and without requiring vast amounts of time from trained radiologists.

In one embodiment, our methodology includes a technique, known as Integrated Gradients, which assigns attribution to the words or phrases in the medical reports which contributed to the assignment of the label to the associated image in <FIG>. In particular, we use a software module <NUM> implementing an Integrated Gradients algorithm which generates weight or attribution to particular words or strings of words in the medical reports which contributed significantly to the label generation. This method is conceptually similar to "attention mechanisms" in machine learning. The output of the software module <NUM> is shown as the attribution <NUM>. Our methodology allows such attributions to be represented to a user for example by showing excerpts from the medical report with pertinent terms highlighted, See <FIG> discussed below. This has tremendous applicability in the healthcare context where providers typically sift through long patient records to find information of interest. For example, if the model identifies that the medial image shows signs of a cancerous lesion, the relevant text in the associated report can be highlighted. Integrated Gradients has been described as having applications in object recognition, diabetic retinopathy prediction, question classification, and machine translation, but its application in this context (attribution of text elements to binary classification of medical images) is new.

The Integrated Gradients algorithm is described in the paper of <NPL>). The methodology will be described conceptually in <FIG> in the context of attribution of individual pixels in an image in a classification of the overall image, and it applies by analogy to individual words in a free-text medical report. Basically, as shown in <FIG>, an Integrated Gradients score IGi (or attribution weight or value) for each pixel i in the image is calculated over a uniform scaling (α) of the input image information content (spectrum of brightness in this example) from a baseline (zero information, every pixel black, α = <NUM>), to the full information in the input image (α = <NUM>), where IGi (score for each pixel) is given by equation (<NUM>) <MAT>.

In the context of a free text medical report, the report is a one dimensional string of words of length L, and each word is represented as a, for example, <NUM> dimensional vector x in semantic space. The dimensions of the semantic space encode information of semantics, co-occurrence statistics of the presence or frequency of a word with other words, and other information content. α = <NUM> means each word in the report has no semantic content and no meaning (or could be represented as a zero vector) and as α goes to <NUM> each word goes to its full semantic meaning.

A more general expression of the Integrated Gradients is <MAT> where the integrated gradient along the i th dimension for an input x and baseline x' is defined per equation (<NUM>). Here, ∂F(x)/ ∂xi is the gradient of F(x) along the i th dimension. Section <NUM> of the Sundararajan et al. paper explain the algorithm further. gradient with respect to each word vector is itself a <NUM>-dimensional word vector. Note that the number <NUM> is somewhat arbitrary, but not an uncommon choice. To get the final Integrated Gradients value for each word, the components are summed. Their sign is retained, so that net positive scores imply an attribution toward the positive class (with value <NUM>) while net negative scores imply an attribution toward the negative or "absent" class (with value <NUM>).

Just as the Integrated Gradients algorithm calculates the attribution value IGi for each pixel in the image example of <FIG>, it calculates the attribution value for each word in the free text report <NUM> related to the label assigned to the associated image <NUM>. In one possible configuration, the free text report can be displayed on a workstation as shown in <FIG>, along with a thumbnail of the associated image(s), and the free text report is color coded or highlighted in any suitable manner to bring the user's attention to the key words or phrases that had the highest attribution value scores and thereby assist the user in understanding why the label was assigned to the associated image. For example, all words in the report with a attribution score above a certain threshold (which could possibly by user-specified) is shown in bold font, red color, larger font size, underlined, or in some other fashion to indicate that these words were the most significant in generating the structured label.

The medical images <NUM> with the structured labels <NUM> as generated by the NLP <NUM> of <FIG> are then used to train a computer vision model (pattern recognizer) to assign or replicate the structured labels to medical images based on image pixels alone. In particular, the training of a computer vision model <NUM> will be described in <FIG>. In this example, the computer vision model <NUM> is in the form of a deep convolutional neural network. The computer vision model can be implemented in several different configurations. In general, deep convolutional neural network pattern recognizers, of the type used in <FIG>, are known in the art of pattern recognition and machine vision, and therefore a detailed description thereof is omitted for the sake of brevity. One implementation is Inception-v3 deep convolutional neural network architecture, which is described in the scientific literature. See the following references: C. <NPL>); <NPL>); see also <CIT>. A fourth generation, known as Inception-v4 is considered an alternative architecture for the computer vision model. See <NPL>). See also <CIT>.

Another alternative for the computer vision model is described in the paper of X.

For training of the computer vision model <NUM>, we use the images <NUM> of <FIG> and labels generated by the convolutional neural network <NUM> as explained in <FIG>. In essence, by training on this body of medical images <NUM> and associated structured labels <NUM> as shown in <FIG>, the computer vision model <NUM> is able to take input images (pixel data alone) and generate structured labels. In other words, the computer vision model does not need associated free-text reports in order to generate structured labels. Rather, it can generate the labels on its own. So, referring now to <FIG>, the computer vision model <NUM> of <FIG>, once trained and validated, is able to take a given input image <NUM>, for example a chest X-ray, which is not associated with a medical report, and generate a structured label <NUM>, for example as a digital assistant to a radiologist.

One example of how the computer vision model might be used is in the context of radiology, in a hospital or clinic environment. In one configuration, the hospital or clinic will have a computer system which is configured with the necessary hardware to implement the computer vision model <NUM> of <FIG> (details of which are not particularly important). In another configuration, the hospital or clinic could also simply request predictions from a computer vision model that is hosted by a third party service provider, in the cloud. This way, the hospital or clinic would not need to run model inference on site, but could configure their software with an application programming interface (API) to make a call to a computer system of a service provider in the cloud which hosts the computer vision model and performs inference on an input image (chest X-ray, for example) and returns via the API a structured label.

A patient has a chest X-ray performed and the digital X-ray image is supplied as input to the computer vision model. The structured label <NUM> is generated and then interpreted by a radiologist. For example the label [<NUM>] in interpreted as positive for pneumothorax, left side, severity moderate, and negative for misplaced nasogastric tube and negative for cancerous lesion or lump. The radiologist inspects the X-ray with the aid of this additional information from the computer vision model and it confirms her own findings and diagnosis. As another example, the radiologist views an X-ray and comes to an initial finding of pulmonary edema and plural effusion, but after considering the structured label [<NUM>] indicative of cardiomegaly and negative for pulmonary edema, she reconsiders her own evaluation of the X-ray, confirms that the computer vision model has correctly evaluated the X-ray and then makes the correct entries and correct diagnosis in the patient's chart, thereby avoiding a potentially serious medical error.

One of the advantages of the system and method illustrated in this document is that it permits the training of the computer vision model <NUM> from a large body of medical images and associated structured labels but the labels are generated automatically and do not require extensive input via human operators. While the convolution neural network (NLP) of <FIG> that is used in generating such labels did make use of a corpus of medical reports with labels assigned by medical experts in initial training, this amount of human involvement is relatively minor in comparison and the methodology of this disclosure allows for the generation of a very large body of training images with structured labels, on the order of thousands or even tens of thousands, and this large body of training images helps the computer vision model achieve both generalize and avoid overfitting to training data.

Referring now to <FIG> is an illustration of an ensemble method of training both a computer vision model (<NUM>) and a diagnosis extraction network <NUM> which is analogous to a natural language processor (NLP) or the 1D deep convolutional neural network <NUM> of <FIG>. Each training example <NUM>, only one of which is shown, consists of a medical image <NUM> (e.g. a chest X-ray) and an associated free-text report <NUM> rendered by a radiologist. These reports are composed of unstructured natural language, and are prone to subjectivity and error. These are the lower quality, "noisy" labels. Some fraction of the training examples will have associated images that have been independently reviewed by a panel of experts and evaluated according a structured label schema or rubric (for instance, a series of binary variables indicating the presence or absence of various findings, as explained above). These labels are considered "ground-truth" labels.

The CNN feature extractor <NUM> is a convolutional neural network or pattern recognizer model (see examples above) that processes the medical image pixels to develop a vector representation of image features related to a classification produced by the feature extractor. The diagnosis extraction network <NUM> learns to extract from the report <NUM> clinical variables of interest using the vector representation of image features as an additional input beyond just the free-text reports <NUM>. This allows the network <NUM> to not only convert the natural language into a useful format, but also correct confusion, bias or error in the original report <NUM>. (Note that the diagnosis extraction network can be initialized with the convolutional neural network NLP model <NUM> (<FIG>) that was described previously, but it is different in that it also receives the output of the CNN feature extractor <NUM>. ) The diagnosis extraction network <NUM> generates structured findings <NUM> which can be used to supervise a multi-label classifier <NUM> (e.g., a deep convolutional neural network pattern recognizer), which operates on the image features alone, when a ground-truth label is not available. In this way, we train a medical image diagnosis model using the abundant but noisy signal contained in the reports, relying on a smaller number of more costly ground-truth labels.

In the methodology of <FIG>, one may optionally include an Integrated Gradients module <NUM> which queries the structured findings and the text in the medical reports <NUM> and generates attribution data <NUM> which can be presented to a user or stored in computer memory for evaluation of the performance of the diagnosis extraction network <NUM>.

In use, after the computer vision model <NUM> of <FIG> has been trained, it can be configured in a computer system and ancillary memory and used to generate a label for an input image, e.g., chest X-ray in the manner shown in <FIG>. In this situation, the multi-label classifier <NUM> generates the labels for the input image <NUM>, even if the image is not associated with a free-text report.

Accordingly, in reference to <FIG>, we have described a machine learning system which includes a computer memory (not shown) storing a set of training data <NUM> in the form of a multitude of training examples, each of which comprises a free-text medical report <NUM> and one or more associated medical images <NUM>. A subset of the training examples contain ground truth structured labels assigned by a medical expert. The system further includes a computer system configured to operate on each of the multitude of training examples <NUM>. The computer system (<FIG>) is configured as follows: it includes a) a feature extractor <NUM> (which may take the form of a deep convolutional neural network) receiving as input the one or more medical images <NUM> and generating a vector of extracted image features <NUM>; b) a diagnosis extraction network <NUM> (essentially analogous to the NLP <NUM>-D convolutional neural network <NUM> of <FIG>) receiving as input a free-text medical report <NUM> and the vector of extracted image features <NUM> and generating a structured label <NUM>; and c) an image classifier <NUM> (e.g., deep convolutional neural network pattern recognizer) trained on the structured labels <NUM> and the vector of extracted features <NUM>. The image classifier <NUM> is further configured to generate a structured label for a further input medical image, for example as explained in <FIG>.

As shown in <FIG>, the system may optionally include an Integrated Gradients module <NUM> generating attribution data for the words in the free text medical reports contributing to the structured labels generated by the diagnosis extraction network <NUM>. In one embodiment, the medical images are chest X-rays. The structured labels may take the form of binary labels for presence or absence of a medical condition or diagnosis, and/or labels for assignment of a particular diagnostic billing code. For example the medical condition could be one or more of airspace opacity (including atelactasis and consolidation), pulmonary edema, pleural effusion, pneumothorax, cardiomegaly, nodule or mass, misplaced nasogastric tube, misplaced endotracheal tube, and misplaced central venous catheter.

Claim 1:
A machine learning system for classifying medical images based on a presence or absence of a medical condition, the system comprising:
a computer memory storing a first corpus of free-text medical reports (<NUM>) each of which having an associated medical image and one or more ground truth structured labels (<NUM>) assigned by a medical expert;
a computer vision model (<NUM>) trained to assign a further structured label (<NUM>) to an input medical image (<NUM>), the computer vision model (<NUM>) including a feature extractor (<NUM>) configured to generate a vector of extracted image features (<NUM>) for the medical image (<NUM>, <NUM>) and an image classifier (<NUM>) configured to generate the further structured label (<NUM>) using the vector of extracted features (<NUM>);
a natural language processor (<NUM>) configured to receive as input a free-text medical report and a vector of extracted image features generated by the feature extractor (<NUM>) of the computer vision model (<NUM>) and to generate a predicted structured label,
wherein the natural language processor (<NUM>) is trained, using the first corpus of free-text medical reports and associated ground truth structured labels (<NUM>), to use the vectors of extracted image features generated by the feature extractor (<NUM>) for the medical images associated with the free-text medical reports in addition to the free-text medical reports to generate the predicted structured labels,
the system further comprising:
a computer memory storing a second corpus (<NUM>, <NUM>) of free-text medical reports (<NUM>, <NUM>) that are associated with medical images (<NUM>, <NUM>) and wherein the natural language processor (<NUM>) is applied to such second corpus (<NUM>) of free-text medical reports (<NUM>, <NUM>) and responsively generates predicted structured labels (<NUM>, <NUM>) for the associated medical images (<NUM>, <NUM>),
wherein the computer vision model is trained on the medical images (<NUM>, <NUM>) and the predicted structured labels (<NUM>, <NUM>) generated by the natural language processor (<NUM>) for the free-text medical reports of the second corpus to use the feature extractor (<NUM>) and the image classifier (<NUM>) to generate a further structured label for a further input medical image, wherein the ground truth, further and predicted structured labels (<NUM>, <NUM>, <NUM>) comprise labels for presence or absence of a medical condition.