Patent Description:
Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output.

Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recurrent neural network uses some or all of the internal state of the network after processing a previous input in the input sequence in generating an output from the current input in the input sequence.

<CIT> relates to a system for medical diagnosis and support services to consumers over network infrastructure using servers- or cloud systems that can be accessed by various clients and a method for providing same.

<CIT> relates to a system, having method and apparatus aspects, for automatically measuring a ratio of respective diameters of the optical disc cup and the optical disc of an eye.

<CIT> relates to methods and systems for obtaining data useful in detecting glaucoma in a human subject.

<CIT> relates to a boosting learning approach using three-dimensional (3D) information to effect automated segmentation of retinal blood vessels.

This specification generally describes a system that generates health analysis data for a patient by processing data that includes one or more fundus images of the patient using a fundus image processing machine learning model.

A health analysis system can effectively analyze the health of a patient using only one or more images of the fundus of the patient's eye and minimal or no other patient data. In particular, the health analysis system can effectively analyze the presence or the probable progression of a specific medical condition using the fundus images. Instead or in addition, the health analysis system can effectively predict which treatments or follow-up actions will be most effective in treating the medical condition. Instead or in addition, the health analysis system can accurately evaluate the risk of the patient for undesirable health events or accurately evaluate the overall health of the patient using the fundus images. Instead or in addition, the health analysis system can accurately predict values of a set of factors that contribute to a risk of a particular set of health events happening to the patient using the fundus images.

In some implementations, the system can present a user of the system with data that explains the basis for the predictions generated by the system, i.e., the portions of the fundus image that the machine learning model focused on to generate a particular prediction. In so doing, the system can allow a medical practitioner or other user to have insight into the prediction process.

The present invention relates to a computer-implemented method, a corresponding system, and computer readable media as defined by the appended claims.

This specification generally describes a system that can generate health analysis data for a patient from an input that includes one or more fundus images of the patient and, optionally, other patient data. A fundus image is a photograph of the fundus of one of the eyes of the patient. The fundus of an eye is the interior surface of the eye opposite the lens and includes, among other things, the retina and the optic disc.

Generally, to generate the health analysis data for a given patient, the system processes the one or more fundus images and, optionally, the other patient data using a fundus image processing machine learning model to generate a model output for the patient and then generates the health analysis data from the model output.

<FIG> shows an example fundus image analysis system <NUM>. The fundus image analysis system <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

For a given patient, the fundus image analysis system <NUM> receives fundus image data <NUM> that includes one or more fundus images of the patient's eye and generates health analysis data <NUM> that characterizes the health of the patient.

In some implementations, the fundus image analysis system <NUM> includes or is in communication with a fundus image capturing system <NUM> that generates the fundus images and provides them as input fundus image data <NUM> to the fundus image analysis system. In particular, the fundus image capturing system <NUM> includes one or more image capturing devices, e.g., an image capturing device <NUM>, that are configured to capture images of the fundus of a patient. Generally, the image capturing device <NUM> is a specialized fundus camera that is configured to capture an appropriate type of fundus image, e.g., using color fundus photography, stereoscopic photography, wide field or ultra wide field photography, or scanning laser ophthalmoscopy (SLO). In some cases, the image capturing system <NUM> includes multiple image capturing devices that capture different types of fundus images.

In other implementations, the fundus image analysis system <NUM> receives the input fundus image data <NUM> from an external system, e.g., over a data communication network.

The fundus image analysis system <NUM> processes the input fundus image data <NUM> and, optionally, other data for the given patient using a fundus image processing machine learning model <NUM>. The fundus image processing machine learning model <NUM> is a machine learning model that is configured to process the input fundus image data <NUM> and, optionally, other patient data <NUM> to generate a model output <NUM> that characterizes the health of the patient.

How many fundus images are in the fundus image data <NUM>, whether the system <NUM> receives other patient data <NUM> and, if so, the nature of the other patient data <NUM> that is received, and the makeup of the model output <NUM> are dependent on the configuration of the fundus image processing machine learning model <NUM>. Fundus image data, example configurations of the machine learning model <NUM>, and example makeups of the model output <NUM> are described in more detail below with reference to <FIG>.

The fundus image analysis system <NUM> also includes a patient health analysis subsystem <NUM> that receives the model output <NUM> and generates the patient health analysis data <NUM>. Generally, the patient health analysis subsystem <NUM> generates health analysis data that characterizes the model output in a way that can be presented to a user of the system. The patient health analysis subsystem <NUM> can then provide the health analysis data <NUM> for presentation to the user in a user interface, e.g., on a user computer of the patient or on a computer of a medical professional, store the health analysis data <NUM> for future use, or provide the health analysis data <NUM> for use for some other immediate purpose.

In some implementations, the fundus image analysis system <NUM> receives requests for patient health analysis data <NUM> from remote users of user computers over a data communication network. For example, a user computer, e.g., a computer on which the fundus image capturing system <NUM> is implemented, may be able to submit a request to the fundus image analysis system <NUM> over the data communication network by providing fundus image data as part of making an Application Programming Interface (API) call to the fundus image analysis system <NUM>. In response to the API call, the fundus image analysis system <NUM> can generate the health analysis data <NUM> and transmit the health analysis data to the user computer over the data communication network.

Additionally, in some implementations, the machine learning model <NUM> is implemented by one or more computers that are remote from the fundus image analysis system <NUM>. In these implementations, the fundus image analysis system <NUM> can access the machine learning model <NUM> by making an API call over a network that includes the input to the machine learning model <NUM> and can receive the model output <NUM> in response to the API call.

While the description in this specification generally describes a single machine learning model <NUM> that generates a particular model output, in some cases the system <NUM> includes or communicates with an ensemble of multiple machine learning models for a given kind of model output. Each machine learning model <NUM> generates the same kind of model output, and the system <NUM> or another system can combine the model outputs generated by the ensemble, e.g., by computing a measure of central tendency of the model outputs. The combined output can then be treated as the model output <NUM> by the system <NUM>.

<FIG> is a flow diagram of an example process <NUM> for generating health analysis data. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a fundus image analysis system, e.g., the fundus image analysis system <NUM> of <FIG>, appropriately programmed, can perform the process <NUM>.

The system receives input fundus image data and, optionally, other patient data (step <NUM>).

Generally, the fundus image data includes one or more fundus images of a patient's eye.

In some implementations, the fundus image data includes a single fundus image, e.g., an image that captures the current state of the patient's fundus.

In some other implementations, the fundus image data includes multiple fundus images that capture the current state of the patient's fundus. For example, the fundus image data can include one or more images of the fundus in the patient's left eye and one or more images of the fundus in the patient's right eye. As another example, the fundus images may include multiple different types of fundus photographs. For example, the fundus images may include two or more of: a color fundus photograph, a stereoscopic fundus photograph, a wide field or ultra wide field fundus photograph, or a scanning laser ophthalmoscopy (SLO) fundus photograph. As yet another example, the fundus images can include multiple images captured using different imaging technology, e.g., optical coherence tomography (OCT) and Heidelberg retinal tomography (HRT).

In yet other implementations, the fundus image data includes a temporal sequence of fundus images that capture how the state of the fundus has evolved over time. That is, the temporal sequence includes multiple fundus images, with each fundus image having been taken at a different time. In some implementations, the fundus images are ordered in the temporal sequence from least recent to most recent.

The other patient data is data that characterizes the patient's eye, data that generally characterizes the patient, or both. For example, the other patient data can include ocular measurement data, e.g., eye pressures, visual fields, visual acuity, central corneal thickness, and so on, patient demographics, e.g., age, gender, ethnicity, family history, and so on, or both.

The system processes the input fundus image data and, optionally, the other patient data using a fundus image processing machine learning model to generate a model output (step <NUM>).

Optionally, prior to processing the fundus image data using the machine learning model, the system can pre-process the fundus images. For example, for a given image, the system can apply any of a variety of conventional image processing techniques to the image to improve the quality of the output generated by the machine learning model. As an example, the system may crop, scale, deskew or re-center the image. As another example, the system can remove distortion from the image, e.g., to remove blurring or to re-focus the image, using conventional image processing techniques.

In implementations where the fundus image data includes a single fundus image, the fundus image processing machine learning model is a feedforward machine learning model that has been configured by being trained on appropriately labeled training data to process the fundus image data and, optionally, the other patient data to generate a model output that characterizes a particular aspect of the patient's health. For example, the fundus image processing machine learning model may be a deep convolutional neural network. An example of a deep convolutional neural network that can be trained to process a fundus image to generate the model outputs described in this specification is described in<NPL>. Other examples of deep convolutional neural networks, including convolutional neural networks with residual connections, that can be trained to process a fundus image to generate the model outputs described in this specification are described in <NPL>.

In implementations where the fundus image data includes multiple fundus images that characterize the current state of the patient's fundus, the fundus image processing machine learning model may be a feedforward fundus image processing machine learning model that has been configured by being trained on appropriately labeled training data to process all of the fundus images to generate a model output that characterizes a particular aspect of the patient's health. For example, the fundus image processing machine learning model may be a deep convolutional neural network that includes multiple towers of convolutional layers. An example of a deep convolutional neural network that can be trained to process multiple fundus images to generate the model outputs described in this specification is described in<NPL>.

In implementations where the fundus image data includes a temporal sequence of fundus images, the fundus image processing machine learning model may be a recurrent fundus image processing machine learning model that has been configured to process each image in the temporal sequence one by one to, for each image, update the internal state of the recurrent fundus image processing machine learning model, and to, after the last image in the temporal sequence has been processed, generate a model output that characterizes a particular aspect of the patient's health. For example, the fundus image processing machine learning model may be a recurrent neural network that includes one or more long short-term memory (LSTM) layers. A recurrent neural network that can be trained to process a sequence of fundus images to generate the model outputs described in this specification is described in <NPL>.

In some implementations, the model output is specific to a particular medical condition. Model outputs that are specific to a particular medical condition are described in more detail below with reference to <FIG>.

In some other implementations, the model output is a prediction of a future state of the fundus of the patient's eye. A model output that is a prediction of the future state of a fundus is described in more detail below with reference to <FIG>.

In yet other implementations, the model output is a prediction of the risk of a particular health event occurring in the future. A model output that is a prediction of the risk of a particular event occurring is described in more detail below with reference to <FIG>.

In yet other implementations, the model output characterizes the overall health of the patient. A model output that characterizes the overall health of the patient is described in more detail below with reference to <FIG>.

In yet other implementations, the model output is a prediction of values of factors that contribute to a particular kind of health-related risk. A model output that is a prediction of values of risk factors is described in more detail below with reference to <FIG>.

The system generates health analysis data from the model output (step <NUM>). Generally, the health analysis data characterizes the model output in a way that can be presented to a user of the system.

In some implementations, the health analysis data also includes data derived from an intermediate output of the machine learning model that explains the portions of the fundus image or images that the machine learning model focused on when generating the model output. In particular, in some implementations, the machine learning model includes an attention mechanism that assigns respective attention weights to each of multiple regions of an input fundus image and then attends to features extracted from those regions in accordance with the attention weights. In these implementations, the system can generate data that identifies the attention weights and include the generated data as part of the health analysis data. For example, the generated data can be an attention map of the fundus image that reflects the attention weights assigned to the regions of the image. For example, the attention map can be overlaid over the fundus image to identify the areas of the patient's fundus that the machine learning model focused on when generating the model output. Generating data that identifies areas of the fundus that were focused on by the machine learning model is described in more detail below with reference to <FIG>.

The system can then provide the health analysis data for presentation to the user in a user interface, e.g., on a user computer of the patient or on a computer of a medical professional, or store the health analysis data for future use.

<FIG> shows the processing of an example fundus image <NUM> by the fundus image processing machine learning model <NUM>. In particular, in the example of FIG. 1B, the fundus image processing machine learning model <NUM> is a deep convolutional neural network that is configured to receive the fundus image <NUM> and to process the fundus image <NUM> to generate a model output that characterizes a particular aspect of the patient's health.

The convolutional neural network illustrated in <FIG> is a simplified example of a deep convolutional neural network and includes a set of convolutional neural network layers <NUM>, followed by a set of fully connected layers <NUM>, and an output layer <NUM>. It will be understood that, in practice, a deep convolutional neural network may include other types of neural network layers, e.g., pooling layers, normalization layers, and so on, and may be arranged in various configurations, e.g., as multiple modules, multiple subnetworks, and so on.

During the processing of the fundus image <NUM> by the convolutional neural network, the output layer <NUM> receives an output generated by the last fully connected layer in the set of fully connected layers <NUM> and generates the model output for the fundus image <NUM>. In the example of <FIG>, the model output is a set of scores <NUM>, with each score being generated by a corresponding node in the output layer <NUM>. As will be described in more detail below, in some cases, the set of scores <NUM> are specific to particular medical condition. In some other cases, the each score in the set of scores <NUM> is a prediction of the risk of a respective health event occurring in the future. In yet other cases, the scores in the set of scores <NUM> characterize the overall health of the patient.

Once the set of scores <NUM> have been generated, the fundus image analysis system generates patient health analysis data that characterizes an aspect of the patient's health from the scores <NUM> and provides the health analysis data for presentation to the user in a user interface, e.g., on a user computer of the patient or on a computer of a medical professional, stores the health analysis data for future use, or provides the health analysis data for use for some other immediate purpose.

<FIG> is a flow diagram of an example process <NUM> for generating health analysis data that is specific to a particular medical condition. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a fundus image analysis system, e.g., the fundus image analysis system <NUM> of <FIG>, appropriately programmed, can perform the process <NUM>.

The system processes the input fundus image data and, optionally, the other patient data using a fundus image processing machine learning model to generate a set of condition state scores (step <NUM>).

Generally, the set of condition state scores are specific to a particular medical condition that the system has been configured to analyze.

In some implementations, the medical condition is a particular eye-related condition.

For example, the particular eye-related condition may be glaucoma. Generally, glaucoma is a condition in which the optic nerve is damaged, which can result in blindness.

As another example, the particular eye-related condition may be age-related macular degeneration. Generally, age-related macular degeneration is a condition in which the macula, an area near the center of the retina, has deteriorated, which may cause partial or total vision loss.

As another example, the particular eye-related condition may be retinal detachment. Generally, retinal detachment is a disorder in which the retina detaches either partially or completely from its underlying layer of support tissue.

As yet another example, the particular eye-related condition may be ocular occlusions. Generally, an ocular occlusion is the blockage or closing of a blood vessel that carries blood to or from some portion of the eye, e.g., to or from the retina.

In some other implementations, the specific condition is not an eye-related condition but is instead a neurodegenerative condition, e.g., Parkinson's or Alzheimer's, or another condition that can effectively be analyzed using fundus imagery.

In some implementations, the set of condition state scores includes a single score that represents a likelihood that the patient has the medical condition.

For example, in the case of glaucoma, the single score may represent a likelihood that the patient has glaucoma.

As another example, in the case of age-related macular degeneration, the single score may represent a likelihood that the patient has age-related macular degeneration.

As another example, in the case of retinal detachment, the single score may represent a likelihood that the patient has retinal detachment.

As another example, in the case of ocular occlusions, the single score may represent a likelihood that the patient has one or more ocular occlusions.

As another example, in the case of neurodegenerative conditions, the single score may represent a likelihood that the patient has the neurodegenerative condition e.g., Parkinson's or Alzheimer's.

In some other implementations, the set of condition state scores includes a respective score for each of multiple possible levels of the medical condition, with each condition score representing a likelihood that the corresponding level is current level of the condition for the patient.

For example, in the case of glaucoma, the set of scores may include a score for no glaucoma, mild or early-stage glaucoma, moderate-stage glaucoma, severe-stage glaucoma, and, optionally, an indeterminate or unspecified stage.

As another example, in the case of age-related macular degeneration, the set of scores may include a score for no macular degeneration, early-stage macular degeneration, intermediate macular degeneration, advanced macular degeneration, and, optionally, an indeterminate or unspecified stage.

As another example, in the case of retinal detachment, the set of scores may include a score for no retinal detachment, initial retinal detachment, i.e., only retinal tears or retinal breaks, advanced retinal detachment, and, optionally, an indeterminate or unspecified stage.

As another example, in the case of ocular occlusions, the set of scores may include a score for no ocular occlusions, minor ocular occlusions, severe ocular occlusions, and, optionally, an indeterminate or unspecified stage.

As another example, in the case of neurodegenerative conditions, the set of scores may include a score for not having the neurodegenerative condition, a score for each of multiple stages of the neurodegenerative condition, and, optionally, an indeterminate or unspecified stage.

The system generates health analysis data from the condition state scores (step <NUM>). For example, the system can generate health analysis data that identifies the likelihood that the patient has the medical condition or identifies one or more condition levels that have the highest condition state scores.

<FIG> is a flow diagram of an example process <NUM> for generating health analysis data that identifies patient follow-up actions. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a fundus image analysis system, e.g., the fundus image analysis system <NUM> of <FIG>, appropriately programmed, can perform the process <NUM>.

The system processes the input fundus image data and, optionally, the other patient data using a fundus image processing machine learning model to generate a set of follow-up scores (step <NUM>).

The set of follow-up scores includes a respective score for each of multiple possible follow-up actions that can be taken by the patient to treat a particular medical condition. For example, the set of possible follow-up actions may include performing a re-screening at a future time, visiting a doctor at a future time, and visiting a doctor immediately. Each follow-up score represents a likelihood that the corresponding follow-up action is the proper action to be taken to effectively treat the medical condition.

The system generates health analysis data from the follow-up scores (step <NUM>). For example, the system can generate health analysis data that recommends that the patient take the follow-up action that has the highest follow-up score.

<FIG> is a flow diagram of an example process <NUM> for generating health analysis data that predicts the likely progression of a medical condition. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a fundus image analysis system, e.g., the fundus image analysis system <NUM> of <FIG>, appropriately programmed, can perform the process <NUM>.

The system processes the input fundus image data and, optionally, the other patient data using a fundus image processing machine learning model to generate a set of progression scores (step <NUM>). The set of progression scores are specific to a particular medical condition that the system has been configured to analyze. The set of condition state scores includes a respective score for each of multiple possible levels of the medical condition, with each condition score representing a likelihood that the corresponding level will be the level of the condition for the patient at a predetermined future time, e.g., in <NUM> months, in <NUM> year, or in <NUM> years.

For example, in the case of glaucoma, the set of scores may include a score for no glaucoma, mild or early-stage glaucoma, moderate-stage glaucoma, and severe-stage glaucoma, with the score for each stage representing the likelihood that the corresponding stage will be the stage of glaucoma for the patient at the future time.

As another example, in the case of age-related macular degeneration, the set of scores may include a score for no macular degeneration, early-stage macular degeneration, intermediate-stage macular degeneration, and advanced-stage macular degeneration, and, optionally, with the score for each stage representing the likelihood that the corresponding stage will be the stage of macular degeneration for the patient at the future time.

As another example, in the case of neurodegenerative conditions, the set of scores may include a score for not having the neurodegenerative condition and a score for each of multiple stages of the neurodegenerative condition, with the score for each stage representing the likelihood that the corresponding stage will be the stage of the condition for the patient at the future time.

The system generates health analysis data from the progression scores (step <NUM>). The health analysis data identifies the likely progression of the medical condition for the patient. For example, the system can generate health analysis data that identifies one or more of the possible condition levels and, for each possible condition level, the likelihood that the corresponding level will be the future level of the condition for the patient.

<FIG> is a flow diagram of an example process <NUM> for generating health analysis data that predicts the proper treatment for a medical condition for a given patient. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a fundus image analysis system, e.g., the fundus image analysis system <NUM> of <FIG>, appropriately programmed, can perform the process <NUM>.

The system processes the input fundus image data and, optionally, the other patient data using a fundus image processing machine learning model to generate a set of treatment scores (step <NUM>).

The set of treatment scores include a respective score for each of multiple possible treatments for a given medical condition, with each treatment score representing a likelihood that the corresponding treatment is the most effective treatment for the condition for the current patient.

For example, the set of treatment scores can include a respective score for each of multiple medications that can be prescribed to a patient that has the medical condition.

As another example, the set of treatment scores can include a respective score for each of multiple treatment plans for a given medical condition, e.g., a respective score for one or more medical procedures and a score for rehabilitation without undergoing a procedure.

The system generates health analysis data from the progression scores (step <NUM>). For example, the health analysis data can identify one or more of the highest-scoring treatments.

<FIG> is a flow diagram of an example process <NUM> for generating health analysis data that includes a predicted fundus image. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a fundus image analysis system, e.g., the fundus image analysis system <NUM> of <FIG>, appropriately programmed, can perform the process <NUM>.

The system processes the input fundus image data and, optionally, the other patient data using a fundus image processing machine learning model to generate a predicted fundus image (step <NUM>).

The predicted fundus image is an image of the fundus of the eye of the patient as it is predicted to look at a particular future time, e.g., in six months, in one year, or in five years.

For example, the fundus image processing machine learning model may be a convolutional neural network that is configured through training to predict, for each pixel in the input fundus image, the color of the pixel at the particular future time.

As another example, when the fundus image data includes a temporal sequence of fundus images, the fundus image processing machine learning model may be a recurrent neural network that is configured through training to, for each pixel in the most recent fundus image in the sequence, predict the color of the pixel at the particular future time. The system can use the predicted color values for the pixels to generate the predicted fundus image.

The system generates health analysis data from the progression scores (step <NUM>). For example, the health analysis data can include the predicted fundus image and, optionally, additional health analysis data.

<FIG> is a flow diagram of an example process <NUM> for generating health analysis data that predicts the risk of a health event occurring. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a fundus image analysis system, e.g., the fundus image analysis system <NUM> of <FIG>, appropriately programmed, can perform the process <NUM>.

The system processes the input fundus image data and, optionally, the other patient data using a fundus image processing machine learning model to generate a set of risk scores (step <NUM>).

In some implementations, the set of risk scores includes a single score that measures a particular kind of risk. For example, the score may measure a predicted cardiovascular risk of the patient, e.g., may be a predicted Framingham risk score that measures the <NUM>-year cardiovascular risk of the patient.

In some other implementations, the set of risk scores may be specific to a particular undesirable health event.

For example, the undesirable health event may be a heart attack, a stroke, mortality, hospitalization, a fall, complications pre-operation or post-operation, and so on. In some of these implementations, the set of risk scores includes a single score that represents a likelihood of the undesirable health event occurring in the future, e.g., within a specified future time window. In others of these implementations, the set of risk scores includes a respective score for each of multiple risk levels, e.g., low, medium, and high, for the health event, with each risk score representing a likelihood that the corresponding risk level is the current risk level of the health event occurring.

In yet other implementations, the set of scores can include multiple scores, with each score corresponding to a respective undesirable health event and representing a likelihood that the corresponding undesirable health event will occur in the future, e.g., within a specified future time window.

The system generates health analysis data from the risk scores (step <NUM>). For example, in implementations where the set of scores includes a single score, the health analysis data can identify the single score. As another example, where the set of scores includes multiple scores, the health analysis data can identify the highest-scoring risk level.

<FIG> is a flow diagram of an example process <NUM> for generating health analysis data that characterizes the overall health of the patient. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a fundus image analysis system, e.g., the fundus image analysis system <NUM> of <FIG>, appropriately programmed, can perform the process <NUM>.

The system processes the input fundus image data and, optionally, the other patient data using a fundus image processing machine learning model to generate a set of wellness scores (step <NUM>).

In some implementations, the set of wellness scores includes a single score that measures the overall health of the patient on a predetermined scale.

In some other implementations, the set of wellness scores may include a respective score for each of multiple wellness labels that each characterize the overall health of the patient. For example, the wellness labels may be "very healthy," "healthy," "somewhat unhealthy," and "very unhealthy. " Each score represents a likelihood that the corresponding wellness label accurately characterizes the current health of the patient. Thus, for example the score for the wellness label "very healthy" represents the likelihood that the patient is very healthy, while the score for the "somewhat unhealthy" label represents the likelihood that the patient is somewhat unhealthy.

The system generates health analysis data from the risk scores (step <NUM>). For example, in implementations where the set of scores includes a single score, the health analysis data can identify the single score. As another example, where the set of scores includes multiple scores, the health analysis data can identify the highest-scoring wellness label.

<FIG> is a flow diagram of an example process <NUM> for generating health analysis data that includes predicted values for one or more risk factors. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a fundus image analysis system, e.g., the fundus image analysis system <NUM> of <FIG>, appropriately programmed, can perform the process <NUM>.

The system receives input fundus image data that includes one or more fundus images (step <NUM>).

The system processes the input fundus image data using a fundus image processing machine learning model to generate a respective predicted value for each of one or more risk factors (step <NUM>).

Each of the risk factors is a factor that contributes to the risk of one of a particular set of health-related events happening to the patient. For example, when the risk is cardiovascular risk, the particular set of health-related events can be a health event that is classified as a major cardiovascular health event, e.g., myocardial infarction, heart failure, percutaneous cardiac intervention, coronary artery bypass grafting, malignant dysrhythmia, cardiac shock, implantable cardiac defibrillator, malignant dysrhythmia, cardiac-related mortality, and so on.

Continuing the example of cardiovascular risk, the risk factors can include one or more of: age, gender, body mass index, systolic blood pressure, diastolic blood pressure, a measure of HbAlc (glycated hemoglobin), or smoking status, i.e., whether or not the patient smokes cigarettes.

In some implementations, the system employs multiple machine learning models that each generate a predicted value for a different subset of the risk factors. For example, one model may generate predicted values for binary risk factors that can only take one of two values, e.g., smoking status and gender, while another model may generate predicted values for continuous risk factors that can take continuous values from some value range, e.g., age, body mass index, and blood pressure. Each of the two models may have similar architectures, but with different parameter values.

The system generates health analysis data from the predicted values (step <NUM>). For example, the health analysis data can identify each generated predicted value. In some cases, the system can use the predicted values to compute a measure of the particular risk and provide the computed measure of risk as part of the health analysis data. For example, the system can provide the predicted values as input to another machine learning model configured to predict the measure of risk or to a hard-coded formula to obtain the computed measure. For example, in the case of cardiovascular risk, the system can compute a Framingham risk score using the predicted values. Alternatively, the system can provide the predicted values as input to a machine learning model that has been trained to predict a risk measure based on values of risk factors.

<FIG> is a flow diagram of an example process <NUM> for generating health analysis data that includes data identifying locations in a fundus image that were focused on by the machine learning model when generating the model output. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a fundus image analysis system, e.g., the fundus image analysis system <NUM> of <FIG>, appropriately programmed, can perform the process <NUM>.

The system processes the input fundus image data and, optionally, the other patient data using a fundus image processing machine learning model to generate a model output (step <NUM>). The model output can be any of the model outputs described above with reference to <FIG>.

In particular, the machine leaning model is a model that includes one or more initial convolutional layers followed by an attention mechanism, which in turn is followed by one or more additional neural network layers.

The initial convolutional layers process each fundus image in the fundus image data to extract a respective feature vector for each of multiple regions in the fundus image.

The attention mechanism determines an attention weight for each of the regions in the fundus image and then attends to the feature vectors in accordance with the corresponding attention weights to generate an attention output. Generally, the attention mechanism attends to the feature vectors by computing a weighted sum or a weighted mean of the feature vectors, with the weight for each feature vector being the attention weight for the corresponding region. To determine the attention weights, the system can use any of a variety of attention schemes to determine the relevance of each of the feature vectors to generating the model output for the fundus image and then normalize the determined relevances to compute the attention weights. Example attention schemes include processing the feature vectors using one or more fully-connected layers to determine the relevance and determining the relevance of a given feature vector by computing a cosine similarity between the feature vector and a learned context vector. An example attention mechanism that can be adapted for use in the fundus image processing machine learning model is described in "<NPL>.

The additional neural network layers that follow the attention mechanism receive the attention output(s) for each of the fundus images and generate the model output from the attention output. For example, when the machine learning model is a recurrent neural network, the additional neural network layers include one or more recurrent layers. When the machine learning model is a convolutional neural network, the additional neural network layers can include convolutional neural network layers, fully-connected layers or other conventional feedforward neural network layers.

The system generates health analysis data from the risk scores (step <NUM>). In particular, as described above, the health analysis data characterizes the model output in a way that can be presented to a user of the system.

In addition, the health analysis data includes data characterizing the areas of the fundus image that the machine learning model focused on to generate the model output. In particular, the health analysis data include data identifying the attention weights assigned to the regions in the fundus image. For example, the system can generate an attention map that identifies, for each pixel in the fundus image, the attention weight assigned to the pixel, i.e., the attention weight for the region of the image that the pixel belongs to. For example, the attention map can be a heat map that represents the attention weights as colors. In some implementations, the system provides the attention map as an overlay of the corresponding fundus image.

Claim 1:
A computer-implemented method comprising:
obtaining a model input comprising one or more fundus images (<NUM>), each fundus image being an image of a fundus of an eye of a patient;
processing the model input using a fundus image processing machine learning model (<NUM>) to generate a model output (<NUM>) comprising a condition state score that represents a likelihood that the patient has a medical condition, wherein the fundus image processing machine learning model has been trained to process an input comprising one or more fundus images to generate a model output that represents a likelihood that a patient has the medical condition;
wherein the fundus image processing machine learning model comprises a deep convolutional neural network comprising one or more initial convolutional layers configured to extract a respective feature vector for each of a plurality of regions in the fundus image, the one or more initial convolutional layers followed by an attention mechanism and subsequently followed by one or more additional neural network layers that are configured to receive an output of the attention mechanism and to process the attention output to generate the model output;
wherein the attention mechanism is configured to:
receive the respective feature vector for each of a plurality of regions in the fundus image generated by the one or more initial convolutional layers of the fundus image processing machine learning model,
compute a respective attention weight for each of the regions, and
generate the attention output by attending to the feature vectors in accordance with the attention weights for the regions in fundus image; and
processing the model output to generate health analysis data for displaying in a user interface that comprises (i) the likelihood that the patient has the medical condition and (ii) data identifying the attention weights generated by the attention mechanism, wherein the data identifying the attention weights is an attention map that specifies the attention weights for the regions in the fundus image; and wherein the attention map is overlaid over the fundus image.