Patent Description:
Clinicians are often overburdened by the amount of data available to understand a condition of a patient, as much of the data comes from historical medical records of the patient and extensive databases of medical knowledge. As a result, a clinician may take much longer providing a diagnosis, thereby increasing costs of treatment and wait times for patients. Furthermore, although clinical guidelines can assist clinicians in identifying a correct diagnosis for a patient, such guidelines may not consider certain mediums of medical data beyond textual data. As a result, a clinician can omit certain conditions from the diagnoses, which can lead to less desirable patient outcomes.

The document "<NPL>, discloses a deep learning model to efficiently detect a disease from an image and annotate its contexts (e.g., location, severity and the affected organs).

The document "<NPL>, describes a deep learning-based latent feature representation with a stacked auto-encoder (SAE) for computer-aided diagnosis of Alzheimer's disease and its prodromal stage, mild cognitive impairment.

According to the present invention, a method, medical device and computer program are presented as defined in the claims. The implementations set forth herein relate to diagnostic inferencing with a multimodal deep memory network. In some implementations, a method performed by one or more processors is set forth as including steps or blocks such as generating a first set of embeddings from an image received at a first neural network, and generating a second set of embeddings from a document received at a second neural network. Each of the image and the document can be associated with a medical patient. The steps can further include applying the first set of embeddings and the second set of embeddings as input across a trained model that includes multiple different diagnosis embeddings. The trained model can include key embeddings and value embeddings that are associated with weights that are based at least in part on an amount of attention given to a portion of medical data from which the key embeddings and the value embeddings are generated. The steps can also include generating weights for the multiple different diagnosis embeddings based on a correlation between the first set of embeddings and the second set of embeddings, and the key embeddings and the value embeddings. Furthermore, the steps can include providing a patient diagnosis for the medical patient at least based on the generated weights for the multiple different diagnosis embeddings. The amount of attention can correspond to attention data that is based on an amount of eye movement exhibited by a user accessing the medical data. Generating the second set of embeddings can include generating an input value embedding from a section heading of the document, and generating input key embeddings from content that is separate from the section heading of the document. The medical data, from which the key embeddings and the value embeddings are generated, can include a medical image with a corresponding textual description. The second neural network can be a bi-directional recurrent neural network, and the document can correspond to an electronic health record. The first neural network can be a convolution neural network.

The method steps can be performed in any ordered, and are not limited by the order set forth herein. Furthermore, the methods steps can be embodied as instructions and stored in a memory of a computing device. The computing device can include one or more processors that, when executing the instructions, cause the computing device to perform the method steps. The methods steps can be embodied in instructions that are stored in a non-transitory computer readable medium.

Additionally the term "controller" is used herein generally to describe various apparatus relating to the operation of one or more electrical components or software components. A controller can be implemented in numerous ways (e.g., such as with dedicated hardware) to perform various functions discussed herein. A "processor" is one example of a controller, which employs one or more microprocessors that may be programmed using software (e.g., microcode) to perform various functions discussed herein. A controller may be implemented with or without employing a processor, and also may be implemented as a combination of dedicated hardware to perform some functions and a processor (e.g., one or more programmed microprocessors and associated circuitry) to perform other functions.

In various implementations, a processor or controller may be associated with one or more storage media (generically referred to herein as "memory," e.g., volatile and non-volatile computer memory such as RAM, PROM, EPROM, and EEPROM, floppy disks, compact disks, optical disks, magnetic tape, etc.). In some implementations, the storage media may be encoded with one or more programs that, when executed on one or more processors and/or controllers, perform at least some of the functions discussed herein. Various storage media may be fixed within a processor or controller or may be transportable, such that the one or more programs stored thereon can be loaded into a processor or controller so as to implement various aspects of the present invention discussed herein. The terms "program" or "computer program" are used herein in a generic sense to refer to any type of computer code (e.g., software or microcode) that can be employed to program one or more processors or controllers.

The implementations set forth herein relate to systems, methods, and apparatuses for providing diagnosis support for clinicians using a multimodal deep learning model. Specifically, the multimodal deep learning model can use both textual data inputs and image data inputs for providing one or more diagnoses for use by a clinician. The textual data can be provided to one or more neural networks of the multimodal deep learning model in order to create embeddings from the textual data (e.g., documents corresponding to electronic health records). The neural networks for creating the embeddings can include a bi-directional recurrent neural network and/or a convolution neural network. The image data can also be provided to one or more neural networks to generate embeddings for the images. The neural networks that process the images can include a convolution neural network. Each of the embeddings for the textual data and the image data can provide descriptors for one or more portions of each document and/or image that is provided to the multimodal deep learning model. The resulting embeddings can be provided as inputs to a trained model, also referred to as an attention model, for determining the embeddings that should be given more consideration than others. For instance, the embeddings, or connections between embeddings, can be assigned weights by the attention model according to their relevance to a particular diagnosis or a clinical decision.

The multimodal deep learning model can include multiple subsystems for providing diagnoses. For instance, the multimodal deep learning model can include an input processing module, a memory component, an attention model, and an output generation module, which can provide a final representation of a diagnosis in a form that is preferred by a clinician (i.e., a user). The input processing module can analyze different types of inputs, and the analysis performed by the input processing module can be selected according to the type of input. For instance, the input processing module can perform separate analyses for text inputs, image inputs, multi-class and multi-label inputs, multi-class and single label inputs, single diagnosis inputs, section-engineered inputs, and/or section specific inputs with limited descriptors. In this way, embeddings for the inputs can be tailored to promote efficiency and accuracy.

The memory component of the multimodal deep learning model can be provided with a predefined structure for more effectively identifying values when responding to a query. For instance, the memory component can be a condensed memory network with an abstract level of learned memory. Associations between memory slots and/or memory values can be pre-defined by a clinician or other expert, or learned from available data (e.g., any data available to the multimodal deep learning model). Furthermore, the memory component can be updated based on frequency data pertaining to each diagnosis available from the multimodal deep learning model. The frequency data can be based on public sources (e.g., a medical websites) and/or private sources (clinical electronic health records (EHRs)). Memory values that are less frequently accessed can be ignored or re-written in order to preserve computational resources for those memory values that remain relevant over time.

<FIG> illustrates a system <NUM> for providing medical diagnoses using a multimodal deep learning model. The system <NUM> can be embodied as one or more computing devices that execute instructions available to the computing devices. The system <NUM> can include one or more remote devices <NUM>, such as a server device <NUM> and/or any other storage device, for providing medical-related data for assisting with identifying accurate medical diagnoses. The server device <NUM> can provide an input module <NUM> with access to electronic health records <NUM>, medical images <NUM>, and/or attention data <NUM>. The electronic health records <NUM> can include medical data corresponding to current and prior patient diagnoses, as well as general sources of medical information. Such sources can include medical-related websites and databases that can include text and images for assisting clinicians with determining patient diagnoses. Furthermore, the medical images <NUM> (e.g., X-rays, CT scans, MRI, etc.) can be included in the electronic medical records <NUM> and/or separate from the electronic medical records <NUM>. For instance, the medical images <NUM> can be associated with a particular patient and include labels that identify a diagnoses of the patient. Furthermore, the medical images <NUM> can include no labels, but otherwise be associated with a patient that has a particular diagnosis, which can be determined from a source of the medical images <NUM> (e.g., an electronic folder or other source that is associated with the patient).

The attention data <NUM> provided by the server device <NUM> can correspond to data that is indicative of an amount of time clinicians spend looking at the electronic health records <NUM> and/or the medical images <NUM> when preparing a diagnosis for a patient. For instance, a computing device operated by the clinician can include a camera that can track the eye movement of the clinician. The computing device can also record the arrangement of the electronic health records <NUM> and/or the medical images <NUM> at a display of the computing device. The computing device and/or the server device <NUM> can use sensor data (e.g., based on signals provided by a camera) related to clinician eye movement, data related to the arrangement of the records and images at the display, data related to screen taps or mouse movement, and/or any other attention-related data to generate correlations that can be used as the basis for the attention data <NUM>. For instance, when a patient experiences a broken rib, the clinician may spend more time examining an x-ray image than a textual document, therefore the attention data <NUM> can indicate that the clinician relied more on the image than the textual document when making the broken rib diagnosis.

In some implementations, multiple electronic health records <NUM> can be associated with a particular diagnosis, and the attention data <NUM> can indicate the electronic health records <NUM> that clinicians accessed the most when making the particular diagnoses. For example, a patient with a diagnoses of lung cancer can be identified in an electronic medical record <NUM> as having complained of chest pain, hemoptysis, and difficulty swallowing. The attention data <NUM> can indicate that the clinician who provided the diagnosis viewed an article on lung cancer more than an x-ray image of the patient's lungs or an article on pneumonia. In some implementations, the attention data <NUM> can identify portions of an individual electronic health record <NUM> that the clinician viewed the most when preparing a diagnosis. For instance, the attention data <NUM> can identify a first section of an article as being given more attention by a clinician than a second section of the article, at least when preparing a particular diagnosis. In this way, the system <NUM> is able to determine, using the attention data <NUM>, the portions of the electronic health records <NUM> and/or the medical images <NUM> that were most relevant for making a particular diagnosis (e.g., a section of a document entitled "conditions" can be more relevant to a diagnosis for a clinician than a "causes" section).

The electronic health records <NUM>, the medical images <NUM>, and/or the attention data <NUM> can be provided to an input module <NUM> of the system <NUM>. The remote device <NUM> can communicate such data over a network <NUM>, which can be a private or public network, such as the internet. The input module <NUM> can use the electronic health records <NUM>, the medical images <NUM>, and/or the attention data <NUM> to generate embeddings for each of the different inputs. The embeddings can correspond to vectors that link different portions of the inputs. The embeddings can be generated using one or more neural networks such as a recurrent neural network, a bi-directional recurrent neural network, a convolutional neural network, and/or any other neural network capable of providing embeddings for an input. In some implementations, a convolutional neural network can learn from existing text descriptors associated with the medical images <NUM> in order to generate embeddings from the images for subsequent use by the system <NUM>.

The embeddings generated by the input module can be used by a controller <NUM> to update a memory module <NUM>. The memory module <NUM> can include a predefined structure to optimize the selection of memory keys and values. For instance, the memory keys and values can be organized according to a frequency that particular information in memory is accessed or otherwise modified. Information that is less frequently accessed can be replaced or otherwise omitted from the memory module <NUM> in order to preserve computational resources and increase an amount of available space at the memory module <NUM>.

Memory slots of the memory module <NUM> can each correspond to a key-value embedding that is generated at the input module <NUM>. For instance, when image data is provided to the input module <NUM>, embeddings for the image data can be generated for one or more portions of the image data. Furthermore, word embeddings for the image data can also be generated and stored as value embeddings that are correlated to key embeddings. Similarly, when textual data inputs are provided to the input module <NUM>, embeddings can be generated for the textual data. The key embeddings can be correlated to content of the textual data and the value embeddings can be correlated to titles, section headings, and/or any other identifier that can indicate a section of text in the textual data.

The controller <NUM> can be tasked with reading and writing to the memory module <NUM>, in order to update that memory module <NUM> and provide responses to queries provided to the input module <NUM>. Furthermore, the controller <NUM> can assign frequency data to the key embeddings and/or value embeddings for particular diagnosis in order to direct the attention to the most relevant memory slots when responding to a query. When a frequency corresponding to a memory slot reaches or exceeds a threshold value (e.g., a low frequency threshold value), the memory slot can be deleted and/or reused in order to save space and improve the speed of the system <NUM>.

<FIG> illustrates an implementation of a multimodal deep memory network <NUM> as discussed herein. The multimodal deep memory network <NUM> can include a key-value memory <NUM> that stores key embeddings <NUM> and value embeddings <NUM>. The key embeddings <NUM> and value embeddings <NUM> can correspond to embeddings generated from different input mediums such as documents <NUM> and/or images <NUM>. The documents <NUM> can be processed according to their various parts (e.g., section headings and sentences) by a neural network, such as a bi-directional neural network process, a convolutional neural network process, a bag of words neural network process, and/or any other process capable of encoding text. Similarly, the images <NUM> can be processed through a convolutional neural network process in which key embeddings <NUM> and value embeddings <NUM> are generated from the images <NUM>.

Addressing the key value memory <NUM> can include measuring a probability that a document <NUM> or image <NUM>, or a portion thereof, is similar to each existing key according to Equation (<NUM>) below, where AΦx(x) is the input n (e.g., a document <NUM> or image <NUM>, or portion thereof). Each probability can then be attached to a memory slot accordingly.

In Equation (<NUM>), Φ can correspond to feature maps of dimension D, and Equation (<NUM>) can provide the equation for Softmax.

Using the probabilities calculated according to Equation (<NUM>), an output vector corresponding to a weighted sum of the memory slot values can be calculated from Equation (<NUM>) below.

After calculating the output vector, the embedding(s) associated with the document <NUM> or image <NUM> can be updated with Equation (<NUM>) below, where R is a d × d matrix.

The steps corresponding to Equations (<NUM>), (<NUM>), and (<NUM>) can be repeated for a fixed number of hops <NUM>. A probability for each available patient diagnosis is computed using the resulting output vector over all possible diagnoses yi, according to Equation (<NUM>) below, where B can be a d × D matrix. The multimodal deep memory network <NUM> can be trained this way, through an end to end process. In some implementations, back propagation and/or stochastic gradient descent are used to learn the parameters A, B and R<NUM>,.

If an input, such as a document <NUM> and/or image <NUM> is related to multiple memory slots, the number of hops <NUM> to identify all the related memory slots can be numerous, and potentially result in some related memory slots not being identified. For instance, an input can be associated with multiple different diagnoses, and some of the diagnoses can be correlated together by the multimodal deep memory network <NUM>. Therefore, a different approach to addressing the key value memory <NUM> can include incorporating multi-layer feedforward neural network (FNNs), which can employ a sigmoid output layer instead of Softmax. Each FNN can calculate a weight between <NUM> and <NUM> for each memory slot of the key value memory <NUM>, in order that the multimodal deep memory network <NUM> can thereafter read the weighted sum of the memory values.

Documents <NUM> provided to the multimodal deep memory network <NUM> can be represented in a Bag of Words (BOW) representation, in which each word wij in the document <NUM> (i.e., di = wi<NUM>, wi<NUM>, wi<NUM>,. , win) is represented as embeddings. The embeddings can be summed to create a result vector provided in Equation (<NUM>) below, where A is the embedding matrix.

Alternatively, the documents <NUM> can be encoded according to a position encoding process in which the position of the words in the documents <NUM> are encoded. The encoding can be implemented through Equation (<NUM>) below, in which · is an element-wise multiplication. Furthermore, lj is a column vector with a structure <MAT>, with J being the number of words in a document <NUM> and d being the dimensions of the embeddings. It should be noted that the key embeddings <NUM> and the value embeddings <NUM> can also be represented in this way.

The multimodal deep memory network <NUM> can process a variety of different inputs, and continue training as new inputs are received. Training input types can include image-only data, text-only data, multi-document data where each document contains multiple diagnoses, multi-document data where each document contains a single diagnosis, binary classification data where a single diagnoses is indicated with a binary value (e.g., yes or no), section-engineered data having single important section or label, and/or a document with a single label (e.g., a section heading) and a limited number of words. In some implementations, what constitutes an important section or feature of an input (e.g., a document or image) can be based on attention metrics or attention data. The attention data can be based on an amount of consideration that was previously given to an input to the multimodal deep memory network <NUM> or a portion of input to the multimodal deep memory network <NUM>. For instance, the attention data can be based on an amount of eye movement that was directed at a document or image while compiling a diagnosis for a patient.

In order to generate a patient diagnosis <NUM> from an input query <NUM>, the input query <NUM> can be processed by the multimodal deep memory network <NUM> to generate embeddings <NUM> from the input query <NUM>. A residual connection <NUM> to the embeddings <NUM> can be provided by the multimodal deep memory network <NUM> in order to mitigate abstraction of the input embeddings <NUM> as an increasing number of hops <NUM> are performed.

The multimodal deep memory network <NUM> can provide, as output, a patient diagnosis <NUM>, which can be embodied as a single classification diagnosis or a sequence of words (e.g., DIAGNOSIS_1, DIAGNOSIS_2, DIAGNOSIS_N, etc.). For instance, the single classification can be identified as a most probable diagnosis according to the multimodal deep memory network <NUM>. The sequence of words approach to providing a patient diagnosis <NUM> can be based on a recurrent neural network. The patient diagnosis <NUM> can be represented in one or more different mediums according to a preference of an end user (e.g., a clinician or patient), or a learned medium that is identified through training of the multimodal deep memory network <NUM>. The different mediums can include documents, videos, images, sound, and/or any other medium through which an end user can understand a predicted patient diagnosis <NUM>.

<FIG> illustrates a method <NUM> for providing a patient diagnosis from a multimodal deep memory network according to a correlation between embeddings provided by the multimodal deep memory network. The method <NUM> can be performed by one or more computing devices, network devices, and/or any other device capable of performing functions of one or more different neural networks. The method <NUM> can include a block <NUM> of generating a first set of embeddings from an image received at a first neural network. The first set of embeddings can be generated using a neural network, such as a convolution neural network having one or more hidden layers. The first set of embeddings can include key embeddings corresponding to image features, and value embeddings corresponding to labels or text associated with the image.

The method <NUM> can further include a block <NUM> of generating a second set of embeddings from a document received at a second neural network. The second neural network can be a recurrent neural network or a bi-directional recurrent neural network that updates vector and/or matrices weights according to forward passes and backward passes of neurons. The second set of embeddings can include value embeddings corresponding to section heading (e.g., a title) of the document and key embeddings corresponding to content that is separate from the section heading (e.g., sentences below the title).

The method <NUM> can further include a block <NUM> of applying the first set of embeddings and the second set of embeddings as input across a trained model that includes multiple different diagnosis embeddings. The trained model can include key embeddings and value embeddings that are weighted at least partially based on an amount of attention given to a portion of medical data from which the key embeddings and the value embeddings are generated. The trained model can include or access a condensed memory network that includes at least some amount of abstracted information. The trained model can include a predefined structure that is updated as memory slots in a memory of the trained model become less frequently accessed. In this way, less frequently accessed data can become more abstracted over time as the trained model undergoes subsequent training, and more frequently accessed data can remain less abstracted in order that more details can be gleaned from the more frequently accessed data.

The method <NUM> at block <NUM> can include generating weights associated with the multiple different diagnosis embeddings. Weights for the different diagnosis embeddings can be generated according to correlations between the embeddings generated from the image and/or document, and the diagnosis embeddings provided by the trained model. The resulting weights can be classified as a predetermined patient diagnosis or converted into a sentence over multiple iterations.

The method <NUM> can further include a block <NUM> of providing the patient diagnosis for the medical patient at least based on the generated weights for the multiple different diagnosis embeddings. The patient diagnosis can be provided in one or more mediums such as a document, image, video, sound, and/or any other medium through which a diagnosis can be communicated. For instance, the patient diagnosis can be embodied in a sentence or classification that is printed onto a document that includes additional detailed information about the patient diagnosis. The additional details can be provided from the trained model, at least based on correlations between the patient diagnosis and the key embeddings and/or value embeddings accessible to the trained model.

<FIG> illustrates a method <NUM> for providing a patient diagnosis using a multimodal deep memory network that provides weights for certain data according to an amount of attention previously given to the data when creating a previous patient diagnosis. The method <NUM> can be performed by one or more computing devices, network devices, and/or any other device capable of performing functions of a neural network. The method <NUM> can include a block <NUM> of accessing a first electronic health record corresponding to a patient diagnosis previously provided by a clinician. The first electronic health record can be an electronic document that includes headings and text (e.g., sentences) that identify the patient diagnosis, as well as information related to the patient diagnosis such as, for example, symptoms and test results. The first electronic health record can be stored in a database on a computing device that is separate from, or the same as, the computing device that operates the multimodal deep memory network. Furthermore, the first electronic health record can be authored by the clinician or generated through an automated process that compiles the first electronic health record from a variety of sources. For instance, an electronic health record can be generated through an automated process that uses historical patient data from a variety of sources in order to generate a single file of health data for a patient.

The method <NUM> can further include a block <NUM> of accessing a first image associated with the patient diagnosis. Each of the first electronic health record and the first image can correspond to a patient, or multiple patients, that received the patient diagnosis. For instance, the patient diagnosis can be Alzheimer's disease and the first image can be data from a magnetic resonance imaging (MRI) scan of the patient. In some implementations, the first image can include labels or other text that identify portions of the first image. Alternatively, the first image can be associated with metadata that can provide descriptions about different portions of the image, and/or identify the clinician, patient, and/or any other data relevant to the patient diagnosis. The first image can be provided from the same source as the first electronic health record, or a different source (e.g., the computing device that controls the MRI scanning).

The method <NUM> can also include a block <NUM> of determining a first attention metric for the first electronic health record and a second attention metric for the first image. The first attention metric and the second attention metric can be data that quantifies an amount of attention or consideration given to the first electronic health record and the second attention metric respectively. For instance, the first attention metric and the second attention metric can be based on sensor data provided by a camera of a computing device. An application at the computing device can receive signals from the camera and use the signals to identify eye movements of a user of the computing device. The application can determine an amount of time spent looking at particular documents and/or images, such as the first electronic health record and the first image.

The first attention metric and the second attention metric can be based on the eye movements of a user that was observing each of the first electronic health record and the first image, respectively. In some implementations, the computing device can track amounts of time that the first electronic health record and the first image were opened at the computing device or a separate computing device. These times can be used as a basis for generating the first attention metric and the second attention metric. In this way, the multimodal deep memory network can assign weights to key embeddings and/or value embeddings according to the consideration given to them. For instance, the first attention metric and the second attention metric can indicate that a clinician spent more time viewing the first image before generating a diagnosis than looking at the first electronic health record (e.g., a medical history document). As a result, a weight of a key embedding and/or value embedding corresponding to the first image can reflect the consideration given to the first image compared to the electronic health record.

The method <NUM> at block <NUM> can include modifying a database to identify a correlation between the patient diagnosis and the first electronic health record and/or the first image, at least based on the first attention metric and/or the second attention metric. The database can be part of a memory module incorporated into the multimodal deep memory network. The database can include key embeddings and value embeddings corresponding to one or more electronic health records and/or images from one or more sources. Furthermore, the database can include medical reference data provided from multiple different sources such as, for example, websites, electronic documents, and/or any other sources for medical data.

The method <NUM> at block <NUM> can include receiving a second electronic health record and a second image, each associated with a patient that is different from the patient associated with the first electronic health record and the first image. The second electronic health record and the second image can be processed by an input module of the multimodal deep memory network in order to generate key embeddings and/or value embeddings. The method <NUM> at block <NUM> can include determining that the second electronic health record and/or the second image include similar content to the first electronic health record and/or the first image, respectively. Block <NUM> can be performed by comparing key embeddings and/or value embeddings. Furthermore, block <NUM> can be performed by processing weighted connections between key embeddings and/or value embeddings.

The method <NUM> can further include a block <NUM> of providing, by the multimodal deep memory network, the patient diagnosis for the different patient at least based on the first attention metric or the second attention metric. For instance, the first image can be correlated to key embeddings and/or value embeddings that identify the patient diagnosis, and the second attention metric can indicate that the clinician spent more time observing the first image than the first electronic health record when making the patient diagnosis. Therefore, the patient diagnosis provided to the different patient can be based on an amount of consideration given by experts to particular medical data. Furthermore, the multimodal deep memory network leverages the attention metrics in order to improve confidence between queries input to the multimodal deep memory network and patient diagnoses output by the multimodal deep memory network.

Storage subsystem <NUM> stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem <NUM> may include the logic to perform selected aspects of method <NUM>, method <NUM>, the multimodal deep memory network <NUM>, and/or to implement one or more of the remote device <NUM>, server device <NUM>, controller <NUM>, and/or any other implementation discussed herein.

All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents, and/or ordinary meanings of the defined terms.

Only terms clearly indicated to the contrary, such as "only one of' or "exactly one of," or, when used in the claims, "consisting of," will refer to the inclusion of exactly one element of a number or list of elements.

Claim 1:
A method performed by one or more processors, the method comprising:
generating a first set of embeddings (<NUM>) from a medical image (<NUM>) of a patient received at a first neural network, the first set of embeddings (<NUM>) including first key embeddings corresponding to image features and first value embeddings corresponding to labels or text associated with the image;
generating a second set of embeddings (<NUM>) from an electronic health record (<NUM>) of the patient received at a second neural network, the second set of embeddings (<NUM>) including second value embeddings corresponding to a section heading of the electronic health record and second key embeddings corresponding to content that is separate from the section heading;
applying the first set of embeddings and the second set of embeddings as input across a trained multimodal deep memory network (<NUM>) that includes a key-value memory (<NUM>) storing multiple different diagnosis embeddings including third key embeddings (<NUM>) and third value embeddings (<NUM>) that are associated with weights that are based at least in part on an amount of attention given by a clinician to a portion of medical data including electronic health records and/or medical images from which the third key embeddings and the third value embeddings are generated, the amount of attention being determined from obtained attention-related data indicating how much attention the clinician gave to the respective portion of medical data;
generating weights for the multiple different diagnosis embeddings based on a correlation between the first set of embeddings and the second set of embeddings, and the third key embeddings and the third value embeddings; and
providing a patient diagnosis (<NUM>) for the patient at least based on the generated weights for the multiple different diagnosis embeddings.