A method performed by one or more processors, the method comprising: generating a first set of embeddings from a medical image of a patient received at a first neural network, the first set of embeddings including first key embeddings corresponding to image features and first value embeddings corresponding to labels or text associated with the image; generating a second set of embeddings from an electronic health record of the patient received at a second neural network, the second set of embeddings including second value embeddings corresponding to a section heading of the electronic health record and second key embeddings corresponding to content that is separate from the section heading; applying the first set of embeddings and the second set of embeddings as input across a trained multimodal deep memory network that includes a key-value memory storing multiple different diagnosis embeddings including third key embeddings and third value embeddings that are associated with weights that are based at least in part on an amount of attention given by a clinician to a portion of medical data including electronic health records and/or medical images from which the third key embeddings and the third value embeddings are generated, the amount of attention being determined from obtained attention-related data indicating how much attention the clinician gave to the respective portion of medical data; generating weights for the multiple different diagnosis embeddings based on a correlation between the first set of embeddings and the second set of embeddings, and the third key embeddings and the third value embeddings; and providing a patient diagnosis for the patient at least based on the generated weights for the multiple different diagnosis embeddings. generating a first set of embeddings from a medical image of a patient received at a first neural network, the first set of embeddings including first key embeddings corresponding to image features and first value embeddings corresponding to labels or text associated with the image; generating a second set of embeddings from an electronic health record of the patient received at a second neural network, the second set of embeddings including second value embeddings corresponding to a section heading of the electronic health record and second key embeddings corresponding to content that is separate from the section heading; applying the first set of embeddings and the second set of embeddings as input across a trained multimodal deep memory network that includes a key-value memory storing multiple different diagnosis embeddings including third key embeddings and third value embeddings that are associated with weights that are based at least in part on an amount of attention given by a clinician to a portion of medical data including electronic health records and/or medical images from which the third key embeddings and the third value embeddings are generated, the amount of attention being determined from obtained attention-related data indicating how much attention the clinician gave to the respective portion of medical data; generating weights for the multiple different diagnosis embeddings based on a correlation between the first set of embeddings and the second set of embeddings, and the third key embeddings and the third value embeddings; and providing a patient diagnosis for the patient at least based on the generated weights for the multiple different diagnosis embeddings. <EOS>