Title: An Investigation of Memorization Risk in Healthcare Foundation Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Healthcare

evaluation

foundation model

memorization
Abstract: Foundation models trained on large-scale de-identified electronic health records (EHRs) hold promise for clinical applications. However, their capacity to memorize patient information raises important privacy concerns. In this work, we introduce a suite of black-box evaluation tests to assess privacy-related memorization risks in foundation models trained on structured EHR data. Our framework includes methods for probing memorization at both the embedding and generative levels, and aims to distinguish between model generalization and harmful memorization in clinically relevant settings. We contextualize memorization in terms of its potential to compromise patient privacy, particularly for vulnerable subgroups. We validate our approach on a publicly available EHR foundation model and release an open-source toolkit to facilitate reproducible and collaborative privacy assessments in healthcare AI.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a suite of tests to study memorization of foundational models in EHR prediction tasks. Some initial experiments are reported.\n\nSTRENGTHS AND WEAKNESSES:\nAbout the general problem setup:\n\nThe paper tackles an interesting problem of studying the performance of foundational medical coding models and their tendency to be biased under several measures.\n\nThe paper is concerned with the models that reveal sensitive information by memorization. I am not sure if I can fully agree with this concern. Generating a trajectory of treatments with a statement that the greater the input information value, the more likely the model is to create the patient trajectory or sensitive output. Isn’t this contradictory? Wouldn’t we want the model to provide accurate trajectories with precise background information? Isn’t it the whole point that medical treatments should follow the prognosis and medical history, and each patient should get the most fitted treatment trajectory — thus, if the information is exactly the same as with another patient, then the trajectory should be identical? Surely, with less information, the model will likely make predictions that are less precise.\nFor instance, the HIV example shows that the model predicts codes that are sensitive, but it does so with some likelihood and based on valid input information. What is the problem here? I think spurious correlations will be likely to exist in any model in some quantity, but the examples given did not convince me that the test suite would reveal something that we could not expect. \n\nResults and their significance\n\nIn general, the manuscript does not communicate the severity of the problem, and the results are not discussed and compared across different models, methods to control for bias, and spurious correlation controls. Many of the results are unclear. For instance, T2: not clear what exactly are the results presented in Table 1 and how they have been computed. The same issue is with the tables in the Appendix. \n\nThe paper has a great initiative, but I think that the identification of the actual problems that may arise need to be sharpened and the empirical results need to be strengthened to include envidence of the tests capturing effects when the problems are controlled (e.g. test a model where sensitive controls are present in training, finetuning or post-processing to improve interpretability vs. a model where these are not present).\n\nQUESTIONS:\nIs there empirical evidence that the tests actually reflect differences between models that are controlled for memorization, sensitive attributes, or spurious correlations? \n\nAre there analyses with a more substantial set of prompting and cases? The present tests seem to be limited.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 3, "DATE": "2025-06-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a suite of tests to detect memorization in Foundation Models trained on structured EHR (Electronic Health Record) data. The goal of the authors is to quantify the risk of LLMs leaking private information. \n\nThe authors create 6 tests and validate them on EHRMamba, a prompt-based model trained on MIMIC-IV. They prompt the model to generate patient medical code sequences varying the amount of ground truth information given to the model in the input prompt. Four of the tests (T1, T2, T5 and T6) use directly the code sequences generated by the EHRMamba, while the other two (T3 and T4) use the embeddings generated by the LLM.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n## 1. Critical problem addressed \n\nThe paper addresses the very relevant yet usually neglected problem of Large Language Models generating patient private information that they were trained on. A working toolkit to quantify and detect leaked information could become a standard diagnostic before releasing medical LLMs.\n\n## 2. Clear structure \n\nThe problem of memorization in healthcare LLMs is introduced clearly and the overall paper is well structured and easy to follow. \n\n## 3. Sound testing framework and setup \n\nThe definitions of generative and embedding memorization are sensible, and the proposed testing setup of varying the amount of private information shown to the model is a good starting point to test for leaked training data.  \n\n## 4. Original metric use \n\nThe use of Wasserstein distance to quantify the similarity between generated and real patient trajectories is interesting and original. \n\n# Weaknesses \n\n## 1. No positive-control validation and anecdotal evidence of tests working\n\nThere is no statistical evidence that the tests would actually detect cases of memorization.  \n\n### T1\n\nT1 is the only test that is stress-tested, but even then, only one synthetic perturbation example is shown in Appendix A. We don’t know if the scores shown would generalize or are just incidental.\n\n### T2\n\nFor T2, another single anecdotal example is presented in lines 211-213: “given a prompt about a 48-year-old individual transferred to the ED with a history of falls and discharged shortly after—representing minimal clinical detail—the model predicts alcohol abuse in 31% of the estimated trajectories.” How do we know that this is a real failure to begin with and not the model capturing underlying valid patterns in the data?\n\n### T3\n\nIn T3, in lines 236-237, the authors state: “AUROC values across all sensitive attributes remain around 0.5, indicating no clear memorization signal and suggesting random performance.” However, no positive example is presented. How do we know that the metric will work on positive examples? \n\n### T4\n\nIn T4, no positive example is shown. Additionally, Figure 3b is analyzed and the difference between both distributions in it is considered not statistically significant, but no statistical test is mentioned. \n\n### T5\n\nIn T5, in lines 283-285, again the authors state that “If the model stops producing sensitive information after altering the identifiers, it signals that the original output resulted from memorizing an individual instance.” but no proof is provided, and only negative examples are shown. \n\n### T6\n\nIn T6, two positive examples are presented in lines 314-318, but even the authors are unsure whether these are valid examples that show the metric working, as they state: “If statistical associations cannot explain these 318 predictions, they reflect memorization failures.”\n\n## 2. Experiments on single model\n\nExperiments have been conducted on one model + one dataset. Generality is untested.\n\n## 3. No mitigation proposed\n\nFramework stops at detection and no mitigation strategy is offered.\n\n## 4. Code not available\n\nThe code shows as not available to me, but this might be a technical issue unrelated to the authors.\n\nQUESTIONS:\n# Questions/Suggestions\n\n1. Add a positive-control benchmark for each test\n\nThis is the main concern of the paper. For this method to be generalizable and used out-of-the-box in any medical LLM, there has to be evidence that these metrics are actually doing something and are able to detect memorization. For the paper to be accepted, there has to be statistical evidence for every test discriminates between positive and negative examples. There might be many ways of proving this, but one could be to overfit a model so that it always leaks information.\n\nIn general, claims should be backed up with more statistical rigor.\n\n2. Demonstrate generality\n\nIt would be ideal to run the experiments on at least another medical model. However, without fixing point 1, these experiments wouldn't have value.\n\n3. Present the prompts used\n\nThese might be in the attached code, but I couldn't access it. Regardless, it would be useful to have them in the paper too, so that results are reproducible and can be better analyzed.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a testing framework designed to evaluate the memorization of private and sensitive health information in models trained on structured electronic health record (EHR) data. The assessment is divided into three categories: generative memorization, embedding memorization, and privacy risk analysis. In the generative memorization test, the authors investigate whether models can regenerate training data through prompting, including whether sensitive details can be reconstructed. The embedding memorization test explores whether the model's embedding space leaks sensitive information or reveals membership in the training dataset. Finally, the privacy risk assessment focuses on individual-level memorization, evaluating whether specific samples reveal personal details or general medical conditions, and whether certain groups are more susceptible to memorization.\n\nSTRENGTHS AND WEAKNESSES:\nThe proposed framework is conceptually straightforward and the tests are intuitive, making the approach accessible and potentially widely applicable. The authors evaluate the framework on a single model, EHRMamba2, and report no strong evidence of memorization. While the methodology and results are generally convincing, a key limitation is the absence of a known positive control, which makes it difficult to verify the sensitivity of the tests in detecting memorization. Additionally, evaluating only one model limits insight into how factors like model size might influence memorization behavior.\n\nSince the framework only requires a sequence of codes as input prompts, it could be adapted for use beyond healthcare. Given the challenges in accessing healthcare data, the authors could strengthen their claims by first demonstrating the framework’s effectiveness in a non-healthcare domain with a known ground truth. This would help establish the framework’s ability to detect memorization before applying it to EHR-trained models.\n\nQUESTIONS:\n1. Can you demonstrate the framework can capture memorization when it exists?\n\n**Formatting Issues**\n- L129: Double comma after [23]\n- Table1 caption: Double period.\n- Figure4 caption: … vertical line … → … red vertical line …\n\nLIMITATIONS:\nMissing a broader discussion about the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a framework for evaluating memorization risks in healthcare foundation models (EHR-FMs) trained on de-identified electronic health records (EHRs). The authors highlight concerns that these models, despite being trained on de-identified data, might still memorize and expose sensitive patient information. The proposed framework includes a suite of black-box evaluation tests, categorized into two main objectives:\n\n1.  Measuring Memorization (Objective I): This objective involves four tests (T1-T4) to quantify the extent to which models reveal training data through both generative and embedding memorization.\n2.  Evaluating Risk of Memorization (Objective II): This objective focuses on distinguishing between patient-level memorization and population-level generalization to assess actual privacy risks. It includes two tests (T5-T6).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe paper presents a comprehensive and much-needed framework for evaluating memorization risks in healthcare foundation models (EHR-FMs). \nQuality: The paper introduces a well-structured suite of six black-box evaluation tests (T1-T6) specifically designed for EHR-FMs, addressing both generative and embedding memorization. \nOpen-Source Toolkit: The release of an open-source toolkit facilitates reproducible and collaborative privacy assessments, which is vital for advancing research and practical application in healthcare AI.\nFocus on Vulnerable Subgroups: The inclusion of T6, which investigates memorization risks for specific, vulnerable subgroups (e.g., elderly, rare diagnoses), demonstrates a thoughtful consideration of ethical implications and potential disparities in privacy risks.\nClarity: The paper clearly outlines its two main objectives: quantifying different types of memorization and assessing associated privacy risks by distinguishing harmful memorization from benign generalization.\nSignificance: The framework's focus on identifying practical privacy threats in black-box EHR-FMs is highly significant, given the sensitive nature of health data and the potential for real-world harm from data leakage.\nGuidance for Model Developers: The proposed tests offer concrete methods for developers to identify vulnerable samples and subgroups, enabling proactive mitigation strategies (e.g., post-training safety layers, red-teaming, retraining) before public release of models.\nOriginality: Novel Framework for EHR-FMs: While memorization has been studied in LLMs , this paper is one of the first comprehensive investigations into memorization risks specifically for structured EHR data and EHR-FMs.\n\n** Weaknesses ** \nBenchmark Model Specificity: The evaluation is primarily conducted on EHRMamba2 , which is a specific type of EHR-FM. While this is a practical choice due to its public availability and reproducible training , the generalizability of the findings to other EHR-FM is not fully explored, although the framework is stated to be adaptable.\nDifficulty in Rigorous Assessment of Metrics and Tests: e.g. in T2, the specific codes within these categories and what constitutes \"sensitive information require expert clinical judgment and could be subject to varying interpretations across different evaluation teams or clinical contexts; in T1, While the d metric quantifies similarity, determining what level of distance constitutes a \"memorized\" or \"harmful\" trajectory in a clinically meaningful way is not explicitly defined and could be difficult to standardize across different evaluations.\n\nQUESTIONS:\n1. For the probing models in T3, what specific types of classifiers were trained on the embeddings, and what were their architectures and training procedures? How was the \"separate test cohort\" for training the probing model selected to ensure its representativeness and independence from the training data of the EHR-FM?\n2. Could the authors provide more detail on the specific \"N-codes\" used for prompting in T1 and T2? What kind of codes were these (e.g., common diagnoses, specific lab results, a mix)? How were they selected to ensure they represent realistic adversarial prompts, and what was the average length or complexity of these prompts in a real-world scenario?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}