Title: Detecting Data Deviations in Electronic Health Records
Venue: NeurIPS 2025
Accepted: yes
Keywords: Electronic Health Records

Data Deviations
Abstract: Data deviations in electronic health records (EHR) refer to discrepancies between recorded entries and a patient’s actual physiological state, indicating a decline in EHR data fidelity. Such deviations can result from pre-analytical variability, documentation errors, or unvalidated data sources. Effectively detecting data deviations is clinically valuable for identifying erroneous records, excluding them from downstream clinical workflows, and informing corrective actions. Despite its importance and practical relevance, this problem remains largely underexplored in existing research. To bridge this gap, we propose a bi-level knowledge distillation approach centered on a task-agnostic formulation of EHR data fidelity as an intrinsic measure of data reliability. Our approach performs layered knowledge distillation in two levels: from a computation-intensive, task-specific data Shapley oracle to a neural oracle for individual tasks, and then to a unified EHR data fidelity predictor. This design enables the integration of task-specific insights into a holistic assessment of a patient’s EHR data fidelity from a multi-task perspective. By tracking the outputs of this learned predictor, we detect potential data deviations in EHR data. Experiments on both real-world EHR data from National University Hospital in Singapore and the public MIMIC-III dataset consistently validate the effectiveness of our approach in detecting data deviations in EHR data. Case studies further demonstrate its practical value in identifying clinically meaningful data deviations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of detecting EHR discrepancies between recorded values and the patient’s true physiological state (ie \"data deviation\") in Electronic Health Records (EHRs). They propose a multi-step framework to estimate whether the data is 'good' based on first training task-specific models (using Shapley values) to measure how much each data point contributes to downstream clinical tasks and then then combining those models into a single, general-purpose predictor. This final model assigns a \"fidelity score\" to new data points. If that score drops sharply after a small change to the input, it signals that the data may be unreliable. The method is tested on both private and public (in the appendix) datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- This area (identifying and potentially addressing incorrect input values) is very valuable for every domain, especially including healthcare which has many opportunities for noise to occur.\n- Section 4.4's case studies did a very good job explaining how one could identify data quality issues (eg two labs that should probably move together actually end up diverging in their series).\n- I appreciated seeing baselines to compare against to illustrate the task & alternative solutions.\n\n\nWeaknesses\n- This area, while important, is hard to measure whether particular contributions are meaningful. For instance, the corruptions to the data you chose might not be meaningful or generalizable to other data collection issues.\n- Some of the baselines were weak. Many are designed to identify global outliers, rather than the e.g. multi-lab time series correlations illustrated in the case study. I would’ve liked to see something like “real vs corrupted” classifier trained to predict actual data points (label=0) vs the corrupted data points you create (label=0).\n\nMinor Weaknesses\n- The writing in the paper often used technical jargon (which made it hard to follow on first pass) when simpler explanations could've worked just as well if not better. As an illustration, instead of \"Knowledge is distilled from the data Shapley oracle  𝑂 𝑑 𝑠 O  ds ​  … into a corresponding neural oracle  𝑂 𝑛 𝑛 O  nn ​  … trained jointly in a multi-task learning setting.\" I would have benefitted from seeing early on \"We train a set of models to mimic the Shapley values efficiently, one for each prediction task.\"\n- Until Section 4.4 I was a little confused about the inputs/outputs of the task setup (eg is it predicting the fidelity of a single data element vs a batch of results vs something else). This could have been made more clear earlier on.\n\nQUESTIONS:\n- None of the models are directly looking at the time series, though such information would probably be very valuable for the underlying question of whether the collected data is a valid reflection of the patient’s state. Was this considered? If so, why was it not leveraged?\n- Intuitively why were the elements of this model combined? For instance, why did you think to use Shapley values as a first step?\n- Am I correct in my understanding that this approach does not verify whether the EHR data is an accurate measurement of patient physiology but rather it measures whether the data appears useful in downstream models? These are not always the same. Misleading data can sometimes improve task predictions if it correlates spuriously with outcomes (e.g., due to confounding).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the novel problem of detecting data deviation in Electronic Health Records, which has implications on the reliability of patient data for downstream tasks. They propose a 2-step knowledge distillation approach that calculator Shapley values for individual tasks, and eventually learns a general neural model for data valuation. They use controlled deviation injection to show the ability for their method to identify anomalies on data from both public and private EHR records.\n\nSTRENGTHS AND WEAKNESSES:\nThe authors study a very interesting and novel problem, and the paper is very well-written. The overall approach considering Shapley scores makes intuitive sense, and the experiments are thorough across several interesting tasks. \n\nSome weaknesses include: \n\nW1. Lack of studying ablations of the proposed method (e.g. removing KD steps) versus just comparing against other baselines. Furthermore, are there any simple rule-based approaches leveraging the domain knowledge that could be used as a baseline? \n\nW2. Clarification whether the primary goal is to remove data for the purpose of downstream prediction, or other uses. Since the method only outputs an anomaly score,  it doesn’t help shed light on why any deviation occurred. The case studies in the paper were really useful, but primarily seemed to highlight an issue with duplication: meanwhile, the paper’s introduction highlights a wider source of data deviation. Could the authors provide more insights from the data on the types of deviation error?\n\nW3. Discuss any limitations around removing data deviation points – similar to arguments around how incorrect data in LMs are actually useful for calibration and misinformation detection (or training on toxic text helps LMs know toxic vs. non-toxic and generate less toxic text), what if a model built on EHR data might actually benefit from being aware of potential data deviations? Are there concerns around robustness?\n\nQUESTIONS:\nOne might ask if the goal is to train a good model on EHR data, then can we just assume that data deviation is random noise that a good model should ignore? Are there any experiments you could run outside of controlled injection studies to show that this actually leads to a performance difference?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the under-explored challenge of data deviation, discrepancies between recorded EHR entries and a patient’s true physiological state, which diminishes data fidelity and jeopardises downstream clinical analytics. It models fidelity as a task-agnostic property and introduces a bi-level knowledge-distillation pipeline: task-specific data-Shapley values (the “Data Shapley Oracle”) are first approximated by neural oracles and then fused into a unified predictor Ψ for patient-level fidelity estimation. Monte-Carlo Shapley sampling combined with neural amortisation slashes the exponential cost of exact valuation, making near-real-time fidelity scoring feasible on large healthcare datasets. An anomaly is flagged whenever Ψ registers a fidelity drop beyond a learned threshold, enabling alerts for documentation errors, specimen-handling issues, or unreliable wearable-device readings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The paper tackles the important problem of detecting anomalies in EHR data.\n- The manuscript is clearly written and generally easy to follow.\n\nWeaknesses\n\n- The “Data Shapley Oracle” is produced via permutation sampling rather than by computing exact Shapley values. This approximation can introduce non-trivial error that may be further amplified when distilled into the proposed Neural Oracle. The paper should report sanity checks that quantify this gap before treating the oracle as ground truth.\n- The amortized Shapley framework may fail under domain shift. Out-of-distribution samples, common in healthcare, where each clinical sub-domain is highly specialized, could yield unreliable value estimates and undermine the proposed approach.\n- Definition 1 specifies a deviation threshold, yet in practice, this threshold behaves like a tunable hyperparameter. In real clinical settings, such cut-offs are hard to set, raising questions about whether the fidelity metric can robustly flag true abnormalities in EHR data.\n- The data fidelity used in this paper for detecting data deviation is highly dependent on the provided value function. However, the paper lacks explanation and analysis of the sanity of the value function. If the value function losts its sanity in providing prediction, then the entire scoring system will be not realiable and meaningless. The paper is recommended to include either empirical assessment or domain experts evaluation for sanity check.\n\nQUESTIONS:\nPlease refer to the weakness.\n\nLIMITATIONS:\nPlease refer to the weakness.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}