Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHR Data
Authors: Michael Wornow
Suhana Bedi
Miguel Angel Fuentes Hernandez
Ethan Steinberg
Jason Alan Fries
Christopher Re
Sanmi Koyejo
Nigam Shah
Venue: ICLR 2025 Poster (2025)
Keywords: ehr
foundation model
long context
clinical prediction making
healthcare
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: An analysis of the impact of context length on foundation models trained on structured electronic health record (EHR) data for clinical prediction tasks.

Abstract: Foundation Models (FMs) trained on Electronic Health Records (EHRs) have achieved state-of-the-art results on numerous clinical prediction tasks. However, prior EHR FMs typically have context windows of $<$1k tokens, which prevents them from modeling full patient EHRs which can exceed 10k's of events. For making clinical predictions, both model performance and robustness to the unique properties of EHR data are crucial. Recent advancements in subquadratic long-context architectures (e.g. Mamba) offer a promising solution. However, their application to EHR data has not been well-studied. We address this gap by presenting the first systematic evaluation of the effect of context length on modeling EHR data. We find that longer context models improve predictive performance -- our Mamba-based model surpasses the prior state-of-the-art on 9/14 tasks on the EHRSHOT prediction benchmark. Additionally, we measure robustness to three unique, previously underexplored properties of EHR data: (1) the prevalence of ``copy-forwarded" diagnoses which create artificial token repetition in EHR sequences; (2) the irregular time intervals between EHR events which can lead to a wide range of timespans within a context window; and (3) the natural increase in disease complexity over time which makes later tokens in the EHR harder to predict than earlier ones. Stratifying our EHRSHOT results, we find that higher levels of each property correlate negatively with model performance (e.g., a 14% higher Brier loss between the least and most irregular patients), but that longer context models are more robust to more extreme levels of these properties. Our work highlights the potential for using long-context architectures to model EHR data, and offers a case study on how to identify and quantify new challenges in modeling sequential data motivated by domains outside of natural language. We release all of our model checkpoints and code.

Published as a conference paper at ICLR 2025
CONTEXT CLUES: EVALUATING LONG CONTEXT MODELS FOR CLINICAL PREDICTION TASKS ON EHRS
Michael Wornow"l Suhana Bedi Miguel Angel Fuentes Hernandez's Ethan Steinberg' Jason Alan Fries', Christopher Ré', Sanmi Koyejo', Nigam H. Shah Stanford University Prealize Health
ABSTRACT
Foundation Models (FMs) trained on Electronic Health Records (EHRs) have achieved state-of of-the-arrr art results on numerous clinical prediction tasks. However, most existing EHR FMs have context windows of <Ik tokens. This prevents them from modeling full patient EHRs which can exceed 10k's of events. Re- cent advancements in subquadratic long-context architectures (e.g.. Mamba) of- fer a promising solution. However, their application to EHR data has not been well-studied. We address this gap by presenting the first systematic evaluation of the effect of context length on modeling EHR data. We find that longer context models improve predictive performance our Mamba-based model surpasses the prior sfate-of-the-ar on 9/14 tasks on the EHRSHOT prediction benchmark. For clinical applications, however, model performance alone is insufficient - robust- ness to the unique properties of EHR is crucial. Thus, we also evaluate models across three previously underexplored properties of EHR data: (1) the prevalence of "oopy--frwwrded" diagnoses which creates artificial repetition of tokens within EHR sequences; (2) the irregular time intervals between EHR events which can lead to wide range of timespans within context window; and (3) the natural increase in disease complexity over time which makes later tokens in the EHR harder predict than earlier ones. Stratifying our EHRSHOT results, we find that higher levels of each property correlate negatively with model performance (e.g.. a 14% higher Brier loss when making predictions for the most versus least irregular patients), but that longer context models are more robust to more extreme levels these properties. Our work highlights the potential for using long-context architectures to model EHR data, and offers case study for identifying new chal- lenges modeling sequential data motivated by domains outside of natural lan- guage. We release our model checkpoints and code at: hhtps://gitub..co//sn fiabllbbllog.ccnnex...o
INTRODUCTION
Foundation Models (FMs) (Bommasani et al. 2021) trained on Electronic Health Records (EHRs) have achieved state-of-the-art results on numerous clinical prediction tasks (Odgaard et al.) 2024] Yang et al. 2002)). Such models can improve patient outcomes via early detection of disease and risk stratification (Steinberg et al. 2023). As an EHR is simply a list of chronologically-orkee events (see Figure (1), it can be modeled as sequence of tokens. Instead of subwords or image patches, however, tokens represent clinical events like diagnoses and procedures (McDermott et al., 2023). This approach has enabled the application of transformer architectures originally developed
clinical
for natural language processing (NIP) such as BERT (Rasmy et al. 202 [. et al. 2020; Odgaard
et al. 2024) and GPT (Steinberg et al. 2021 Pang et al. 2024 Kraljevic et al.) 2024) EHR data.
A critical choice in FM design is context length i.e. how many tokens of input the model can ingest. Longer context lengths have shown a consistent positive impact on FM performance across various domains by enabling models to reference and reason over more information (Xiong et al.) 2023 Given the typical hospital's limited compute resources, however, transformer- transformer-based based EHR FMs have been limited to processing short context lengths (i.e., 512 tokens) due to the quadratic scaling of attention with input length (Vaswani et al., 2017). As a single patient's EHR can contain 10k's of tokens, this greatly limits the amount of data that EHR FMs consider. This is especially

---

Published as a conference paper at ICLR 2025
A putient's EHi la timelins of citrical everts
For cortain models, context longth improves moon AUROC across EHRBHOT lasks
ALROC
Freecams
0.77
0.79
0.35
can transform the Enii into aequence toker modeling
1yea
0.11
Farda
cONDOnT pecitoo soctamos concisens coccasaaso routeraa couns saa NOTTEEE
2000
4003
6003
5000
12000
12003
EHR has urique properties distinet natural language
properties EHR
Natural Language
KOIELLICOOINTE VNRICOENED KORKOUELE
nour Hard Charciee
impostant cilveres
diagrosss
billing pursocers
parpossen
Eeer
Irregueer intervale between sokess
and
Iugplet Andy
Redusodd value
PPL
prior takera due Increated comp exity over dime
ICOODEE
The central claims of this paper. (a) EHRS are sequences: An EHR is simply a timeline of clinical events that occur to patient, and thus can be naturally represented sequence of tokens. (b) Long context improves performance: AUROC context lengths, with Hyena (red) being the notable exception. Overall, Mamba (green) at a context length of 16k achieves the highest average AUROC across 14 diverse clinical prediction tasks. (c) EHR data has distinct properties: In contrast to natural language, EHR data has unique properties whose implications remain under-explored in the ML literature. Here, we highlight three such attributes copy- forwardig..,.gg.. irregular time intervals between tokens, and discase progression. (d) EHRs properties present unique modeling challenges: Stratifying patients by the degree to which they exhibit each EHIR-specifi property. we find that higher Brier scores (i.e., worse model performance) are associated with patients who have more repetitive (top) or irregular (middle) EHRs. Additionally, the perplexity of tokens later in patient's 's timeline tends to be higher, even when conditioning prior tokens (bottom).
Figure I:
clinical prediction tasks tends to increase with longer
true for the sickest patients i.c. the ones of most interest to a hospital for prediction tasks as they typically have high healthcare utilization and thus have very long timelines, as can be seen in the CDF plots of patient sequence length in Appendix Figure 6
Recently developed subquadratic architectures such as Mamba (Gu & Dao 2024 and Hyena (Poli et al. 2023a) that are optimized for long contexts offer a potential solution. As EHR FMs begin driving real-world care decisions, is essential to better understand the implications of adapting these long context architectures for clinical prediction making.
However, their effectiveness on EHR data remains unclear. In contrast to natural language, EHR data exhibits specific types of token repetition and noise that complicate the expected benefits of longer contexts. We identify and present the first quantitative analysis of three such underexplored properties, outlined in Figure :::
1. Copy-forwarding key diagnoses are repeated across multiple visits due to billing prac- tices, leading to artificial repetition of tokens in the EHR (Thornton et al. 2013
2. Irregular time intervals between tokens - unlike in natural language where consecutive tokens are trivially 1 position apart, consecutive clinical events can be days years apart, thus creating a wide range of timescales within a single context (McDermott et al.) 2023).
3. Disease progression - later tokens in a patient's timeline are harder to predict as discase complexity tends to increase with age (Fabbri al. 2015), when conditioning on
2

---

Published as a conference paper at ICLR 2025
prior tokens; this contrasts with natural language, in which later tokens in a prompt tend to exhibit lower perplexities (Peng et al., 2023b).
While several papers have introduced transformer-basee EHR FMs, they typically only evaluate at a single context length of 512 tokens, as shown in Table I Evaluations of subquadratic architectures on EHR data have also been limited to one context length and do not consider "longitudinal" (i.c. full-length) EHRs (Fallahpour et al. 2024). To knowledge, there has been systematic evalua- tion of the impact of context length on state-of-the-an transformer and non-transformer architectures trained on longitudinal EHR data for clinical prediction tasks.
To address these gaps in the literature, our paper makes the following three contributions:
State-of-the-art (SOTA) Clinical Prediction Making with Subquadratic Architectures: We train and evaluate two transformer- based - GPT (Brown et al. 2020) and Llama TTem 2024) and two subquadratic Mamba (Gu & Dao 2024) (state space models) and Hyena (Poli et al, 2023a) (long convolutions) - architectures. We are among the first to train the latter three at the scale of millions of patients' EHRs. We achieve SOTA AUROC scores on 9/14 tasks from the EHRSHOT clinical prediction benchmark using a Mamba-based model. These results highlight the potential for subquadratic models to process EHR data.
Increased Performance with Longer Contexts: We evaluate the impact of context length (ranging from 512 to 16k tokens) on 14 clinical risk prediction tasks. As shown in Fig- ure [p. model performance tends increase with longer contexts (with the exception of Hyena, whose performance degrades sharply). While we observe smaller gains than in other fields, these results represent a first step towards improved clinical prediction making by leveraging larger amounts of medical history.
Quantifying Difficulties in Modeling EHRs Natural Language: Beyond AUROC, we measure how 3 EHR- speciicccfic properties - copy copy-forwarding.. irregular inter-token time intervals, and disease progression impact models at different context lengths. As shown in Figure Td. these E.R--specffic properties negatively correlate with model performance, e.g. patients with the most irregular timelines achieve a Brier score 14% worse than pa- tients with the least irregular timelines. However, we find that longer context models are more robust to patients exhibiting higher degrees of these properties.
Our work aims to realize the benefits of long context models in healthcare. More broadly. as se- quence modeling architectures designed for natural language increasingly applied to external domains such as molecular sequences (Nguyen et al. 2023a 2024), climate (Bodnar et al.) 2024 Nguyen al. 2023b), and time series (Cohen et al.| 2024)) we hope analysis serves general blueprint for taking a data-centric lens on adapting such models for non-NLP domains. We release the full weights of our pretrained models on HuggingFace and our code the Github repo context_cluee
here: http:s//gthbbb.ccmm///mmaaaaal//
2 BACKGROUND
In this section, motivate the application of long-context foundation models electronic health record data and summarize related work.
2.1 FOUNDATION MODELS FOR EHRS
Foundation Models (FMs) are large-scale deep learning models trained on extensive amounts of unlabeled data via unsupervised learning (Bommasani et al.] (02T). An electronic health record (EHR) provides comprehensive documentation of patient interactions with the healthcare system, including diagnoses, medications, procedures, lab results, etc. (Ambinder (005). this work, we only consider structured EHR data - i.e. ignore notes and images structured EHR data is simpler to deidentify and thus share with the community for open science (Negash et al. 2023
As seen in Table I many architectures for sequence modeling have been re-applied EHR data. Most utilize transformer based architectures such as BERT (Devlin et al.) 2019) or GPT (Brown) et al. 2020) with context length of 512. Pretrained millions of EHRS using objectives such as
3

---

Published as a conference paper at ICLR 2025
causal or masked language modeling, these EHR FMs state-of-the many clinical predic- tion tasks Yang et al. 2023 Odgaard et al. 2024 Wornow et al. 2023
Model
Context Length(s)
Architecture(s)
Subquadratic?
CETIR BERT (Pang et al
2021
300
BERT
Rasmy el
BEHRT Li atal 2020
512
BERT BERT
CORE-BEEKI Outeard al
2024
512
ForeSight KKrljevvcc
256
GPT
CLMBR (Stomber al
202
512
GPT
Pangge
GPT
ETHOS Renc ml [TA4]
2048
GPT
THanformEHR Yang ot
2025
MOTOR Steinbere
2023
512
Custom
UniHPF Fur al 2024/5
8192
Custom
GenHPF (Hur 03 (U745)
8192
Custom
EHHManbba FFllalioor et
2024
2048
Mamba
Our Work
512
16,384
Mamba, Llama. Hyean, GPT
Table I: Comparison prior work on sequence modcling for EHR data
2.2 LONG CONTEXT FMS
Context length is the number of input tokens that a model can ingest. Longer contexts have shown positively impact FM performance by enabling models reason over more information (Xiong
et al. 2023). Tooeenllvel perplexity typically decreases as context length increases, reflecting im-
proved model comprehension of longer sequences (Press al. 2022 Chen et al. 2023 Peng al.,
2023b)
Theoretically, conditioning on more of a patient's medical history should also enable better clinical decisions. Unfortunately, transformers scale quadratically with context length (Vaswani et al. 2017), which makes processing long sequences computationally expensive. This an especially important consideration for sesource-constringe hospitals hoping to deploy such models. To remedy this, subquadratic architectures optimized for processing longer contexts have been proposed (Tay al.) 2020 Wang et al. 2024). They replace the ( attention mechanism in transformers with linear or lonliinear alternatives such as state space models (Gu Dao 2024, Goel et al. 2022), long convolutions (Poli et al. 2023ab.. linear attention (Peng et al. [2023a] Katharopoulos et al. 2020), or recurrent subunits (De et al. 2024).. Despite strong results in NLP (Xu (202)) and biology (Nguyen
et al. 2023a), these architectures remain largely untested on EHR data.
2.3
RELATED WORK
The impact of context length on EHR FMs for clinical prediction tasks remains largely unexplored. Many papers have evaluated the trade-offs of BERT (Odgaard al. 2024 Rasmy et al. 2021; Li
et al. 2020) and GPT-based (Kraljevic et al., 2024; Pang et al. 2024) architectures on EHR data. However, they typically only consider one context length up to 512 tokens. In contrast, our work examines the impact of multiple context lengths up to 16,384 tokens.
These works also do not consider state-of-the-arr art subquadratic architectures. To our knowledge, only one work EHRMamba (Fallahpour et al.) (2024) has done so. However, the authors only consider a single context length 2048, and do not train or evaluate on longitudinal (i.e. full-length) EHRs, instead focusing on the more limited ICU setting. In contrast, our work evaluates Mamba (u Dao 2024) on 8x longer context lengths and longitudinal EHR tasks.
Several studies have combined fixed context window transformers with preliminary retrieval step that selects the most relevant events across a patient's entire timeline (Kim et al.) 2003] Zhu et al. 2024]. However, they only consider fixed context windows and benchmark against weaker long context models such as S4 (Gu et al. 2022) and Performer (Choromanski et al. 2022)
3 METHODS
Our goal to measure how non-transformer architectures, context length, and the unique properties of EHR data impact performance clinical prediction tasks. We pretrain 16 models across four
4

---

Published as a conference paper at ICLR 2025
architectures and six context lengths the structured EHR data of 2.5M patients. We evaluate each model on 14 binary classification tasks from the EHRSHOT benchmark (Wornow et al. 2023), as detailed in Section 3.2 We stratify our results the degree to which each patient exhibits 3 EHR- specific properties token repetition due to copy-forwarding. irregularity of time intervals between tokens, and increased complexity of tokens due disease progression which we hypothesize may influence the efficacy of longer context models,
3.1
MODEL. TRAINING
Here, we provide details on our training dataset, tokenization strategy, and model architectures.
3.1.1 PROBLEM SETUP
In this paper, we focus exclusively on the structured data within a longitudinal (i.e. full-Iength) EHR -i.e., diagnoses, medications, lab tests, procedures, visits, and other observational data. Our dataset consists of n patients X {X1.... "i....,,,,,,, ). J. For each patient i have their structured EHR data X,, which is composed of a sequence of chronologically ordered clinical events Xzy:
X; = (X11, X12. X
We refer to X; as a "patient timeline", where each clinical event is a tuple the form (tij, Cijy May). Here, by is the timestamp, Gy E C is medical code drawn from a fixed medical ontology (C), and Viy E Ve U Va U is an optional value, either categorical (Ve) or numeric (Vr):
Xij = (1ags Cijy viy)
Events are sorted by time such that 4ig (14+1) Vj. This formulation of EHR data is also referred to as the "event stream format" (McDermott et al. 2023).
For our experiments, use dataset of deidentified longitudinal EHRs sourced from academic medical center that have been formatted under the OMOP Common Data Model (Sciences & In- formatics 2021). We refer to this dataset as EHR-OMOP. We use 2.5M patients (covering 3.5B clinical events) for training, and hold out 0.5M patients as a validation set. The average patient has 1,364 total and 237 unique events, Additional information can be found in Appendix Section A
3.1.2 TOKENIZATION
Given a patient timeline X₁, must convert it into sequence of tokens T; that models can ingest. Thus, we must map each X,, = (tag. Caj. Uty) to some set of token(s) Ti, = (T.si.....Tss }- We the same vocabulary used by the prior SOTA model the benchmark EHRSHOT (Wornow et al. 2023). Each clinical "event" in a patient's timeline has a single "code" associated with it. Each "code" then gets converted into single "token" within our vocabulary via the following process. First, all unique codes e E C that occur at least once in our training dataset are assigned unique token. Second, all codes that are associated with categorical values are assigned a unique token for each possible associated categorical value. Third, all codes associated with numerical values are assigned unique token for each decile within the range of values attained in our training dataset. After sorting all tokens by their information content, the top 39811 tokens were kept as our vocabulary, and all models share this same vocabulary. Please see Appendix Section for additional details the token generation and selection process.
use for evaluation,
3.1.3 ARCHITECTURES
We evaluate four models GPT (Brown et al. 2020). Llama (Team] 2024), Mamba (Gu & Dao, 2024), and Hyena (Poli et al.) 2023a) - at the 120 million parameter scale using their default HuggingFace implementations. (see Appendix Section C for details on each architecture and Ap- pendix Table 6 for exact configurations). We evaluate each model across various context lengths L E L, with L {512, 1k, 2k, 4k) for the transformer-basee models (GPT and Llama) and L = {1k, 4k, 8k, 16k} for the subquadratic models (Mamba and Hyena). The ranges are differ- given the poor computational scaling of transformers and our limited compute.
5

---

Published as a conference paper at ICLR 2025
Moan Times Betwcen Events
Standard Deviation of Times Between Events
IOR of Times 3atween Events
23000
E- ovir
Enn ONOP
20000
30000
:5000-
25200
20000
20200
10000
15:00
102000
5900
5008
18
Monoh
1.7
Woot
M
Decarto
Decade
Figure 2: EHR data exhibits high degree of variation in time intervals between events. From left to right, we measure the mean, standard deviation. and inter-quartile range (IQR) of time intervals between events, reflecting the irregular timing of clinical interactions "EHR-OMO OMOP validation set. The x-axis (log scalc) represents the metric in seconds, ranging from 10' to 10 The y-axis measures the number of sequenees with those values. Here, we focus event intervals capture the temporal structure of clinical encounters and highlight patterns in patient healthcare utilization.
(blue) is the 0.5M patients in the EHR-
For pretraining, we employ an autoregressive next-token prediction objective with cross entropy loss. We sample one subsequence of min( [ tokens from each patient i's timeline per epoch and train each model for billion tokens.
3.2 EVALUATION
We the EHRSHOT clinical prediction benchmark for all of our downstream evaluations (Wornow et al. 2023). EHRSHOT consists of 15 clinical prediction tasks based on a dataset of 7 patients' longitudinal EHRs. The primary evaluation metric is AUROC, and Brier scores also reported. We only consider binary classification tasks, thus we exclude the multilabel Chest X-Ray Findings task. We use the remaining 14 tasks from the EHRSHOT benchmark for our evaluations, which are broadly grouped into three categories: Operational Outcomes includes predicting ICU Transfer, 30-day Readmission, and Long Length-of-Stayy Anticipating Lab Test Results involves predicting if thrombocytopenia, hyperkalemia, hypoglycemia, hyponatremia, or anemia lab will be abnormal; and Assignment of New Diagnoses requires predicting whether a patient will get a new diagnosis of hypertension, hyperlipidemia, pancreatic cancer, celiac disease, or lupus within the next year. For additional details on all 14 tasks, including precise definitions, label counts, statistics on the number of tokens per patient, and evaluation methodology, please see Appendix Section A
For our evaluations, we use the same context length that was used during pretraining. We thus sample the last min |T,|} tokens for each patient prior to the relevant prediction time for task, then take the embedding of the last token in that sequence as our representation for that patient. We evaluate our models under the zero-shot, few-shot, and "All" data setting, with detailed results for zero- and few-shot evaluation provided in Appendix Sections C and H All EHRSHOT scores reported in the main results use the "All" data setting. To be consistent with the original EHRSHOT benchmark, we do not finetune our base models instead, we train a logistic regression head on top of the frozen representations created for each patient. Additional details are in Appendix Section A
3.3 EHR-SPECIFIC PROPERTIES
In the following subsections, we define metrics to quantify three properties of EHR data that distin- guish it from modalities such as natural language repetitiveness due to copy-forwarding irregular intervals of time between events, and a natural trend towards increased token complexity over time due to disease progression. Please see Figure I for overview. We believe this analysis provides an interesting counterpoint to most ML research being conducted on natural language sequences.
For all three metrics, first apply them to the EHR-OMOP validation dataset to measure the extent to which a large corpus of real-wont world EHR data exhibits these properties. Second, we apply two of the EHR-specific metrics repetitiveness and irregularity to the EHRSHOT dataset to stratify
6

---

Published as a conference paper at ICLR 2025
Foemaiis With Ohen fepereeeet fdle
Deumplie With Oven
Fermale V0B Ohen cortooatte
0:25
088
00
33
022
24
03
Figure 3: EHR data exhibits a higher degree of repetition than natural language, as measured by n-gram repetition rates. From left to right, we measure n = 1, 2, 3,4. "EHR- OMOP" (blue) is the 0.5M patients in the EHR- OMRPP validation dataset, "WikiText" (orange) is the WikiText- 103 training dataset of high quality Wikipedia articles (Merity at. 2016). We analyze n-gram repetition at the event level to reflect the structure of recurring clinical events, capturing patterns unique EHR The x-axis represents the n-grrmm repetition rate (i.e., percentage of n-grams that are repeated at least once within a sequence, where higher is more repetitive) and the y-axis shows the frequency of sequences with that repetition rate in each dataset.
individual patients based on how much they exhibit each property. This stratification allows us to assess how model performance varies across different levels of these properties, and to what extent longer context models can maintain robust performance.
3.3.1 COPY-FOWWARDING LEADS TO NOISY TOKEN REPETITION
EHR v. NLP. Copy-forwarding refers to the practice of recording the same diagnosis across multiple visits, typically for chronic conditions or billing purposes (Thornton et al. 2013 Calder et al., 2024] Weis & Levy 2014). This leads to higher levels of event repetition within the EHR. We hypothesize that repetition could worsen model performance by crowding information out of a limited context window. A long context model might be better equipped to handle this range of possibilities.
Metrics. To quantify the prevalence of copy-forwarding repetition rate (RR), i.e., the proportion of n-grams in the sequence that are repeated at least once. Please see Appendix Section F.1 for details. A higher RR implies a more repetitive sequence.
in sequence, we calculate its n-gram
3.3.2 TIME INTERVALS BETWEEN EVENTS ARE HIGHLY IRREGULAR
EHR NLP. In natural language, consecutive tokens uniformly have the same "distance" of position. In EHR data, however, a patient might wait days, weeks, or even years between visits to the hospital (MeDermott et al. 2023). This means consecutive EHR events have vastly different "distances" in time. We hypothesize that patients with more "irregular" sequences, i.c., a greater variety of inter-event time intervals, are more difficult model they present a more complex mix of timespans over which model must reason. This could pose particular challenges to long context models given they observe an even broader range of events (and thus inter-event timespans).
Metrics. We quantify irregularity as the standard deviation of time intervals between every pair of consecutive events. A higher standard deviation implies more irregular sequence. Please see Appendix Section F2 for more details.
3.3.3 DISFASE PROGRESSION CAUSES INCRFASED TOKEN COMPLEXITY OVER TIME
EHR v. NLP. Disease progression refers to the evolving nature of a patient's health over time. As people age, they experience an increase in the variety, frequency, and complexity of diseases they experience due to declining immunity and the increased likelihood of developing comorbidities (Fabbri et al., 2015). In natural language, earlier tokens tend to help in predicting later tokens, and thus perplexity is inversely correlated with token's 's position in a prompt (Kaplan al. 2020). Since disease becomes more complex over time, however, it was unclear if this trend holds for EHR data.
Metrics. To quantify disease complexity over time, we apply trained EHR FMs to calculate the median perplexity at each token position across a sample of 20,000 patients from the EHR-OMOP validation set. Please Appendix Section F3 for additional experimental details.
7

---

Published as a conference paper at ICLR 2025
Context Length
Model
Q1
Q2
Q3
Q4
Repelitiveness (I-gram RR)
Mamba
lk
0,0737
16k
0.0605
0.0670
0,0700
0.0746
Llama
512
0.0640
0.0710
0.0743
0.0792
4k
0.0627
0.0687
0.0721
0.0770
CLMBR-I-base
512
0.0647
0,0719
0.0751
0.0805
Ireeuulrity
Mamba
Ik
0.0693
0.0729
0.0731
0.0764
(Standard Devistion)
0.0641
Llama
4k
0.0664
0.0705
0.0694
0.0740
CIMBB--thase
512
0.0683
0.0741
0.0721
0.0777
Table 2: Comparison of average Brier scores of models across all 14 EHRSHOT tasks. Patients bucketed by repetitiveness (top) and irregularity (bottom). Q1/Q2/Q3/Q4 are the 1st through 4th quartiles of patients ranked by each metric. example, QI contains the least repetitive irregular patients while Q4 contains the most repetitive imregular patients. Bolded values show a statistically significant win rate of at least 50% of the longer context model over the shorter context model specific quartile. Only Mamba, Llama, and CLMBR-t-bar (the prior SOTA) shown for space Appendix Table 14 for results all models.
RESULTS
First, we evaluate each of our models the 14 EHRSHOT clinical prediction tasks. Overall results are shown in Figure Db, and per-task results in Appendix Figure 9 Our best performing model is Mamba with context length of 16k tokens. It achieves the highest average AUROC across all tasks, beating the prior state-of-the-ae by 0.03 points. Second, we analyze how three EHR-specific properties event repetition from copy-forwarding, irregularly spaced inter-event times, and disease progression impact model performance. After stratifying EHRSHOT patients into quartiles by each property, find that each property negatively correlates with model performance. However, longer context models exhibit more robustness as they perform better across all quartiles.
4.1 LONGER CONTEXTS IMPROVE PREDICTION MAKING FOR CERTAIN ARCHITECTURES
Our best performing model is Mamba at its maximum context length of 16k tokens, with mean AUROC of 0.007 (+0.03 points over prior SOTA). This can be seen in Figure Tp. Each line represents separate model architecture. The y-axis is mean AUROC across the 14 EHRSHOT tasks, and the x- axis the context length. The dotted purple line is the AUROC (0.777) achieved by the best overall prior model, CLMBR-I-base, which had context length of 512 tokens (Wornow al. 2023
Several trends appear Figure Tp. Both Mamba (green) and Llama (orange) show increased per- formance at longer context lengths. demonstrating the value of additional EHR data when making clinical predictions. In contrast, Hyena (red) exhibits a sharp decrease in performance after exceed- ing context length of 4k. This shows that including more tokens into the context does not always improve performance across architectures. The impact of context length on GPT (blue) appears less clear, which could be due its usage of absolute positional embeddings (see Section 4.4 for additional analysis). Results on individual tasks are in Appendix Figure
To more explicitly model the passage of time, we also train version of our models using the Artificial Time Tokens (ATT) technique proposed in CEHR-BERT (Pang et al. 202T). However, as shown in Appendix Figure 12. we see slightly worse performance with this tokenization strategy.
4.2 COPY-FORWARDING CREATES NOISY REPETITION HARMING MODEL PERFORMANCE
EHR-OMOP Analysis. We measure the n-gram repetition rate (RR) across all 0.5M EHR-OMOP
validation patients and plot the frequency of each observed RR in Figure 3 in blue. We perform dataset and overlay them in orange as "WikiText" as a point of comparison (Merity al.) 2016). While significant number of patients have no repeated n-grams in their records due to their short length (see Appendix Figure g for a recreation of this plot that excludes patients with less than 20 total events), see that EHR data still exhibits much higher degree of repetition than does natural language, especially when considering the repetition of 3-grams and 4-grams. For more details on n-gram RRs, see Appendix Section
the same calculations on the WikiText- 103
EHRSHOT Stratification. Next, we evaluated how the repetitiveness of patient's timeline affects model performance the EHRSHOT benchmark using Brier score. Using 1-gram repetition rate
8

---

Published as a conference paper at ICLR 2025
Figure 4: Median perplexity (PPL) by token position for different models GPT (far left), Hyena (middle left), Llama (middle right), Mamba (far right) across varying context lengths (lines). The x-axis represents token position, and the y-axis shows the median PPI. at each position measured across 20k EHR-OMOP patients. We analyze PPL. by token rather than by event to capture the model's 's handling of the specifie information content in each encoded Note that the upward trend in PPL is almost immediate, even within the first hundred tokens of each model's context window.
the metric, patients were grouped into quartiles from Q1 (lowest) to Q4 (highest). D (top) show that increased repetition reduces the performance of CLMBR-t-base.
We repeated this analysis with the EHR FMs trained in this work 2 (top). Model performance consistently degrades as repetition increases, indicating that highly repetitive sequences are more challenging model. Notably, longer context versions of Mamba and Llama achieve significantly lower Brier scores across all quartiles compared to their shorter counterparts.
4.3 IRREGULAR INTER-TOKEN TIME INTERVALS ARE HARDER TO MODEI.
EHR-OMOP Analysis. We first quantify the degree which EHR data exhibits irregularity in the intervals of time between consecutive events. TigureDshow three different metrics for irregularity the mean, standard deviation, and interquartile range of inter-event times for each individual patient for the EHR-OW OMHO validation set in blue. The x-axis of each plot is log scale, illustrating the large range of inter-event times across patients. Most patients appear have a standard deviation of inter-event times between 107 and 10t seconds (i.e. 115 days to 3.2 years).
EHRSHOT Stratification. Next, we measured how patient timeline irregularity impacts model performance the EHRSHOT benchmark using Brier score. Evaluating CLMBR-I-base across quartiles of patient irregularity (using the standard deviation inter-event times as the metric), found that performance generally degrades (higher Brier scores) as irregularity increases T (middle), indicating that irregular sequences are harder to model.
Table extends this analysis the EHR FMs trained in this work. While model performance still degrades with increased irregularity, longer context versions of Mamba and Llama consistently outperform their shorter counterparts across all quartiles.
4.4 DISEASE PROGRESSION EFFECTS ARE BETTER MODELED WITH LONGER CONTEXTS
EHR-OMOP Analysis. Figure 4 shows that tokens later in a patient's 's timeline are more difficult to predict (higher perplexity), even when conditioning on all prior tokens. This contrasts with natural language, where later tokens tend to have lower perplexity (Kaplan et al. 2020 Peng et al. 2023b). We hypothesize this is because diseases naturally become more complex and varied with aging. This degrades the predictive utility of past medical history as primary diagnoses change over time.
Longer context versions of Mamba and Llama consistently achieve lower perplexities across all to- ken positions compared to shorter contexts, with the gap widening at later tokens. This suggests that a more complete view of the patient's 's timeline helps handle increasing token complexity due to ag- ing. In contrast, Hyena's longer context models perform worse, replicating our original EHRSHOT results. For GPT, results are mixed: longer contexts (2k and 4k) achieve lower perplexities at later tokens but exhibit significant spikes. This appears to be caused by GPT's usage of absolute po- sitional embeddings- replacing them with rotary positional embeddings (ROPE) (Su et al., 2024) mitigated these spikes seen in Appendix Figure II Thus, despite its popularity the EHR FM community (see Table []. we recommend discontinuing the GPT architecture in favor of Llama or other more modern decoder-only architectures.

---

Published as a conference paper at ICLR 2025
5 DISCUSSION
In this study, we evaluated the impact of context length on clinical prediction tasks across four models-Mamba, Llama, GPT, and Jlyena-trained on longitudinal EHR data. We are the first to pretrain and release the full weights of these non-GPT architectures the scale of millions of EHRs. With a context length of 16k tokens, Mamba achieved the highest average AUROC across 14 pre- diction tasks on the EHRSHOT benchmark, surpassing the prior state-of-the-art by +0.03 points. In addition to the best performance, Mamba also offers faster training, quicker inference, and the po- tential support longer contexts (Gu & Dao, 2024). Notably, longer context versions of Mamba and Llama performed well in handling EHR-specific issues like token repetition due to copy fopy-forwa,,,ng, irregular inter-token time intervals, and increased token complexity from disease progression. This improvement, however, wasn't It universal, as Hyena's 's performance declined significantly beyond 4k tokens, underscoring the need to empirically validate each architecture for long context use.
While our findings highlight the potential for long-context models to successfully model EHR data, several limitations should be considered. First, we did not eval- uate transformer-based models at context lengths beyond 4k tokens due to limited computational resources. Running vanilla 16k transformer takes roughly 16x more compute/memorry than at context length of 4k, which was a core motivator for the development of the subquadratic architec- tures evaluated in this work. Second, model sizes were kept consistent across architectures to isolate the impact of context length. Preliminary findings suggest smaller Mamba models with 16k tokens perform well, which may reduce the need for larger models unsuitable for resource-constrained set- tings. Future work should quantify the impact of model size on performance. Third, our evaluations focused on clinical risk prediction tasks. but broader clinical tasks (c.g.. phenotyping, treatment selection) merit further consideration. Fourth, our pretraining dataset was sourced from a single institution due to data privacy concerns, which may limit generalizability. Fifth, we explored only three EHR-specific properties. Future research could extend this to more attributes of EHR data- e.g.. partial observation due to underdiagnosis or miscoding (Pivovarov et al. 2014/Che et al. 2018 multimodal signals (Soenksen et al. 2022). and event associasdd metadata (MeDermot et al. 2023)
Limitations/ / Future Work:
Sixth, we focused on the impact of these EHR- speciicci properties on downstream evaluations, but they may also have effects on pretraining convergence and stability, which we leave future work. Seventh, while the metrics we introduce offer a novel lens for examining EHR data, they are fairly simple and could be improved with additional context. For example, having our repetition metric distinguish between meaningful and non-meaningful repetition (e.g., repeated lab test in an ICU stay likely more informative than a repeated diagnosis code of chronic condition like hyperten- sion) could improve model performance in high-repetition settings. And for the irregularity metric, disease status may influence the regularity of time intervals between events (e.g. cancer patient may exhibit more regular visits than a patient suffering from acute cardiovascular events), which future work could explore by stratifying results based on specific disease phenotypes. Eighth, other promising transformer alternatives, such as linear attention models (Arora et al. 22024), hybrid ar- chitectures (Poli et al. 2023bb Lieber et al., 2024)). and recurrent models (Peng et al. (023a), should be explored in future work that builds upon the framework introduced here.
CONCLUSION
Long context models have unlocked a broad range of natural language applications through their ability to ingest and reason over massive amounts of information. Translating these gains to EHR data could benefit patients by enabling the modeling of an entire lifetime. Thus, we present the first systematic evaluation of how context length impacts EHR modeling. We find that long context subquadratic models such as Mamba are capable of achieving state-of-the-art results on clinical prediction tasks. This represents a sharp break from prior work in EHR FMs, as shown in Table [ which generally utilized BERT-based models limited to context windows of 512 tokens. We also find that longer context models are more robust to three distinct aspects of EHR data that had been underexplored in prior literature on sequence modeling. We hope our work inspires future efforts to identify interesting sequence modeling challenges from Non-NLP domains and encourages further research towards applying non-transformer architectures to structured EHR data.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
NHS acknowledges support by the Mark and Debra Leslie Endowment for AI in Healthcare, the Clinical Excellence Research Center at Stanford Medicine, and Technology and Digital Solutions at Stanford Healthcare. JF was supported in part by a Stanford AIMI-HAI Partnership Grant. CR acknowledges the support of NIH under No. U54EB020405 (Mobilize). NSF under Nos. CCF2247015 (Hardware-Aware), CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Veloc- ity), and 1937301 (RTML); US DEVCOM ARL under Nos. W911NF-23-2-016 (Long-context) and W91 N (Interactive Human-AI Teaming); ONR under Nos. N000142312633 (Deep Signal Processing): Stanford HAI under No. 247183; NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASE, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research pro- gram, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project: Meta, Google, and VMWare. SK acknowledges support by NSF 2046795 and 2205329, IES R305C240046, the MacArthur Foundation, Stanford HAI, OpenAl, and Google. The U.S. Govern- ment is authorized reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations ex- pressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, the U.S. Government.
REFERENCES
Edward P Ambinder. Electronic health records. J. Oncol. Pract., 1(2)57-63, July 2005.
Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the tecoul-th tradeoff. arXiv preprint arXiv:2402./8660 2024.
Cristian Bodnar, Wessel P Bruinsma, Ana Lucie, Megan Stanley, Johannes Brandstetter, Patrick Garvan, Maik Riechert, Jonathan Weyn, Haiyu Dong, Anna Vaughan, et al. Aurora: A foundation model of the atmosphere. arXiv preprint arXiv:2405. 13063, 2024.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ B. Altman. Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quiney Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Dur- mus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang. Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keel- ing, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Ku- ditipudi, and et al. On the opportunities and risks of foundation models. CoRR. 0bs/2108.072566 2021. URL btttps://aaat
org/ abs / 08.10..000 07258
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess. Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL.
https: ptmpss//aae /tt::///axiivv org/ abts / 14165
Madison B Calder, Matt Hanson, Melissa Jost, and Kristen D Kelley, Time and note characteristic effects of an electronic health record template for internal medicine resident notes. Journal of Graduate Medical Education, 300)
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):6085, 2018.
11

---

Published as a conference paper at ICLR 2025
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023. URL https://axxiv.org abs abs/2306.1555
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser. David Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers, 2022. URL https://
arxiv.org/abs 2009. 14794
Ben Cohen, Emaad Khwaja, Kan Wang, Charles Masson, Elise Ramé, Youssef Doubli, and Oth- mane Abou-Amal. Toto: Time series optimized transformer for observability. arXiv preprint
arXiv:2407.07874, 2024.
Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Al- bert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, al. Griffin: Mix- ing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.19427. 2024.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv.orrt
abs/ 2ess/1110.0m 04805
Elisa Fabbri, Marco Zoli, Marta Gorvalez-Freire, Marcel E Salive, Stephanie Studenski, and Luigi Ferrucci. Aging and multimorbidity: new tasks, priorities, and frontiers for integrated gerontological and clinical research. Journal of the American Medical Directors Association, 16 (8):640-647, 2015.
Adibvafa Fallahpour, Mahshid Alinoori, Arash Afkanpour, and Amrit Krishnan. Ehrmamba: To- wards generalizable and scalable foundation models for electronic health records. arXiv preprint 14557 2024.
Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It's raw! audio generation with state- space models. In International Conference on Machine Learning, pp. 7616-7633, PMLR, 2022.
Albert Gu and Tri Dao. Mamba: Linear sequence modeling with selective state spaces, 2024. URL htttps//////xx http://awww..oooaa///33/// 00752
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, 2022. URL https::/aaxxx..oottttotm// haape///ab//11110 00396
Kyunghoon Hur, Jungwoo Oh, Junu Kim, Jiyoun Kim, Min Jae Lee, Eunbyeol Cho. Seong-Eun Moon, Young-Hak Kim, Louis Atallah, and Edward Choi. Genhpf: General healthcare predictive framework for multi-task multi-source learning. IEFF Journal of Biomedical and Health Infor-
matics, 28(1):502-513, Janwary 2024a. ISSN 2168-2208. doi: 10. 00.11999hhi.20233337799 URL
http: dx. doi. www.:.oorgg11.. http:///........rr///00.1111/1/0 1BH 3327751
Kyunghoon Hur, Jungwoo Oh, Junu Kim, Jiyoun Kim, Min Jae Lee, Eunbyeol Cho, Seong-Eun Moon, Young-Hak Kim, and Edward Choi. Unihpf Universal healthcare predictive framework with zero domain knowledge, 2024b. URL httphttt///// pattts://arxivi org abs /tabb///21l. 08082
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 20Xi::2001.08661 2020.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference ma- chine learning, pp. 5156-5165. PMLR, 2020.
Junu Kim, Chaeeun Shim, Bosco Scong Kyu Yang. Chami Im, Sung Yoon Lim, Han-Gil Jeong, and Edward Choi. Gererae-purpose retrievel-enhanced medical prediction model using near-infh infrninfinite history. arXiv preprint arXiv:23 10,20444, 2023.
12

---

Published as a conference paper at ICLR 2025
Zeljko Kraljevic, Dan Bean, Anthony Shek, Rebecca Bendayan, Harry Hemingway, Joshua Au Ye- ung. Alexander Deng, Alfred Baston, Jack Ross, Esther Idowu, et al. Foresight a generative pretrained transformer for modelling of patient timelines using electronic health records: retro- spective modelling study. The Lancet Digital Health, ((4):e211-2290 2024.
Yikuan Li, Shishir Rao, José Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza Khorsiiii. Behrt: transformer for electronic health records. Scientific reports, 10(1):1-12, 2020.
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer- mamba language model. arXiv preprint arXiv:2403. 19887, 2024.
Matthew McDermott, Bret Nestor, Peniel Argaw, and Isaac S Kohane. Event stream gpt: data pre- processing and modeling library for generative, pre-trained transformers over continuous-tine sequences of complex events. Advances in Neural Information Processing Systems, 36:24322- 24334, 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Bekelu Negash, Alan Katz, Christine Neilson, Moniruzzaman Moni, Marcello Nesca, Alexan- der Singer, and Jennifer E Enns. De-identification of free text data containing personal health scoping review of reviews. Int. J. Popul. Data Sci., 8(1):2153, December 2023.
information:
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Callum Birch-Sykes, Michael Wornow, Aman Patel, Clayton Rabideau, Stefano Massaroli, Yoshua Bengio, Stefano Ermon, Stephen A. Baccus, and Chris Ré. Hyenadma: LLog-rangg genomic sequence modeling at single nucleotide resolution, 2023a. URL
org. abs 2306 15794
Eric Nguyen, Michael Poli, Matthew G Durrant, Armin W Thomas, Brian Kang, Jeremy Sullivan, Madelena Ng, Ashley Lewis, Aman Patel, Aaron Lou, et al. Sequence modeling and design from molecular to genome scale with evo. BioRxiv, pp. 2024 02, 2024.
Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. Climax: A foundation model for weather and climate. arXiv preprint arX1v:2301.10343. 2023b.
Mikkel Odgaard, Kiril Vadimovic Klein, Sanne Møller Thysen, Espen Jimenez-Solem, Martin Sille- sen, and Mads Nielsen. Core-behrt: A carefully optimized and rigorously evaluated behrt. arXiv preprint arXiv:2404. 15201, 2024.
Chao Pang, Xinzhuo Jiang, Krishna S Kalluri, Matthew Spotnitz, RuiJun Chen, Adler Perotte, and Karthik Natarajan. Cehr-bert: Incorporating temporal information from structured ehr data to improve prediction tasks, 2021. htRh:ttps: httttps://aaxi..oogg//////
httpaa/////.i1.11 08585
Chao Pang. Xinzhuo Jiang, Nishanth Parameshwar Pavinkurve, Krishna S. Kalluri, Elise L. Minto, Jason Patterson, Linying Zhang, George Hripcsak, Gamze Gürsoy, Noémie Elhadad, and Karthik Natarajan. Cehr-gpt: Generating electronic health records with chronological patient timelines, 2024. URL UtLpttps:: httpppppp/pa......teeast abs 2400. 04400
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung. Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. Rwkv: Reinventing rnns for the transformer era, 2023a. URL http:: arxiv. org/ httpaaa/./3.5..3..3..
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yar:: Efficient context win- dow extension of large language models, 2023b. URL https///axxii..oo///////000
00071
Rimma Pivovarov, David Albers, Jorge L Sepulveda, and Noémie Elhadad. Identifying and miti- gating biases in ehr laboratory tests. Journal of biomedical informatics, 51:24-34, 2014.
13

---

Published as a conference paper at ICLR 2025
Michael Poli, Stefano Massaroli, Eric Nguyen. Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference Machine Learning, pp. 28043-28078. PMLR,, 2023a.
Michael Poli, Jue Wang, Stefano Massaroli, Jeffrey Quesnelle, Ryan Carlow, Eric Nguyen, and Armin Thomas. StripedHyena: Moving Beyond Transformers with Hybrid Sig- nal Processing Models. GitHub repository, 12 2023b. URL bittps://githie com/
Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases 12409
enables input length extrapolation, 2022.
Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ digital medicine, 4(1):86, 2021.
Pawel Renc, Yugang Jia, Anthony E Samir, Jaroslaw Was, Quanzheng Li, David W Bates, and Arkadiusz Sitek. Zero shot health trajectory prediction using transformer. NPJ Digital Medicine, 7(1):256, 2024.
Observational Health Data Sciences and Informatics. The book of ohdsi, Jan 2021. URL htttps://cddi..giie www.thub..ooroooe wtw..//ecccccoooossaaooooaoaoooe Commun ty. html @hdsis-progress:
/
Luis R Soenksen, Yu Ma, Cynthia Zeng, Leonard Boussioux, Kimberly Villalobos Carballo, Liangyuan Na, Holly M Wiberg, Michael I. Li, Ignacio Fuentes, and Dimitris Bertsimas. In- tegrated multimodal artificial intelligence framework for healtheare applications. NPJ digital medicine, 5(1):149, 2022.
Ethan Steinberg, Ken Jung. Jason A Fries, Conor K Corbin, Stephen R Pfohl, and Nigam H Shah. Language models are an effective representation learning technique for electronic health record data. Journal of biomedical informatics, 113: 103637,, 2021.
Ethan Steinberg, Jason Fries, Yizhe Xu, and Nigam Shah. Motor: A time-to-event foundation model for structured medical records. arXiv preprint 2rXiv:2301.031500 2023.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputingg, 568: 127063, 2024.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A sur- vey.(2020). arXiv preprint cs. n..LG/2009.06732 2020.
Llama Team. The llama 3 herd of models, 2024. URL http:sss/aaaiii.orrg/a//////////
21783
J Daryl Thortoon, Jesse Schold, Lokesh Venkateshaiah, and Bradley Lander. Prevalence of copied information by attendings and residents in critical care progress notes. Critical Care Medicine, 41(2):382-385 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017. URL https::/aara
org/abs/1706. 03762
Xiao Wang, Shiao Wang, Yuhe Ding, Yuehang Li, Wentao Wu, Yao Rong, Weizhe Kong, Ju Huang, Shihao Li, Haoxiang Yang, et al. State space model for new-generation network alternative to transformers: A survey, arXiv preprint arXiv:2404.0955: 00540016. 2024.
Justin M Weis and Paul C Levy. Copy, paste, and cloned notes in electronic health records. Chest, 145(3):632-66 2014.
Michael Wornow, Rahul Thapa, Ethan Steinberg, Jason Fries, and Nigam Shah. Ehrshot: An ehr benchmark for few-shot evaluation of foundation models. Advances in Neural Information Pro- cessing Systems, 10667125--61337 2023.
14

---

Published as a conference paper at ICLR 2025
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang. Yashar Mchdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models, 2023. URL httts::/arxiisi www.///////./oeoooee..aaaeeeeeee
16039
Zhichao Xu. Rankmamba: Benchmarking mamba's document ranking performance in the era of transformers, 2024. URL https://axxix..ooottt// wttp:xii..rg///oo.. abs/ 2203. 18276
Zhichao Yang. Avijit Mitra, Weisong Liu, Dan Berlowitz, and Hong Yu. Transformehr: transformer- based encoder-decoder generative model to enhance prediction of disease outcomes using elec- tronic health records. Nature communications, 14(1):7857, 2023.
Yinghao Zhu, Changyu Ren, Zixiang Wang, Xiaochen Zheng, Shiyun Xie, Junlan Feng, Xi Zhu, Zhoujun Li, Liantao Ma, and Chengwei Pan. Emerge: Integrating rag for improved multimodal ehr predictive modeling. arXiv preprint arXiv:4466.00336 2024.
15

---

Published as a conference paper at ICLR 2025
A DATASET
Our primary dataset, "EHR-OUE SCHR--MOPP, is sourced from academic medical center. It contains dei- dentified longitudinal EHR data formatted according to the Observational Medical Outcomes Part- nership Common Data Model (OMOP-CDM) (Sciences & Informatics) 2021 All data is stripped of protected health information and deidentified at the institution level to comply with HIPAA and the Safe Harbor standard. The dataset stored HIPAA- -iopplactll compute environment. All patients included in EHR-OMOP sign form consenting their records to be included in research pur- poses like this work. This study was conducted under an eesiiuttion-wide IRB protocol that makes this deidentified dataset available for research purposes.
We use roughly 2.5M patients from EER-OMOP for pretraining our models, and hold out 0.5M patients for conducting validation experiments.
Unique (hentsPaaeee(kon
1.0000
140000
120000
120000
10000
L80000
300000
0000
UP
10
3P
evera
Unique Eventalhubent
30000
2000
40000
1,000
20000
L000
500
20000
157 190 12T
everta
Figure 5: Distributions of patient data from the EHR-OMOP dataset across (A) training and (B) validation splits, showing both event-level and code-level counts. The x-axis is log-scaled to capture the wide range in the number of events per patient, the number of unique patients per code, and the distribution events associated with each code.
Training Split
Value
Validation Split
Value
Overall counts
Overall counts
Number of events
3,501,210,238
Number of events
749,0013,035
Unique codes
3,144,978
Unique codes
881,012
Unique patients
2,567,450
Unique patients
550,305
Events per patient
Events per patient
Minimum
Minimum
Mean
1,364
Mean Median
1,361
Median
121 638,708
Maximum
890,048
Maximum
Unique events per patient
Unique events per patient
Minimum
Minimum
Mean
237
Mean
237
Median
76
Median
76
Maximum
26,131
Maximum
18,561
Table 3: Summary statistics for the EHR-OMOP training (left) and validation (right) splits.
16

---

Published as a conference paper at ICLR 2025
B EVALUATION
B.I TASKS
For all of our model evaluations, we 14 binary clinical prediction tasks sourced from the EHRSHOT benchmark (Wornow et al. 2023). The definitions of these tasks are detailed in Ap- pendix Table
We also provide label and patient counts in Appendix Table Sable for each task.
Task
Type
Prediction Time
Time Horizon
Operational Outcomes
Long Length of
Binary
11:59pm day of admission
Admission duration 30 days post discharge Admission duration
30-day Readmission
Binary
11:59pm on day of discharge
ICU Transfer
Binary
11:59pm day of admission
Anticipating Lab
Thrombocyooenia
Binary
Immediately before result Immediately before result
Next result
Hyperkalemia
Binary
Next result
Hypoglycemia
Binary
Immediately before result
Next result
Hyponatremia
Rinary
Immediately before result
Next result
Anemia
Binary
Immediately before result Next result
Assignment New Diagnoses
Hypertension
Binary
11:59pm day of discharge
year post-discharge
Hyperlipidemia
Binary
11:59pm on day of discharge
year pust-discharge
Pancreatic Cancer
Binary
11:59pm day of discharge
year post. discharge
Celinc
Binary
year post-discharge 11:59pm day of discharge
11:5pp on day of discharge
Lupus
Binary
year post-discharge 11:59pm on day of discharge
Acute MI
Binary
1 year post- sist-dishhrgge
Table 4: The 14 clinical prediction tasks used for evaluating models in this work. Prediction Time is the precise time point (up minute precision) in patient's timeline when the prediction is made. Time Horizon is the length of time considered after prediction time determine whether an event occurs, ie. we only consider a patient "positive" for a new diagnosis of pancreatic cancer if she receives that diagnosis within a year of being discharged. Table reproduced verbatim from (Wornow et al. 2023
The definitions for each task are provided below (reproduced verbatim from (Wornow et al. 2023
Operational Outcomes. These tasks are related to hospital operations. They are defined as follows:
Long Length of Stay: Predict whether patient's total length of stay during a visit to the hospital will be at least 7 days. The prediction time is at 11:59pm the day of admission, and visits that last less than one day (i.c. discharge occurs on the same day of admission) are ignored.
30-day Readmission: Predict whether a patient will be re-admitted to the hospital within 30 days after being discharged from visit. The prediction time is at 11:59pm the day of admission, and admissions where a readmission occurs on the same day as the corresponding discharge ignored.
Predict whether a patient will transferred to the ICU during visit the hospital. The prediction time is at 11:59pm the day of admission, and ICU transfers that occur on the same day as admission are ignored.
ICU Transfer:
Anticipating Lab Test Results. These tasks related to lab value prediction. The prediction time is immediately before the lab result is recorded. They are defined as follows:
Thrombocylopenia: Predict whether a thrombocytopenia lab comes back normal (>=150 10 or abnormal (any other reading). We consider all lab results coded as LOINC/LP39321885, L328C/L or LOINC/777-3.
Hyperkalemia: Predict whether a hyperkalemia lab comes back as normal (<=5.5 mmol/L). or abnormal (any other reading). We consider all lab results coded as LOINC/LG7931-,, LOINC/LP386618-5, LOINCILG10990-6, LOINC/6298-4, or LOINC/2823-3.
17

---

Published as a conference paper at ICLR 2025
Train
Val
Test
Task Name
Patients (# Positive)
# Labels
Parients (# Positive)
# Labels (# Positiv))
Patients (# Positive)
# Labels
(# Positive)
(# Positive)
Operational Outcomes
Long Length of Stay
1377 (464)
2569 (681)
1240 (395)
2231 (534)
1238 (412)
2195 (552)
30-day Readmission
1337 (164)
2608 (370)
1191 (159)
2206 (281)
1190 (151)
2189 (260)
ICU Transfer
1306 (107)
2402 (113)
1157 (84)
2052 (92)
1154 (75)
2037 (85)
Anticipating Lab Test Results
Thrombocytopenia
2084 (906)
(22714)
1981 (807)
(17867)
1998 (853)
(19137)
Hyperkalemia
2038 (456)
76349 (1829)
1935 (428)
60168 (1386)
1958 (405)
63653 (1554)
Hypoglycemia
2054 (511)
122108 (1904)
1950 (433)
95488 (1449)
1970 (435)
100568 (1368)
pypouttremia
2035 (1294)
81336 (23877)
1930 (1174)
64473 (17557)
1956 (1224)
67028 (19274)
Anemia
2092 (1484)
70501 (49028)
1992 (1379)
56224 (38498)
2002 (1408)
58155 (39970)
Assigument of New Diaguoses
Hypertension
792 (129)
1259 (182)
781 (128)
1247 (175)
755 (129)
1258 (159)
Hyperlipidmia
923 (137)
1684 (205)
863 (140)
1441 (189)
864 (133)
1317 (172)
Pancrentic Cancer
1376 (128)
2576 (155)
1242 (46)
2215 (53)
1246 (40)
2220 (56)
Celiac
1392 (48)
2623 (62)
1252 (8)
2284 (11)
1255
2122 (21)
Lupus
1377 (79)
2570 (104)
1238 (24)
2225 (33)
1249 (19)
2243 (20)
Acute
1365 (130)
2534 (175)
1234 (112)
2176(145)
1235 (115)
2127 (144)
Table 5: The number of unique patients total labels each split of the 14 EHRSHOT tasks evaluated this work. The prevalence of positive patients/labels is shown in parenthesis. Table reproduced from (Wornow ct al. 2023), with updates to reflect the latest version of the EHRSHOT dataset.
Hypoglycemia: Predict whether a hypoglycemia lab comes back as normal (>=3,9
mmol/L. abnormal (any other reading). We consider all lab results coded as SNOMED/33747003, LOINC/LPP46145-3, or LOINC/14749-6.
Hyponatremia: Predict whether hyponatremia lab comes back as normal (>=133 135 mmol/L) or abnormal (any other reading). We consider all lab results coded as LOINC/LG1136335 LOINC/2951-2 or LOINC/2947-0.
Anemia: Predict whether an anemia lab comes back as normal (>=120 120 g/L) or abnormal (any other reading). We consider all lab results coded as LOINC/L.P322452.1.
Assignment of New Diagnoses. These tasks related to predicting the first diagnosis of a disease. The prediction time is at 11:59pm on the day discharge from an inpatient visit, and we count any diagnosis that occurs within 365 days post-discharge positive outcome. We ignore all discharges in which the patient already has an existing diagnosis of a disease. The tasks are defined as follows:
Hypertension: Predict whether the patient will have her first diagnosis of essential hy- pertension within the next year. We define hypertension as an occurrence of the code SNOMED/5962100 as well as its children codes in our ontology.
Hyperlipidemia: Predict whether the patient will have her first diagnosis of hyperlipi- demia within the next year. We define hyperlipidemia as an occurrence of the code SNOMED/55822004, as well as its children codes in our ontology.
Pancreatic Cancer: Predict whether the patient will have her first diagnosis of panereatic cancer within the next year. We define pancreatic cancer as an occurrence of the code SNOMED/372003004, as well its children codes in our ontology.
Celiac: Predict whether the patient will have her first diagnosis of celiac disease within the next year. We define celiac disease as an occurrence of the code SNOMED/396331005, as well as its children codes in our ontology.
Lupus: Predict whether the patient will have her first diagnosis of lupus within the next year. We define lupus as an occurrence of the code SNOMED/55464009, well as its children codes in our ontology.
Acute MI: Predict whether the patient will have her first diagnosis of an acute myocardial infarction within the next year. We define myocardial infarction as an occurrence of the code SNOMED/57054005, well as its children codes in our ontology.
18

---

Published as a conference paper at ICLR 2025
B.2 EVALUATION PROCEDURE
Each model m E M oulputs an embedding for each token in its input sequence. Our goal is aggregate these outputs into unified representation R. for each patient which captures key patterns in their disease trajectory. We will then use this representation R2 to finetune a logistic regression head for our downstream binary classification prediction tasks.
We define two functions. First, we define S R Roxd + R &xd to select subset of & vectors from of vectors. Second, define A Raxd Rd aggregate set of n d-dimensional vectors into a single vector. Thus:
R² =
Initial experiments indicated that setting to simply return the vector the sequence (i.e. the most recent token in patient's timeline) and S to the most recent I. tokens in patient's timeline prior to the timepoint at which the prediction for task is made performed the best. Thus, we have:
R = (
Typ,
Finally, we fit logistic regression head H top of these representations in order apply them to binary prediction tasks. This yields a final prediction P; of:
P. (R))
which provides the model's estimate for the probability that specific clinical event occurs within task-defined window time for this patient based on their current representation R;
B.3 PATIENT STATISTICS
In Appendix Figure we plot the CDF the number of raw clinical events and tokens preceding each prediction time for a given task across train/val/test splits. The blue line represents all prediction times, the orange line corresponds only predictions associated with a positive label. Note that not every clinical event corresponds token in our vocabulary, hence many events are dropped during the tokenization process.
B.4 TASK-LEVER RESULTS
We present plots of each model's performance the 14 individual EHRSHOT tasks in Appendix Figure / Additionally, we provide raw numbers on the AUROC differences between each model
and the prior SOTA model, CLMBR-t-base, for each task Appendix Tables 7 8 IO We
report bootstrapped 95% confidence intervals over 1,000 resamples of the test set for each AUROC difference. Across all context lengths, our results for Mamba are shown in Appendix Table Llama in Appendix Table 8 GPT in Appendix Table 9 and Hyena in Appendix Table 10
C MODEL ARCHITECTURES
In this section, we present the mathematical formulations and detailed architectural descriptions of the four models used our experiments: GPT, Mamba, Llama, and Hyena.
C.1 GPT
GPT (Generative Pre-trained Transformer) is transformer-based autoregressive model that uses self-attention to process input sequences. (Brwwn et al.) 2020) The main operation is the scaled dot-product attention:
QK
Attention(Q. K,V) softmax
(1)
Vdk.
19

---

Published as a conference paper at ICLR 2025
0m
(a) Train (raw clinical events)
Train (tokens)
(c) Val (raw clinical events)
(d) Val (tokens)
(e) Test (raw clinical events)
(I) Test (tokens)
Figure 6: For each EHRSHOT task, we plot the CDF of the number of raw clinical events (left column) and tokens (right column) available to the model when making its prediction. In other words, the number of events/tokens preceding each label's prediction time point. The blue line represents all prediction times, while the orange line represents only predictions associated with positive label. Note that unlike the raw event counts, all token counts are capped at the maximum context length of the models we test (16k), hence the spike at the end of the CDF.
20

---

Published as a conference paper at ICLR 2025
Here, Q, K, and V are the query, key, and value matrices, respectively, and dę is the dimensionality of the key vectors. The transformer block consists of multi-head attention and a position- wise feed- forward network:
MultiHead K,K) V) Concat(headı, -..; head,, W
head; Attention(QW, KW. VW.)
where W W W.. and Wo are learned projection matrices. After attention, GPT applies position-wii -wise cerw-fowward network consisting of two fully connected layers with ReLU activations:
(FN(3) ReLUU(>W, + b22 + b2
The quadratic complexity of self-attention with respect to input length makes it challenging to scale GPT to long context lengths. In experiments, we use GPT variants with context lengths up to 4096 tokens.
C.2 LLAMA
Llama is transformer-based model that shares the core structure of GPT but incorporates opti- mizations for training efficiency and scalability (Team 2024). The model uses the same attention mechanism as GPT, but with several architectural modifications, such as an increased hidden state dimension, fewer normalization layers, and relative positional embeddings to improve its perfor- mance.
The forward pass for each transformer block in Llama follows the same formulation as GPT, com- bining self-attention with feed-forward network:
h = LayerNorm(h; MultiHead(h,, he, h,))
(5)
h = LayerNorm(be+1 + FFN((,-1))
(6)
Llama utilizes rotary positional embeddings (RoPE) (Su et al. 2024), which encode relative posi- tional information directly into the self-attention mechanism without requiring absolute positional encodings:
RoPE(q, k) cos(ii)g + sin(i8)k
Here, g and k are the query and key vectors, and is frequency parameter. We evaluate Llama on context lengths of up to 4096 tokens.
C.3 MAMBA
Mamba state-space model (SSM) architecture designed to handle long sequences eff- ciently. It replaces self-attention with state-space layers, which provide linear scaling with respect to input length. Mamba leverages the continuous-time state-space model to capture long-range de- pendencies:
X1+1 Ax, Bu,
(8)
y: Cx + Du,
(9)
where X; is the hidden state, u, is the input at time t, Yt is the output, and A, B, C, and D are learned matrices. This allows Mamba model long sequences with linear complexity, making ideal for processing the lengthy and complex event streams in EHR data.
In experiments, evaluate Mamba with context lengths of up to 16k tokens. Mamba's 's effi- ciency allows it to process long patient histories without the computational overhead of traditional transformer models.
21

---

Published as a conference paper at ICLR 2025
C.4 HYENA
The Hyena architecture introduces an efficient mechanism for handling long sequences by utilizing implicit long convolutions and multiplicative gating (Poli et 2023a).
The imput sequence is denoted by x(t), where t represents the sequence position. The convolution operation applied in Hyena can be described by the following equation:
L-1 y(t) - Σ h(1)-x(t- x(t-i)
where x(1) is the input at time step i, h(1) is the convolution filter of length L, y(t) is the output at time step 1,and L is the length of the filter.
The key difference between Hyena and traditional attention mechanisms is the use of implicit con- volutions, which avoid the quadratic complexity of the attention mechanism.
To further enhance the expressivity of the model, Hyena applies multiplicative gating after the con- volution operation. This gating mechanism can be expressed as:
z(t) = oWW, y(t)) O W2 y(t)
where:
z (t) is the gated output,
Ø is non-linear activation function (e.g., sigmoid),
W₁ and W2 are learnable weight matrices,
represents element-wise multiplication.
This combination of implicit long convolutions and multiplicative gating allows the Hyena model to process sequences with linear complexity in their length.
D TOKENIZATION
We follow the tokenization strategy used by the CLMBR-t-base model which had achieved the highest average AUROCs the EHRSHOT benchmark (Wornow et al. 2023). This tokenization strategy described in detail in (Steinberg et al. 2021).
Given patient timeline X,, our goal is to convert it into sequence of tokens T, that our models can ingest. Thus, we must map each Xij = (tij. Cij, Vty) to some set token(s) T.ş (Tay1. Tips) where Tisk T.
For encoding the t,y component each clinical event X we utilize positional encodings based on the token position j. prior studies have shown minimal benefits from directly embedding absolute time information Yang et al. 2023).
For handling the U/ component of Xij. define the following function g to map clinical events to tokens by handling each of the three possible cases for the types of values that U/y can take on separately:
(g((e)) go gelCigs vey)
if vig 0, if if Ve. if if E Vn.
((x;,)
gn (Ciji vi))
Thus, the same clinical event (e.g. lab test for anemia) be mapped to arbitrary large set of finer-grained tokens (e.g. one token for all lab tests, one each for mild/modderatt/severr,, one each for 10-point scale, etc.).
22

---

Published as a conference paper at ICLR 2025
Following (Steinberg al. 2021)
choose to employ deciling strategy for all numerical Biy, and we map each unique categorical Viy to its own token.
Let D C {a e 2
< 9} be function that maps vy to the decile it belongs to when considering all possible values that Cij is associated with in the training set. And let G(.) be a function that maps its input to some unique integer in the domain of our tokenizer's vocabulary.
Thus, we have that:
Do (ct) = G(ci))
9e (eg Vigy G(ay; (ig
9n (Gg m4y) = G(cy, D(cij, U;y))
Within our dataset, employing this tokenization strategy results in hundreds of thousands of potential unique codes. Many such codes, however, occur very infrequently. Thus, we select the top 39811 frequently occurring codes, following the same procedure outlined in (Steinberg al. 2021 In addition, seven special tokens - [BOS], [EOS], [UNK]. [SEP], [PAD], [CLS], and [MASK] are included, resulting in total vocabulary size of 39818 tokens. This yields an identical vocabulary to the one used by CLMBR-t-brre in the original EHRSHOT benchmark (Wornow et al. (023).
For positional embeddings, we use the default strategies for the various architectures we evaluate e.g. absolute positional embeddings for GPT, rotary positional embeddings for Ilama, none for Hyena beyond the Hyena positional embedding, and none for Mamba.
For completeness, we also evaluate the impact of injecting explicit temporal information into the patient timeline via Artificial Time Tokens (ATTs), as proposed in CEHR-BERT Pang et al., 2021 and used in other works (Pang et al. 2024 Renc al. 2024). In brief, we create artificial tokens to represent various time intervals (days, weeks, months, etc.) and inject these tokens between consecutive visits to represent the interval of time between them:
Dn if gap < 7 days (e.g., Di....,
W
if days $ gap < 28 days (e.g., W1, ..., Wa), if 28 days S gap < 365 days (e.g., M1, ..,, M12).
ATT
Mr
LT if gap 2 365.
Furthermore, to clearly define the start and end of each visit, we enclose each visit VÚ with special tokens (Visit Start) and VE (Visit End). This approach allows us to represent patient timeline as structured sequence:
P (VS, U1, VE, ATT, VS, VE, .TT...., VS, VE}
This enhancement directly embeds temporal patterns within the token sequence, providing con- textual information about the intervals between clinical events. The results these models trained using ATT tokens shown in Appendix Figure The figure shows that this tokenization strategy actually tended to reduce the performance of our models, and our best performing model remains Mamba- 166 16k without ATT'..
E TRAINING
In this section, describe the training of models used in experiments. All model base config- uration were taken from Huggingface, and can be found uder:
GPT:
Hyena: http://hgggggggoo.//////aaaaaaan
Mamba: http://inngiigggaccoossseeeeseem
23

---

Published as a conference paper at ICLR 2025
Potsont Toee
Tekenscatizn
LEWO Nelli S0M
Core
Fhe EH
Reasurch
Nacioall manide
Cecdent Lerethe
DB Specific Propa
Figure 7: high-level overview of experimental pipeline, from data generation final evalua- tion results.
Llama: http::/hhwgginnaaaccccoootttaaa/ tt Instructt
Their base configurations were modified standardize in terms of parameter count make fair comparison between them. These configuration changes are shown in Table
Model
Configuration
Value
GPT
positions learning rate dim model num layers
{512, 1k, 2k,
768
12
num heads
12
Total Parameters
116M
Hyena
seq len
1k, 4k, 8k, 16k
learning rate
2e-4
dim model
num layers
16
Total Parameters
125M
Mamba
max seq len
1k, 4k, 8k, 16k
learning rate
2c-4
dim model
768
num layers
24
hidden layers
24
state size
Total Parameters
121M
Llama
position embeddings
{512, 1k, 2k, 4k }
learning rate
2c-4
hidden size intermediate size attention heads num hidden layers num key value hcads Total Parameters
768
2688
12
123M
Table Model configurations used for training. All models are designed to be roughly 120 million parameters. We use the same tokenizer and vocabulary size for all models.
For the pretraining of our models, we randomly sample patient timeline of length equal to the lesser of the timeline length of the model's context length. To improve training stability and ensure GPU memory optimization, we employed gradient accumulation across multiple batches with total number of tokens per step of 65,536.
All models were trained using the AdamW optimizer with the following parameters: B1 0.9. 32 = 0.95, = 0. We performed a hyperparameter sweep over learning rates between le and le 3 for each model architecture before settling on the learning rates shown in Appendix Table 6 We employed a learning rate warm-up for the first 40,000 steps, after which the learning rate decayed to 1e 5 training progressed. This approach ensured smooth convergence while avoiding abrupt changes in training dynamics. Perplexity stabilized after one epoch, and we trained all models for 2 billion total tokens.
24

---

Published as a conference paper at ICLR 2025
The training was conducted PHI-compliant shared cluster equipped with heterogeneous mix of GPUs.The majority of experiments in this work were conducted on a set of V100s, with limited access to another 4 NVIDIA H100s and 16 NVIDIA A100s. The use of secure, PHI-complian en- vironment ensured that all patient health information remained confidential and protected throughout the training process, adhering stringent data privacy regulations.
F EHR-SPECIFIC PROPERTY METRICS
We define several metrics for quantifying the specific properties of longitudinal EHR data, such as the irregularity of inter-event time intervals, the repetitiveness of event sequences, and the complex- ity of tokens due disease progression. These metries help understand the challenges posed by EHR data when used predictive models.
F.1 REPETITIVENESS
Due to liability, documentation requirements, billing practices, and other administrative processes, EHR data tends to have a high prevalence of "copy- information - i.c. data that is copied- and-pasted from one visit the next (Thornton al. 2013; Calder al.) 2024 Weis & Levy 2014). To quantify the level of "copy-fore fooya-forwrrdigg" within a sequence, we calculate the n-gram repetition rate (RR) for each EHR sequence our dataset using n 1,
We define the n n-gram repetition rate as the proportion of n-grams in a given sequence that are repeated least once. A higher repetition rate means a sequence more repetitive. Formally, we define the n-gram repetition rate as follows:
RR, (z) Eneu(a) I(((,,x) 1]
U(x)
where ((()) the set of unique n-grams in the sequence and Cu,,x) R is the count of occur- rences of the n-gram E 4 in the sequence æ. We define I[[] as the indicator random variable that if the condition inside the brackets is true. and 0 otherwise.
We calculate n-gram repetition rates for n 2, 3, 4 across all 0.5M patients in our EHR-OMOP validation dataset. In Figure X we compare the observed repetition rate our EHR dataset to the repetition rates observed in the WikiText- corpus to demonstrate the higher levels of repetition EHR sequence data. We repeat our analysis Appendix Figure S but first remove patients with less than 20 total clinical events in order to give more accurate picture of the level of repetition seen in the timelines patients with "meaningful" levels engagement with the healthcare system.
F.2 IRREGULARITY
Irregularity in EHR data arises from uneven time intervals between clinical events for each patient McDermott et al. 2023). We define three metrics to quantify the irregularity of a given patient's s EHR sequence. These metrics help capture the variability in timing between events, which is critical for models dealing with irregular time intervals.
Standard deviation of inter-event times:
Let X₃ represent the sequence of clinical events for patient& i. Let t.y represent the timestamp of the j-th event in X₃. Then the irregularity I1 of patient using the standard deviation of inter-event times is given by:
Aay = 40+1) -
lij,
Vj (1,...,, -1) 1}
X
-1
14 =
Atij
[Xil -
j=l
Ix
100
Σ
(Atij - pi)²
X
j=1
25

---

Published as a conference paper at ICLR 2025
Figure 8: Distribution of 7-gram repetition rates across patients the EHR-OMOP validation set. We repeat our analysis from Figure in the main text (reproduced in the bottom in orange), but also include a version in which we first filter out all patients with less than 20 total events before generating our plots (top row in blue). This helps to clearly show that patients with "meaningful" length encounters with the healthcare system tend to have highly repetitive EHR timelines. The x-axis represents the n-gram repetition rate (i.e. percentage of n-grams that repeated least once within a patient's 's EHR), and the y-axis shows the number of patients in each bin.
Mean inter-event time: We can also estimate irregularity as I1(0, which represents the mean time between events and given by:
IX.-1 Σ j=l
Ats
[x -
Interquartile range (IQR): We also estimate irregularity as iQR which represents the in- terquartile range of the time intervals between events and is given by:
(0)
1QQ (rs -((xx1-1)) (tts At((x.)--))
where Q.(-)) returns the n--hh percentile of its arguments.
F.3 INCREASED TOKEN COMPLEXITY DUE TO DISEASE PROGRESSION
As patients age, their diseases become more complex and varied. Thus, we should expect to see tokens later in a patient's timeline to have higher perlexity than tokens earlier in a patient's timeline. all
In natural language, the uncertainty of later tokens in document is reduced by conditioning prior tokens, such that later tokens in a prompt typically exhibit substantially lower perplexity than earlier words (Kaplan et al. 2020). We found that this trend did not hold with EHR data, per the experimental set-up described below.
To quantify how the complexity of disease changes time, used the median perplexity mea- sured at each token position across patient EHRs. Under our hypothesis of disease progression, later tokens should have higher perplexities, even when conditioning all prior tokens in patient's medical history.
Perplexity measures the uncertainty in model's predictions and computed
26

---

Published as a conference paper at ICLR 2025
1
Perplexity(x)- exx
log P(a: x<i))
Where vj is the current token and P(x; Tee)) is the predicted probability of the token given the preceding tokens.
More specifically, we start by sampling 20,000 patients from the EHR-OMOP validation set and tokenizing their full timelines. We use this set of patients for all of our subsequent evaluations.
We then select one of our trained models (e.g. Llama with a context length of 512). We use this model to inference on the full length of each of these 20,000 patients' timelines. This yields a perplexity score for every token. For patient timelines that are longer than the model's context window, we use sliding window of 32 tokens.
After running inference on all 20,000 patients with this model, we then calculate the median per- plexity output by the model at each token positions. We use median rather than mean to reduce the influence of outliers, which we found to be problematic in early testing. We use these median per- plexity scores as our official measurement for that token position's 's perplexity under that model. For our plots, we apply an exponential moving average over the past 250 token positions for smoothing.
F.4 EHRSHOT STRATIFICATION
To stratify model performance on EHRSHOT by the repetitiveness the underlying patient, we first calculate the 1-gram repetition rate (RR) for each patient in the EHRSHOT test set. After grouping the EHRSHOT test patients by the tasks they belong to, we then stratify the patients within each task by their associated 1-gram RR. We sort patients into 4 quartiles, with QI containing patients with the lowest RRs (i.e. the least repetitive patients) and Q4 containg patients with the highest RRs (i.e. the most repetitive patients). For each model and each quartile, then calculate the average Brier score achieved by that model on all patients within the quartile. This yields one Brier score per quartile per model per task. We chose the Brier score our performance metric because certain strata exhibited uniform labels, which rendered AUROC calculations infeasible. We repeat this process across all tasks and models.
To obtain a single "Q1" Brier Score for a specific model, we take an unweighted average of the previously calculated mean Brier score for the QI patients for each task. We repeat this process for Q2/Q3/Q4 to fill out the full row in the table for specific model.
For testing the statistical significance of whether two models achieve different Brier scores for the same quartile, we perform 1,000 bootstrap samples over the EHRSHOT test set.
G FEW-SHOT LEARNING ON EHRSHOT
We define k-shot evaluation model on a specific task T as follows:
1. Training:
For each task T, we sample k positive and k negative examples from the training split of T to train the model M.
2. Validation: An additional & positive and k negative examples sampled from T"s vali- dation split to tune hyperparameters for M on
3. Testing: The best-performing version of M, based validation results, is evaluated the entire held-out test split of T. AUROC recorded as the performance metric.
For tasks where the total number of unique positive examples is fewer than k, all positive examples are included in the training set, and positive examples are randomly resampled until training examples are achieved.
G.1 EXPERIMENTAL SETUP
We considered values of ls E {8,16, 16, 32, 64, 128] for all 14 EHRSHOT tasks, with one exception: for the Celiac prediction task, limited k 64 due to the dataset's constraint of only 62 posi-
27

---

Published as a conference paper at ICLR 2025
tive training examples. This approach ensures faimess in evaluating performance across tasks with varying dataset sizes and class imbalances.
G.2 RESULTS
As shown in Appendix Tables 13 and 12 and Appendix Figure 10 our few-shot learning results indicate that model performance, measured by AUROC, improves consistently l increases. Longer-context models, particularly Mamba, demonstrated notable gains even lower values of k, underscoring their robustness in data-limited scenarios. This trend was consistent across most benchmark tasks, underscoring the utility of long-context architectures in low-resource settings. Our key observations are follows:
Performance Gains with Context Length: Longer context lengths generally led to better performance, with Mamba models achieving the highest AUROC scores across several k- shot settings, especially 16,384 tokens.
Impact of Few-Shot Sample Size (k): All models showed improved performance with increasing k but Mamba and Llama benefited more significantly at higher values of (64 and 128), consistently outperforming other models across tasks.
H ZERO-SHOT LEARNING ON EHRSHOT
We also evaluate subset of models under the zero-shot setting, ie we simply run inference each model without any finetuning. This offers the practical benefit of not having to train or store any fine-tuned task-specific model heads.
H.1 EXPERIMENTAL SETUP
We follow the procedure outlined in the ETHOS paper (Renc et al. 2024) for making our zero-shot predictions. In brief, we generate 20 synthetic timelines for each patient at the prediction time, measure the percentage of timelines in which the positive event for task occurs, and then use that percentage as the probability that the patient experiences that positive event. For our zero-shot evaluations, choose two strongest models (Mamba and Llama) at their minimum and max- imum context lengths, and evaluate them on three representative EHRSHOT tasks new diagnosis of hypertension, 30-day readmission, and new diagnosis of acute MI.
H.2 RESULTS
As shown in Appendix Table 15 our zero-shot results significantly lag behind the performance of our few-shot and finetuned models. None of the zero-shot models beat the prior SOTA model on any of the three tasks evaluated. Additionally, results across context lengths appear mixed. This underscores the importance of finetuning for clinical prediction making, and suggests that our training pipeline is not optimally designed for zero-shot evaluations.
28

---

Published as a conference paper at ICLR 2025
Tarsa
rumba
Pyero
Mar SOTA
mandrr TANA, #ERnran-
5100
311
Calaed
U4e7 2015 Aespure..r
Pasls 11.0% MPEPLTRRES
2010
0.4
0.0
347
3500
0216
5001
1228
1508
2052
8000
AM0
3003
7500
8030
3750 1757 185
3.70
375
3.75
312
505
332
Leng
6.80
6905
0.75
0.00
3.3
342
7300
M00
W061
1908
200
210
8000
1972
2000
M003
250
800 1050
Tel0day/sar Fardh B1RTraRATT
6.6%
0.70
0.02
Figure 9: AUROC by context length and architecture across all 14 tasks evaluated from EHRSHOT. The highest scoring model for each task is listed above its plot. Note that the "Prior SOTA" is selected on a task- by-task basis, and thus is not necessarily the same model across plots.
29

---

Published as a conference paper at ICLR 2025
Model
Comext Longth
CLMBB--1bbse
C1
Siguificant
mamba
1034
Admission
(-0.390, 0.019)
Long
-0.003
(-8.115, 0.010)
mumba
1024
30-3ay Readmission
0.001
(.0001.0.011))
mambs
1074
Anemia
0.000
1.0001,0.000
1024
1yperkalemia
(-0.006.0001
numba
1034
Hypoglyormia
0.001
600001.0.011)
mamba
1074
Hypanstremia
0.014
(0.007, 0.022)
mumba
1024
Thrombopttopeni
-0,055
(40.010- 0.001)
1024
0.102
(-0.076
mumba
1024
Iyperliiidemii
0,020
(-0.010, 0.0800
numbs
1024
Hyperiension
-0,011
(-0.034, 0.011)
1074
Tapus
(-0.115,
mumba
1024
Pancreatic Cancer
0.032
(-0.008, 0.071)
mamba
4096
ICU Admission
0.004
(-0.034, 0.029)
mamba
4096
Long LOS
0.005
0.0100 0.021)
Readmission
(-0,005.
mamba
4006
0.002
(0.001, 0.003)
mumba
4096
Hyperkalemia
0.024
(0.014, 0.034)
numba
4096
Hypoghoeemia
0.001
(-0,012.000 0.013)
Hyponstremia
0.066
mumba
4096
Thrombooytopenia
0.007
(0.002, 0.011)
mumba
4096
Aculle MI
0.014
(-0.009, 0.036)
mamba
4096
Celiac
0.198
00,115, 0.288)
Hyperlipidemia
(-0,034.0005
numba
4096
Hyperteenion
00,000
(-0.033. 0.110)
mamba
4096
Lupus
0.003
(-0,991_ 0.086)
mumba
4096
Pancreatic Cuncer Admission
0.019
(0.017, 0.081)
(-0.033,
mamba
8192
Long LOS
0,009
0.000 0.024)
mumba
8192
30 Readmission
0.003
(-0.010, 0.016)
mamba
8192
Anemia
0.001
00.000
mumba
8192
Hyperkalemia
0.029)
mumba
8192
Hyppogyyemmia
00002
1.0.044..0010
mamba
8192
Hyponatremia
0.063
(0.053, 0.072)
mamba
8192
Thrombocytoeeni
0.004
0.001 0.008)
0.036)
mamba
8192
Cellac
0.173
(0,033, 0.312)
mamba
$192
Hyperlipidemia
0,030
(-00111 0.068)
mamba
8192
Hypertension
-0.016
mamba
8192
Lupus
0.113)
mumba
8192
Pancreatic Cancer
0.027
0.0.00 0.062)
mamba
10384
ICU Admission
0.007
(-D.125, 0.040)
mamba
16384
Long LOS day Readmission
0.013
0.029)
0.017)
mambe
10384
Aneuia
0,002
(0.001, 0.005)
mamba
16384
Hyperkalemia
0,030
(0.019, 0.042)
mamba
16381
Hypogboemmia
0.006
0.000, 0.019)
Hyponatremia
0,070
0.0799)
mamba
16384
thromnboytooeeia
0,008
(0.004, 0.013)
namba
16384
Acute MI
B.16
(-D.005, 0.036)
Celinc
0.335)
Hypetlipidemia
0.058)
mambe
16384
Hypertansion
0.003
(-0.018, 0.025)
mamba
10384
Lupus
D.037
(-0.056. 0.132)
mamba
16384
Pancreutic Cuncer
0.053
(0,024, 0.087)
Table 7: Performance of Mamba across all context lengths the 14 EHRSHOT tasks. The column "A over CLMBR- t-asse contains the increase in AUROC relative CLMBR- the prior SOTA model on EHRSHOT. The column "95% CI" contains a bootstrapped confidence interval calculated over 1,000 samples of the test set. The column "Significant" contains a checkmark if the CI does not intersect with O.
30

---

Published as a conference paper at ICLR 2025
Length
CLMBR.<base
Signiticant
Ioma
512
Admission
(4,052, 0.015)
Nana
Long LOS
(-0044
Bama
512
30-day Reudmission
0.012
10,000 0.004)
Hama
$12
Anemia
-0.004
(4005 -0.00))
[lyperkalemia
Nama
512
Hypoglyoemia
-0.01]
(-0002) 0.001)
Hama
512
Hyponstremis
-0.010
(40016, -0.004)
laaa
512
Thoombayylpenna
-0.001
0.004
MI
(.037)00.37)
Bama
Cillic
(0.111, 0.356)
Rama
512
Ilsperlipidemia
0.001
(-0.018, 0.020)
Nama
512
Flypetienionn
-0.035
(-0087, -0.012)
Lupus
lama
512
Pancrentic Cancer
0.001
(-0004, 0.046)
Bama
1024
ICU Admission
-0.005
(-0.422 0.032)
laaa
1024
Long LOS
00,013
(40.044 0.005)
Bama
Anemia
-0.044
(.0005 -0.00))
laaa
1024
Elyperkalemia
0.010
10.022, 0.019)
Nama
1024
Elyroglyeemia
-0.00
(-0.014 0.008)
1024
Flyponsenemis
[-0.10,
lama
1024
Thrombacylopenia
00,05
0.009, 0.0000
Rama
1024
Acute MI
0.007
(-0.014.0.039 0.029)
laaaa
1024
Celine
0.200
(0.149, (0.149.0.359)
[lperripidemia
(-00166
Hama
1024
Elypertension
-0.014
(-00033 0.003)
llama
1024
Lupus
00.04
[-0.102, 0.079)
laaa
1024
Pancreatic Cancer Admission
00.007
(-0.053, 0.037)
(-0.011 0.035)
llama
2048
Long LOS
0.014
[-00000 0.029)
lama
2018
30 Readmission
0.010
(-0.000, 0.023)
Ilama
2048
Anemia
0.0013
Ilama
Hlpperkalemia
10,055 0.025)
lama
2048
Elypoglyeemia
0.011
(.0002 0.025)
llaaa
2048
Hyponutremia
0.013
(00055 (0020)
llama
20188
Thrembacyiupenia
00,00
(-00000 0.004)
(-0001,
llaaa
2048
Celiac
0.222
10.083 0.243)
laaaa
2018
Elyperiipidmmia
0.021
(-0.005 10,019)
llaaa
2048
Hypenension
1-00025
Ilama
Lupus
(-0049 (,119
llama
2048
Puncreatic Cancer
0.007
(-0002 0,053)
Baaa
4196
ICU Admission
-0.03
1-00025 (0.121)
Ilama
4096
LOS
1-00008, 0,010)
10,02,
Haaa
.196
Anemia
0.001
(0000) (.002)
llaaa
4096
Hyperialemia
0.04
(0006, (.033)
laaaa
4096
Hypoglyeemia
0.012
(-0,000, 0.022)
laaaa
Hyponutremia
(0,028,
laaaa
.0966
hhoombocylppnna
0.000
[40.04, 0.005)
Boaa
4196
Acute MI
0.015
[40.008 0.135)
llaaa
Celine
Hyperiipidemia
(40.002
Hama
.1966
Hypentnsion
0.004
(-0013, 0.021)
liaaa
4096
Lupes
-0.023
1-0.09) 0.049)
lama
4096
Puncreatic Cancer
(40.066 0.033)
Table 8: Performance of Llama across all context lengths the 14 EHRSHOT tasks. The column "A over LLMBB-t-base contains the increase in AUROC relative CLMBR-t- the prior SOTA model on EHRSHOT. The column "95% CI" contains a bootstrapped confidence interval calculated over 1,000 samples of the test set. The column "Significant" contains a checkmark if the CT does not intersect with O.
31

---

Published as a conference paper at ICLR 2025
Length
CLMBR&<base
Signiticant
512
Admission
(4.005,
8/02
Long LOS
(-0017, 0.812)
1912
512
30-day Reudmission
-0.002
(4.013, 0.009)
1002
$12
Anemia
-0.003
(40004-000) -0.002)
sp12
[lyperkalemia
1012
512
Hyprolyyemia
-0.00
(..014.0.012)
1o02
$12
Hyponstremis
0.037
(0028 0.046)
512
Thoombayylpenna
0.020
(0011.0.0255)
MI
1y02
Cillnc
(0.03, 0.795)
sp12
512
Itsperlipidemia
00.041
0.043)
512
Flyperiension
-0.003
(-0.021.0:044 0.014)
Lupus
(40.110,
ap12
512
Pancrentic Cancer
(-00028, 0.054)
1012
1024
ICU Admission
-0.031
(-0.052, 0.009)
1024
Long LOS
00004
(40002 0.004)
up12
(40.009..013
1002
Anemia
(-0012.
gp12
1024
Elyperkalemia
0.022
(00011 0.033)
1024
Elyroglyuemia
-0.009
(-00022 0.004)
1024
Flyponsenemis
Epe2
1024
Thrombacylopenia
0.013
10.09, 0.019)
up12
1024
Acute MI
-0.003
(-0.027, 0.021)
1024
Celine
(0.007.0.2244
sp12
[lperripidemia
(-0053)
ap12
1024
Elypertension
-0.026
(-0049, -0.005)
sp12
1024
Lupus
00,016
[-0.090, 0.062)
102.1
Pancreatic Cancer Admission
0.022
(-0.009, 0.050)
1912
(-0.040,
Epc2
2048
Long LOS
0,008
[-0.02, 0.005)
sp12
2018
30 day Readmission
0.002
(-0.011, 0.014)
2048
Anemia
0.003)
sps2
Elyperkalemia
(-0,000
ap12
2048
Elypoglyeemia
0.001
(-00133 0.013)
2048
Hyponutremia
0.023
(0013, 0.025)
2018
Thrembacyiupenia
(0.06,
1p12
Eps2
2048
Cellac
0.227
10.337 0.433)
gp12
2018
Elyperiipidemia
0.005
(-0.014, 0.025)
2048
Hypenension
(-11021,
gpc2
Lupes
10,00,
Ep12
2048
Puncreatic Cancer
0.001
(-00022 0.037)
4196
ICU Admission
0,011
1-000211 0.044)
LOS
1-0.024, 0.014)
spd2
(-0.009, 0.015)
8p62
.196
Anemia
-0.005
(-0006 .0004)
gpx2
4096
Hyperialemia
0.011
(0.001 0.021)
4096
Hypoolyyemia
0.003
(-0.011 0.015)
gp62
Hyponutremia
(0.066,
Eps2
4096
hhrobboyloppnia
0.014
10.09, 0.018)
4196
Acute MI
0.006
[-0,02 0.033)
gps2
Celine
8p62
Hyperiipidemia
(-0.018, 0.043)
8062
4096
Hypenension
0.004
(-0015, 0.034)
4096
Lupes
-0.005
1-0.095. 0.085)
4096
Puncreatic Cancer
(40.08, 0.062)
Table 9: Performance of GPT across all context lengths the 14 EHRSHOT tasks. The column "A over CLMBR- --bsee" contains the increase in AUROC relative CLMBR- the prior SOTA model on EHRSHOT. The column "95% CI" contains a bootstrapped confidence interval calculated over 1,000 samples of the test set. The column "Significant" contains a checkmark if the CI does not intersect with O.
32

---

Published as a conference paper at ICLR 2025
Context Length
CLMBR&+base
Significant
byans
1024
Admission
-0.026
(4.064 0.013)
hyens
Long LOS
(-0020, 0.811)
hyyna
1024
30-day Reudmission
-0.001
(0.012,0.010)
byens
1034
Anemia
-0.002
(40003 -0.001)
byena
[lyperkalemia
hyena
1024
Hyyroglycmii
-0.044
0.008)
byens
1024
Hyponstremis
0.045
(0006. 0.055)
hyena
1024
Thoombayylppnna
0.019
0.00440.004)
hyena
MI
(4.015.0.03))
byens
Cillicc
0.367)
hyena
1024
Ilsperlipidemia
0.018
(-0.000, 0.037)
hyena
1024
Flyperiension
-0.026
(-0083,, -0.003)
byens
1024
Lupus
-0.02
hyena
1024
Pancrentic Cancer
[-0.022, 0.060)
hyena
4096
ICU Admission
-0.036
(-0.058, 0.004)
byena
4096
Long LOS
00,012
(40.000 0.005)
hyena
0.013)
hyens
Anemia
-0.005
(.0006 -0.004)
hyena
4096
Elyperkalemia
0.022
10.013,000 0.003)
hyena
4096
Elyroglyeemia
-0.013
(-0.027 0.001)
byena
Flyponsenemis
hyena
4096
Thrombacylopenia
0.018
10013, 0.023)
hyena
4096
Acute MI
0.013
(-0.013,0000
byena
4096
Celine
0.16
(0.077.0.370)
hyena
[lperripidemia
hyem
.096
Elypertension
-0.023
(-00500, 0.002)
hyena
4096
Lupus
00,019
[-0.110, 0.056)
hyena
4096
Pancreatic Cancer Admission
0.038
(-0.011. 0.092)
hyena
hyena
8192
Long LOS
0,023
(00011 0.004)
hyena
$192
30 Readmission
00.017
(-0.033, 0.002)
byena
Anemia
40.06
0.014)
hyena
Hlpeerkalemia
10.000,00
hyema
8192
Elypoglyeemia
-0.41
(-0056, 00025))
hyena
$192
Hyponutremia
0.099
(0.029, 0.029)
hyena
8192
Thrembacylupenia
(-0,001, 0.010)
hyena
hyena
8192
Celiac
0.134
1-00013, 0.352)
hyena
$192
Elyperiipiemiia
0.014
[-0.026, 0.052)
Hypenension
-0.06
.0.080)
hyena
Lupes
(-0,18%
hyena
8192
Puncreatic Cancer
0,003
[-0008, 0.018)
hyena
163344
ICU Admission
-0.110
(-0.477 .0075)
byena
163384
LOS
-0.08
(-0.688 0.029)
hyena
(-0.67,
hyena
16:284
Anemin
-0.47
(-0051, ..043)
hyena
16388
Hyperialemia
00.03
(-0054, 0.023)
hyena
16384
Hypoolyyemia
00.93
(-0.009, 0.075)
hyena
Hyponstremia
(-0,002
hyena
16334
hhoobooyloppnia
0.003
[-0.005, 0.011)
hyena
16384
Acute MI
-0.100
(-0145, -0.53)
byena
163844
Celine
hyena
Hyperiipidemia
(-0.099
hyena
16384
Hypentesionn
-0.071
(-0.125, .0.025)
hyena 10384
Lupes
-0.145
(40268, .0.17)
hyena
16384
Puncreatic Cancer
(40.148, 0.005)
Table 10: Performance of Hyena across all context lengths the 14 EHRSHOT tasks. The column "A over L.MBR-t-base" contains the increase in AUROC relative CLMBR-t-bar the prior SOTA model on EHRSHOT. The column "95% CI" contains a bootstrapped confidence interval calculated over 1,000 samples of the test set. The column "Significant" contains a checkmark if the CI does not intersect with O.
33

---

Published as a conference paper at ICLR 2025
k
Model
Context Length
16
128
gp12
gp12
1024
0.634
0.697
0.732
0.758
0,774
0.813
gpt2
2048
0.704
0.771
0.792
gpt2
4096
0.657
0,706
0.742
0.769
0,791
0.828
Ilama
512
0.672
0.716
0.741
0.767
0,786
0.822
Ilama
1024
0.662
0.707
0.737
0.769
0,788
0.821
Iama
0.714
Ilama
4096
0.665
0,709
0.756
0.782
0.800
0.826
mamba
1024
0.668
0.719
0.745
0.774
0.786
0.820
mamba
4096
0.681
0,730
0.754
0.784
0.796
0.828
mamba
16384
0.685
0.734
0.761
0.791
0.804
0.831
hyena
1024
0,655
0,705
0,739
0.761
0.778
0.813
hyena
4096
0.631
0.681
0.725
0.747
0.773
0.811
hyena
8192
0.622
0.669
0.698
0.727
0.750
0.788
hyena
16384
0.587
0.629
0.651
0.676
0.705
0.755
Table 11: Few-Shot Evaluation: Average AUROC score for each model and context length across all Operational Outcomes tasks and k-shot settings. The highest AUROC across all models for each A: is bolded underlined, the maximum value within each model across context lengths for each is bolded.
Model
Context Length
128
gp12
512
0.603
0.634
0.670
0.695
0,713
0,730
gpt2
1024
0.610
0.644
0.672
0.691
0.711
0.719
gp12
2048
0.621
0.654
0,684
0.709
0,726
0,756
gpt2
4096
0.616
0.642
0.678
0.700
0.722
0.734
Ilama
512
0.606
0.635
0.665
0.687
0.721
0.739
1024
llaaa
2048
0.624
0.683
0.675
0.694
0.728
0.751
Ilama
4096
0.621
0.646
0.679
0.721
0.741
mamba
1024
0.628
0.652
0.682
0.698
0.716
0.725
mamba
4096
0.630
0.658
0.704
0.726
0.747
mamba
8192
0.633
0.657
0.690
0.706
0.723
0.747
mamba
16384
0.647
0.668
hyena
1024
0.621
0.651
0.682
0.697
0.717
0.740
hyena
4096
0.608
0.638
0.666
0.680
0.709
0.745
hyena
8192
0.585
0.608
0.638
0.657
0.671
0.699
hyena
Table 12: Few-Shot Evaluation: Average AUROC score for each model and context length across all As- signment of New Diagnoses tasks and k-shot settings. The highest AUROC across all models for each bolded underlined, and the maximum value within each model across context lengths for each is bolded.
34

---

Published as a conference paper at ICLR 2025
Membe
Operational Ouicomes
Antrppoiin Las rost Rowets
Ass grmmerr how Uhagrisses
380
380
Feurglny per
Liama
Operational Outcomen
Articppting Lab Tol Roout
Asa grmaerr how Diegricss
0.55
AM
03
337
3.0
335
Operational Outcomes
Antiopacing Rosurs
AdS gameet Now Dlagroses
X75
3.5
Figure 10: Few-Shot Evaluation: Average AUROC scores for each model and context length across all few-shot settings, aggregated for each EHRSHOT clinical prediction task group: Operational Outcomes, An- ricipating Lab Test Results, and Assignment of New Diagnoses. Each row a different model (from top to hottom: Mamba, Llama, GPT, Hyena) and each column is a task group. The x-axis shows the number of few-shot examples (k-shot), while the y-axis displays AUROC. Each line represents a different context length. Solid lines AUROCs average across all subtasks within task group. while lighter lines the few-shot results individual subtask.
35

---

Published as a conference paper at ICLR 2025
gpt-base (with ROPE) Token position vs. Median PPL
gpt apt-base-512
22.5
gpt apt-base-1024
gpt-bass--2048
20.0
gpt-base-4096
17.5
Median PPL (EMA)
15.0
12.5
10.0
7.5
5.0
2.5
2500
5000
7500
10000
12500
15000
17500
Token position
Figure 11:
Reproduction of Figure for the GPT architecture, but with rotary positional embeddings (ROPE) instead of absolute positional embeddings. All other aspects of the GPT architecture are kept the same. With ROPE, the perplexity curves appear more stable and do not exhibit the 10+ point perplexity spikes in Figure but still mirror the trend of increased perplexity with increased sequence length.
Mean AUROC Across All Tasks vs. Context Length
0.81
0.80
0.79
0.78
AUROC
0.77
0.76
Models
gpt-att
0.75
mamba
hyena
0.74
hyena-att
llama-att
Prior SOTA
2000
4000
6000
8000
10000
12000
14000
16000
Context Length
Reproduction of Figure Db. but with models trained using Artificial Time Tokens (ATTs) (as defined in CEHR- (Pang et 2021)) shown dotted lines, and models trained without ATTs solid lines. Overall, we see better performance without using ATT tokens. While the dotted lines closely follow the solid lines for Mamba and Hyena, the transformer models appear to have less stable performance at smaller contexts, potentially due the injection of more tokens within each patient's 's timeline.
Figure 12:
36

---

Published as a conference paper at ICLR 2025
Model
Context Length
gp12
512
0.649
0.669
0.704
0.733
0.766
0.845
gpt2
1024
0.639
0.665
0.694
0.730
0.763
0.843
gpt2
2048
0.643
0.667
0.696
0.726
0,761
0.841
gpt2
Ilama
512
0.647
0.672
0.735
llama
1024
0.635
0.665
0.696
0.728
0.762
0.831
Ilama
2048
0.643
0.669
0.707
0.741
0.772
0.839
Ilama
4096
0.647
0.709
mamba
1024
0.656
mamba
4096
0.640
0.669
0,706
0.734
0,770
0.852
mamba
8192
0.638
0.666
0.701
0.733
0.768
0.849
mamba
16384
0.644
0.666
0.705
0.738
0.776
0.858
hyena
0.647
hyena
4096
0.632
0.655
0.688
0.725
0.759
0.850
hyena
8192
0.615
0.642
0.672
0.697
0.737
0.833
hyena
16384
0.575
0.594
0.615
0.634
0.668
0.799
Table 13: Few-Shot Evaluation: Average AUROC score for each model and context length across all An- ticipating Lab Results tasks and k-shot settings. highest AUROC across all models each bolded underlined, and the maximum value within each model across context lengths for each is bolded.
Context Length
Metric
Model
Q2
Q3
Q4
Repetitiveness (I-gram RR)
Mamba
Ik
0.0644
0.0737
0.0744
0.0790
16k
0.0605
0.0670
0.0700
0.0746
Llama
512
0.0640
0.0710
0.0743
0.0792
GPT
4k
0.0643
0.0692
0.0711
0.0765
Hyena
Ik
0.0636
0.0681
0.0718
0.0776
16k
0.0733
0,0759
0.0780
0.0822
CLMBR-4-base
512
0.0647
0.0719
0.0751
0.0805
Irregularity (Standard Deviation)
Mamba
1k
0.0693
0.0729
0.0731
0.0764
0.0641
Llama
4k
0.0664
0.0705
0.0694
0.0740
GPT
512
0.0654
0.0693
0.0703
0.0736
4k
0.0653
0.0699
0.0701
0.0759
Hyena
1k
0.0666
0.0702
0.0692
0.0751
16k
0.0698
0.0755
0.0788
0.0853
CLMBR-4-base
0.0683
0.0741
0.0721
0.0777
Table 14: Comparison of average Brier scores for all models across all 14 EHRSHOT tasks. Patients
bucketed by repetitiveness (top) and irregularity (bottom). Q1/Q2/Q3/Q4 are the 1st through 4th quartiles of patients ranked by each metric. For example, QI contains the least repetitive least irregular patients while Q4 contains the most repetitive most irregular patients. Bolded values show statistically significant win rate of least 50% of the longer context model over the shorter context model specific quartile. This is identical Table 2 but with all models shown.
37

---

Published as a conference paper at ICLR 2025
Model
Context Length
AUROC
Hypertension
CLMBR-t-base
512
0.718
Mamba
1024
0.660
Llama
512
0.642
Llama
4096
0.609
Mamba
16384
0.563
30-day Readmission
CLMBR-I-basc
512
0.810
Mamba
1024
0.720
Llama
4096
0.710
I.lama
512
0.705
Mamba
16384
0.643
Acute MI
CLMBR-E-base
512
0.729
Mamba
16384
0.531
Mamba
1024
0.525
Llama
4096
0.52
Llama
512
0.51
Table 15: Zero-Shot Evaluation: AUROC scores for each model and context length for zero-shot evaluations three EHRSHOT clinical prediction tasks. The zero-shot evaluations followed the procedure outlined (Rene et 2024). Namely, 20 synthetic timelines were generated for each patient at cach prediction timepoint. The probability that a patient experienced positive event was calculated as the percentage of generated timclines that contained that positive event within the appropriate time horizon as defined by the relevant task.
38


Reviews:
[{"review_id": "z0pnflZ41R", "reviewer": "Reviewer_ACwG", "summary": "This paper explores enhancing EHR data processing models by assessing the impact of context length and model architecture on clinical prediction tasks. Four architectures—GPT, Llama, Hyena, and Mamba—were tested with various context lengths, with Mamba (16k token context) achieving state-of-the-art results in 9 of 14 EHRSHOT benchmark tasks. The study identifies three EHR-specific factors influencing model performance: copy-forwarding (repeated diagnoses), irregular time intervals, and disease progression. Findings show longer contexts generally improve performance, with robustness in EHR data handling, though results vary across architectures, as Hyena showed performance drops beyond 4k tokens. This research advances medical data prediction and offers insights on processing long-sequence data, despite limitations like transformer model computational demands and single-institution data.", "questions": "Since the author's testing only showed good results with small Mamba models, it's uncertain whether larger Mamba models would perform better than traditional GPT.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Introduce non-Transformer architectures (such as Mamba) to process medical data.\n\n2. The paper demonstrates the potential of the Mamba model’s long-context capabilities for future clinical applications.\n\n3. The advantage of this paper is that Mamba can perform well with linear complexity.", "weaknesses": "1. Both GPT and LLaMA have a maximum context length of only 4096 tokens, so it's not appropriate for the authors to conduct tests with 16K length as a benchmark.\n\n2.The authors should provide additional details about what the 14 tasks in EHRSHOT include to facilitate better comparison.\n\n3. The paper mentions using the standard deviation of time intervals to divide patients into four groups (Q1-Q4). Regarding the groups with the most regular time intervals and the most irregular time intervals, the standards vary across different diseases, and testing of time intervals should be conducted according to specific diseases.\n\n4. The paper mentions achieving an AUROC of 0.807, but it's confusing that the specific content of the 14 tasks is not listed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0ZtAa1joXa", "reviewer": "Reviewer_wRBt", "summary": "This paper analyzes qualities unique to EHR data that are challenging when scaling up sequence lengths. They identify the key properties of copy-forwarding (EHR data is repetitive), irregular time intervals (time intervals vary greatly between tokens), disease complexity (EHR data actually has generally increasing perplexity overtime--unlike text data--as far out future trajectories are hard to predict). They evaluate their models in zero-shot, few-shot, and linear probing regimes, varying context lengths and architectures, and even sharing compute time estimates via token throughputs.\n\nThe authors share metrics for quantifying the severity of these three properties on EHR datasets, and demonstrate that severity of these properties correlates with worse performance. They additionally show that longer sequence lengths can help.\n\nEvery weakness I pointed out in the initial review has been thoroughly addressed by the authors, this paper is both a clear accept and a major contribution in the EHR machine learning space as it provides a broad benchmark and insights on training autoregressive models in this space.", "questions": "I included my questions in the Weakness section, but I'll summarize the actionables below:\n\n1. **The most critical improvement to this paper**: Include a few-shot/transfer-learning results, the use of mean pooling the last L tokens for all evaluations is not a well supported embedding strategy. Alternatively, you have a zero-shot capable model, why not evaluate it in the zeroshot setting as past works do [1,2,3]? At the moment the limited embedding strategy greatly diminishes the generalizability of your results. If this were resolved, I would significantly increase my rating for the paper.\n2. Add compute time results, and analysis of the diminishing or increasing predictive performance returns as you increase compute times (via increasing sequence lengths) across methods.\n3. A more complete Analysis of the erratic GPT-model Perplexity behavior\n4. Add a comparison with linear attention models\n\n\n[1] Renc, Pawel, et al. \"Zero shot health trajectory prediction using transformer.\" NPJ Digital Medicine 7.1 (2024): 256.\n\n[2] Kraljevic, Zeljko, et al. \"Foresight—a generative pretrained transformer for modelling of patient timelines using electronic health records: a retrospective modelling study.\" The Lancet Digital Health 6.4 (2024): e281-e290.\n\n[3] McDermott, Matthew, et al. \"Event Stream GPT: a data pre-processing and modeling library for generative, pre-trained transformers over continuous-time sequences of complex events.\" Advances in Neural Information Processing Systems 36 (2023): 24322-24334.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The authors define metrics for evaluating the severity of these properties on any dataset.\n\n2. They demonstrate that as the RR metric and irregularity metric increase, model performance decreases (via patient stratification experiments).\n\n3. The demonstration that perplexity does not generally decrease with sequence length is a major deviation from text data where it is well known to decrease with sequence length. This is because future EHR data is less predictable from previous tokens. The assumption that perplexity reduces with context length for EHR \n\n4. The authors demonstrate that for all three properties, increasing sequence length helped improve performance (either via improved brier scores or perplexity).", "weaknesses": "The authors provide in Figure 4 plots of perplexity over tokens for different context-length models. The GPT model has wildly varying perplexity over token positions which is described by the authors as being caused by \"training instability\". I think a more thoughtful analysis of the issue here is required, because it would otherwise look like the cause is a bug.\n\nWhy isn't transfer-learning or few-shot included. A major problem in the evaluation is that representations are obtained by averaging the last L token representations from the transformer (This is a one-liner in the appendix and really should be added to the main paper and be clearly communicated in the limitations section). It would be great to see these results in the few-shot setting. I imagine that the performance improvements as you increase sequence length would be even more extreme.\n\nThis paper should include comparisons to linear attention models that practitioners are interested in.\n\nThis paper does not communicate compute budgets, such as wall-times and the hardware used for these sequence lengths. Could a plot communicating performance vs the compute-load be provided to help justify whether these improvements are worth the added compute time.\n\nSince you trained an autoregressive sequence model, you could perform a zero-shot evaluation where given the past tokens, you autoregressively generate the future tokens and analyze this generated future trajectory for the binary classification task. This paper does not demonstrate whether these results generalize to the zero-shot. I think that analysis is out of scope for this work but should be mentioned in the limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9xf19roLhX", "reviewer": "Reviewer_UYtp", "summary": "The manuscript benchmarks four foundation model architectures on Electronic Health Records (EHRs) to investigate the impact of context length on the model performance of downstream tasks. Moreover, the authors identified and quantified three challenges present in EHR data and showed that long-context models are better at mitigating the challenges.", "questions": "- Statistics of code, event, and token are confusing. \n\n \n- Diagnosis codes for chronic diseases can frequently appear in a patient’s EHR, often for billing purposes rather than indicating an active health issue. For instance, if a patient with a chronic condition (e.g., COPD or obesity) visits the hospital for an unrelated condition, the chronic disease code may not appear in that visit. In contrast, for acute diseases, the presence of a code in the record typically indicates an active case during that visit. How can token repetition be modeled effectively for these two types of diseases?\n\n\t\t\n- In line 261, the authors reported that the vocabulary has 39818 tokens. Is this the vocabulary of EHRSHOT, or does EHR-OMOP also share it? \n\t\t\n- Meanwhile, in line 950 of Appendix C, it is reported that 39811 codes are selected. Is \"code\" here equivalent to \"token\"? \n\t\t\n- In Table 3 in Appendix A, The number of unique codes in EHR-OMOP is much more than the token vocabulary mentioned earlier. Is this the original vocabulary without removing infrequent codes?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors benchmark both transformer-based and subquadratic architectures on EHR data\n2. The authors identified and quantified three challenges present in EHR data.\n3. The authors conduct experiments to show the effectiveness of long-context models on EHR data.", "weaknesses": "1. The design of token repetition measurement is less convincing. \n\t- Are the proposed models applicable to ICU patients? If yes, routine vital signs and regular lab tests can repeat a lot, but they can continuously show patients' health status. It is tricky to determine whether they are informative. \n\n\n2. the comprehensiveness of experimental design is limited\n\t- The investigated methods are limited. They are general architectures for foundation models. However, foundation models designed for EHR, such as [1] and [2], are not included.\n\t- The authors claimed that irregular time gaps hinder the performance of the models. This is reasonable because the time gap is not encoded during tokenization. It could be interesting to see whether encoding time information would be helpful for some stratified groups although the gain may be minimal overall.\n\n3. The experiment result reported is limited for a comprehensive understanding\n\t- Prior SOTA mentioned in the manuscript (CLMBR-t-base) is also a transformer-based model. However, the context length of this model is not discussed. Additionally, it is not trained with variable context lengths.\n\t- Table 2 provides stratified results of the same experiment as Figure 1 (b) and (d). However, it is confusing that CLMBR-t-base and Hyena don't appear in this table. \n\t- The author hypothesizes that some degree of copy-forwarding can be helpful by emphasizing important diagnoses. This is observed from the CLMBR-t-base but cannot be validated by other models. Moreover, the Brier score of the CLMBR-t-base seems smaller than other models.\n\t- Standard deviation is not provided when comparing different models. The authors only conduct statistical tests between short- and long- context counterparts. However, neither standard deviation nor statistical testing is reported when comparing different methods.\t\n\t- (Minor) The impact of the long-context and proposed properties on pretraining is not discussed. The downstream tasks are enough to show the conclusion but it will be better to see if these factors affect training process.\n\n4. There are some typos in the manuscript\n\t- In line 139, there's a corrupted citation\n\t- 3.3.3 title: Diseae -> Disease\n\t- Reference [Yang et al., 2023a] and [Yang et al., 2023b] are the same\nReferences:\n[1] Guo, L. L., Steinberg, E., Fleming, S. L., Posada, J., Lemmon, J., Pfohl, S. R., ... & Sung, L. (2023). EHR foundation models improve robustness in the presence of temporal distribution shift. Scientific Reports, 13(1), 3767.\n[2] Fallahpour, A., Alinoori, M., Afkanpour, A., & Krishnan, A. (2024). EHRMamba: Towards Generalizable and Scalable Foundation Models for Electronic Health Records. arXiv preprint arXiv:2405.14567.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZcB56STQEf", "reviewer": "Reviewer_TDwP", "summary": "This paper investigates the challenges of using language models to model EHR data. By comparing EHR data with natural language, this paper identifies three unique and significant properties of EHR data (copy-forwarding, irregular time intervals, and disease progression) that make modeling EHR sequences more complex than natural language. To provide evidence for these properties, this paper evaluated the performance and robustness of three language models by varying the repetitiveness, irregularity, and context length of EHR data. The models are pre-trained using a private dataset and evaluated with a publicly available dataset, EHRSHOT.", "questions": "Some technical details seem to be missing in the paper. For example, it is unclear what the 14 tasks in EHRSHOT are, whether some tasks have highly imbalanced label distributions, and if the sample sizes for these 14 tasks are the same. Since Table 2 reports the mean performance across all 14 EHRSHOT tasks, the lack of such information makes it challenging to assess the actual performance.\n\nWhen finetuning models, the author wrote \"To be consistent with the original EHRSHOT benchmark, we do not finetune our base models – instead, we train a logistic regression head on top of frozen representations generated for each patient by our base models.\" However, there is no evaluation of CLMBR-T-Base, which is the foundation model released together with EHRSHOT.  Why was CLMBR-T-Base not included in the experiment?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The identified properties of EHR data are convincing. The evaluation of the effects of these properties provides valuable insights into using long-context models to model EHR data. The observations and conclusions in this paper will be helpful for future work to build better foundation models for EHR. \n\nThe authors have released the code and plan to release the model checkpoints later. The release of pre-trained and fine-tuned foundational models will benefit the community, considering the small number of such models currently publicly available.", "weaknesses": "Although the authors test the performance of different language models, the tokenization strategies of these models remain the same as the one used in EHRSHOT. It would be helpful to see if using other tokenization strategies could improve performance. For example, Section 4.3 indicates that irregular inter-token time intervals are harder to model. This conclusion is based on EHRSHOT’s tokenization, which doesn’t encode time intervals. However, there are other tokenization strategies, such as those used by ExBEHRT and EHRMamba, that do encode time intervals.\n\nThis paper uses only one EHR dataset to evaluate language models, which somewhat limits its conclusions to the EHRSHOT dataset. While I understand that unifying different EHR datasets is highly intensive work, it would be valuable to see whether similar observations consistently appear in other EHR datasets, such as MIMIC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UvdOJWvE7h", "reviewer": "Reviewer_uVBS", "summary": "This paper investigates the advantages of long context models in the healthcare domain by evaluating the impact of context length on clinical prediction tasks using four models—Mamba, Llama, GPT, and Hyena—trained on millions of longitudinal EHR records. Additionally, the study assesses model robustness against three properties of EHR data: (1) \"copy-forwarded\" diagnoses that lead to artificial token repetition, (2) irregular time intervals between EHR events causing variable timespans within context windows, and (3) the increasing complexity of diseases over time, which complicates the prediction of later tokens compared to earlier ones. These factors highlight challenges associated with EHR data in clinical prediction tasks. The results indicate that a higher prevalence of each property negatively impacts model performance, while models with longer context inputs tend to be more robust (although not consistantly) to these issues.", "questions": "Could you elaborate on the paper's novelty?\nIs the dataset utilized in the study publicly available?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The analysis is solid in its technical execution and experimental design\n2. Generally, a rich technical/experimental paper", "weaknesses": "1.\tThe analysis is solid in its technical execution and experimental design; however, it does not introduce any new methods, models, or techniques, which limits its novelty.\n2.\tThe importance of long context is questionable given the low frequency of extremely long contexts in EHRs. It seems somewhat expected that providing more information for each sample would improve results.\n3.\tWhile the paper states that disease progression increases token complexity over time. I am not very convinced how this property can be problematic with shorter input context compared to longer input context models. \n4.\tThe paper suggests that model performance improves with longer contexts; however, this is not consistently reflected in the figures. If this is indeed the case, it stands to reason that having more information about a patient (i.e., longer context) should facilitate easier predictions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)