Time-to-Event Pretraining for 3D Medical Imaging
Authors: Zepeng Frazier Huo
Jason Alan Fries
Alejandro Lozano
Jeya Maria Jose Valanarasu
Ethan Steinberg
Louis Blankemeier
Akshay S Chaudhari
Curtis Langlotz
Nigam Shah
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal learning
medical imaging
Electronic Health Records
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: With the rise of medical foundation models and the growing availability of imaging data, scalable pretraining techniques offer a promising way to identify imaging biomarkers predictive of future disease risk. While current self-supervised methods for 3D medical imaging models capture local structural features like organ morphology, they fail to link pixel biomarkers with long-term health outcomes due to a missing context problem. Current approaches lack the temporal context necessary to identify biomarkers correlated with disease progression, as they rely on supervision derived only from images and concurrent text descriptions. To address this, we introduce time-to-event pretraining, a pretraining framework for 3D medical imaging models that leverages large-scale temporal supervision from paired, longitudinal electronic health records (EHRs). Using a dataset of 18,945 CT scans (4.2 million 2D images) and time-to-event distributions across thousands of EHR-derived tasks, our method improves outcome prediction, achieving an average AUROC increase of 23.7% and a 29.4% gain in Harrell’s C-index across 8 benchmark tasks. Importantly, these gains are achieved without sacrificing diagnostic classification performance. This study lays the foundation for integrating longitudinal EHR and 3D imaging data to advance clinical risk prediction.

Published as a conference paper at ICLR 2025
TIME-TO-EVENT PRETRAINING FOR 3D MEDICAL IMAGING
Zepeng Jason Alan Fries'; :lejandro Lozano
Jeya Maria Jose Valanarasu Ethan Steinberg 16 Louis Blankemeier',
Akshay S. Chaudharf Curtis Langlotz4.58.0 Nigam H. Shah 4.77999
Center for Biomedical Informatics Research, Stanford University
Department of Biomedical Data Science, Stanford University
Department of Computer Science, Stanford University "Stanford Medical Center, Stanford Health Care
$ Stanfoor Center for Artificial Intelligence in Medicine and Imaging Prealize Health
Technology and Digital Solutions, Stanford Health Care DDepartment of Medicine, Stanford School of Medicine
p Centere Artificial Intelligence Institute, Stanford University 10 Department of Radiology, Stanford University
Clinical Excellence Research Center, Stanford School of Medicine tzphuo, jfries, lozanoe } @@stanford.eel
ABSTRACT
With the rise of medical foundation models and the growing availability of imag- ing data, scalable pretraining techniques offer promising way to identify imaging biomarkers predictive of future disease risk. While current self-supervised meth- ods for 3D medical imaging models capture local structural features like organ morphology, they fail to link pixel biomarkers with long-term health outcomes due to a missing context problem. Current approaches lack the temporal context necessary identify biomarkers correlated with disease progression, as they rely on supervision derived only from images and concurrent text descriptions. To address this, we introduce fime-to-event pretraining, pretraining framework for 3D medical imaging models that leverages large-scale temporal supervision from paired, longitudinal electronic health records (EHRs). Using a dataset of 18,945 CT scans (4.2 million 2D images) and time-to-even to-event distributions across thousands of EER--drrive tasks, our method improves outcome prediction, achieving an av- erage AUROC increase of 23.7% and 29.4% gain in Harrell's C-index across 8 benchmark tasks. Importantly, these gains are achieved without sacrificing diag- nostic classification performance. This study lays the foundation for integrating longitudinal EHR and 3D imaging data to advance clinical risk prediction.
INTRODUCTION
Foundation models for medical imaging have the potential transform healthcare by assisting doc- tors in complex clinical decision making (Saab et al., 2024; Sox et al., 2024) and identifying novel pixel biomarkers predictive of future disease risk (Pai et al., 2024; Sriram et al., 2021). Such models rely on self-supervised learning (SSL) for obtaining supervision the scale required to train on growing collections of 3D data (e.g., CT scans, MRIs). SSL captures local structural features by leveraging pretraining signal directly from images cross moos- pairs (e.g.. images and their text descriptions) (Zhang et al., 2022). While excelling at segmentation tasks and diagnostic classifi- cation of pathologies (Pinto-Coelho, 2023), SSL fails to leam prognostic biomarkers because the current pretraining regimens suffer from a missing context problem (see Figure 1). This issue arises when supervision sources are restricted to narrow time windows around the image, thus excluding
Authors contrihuted equally. detailed contribution breakdown Appendix C.

---

Published as a conference paper at ICLR 2025
long-term temporal patterns that are correlated with disease progression, which limits model's ability to identify prognostic biomarkers.
These long-term temporal patterns, which we refer to longitudinal context, are readily available in a patient's electronic health record (EHR) which is routinely used by clinicians to guide the interpretation of images and inform treatment planning (Leslie et al., 2000; Holste et al., 2024). Longitudinal EHRs contain temporal information about the progression of disease, as well as years of patients' health outcomes. However, the full breadth of these outcome data, in terms of temporal structure and task diveeritty, is rarely used as a source of supervision when training image foundation models. Current approaches for training 3D imaging models typically restrict labels to diagnosis codes sourced from the same or nearby temporal context as the image and its textual description (Blankemeier et al., 2024). Because EHR data readily available, it offers an untapped resource for large-scale pretraining of medical image models in a manner that uses long-term temporal context. More generally, image foundation models must reflect the settings in which they will be used, which is to assist in prognosis in the clinic (Negro-Calduch et al., 2021; Yala & Hughes, 2023).
However, performing effective risk estimation using imaging models involves navigating several challenges. Developing such models requires capturing correlations between pixels and outcomes spanning years, which is difficult with current SSL methods. Direct approaches to identifying pixel- level biomarkers, such sequential image capturing (Lu et al., 2021; Bera et al., 2022). difficult to scale for collecting large, high-quality datasets. Moreover, conducting risk estimation requires addressing right censoring, where the outcome of interest remains unobserved by the study's 's end. Naively excluding censored patients introduces bias and reduces available training data.
In medicine, time-to-event (TTE) modeling (also known as survival modeling) is commonly used to estimate future risk of an outcome at specific time point conditioned on feature representations. Although TTE models offer many theoretical advantages, including the ability estimate instanta- neous risk at any given time point (Collett, 2023) and naturally handling right censoring (Kleinbaum & Klein, 1996), their use image pretraining remains underexplored. Prior deep learning studies exploring TTE modeling in medical imaging have been restricted to small-scale, single-task appli- cations, typieally using 2D, end-to-end models (Zhu et al., 2016; Shu al., 2021; Lu et al., 2021). 3D medical imaging data remains underutilized resource, offering a wealth of biomarkers that could enhance clinical decision tools, particularly for the opportunistic detection of underdiagnosed conditions (Aali et al., 2024). However, large-scall scale TTE pretraining for 3D imaging has not yet been investigated, likely because multimodal medical datasets linking 3D images with longitudinal EHR data have only recently become available (Huang et al., 2024).
In this work, we propose time-to-event pretraining for medical imaging models as a way to address the missing context problem. Our central claim is that temporal supervision, defined by TTE distri- butions sourced from longitudinal EHRs, provides a readily available, scalable source of contextual information for pretraining that better captures prognostic pixel biomarkers. Moreover, by naturally handling right-censorship. TTE-based methods improve data efficiency and mitigate censorship bias. Our contributions are as follows:
We present the first large-scale evaluation of time-to-event pretraining for 3D medical imaging encoders. We use public dataset of 18,945 chest CT scans (equivalent to 4.2 million 2D images) linked to longitudinal EHR data containing 225M clinical events with a median follow-up time of years.
Our approach converts longitudinal EHR data into source of time-to-event supervision, thus predicting not only if a clinical event will occur but also when. This richer pretraining signal goes beyond diagnostic classification explored in prior work and enables generating many pretrain- ing tasks (8,19) 192 in this work) that capture the temporal event structure available in longitudinal EHR data. This choice also increase per-image label density by average of 3 times over prior approaches.
Our approach substantially improves performance in predicting future medical outcomes, achiev- ing on average a 23.7% increase in AUROC and a 29.4% improvement in Harrell's C-index over baseline models for 8 benchmark tasks without negatively impacting diagnostic classification performance in 8 external tasks. Our approach also improves model calibration, measured by the Integrated Brier Score, by average of 54%.
2

---

Published as a conference paper at ICLR 2025
All experiments conducted using public medical datasets ensure full reproducibility. We also make all of our experiment code and pretrained model checkpoints available for download 2 to contribute to the community for continued pretraining of imaging foundation models with prognosis as added henefit.
Longitudinal Electronic Health Record (EHR)
Visits
Diagnoses
Procedures
Notes
Medications
Medical Devices
Measurements
Social Determinants:
2020-07
2020-10
2021-01
2021-04
2021-07
2021-10
2022-01
202204
2022-07
2022-10
2023-01
Sources of Pretraining Supervision
Time-to-Event Pretraining Supervision (This Work)
Diagroses Destcas
Medications
I0630/E26.0
Maasirements
Image
Image
Text
Image + Text Diagnosis Codas
Imago Code Event Times (Diverse Types)
Figure I: The missing context problem in medical imaging. Existing supervision sources (red boxes) are localized the image itself (i.e., pixel features and descriptions of those features via text) or immediate clinical context via diagnosis codes. Doing so misses future information on disease pro- gression (black boxes), which reduces the ability to learn correlations necessary for identifying prognostic pixel biomarkers. Time-to-event pretraining provides principled framework for incor- porating the vast amount temporal supervision available in EHR data to estimate future risk in the presence of right censorship well as leverage large, diverse number of clinical tasks, beyond just diagnoses, for pre-training.
RELATED WORK
Time-to-Event Modeling with Medical Images Time-to-event (TTE) modeling, also known as survival analysis, predicts the distribution of time until a specific event occurs, such as death. TTE models primarily include accelerated failure time models, which assume various probability distri- butions, e.g., exponential (Saikia & Barman, 2017), Weibull (Breheny, 2015), and Cox proportional hazard (Cox-PH) models (Cox. 1972) which a semi camiiiaaametricc approach with constant hazard ratio assumption. Non-parametric models like random survival forests (Ishwaran et al., 2008) cap- ture non-linear interactions. the deep learning era, methods such as DeepSurv (Katzman et al., 2018 8) and MOTOR (Steinberg et al., 2024) provide higher level feature learning, making TTE mod- cling easier to extend to complex inputs such as medical images. Prior TTE methods for imaging assume 2D and 2.5D model architectures and have focused on end-to-end training for small-scale, single-task models. DeepConvSurr (Zhu et al., 2016) was the first work to replace the log-partial hazard (the exponential component of Cox model) with CNN, enabling survival prediction di- rectly from 2D images. Similarly, Shu et al. (2021) and Lu et al. (2021) modeled the log-partial hazard with a CNN-RNN encode sequential images.
Pretraining for 3D Medical Image Models Although early work in medical imaging relied on supervised pretraining using general-domain datasets such as ImageNet (Xie & Richmond, 2018; Ke et al., 2021), seff-supervised learning (SSL) is now the predominate approach to scaling pretraining medical imaging models. Popular approaches include reconstruction or de-noising objectives via masked autoencoders (MAE) (He et al., 2022), contrastive losses defined over paired samples (Chen et al., 2020; Sowrirajan et al., 2021) or leveraging multimodal pairs, such as medical images and their aligned text descriptions (Radford al., 2021; Zhang et al., 2022) unpaired medical images and text (Wang et al., 2022). SSL methods face added challenges when extended to 3D imaging, where instances (e.g., CT scans) contain over 100 times more pixel data than 2D counterparts like radio- graphs. Tang et al. (2022) combined unimodal contrastive learning, masked volume in- painting, and
hhttps://gite thub. com/ som- shahl hhttps://hugging wtt.:/aaa..///aaoaraaaaaaao
3

---

Published as a conference paper at ICLR 2025
Timel
Soat lindes
Cersenol
@ Evert
10D10/RMM
(P7771900
Paterts
RoNpmm/1 2000
Medical Cuals
0
15-1120117
04-0-2003
Entropy-Rante Vertex Cover for Task Selection
(Mentte)
Patient Electonic Healh Record (EHR)
Create Time to Event Pretraining Tasks (922,192)
TIME-TO-EVENT
Warfarts
Vision Encodor
Time-to- -oo--ent-ttt Head (PEANN)
P(T ()M)))
lang
Px
P:
Acste farkure
24
So
TTE Pretraining Tasks
Precewis Survival Curve
Linear
Categorical Claseification
AUROC
Vision Encoder
finetuning
Time-to-Event (CoxPH)
Time-to-Event Estimation
Integrated lhiee Soor Time Dependent C 913tGboos
3D Image Inputs
Pretrained Backbone
Task Heads (Probes)
Task Categories
Evaluation Metrics
Figure 2: Overview of the proposed time-to-event pretraining pipeline. Patients' longitudinal EHR timelines transformed into large-scale, time-to-event (TTH) pretraining tasks. These tasks, which reflect informative temporal patterns for medical outcome prediction, are then used for continued pretraining (full fine-tuning) of a 3D vision encoder. The resulting encoder is then frozen and adapted to downstream tasks via different task heads for classification or TTE estimation.
image rotation prediction to learn 3D structural information. Chen et al. (2023) explored 3D masked image modeling, demonstrating faster convergence compared simple contrastive methods (Sim- CLR and MAE). Valanarasu et al. (2023) used reconstruction loss to restore 3D CT volumes from tokens corrupted by noise, downsampling, and local masking. Finally, Blankemeier al. (2024) introduced Merlin, dual-objective framework that leverages EHR data by first using contrastive learning to align radiology reports with CT volumes, followed by training on disease phenotype classification using labels derived from diagnostic codes recorded during contemporaneous hospital visits.
Current SSL approaches excel at capturing structural features in medical images, such organ morphology for image segmentation. However, these learned features are static and largely fail to identify dynamic patterns and biomarkers that predictive of future health risks. Our contributions help bridge this gap in several key ways by leveraging future medical events to guide pretraining, resulting in more powerful image encoder for outcome prediction tasks. First, employ TTE objective to capture future patient health dynamics while accounting for right censoring, leverag- ing longitudinal EHR data to greatly expand the scale, diversity, and temporal scope of pretraining supervision. Second, we evaluate pretraining using native 3D imaging architectures which are un- derexplored in prior TTE work. Finally, we evaluate the impact of TTE modeling in imaging at larger scale than prior work, utilizing 18,945 CT scans equivalent to 4.2 million 2D images-ant 8, 8,192 unique TTE tasks derived from longitudinal EHRs.
3 PRELIMINARIES
Time-to-Event Modeling. The objective of time-to-event modeling is estimate the distribution of times T until an event of interest occurs. Observable data denoted as O {(T), X?) i= 1,.., n.. where X; are the features of observation and A, = I(T; < C.) is an event indicator function whose value is 1 when the actual event time is observed. One complexity is that medical data is often reght-censored, where the survival time T not observed due to loss of followup. With right-cen,oring, we do not observe and instead observe time T = min(T, C), where the C the censoring time. When A; 0, we do not know the true survival time, but we know that it
4

---

Published as a conference paper at ICLR 2025
is greater than Tj. The mixture of known survival times and censored survival times necessitates methods that can estimate T in an unbiased manner despite the censorship.
Different time-to-event models use different definitions to model the instantaneous hazard A(t), where t represents the continuous time at which the hazard rate is evaluated. We use a piecewise exponential function (Kitchin et al., 1983) that splits the timeline into P distinct intervals, pieces, each with a constant hazard rate A()). Each piece p(t) has Aipte) Cip- where C' depends on the patient's CT image and the specific interval p.
Piecewise Exponential Neural Network. To make use of deep learning models, we use a neural network to define Cip- Using a piecewise exponential neural network (PEANN) (Fornili al., 2014) greatly simplifies large-scale pretraining compared to Cox PH-based methods, which require creat- ing batches where samples are paired with at least one uncensored patient (Harrell Jr et al., 1996). We first use neural network to derive CT image representation Mip for image i's information at time piece p. We then apply a linear layer followed by an exponential function to define hazard Aip = Cip exp(A* Mip b) where A and b are learned parameters. With the hazard calculated, we can then input it into the survival function:
P
S1(1)= = II exp [- (min(t, Ex)) - Spp][( 2 Sp)
where each time piece p has starting point Sp and end point Ep and I(-) is an indicator function. The standard survival likelihood loss function for image is then:
L. -
where A; is event indicator whose value is when the actual future event is observed for image i and ft is the probability density function f. = St
as:
A full derivation of the loss function, including calculating the derivative and plugging in the survival function, can be found in Appendix M.
4 TIME-TO-EVENT PRETRAINING
In this work, we are interested in training a 3D image encoder to learn representations optimized for estimating the distribution of event times for future clinical outcomes. Using years of follow- up EHR data after image capture, we generate time-to-event pretraining tasks that provide large- scale training signal for estimating this distribution. We outline our approach for time-to-event pretraining in Figure 2. Given the computational costs of pretraining 3D image architectures from scratch, we evaluate the benefits of TTE supervision via continued pretraining of an existing neural network fe. where 0 represents the parameters of the pretrained backbone.
Creating TTE Pretraining Tasks EHR data captures a vast amount of structured information on patient demographics, diagnoses, procedures, medications, medical devices, social determinants of health, and other aspects of medical care. These data are encoded as timestamped, standardized identifiers called medical codes that map to ontologies (c.g.. ICD10, RxNorm, CPT). These ontolo- gies collectively represent a knowledge graph in the form of a directed acyclic graph (DAG). By treating each code as separate task and organizing EHR data chronologically, we generate ex- tensive training signals to model longitudinal health trajectories. This approach enables tasks such as predicting when a patient might develop lung cancer or be prescribed warfarin (a blood thinner used to prevent and treat blood clots), represented as time-to-event distributions conditioned on im- age features. However, naively treating each medical code as a task leads to millions of candidate pretraining tasks (4.3 million in dataset). Many of these tasks are low-frequency otherwise dundant given an ontology structure. To select pretraining tasks, we follow Steinberg et al. (2024)'s conditional Shannon entropy selection measure, which treats ontology- tex cover problem (West et al., 2001). We select a set of medical codes that maximizes conditional entropy given task budget, ontology DAG, and frequency distribution of observed medical codes.
task selection as ver-
We then define our TTE pretraining procedure by predicting the time until the first occurrence (if there are multiple recurring ones) of medical code as defined above, as shown in Appendix B.
5

---

Published as a conference paper at ICLR 2025
The survival function in each time piece is modeled from the time piece's starting time, where the first time piece's 's starting time is the index time (here the time of the CT scan exam). When there is no event in time piece the loss will be O. Each TTE task (n=8,192) is modeled independently (Appendix O). We also apply censorship at patient death, which is the only competing risk. Given TTE labels for input image X; use the loss described in Eq. 2 for full fine-tuning of fo-
Task Adaptation. Once TTE pretraining is complete, we use the frozen encoder Sa to generate feature embeddings for each input image X,, such that Z; = fo(;)) These embeddings are then passed to a task-specific head hg. which can either be classification head or a time-to-event head, depending on the task. We found that CoxPH task head (DeepSurv), which directly optimizes for Harrell's C-index, performed better in practice than fine-tuning a PEANN task head, thus we used CoxPH for all TTE task adaption. The classification head outputs prediction probabilities pi Z,) for discriminative tasks, while the survival head produces a time-dependent hazard score (t) Z,) for time-to-event (TTE) tasks. The model's outputs are evaluated using lask-appropriate metrics, e.g., AUROC for classification or Harrell's C-index and dime-dependent C-statistics for TTE tasks.
5 EXPERIMENTS
Hypotheses. Our experiments measure the impact of TTE pretraining on image encoder's 's abil- ity to generate representations useful for medical prognosis. We explore the following hypotheses:
1. TTE supervision improves data efficiency by utilizing temporal information from future EHRs and censored patients, increasing available task labels per-pretraining instance.
2. Existing supervised and SSL pretraining methods struggle to learn strong pixel biomarkers for disease risk due to missing temporal connections with pathologies that will be detected in the future. TTE supervision, in contrast, provides simple approach leveraging complex temporal information found in longitudinal EHRs for purposes of learning prognostic pixel biomarkers.
3. TTE supervision, when conducted post hoc, continued pretraining, does not negatively im- pact performance on standard categorical classification tasks as used for diagnostic image la- beling.
Setup. All image and EHR preprocessing details outlined in Appendix A. For our PEANN, use 8 piecewise time bins and 8,192 pretraining tasks (see Appendices O.P) per the pest-performing hyperpa- rameters from Steinberg et al. (2024). Each time bin created by uniformly dividing the range between the cohort's earliest and latest timestamps. We uti- lized two compute nodes from an on-premise cluster, each with 24 Intel Xeon 2.7GHz CPU cores, 200 GB RAM and 4 Nvidia 1100 GPUs or 4 H100 GPUs.
INSPECT
RSPECT
# Patients
19,402
7,279
# Train
18,945
5,823
Valid 1,089
364
# Test 3,214
1,092
Imaging
EHR
X
TTE Tasks
X
Diag. Tasks
/
Scan Type
Chest CT
Chest CT
Datasets & Evaluation Tasks. We use two 3D medical imaging datasets (see Table 1). INSPECT (Huang et al., 2024) is multimodal dataset of paired CT scans and radiology notes where each patient is linked to their longitudinal EHR This provides an average of 5 years follow-up data Post-CT scan and, in aggregate, contains 225 million medical events. RSPECT (Colak et al., 2021) is an image-only dataset of CT scans annotated by radiologists for imaging biomarkers related to pul- monary embolism and cardiac function. We evaluate 3 task categories in this work:
Table 1: Dataset summary statistics.
Estimate the distribution of event times for a specific outcome (e.g, mortality). We predict the first occurrence of an event, as the first occurrence (or only occurrence in cases such as mortality) is of higher clinical utility than subsequent events.
Prognostic TTE:
http://ttaaarraaiiii.a.eeeeeeeee htt.tteeeeeeeetteeeeeeeeeeeeeeee datasstt ?term- NSPECT 6gapt

---

Published as a conference paper at ICLR 2025
Prognostic Classification: Binarized classification formulation of TTE tasks using bucketed time bins. We use 1, 6, and 12 month bins. Unlike in TTE, censored patients are excluded in this category.
Diagnostic Classification: Standard classification using diagnostic image label categories.
INSPECT defines 3 prognostic binary tasks: hospital mortality, hospital readmission, and pulmonary hypertension (PH); and I binary diagnostic classification task (pulmonary embolism). Using the provided EHR data define 5 additional prognostic TTE tasks for lung pathologies: ATX (Atelec- tasis), CMG (Cardiomegalyy, CONS (Consolidation) EDM (Edema), and PEFF (Pleural Effusion). These tasks were selected based on their use in common chest medical imaging datasets (Irvin al., 2019). Appendix details how labels were assigned. RSPECT provides whole-volume labels for 9 diagnostic tasks, but longitudinal outcome data. RSPECT does not provide a public test set, so we impose a 80/5/15 split on train (n=7,279) for our experiments.
Architectures. We evaluate three model architectures: SwinUNETR, DenseNet, and ResNet. SwinUNETR (Tang et al., 2022) was originally designed for medical image segmentation tasks, combining elements from the Swin Transformer and the UNETR (U-Net Transformer) architec- tures. Weights are learned using reconstruction loss on 10,050 3D brain/chest CT and MRI data (Valanarasu et al., 2023), a much larger pretraining dataset than used by other public 3D models, e.g., Wasserthal et al. (2023). Following Merlin, adapt DenseNet-122 (Huang al., 2016) and ResNet-152 (He et al., 2015) by inflating their 2D pretrained ImageNet weights (specifically the filters and pooling kernels) DenseNet parameters were initialized using 2D weight inflation, but the architecture is fully 3D, using 3D convolutions in each Res-block or Dense-block, different from 2.5D methods (Hung al., 2024). This process enables us to input 3D CT images into the models for training. We also evaluated Merlin's pretrained ResNet- 152 backbone, but found performed similar to our base ResNet- (see Appendix E), thus we use the base weight- weight-inflated model for consistency across experiments.
described in Carreira & Zisserman (2017). Note that ResNet and
Model Baselines.
We evaluate the following continued pretraining approaches on all architectures:
base:
Baseline performance of the 3D pretrained SwinUNETR, DenseNet- 121, and ResNet-152 models without continued pretraining on INSPECT. See Appendix H for summary of the source pretraining datasets.
Continued pretraining of the base model via multitask, supervised learning using the INSPECT evaluation task labels. This controls for exposure to our training dataset in a consistent manner across architectures.
base /MTL:
base/visits Continued pretraining using the same 8,192 tasks used for TTE supervision, but restricting label ussignment to the same visit as the CT scan. This ablates the TTE component to learn temporal information and aligns more closely with the EHR-based supervision used by Merlin.
base TTEe TTE: Continued pretraining using TTE labels for 8,192 tasks occurring after the CT scan index timestamp.
All models except base use the same INSPECT training set exam- ples for continued pretraining. Pretraining task labels are assigned per-CT scan and vary density based on pretraining approach (Fig- ure 3). Note that TTE supervision enables leveraging patient's 's en- tire future EHR, providing times more training labels on average per CT-scan over per-visit labels. TTE also captures temporal struc- ture and time-varying disease risk, providing supervision signal that is absent in predominant SSL methods for imaging.
1.0
0.8
1368
750
0.4
8
82
75th
TTE
50th
Visit
25th
0.0
For evaluating our pretrained encoder, we use the frozen encoder and lightweight task head strategy outlined in Section 4. For clas- sification tasks, we use logistic regression as the classification head (linear probe) and for TTE tasks, we employ DeepSurv (Katzman et al., 2018) as the survival head. All model search hyperparameters for pretraining and adaptation in Appendix F.
1000
2000
3000
Labels Per:Image
Figure 3: Label density CDF by pretraining approach.
7

---

Published as a conference paper at ICLR 2025
Metrics. We evaluate discrimination performance using AUROC
for the-thhrsholded binary classification tasks and Harrell's C-index (Harrell et al., 1982) for TIE tasks. Harrell's C-index is type of C statistic that summarizes the ability of predictive model to rank patients. Harrell's C-index requires the predictive model to output a single risk per patient, as opposed to risk over time, thus suitable for our DeepSurv head evaluation. We the standard formulation for Harrell's C-index, where it first finds all possible pairs P' and one patient is known have the event before another, then splits that set into Poconeer (the higher predicted risk patient has the event first), Pued (both patients are tied for the predicted risk), and P incorreet (lower predicted risk patient actually has the event first): CH = evaluation metrics for time-dependent C-statistics and the integrated Brier score.
Pamer-1-055. Page Peomat Phart Paammr Appendix D contains additional TTE
All performance results and 95% confidence intervals reported using test set bootstrap of n 1000 replicates. Statistical significance was computed using a two-tailed Z-test (p-value at 0.05 for rejecting the null hypothesis that the difference between two sample means is zero). A complete set of statistical tests are in Appendix K.
Additional Experiments. See the appendix for further experimental ablations including: sub- group performance (Appendix R), task head capacity (Appendix L), and full fine-tuning vs. frozen backbone adaptation (Appendix J).
6 RESULTS
Evaluating Prognostic Performance. Table 2 reports performance for the prognostic binary clas- sification formulations of outcome prediction for the original INSPECT tasks. This provides simplified view of high and low-risk patients across time. We find that TTE pretrained models outperform all baselines across all architectures in our experiments. Here TTE pretraining pro- vides an average of 22.6% performance over the base pretrained model and 15.4% average in- crease over base/visit. The base/MII bascline also outperforms the base model, but underper- forms TTE, highlighting the benefits of increasing pretraining tasks. Comparing base/TTE versus base base/visit /vissit (8,192) but base/TTr TTE substantially improves the density of the label per image during pretraining (Figure 3). Table reports performance of all original INSPECT tasks and our five new outcomes using Harrell's C-index. Here TTE pretraining largely outperforms all of our baselines across all architectures.
more informative, as both approaches
the same number of pretraining tasks
Mortality
Readmission
PH
Model
IM
6M
12M
IM
6M
12M
12M
SwinUNETR bace
0.693
0.684
0.685
0.507
0.538
0.569
0.597
SwinUNETR base/NTL
0.676
0.700
0.697
0.502
0.543
0.551
0.606
SwinUNETR beae/visit
0.693
0.716
0.670
0.560
0.554
0.528
0.560
SwinUNETR base/TTe TTE
0.827
0.808
0.788
0.582
0.612
0.607
0.672
DenseNet-121 121 base
0.616
0.568
0.575
0.512
0.532
0.524
0.506
DenseNet- 121
base/NTL.
0.665
0.596
0.602
0.533
0.557
0.547
0.577
DenseNet- 121 base/ base/vieit
0.649
0.698
0.692
0.504
0.538
0.567
0.599
DenseNet-122 121
base/TTE
0.770
0.730
0.725
0.629
0.643
0.637
0.689
ResNet- 152 base
0.583
0.557
0.554
0.536
0.537
0.509
0.537
ResNet- 152 base/NTL
0.726
0.715
0.647
0.563
0.564
0.566
0.570
ResNet-152 base/vissi
0.691
0.636
0.643
0.562
0.564
0.566
0.567
ResNet-152 basa/TTE
0.804
0.798
0.792
0.602
0.649
0.657
0.718
Table 2: Prognostic binary classification performance for INSPECT tasks on Logistic Regression
(1, 6, 12 month time horizon bins) reported the mean AUROC of a test set bootstrap (n=1000). Bold indicates the best performer. Underlined indicates no statistically significant difference versus the /TTE models.
8

---

Published as a conference paper at ICLR 2025
Harrell's C-Index 1
Model
Mort.
Readm.
PH
ATX
CMG
CONS
EDM
PEFF
SwinUNETR base
0.717
0.653
0.696
0.558
0.662
0.549
0.697
0.641
SwinUNETR
0.672
0.671
0.665
0.681
0.677
0.668
0.715
0.668
SwinUNETR base/viste
0.671
0.716
0.697
0.719
0.717
0.718
0.717
0.714
SwinUNETR base/TTB
0.738
0.723
0.724
0.739
0.739
0.738
0.738
0.738
DenseNet- 121
0,505
0,505
0,541
0,505
0,505
0,506
0.505
0,536
DenseNet- 121
base/NIL
0.589
0.590
0.591
0.593
0.589
0.586
0.587
0,590
DenseNet- 121 base/visii
0.675
0.675
0.699
0.675
0.662
0.676
0.676
0.675
DenseNet-121 base/TTE
0.732
0.723
0.726
0.720
0.711
0.712
0.725
0.723
ResNet-152 base
0.505
0.560
0,505
0.577
0.505
0,559
0.505
0,536
ResNet-155 base/MTL
0.701
0.686
0.656
0.663
0.562
0.656
0.660
0.572
ResNet- 152 base/visit
0.656
0.702
0.643
0.703
0.705
0.703
0.700
0.716
ResNet-1152 base/TTE
0.732
0.739
0.735
0.728
0.727
0.727
0.737
0.737
Table 3: Prognostic TTE performance for INSPECT, measured by Harrell's C-Index. Bold indi-
cates the best performance. Underlined indicates no statistically significant difference versus the "base/TTE models.
Evaluating Diagnostic Performance. We are also interested in assessing how TTE continued pretraining may impact standard image classification tasks, corresponding diagnostic labeling of medical imaging for current disease biomarkers. Table 4 outlines 8 image biomarker tasks. Here, performance of all 3D model architectures is poor, especially base models. For almost all diagnos- tic tasks, TTE pretraining performs the same (i.e., statistically indistinguishable, detailed numbers shown in Table 17) as other tested pretraining approaches. This aligns with the intuition that visit-level labels reflect current clinical events, should encode the same level of diagnostic informa- tion as TTE pretraining. This also aligns with the small performance gains reported in prior work when tasks derived from EHR codes to supervise image models (Blankemeier et al., 2024). Note that while these tasks reflect observable pixel biomarkers present images, there also overlap with pixel biomarkers indicative of future risk. For example, RV/LV ratio 2 1, defined the ratio of the right ventricular (RV) diameter to the left ventricular (LV) diameter, is indicative of increased risk of mortality (Lu et al., 2012). Here TTE pretraining yields statistically significant performance improvements across all architectures and pretraining methods.
PE
RV/LV Ratio
Model
Left
Cent.
Right
Chronic
Acute
Indet.
<1
21
SwinUNETR base
0.573
0.571
0.573
0.507
0.581
0.721
0.525
0.578
SwinUNETR base/WTL
0.583
0.633
0.591
0.504
0.594
0.732
0.517
0.598
SwinUNETR base/visit
0.545
0.664
0.562
0.556
0.595
0.741
0.521
0.546
SwinUNETR base/TTE
0.634
0.651
0.633
0.525
0.716
0.781
0.643
0.636
DenseNet-122 121 base
0.596
0.615
0.596
0.565
0.676
0.724
0.581
0.607
DenseNet-
0.612
0.638
0.615
0.504
0.664
0.713
0.594
0.636
base//t
DenseNet-121 base/visit
0.595
0.633
0.605
0.571
0.677
0.748
0.573
0.615
DenseNet-121 base/
0.647
0.716
0.644
0.586
0.665
0.762
0.623
0.686
ResNet- 152 base
0.654
0.687
0.618
0.507
0.695
0.704
0.593
0.626
ResNet- base/HIL
0.658
0.693
0.621
0.506
0.685
0.703
0.587
0.642
ResNet- 152 base/vinit
0.646
0.665
0.625
0.548
0.743
0.718
0.566
0.665
ResNet- 152 base/ITE 0.657
0.687
0.642
0.504
0.588
0.722
0.597
0.708
Table Diagnostic binary classification performance for RSPECT, reported as the mean AUROC of a test set bootstrap 10=1000). Bold indicates the best performer. Underlined indicates no statistically significant difference versus the * "base/TTE models.

---

Published as a conference paper at ICLR 2025
7 DISCUSSION AND CONCLUSION
Medical images hold significant, untapped potential as sources of imaging biomarkers to predict fu- ture disease risk. However, current SSL. approaches for imaging largely fail to capture the temporal dynamics of long-term disease progression, inherently capturing only static, structural information. Building on this observation, our work explores a TTE pretraining technique that directly incorpo- rates future temporal information at scale, yielding several insights.
Current Pretraining Struggles Learn Prognostic Pixel Biomarkers. Existing supervised and SSL pretraining methods consistently showed lower AUROC across prognostic tasks in our ex- periments. When compared against base models, TTE pretraining showed an average increase in AUROC of 0.128 (95% CI: [0.075, 0. 0,15 Harrell's C-index 0.166 (95% CI [0.138, 0.490]) over three architectures on prognostic binary classification and prognostic TTE tasks respectively. This suggests that off-the- pretraining methods struggle to learn pixel biomarkers associated with fu- ture disease risk, likely due to missing temporal links to future pathologies. This underscores Huang et al. (2020)'s findings that predicting disease prognosis with high accuracy and certainty remains challenging task. Therefore, incorporating explicit temporal supervision into the pretraining process may be essential for improving prognostic tasks' performance.
TTE Pretraining Improves Prognostic Performance. When compared against base/Mi /ore and base/visit, TTE supervision improves performance on Harrell's 's C-index on prognostic TTE task tasks by 0.093 (95% CI [0.086, 0.554]) and, 0.038 (95% CI [0.017, 0.370J) respectively. Addi- tionally TTE supervision does not negatively impact diagnostic binary classification for significant difference between base and TTE across RSPECT tasks, shown in Table 17.
TTE Supervision Improves Training Data Efficiency. By leveraging temporal information from the future in EHRs and censored patients, TTE pretraining increases label density to boost AU- ROC for prognostic binary classification tasks by 0.093 (95% CI [0.053, 0.134) and 0.092 (95% CI [0.051, 0.135]) when compared against base/MTL and base base/visit /visit models respectively. TTE supervision increases available task labels per-training instance by 3x on average (Figure 3). This underscores the potential of TTE pretraining as a viable objective for scaling medical imaging AI, given that expert-level annotation is time-consuming, expensive, and difficult to collect (Dgani et al., 2018; Tajbakhsh et al., 2021; Aljabri et al., 2022). Increasing the data efficiency of a given training example also contributes to reducing compute costs.
Limitations. First, our study focuses exclusively on evaluating 3D vision encoders, which de- mand significant memory and computational resources- -80BB800BBB memory GPUs for SwinUNETR and 40GB memory GPUs for DenseNet/ResNet architectures. Second, while our study surpasses TTE in terms of scale, our pretraining dataset remains relatively
previous work in imaging- small compared to modern, general-purpose datasets (Schuhmann et al., 2022) and recently released medical datasets (Xie et al., 2024). Furthermore, we focus exclusively on a single modality, CT scans. Expanding both the scale and diversity of the pretraining data mixtures (c.g., other imaging modalities including both 2D and 3D. historical EHR and clinical text) could enhance performance or lead to a deeper understanding of the trade-offs between different architectures. Finally, since this work focuses on evaluating encoder quality, we only evaluated frozen encoders with smaller, lightweight, supervised task heads for adaptation. Alternative adaption methods under different sample assumptions, e.g., zero/few-shot learning, may reveal different performance trade-offs.
Conclusion. This work presents the first empirical study of using time-to-event pretraining for 3D medical vision models. By leveraging longitudinal EHRs and defining a time-to-event pretraining objective (comprising 8,192 tasks), we embed long term outcome information into the image model during pretraining. Doing so results in an average threefold increase in training labels compared to limiting just to patient's 's current EHR visit. We observe substantial improvements in prediction performance for future events, with increases of up to 31.6% in AUROC and a 40.5% improvement in Harrell's C-index, without negatively impacting standard binary classification tasks (e.g., image labeling for diagnostic tasks). Our results reveal clear need for pretraining datasets for 3D medical foundation models to include tasks that capture long-term temporal structure, demonstrated here through our use of time-to-event supervision. This study the utility of using longitudinal EHR records as a supervision source for cature-guided pretraining of 3D medical imaging models lays the groundwork for innovative ways combine EHR and imaging modalities for clinical risk prediction.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Research reported in this publication was supported by the National Heart, Lung, and Blood Institute of the National Institutes of Health Award RO1HL155410, as well as NIH grants R01HL167974,
R01HLI69345, P41 EB027060; ARPA-H contract AASSXX0000224001 and NIH contracts 75N92020C00008 and 75N92020C00021. Further support was provided by the Stanford Institute for
Artificial Intelligence (HAI) and the Stanford Center for Artificial Intelligence in Medicine and Imaging (AIMI) in the form of an HAI Seed Grant and AIMI-HAI Partnership Grant. We would also like to thank the Clinical Excellence Research Center (CHRC) at Stanford for their support.
REFERENCES
Asad Aali, Andrew Johnston, Louis Blankemeier, Dave Van Veen, Laura T Derry, David Svec, Jason Hom, Robert Boutin, and Akshay S Chaudhari. Detecting underdiagnosed medical conditions with deep learning- based opportunistic ct imaging. arXiv preprint arXiv:2409.1 1666, 2024.
Manar Aljabri, Manal AlAmir, Manal AIGhamdi, Mohamed Abdel-Mottaleb, and Fernando Collado-Mesa. Towards better understanding of annotation tools for medical imaging: survey. Multimedia tools and applications, (1(18)255877-551 2022.
Shekoofeh Azizi, Laura Culp, Jan Freyberg, Basil Mustafa, Sebastien Baur, Simon Kornblith, Ting Chen, Patricia Mac Malliamams,, S Sara Mahdavi, Ellery Wulczyn, et al. Robust and efficient medical imaging with self- seff-supervision ar preprint S.XXxx2205509723., 2022.
Kaustav Bera, Nathaniel Braman, Amit Gupta, Vamsidhar Velcheti, and Anant Madabhushi. Pre- dicting cancer outcomes with radiomics and artificial intelligence in radiology. Nature reviews Clinical oncology, 19(2):132-146) 2022.
Louis Blankemeier, Joseph Paul Cohen. Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck, Eduardo Reis, Cesar Truyts, et al. Merlin: vision language foundation model for 3d computed tomography. arXiv preprint 2024. 2024.
arXiv.2406.06512,
Patrick Breheny. Accelerated failure time models. University Lecture, 2015.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinctics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
6299-6308, 2017.
Trenton Chang, Michael Sjoding, and Jenna Wiens. Disparate censorship & undertesting: A source of label bias in clinical machine learning. In Machine Learning for Healthcare Conference, pp. 343-390. PMLR, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Zekai Chen, Devansh Agarwal, Kshitij Aggarwal, Wiem Safta, Mariann Micsinai Balan, and Kevin Brown. Masked image modeling advances 3d medical image analysis. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 1970-1980, 2023.
Errol Colak, Felipe C Kitamura, Stephen B Hobbs, Carol C Wu, Matthew P Lungren, Luciano M Prevedello, Jayashree Kalpathy-Crrne Cramerr Robyn L Ball, George Shih, Anouk Stein, et al. The rsna pulmonary embolism et dataset. Radiology: Artificial Intelligence, 3(2)e200254, 2021.
David Collett. Modelling survival data in medical research, Chapman and Hall/CRC, 2023.
David R Cox. Regression models and life-tables. Journal of the Royal Statistical Society: Series B (Mettodological), 34(2):187-202, 1972.
Yair Dgani, Hayit Greenspan, and Jacob Goldberger. Training neural network based on unreliable human anmotation of medical images. In 2018 IEEE 15th International symposium on biomedical imaging (ISBI 2018), , pp. 39-42. IEEE, 2018

---

Published as a conference paper at ICLR 2025
Marco Foriili, Federico Ambrogi, Patrizia Boracchi, and Elia Biganzoli. Piecewise exponential artificial neural networks (peann) for modeling hazard function with right censored data. In Computation Intelligence Methods for Bioinformatics and Biostatistics, pp. 125-136, 07 2014. ISBN 978-3-319-09044 doi: 10. 10.1007/978-3 0-119788--31--99422--
Erika Graf, Claudia Schmoor, Willi Sauerbrei, and Martin Schumacher. Assessment and comparison of prognostic classification schemes for survival data. Statistics medicine, 18(17-18):2529-
2545, 1999.
Frank E Harrell, Robert M Califf, David B Pryor, Kerry L Lee, and Robert A Rosati. Evaluating the yield of medical tests. Jama, 247(18):2543-2546 1982.
Frank E Harrell Jr, Kerry L Lee, and Daniel B Mark. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in medicine, 15(4)3311337, 1996.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. arxiv e-prints. arXiv preprint uIXiv:1512.03385 10, 2015.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Patrick J Heagerty and Yingye Zheng. Survival model predictive accuracy and curves. Biometrics, 61(1):92-105, 2005.
Gregory Holste. Mingquan Lin, Ruiwen Zhou, Fei Wang, Lei Liu, Qi Yan, Sarah H Van Tassel, Kylc Kovacs, Emily Chew, Zhiyong Lu, et al. Harnessing the power of longitudinal medical imaging for eye disease prognosis using transformer-based sequence modeling. NPJ Digital Medicine, (1):216, 2024.
Gao Huang, Zhuang Liu, and Kilian Q Weinberger. Densely connected convolutional networks. arXiv preprint arXiv: 1edii::66....0 608.06993 2016.
Shigao Huang, Jie Yang, Simon Fong, and Qi Zhao. Artificial intelligence in cancer diagnosis and prognosis: Opportunities and challenges. Cancer letters, 471:61-71, 2020.
Shih- Shih-Cheng Huang. Zepeng Huo, Ethan Steinberg, Chia-Chun Chiang, Curtis Langlotz, Matthew Lungren, Serena Yeung, Nigam Shah, and Jason Fries. Inspect: A multimodal dataset for pa- tient outcome prediction of pulmonary embolisms. Advances in Neural Information Processing Systems, 36, 2024.
Alex Ling Yu Hung, Haoxin Zheng, Kai Zhao, Xiaoxi Du, Kaifeng Pang, Qi Miao, Steven S Raman, Demetri Terzopoulos, and Kyunghyun Sung. Csam: 2.5 cross-slice attention module for anisotropic volumetric medical image segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5923-5932, 2024.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-lleus, Chris Chute, Hen- rik Marklund, Behzad Haghgoo. Robyn L. Ball, Katie S. Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lungren, and Andrew Y. Ng. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. CoRR, abs/1901 000031,,,, 2019. URL http:// httt:///rxiiv/ /ttp//avvvv///////////.e///.wev. org/ abs/ 11901.1.. 07031..
Hemant Ishwaran, Udaya B Kogalur, Eugene H Blackstone, and Michael S Lauer. Random survival forests. 2008.
Edward L Kaplan and Paul Meier. Nonparametric estimation from incomplete observations. Journal of the American statistical association, 53(282):457-481, 1958.
Jared L Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger. Deepsurv: personalized treatment recommender system using a cox proportional hazards deep neural network. BMC medical research methodology, 18:1-12, 2018.
12

---

Published as a conference paper at ICLR 2025
Alexander Ke, William Ellsworth, Oishi Banerjee, Andrew Y Ng. and Pranav Rajpurkar. Chextrans- fer: performance and parameter efficiency of imagenet models for chest x-ray interpretation. In Proceedings of the conference on health, inference, and learning, pp. 116-124, 124, 2021.
John Kitchin, Naftali A Langberg. and Frank Proschan. new-method for estimating life distribu- tions from incomplete data. Statistics & Risk Modeling, 1(3):241-256 1983.
David G Kleinbaum and Mitchel Klein. Survival analysis a self- lelfnleannngg text. Springer, 1996.
Adones Leslie, AJ Jones, and PR Goddard. The influence of clinical information the reporting of ct by radiologists. The British journal radiology. 73(874): 10520-1555 2000.
Lin Lu, Laurent Dercle, Binsheng Zhao, and Lawrence H Schwartz. Deep learning for the prediction of early on-treatment response in metastatic colorectal cancer from serial medical imaging. Nature communications, 1261))6654, 2021.
Michael T Lu, Shadpour Demehri, Tianxi Cai, Layla Parast, Andetta R Hunsaker, Samuel Z Gold- haber, and Frank Rybicki. Axial and reformatted four-chamber right ventricle-to-lefi ventricle diameter ratios on pulmonary et angiography as predictors of death after acute pulmonary em- bolism. American Journal of Roentgenology, 198(6)11353-336) 2012.
Elsa Negro-Calduch, Natasha Azzopardi-
Ramesh S Krishnamurthy, and David Novillo- Ortiz. Technological progress in electronic health record system optimization: Systematic review of systematic literature reviews. International journal of medical informatics, 152:00 2021.
Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464))447-4533 2019, 2019.
OHDSI. Omop common data model. https//wlalllllloo.cccooooooooos 2023. Accessed: 2023-06-07.
Suraj Pai. Dennis Bontempi. Ibrahim Hadzic, Vasco Prudente, Mateo Sokač, Tafadzwa L Chaunzwa, Simon Bernatz, Ahmed Hosny, Raymond H Mak, Nicolai Birkbak, et al. Foundation model for cancer imaging biomarkers. Nature machine intelligence, 6(3):354-36) 2024 2024.
Stephen R Pfohl, Heather Cole-Lewis, Rory Sayres, Darlene Neal, Mercy Asiedu, Awa Dieng, Ne- nad Tomasev, Qazi Mamunur Rashid, Shekoofeh Azizi, Negar Rostamzadeh, al. A toolbox for surfacing health equity harms and biases in large language models. Nature Medicine, pp. 1-11, 2024.
Luis Pinto- Finto-Coelho. How artificial intelligence is shaping medical imaging technology: A survey of innovations and applications. Bioengineering, 10(12):1435, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 9748-8763. PMLR, 2021.
Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. Capabilities gemini models medicine. arXiv preprint 140.:2400.18416 2024.
Rinku Saikia and Manash Pratim Barman. A review on accelerated failure time models. Stat Syst, 12(2):311333 2017.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25222 2022.
Bob Segert. Athena's 's OMOP vocabulary list. https: pettts://atheaee /athena. wwdsi. sist, 2023. Accessed: 2023-08-16.
13

---

Published as a conference paper at ICLR 2025
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based local- ization. In Proceedings of the IEEE international conference computer vision, pp. 618-626, 2017.
Michelle Shu, Richard Strong Bowen, Charles Herrmann, Gengmo Qi. Michele Santacaticrina, and Ramin Zabih. Deep survival analysis with longitudinal x-rays for covid-19. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9046-4050 2021.
Hari Sowrirajan, Jingbo Yang. Andrew Y Ng, and Pranav Rajpurkar. Moco pretraining improves representation and transferability of chest x-ray models. In Medical Imaging with Deep Learning, pp. 728-744. PMLR, 2021.
Harold C Sox, Michael C Higgins, Douglas K Owens, and Gillian Sanders Schmidler. Medical decision making. John Wiley & Sons, 2024.
Anuroop Sriram, Matthew Muckley, Koustuv Sinha, Farah Shamout, Joelle Pineau, Krzysztof J Geras, Lea Azour, Yindalon Aphinyanaphongs. Nafissa Yakubova, and William Moore. Covid- 19 prognosis via self-supervised representation learning and multi-image prediction. arXiv preprint arXiv:2101.0449 2021.
Ethan Steinberg, Jason Alan Fries, Yizhe Xu, and Nigam Shah. MOTOR: A time-to-event founda- tion model for structured medical records. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreviee net
Nima Tajbakhsh, Holger Roth, Demetri Terzopoulos, and Jianming Liang. Guest editorial annotation-efficient deep learning: the holy grail of medical imaging. IEEE transactions on medical imaging, 40(10))2526-2533, 2021.
Yucheng Tang, Dong Yang. Wenqi Li, Holger R Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. Self-supervised pre-training of swin transformers for 3d medical image analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 20730-20740, 2022.
Jeya Maria Jose Valanarasu, Yucheng Tang, Dong Yang, Ziyue Xu, Can Zhao, Wenqi Li, Vishal M Patel, Bennett Landman, Daguang Xu, Yufan He, et al. Disruptive autoencoders: Leveraging low-level features for 3d medical image pre-training. arXiv preprint arXiv:2307.16896, 2023.
Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medelip: Contrastive learning from unpaired medical images and text. In Proceedings of the 2022 Conference on Empirical Methods in Natural I Language Processing, pp.
2022.
Jakob Wasserthal, Haans-Chhissiin Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexander W Sauter, Tobias Heye, Daniel T Boll, Joshy Cyriac Shan Yang, al. Totalseg- mentator: robust segmentation of 104 anatomic structures in et images. Radiology: Artificial Intelligence, 5(5), 2023.
Douglas Brent West et al. Introduction to graph theory, volume 2. Prentice hall Upper Saddle River, 2001.
Yiting Xie and David Richmond. Pre-training on grayscale imagenet improves medical image clas- sification. In Proceedings of the European conference on computer vision (ECCV) workshops, pp. 0-0, 2018.
Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong- Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, et al. Medtrinity-25mn large-scale multimodal dataset with multigran- ular annotations for medicine. arXiv preprint arXiv:2408.0m 029000 2024.
Adam Yala and Kevin S Hughes. Rethinking risk modeling with machine learning. Annals of 2023.
Surgical Oncology,
14

---

Published as a conference paper at ICLR 2025
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Con- trastive learning medical visual representations from paired images and text. Machine Learning for Healthcare Conference, pp. 2-25. PMLR, 2022.
Xinliang Zhu, Jiawen Yao, and Junzhou Huang. Deep convolutional neural network for survival analysis with pathological images. In 2016 IEEE international conference on bioinformatics and biomedicine (BIBM), pp. 844-547, IEEE, 2016.
15

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Research involving de-identifed publicly available data does not require Institutional Review Board (IRB) approval. However, uphold standards and safeguard patient privacy, we follow the health- care machine learning reproducibility recommendations set by the Medical AI Research Foundations (Azizi al., 2022). Below, we outline specific ethical considerations and include references to more detailed discussions in the appendix.
In accordance with applicable privacy laws and institutional guidelines. all data utilized for model training and evaluation has been de-identified by the original dataset authors. Both datasets used in this study, INSPECT (Huang et al., 2024) and RSPECT (Colak et al., 2021), underwent thorough removal of protected health information prior to their public release and were approved by an Institutional Review Board.
Data Deidentification:
Security, Data Storage, and Compliance: All authors involved in data handling have completed institutional training on HIPAA and data privacy before engaging with the data. All training data was stored in HIPAA-compliant compute environment.
Algorithmic Bias Healthcare machine learning models can be susceptible to algorithmic bias, lead- ing unfavorable outcomes for underrepresented subgroups (Obermeyer et al., 2019). Bias mitiga- tion medical foundation models remains an ongoing research challenge (Pfohl et al., 2024) and not covered in this study. However, take two steps to mitigate risk. First, all of our contin- ued pretrained model releases includes Data Use Agreement (DUA) that explicitly prohibits direct conduct
medical care. Second, in line with the recommendations from Chang et al. (2022), analysis in Appendix R to assess performance across sensitive subgroups, ensuring our pretraining technique does not unfairly disadvantage any group compared existing methods. We evaluate per- formance using the AUROC (bootstrapped, n=1000) on 7 binary prognostic tasks, comparing TTE against base, showing that TTE pretraining does not reduce performance for sensitive groups and generally improves risk ranking across all groups.
REPRODUCIBILITY STATEMENT
The code artifact necessary for reproducing the experiments in this paper can be found in the supplemental materials as a zip file. The anonymous Github link pretraining-DAGC. Our hyperparameter search grids can be found in Appendix F. All base pretraining weights are publicly available as detailed in Table 10. To ensure reproducibility, all experiments researcher accessible, public medical datasets.
16

---

Published as a conference paper at ICLR 2025
APPENDIX
A DATA PREPROCESSING
CT Scans. Each CT scan is preprocessed by extracting pixel data and applying linear trans- formation to Hounsfield Units (HU) using the rescale slope and intercept values from the original DICOM records. To retain fine-grained details, we ensure axial slices have thickness between mm and mm. Finally, we pad and center crop the images to 224 224 pixels.
Electronic Health Records. INSPECT's EHR data is provided in the Observational Medical Out- comes Partnership (OMOP) Common Data Model (CDM) schema, standardized framework that harmonizes healthcare data from various sources to support large-scale analysis (OHDSI, 2023). We use the Athena OHDSI Vocabularies Repository (Segert, 2023) (OMOP vocabulary version: v20240830) as our knowledge graph for generating tasks.
B INSPECT NEW TTE TASK DEFINITIONS
We have selected a set commonly used pulmonary disease tasks (Irvin et al., 2019), that are coded in INSPECT dataset's EHR events. We use these common tasks benchmark the time-to- event performance of our proposed TTE formulation. Each patient is assigned time-to-event=True after CT's capture for pulmonary hypertension, atclectasis, cardiomegaly, consolidation, edema, and pleural effusion. These labels indicate either the time until the first occurrence of each condition or the time until the patient censored. The description can found in Figure 4 for diagnostic and prognostic labels.
Current Events
Future Events
CPT 99490
CPT 99490
ICD 126.99
CPT 99590
CPT 99590
Visit
Visit
Y2
Y5
Censoning
Figure 4: Overview of Label Definitions: Diagnostic tasks use labels derived from the same hospital visit as the CT scan. Prognostic tasks involve future medical events from patients' EHR timelines and are categorized into binary prognostic labels and time-to-event (TTE) prognostic labels. Note that for TTE tasks, only the time until the first occurrence labeled.
C CONTRIBUTION TABLE
We wish to ensure that we can accredit contributions to all the authors in clear and fair manner and choose to follow the example from here
Experiment Design
Coding Analysis
Writing
Visualization
Project Management
Resources
Funding
Zopeng Hupt
Jason Nan Frees
Rejendro
Ethan Stenberg
Inus Ramoermeerr
Acshay Cheutheri
Nigam
Figure Overview of author contributions. denotes equal contribution.
17

---

Published as a conference paper at ICLR 2025
D ADDITIONAL TIME-TO-EVENT MODEL METRICS
We calculate additional metrics for time-to-event task modeling: the integrated Brier score (IBS) Graf al. (1999), shown Table 5, and time-dependent C-statistics Heagerty & Zheng (2005), shown in Table 6. IBS is intended to assess the discrepancy between predicted survival probabili- ties and observed outcomes. At each time point, the Brier score calculates the squared difference between the predicted survival probability and the actual event censoring status. By integrating over the time range from the start to a specified maximum horizon, summarizes the model's pre- dictive performance over the entire period of interest. For time-dependent C-statistics, we adopt the incident and dynamic definitions for time-dependent sensitivity and specificity Heagerty & Zheng (2005) to derive time-dependent receiver operating curve AUC(t). Time-dependent concordance is calculated as Ced J. (UC()))(()dt J. "(t)dtt where w(t) = f(t) S(t) and f(t) is defined as the event rate
time and S(t) is the survival probability at time t. both of which are estimated using Kaplan-Meice estimator Kaplan & Meier (1958). AUC(!) is the integration of ROC curve under time bin, where sensitivity and specificity are following the definitions Heagerty & Zheng (2005). We should note that the time-dependent C-statistics is models, because it upweighs the long time horizon examples which are generally harder to predict for models like traditional CoxPH.
strict metric, especially for non-piecewise survival
Integrated Brier Score |
Model
Mort.
Readm.
PH
ATX
CMG
CONS
EDM
PEFF
SwinUNETR base
0.071
0.070
0.072
0.071
0.067
0.073
0.074
0.069
SwinUNETR base/MILL
0.081
0.078
0.079
0.081
0.078
0.083
0.081
0.080
SwinUNETR base base/visit
0.073
0.068
0.072
0.070
0.066
0.073
0.074
0.070
SwinUNETR base/TTE
0.069
0.067
0.066
0.069
0.067
0.073
0.079
0.069
DenseNet- 121 base
0.219
0.803
0.702
0.779
0.217
0.533
0.335
0.822
DenseNet- 121
base/WTL
0.071
0.067
0.068
0.060
0.065
0.072
0.083
0.068
DenseNet- 121 base/visit
0.063
0.068
0.071
0.071
0.060
0.075
0.075
0.072
DenseNet-122 121 base/'TEE
0.028
0.021
0.022
0.014
0.014
0.029
0.023
0.014
ResNet- 1.52 base
0.335
0.078
0.479
0.174
0.160
0.166
0.140
0.095
ResNet- 152 base//HTI
0.076
0.074
0.078
0.075
0.074
0.077
0.078
0.077
ResNet-155 base/vieit
0.075
0.077
0.074
0.070
0.067
0.077
0.074
0.075
ResNet- 152 base/ITE
0.068
0.069
0.066
0.065
0.064
0.070
0.070
0.066
Table 5: Prognostic TTE performance for INSPECT, measured by Integrated Brier Score. Bold indicates the best performer. Underlined indicates statistically significant difference versus the
"base/TTT (TE models.
18

---

Published as a conference paper at ICLR 2025
Time-dependent C-Statistics 1
Model
Mort.
Readm.
PH
ATX
CMG
CONS
EDM
PEFF
SwinUNETR base
0.649
0.620
0,636
0,602
0.619
0.555
0.637
0.619
SwinUNETR base/NTL
0.621
0.624
0.631
0.619
0.625
0.631
0.649
0.633
SwinUNETR case/visit
0.625
0.645
0.635
0.650
0.648
0.644
0.648
0.638
SwinUNETR base/TTE
0.672
0.657
0.645
0.662
0.660
0.663
0.667
0.667
DenseNet- 121 base
0.501
0.504
0.542
0.501
0.502
0.508
0.500
0.531
DenseNet- 121 basa/NTL.
0.568
0.587
0.585
0.584
0.586
0.586
0.581
0.573
DenseNet- 121 base/visit
0.632
0.627
0.637
0.624
0.618
0.612
0.624
0.629
DenseNet-122 121 base/TTE
0.677
0.657
0.658
0.651
0.644
0.651
0.658
0.657
ResNet- base
0.503
0.557
0.517
0.580
0.505
0.563
0.501
0.531
ResNet- 152 base/KTL
0.632
0.620
0.621
0.615
0.604
0.620
0.618
0.601
ResNet-152 base/visit
0.622
0.636
0.621
0.638
0.631
0.635
0.640
0.641
ResNet- 152
base/TTE
0.678
0.656
0.674
0.667
0.662
0.664
0.670
0.669
Table 6: Prognostic TTE performance for INSPECT, measured by time-dependent C-statistics. Bold indicates the best performer. Underlined indicates statistically significant difference versus the
"base/TTT /TE models.
19

---

Published as a conference paper at ICLR 2025
E MERLIN RESNET- 152 RESULTS
For experimental consistency, we selected the ResNet-152 base model as the starting point for our continued pretraining experiments, however we also conducted an initial study on the Merlin 3D foundation model (Blankemeier et al., 2024). Merlin employs a dual-objective pretraining strategy: first, it uses contrastive loss on radiology notes associated with abdominal CT scans, followed by supervised training where EHR diagnosis codes are used as phenotype classification labels. We compare their reported best performing backbone, ResNet-152, with our weight-inflatee (2D Im- ageNet weights) baseline ResNet-152 base- Table 7 reports adaption performance for prognostic classification, prognostic TTE and diagnostic classification tasks, where ResNet- 152 base performs similarly to Merlin. While Merlin performs better in mortality classification (though not in the TTE formulation), most tasks show no statistically significant differences across the corresponding met- rics. Note that Merlin's 's pretraining data consists solely of abdominal CT scans, which likely limits its generalizability to a broader CT data distribution, particularly for capturing lung disease features during pretraining.
Task Category
Task
Metrie
Merlin
ResNet base
Best
PE: (0M)
0,518
0.573
ResNet bare
Mortality (IM)
0,632
0.583
Merlin
Mortality (6M)
0,586
0.557
Merlin Merlin
Mortality (12M)
AUROC
0,590
0,554
Prognostic Classification
Readmission (IM)
0,538
0,536
Readmission (6M)
0.513
0.537
ResNet base
Readmission (12M)
0,538
0,509
PH (12M)
0.514
0.537
Mortality
0.549
0.505
Readmission
0.545
0.560
ResNet
PH
0.615
0,505
ATX
Harrel''s C-index
0.565
0.577
ResNet base
Progostic TTE
CMG
0,575
0.505
Merlin
CONS
0.601
0.559
EDM
0.601
0.505
Merlin
PEFF
0.549
0.536
PE Left
0,603
0.654
ResNet
PE Cent.
0,572
0.687
ResNet base
PE Right
0.562
0.618
ResNet base
PE Chromic
AUROC
0,550
0,507
Diagnostic Classification
PE Acute
0.647
0.695
PE Indeterminate
0.693
0.704
RV/LV <1
0.515
0.593
ResNet base
RV/LV 21
0.572
0.626
ResNet base
Table 7: Performance comparison of Merlin and ResNet across various tasks. Each task is eval- uated using either AUROC for binary classification tasks Harrell's C-index for TTE tasks. Bold indicates the best performer. Underlined indicates no statistically significant difference between models (i.e., 0>0055). The Best column denotes the model with the highest performance for each task. Note: Merlin is pretrained on abdominal CT's.
20

---

Published as a conference paper at ICLR 2025
F MODELING HYPERPARAMETERS
We have detailed our hyperparameter set, both in the image backbones and the probing methods, in Table
Model
Hyperparameters
Values
SwinUNETR
learning rate
10-4, 10"+,10 10-5 10 0.1, 0.2, 0.3 2x2x2
dropout prob
patch size
window size
7x7x7, 8x8x8
augmentation strategies
Random rotations, flips
DenseNet
learning rate depth
10 10-3 3, 10
10 10
Image backbones
121, 169
num. of dense blocks
3,4
dropout prob
0.1, 0.2, 0.3
augmentation strategies
Random rotations, flips
ResNet
learning rate
10-3, 10-4, 10-5, 10-6 152
depth
residual block
BottleneckBlock
dropout prob
0.1, 0.2, 0.3 Random rotations, flips
augmentation strategies
Logistic Regression
penalty
L-1, L-2
learning rate
0.01, 0.1, 0.2 lbfgs, liblinear roc_auc, accuracy scikit-learn 1.3.2
solver
scoring
software version
Probing heads
DeepSury
num. nodes per layer depth
32, 64, 128, 256
2, 4
learning rate
10-4, 10-5, 10-6
dropout prob
0.1, 0.2, 0.3
software version
pycox 0.2.3
Table 8: Hyperparameter search grids and software versions of methods under comparison.
21

---

Published as a conference paper at ICLR 2025
G CLINICAL TASK SUMMARY
Since INSPECT and RSPECT both focus on patients suspected of having pulmonary embolisms, we focus on pulmenary-related pathologies in our experiments. Table 9 outlines the clinical significance of these tasks.
Abbr.
Name
Clinical Meaning
Pulmonary
A condition characterized by high blood pressure in pulmonary arteries, lead- ing to right heart strain and potentially heart failure. Predicting on long term out- come is of higher clinical usage.
Hypertension
PE Pulmonary
A lff-thhreatenig condition where a blood clot blocks the pulmonary arteries, leading to impaired blood flow to the lungs and acute respiratory symptoms. A collapse of lung tissue that prevents normal oxygen absorption. oflen resulting from obstruction or fluid accumulation.
Embolism
ATX
Atelectasis
CMG
Cardiomegaly
An abnormal enlargement of the heart, typically caused by high blood pressure heart valve discase, which lead heart failure.
CONS
Consolidation
A region of lung tissue filled with liquid instead of air. often seen in pneumonia. causing reduced oxygen exchange.
EDM
Edema
The abnormal accumulation of fluid in tissues or alveolar spaces, often related to heart failure or lung injury, causing difficulty in breathing.
PEPF
Pleural
The buildup of excess fluid between the layers of the pleura surrounding the lungs, often due infection, heart failure, or malignancy, leading to difficulty breathing.
Effusion
Table 9: Clinical definitions of pulmonary conditions used to define TTE tasks in the INSPECT dataset.
H base PRETRAINING DATASETS
The pretrained image backbone base. i.c. SwinUNETR, DenseNet and ResNet, have their corre- sponding pretrained dataset before take them in for continued pretraining. The details are de- scribed in Table 10.
Architecture
Method
Loss
Dim.
Dataset
Size
SwinUNETR
Self-supervised
MAE
3D
Custom Medical
10,050
DenseNet- 121 bane 1
Supervised
BCE
2D
ImageNet
1,281,167 1,281,167
Supervised
BCE
2D
ImageNet
Table 10: Summary of model architectures, pretraining approaches, and off-the-shelf weights. Cus- tom Medical includes 3D Chest, Abdomen, and Head/Neck CT and MRI scans (Valanarasu et al., 2023).
I EXPERIMENT RUNTIME COST
For our experiments, we have trained the models in PHI--ompliant on-premise server that uses 2 compute nodes, each with Intel Xeon 2.7GHz CPU cores, 200 GB RAM and Nvidia A100 GPUs 4 H100 GPUs. The total estimate cost for model base/TTe regime pretrianing is shown Table 11.
Architecture
Number of GPUs
Estimated wall-clock time Estimated GPU hours
SwinUNETR DenseNet- N12 1
4 H100 (80GB)
15 days
1,440 GPU hours
4 A100 (40GB)
9 days
864 GPU hours
ResNet- 152 /5
4 A100 (80GB)
10 days
960 GPU hours
Table 11: Summary of compute cost across architectures
22

---

Published as a conference paper at ICLR 2025
J FULL PARAMETER FINE-TUNING ANALYSIS
To illustrate the data efficiency of TTE pretraining method, we conduct single task full parameter fine tuning both INSPECT and RSPECT dataset, starting from base model checkpoint. This experiment is to show what is the cost to take off-the-shelf model to evaluate on the same set of tasks as ours but following traditional machine learning pipeline conduct full parameter fine- tuning towards each task. Due to the high number of tasks, i.e. 7 binary classification tasks and TTE tasks from INSPECT data, as well 8 binary classification tasks from RSPECT data, we only choose one representative tasks from each data set and conduct the fine-tuning. We train until the model has early stopping till the plateau on validation set measured by cross entropy loss. The results shown in Table 12. We conclude that single task full parameter fine-tuning does not scale as well as TTE pretraining, and general the performance no better than, not much worse than TTE pretraining.
Full param fine-tuned
I Linear probe results (AUROC)
Dataset (fine-tuning task)
Architecture
Delta
results (AUROC)
SwinUNETR
0.616
0.597
+.188
SwinUNETR, Raza /TTE
0.637
0.672
5.20 %
INSPECT (12 month PH)
DenseNet- 12 base
0.553
0.506
9..28
DenseNet- 12 base////TT
0.631
0.689
88.41 %
ResNet- 1044-52,, 152
0.552
0.537
2.79 %
ResNet- 1 152
0.649
0.718
-9.61 %
SwinUNETR, bane SwinUNETRpaee//
0.606
0.578
+.484
0.631
0.636
0.799
RSPECT (RV/L& 2 1)
DenseNet- -aaeNNett-11aass 121
0.643
0.607
5,93
DenseNet- 121 12
0.611
0.686
-0.93 %
ResNet- 152base ResNet- 152
Miiket-122par
0.653
0.626
4.31
0.588
0.708
11.95 %
Summary compute cost across architectures for full parameter fine-tuning on one sin- gle task from each dataset. On average the full parameter fine-tuning computation cost for both INSPECT and RSPECT data ranges from 57.5 (standard deviation: 61.4) to 31.3 (standard devia- tion: 16.7) GPU hours respectively, which is much more expensive than linear probe, i.e. logistic regression on CPU, which takes minute level computation cost. In addition, the models when fine- tuned single label task, tend to overfit given the large amount of parameter counts. The above results are from more stringent regularization training.
Table 12:
23

---

Published as a conference paper at ICLR 2025
K CONFIDENCE INTERVALS
Here we are showing the confidence interval differences between all the baselines against the pro- posed method (base/TTe) in both classification tasks (measured by AUROC, in Table 13) and time- lo-event tasks (measured by Harrell's C-index, in Table 14, time-dependent C-statistics, in Table 15 and integrated Brier score in Table 16). The RSPECT dataset confidence interval difference is detailed in Table 17.
Model
Morality
Readmission
PII
IM
6M
12M
IM
6M
12M
12M
SwinUNETR
6-0.151, -0.055*
(4.155, -.0R2)"
(4.193,
-0113
(-01553, -0011"
(-0.099, 4..001"
(4.072 -001I**
(-0084, -0.05
SWinUNETR
60.103, 0,023y"
(0.123, 0.053
(0112
0042)*
(0.140, 0,013""
(0.993 0.022""
(0.774
0005)*
(-0083,
0.001"
SWilLNETR
6-0113, -0,04y" (0.0)
0.064) (40144)
(-0124, 20055)*
(-0.155, 0.05))
-2004) (-0322,
.0084) (4.075,
(40055 -0011)"
SwinUNETR
(0.0)
(00)
(6,0)
(0.0)
DenseNet- 121
60.193, 0.115
(0.155, 0,092)
6-.173,
0
0.066" (-0.994
(-0.85,
(-.1144,
0.044
(0.123,
0063y*
(4.16),
0.990*
DenocNet-121 Nem
(9,119,
40.133
0.095
(a117)
.ooay"
(48.122
(-.123,
-065))
(4.162,
40022
DonseNet-121 beae/ttact
-0,062y*
4.133
(4.193, 1335
00,055
0.065) (-0.145,
0006)** (0.154
DenscNot-122 21 teex/TT
(0.0)
(000)
(000)
(0D)
(0.0)
(0.0)
(00))
ResNes-152.. kear
(-0.235,
(-0.257,
0.172
(-0232, 1644
(-0.115
0.006
(-0.116,
0.022y
(4.102
20010
20234,-0151
ResNet: 152 32T
0,094y 00.174,
(0.155,
0.085**
(a.14,
0.83)*
(0,033
0054)
(0.113,
0.022**
(0.112.
00255
(4.171.
0000
ReiNet- 152
64.132, 4,006)"
(40.123, ..06333
(-1113,, .0056)
(-8056, JL062y
(-0.132, 4.032)
-0877
(4.113, (002))
RecNer- 152
(0.00
(0.0)
(0.0)
(0.0)
(0.0)
(0.0)
(0.)
Table 13: 95% confidence interval differences for classification performance on INSPECT dataset for proposed method (bano/rre) and baselines, measure by AUROC. The indicates statistical sig- nificance under p-value at 0.05 for null hypothesis. (PE: pulmonary embolism: PH: pulmonary hypertension)
Harrell's Cindex 7
SkiaLNETR
(0.381, 10178
ca615, -0.16
(-892, 4.134
(-2222 4,822"
(0,210, 4.121"
60272 -.164"
(6.55,
..190
(4.33--0.55P -0264
SkinLNNITT NETR
741776
-00017"
(4E404 200617
(-887) -0 S00
(OA0,
41000
603997
(4608 a00Ty"
(4005, -10000
SkiaUNETR beaittat
00.00,, 0.011"
(-044), 0.053)
O3077 (A895,
(4.144 0,011"
0,078* (0.333,
(0214, 0.127)
(0,05 0,000)
(4.7%%
Sxtnl 2a
(0,00
10.00
(0.0)
(0.9)
(0.0)
De1aeNe-121 musim
(0.155,
4003/
(-8.79
0.001*
(9.55,
0.146
(0644,
0.477*
(4965,
00345**
541 40 1945*
108 (406
(-0 189 (.046)
64463, 40055)
4.079' (4677)
(4642)) -005KP
(40.50
-0071"
(4023% -1099
DesseNet-1 (2)
(0.0)
(0,0)
00.
(0.9)
(0.0)
(0.9)
(0.0)
(3,00
RexNes-S0 150
--,332, 1646
(-0443,
-0357
(-0275,
.00301
(0.11,
0,2281
(0.518,
0.000
(6:352
(-0.250.
0.023
(4.574,
ResNes mmrm
(-0.22,
0.177)
(.113),
00071
(.0560,
0.284)
0.15**
0.005
(4,672
.0273
(4.42)
ResNes suar
24.247
(.554),
0000))
(00178 0012)"
(-0250, 00191*
(0.227, 0.030*
(0.122 0.055
(.0.055 0.832)
Table 14: 95% confidence interval differences for time-to-event performance on INSPECT dataset for proposed method (bSas/TTE) and baselines, measured by Harrell's C-Index. The indicates sta- tistical significance under p-value at 0.05 for null hypothesis.
24

---

Published as a conference paper at ICLR 2025
Madel
Mon
Headr
PI1
AIX
CINS
00038
3,99,
(-0
(-R032
(-aR0)
6003/
0,000
StaatNNE 1R
2,85,
(L06))
High
SwaiUNETE suat
60000
0003-
0102
6000
DexseNaail E2
2,222
0. 1681
1-0.46
1201
100
1.555
1859
[-0.68
(-0
0.140
1811
DerssNNa- 121
[-00050
111000
(-11003
-0122)
(-0
-0013
-0100
-
DerscNea-11 121
16:00
(1.00
(0.00
30.03
01.00
20.00
RosNet- 152
3,59,
(L160
1.568
01300
0100
6037
5.07.
01221-
0.00L 6033
caPs, 1 110651
RoiNic 152
0.00
0012
CJ03S, (L00G)
1100)
CJ94
0.044
00044-
ILICS (L001)
C/025 IL00C)
DenacNoc-1 com
10.00
(0.0)
(0.0)
(0.90
(0.
(0.00
(9.0)
(0.0)
Table 15: 95% confidence interval differences for time-to-event performance on INSPECT dataset for proposed method (base/TTE /TTE) and baselines, measured by time-dependent C-statistics. The indi- cates statistical significance under p-value at 0.05 for null hypothesis.
Meddl
Inlegrated Brier Scute
Moet
Reada.
281
AIX
CMG
CONS
EDM
PEFF
10000,,
(DARN, 0:014>"
(0:005,
10015,
SwisUNETR m
a.122)
6.027Y
0.025)
SwIUUNET suas/
0.034)
(0,00N 0.001>"
(00045, U03T!*
(1014, ou1sy"
00.015, D005)*
(-O u000)
SwidUNEEM
(0.00
20.
20,
20.
DesoeNa-121 421
10. 0.206/
0.
(0.751,
0.791
18188
0,218'
10.685,
D.625)*
10.295 D.520
10.764, 0.822)
DenseNer 121
(1411) 0.001
loves,
(000O, 0.09y*
2000,
0/000*
D0002 (0.3844
(0.04 BURGY
10000
DensoNe-131 www/worr
00002 0.0027*
(0100 00000
(0.002,
ouray*
(0A0 0.000*
(0.0077 8005)**
(0005), BUFT)*
200033, OUITS)
DeroeNas-13 121 /mee
0)
10,
(0.00
(0. 03
10,
10.
20,
20.
RooNd-152
10.266.
(0.00,,
(0.000
(0.86
10.381,
OL0GRY
RosNee-
MUNI, DUE1))
(D4055,
0.085)
(-00004 (0026)
BeaNees-12. 152
(0,003, 0.0L7)*
(0.022 0,018y*
(C.007,
0.02)
(0.000 0L020)
(0.012 0,018)*
10,208, D.022)
(0,011, DUDLD)
10,010, 0.023)"
RosNor-132
(0.00 0)
0%, M3
(0.9)
(0.0) 03
09, 02
20.0)
30.0)
(0.0)
Table 16: 95% confidence interval differences time-to-event performance on the INSPECT dataset for the proposed method (base/Tre and baselines, measured by integrated Brier score. The indi- cates statistical significance under p-value 0.05 for null hypothesis.
PR:
RVVE Ratio
Lch
Righ
Cleosic
Indes
SwisLNETT
[-0094, 40029'
(-01399,
4,012
[-0,59, -0015)*
[-0.199 0.108)
(-0.274, -0003
(40,31,
-0000)"
(-0118) -0005
(-0/922,
400211"
SUIJUNETR
1-0097 0006)"
rO009, 0.25)
LIB85, (-0.122 0.011)
10.00, (U57)
(1120% QUEEUI
10.15% (L0G0)
COL174, -00711
(-008
SwisUNETR
[-0.156. 0.0171
(4.096, 0.115)
(4.120 0.178)
6-0290) 0.127)
(-.206 0.144)
(-0.203, -0007y+
(-0.210. 1003
SoiaUNETT
10.0 0
(0.0)
80
(0.
(0.0)
10,
00.0)
20.
DenseNer sue
10003, 00121*
60,170
0.17
10077,
(BUNS, 0073)
(.05)) (40082,
(0.152 D0ET1"
-016)
-0.13)
DenseNes 121 sun//
00013* 1-0,091
00005> 640155,
(.008),
14.107, (.085)
(4003,, -0007Y
(40.199
DensoNa--13 121
(0.00
J0,
(0.00
RasNee 152
LIJ03M 9.064)
(O006)
0.85))
(1L55 Q.015)
(0032, (.45))
(BU2S, 0L233)
L..116, (LD6S)
CO.00, 0.83))
(40.422
(1826)
RcoNi--15. 52
(0.040
0.007)
(OU06),
0.086)
G00057
0.015)
(-0034,
0.641)
C-0022 0.213)
(0.120 (.075)
(-0055,
0.06)
(-0.122,
.0012)
R55Noe-152 suacmaat
(-00001 40005)
00.000, 0U862
BUDF 2.28Y
140000 (1005)
(40073, 40003
R52 17e
10, 0)
(0.0)
10.
(0.0)
(0.0)
(0. D0
(0.0)
10,
Table 17: 95% confidence interval differences classification performance of different methods on RSPECT dataset for diagnosis labels, measured by AUROC. * indicates the statistical significance under value 0.05
25

---

Published as a conference paper at ICLR 2025
L TASK HEAD CAPACITY ON RANDOMLY INITIALIZED BACKBONES
We are interested in knowing the different probing method's utility, with random initialization from different model architectures (i.e. image backbone provides no predictive power thus only relying on probing methods), how much does probe methods provide for prediction, and what is delta of each model from that initialization. The comparisons are shown in Figures 7, 8.
Random Init
SNiTU
Mortality IM-
Mortality 6M
Mortality
Readmission 1M
Readmission
Readmission 12M
Pulm HTN 12M
359
635
0.8
0.5
052
055
3.3
AUROC
D43
0.9
0.5
010
335
0.0 (Higher better)
3.6
6/9
6.5
(Higher botter)
(Higher better)
Figure Comparison of models towards random initialization and each model's delta on AUROC
Random Init.
Base
Visit
TTE
SwinUNETR
DenseNet
ResNet
Mortality
Readmission
Pulm HTN
Atelectasis
Cardiomegaly
Consolidation
Edema
Pieural Effusion
655
0.60
0.70
0.50
0.55
0.60
0.05
0.70
0.30
6.6
0.65
0.70
0.75
Harrell's s C-index
Harrell's C-index
Harrell's C-index (Higher is better)
(Higher is better)
(Higher is better)
Figure 7: Comparison of models towards random initialization and each model's delta on Harrell's C-index
26

---

Published as a conference paper at ICLR 2025
Random Init.
Base
MT
Visit
TTE
SwinUNETR
DenseNet
ResNet
Mortality
Readmission
Pulm HTN
Atelectasis
erdiomyggaly
Consolidation
Edema
Pleural Effusion
0.07
0.08
0.00
0:10
0.0
0.2
2/4
0.8
D-4
0.6
3.8
Integrated Brier Score (Lower is beller)
Integrated Brier Scare (Lower is better)
Integrated Brier Score (Lower is better)
Figure 8: Comparison of models towards random initialization and each model's 's delta integrated Brier score
27

---

Published as a conference paper at ICLR 2025
M PIECEWISE EXPONENTIAL LOSS FUNCTION
In our methods section, we briefly describe that we use a piecewise exponential model, and thus, use the corresponding piecewise exponential survival loss our loss function. In this appendix section, we provide the exact formulas used to implement that survival loss.
For the piecewise exponential, we select a number of pieces and each piece covers the response period starting at Sp and ending at Ep. For every piece and image in our dataset, our model needs output Aip, the instantaneous hazard rate for the patient described by that image during the response time period p. These À values are then used to define a survival function for each image i:
P (t) = II
exp (min(t, Ep) - Sp)) Sp)], p=1
(3)
This survival function has an associated PDF, f, that is simply the negative of the derivative of the survival function.
as:
f. Öt
(4)
Applying the derivative operator, we obtain that
S.(0 Σ Aip TS. Sts Sp)
(5)
We define the loss function for training by using the survival function to implement the standard survival likelihood equation and taking the product of that likelihood over the entire dataset. A; event indicator whose value is when the actual event is observed. We can then plug our definitions for the survival function S and the pdf f to obtain our final loss function:
II (S.(1) (f((t)
[S)
Σ Aip S15)
(6)
(=
P
P
A.
exp
(mi(mi( F.p) - -p)))((1 Sp)]]
Σ
Aip [S.
S15
28

---

Published as a conference paper at ICLR 2025
N KAPLAN-MEIER CURVES FOR TTE TASKS
We here plot the stratified groups in terms of the diagnosis of pulmonary embolism for Kaplan-Meier among all of our TTE tasks, shown in Figures 10, 11, 12, 13, 14, 15, 16.
Kaplan-Meier curve for mortality
Pos. Group Median TTE: 20 days
1.0
PE Neg. Group Median TTE: 17 days
Survival Probability
0.6
0.4
PE Positive (95% CI)
PE Negative (95% CI)
1
12
15
Time Event Censorship (Months)
Figure 9: Kaplan-Meier curve for Mortality
Kaplan-Meier curve for readmission
PE Pos: Group Medias TTE: days
1.0
PE Neg Group Median TTE: 14 days
Survival Probability
0.8
0.4
PE Positive (95% CI)
PE Negative (95% CI)
12
15
Time Event Censorship (Months)
Figure 10: Meilan-Meier curve for Readmission
29

---

Published as a conference paper at ICLR 2025
Kaplan-Meier curve for PH
Pos. Group Madian days
1.0
Neg. Group days
Survival Probability
0.6
PE Positive (95% CI)
PE Negative (95% CI)
0.2
1
3
9
12
15
Time to Event or Censorship (Months)
Figure 11: Kaplan-Meier curve for Pulmonary Hypertension (PH)
Moplan-Meier curve for Atelectasis
PE Pos. Group Median TTE: 18 days
PE Neg. Group Median TTE: 15 days
Survival Probability
0.8
0.4
PE Positive (95% CI)
PE Negative (95%
0.2
Time Event or Censorship (Months)
Figure 12: Kaplan-Meies Meier curve for Atelectasis
Kaplan-Meier curve for Cardiomegaly
1.0
Survival Probability
PE Pos Group Median TTE: 18 days
0.6
PE Neq, Group Mediah TTE: 15 days
0.4
PE Positive (95% CI)
PE Negative (95% CI)
12
15
Time Event Censorship (Months)
Figure 13: Kaplan-Meier curve for Cardiomegaly
30

---

Published as a conference paper at ICLR 2025
Aaplan-Meier curve for Consolidation
Pos Median TTE: days Neg, Group days
1.0
Survival Probability
0.6
PE Positive (95% CI)
PE Negative (95% CI)
0.2
12
15
Time to Event or Censorship (Months)
Figure 14: Kaplan-Meich Meilerr-Miirreeee curve for Consolidation
Kaplan- Meier curve for Edema
PE Pos. Group Median TTE: 18 days
PE Neg. Group Median TTE: 15 days
Survival Probability
0.8
0.4
PE Positive (95% CI)
PE Negative (95%
0.2
15
Time Event or Censorship (Months)
Figure 15: Kaplan-Meier curve for Edema
Kaplan-Meier curve for Pleural Effusion
Pos. Group Median TTE: 18 days
1.0
PE Neg. Group Modian TTE: 16 days
Survival Probability
0.6
0.4
PE Positive (95% CI)
PE Negative (95% CI)
1
3
12
15
Time Event or Censorship (Months)
Figure 16: Kaplan-Meier curve for Pleural Effusion
31

---

Published as a conference paper at ICLR 2025
O PRETRAINING TASK EVENT TIME DISTRIBUTIONS
We have employed 1,192 future clinical events in our pretraining mixture and here we plot the distribution the time event w.r. each of the 8,192 events across all the cohorts in INSPECT data. We have maximum and median time to event distributions for the 8192 labels and all time to event distribution for each occurrence in Figures 17, 18, 19.
Distribution of Max Time-to-Evee Event of 8192 future events
500
400
300
200
100
50
100
150
200
250
Time-to-Event (Months)
Figure 17: Maximum Time to Event across INSPECT cohort for 8192 future events (Per event)
Distribution of Median Time-to-Eveed of 8192 future events
50
Frequencyy per event
40
30
20
0.00
0.05
0.10
0.15
0.20
0.25
Time-to-Event (Months)
Figure 18: Median Time Event across INSPECT cohort for 8192 future events (Per event)
Frequency per oczurennce (log ozale)
Distribution Time-to-Event of 8192 future events
10a
10a
50
100
150
200
250
Time-to-Event (Months)
Figure 19: All Time to Event across INSPECT cohort for 8192 future events (Per occurrence)
32

---

Published as a conference paper at ICLR 2025
P EXAMPLES OF PRETRAINING CLINICAL EVENTS
We have list the set of medical ontologies that are availabe in the EHR modality, in Table 18. We fur- ther stratify the frequency of our 8,192 pretraining clinical events across all cohorts the INSPECT dataset into quintiles and present the top 3 examples in each quintile, in Table 19.
Ontology
OMOP Extension
Medicare Specialty
CPT4
CVX
ICD9Proc
RxNorm
SNOMED
RxNorm Extension
Cancer Modifier
ICDIOPCS
CMS Place Service
Visit
Ethnicity
Gender
ICDO3
Race
LOINC
HCPCS
Table 18: OHDSI Athena ontolgies used in our benchmark
Medical Code
Description
Quintile
3NOMED/6085300
Disorder of magnesium metabolism Urine dipstick for urobilinogen
SNOMED//66321100 LOINC/2472-9
Immunoglobulin M (IgM) in serum or plasma
RxNorm/905216
Hydralazine Hydrochloride 50 MG Finding of menstrual bleeding Losartan potassium 100 MG
2
SNOMED/27631900
RxNomm/979463
2
SNOMED/S9659001
Amylase measurement, serum
3
SNOMED/400166009
Acquired keratoderma Late effect of complications of procedure (disorder)
3
SNOMED/70209001
3
LOINC/32355-0
Bacteria identified in Specimen by Respiratory culture
SNOMEDD/66750017
Acute heart failure
4
SNOMED/297971001
Finding of sensation of skin
1C0MEDD/11664400
Procedure on body system
5
SNOMED/118717007
Procedure organ
5
SNOMED/1 11772000
Procedure on cardiovascular system
5
Table 19: Top 3 events in each quintile
33

---

Published as a conference paper at ICLR 2025
Q FEATURE ATTRIBUTION
We use GradCAM (Selvaraju et al., 2017) to visualize the Gradient- weighted Class Activation Map- ping for our DenseNet-13 1eteeet121mmodee and its baselines, w.r.f. to the TTE tasks curated, where we selected the CTs with the corresponding TTE labels that actually happened for the patient after the CT capture, in Figure 20, We can observe that the TTE version of the model can focus the pathology in more reasonable region rather than scattered features learned by different baselines.
DredCAM [base]
GradiCM (MTT)
GredGAM (Via))
GrediAM (TT))
Melkettats
0.4
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.25
(a) GradCAM for Atclectasis= True after CT's capture for models and baselines
Actual
GradCAM (0a2e)
GradCAM (MTL)
GradCAM (VG.)
GraCCM (TTE)
Consolidetion occured: Then
Comolitetto occured:
Cansoliaaiion occuped:
Consniiatiio bonared: Thue
Consalidetien encumd: The
0.2
0.25
0.25
0.0
0.00
0,0
(b) GradCAM for Consolidation= after CT's capture for models and baselines
Pleurs] Pleura_Fhaadon Eifidoon recurdd Thee
Pleura Ieturleffuude Effludion sccured: Thu
Pleural Efftutttoo occured: Thu
Reeural_ftuuion
Pleursl, Eifieiiiiinn occunet: Tue
Pleursl_tfluiin
Pleursl Tittttutioon Pleurs__ttuiion orouned True
0,25
0.2
0.25
0.5
0.0
0.00
0.0
0.00
0.0
(c) GradCAM for Pleural Effusion=TTrue after CT's capture for models and baselines
GraZIAM (bese)
FredCAM [MTL
GreZCAM (Mti
GrearAM (TTE)
Cardiomegaly
0.2
0.25
0.2
0.5
(d) GradCAM for Cardiomegaly= after CT's capture for models and baselines
GradCAM Edeemn occured Tram
GradCAM Eidems encured: Thue
GradCAM Endemn occumed: Thue
Edema occure: Tape
Edems occured: Thue
0.5
0.5
0.1
0.0
0.0
(e) GradCAM for Edema=True after CT's capture for models and baselines
Actaal HH sccured: Thuc
GredCAM (bave) PH occures: Thue
EredIAM (MTL) accured: True
GredIAM (vict) PH oocures: True
GRedCAM (TTE] PH occuned: Tlue
0.50
0.5
0.5
0.25
0.25
0.25
0.0
(f) GradCAM for Pulmonary Hypertension= True after CT's capture for models and baselines
Figure 20: GradCAM visualizations of sampled CTs for various occurred conditions: Atelectasis, Consolidation, Pleural Effusion, Cardiomegaly, Edema, and Pulmonary Hypertension.
34

---

Published as a conference paper at ICLR 2025
R SUBGROUP PERFORMANCE
We conduct an analysis to assess performance on our best performing model, SwinUNETR, across sensitive subgroups, ensuring our pretraining technique does not unfairly disadvantage any group compared to existing methods. The stratified subgroup counts for this experiment under test split of INSPECT dataset in Table 20. We evaluate performance using the AUROC (bootstrapped, n=1000) on 5 binary prognostic tasks, comparing TTE against base, showing that TTE pretraining does reduce performance for sensitive groups and generally improves risk ranking across all groups. The details are in Tables 21, 22, 23, 24, 25, 26, 27.
Group
Concept
Counts
Female Male
1844
Gender
1370
18-39
448
39-69
1667
Age
69-89
993
>8
106
Asian
554
Black
192
Race
Native
74
White
1730
Unknown
664
Hispanic
495
Ethnicity
Not Hispanic
2612
Unknown 107
Table 20: Subgroup stratified counts for the test split under INSPECT dataset
AUROC
Group
Concept
SwinUNETR
SwinUNETR base
SwinUNETR base/ /viait
SwinUNETR basn
Female
0.666 (0.607, 0.722)
0.648 (0.588, 0.705)
0.690 (0.636, 0.739)
0.828 (0.789, 0.866)
Gender
Male
0.735 (0.670, 0.800)
0.712 (0.637, 0.785)
0.696 (0.634, 0.753)
0.841 (0.797, 0.879)
18-39
0.753 (0.591. 0.887)
0.715 (0,553, 0.863)
0,643 (0.512, 0.765)
0.887 (0.808, 0,952)
39-69
0.701 (0.639, 0.755)
0.688 (0.626, 0.751)
0.683 (0.628, 0.735)
0.846 (0.809, 0.882)
Age
69-89
0.661 (0.580, 0.738)
0.643 (0.558, 0.720)
0.730 (0.672. 0.790)
0.809 (0.752, 0.864)
>89
0.662 (0.435. 0.869)
0.648 (0.434, 0.877)
0.723 (0.445, 0.916)
0.647 (0.426, 0.847)
Asian
0.675 (0.547, 0.787)
0.642 (0.522, 0.760)
0.668 (0.562, 0.774)
0.831 (0.732, 0.912)
Black
0.533 (0.328, 0.738)
(0.313, 0.707)
0.660 (0.523, 0.785)
(0.690, 0.905)
Race
Native
0.667 (0.040, 1.000)
0.689 (0.040, 1.000)
0.743 (0.333, 0.997)
0.773 (0.555, 0.939)
White
0.749 (0.694, 0. 7.797)
0.732 (0.673, 0.786)
0.718 (0.665, 0.769)
0.855 (0.816, 0.889)
Unknown
0.649 (0.561, 0.733)
0,636 (0.553, 0.726)
0,670 (0.598, 0.740)
0.778 (0.719, 0.830)
Hispanic
0.645 (0.532, 0.754)
0.646 (0.524, 0.758)
0.685 (0.588, 0.778)
0.785 (0.701, 0.866)
Ethnicity
Not Hispanic
0.701 (0.648, 0.751)
0.681 (0.630. 0.730)
0.699 (0.655. 0.741)
0.842 (0.809, 0.873)
Unknown
0.731 (0.563, 0.862)
0.720 (0.566, 0.848)
0.688 (0.572, 0.800)
0.832 (0.744, 0.912)
Table 21: 1-Month Mortality prognosis breakdown by subgroups, reported as the mean AUROC of a test set bootstrap (n=1000) with 95% CI. Bold indicates the best performance across all models.
35

---

Published as a conference paper at ICLR 2025
AUROCT
Group
Concept
SwinUNETR base
SwinUNETR base
SwinUNETR baso/vicis
SwinUNETR base/IIE
Female
0.661 (0.620, 0.702)
0.676 (0.632, 0.717)
0.699 (0.653, 0.743)
0.813 (0.779, 0.842)
Gender
Male
0.710 (0.657, 0.762)
0.739 (0.688, 0.790)
0.727 (0.683, 0.769)
0.819 (0.779, 0.858)
18-39
0.799 (0.727. 0.861)
0.803 (0.729. 0.867)
0.723 (0.634, 0.810)
0.887 (0.820, 0.944)
39-69
0.685 (0.641, 0.725)
0.700 (0.653. 0.747)
0.690 (0.645, 0.734)
0.811 (0.775, 0.842)
Age
69-89
0.635 (0.571, 0.691)
0.657 (0.596, 0.717)
0.755 (0.693, 0.805)
0.799 (0.752, 0.844)
>89
0.680 (0.502. 0.844)
0.738 (0.562. 0.880)
0.687 (0.473, 0.870)
0.732 (0.528, 0.883)
Asian
0,666 (0.583, 0,740)
0.693 (0.608, 0.771)
0,682 (0.591. 0.770)
0.755 (0.667, 0,840)
Black
0.573 (0.414, 0.728)
0.615 (0.478. 0.758)
0.666 (0.536. 0.793)
0.818 (0.720, 0,905)
Race
Native
0.806 (0.500, 1.000)
0.794 (0.488, 0.986)
0.795 (0.586, 0,966)
0.882 (0.710, 0.990)
White
0.713 (0.674, 0.751)
0.724 (0.684, 0.7)
0.733 (0.694, 0.772)
0.834 (0.802, 0.862)
Unknown
0.664 (0.612. 0.713)
0,680 (0.627. 0.728)
0.707 (0.650. 0.761)
0.779 (0.739, 0.818)
Ilssaaiic
0.651 (0.554, 0.743)
0.673 (0.579, 0.762)
0.679 (0.597, 0.756)
0.767 (0.694, 0.833)
Ethnicity
Not Hispanic
0.688 (0.653, 0.724)
0.709 (0.675, 0.743)
0.724 (0.688, 0.759)
0.820 (0.795, 0.845)
Unknown
0.693 (0.624, 0.758)
0.689 (0.625, 0.753)
0.726 (0.647, 0.795)
0.792 (0.741, 0.840)
Table 22: 6-Month Mortality prognosis breakdown by subgroups, reported as the mean AUROC of a test set bootstrap (n=1000) with 95% CI. Bold indicates the best performance across all models.
1
Group
Concept
SwinUNETR base
SwinUNETR /N
SwinUNETR bask/vints
SwinUNETR base/TTK
Female
0.655 (0.616, 0.694)
0.669 (0.630. 0.706)
0,660 (0.617, 0.702)
0.796 (0.762, 0.825)
Gender
Male
0.717 (0.669, 0.762)
0.732 (0.685, 0.775)
0.672 (0.626, 0.717)
0.796 (0.761, 0.833)
18-39
0.828 (0.760, 0.886)
0.839 (0.777, 0.893)
0.681 (0.589, 0.764)
0.876 (0.805, 0.936)
39-69
0.682 (0.636, 0.722)
0.694 (0.653, 0.731)
0.644 (0.598, 0.686)
0.791 (0.759, 0.825)
Age
69-89
0.632 (0.573, 0.687)
0.647 (0.589, 0,702)
0.706 (0.649. 0.763)
0.784 (0.738, 0.827)
>-89
0.605 (0.434, 0.771)
0.616 (0.429, 0.796)
0.632 (0.440, 0.809)
0.660 (0.481, 0.819)
Asian
0.675 (0.664
(0.604, 0.746)
0.695 (0.624, 0.763)
0.657 (0.567. 0.741)
0.754 (0.684, 0.824)
Black
0.624 (0.498, 0.746)
0.645 (0.522, 0.766)
0.620 (0.505, 0.730)
0.781 (0.683, 0.866)
Race
Native
0.716 (0.451, 0.950)
0.690 (0.394, 0.937)
0.745 (0.487. 0,952)
0,840 (0.650, 0.981)
White
0.716 (0.675, 0.755)
0.724 (0.687. 0.762)
0.676 (0,632. 0.714)
0.822 (0.793, 0.851)
Unknown
0.657 (0.609, 0.703)
0.673 (0.625, 0.719)
0.672 (0.621, 0.725)
0.750 (0.707, 0.788)
Hispanic
0.643 (0.553, 0.728)
0.660 (0.571, 0.746)
0.631 (0.551, 0.701)
0.708 (0.630, 0.780)
Ethnicity
Not Hispanic
0.689 (0.657, 0.721)
0.702 (0.668, 0.736)
0.675 (0.640, 0.7
0.808 (0.782, 0.835)
Unknown
0.698 (0.638, 0.755)
0.706 (0.647, 0.761)
0.695 (0.621, 0.764)
0.764 (0.713, 0.811)
Table 23: 12-Month Mortality prognosis breakdown by subgroups, reported as the mean AUROC of a test set bootstrap (n=1000) with 95% CL. Bold indicates the best performance across all models.
AUROCT
Group
Concept
SwinUNETR
SwinUNETR baso/HTL
SwinUNETR base/vicit
SwinUNETR base/TTE
Female
0.539 (0.459, 0.623)
0.534 (0.457, 0.614)
0,551 (0.477, 0.630)
0.599 (0.522, 0.671)
Gender
Male
0.498 (0.391, 0.607)
0.492 (0.386, 0.600)
0.532 (0.448, 0.613)
0.587 (0.499, 0.675)
18-39
0.452 (0.280, 0.634)
0.438 (0.260, 0.613)
0.598 (0.456, 0.737)
0.603 (0.423, 0.760)
39-69
0.458 (0.373. 0.546)
0.457 (0.374, 0.546)
0.523 (0.446, 0.597)
0.558 (0.482, 0.628)
Age
69-89
0.687 (0.573, 0.794)
0.689 (0.581, 0.787)
0.492 (0.370, 0.608)
0.676 (0.547, 0.787)
>-99
0.459 (0.013, 0.907)
0.432 (0.013, 0.878)
0.772 (0.585, 0.915)
0.363 (0.200, 0,533)
Asian
0.594 (0.442, 0.742)
0.603 (0.428, 0.748)
0.674 (0.485, 0.854)
0.619 (0.460, 0.763)
Black
0.374 (0.025, 0.848)
0.397 (0.027. 0.865)
0.523 (0,338, 0.775)
0.524 (0.201, 0.945)
Race
Native
0.860 7606 0.940)
0,880 (0.780, 0.960)
0.287 (0.146, 0.447)
0.941 (0.875, 1.000)
White
0.490 (0.41 0.573)
0.478 (0.395, 0.558)
0,521 (0.449, 0.591)
0.614 (0.540, 0.686)
Unknown
0.491 (0.408. 0.571)
0.489 (0.410. 0.575)
0,613 (0.525, 0.700)
0.525 (0.444, 0.604)
Hispanic
0.670 (0.4799, 0.836)
0.662 (0.468, 0.842)
0.515 (0.406, 0.626)
0.587 (0. (0.400, 0.774)
Ethnicity
Not Hispanic
0.515 (0.439. 0,586)
0.513 (0.442, 0,587)
0,542 (0.475, 0.610)
0.620 (0.557, 0,680)
Unknown
0.433 (0.347, 0.522)
0.426 (0.340, 0.51
0.666 (0.563, 0.761)
0.480 (0.380, 0.576)
Table 24: 1-Month Readmission prognosis breakdown by subgroups, reported as the mean AUROC of test set bootstrap (n=1000) with 95% CI. Bold indicates the best performance across all models.
36

---

Published as a conference paper at ICLR 2025
AUROC
Group
Concept
SwinUNETR base
SwinUNETR ba
SwinUNETR base/visit
SwinUNETR base/TIE
Female
0,571 (0.522, 0.619)
0,577 (0.528, 0.626)
0.488 (0.436. 0.541)
0.607 (0.558, 0.654)
Gender
Male
0.525 (0.466,, 0.585)
0.526 (0.470, 0.583)
0.608 (0.541, 0.670)
0.607 (0.556, 0.660)
18-39
0,533 (0.422, 0.630)
0.542 (0.440, 0.636)
0.541 (0,433, 0.654)
0.660 (0.569, 0.753)
39-69
0,516 (0.467, 0.564)
0,519 (0.472, 0.566)
0.519 (0.461, 0.573)
0.590 (0.544, 0.634)
Age
69-89
0.616 (0.542, 0.691)
0.624 (0.549, 0.694)
0.537 (0.456, 0.616)
0.610 (0.535, 0.689)
2-89
0.604 (0.000, 0.984)
0.616 (0.000. 0.984)
0.792 (0.601, 0.956)
0.553 (0. 16., 0.921)
Asian
0.548 (0.456, 0.641)
0.556 (0.459, 0.648)
0.549 (0.401, 0.693)
0.591 (0.497, 0.678)
Black
0,510 (0,305, 0.706)
0,509 (0.305, 0.698)
0.525 (0.365, 0.697)
0.557 (0.382, 0.734)
Race
Native
0.841 (0.612, 0,989)
0.850 (0.659, 0.985)
0.598 (0.431, 0.774)
0.835 (0.588, 0,977)
White
0,554 (0,504, 0.604)
0,561 (0,511, 0.610)
0.529 (0.479. 0,580)
0.626 (0.575, 0,675)
Unknown
0,504 (0,451, 0.559)
0.502 (0.451, 0.553)
0.591 (0.522, 0.652)
0.594 (0.544, 0.642)
Hispanic
0.610 (0,516, 0.704)
0.606 (0.510, 0.695)
0.594 (0.503, 0.685)
0,556 (0.455, 0.648)
Ethnicity
Not Hispanic
0.553 (0,507, 0.596)
0,560 (0.519. 0.601)
0.528 (0.475. 0,574)
0.620 (0.581, 0.661)
Unknown
0.469 (0.401, 0.532)
0.472 (0.413, 0.534)
0.600 (0.511, 0.687)
0.605 (0.541, 0.665)
Table 25: 6-Month Readmission prognosis breakdown by subgroups, reported as the mean AUROC of test set bootstrap (n=1000) with 95% CI. Bold indicates the best performance across all models.
AUROC1
Group
Concept
SwinUNETR base
SwinUNETR base/HTL
SwinUNETR bass/viett
SwinUNETR base/TTE
Female
0.588 (0.539, 0.629)
0.565 (0.520, 0.609)
0.492 (0.444, 0.538)
0.607 (0.562, 0.650)
Gender
Male
0.563 (0.512, 0.612)
0.547 (0.497. 0.598)
0,555 (0.501. 0.610)
0.601 (0.552, 0.650)
18-39
0.557 (0.467, 0.645)
0.554 (0.462, 0.639)
0.496 (0.402, 0.58
0.653 (0.573, 0.730)
39-69
0.562 (0.519. 0,605)
0.530 (0.484, 0,571)
0,509 (0.461. 0.557)
0.596 (0.554, 0,640)
Age
69-89
0.611 (0.544, 0.673)
0.600 (0.529, 0.664)
0.529 (0.464, 0.596)
0.592 (0.523, 0.659)
>88
0.589 (0.019. 0.913)
0.675 (0.180, 0,981)
0.678 (0.466, 0,875)
0,564 (0.130, 0.855)
Asian
(0.524, 0.694)
0.587 (0.497, 0.679)
0,473 (0.383, 0.561)
0.635 (0.555, 0.716)
Black
(0.336, 0.680)
0.486 (0.325, 0.642)
0.488 (0.344, 0.635)
0.634 (0.472, 0.787)
Race
Native
0,901 (0.763, 0.987)
0.843 (0.679. 0.976)
0.631 (U,462. 0.792)
0.861 (0.658, 1.000)
White
0.589 (0.542, 0.632)
0.564 (0.519, 0.607)
0.516 (0.472, 0.562)
0.626 (0.585, 0.668)
Unknown
0.523 (0.481, 0.569)
0.515 (0.470. 0.560)
0.566 (0.510, 0.621)
0.560 (0.512, 0.607)
Hispanic
0.522 (0.440, 0.604)
0.534 (0.447, 0.622)
0.602 (0.510, 0.689)
0.479 (0.401, 0.557)
Ethnicity
Not Hispanic
0.597 (0.560, 0.634)
0.570 (0.534, 0.608)
0.506 (0.464, 0.545)
0.635 (0.601, 0.669)
Unknown
0.509 (0.454, 0.565)
0.501 (0.437, 0.557)
0.554 (0.472, 0.631)
0.589 (0.529, 0.644)
Table 26: 12-Month Readmission prognosis breakdown by subgroups, reported as the mean AUROC of a test set bootstrap (n=1000) with 95% CI. Bold indicates the best performance across all models.
AUROC
Group
Concept
SwinUNETR base
SwinUNETR boee HMTL.
SwinUNETR baze/viais
SwinUNETR baxw/ TTR
Female
0,597 (0.547, 0.645)
0.607 (0.556, 0.655)
0.559 (0.515, 0.602)
0.665 (0.618, 0.710)
Gender
Male
0.563 (0.504, 0.622)
0.572 (0.512, 0.631)
0,542 (U.496. 0.592)
0.673 (0.625, 0.721)
18-39
0.543 (0.379, 0.709)
0.553 (0.386, 0. 0.703)
0.515 (0.431, 0.594)
0.758 (0.650, 0.862)
39-69
0.606 (0.552, 0.660)
0.615 (0.562, 0.669)
0.552 (0.506. 0.596)
0.661 (0.614, 0.705)
Age
69-89
0.547 (0.486, 0.609)
0.556 (0.497, 0.616)
0.579 (0.516, 0.639)
0.644 (0.584, 0.705)
>89
0.539 (0.361, 0.731)
0.548 (0.361, 0.728)
0.534 (0.404, 0.669)
0.464 (0.294, 0.631)
Asian
0.613 (0.521, 0.707)
0.620 (0.523, 0.714)
0.547 (0.470, 0.627)
0.667 (0.576, 0.750)
Black
0.706 (0.532, 0.850)
0.727 (0.562, 0.876)
0.373 (0.249, 0.501)
0.767 (0.644, 0,872)
Race
Native
0.378 (0.074. 0.711)
0.326 (0.046. 0.644)
0,218 (0.075, 0.369)
0.583 (0.312, 0.833)
White
0.579 (0.529, 0.630)
0.593 (0.543, 0.643)
0,580 (0.539, 0.625)
0.687 (0.643, 0.730)
Unknown
0.603 (0.558, 0.647)
0.609 (0.566. 0.655)
0.581 (U.532, 0.633)
0.641 (0.595, 0.686)
Hispanic
0.602 (0.511. I, 0.687)
0.615 (0.521, 0.703)
0.500 (0.414, 0.584)
0.641 (0.550, 0.722)
Ethnicity
Not Hispanic
0,584 (0.538, 0.626)
0,589 (0.547, 0.632)
0.562 (0. 0,598)
0.679 (0.643, 0.715)
Unknown
0.618 (0.566. 0.672)
0.626 (0.570. 0.677)
0.597 (0.526. 0.662)
0.646 (0.594, 0.696)
Table 27: 12-Month Pulmonary Hypertension prognosis breakdown by subgroups, reported as the mean AUROC of test set bootstrap (n=1000) with 95% CI. Bold indicates the best performance across all models.
37


Reviews:
[{"review_id": "4ckH4Q6Jhv", "reviewer": "Reviewer_rQZZ", "summary": "The paper introduces a future-guided pretraining approach using time-to-event supervision to enhance the prognostic capabilities of 3D medical imaging models. By incorporating longitudinal EHR data into the pretraining process and predicting time-until-event, the model outperforms traditional methods across multiple standard tasks, as demonstrated by thorough experiments.", "questions": "1. Why start from 3D image scans instead of 2D medical images? Is this due to the dataset choice, or has similar work already been done on 2D data?\n2. How does the choice of time segmentation for EHR data affect model results during pretraining? Specifically, my understanding is that the model predicts the probability of a patient experiencing a certain event at intervals like 1, 2, or 3 years post-scan. How does the granularity of these time segments impact the performance of the pretrained encoder?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Innovative Approach: The method creatively leverages EHR data following a medical scan to assist model pretraining, demonstrating better performance compared to imaging-only pretraining.\n2. Comprehensive Evaluation: Extensive comparisons across multiple tasks validate the robustness and efficiency of the TTE-based approach across different architectures.", "weaknesses": "1. Dependence on Large EHR Datasets: This approach relies on extensive, high-quality EHR data, which many medical datasets do not include.\n2. Limited Modality Scope: Tested only on CT images; broader modality testing could validate versatility across imaging types.\n3. Interpretability: The TTE pretraining’s impact on specific pixel-level biomarkers is less clear; additional analysis on feature attribution could help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EqSK9s3UZt", "reviewer": "Reviewer_Wo2C", "summary": "The authors proposed to utilize the time-to-event information in EHR that paired with the imaging data as a form of supervision for the pre-training purpose. A public dataset with both 3D images and EHR notes is employed for the pre-training and downstream applications. Another dataset without the time events is also used for the evaluation of model adaptation. The manuscript is easy to follow. However, it also suffers from several critical flaws, which are detailed below.", "questions": "See above", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Propose utilizing the time events as pre-training tasks specially designed for prognosis tasks in downstream applications. \n- The manuscript is overall easy to follow", "weaknesses": "- The proposed method is limited in generalization since it will require longitudinal time-to-event EHR data as the supervision for the pre-training. In comparison to the common self-supervised pre-training, the proposed methods are harder to scale up.\n\n- There is no comparison evaluation between the proposed method and prior methods in model pre-training. Only the results of the proposed method with different model architectures are reported. It will be difficult to appreciate the benefits of the proposed method.\n\n- The selected model architecture also raises questions since there are many popular model networks in medical imaging, e.g., 3D-UNet, ViT, etc. It will be helpful to see their performance compared to the vanilla ResNet. \n\n- Baselines without the pre-training process should also be reported.\n\n- The current setting utilizes public data for both pre-training and downstream applications. Having a separate evaluation dataset of a prognosis task will be helpful. \n\n- The proposed method is limited in technical innovation, though utilizing the time-to-event data as a form of supervision is relatively new in the pre-training. Mostly existing techniques are adopted for the pre-training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WhU41kJNeS", "reviewer": "Reviewer_iczn", "summary": "*Edit: Score increased from 6 to 8 during discussion period.*\n\nThis paper presents a self-supervised learning (SSL) method for 3D medical imaging data that leverages electronic health records (EHR) to provide extra sources of supervision via time-to-event modeling. The proposed method, future-guided pretraining, performs time-to-event (TTE) survival modeling of various medical events in the longitudinal EHR associated with each 3D scan. The authors show that future-guided pretraining consistently improves downstream TTE modeling and prognostic classification tasks – also improving data efficiency – without degrading standard diagnostic classification performance.", "questions": "- What exactly does it mean that Steinberg et al.’s method was used to “[sample tasks to maximize entropy given the frequency distribution of medical codes populating the DAG”? I feel that a basic plain-language description of the motivation for this procedure is needed first: why is this method being applied at all? Are there way more than 8k events and the goal is to settle on a subset of 8k “meaningful”/common ones for pretraining? I don’t understand the motivation.\n- Unless I am misunderstanding, this is the only description of the TTE pretraining procedure and labels used: “We define our TTE task labels by predicting the time until the next occurrence of a medical code.” The previous Section 3 described deep survival modeling in the abstract, so I expected Section 4 to more concretely describe how TTE pretraining works. Is this a “competing risks” approach, where multiple events are being modeled simultaneously (in “multi-label” fashion)?\n- What are the 8,192 EHR tasks/events? I’m aware it would be cumbersome or impossible to list and define them all, but any reasonable attempt to convey information about them would be useful. What kinds of “events” are they? What are some examples?\n- Related to the above point, are the downstream labels *also* present in the set of TTE pretraining tasks? If so, isn’t there concern of “label leakage”, where the model has been pretrained on label information present in the downstream training dataset? Please clarify this.\n\n**Minor comments/questions:**\n- Line 13: Maybe “build” instead of “capture” since you use this word in the next sentence.\n- In-text citation style seems off – should be parenthetical (\\pcite{}) in most cases when used at end of sentence/clause: “Sox et al. (2024)” -> “(Sox et al., 2024)”\n- Change “e.g.” -> “e.g.,” throughout\n- Would include more recent references [1,2] when discussing deep prognosis models on longitudinal medical imaging (first paragraph of Section 2)\n- “i.e. 8192” -> “i.e., 8.192”\n- “Our approach improves training data efficiency, increasing training labels by an average of 3x over labels assigned to patients based on their current EHR visit.” This is a bit unusual to highlight as a main contribution – I don’t think readers will understand what “increasing training labels” means without having read the entire paper (nor why this impact data efficiency). Perhaps clarify language here to indicate that your approach provides 3x as many sources of supervision during SSL + that this is what provides data efficiency benefits.\n- “Pretraining task labels as assigned per-CT scan and vary in density based on pretraining approach, see Figure 2.” Perhaps “as assigned” is meant to be “are assigned”? Also change “, see Figure 2” -> “(Figure 2)”.\n- Be consistent with “c-statistic” vs. “C-statistic”\n\n**References**\n[1] Holste, Gregory, et al. \"Harnessing the power of longitudinal medical imaging for eye disease prognosis using Transformer-based sequence modeling.\" NPJ Digital Medicine 7.1 (2024): 216.\n[2] Sriram, Anuroop, et al. \"Covid-19 prognosis via self-supervised representation learning and multi-image prediction.\" arXiv preprint arXiv:2101.04909 (2021).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The presentation quality is very high. Care has been taken to logically organize the paper, clearly articulate key points, and straightforwardly present results with concise figures and tables.\n- The core idea is creative, making use of the wealth of longitudinal EHR data associated with each 3D volume for pretraining.\n- Discussion or related work and background is particularly strong.\n- Experiments are sufficiently thorough and easy to interpret – results are convincing.", "weaknesses": "- The actual description of the TTE pretraining approach is brief (lines 184-191) and somewhat unclear. I would advise the authors to flesh out this section. See specific questions below.\n- A description or list of the 8,192 EHR pretraining tasks is never provided. I’m aware there may not be a convenient place to list this many items, but a general description of categories of events or a few illustrative examples would be helpful. Without this information, it’s impossible to assess whether, e.g., one the TTE pretraining tasks is *also* used as a downstream TTE modeling task. In this case, there may be concerns of “label leakage”.\n\nI’m happy to increase my score once these issues are addressed – this is an otherwise strong submission.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)