Time-to-Event Pretraining for 3D Medical Imaging

Authors: Zepeng Frazier Huo, Jason Alan Fries, Alejandro Lozano, Jeya Maria Jose Valanarasu, Ethan Steinberg, Louis Blankemeier, Akshay S Chaudhari, Curtis Langlotz, Nigam Shah
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Multimodal learning, medical imaging, Electronic Health Records

Abstract:
With the rise of medical foundation models and the growing availability of imaging data, scalable pretraining techniques offer a promising way to identify imaging biomarkers predictive of future disease risk. While current self-supervised methods for 3D medical imaging models capture local structural features like organ morphology, they fail to link pixel biomarkers with long-term health outcomes due to a missing context problem. Current approaches lack the temporal context necessary to identify biomarkers correlated with disease progression, as they rely on supervision derived only from images and concurrent text descriptions. To address this, we introduce time-to-event pretraining, a pretraining framework for 3D medical imaging models that leverages large-scale temporal supervision from paired, longitudinal electronic health records (EHRs). Using a dataset of 18,945 CT scans (4.2 million 2D images) and time-to-event distributions across thousands of EHR-derived tasks, our method improves outcome prediction, achieving an average AUROC increase of 23.7% and a 29.4% gain in Harrell’s C-index across 8 benchmark tasks. Importantly, these gains are achieved without sacrificing diagnostic classification performance. This study lays the foundation for integrating longitudinal EHR and 3D imaging data to advance clinical risk prediction.

Decision: accept (poster)
Reviewer scores: 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1729390456546

thread_type:
official_review

<thread_object>
creator: reviewer_iczn

<content>
summary:
*Edit: Score increased from 6 to 8 during discussion period.*

This paper presents a self-supervised learning (SSL) method for 3D medical imaging data that leverages electronic health records (EHR) to provide extra sources of supervision via time-to-event modeling. The proposed method, future-guided pretraining, performs time-to-event (TTE) survival modeling of various medical events in the longitudinal EHR associated with each 3D scan. The authors show that future-guided pretraining consistently improves downstream TTE modeling and prognostic classification tasks – also improving data efficiency – without degrading standard diagnostic classification performance.

strengths:
- The presentation quality is very high. Care has been taken to logically organize the paper, clearly articulate key points, and straightforwardly present results with concise figures and tables.
- The core idea is creative, making use of the wealth of longitudinal EHR data associated with each 3D volume for pretraining.
- Discussion or related work and background is particularly strong.
- Experiments are sufficiently thorough and easy to interpret – results are convincing.

weaknesses:
- The actual description of the TTE pretraining approach is brief (lines 184-191) and somewhat unclear. I would advise the authors to flesh out this section. See specific questions below.
- A description or list of the 8,192 EHR pretraining tasks is never provided. I’m aware there may not be a convenient place to list this many items, but a general description of categories of events or a few illustrative examples would be helpful. Without this information, it’s impossible to assess whether, e.g., one the TTE pretraining tasks is *also* used as a downstream TTE modeling task. In this case, there may be concerns of “label leakage”.

I’m happy to increase my score once these issues are addressed – this is an otherwise strong submission.

questions:
- What exactly does it mean that Steinberg et al.’s method was used to “[sample tasks to maximize entropy given the frequency distribution of medical codes populating the DAG”? I feel that a basic plain-language description of the motivation for this procedure is needed first: why is this method being applied at all? Are there way more than 8k events and the goal is to settle on a subset of 8k “meaningful”/common ones for pretraining? I don’t understand the motivation.
- Unless I am misunderstanding, this is the only description of the TTE pretraining procedure and labels used: “We define our TTE task labels by predicting the time until the next occurrence of a medical code.” The previous Section 3 described deep survival modeling in the abstract, so I expected Section 4 to more concretely describe how TTE pretraining works. Is this a “competing risks” approach, where multiple events are being modeled simultaneously (in “multi-label” fashion)?
- What are the 8,192 EHR tasks/events? I’m aware it would be cumbersome or impossible to list and define them all, but any reasonable attempt to convey information about them would be useful. What kinds of “events” are they? What are some examples?
- Related to the above point, are the downstream labels *also* present in the set of TTE pretraining tasks? If so, isn’t there concern of “label leakage”, where the model has been pretrained on label information present in the downstream training dataset? Please clarify this.

**Minor comments/questions:**
- Line 13: Maybe “build” instead of “capture” since you use this word in the next sentence.
- In-text citation style seems off – should be parenthetical (\pcite{}) in most cases when used at end of sentence/clause: “Sox et al. (2024)” -> “(Sox et al., 2024)”
- Change “e.g.” -> “e.g.,” throughout
- Would include more recent references [1,2] when discussing deep prognosis models on longitudinal medical imaging (first paragraph of Section 2)
- “i.e. 8192” -> “i.e., 8.192”
- “Our approach improves training data efficiency, increasing training labels by an average of 3x over labels assigned to patients based on their current EHR visit.” This is a bit unusual to highlight as a main contribution – I don’t think readers will understand what “increasing training labels” means without having read the entire paper (nor why this impact data efficiency). Perhaps clarify language here to indicate that your approach provides 3x as many sources of supervision during SSL + that this is what provides data efficiency benefits.
- “Pretraining task labels as assigned per-CT scan and vary in density based on pretraining approach, see Figure 2.” Perhaps “as assigned” is meant to be “are assigned”? Also change “, see Figure 2” -> “(Figure 2)”.
- Be consistent with “c-statistic” vs. “C-statistic”

**References**
[1] Holste, Gregory, et al. "Harnessing the power of longitudinal medical imaging for eye disease prognosis using Transformer-based sequence modeling." NPJ Digital Medicine 7.1 (2024): 216.
[2] Sriram, Anuroop, et al. "Covid-19 prognosis via self-supervised representation learning and multi-image prediction." arXiv preprint arXiv:2101.04909 (2021).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648537549

thread_type:
official_review

<thread_object>
creator: reviewer_wo2c

<content>
summary:
The authors proposed to utilize the time-to-event information in EHR that paired with the imaging data as a form of supervision for the pre-training purpose. A public dataset with both 3D images and EHR notes is employed for the pre-training and downstream applications. Another dataset without the time events is also used for the evaluation of model adaptation. The manuscript is easy to follow. However, it also suffers from several critical flaws, which are detailed below.

strengths:
- Propose utilizing the time events as pre-training tasks specially designed for prognosis tasks in downstream applications. 
- The manuscript is overall easy to follow

weaknesses:
- The proposed method is limited in generalization since it will require longitudinal time-to-event EHR data as the supervision for the pre-training. In comparison to the common self-supervised pre-training, the proposed methods are harder to scale up.

- There is no comparison evaluation between the proposed method and prior methods in model pre-training. Only the results of the proposed method with different model architectures are reported. It will be difficult to appreciate the benefits of the proposed method.

- The selected model architecture also raises questions since there are many popular model networks in medical imaging, e.g., 3D-UNet, ViT, etc. It will be helpful to see their performance compared to the vanilla ResNet. 

- Baselines without the pre-training process should also be reported.

- The current setting utilizes public data for both pre-training and downstream applications. Having a separate evaluation dataset of a prognosis task will be helpful. 

- The proposed method is limited in technical innovation, though utilizing the time-to-event data as a form of supervision is relatively new in the pre-training. Mostly existing techniques are adopted for the pre-training.

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716004393

thread_type:
official_review

<thread_object>
creator: reviewer_rqzz

<content>
summary:
The paper introduces a future-guided pretraining approach using time-to-event supervision to enhance the prognostic capabilities of 3D medical imaging models. By incorporating longitudinal EHR data into the pretraining process and predicting time-until-event, the model outperforms traditional methods across multiple standard tasks, as demonstrated by thorough experiments.

strengths:
1. Innovative Approach: The method creatively leverages EHR data following a medical scan to assist model pretraining, demonstrating better performance compared to imaging-only pretraining.
2. Comprehensive Evaluation: Extensive comparisons across multiple tasks validate the robustness and efficiency of the TTE-based approach across different architectures.

weaknesses:
1. Dependence on Large EHR Datasets: This approach relies on extensive, high-quality EHR data, which many medical datasets do not include.
2. Limited Modality Scope: Tested only on CT images; broader modality testing could validate versatility across imaging types.
3. Interpretability: The TTE pretraining’s impact on specific pixel-level biomarkers is less clear; additional analysis on feature attribution could help.

questions:
1. Why start from 3D image scans instead of 2D medical images? Is this due to the dataset choice, or has similar work already been done on 2D data?
2. How does the choice of time segmentation for EHR data affect model results during pretraining? Specifically, my understanding is that the model predicts the probability of a patient experiencing a certain event at intervals like 1, 2, or 3 years post-scan. How does the granularity of these time segments impact the performance of the pretrained encoder?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732181410075

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to all reviewers

comment:
Dear reviewers,

We thank you for the constructive feedback and questions. We appreciate all the reviewers for their thoughtful feedback and constructive criticisms of our manuscript. We are grateful for all reviewers for acknowledging:
- Innovative approach (rQZZ, Wo2C, iczn), 
- Presentation and flow of the paper (Wo2C, iczn)
- Comprehensive experiment (rQZZ, iczn)
- Background and discussion strength (iczn)

At the same time, we appreciate the criticism and suggestions for our paper:
- Dependency on EHR data (rQZZ, Wo2C)
- Input modality as only CT (rQZZ)
- Interpretability on feature attribution (rQZZ) and label construction (iczn)
- Time-to-event objective clarity, including time piece segmentation (rQZZ), details of 8,192 pretraining tasks (iczn)
- Lack of pretraining baselines and wider model architectures (Wo2C)
- Lack of private data for external validation (Wo2C)

In summary, we have added respective sections in our new manuscript to address these concerns:
- We have rewritten the Introduction section to address the core contribution claims
- We added Figure 1 to elucidate the state of affairs, where existing methods do not take EHR into account but only local/short-horizon information as supervision, framed as a missing context problem. We further changed the paper title to reduce the confusion brought by reviewers.
- We added Appendix P  for feature attribution 
- We added Appendix O and Appendix N to illustrate the examples of pretraining tasks and the aggregated distribution of event times in our cohort
- We added Appendix L to detail the piecewise exponential loss function formulation
- We added Appendix D for baseline comparison with a current CT foundation model Merlin’s performance
- We reorganized the evaluation metrics. Specifically we added Harrell’s C-index as TTE evaluation in Table 3 (which doesn’t evaluate on segmented timepieces but one hazard score in non-piecewise fashion) and moved time-dependent C-statistics into Appendix C, given this metrics up-weighs the long time horizon examples and thus gives unfair evaluation on non-piecewise models (i.e. our CoxPH survival head).

In addition to that, we reorganized the paper and added extra contents to make the manuscript more clear:
- We conducted our classification task with Logistic Regression to align common practices of ‘linear probing’ for foundation model adaptation (therefore expect the Table 2 and 4 to have updated numbers)
- We added Appendix I for full parameter fine-tuning experiment to show the generalizability of our pretraining method compared to per-task fine-tuning
- We added Append H for computation cost of our models
- We added Appendix M for Kaplan-Meier curves to illustrate long term survival probabilities for respective TTE tasks

We have highlighted the changed text to the color of orange for ease of comparison.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732529928065

thread_type:
official_comment

<thread_object>
creator: area_chair_qzpa

<content>
title:
Please engage in discussion

comment:
Dear all,

Many thanks to the reviewers for their constructive reviews and the authors for their detailed responses.

Please use the next ~2 days to discuss any remaining queries as the discussion period is about to close.

Thank you.

Regards,

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a novel framework called “Time-to-Event (TTE) pretraining” for 3D medical imaging models, aiming to improve outcome prediction by leveraging longitudinal supervision from electronic health records (EHRs). Traditional pretraining methods for 3D medical imaging focus on capturing structural features, but they fail to establish links between pixel-level biomarkers and future health outcomes due to the absence of temporal context. To address this, the authors propose a method that incorporates future event information from EHRs into the pretraining process.

The key contribution of the paper is the integration of time-to-event modelling with 3D imaging data. Using a large dataset of 18,945 chest CT scans linked to longitudinal EHR data, the authors constructed 8,192 unique pretraining tasks. Each task predicts not only whether a clinical event will occur but also when it is likely to happen, using distributions derived from EHR records. This approach significantly increases label density, resulting in a richer and more informative pretraining signal. The authors report good performance improvements, achieving a 23.7% increase in AUROC and a 29.4% improvement in Harrell’s C-index across eight benchmark tasks, while maintaining strong performance on standard diagnostic classification tasks. This demonstrates the framework’s capacity to better identify prognostic pixel biomarkers in 3D medical images.

The strengths of the proposed approach are its scalability, use of large-scale multimodal data, and enhanced predictive performance. By incorporating temporal supervision, the authors provide a more comprehensive view of disease progression, enabling models to capture subtle imaging biomarkers that are associated with future health risks. The methodology also handles right-censored data, a common challenge in survival analysis, and utilises large datasets to pretrain models, contributing to generalisability. The use of open datasets and publicly available code promotes transparency and reproducibility in the research community.

However, the study has certain limitations. First, the reliance on large computational resources poses practical challenges for institutions without access to high-memory GPUs, as highlighted by the authors themselves too. Additionally, the pretraining dataset, while substantial, is still smaller than datasets used in other large-scale model pretraining efforts, which may affect generalisation across different populations and imaging modalities. The study is also focused on CT scans, possibly limiting its applicability to other imaging types like MRIs or X-rays; but that is secondary.

In conclusion, the paper offers a decent contribution to the field of medical imaging by introducing a time-to-event pretraining strategy that bridges the gap between imaging data and long-term clinical outcomes.