Title: The Boundaries of Fair AI in Medical Image Prognosis: A Causal Perspective
Venue: NeurIPS 2025
Accepted: yes
Keywords: fairness

time-to-event

survival analysis

bias

distribution shift

causal
Abstract: As machine learning (ML) algorithms are increasingly used in medical image analysis, concerns have emerged about their potential biases against certain social groups. Although many approaches have been proposed to ensure the fairness of ML models, most existing works focus only on medical image diagnosis tasks, such as image classification and segmentation, and overlooked prognosis scenarios, which involve predicting the likely outcome or progression of a medical condition over time. To address this gap, we introduce FairTTE, the first comprehensive framework for assessing fairness in time-to-event (TTE) prediction in medical imaging. FairTTE encompasses a diverse range of imaging modalities and TTE outcomes, integrating cutting-edge TTE prediction and fairness algorithms to enable systematic and fine-grained analysis of fairness in medical image prognosis. Leveraging causal analysis techniques, FairTTE uncovers and quantifies distinct sources of bias embedded within medical imaging datasets. Our large-scale evaluation reveals that bias is pervasive across different imaging modalities and that current fairness methods offer limited mitigation. We further demonstrate a strong association between underlying bias sources and model disparities, emphasizing the need for holistic approaches that target all forms of bias. Notably, we find that fairness becomes increasingly difficult to maintain under distribution shifts, underscoring the limitations of existing solutions and the pressing need for more robust, equitable prognostic models.

The Boundaries of Fair AI in Medical Image

Abstract

As machine learning (ML) algorithms are increasingly used in medical image analysis, concerns have emerged about their potential biases against certain social groups. Although many approaches have been proposed to ensure the fairness of ML models, most existing works focus only on medical image diagnosis tasks, such as image classification and segmentation, and overlooked prognosis scenarios, which involve predicting the likely outcome or progression of a medical condition over time. To address this gap, we introduce FairTTE, the first comprehensive framework for assessing fairness in time-to-event (TTE) prediction in medical imaging. FairTTE encompasses a diverse range of imaging modalities and TTE outcomes, integrating cutting-edge TTE prediction and fairness algorithms to enable systematic and fine-grained analysis of fairness in medical image prognosis. Leveraging causal analysis techniques, FairTTE uncovers and quantifies distinct sources of bias embedded within medical imaging datasets. Our large-scale evaluation reveals that bias is pervasive across different imaging modalities and that current fairness methods offer limited mitigation. We further demonstrate a strong association between underlying bias sources and model disparities, emphasizing the need for holistic approaches that target all forms of bias. Notably, we find that fairness becomes increasingly difficult to maintain under distribution shifts, underscoring the limitations of existing solutions and the pressing need for more robust, equitable prognostic models.

1 Introduction

Machine learning (ML) algorithms trained on real-world medical images may inherently exhibit bias, leading to discrimination against certain social groups [57, 42]. This is especially concerning in the medical field, where biased algorithms can result in inequitable treatment recommendations, misdiagnoses, or unequal access to care [46]. Therefore, ensuring the fairness of ML models and identifying hidden biases within medical images is critical for advancing health equity [54].

Existing research on fairness in ML and medical image analysis has primarily focused on medical image diagnosis tasks, such as image classification and segmentation [82, 64]. These diagnostic tasks typically aim to determine the presence or absence of a condition. However, prognosis scenarios, which involve predicting the likely outcome or progression of a medical condition over time, have been

Data

Algorithm

Evaluation

TTEOutcomes

TTEPrediction

Fairness

Metrics

Pre-training

ChestX-ray

FundusImage

Brain MRI

In-hospital

DeepHit

Pre-

Time-dependent

DistributionShift

Mortality

processing

C-index

StatisticalTest

Nnet-

In-

IntegratedBrier

LateAMD

Survival

processing

Score

CausalAnalysis

Alzheimer's

PMF

Post-

Time-dependent

Fairness-accuracy

2D

3D

Disease

processing

AUC

Trade-off

largely overlooked. In ML, prognosis is framed as time-to-event (TTE) prediction or survival analysis, where the goal is to predict the time until a critical event. Unlike classification, TTE prediction provides a richer and more dynamic approach to modeling medical outcomes by accounting not only for the presence or absence of a condition but also for the timing and progression of key health events. This makes it especially valuable for predicting long-term outcomes such as survival or disease recurrence [21]. Despite its importance, there remains a significant gap in the literature regarding fairness in TTE prediction, particularly in the context of medical imaging (see Appendix A).

Developing fairness methods for TTE prediction in medical imaging presents several unique challenges compared to tasks like image classification or segmentation. First, there is a lack of public datasets that pair medical images with TTE outcome information. While many public datasets include sensitive attributes, they primarily focus on diagnostic tasks and do not provide the temporal data necessary for training TTE prediction models. Second, there is limited understanding of how biases in medical images specifically affect the fairness of TTE prediction models, complicating the development of fair algorithms. Third, the absence of a universally accepted fairness metric for TTE prediction poses a challenge in assessing and comparing fairness across studies [36, 53, 79]. Existing fairness metrics are often not well-suited to the complexities of medical applications (see Section 2.1). Furthermore, varying experimental designs and evaluation approaches in the literature hinder direct comparisons between fairness methods, leading to fragmentation in the field and impeding the establishment of best practices for fair TTE prediction in medical imaging.

To address the challenges of understanding fairness in TTE prediction for medical imaging, we introduce FairTTE, the first comprehensive framework specifically designed for this task (Figure 1). Our approach begins by applying causal reasoning to develop a general framework for understanding and quantifying bias in TTE prediction. This framework decomposes the data likelihood into distinct components, enabling a fine-grained analysis of various sources of bias, including disparities in image features, censoring rates, TTE labels, mutual information between images and TTE, and mutual information between images and censoring indicators. This principled decomposition allows us to explain why standard TTE prediction methods and fairness algorithms frequently fail to mitigate bias in practice (see Section 2.2). Using this framework, we characterize not only the presence but also the type and extent of bias across different datasets, offering a detailed understanding of fairness challenges in medical image prognosis. To complement the framework, we construct a collection of large-scale, publicly available medical image datasets tailored for fair TTE prediction. These datasets span a wide range of imaging modalities, including fundus images, chest X-rays, and brain MRIs, and cover diverse TTE outcomes, such as mortality, late age-related macular degeneration (AMD), and Alzheimer's disease. Finally, leveraging both the datasets and causal framework, we establish a realistic and comprehensive investigation study for fairness in TTE prediction. This investigation incorporates state-of-the-art (SOTA) algorithms across various learning strategies and, to our knowledge, represents the first attempt to provide a fine-grained analysis of fairness in TTE prediction, making a significant contribution to the growing field of fair ML in medical imaging.

Our work also explores fair TTE prediction methods across various settings, considering factors such as pre-training and distribution shifts that may influence model fairness. Building on our causal framework and through the training of over 20,000 models, we derive several key insights to inform future research in this area, as follows:

Bias is prevalent across TTE prediction models trained on different imaging modalities, with consistent performance disparities observed between demographic groups.

All medical image datasets examined in our study exhibit some sources of bias that adversely affect the fairness of TTE prediction models.

Most SOTA fairness methods struggle to consistently mitigate bias-while they can improve fairness in some settings, these gains are often accompanied by reductions in predictive accuracy.

Pre-training improves model accuracy but has minimal impact on fairness.

Different types of distribution shifts affect fair TTE prediction in distinct ways, underscoring the growing difficulty of maintaining both fairness and utility under realistic clinical scenarios.

2 Unified Framework for Fair TTE Prediction

2.1 Fair TTE Prediction Setup

We first introduce the notations used throughout the paper and then formulate the fair TTE prediction task. In this study, we focus on analyzing methods designed for achieving group fairness in rightcensored TTE prediction.

Notations. A TTE dataset contains observations for each individual along with their corresponding (right-censored) TTE outcomes. Specifically, each individual's data is represented by a tuple of random variables (RVs) ( X,Y, ∆ , A ) , where X denotes a set of features, Y represents an observed time, ∆ is an event indicator, and A is a sensitive attribute for the individual. If ∆ = 1 (indicating that the event has occurred), then Y represents a true survival time; otherwise, Y is a censoring time. We denote T as a true (possibly unobserved) survival time associated with X , and C as a true (possibly unobserved) censoring time associated with X . We use lowercase letters x , y , δ , a , t , c to denote the respective realizations of these random variables. It is important to note that for each individual, we do not observe both T and C ; instead, we observe exactly one of them. More precisely, Y = min { T, C }

′

and ∆ = ✶ { T ≤ C } , where ✶ is an indicator function. Let S ( t ′ | x ) = 1 -∫ t 0 P ( t | x ) dt represent the survival function at time t given the feature x . In fair TTE prediction, our goal is to accurately predict S ( t ′ | x ) while adhering to fairness constraints.

Performance Metrics for TTE Prediction. Consider a TTE prediction model h : X → T where X and T are the spaces of x and t , we denote the performance metric for this model on the dataset D = { X i , Y i , ∆ i , A i } | D | i =1 as Er( f, h, D ) where f is the ground-truth labeling function (i.e., f = P ( t | x ) ). Note that this notation is sufficiently flexible to encompass censoring data and different metrics in TTE prediction. Furthermore, while prediction error (smaller values indicate better performance, e.g. , Brier score [16]) is used as the metric in our theoretical analysis, prediction accuracy (larger values indicate better performance, e.g. , concordance index [19]) can also be applied.

Fairness Metrics for TTE Prediction. Various fairness metrics have been proposed recently for TTE prediction, which can be roughly classified into three categories based on their objectives: (i) ensuring similar predicted TTE outcomes for similar data points [36, 53, 77, 76, 72], (ii) ensuring similar predicted outcomes for data points from different groups [36, 53, 80], and (iii) ensuring similar predictive performance across different groups [11, 22, 78, 79]. Notably, some of these metrics are less applicable in the context of medical imaging. For instance, metrics in the first category require a well-defined similarity measure between data points, which is difficult to establish for medical images. Metrics in the second category may also be inappropriate when sensitive attributes are strong risk factors for the TTE outcome. For example, age is often a key predictor of various TTE outcomes, such as mortality. In this case, asking for similar predicted survival times for young and elderly individuals would be nonsensical.

Therefore, in this study, we focus on fairness metrics from the third category, which aim to ensure that the model maintains equal predictive performance across different groups. Specifically, given a performance metric Er for TTE prediction task, we define fairness metric F Er as follow:

where A is the set of groups considered in TTE prediction task, D a = { X i , Y i , ∆ i , A i | A i = a } | D a | i =1 and D a ′ = { X i , Y i , ∆ i , A i | A i = a ′ } | D a ′ | i =1 are the subsets containing data from groups a and a ′ , and f a = P ( t | x, a ) and f a ′ = P ( t | x, a ′ ) are the ground-truth labeling functions for D a and D a ′ , respectively. If the labeling function f is independent of the sensitive attribute, then f = f a = f a ′ . Note that the fairness metric in Eq. (1) is defined as the maximum performance gap between any two groups, and as such, it can be applied to any performance metric used in TTE prediction.

2.2 Causal Structure for Fair TTE Prediction

To conduct a fine-grained analysis to understand fairness in TTE prediction, we leverage the structural causal model (SCM) [48] to represent the data generation process underlying TTE data. Specifically, we examine how the sensitive attribute A influences the TTE prediction model, which aims to approximate the data likelihood P ( t | x ) using the SCM framework. To construct the causal graph, we introduce an unobserved underlying health condition of the patient, denoted as Z . In this context, the patient's feature set X can be viewed as noisy and partially observed information derived from Z .

Following the approach in [25, 69], we partition X into two components: X Z , representing target-related features directly influenced by Z , and X A , representing features related to the sensitive attribute, directly influenced by A . By construction, X A encodes sensitive information as it is predictive of the sensitive attribute A . Using these definitions, we construct causal graphs for TTE data under two settings: unbiased and biased scenarios, illustrated in Figure 2. In the unbiased scenario (Figure 2a), the sensitive attribute A is irrelevant to the TTE outcome. It only influences X A , and not any other variable in the graph. Therefore, capturing the in-

̸

X

A

A

C

X

A

A

C

△

X

X

X

Z

Z

T

Y

XZ)

Z

T

Y

(a) Unbiased Setting

(b)Biased Setting

variant feature X Z across groups is sufficient for learning a fair model. Specifically, we have, P ( t | x z ) = P ( t | x z , a ) , ∀ a ∈ A . In contrast, in the biased scenario (Figure 2b), the sensitive attribute A influences additional variables in the causal graph beyond X A , leading to dependency between A and TTE outcome. This results in the conditional distribution of the outcome given X Z varies across groups P ( t | x z , a ) = P ( t | x z , a ′ ) , a, a ′ ∈ A and/or the distribution of X Z differs across groups P ( x z | a ) = P ( x z | a ′ ) . These discrepancies reflect the presence of bias in the data and challenge the assumption that learning invariant features alone is sufficient for fair TTE prediction. 1

̸

It is important to note that disparities in data distributions across groups, as reflected in the causal graph, do not necessarily indicate that the sensitive information captured during model training is spurious or inappropriate in every context. In some cases, genuine biological differences between groups may exist, making the sensitive attribute relevant for disease prediction. In such situations, incorporating group-specific information can be beneficial, as it allows the model to capture distinct disease mechanisms that reflect true underlying biological variation.

However, observed biases in the training data may arise from spurious correlations that reflect historical inequities in healthcare access, diagnosis, and treatment practices (illustrated by the

X

A

A

△

X

A

A

△

X

U

X

XZ

Z

T

Y

X

Z

I

一

Z

T

Y

(a)Training Dataset

(b)TestingDataset

blue arrows in Figure 3a). When models are trained on such data, they risk encoding and amplifying these biases, thereby perpetuating unfairness in clinical decision-making. From this perspective, fairness and distribution shift are intrinsically connected: achieving fairness requires models that can generalize to a test distribution in which unfair causal pathways are absent and only fair causal pathways remain (Figure 3b) [37, 71, 62]. 2

Group-specific Distribution Disparity and Fairness. As shown in Figure 2b, the effect of A on other RVs can induce distribution disparity across groups. We now formalize how it affect fairness in Theorem 1 below.

Theorem 1. Given a performance metric Er satisfying triangle inequality and symmetry properties, i.e., | Er( h, h ′ , D ) -Er( h, h ′′ , D ) | ≤ Er( h ′ , h ′′ , D ) and Er( h, h ′ , D ) = Er( h ′ , h, D ) , then we have:

with

where H represents the hypothesis class of the TTE prediction model h , η ( H , f a , f a ′ ) denotes the minimum joint prediction error on D a and D a ′ , and D ( H , D a , D a ′ ) is the largest distance between two groups a and a ′ , with respect to the hypothesis class H . Note that the requirements on symmetric and triangle inequality properties are relatively mild and can be met by many performance metrics commonly used in practice (see Appendix B).

Discussion of Theorem 1. The term D ( H , D a , D a ′ ) aligns with the concept of subgroup separability-the ability to predict group membership from image features-in the group fairness literature [30]. Low subgroup separability implies that the image feature distributions are similar across groups, which corresponds to a small value of D ( H , D a , D a ′ ) . Prior work has shown that unfairness tends to be less severe in datasets with low subgroup separability [30, 73]. Our results extend these findings by demonstrating that this relationship also holds in the context of TTE prediction, while further revealing that model fairness is influenced by additional factors beyond subgroup separability. We observe that while it is often possible to reduce D ( H , D a , D a ′ ) during training-for example, by learning fair representations across groups-minimizing the term η ( H , f a , f a ′ ) may remain challenging. In particular, when the labeling functions f a and f a ′ differ significantly across groups (Figure 2b), even the optimal choice of hypothesis class H may not ensure a small η . In such cases, the upper bound on fairness error becomes large, implying that F Er ( h ) may also be large. Conversely, when f a and f a ′ are similar-as in Figure 2a or when the only causal pathway from A is to X in Figure 2b-the upper bound becomes small, indicating that F Er ( h ) can also be small. In these settings, fairness can be achieved by learning invariant representations across groups. We formally state this observation as follows.

Proposition 2. Let g : X → Z be a mapping from the input space X to a representation space Z . Assume the following conditions hold:

(i) Data distribution disparity across groups is a covariate shift, i.e., P ( t | x, a ) = P ( t | x ) , ∀ a ∈ A .

(ii) Representation Z is sufficient for an arbitrary group a ∈ A , i.e., I a ( Z, T ) = I a ( X,T ) , where I a ( · , · ) denotes the mutual information computed over the distribution D a .

Then, distribution shift across groups w.r.t. representation Z is also a covariate shift, that is,

Discussion of Proposition 2. This proposition suggests that under covariate shift ( i ) , fair TTE prediction can be achieved by learning a fair representation Z across groups (i.e., P ( z | a ) = P ( z ) , ∀ a ∈ A ). More clearly, P ( z | a ) = P ( z ) , ∀ a ∈ A guarantees small D ( H , D a , D a ′ ) while P ( t | z, a ) = P ( t | z ) , ∀ a ∈ A indicates small η ( H , f a , f a ′ ) . We note that the sufficiency condition imposed on Z ( ii ) is practical: it only needs to hold for one group, and during training we have access to labeled data. Moreover, the dimension of T is often smaller than that of Z , making the sufficiency assumption more attainable in practice.

To further investigate the influence of sensitive attributes on the fairness of TTE prediction models, we utilize our proposed causal framework to decompose the group-specific labeling functions. Following Bayes' theorem, we rewrite the labeling function f a w.r.t. each group a as:

Note that in practical TTE prediction scenario, we observe Y and ∆ rather than T . Therefore, the TTE prediction model estimates P ( y, δ | x ) instead. Replacing t with y and δ in Eq. (3), we obtain:

̸

where the first and second terms represent the pointwise mutual information (PMI) between X Z and Y (conditioned on ∆ ), and between X Z and ∆ , respectively. The third term corresponds to the TTE distribution, while the fourth term accounts for the censoring rate. As shown in Figure 2b, when A is correlated with other random variables in the causal graph, it alters these four terms, resulting in P ( y, δ | x, a ) = P ( y, δ | x, a ′ ) .

Sources of Bias 3 in Fair TTE Prediction. Based on Theorem 1 and the decomposition formula in Eq. (4), we can identify five primary sources of bias across groups in fair TTE prediction: 1) disparity in image feature distributions, 2) disparity in mutual information between X Z and Y , 3) disparity in mutual information between X Z and ∆ , 4) disparity in the TTE distributions, and 5) disparity in the censoring rates. In practice, we observe that TTE datasets often contain multiple sources of bias rather than just one. These five cases represent the fundamental sources of bias and are crucial for understanding the complex biases present in real-world TTE data.

3 Experimental Setup for Fair TTE Prediction

We propose FairTTE, a reproducible and user-friendly framework for evaluating fairness algorithms in TTE prediction within medical imaging. Our framework includes large-scale experiments conducted on three real-world medical image datasets, covering diverse imaging modalities and TTE outcomes, with up to three sensitive attributes considered for each dataset. Using these datasets, we evaluate three TTE prediction models and five fairness algorithms in the context of fair TTE prediction.

Datasets. FairTTE includes MIMIC-CXR [27] for predicting in-hospital mortality from chest X-ray images, ADNI [49] for predicting Alzheimer's disease from brain MRI images, and AREDS [14] for predicting late AMD from color fundus images. We selected these datasets for our benchmark based on several key criteria: the availability of temporal information to derive TTE outcomes, the diversity of medical imaging modalities, the presence of various potential sources of bias, the availability of sensitive attributes, and the range of dataset sizes. Table A1 provides basic information about these datasets, with additional details on data access and TTE label construction provided in Appendix C.

Algorithms. TTE prediction models: We employ DeepHit [43], Nnet-survival [15], and PMF [40] as the base models for TTE prediction to investigate the fairness issue in medical image prognosis. Fairness algorithms: We incorporate five SOTA fairness algorithms and adapt them to the TTE prediction setting to promote equitable predictions. These algorithms span a wide range of learning strategies and are categorized into three main groups: 1) pre-processing: subgroup rebalancing (SR) [31], 2) in-processing: domain independence (DI) [71], fair representation learning (FRL) [70], distributionally robust optimization (DRO) [22], and post-processing: controlling for sensitive attributes (CSA) [51]. To the best of our knowledge, this is the first study to comprehensively evaluate such a diverse set of fairness algorithms for TTE prediction in medical imaging. Detailed descriptions of each algorithm are provided in Appendix D.

Evaluation Metrics. Compared to classification tasks, evaluating TTE prediction, particularly in medical contexts, requires careful consideration. The primary challenge in assessing accuracy in TTE prediction is censoring , which complicates the use of standard evaluation metrics. As a result, no single evaluation metric is universally ideal for all TTE prediction scenarios.

To address these challenges, we adopt multiple evaluation metrics to ensure a comprehensive assessment of TTE predictive performance. These metrics can be grouped into two categories: 1) ranking-based metrics which evaluate the ranking of patients in the dataset based on their corresponding survival times: time-dependent C-index ( C td ) [1], time-dependent AUC ( AUC td ) [67] and 2) squared error which measures the error between estimated survival times and ground-truth values: Integrated Brier score ( IBS ) [16]. For each performance metric, we consider a corresponding fairness metric. A detailed description of all metrics can be found in Appendix E.

Model Selection. Prioritizing fairness often involves a trade-off with utility, as the model's objective shifts from utility to balancing both utility and fairness. Consequently, model selection becomes critical in determining the optimal trade-off. In our setting, we perform a hyperparameter search

and select the best fair TTE prediction models based on fairness performance on the validation set, allowing for up to a 5% reduction in predictive performance compared to the base TTE models.

Implementation Details. We use a 2D EfficientNet [61] backbone for the AREDS and MIMIC-CXR datasets, and a 3D ResNet-18 backbone [65] for the ADNI dataset. These lightweight backbones are chosen to mitigate overfitting. In addition to training models from scratch, we also explore pre-training using weights from models trained on the ImageNet [10] and Kinetics [35] datasets. To ensure stability across random initializations, we perform a hyperparameter search with 10 random seeds for each combination of dataset, sensitive attribute, algorithm, and evaluation metric. More implementation details can be found in Appendix E.2 and in our code repository. 4

4 Experiment and Result

Bias Across TTE Prediction Models in Diverse Imaging Modalities and Outcomes. We first train TTE prediction models, including DeepHit, Nnet-survival, and PMF, on various datasets and sensitive attributes, then select the best models based on their performance on the validation sets. For each dataset and sensitive attribute pair, we report predictive performance using metrics including C td , AUC td , and IBS , along with fairness as the performance gap between the best and worst groups. As shown in Figure 4, these performance gaps (measured by C td ) are prevalent across all datasets and sensitive attributes. From this figure, we also observe that these gaps are more pronounced for age and race compared to sex across all datasets. While bias in model predictions has been extensively discussed in the context of medical classification and segmentation, as well as TTE prediction for tabular data, it has not been systematically quan-

1.0

Group 0

Group 1

95%CI

0.9

2

0.8

0.7

0.6

Age

Sex

Age

Race

Sex

Age

Race

Sex

ADNI

AREDS

MIMIC-CXR

tified for TTE prediction in medical imaging. This study provides the first comprehensive analysis across a wide range of imaging modalities, TTE outcomes, and sensitive attributes.

Statistical Tests. We conduct statistical tests to verify the robustness of our findings against the variability in TTE prediction models and hyperparameters. Specifically, for each combination of dataset, sensitive attribute, and evaluation metric, we perform two-sided Wilcoxon signed-rank test [6]

on the results across all TTE prediction models and hyperparameters to identify significant differences in predictive performance between groups (p-value < 0 . 05 ). As illustrated in Figure A7, our analysis reveals significant performance disparities between groups across all experimental settings, highlighting the importance of considering fairness in TTE prediction tasks.

Quantifying Sources of Bias. To better understand the unfair behavior of TTE prediction models, we quantify the degree of each source of bias in various datasets and sensitive attribute settings. Specifically, to estimate disparities in PMI between RVs across groups, we compute normalized mutual information scores. To evaluate disparities in image features and TTE outcomes across groups, we calculate the Wasserstein distance [68]. Lastly, disparities in censor-

ADNI

Age

Sex

Age

REDS

S

Race

ARE

Sex

IC-CXR

Age

Race

Sex

0

1

0.0

0.1

0

1

0.0

0.30.0

0.1

Censoring

TTE

Image

PMI(c,y)

PMI(c,8)

ing rates are quantified by examining differences in the ratio of censored data across groups. Detailed methods for quantifying sources of bias can be found in Appendix F.4.

Figure 5 illustrates the degree of bias across these five sources for each dataset and sensitive attribute setting. The results reveal a correlation between the degree of bias and the fairness performance of TTE prediction models. For instance, in the AREDS dataset, the degree of bias in terms of disparities in censoring rates, in PMI between X Z and Y , and in PMI between X Z and ∆ is significantly higher when considering race as the sensitive attribute compared to sex or age. This finding aligns with Figure 4, where the performance gap between racial groups is considerably larger than the gaps between sex or age groups. Additionally, we observe that settings with greater disparities in image features across groups exhibit larger performance gaps. This observation aligns with prior findings in medical image classification tasks, where higher subgroup separability is associated with greater performance disparities [30, 73].

The Role of Pre-Training. Pre-training on large datasets has been proven effective in many applications [2]. In this study, we investigate the impact of pre-training on both the accuracy and fairness of TTE prediction models. As shown in Figure 6, pre-training outperforms training models from scratch in terms of predictive performance. This improvement is particularly notable for the ADNI dataset, which is relatively small and contains more complex data structures (3D images) compared to the AREDS and MIMICCXR datasets (2D images). However, in terms of fairness, we do not observe a significant improvement with pre-training compared to training from scratch, as shown in Figure A9. Specifically, the p-values from one-sided Wilcoxon signed-rank tests are larger than 0.05 in 18 out of 24 settings, suggesting that pre-training does not lead to more equitable predictions in most cases. These results imply that while pre-training enhances accuracy, combining it with fairness algorithms may be necessary to achieve both fairer and more accurate TTE prediction.

0.3

Group0

0.05

Group1

(ADNI)

0.03

(Others)

0.15

△Ctd

△Ctd

0

0

Age

Sex

Age

Race

Sex

Age

Race

Sex

ADNI

AREDS

MIMIC-CXR

Performance of Fairness Algorithms in TTE Prediction. We conduct an experiment to evaluate the effectiveness of fairness algorithms in reducing disparities between groups in TTE prediction. In this experiment, we use DeepHit as the base TTE prediction model and integrate various fairness algorithms to achieve equitable predictions. To ensure a comprehensive evaluation, we select algorithms from pre-processing (i.e., SR), in-processing (i.e., DI, FRL, DRO), and post-processing approaches (i.e., CSA). As shown in Table 1, existing fairness methods reduce performance gaps between groups compared to DeepHit in settings where bias sources are substantial (e.g., ADNI with age, AREDS with race, and MIMIC-CXR with age). However, when bias sources are small, enforcing fairness can sometimes exacerbate performance disparities. Moreover, no method consistently outperforms DeepHit across all settings. To verify the significance of our observations, we perform a Friedman test followed by a Nemenyi post-hoc test [9] across all algorithms, datasets, and sensitive attribute settings. As shown in Figure A10, the results confirm that current fairness algorithms do not significantly outperform DeepHit in mitigating bias. Additionally, we observe trade-offs in predictive performance in most cases where fairness interventions reduce group disparities: while certain sources of bias (e.g., disparities in image features) are mitigated, others often persist. These findings highlight the need for developing new fairness algorithms capable of addressing multiple sources of bias without compromising predictive performance.

Fair TTE Prediction Under Distribution Shift. Fairness algorithms are known to lose effectiveness under distribution shifts, particularly in medical image diagnosis tasks [73, 29]. Building on this insight, we investigate the impact of distribution shifts on fair TTE prediction. As illustrated in Figure 3, we define distribution shifts as scenarios where correlations between the sensitive attribute and other random variables in the causal graph exist in the training dataset but disappear in the testing dataset. To simulate such shifts, we manipulate the data generation process by corrupting images, TTE labels, or censoring indicators for one group while keeping the other group unchanged. The

real-world motivations and detailed descriptions of these causal distribution shifts are provided in Appendices G.2 and F.5 .

Figure 7 compares the predictive performance of models under in-distribution and distribution shift learning scenarios for the MIMIC-CXR dataset, where sex is considered the sensitive attribute. The complete results for all settings are provided in Appendix H.6. Our findings indicate that different types of distribution shifts impact model performance in distinct ways. Specifically, adding noise to TTE labels (shift in Y ) leads to a significant decline in IBS , as the increased label uncertainty makes accurate predictions more challenging. Flipping censoring indicators (shift in ∆ ), on the other hand, severely degrades ranking-based metrics such as C td and AUC td , as it reduces the availability of comparable pairs during training, making it harder for the model to learn an effective ranking function. Additionally, introducing noise to medical images (shift in X ) negatively affects all performance metrics, as degraded image quality limits the model's ability to extract meaningful features. These observations align with our expectations-corrupting TTE labels directly impacts metrics measuring error between predicted and ground-truth TTE, while censoring flips disrupt ranking-based evaluations by impairing the model's ability to differentiate survival times.

。

DI

.

CSA

。

DRO

。

DeepHit

○FRL

·SR

(a) Shift in X

0.760

0.776

。

0.220

。

0.768 -

0.752

.

。

0.210

0.760

0.744

.

0070

.

0.780

0.785

0.750 0.756 0.762

0.200 0.208 0.216

Out-of-distribution

(b)Shift in Y

0.216

0.786

。

0.756

0.208

0.783 -

.

。

0.750

0.200

0.780

.

0.780 0.785

0.750 0.756 0.762

0.200 0.208 0.216

(c) Shift in

0.762

8070

0.785

?

0.756

0.204

0.780 -

：

0.775

0.750

.

0.200

0.780 0.785

0.750 0.756 0.762

0.200 0.208 0.216

AUCtd

Ctd

IBS

In-distribution

Regarding fairness, our observations reveal mixed results. In some cases, distribution shifts negatively impact fairness, while in others, they lead to improvements. Notably, adding noise to the data of one group generally degrades predictive performance on that group. However, we also observe that this noise can similarly affect the other group-sometimes even more severely-thereby reducing the performance gap between groups and unintentionally improving fairness. These findings underscore the complexities of achieving fairness in the presence of distribution shifts and highlight the need for developing TTE algorithms that are both fair and robust in such challenging scenarios.

5 Limitation and Broader Impact

While FairTTE represents a significant step forward in advancing fairness research in medical image prognosis by providing the first comprehensive benchmark across multiple datasets and fairness algorithms, we acknowledge several limitations in the current study. Specifically, our analysis focuses on the standard TTE prediction setting, which assumes a single clinical risk and non-informative right censoring. In real-world clinical applications, however, patients often face multiple competing risks and may drop out of studies for reasons that introduce informative censoring, making the fairness landscape considerably more complex. Addressing fairness in such settings remains an important direction for future research. Additionally, although we adopt group fairness definitions (i.e., minimizing performance gaps across subgroups), we recognize that strictly enforcing these criteria can, in some clinical scenarios, degrade overall utility or harm performance for all groups. In such scenarios, alternative fairness notions may be more appropriate, depending on the clinical context and ethical objectives.

Selecting an appropriate fairness criterion in the medical domain requires careful consideration of the clinical context, ethical principles, and statistical validity. An effective fairness notion should align with the model's intended use, its potential impact on different patient subgroups, and real-world constraints. In our study, we focus on statistical group fairness (i.e., predictive performance gaps across subgroups) which is widely examined in the medical image analysis literature. This choice is grounded in the assumption that any causal pathway from sensitive attributes represents an unfair influence in the causal graph and should be mitigated through fairness constraints. However, we acknowledge that in practical clinical scenarios, such causal pathways may reflect fair and clinically meaningful relationships. Enforcing group fairness in these cases may inadvertently remove relevant information and degrade predictive performance.

Moving forward, we suggest several directions for fairness research in medical image analysis: (1) Identifying the causal nature of bias to distinguish between fair and unfair sources, enabling models to address specific pathways appropriately; (2) Developing fairness metrics and mitigation strategies that preserve clinically relevant (fair) pathways while minimizing the effect of unfair ones; and (3) Collaborating with clinicians and domain experts to define context-specific fairness objectives that are aligned with both clinical utility and ethical standards.

6 Conclusion

We systematically investigate fair TTE prediction in medical imaging, introducing a unified framework to define and quantify bias sources and establishing a comprehensive and realistic evaluation. Our framework includes three TTE prediction methods, five fairness algorithms, and three large-scale public datasets spanning diverse modalities and outcomes. Extensive experiments reveal critical insights, including the link between bias sources and model unfairness, the inconsistent effectiveness of existing fairness methods, the role of pre-training strategies, and the challenges of causal distribution shifts. We hope our fine-grained analysis encourages more rigorous evaluations and drives the development of new fairness algorithms for TTE prediction.

References

Laura Antolini, Patrizia Boracchi, and Elia Biganzoli. A time-dependent discrimination index for survival data. Statistics in medicine , 24(24):3927-3944, 2005.

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021.

Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sliced and radon wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision , 51:22-45, 2015.

Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney. Optimized pre-processing for discrimination prevention. Advances in neural information processing systems , 30, 2017.

George H Chen. An introduction to deep survival analysis models for predicting time-to-event outcomes. arXiv preprint arXiv:2410.01086 , 2024.

William Jay Conover. Practical nonparametric statistics , volume 350. john wiley & sons, 1999.

Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining , pages 797-806, 2017.

Matthew D Davis, Ronald E Gangnon, Li Yin Lee, Larry D Hubbard, BE Klein, Ronald Klein, Frederick L Ferris, Susan B Bressler, Roy C Milton, et al. The age-related eye disease study severity scale for age-related macular degeneration: Areds report no. 17. Archives of ophthalmology (Chicago, Ill.: 1960) , 123(11):1484-1498, 2005.

Janez Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research , 7:1-30, 2006.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Hyungrok Do, Yuxin Chang, Yoon Sang Cho, Padhraic Smyth, and Judy Zhong. Fair survival time prediction via mutual information minimization. In Machine Learning for Healthcare Conference , pages 128-149. PMLR, 2023.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2024.

Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference , pages 214-226, 2012.

Frederick L Ferris, Matthew D Davis, Traci E Clemons, Li-Yin Lee, Emily Y Chew, Anne S Lindblad, Roy C Milton, Susan B Bressler, Ronald Klein, et al. A simplified severity scale for age-related macular degeneration: Areds report no. 18. Archives of ophthalmology (Chicago, Ill.: 1960) , 123(11):1570-1574, 2005.

Michael F Gensheimer and Balasubramanian Narasimhan. A scalable discrete-time survival model for neural networks. PeerJ , 7:e6257, 2019.

Erika Graf, Claudia Schmoor, Willi Sauerbrei, and Martin Schumacher. Assessment and comparison of prognostic classification schemes for survival data. Statistics in medicine , 18(17-18):2529-2545, 1999.

Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, and Omar Badri. Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1820-1828, 2021.

Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems , 29, 2016.

Frank E Harrell, Robert M Califf, David B Pryor, Kerry L Lee, and Robert A Rosati. Evaluating the yield of medical tests. Jama , 247(18):2543-2546, 1982.

Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning , pages 1929-1938. PMLR, 2018.

Gregory Holste, Mingquan Lin, Ruiwen Zhou, Fei Wang, Lei Liu, Qi Yan, Sarah H Van Tassel, Kyle Kovacs, Emily Y Chew, Zhiyong Lu, et al. Harnessing the power of longitudinal medical imaging for eye disease prognosis using transformer-based sequence modeling. NPJ Digital Medicine , 7(1):216, 2024.

Shu Hu and George H Chen. Fairness in survival analysis with distributionally robust optimization. Journal of Machine Learning Research , 25(246):1-85, 2024.

Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 590-597, 2019.

Ray Jiang, Aldo Pacchiano, Tom Stepleton, Heinrich Jiang, and Silvia Chiappa. Wasserstein fair classification. In Uncertainty in artificial intelligence , pages 862-872. PMLR, 2020.

Yibo Jiang and Victor Veitch. Invariant and transportable representations for anti-causal domain shifts. Advances in Neural Information Processing Systems , 35:20782-20794, 2022.

Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, a freely accessible electronic health record dataset. Scientific data , 10(1):1, 2023.

Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chihying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042 , 2019.

Charles Jones, Daniel C. Castro, Fabio De Sousa Ribeiro, Ozan Oktay, Melissa McCradden, and Ben Glocker. A causal perspective on dataset bias in machine learning for medical imaging. Nature Machine Intelligence , 6:138-146, 2024.

Charles Jones, Fabio de Sousa Ribeiro, Mélanie Roschewitz, Daniel C Castro, and Ben Glocker. Rethinking fair representation learning for performance-sensitive tasks. In The Thirteenth International Conference on Learning Representations , 2025.

Charles Jones, Mélanie Roschewitz, and Ben Glocker. The role of subgroup separability in group-fair medical image classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 179-188. Springer, 2023.

Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrimination. Knowledge and information systems , 33(1):1-33, 2012.

Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairness-aware classifier with prejudice remover regularizer. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 23 , pages 35-50. Springer, 2012.

Edward L Kaplan and Paul Meier. Nonparametric estimation from incomplete observations. Journal of the American statistical association , 53(282):457-481, 1958.

Jared L Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger. Deepsurv: personalized treatment recommender system using a cox proportional hazards deep neural network. BMC medical research methodology , 18:1-12, 2018.

Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 , 2017.

Kamrun Naher Keya, Rashidul Islam, Shimei Pan, Ian Stockwell, and James Foulds. Equitable allocation of healthcare resources with fair survival models. In Proceedings of the 2021 siam international conference on data mining (sdm) , pages 190-198. SIAM, 2021.

Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn: Training deep neural networks with biased data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9012-9020, 2019.

Newton M Kinyanjui, Timothy Odonga, Celia Cintas, Noel CF Codella, Rameswar Panda, Prasanna Sattigeri, and Kush R Varshney. Fairness of classifiers across skin tones in dermatology. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 320-329. Springer, 2020.

Oleksandr Kovalyk, Juan Morales-Sánchez, Rafael Verdú-Monedero, Inmaculada SellésNavarro, Ana Palazón-Cabanes, and José-Luis Sancho-Gómez. Papila: Dataset with fundus images and clinical data of both eyes of the same patient for glaucoma assessment. Scientific Data , 9(1):291, 2022.

Håvard Kvamme and Ørnulf Borgan. Continuous and discrete-time survival prediction with neural networks. Lifetime data analysis , 27(4):710-736, 2021.

Håvard Kvamme, Ørnulf Borgan, and Ida Scheel. Time-to-event prediction with neural networks and cox regression. Journal of machine learning research , 20(129):1-30, 2019.

Agostina J Larrazabal, Nicolás Nieto, Victoria Peterson, Diego H Milone, and Enzo Ferrante. Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proceedings of the National Academy of Sciences , 117(23):12592-12594, 2020.

Changhee Lee, William Zame, Jinsung Yoon, and Mihaela Van Der Schaar. Deephit: A deep learning approach to survival analysis with competing risks. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Yan Luo, Yu Tian, Min Shi, Louis R Pasquale, Lucy Q Shen, Nazlee Zebardast, Tobias Elze, and Mengyu Wang. Harvard glaucoma fairness: a retinal nerve disease dataset for fairness learning and fair identity normalization. IEEE Transactions on Medical Imaging , 2024.

David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. In International Conference on Machine Learning , pages 3384-3393. PMLR, 2018.

Trishan Panch, Heather Mattie, and Rifat Atun. Artificial intelligence and algorithmic bias: implications for health systems. Journal of global health , 9(2), 2019.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CambridgeUniversityPress , 19(2):3, 2000.

Ronald Carl Petersen, Paul S Aisen, Laurel A Beckett, Michael C Donohue, Anthony Collins Gamst, Danielle J Harvey, CR Jack Jr, William J Jagust, Leslie M Shaw, Arthur W Toga, et al. Alzheimer's disease neuroimaging initiative (adni) clinical characterization. Neurology , 74(3):201-209, 2010.

Sebastian Pölsterl. scikit-survival: A library for time-to-event analysis built on top of scikit-learn. Journal of Machine Learning Research , 21(212):1-6, 2020.

Devin G Pope and Justin R Sydnor. Implementing anti-discrimination policies in statistical profiling models. American Economic Journal: Economic Policy , 3(3):206-231, 2011.

Shi-ang Qi, Weijie Sun, and Russell Greiner. SurvivalEVAL: A comprehensive open-source python package for evaluating individual survival distributions. Proceedings of the AAAI Symposium Series , 2:453-457, 01 2024.

Md Mahmudur Rahman and Sanjay Purushotham. Fair and interpretable models for survival analysis. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1452-1462, 2022.

María Agustina Ricci Lara, Rodrigo Echeveste, and Enzo Ferrante. Addressing fairness in artificial intelligence for medical imaging. nature communications , 13(1):4581, 2022.

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations , 2019.

Laleh Seyyed-Kalantari, Guanxiong Liu, Matthew McDermott, Irene Y Chen, and Marzyeh Ghassemi. Chexclusion: Fairness gaps in deep chest x-ray classifiers. In BIOCOMPUTING 2021: proceedings of the Pacific symposium , pages 232-243. World Scientific, 2020.

Laleh Seyyed-Kalantari, Haoran Zhang, Matthew BA McDermott, Irene Y Chen, and Marzyeh Ghassemi. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature medicine , 27(12):2176-2182, 2021.

Ammar Shaker and Carolin Lawrence. Multi-source survival domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 9752-9762, 2023.

Saeed Sharifi-Malvajerdi, Michael Kearns, and Aaron Roth. Average individual fairness: Algorithms, generalization and experiments. Advances in neural information processing systems , 32, 2019.

Changjian Shui, Gezheng Xu, Qi Chen, Jiaqi Li, Charles X Ling, Tal Arbel, Boyu Wang, and Christian Gagné. On learning fairness and accuracy on multiple subgroups. Advances in Neural Information Processing Systems , 35:34121-34135, 2022.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR, 2019.

Enzo Tartaglione, Carlo Alberto Barbano, and Marco Grangetto. End: Entangling and disentangling deep representations for bias correction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13508-13517, 2021.

Matthew Thorpe. Introduction to optimal transport. Notes of Course at University of Cambridge , 2018.

Yu Tian, Min Shi, Yan Luo, Ava Kouhana, Tobias Elze, and Mengyu Wang. Fairseg: A large-scale medical image segmentation dataset for fairness learning using segment anything model with fair error-bound scaling. In The Twelfth International Conference on Learning Representations , 2023.

Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages 6450-6459, 2018.

Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data , 5(1):1-9, 2018.

Hajime Uno, Tianxi Cai, Lu Tian, and Lee-Jen Wei. Evaluating prediction rules for t-year survivors with censored regression models. Journal of the American Statistical Association , 102(478):527-537, 2007.

Cédric Villani and Cédric Villani. The wasserstein distances. Optimal transport: old and new , pages 93-111, 2009.

Tan Wang, Zhongqi Yue, Jianqiang Huang, Qianru Sun, and Hanwang Zhang. Self-supervised learning disentangled group representation as feature. Advances in Neural Information Processing Systems , 34:18225-18240, 2021.

Yuyan Wang, Xuezhi Wang, Alex Beutel, Flavien Prost, Jilin Chen, and Ed H Chi. Understanding and improving fairness-accuracy trade-offs in multi-task learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 1748-1757, 2021.

Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness in visual recognition: Effective strategies for bias mitigation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8919-8928, 2020.

Zichong Wang, Jocelyn Dzuong, Xiaoyong Yuan, Zhong Chen, Yanzhao Wu, Xin Yao, and Wenbin Zhang. Individual fairness with group awareness under uncertainty. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 89-106. Springer, 2024.

Yuzhe Yang, Haoran Zhang, Judy W Gichoya, Dina Katabi, and Marzyeh Ghassemi. The limits of fair medical imaging ai in real-world generalization. Nature Medicine , 30(10):2838-2848, 2024.

Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. Fairness constraints: Mechanisms for fair classification. In Artificial intelligence and statistics , pages 962-970. PMLR, 2017.

Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society , pages 335-340, 2018.

Wenbin Zhang, Tina Hernandez-Boussard, and Jeremy Weiss. Censored fairness through awareness. In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 14611-14619, 2023.

Wenbin Zhang, Zichong Wang, Juyong Kim, Cheng Cheng, Thomas Oommen, Pradeep Ravikumar, and Jeremy Weiss. Individual fairness under uncertainty. In ECAI 2023 , pages 3042-3049. IOS Press, 2023.

Wenbin Zhang and Jeremy C Weiss. Fair decision-making under uncertainty. In 2021 IEEE international conference on data mining (ICDM) , pages 886-895. IEEE, 2021.

Wenbin Zhang and Jeremy C Weiss. Longitudinal fairness with censorship. In proceedings of the AAAI conference on artificial intelligence , volume 36, pages 12235-12243, 2022.

Zhouting Zhao and Tin Lok James Ng. Fairness-aware processing techniques in survival analysis: Promoting equitable predictions. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 460-476. Springer, 2023.

Yuyin Zhou, Shih-Cheng Huang, Jason Alan Fries, Alaa Youssef, Timothy J Amrhein, Marcello Chang, Imon Banerjee, Daniel Rubin, Lei Xing, Nigam Shah, et al. Radfusion: Benchmarking performance and fairness for multimodal pulmonary embolism detection from ct and ehr. arXiv preprint arXiv:2111.11665 , 2021.

Yongshuo Zong, Yongxin Yang, and Timothy M Hospedales. Medfair: Benchmarking fairness for medical imaging. In The Eleventh International Conference on Learning Representations , 2023.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims presented in the abstract and introduction accurately reflect the paper's contributions and scope. They are supported by both theoretical results-formally stated and proven-and comprehensive experimental evaluations across multiple datasets, metrics, and fairness algorithms. We clearly distinguish between established findings and broader motivations, and we explicitly state assumptions and limitations to ensure that the claims are appropriately contextualized and not overstated.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We explicitly discuss the assumptions made in our theoretical framework-such as triangle inequality and symmetric properties for evaluation metric-and their potential violations in real-world scenarios. We also acknowledge limitations in generalizability, as our experiments are conducted on a finite set of publicly available medical imaging datasets. These limitations help contextualize our contributions and inform directions for future research.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions for Theorem 1 and Proposition 2 are explicitly stated in Section 2.2, and complete, formal proofs are provided in Appendix B. All theoretical results are clearly numbered, cross-referenced, and presented with the necessary assumptions to ensure clarity and correctness. Where appropriate, we also include intuitive explanations in the main text to aid understanding.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experimental design and implementation details necessary to reproduce the main results are thoroughly documented in Section 3 and Appendix F. This includes dataset descriptions, preprocessing steps, model architectures, training protocols, hyperparameters, and evaluation metrics. We also describe our experimental setup across all dataset and sensitive attribute combinations. These details are sufficient for reproducing our findings, even without access to the code or data at submission time. Code and documentation will be released publicly to further facilitate reproducibility upon publication.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed

instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide open access to all code, documentation, and scripts necessary to reproduce our main experimental results. Detailed instructions for setting up the environment, accessing and preprocessing the datasets (including MIMIC-CXR, ADNI, and AREDS), and running the experiments are included in Appendix F and will be made publicly available upon publication. For datasets with access restrictions, we provide clear guidance on how to obtain access through the appropriate data use agreements in Appendix C. Our released code includes scripts for training, evaluation, and statistical analysis, along with configuration files for all reported experiments to ensure faithful reproduction of our results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental settings, including data splits, hyperparameter configurations, optimization strategies, and model selection procedures, are comprehensively detailed in Appendix F. These include descriptions of how hyperparameters were tuned, the choice of optimizer, training schedules, and evaluation protocols. This information is sufficient to understand and contextualize the reported results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report error bars using bootstrapping across multiple runs to capture variability due to model initialization, train/validation splits, and random seeds. Specifically, we calculate 95% confidence intervals based on bootstrapped samples to quantify uncertainty in model performance. Additionally, we assess statistical significance using well-established non-parametric tests, including the Wilcoxon signed-rank test for pairwise comparisons, and the Friedman test followed by the Nemenyi post-hoc test for comparing multiple methods across datasets and sensitive attributes.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Details on compute resources used for all experiments are provided in Appendix F.1. This includes the type of hardware, memory specifications, and estimated runtime for each experimental run. This information is sufficient to support reproducibility and assess the computational requirements of our work.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have thoroughly reviewed the NeurIPS Code of Ethics and confirm that our research adheres to all applicable guidelines. Our study involves no human subjects, privacy risks, or potentially harmful applications, and we have taken appropriate steps to ensure transparency, reproducibility, and responsible use of data and models.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents foundational research on the methodological evaluation of fairness in TTE prediction using medical imaging. The work is primarily intended to inform academic research and improve scientific understanding of bias in prognostic modeling. It does not introduce new systems intended for deployment, nor does it involve the creation or use of data or models with direct societal impact. As such, we consider the broader societal implications beyond the scope of this work and have not included a dedicated discussion.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All external assets used in this paper-including datasets (MIMIC-CXR, ADNI, and AREDS) and pretrained models (e.g., RETFound and EfficientNet)-are properly cited with their corresponding publications. We use publicly available versions of these datasets and models in accordance with their respective licenses and terms of use. We include dataset description and access URLs in the Appendix C.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We release new assets as part of this work, including benchmark code, evaluation scripts, and structured documentation for fair TTE prediction across multiple medical imaging datasets. These assets are well-documented and include detailed instructions for dataset preprocessing, model training, evaluation, and reproduction of results. All assets are anonymized for the review process and will be made publicly available upon publication. No personally identifiable information or private data is included, and all released components comply with the data use agreements of the underlying datasets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This study does not involve crowdsourcing or research with human subjects; therefore, IRB approval is not required.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard component; therefore, a declaration of LLM usage is not applicable.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Appendix Contents

A Related Works

TTEPrediction. TTE prediction models can generally be classified into two categories: continuoustime and discrete-time models, each with distinct approaches for handling event timing. Continuoustime models treat time as a continuous variable and often extend traditional models like Cox regression. For example, DeepSurv [34] extends the Cox regression by using a deep neural network with non-linear activation functions in hidden layers. Cox-Time [41] further builds on DeepSurv, introducing time-dependent predictors that allow for the estimation of time-varying effects. In contrast, discrete-time models treat time as a series of distinct intervals and typically use classification techniques. DeepHit [43] learns survival times directly without assuming a specific underlying stochastic process, parameterizing the discrete probability mass function. Another method, Nnetsurvival [15], parametrizes the discrete hazard function using a neural network and optimizes the negative log-likelihood loss.

Fairness in Machine Learning. Fairness in machine learning has gained significant attention in recent years, with a focus on ensuring models are unbiased and equitable across individuals and groups. Fairness metrics. Fairness metrics can be broadly categorized into two types: group fairness [13, 7, 74, 18] and individual fairness [13, 59]. Group fairness ensures that models are fair across different demographic groups, while individual fairness emphasizes that similar individuals should be treated similarly. Fairness algorithms . To address fairness and bias issues, bias mitigation methods are generally classified into three approaches: pre-processing, which focuses on modifying the input data before model training [31, 4]; in-processing, which incorporates fairness constraints during model training [32, 75, 45, 60]; and post-processing, which adjusts model outputs to improve fairness [18, 24].

Fairness in Medical Imaging. In medical image analysis, machine learning (ML) models have been shown to exhibit systematic biases related to various attributes such as race, gender, and age [57, 42]. These biases are prevalent across different medical imaging modalities, including chest X-rays [56], CT scans [81], and skin dermatology images [38]. While several efforts have been made to benchmark fairness algorithms on medical images, existing datasets [23, 39, 66, 17] and benchmarks [82, 64] primarily focus on diagnostic tasks like image classification and segmentation. Unfortunately, they often overlook the crucial domain of medical prognosis, which involves predicting TTE outcomes.

Fairness in TTE Prediction. Despite significant advances in TTE prediction, research on fairness in this area remains limited. Fairness metrics for TTE prediction. Fairness metrics for TTE prediction have only recently been defined. These metrics can be roughly classified into three categories based on their objectives: (i) ensuring similar predicted TTE outcomes for similar data points [36, 53, 77, 76, 72], (ii) ensuring similar predicted outcomes for data points from different groups [36, 53, 80], and (iii) ensuring similar predictive performance across different groups [11, 22, 78, 79]. Fairness algorithms for TTE prediction. Building on these metrics, several methods have been proposed to achieve fair TTE prediction. One approach incorporates fairness as a regularization term during model training [36, 53, 11], ensuring that the model accounts for fairness constraints throughout its optimization process. Another approach focuses on improving worst-group accuracy by leveraging distributionally robust optimization techniques [22, 20, 55], which aim to enhance performance for underrepresented or disadvantaged groups. In addition to these in-processing methods, recent work has also explored pre- and post-processing strategies to address fairness in TTE prediction [80]. However, these efforts are limited to tabular data and fail to consider medical images, which are essential and pervasive in medical prognosis tasks.

B Missing Proof

Proof for Theorem 1 For any a, a ′ ∈ A , we have:

where h ∗ = arg min h ′ ∈H (Er( f a , h ′ , D a ) + Er( f a ′ , h ′ , D a ′ )) . We have (1) ≤ by using inequality | a + b | ≤ | a | + | b | ; (2) ≤ by using triangle inequality for Er metric and | Er( h, h ∗ , D a ) -Er( h, h ∗ , D a ′ ) | ≤ max h ′ ,h ′′ ∈H | Er( h ′ , h ′′ , D a ) -Er( h ′ , h ′′ , D a ′ ) | = D ( H , D a , D a ′ ) ; (3) = because η ( H , f a , f a ′ ) = (Er( f a , h ∗ , D a ) + Er( f a ′ , h ∗ , D a ′ )) by definition. Subtracting Er( f a , h, D a ) from both sides and taking max operator, we have:

Discussion on the assumption of performance metric. The proof of Theorem 1 relies on the assumption that the performance metric Er satisfies the properties of triangle inequality and symmetry. This assumption is relatively mild and holds for many commonly used performance metrics. For instance, [58] introduced the symmetric discordance index (SDI), a ranking-based metric that adheres to these properties, demonstrating the practical applicability of this assumption in practice.

Proof of Proposition 2 For a ∈ A with I a ( Z, T ) = I a ( X,T ) , we have:

We have (1) ≥ by using Jensen's inequality. ∀ a ′ ∈ A , taking expectation w.r.t. P ( x, t | a ′ ) over both sides, we have:

We have (1) ≤ because log P ( t | x, a ) -E P ( z | x ) [log P ( t | z, a )] ≥ 0 according to Eq. (5); (2) = because I a ( T, Z ) = I a ( T, X ) . Based on Eq. (6), we have:

We also have:

We have (1) ≤ by using data processing inequality. Finally, we have:

We have (1) = because the shift between two domains w.r.t. input space X is covariate shift; (2) = by using Eq. (7) and Eq. (8) and the fact that KL-divergence is non-negative. Note that Eq. (9) implies that the shift between these two domains w.r.t. representation space Z is also covariate shift (i.e., P ( t | z, a ) = P ( t | z, a ′ ) = P ( t | z ) , ∀ a, a ′ ∈ A ).

C Dataset Details

C.1 AREDS

C.1.1 Dataset Description

The Age-Related Eye Disease Study (AREDS) [14] was a clinical trial conducted between 1992 and 2001 across 11 retinal specialty clinics in the United States. The primary objective was to study the risk factors for age-related macular degeneration (AMD) and the impact of dietary supplements on AMDprogression. The study followed 4,757 participants, aged 55-80 at enrollment, for a median of 6.5 years. Participants were selected with a broad range of AMD severity, from no AMD to late-stage AMD in one eye. At each visit, certified technicians captured color fundus photography images using a standardized imaging protocol, although adherence to the protocol varied, leading to visits at irregular intervals. AMD severity scores were determined by expert graders at the University of Wisconsin Fundus Photograph Reading Center, with late AMD defined as the presence of neovascular AMD or atrophic AMD with geographic atrophy (severity scores from 10 to 12 using severity scale [8]). For this study, we include images from both the left and right eyes of each participant and make predictions separately for each eye. In cases where multiple images were captured per eye during a visit, we select only one image for analysis. Demographic information, including age, sex, and race, is available in the dataset and is used as sensitive attributes in our study.

C.1.2 TTE Outcome Construction

In our study, TTE outcomes for fundus images are defined as the duration, in years, from the date an image was captured to the first recorded diagnosis of late AMD in the corresponding eye. For eyes without a recorded diagnosis of late AMD, the censoring dates are set to the time of the last imaging visit. To ensure that the model forecasts the future risk of developing late AMD, all images taken during the final visit for a given eye were excluded from the dataset, as this visit was used solely to determine the TTE outcome. By removing the final visit, we ensured that no images were included with late AMD already present at the time of acquisition. This process resulted in a final dataset of 129,708 fundus images, each paired with corresponding TTE information, enabling a robust analysis of the TTE prediction for AMD progression.

C.1.3 Data Access

AREDS is a publicly available dataset hosted in the National Center for Biotechnology Information (NCBI) database of Genotypes and Phenotypes (dbGAP) through controlled access. Researchers can request access to this dataset at dbGAP. Once the application is approved, researchers can access data

at the following address: https://www.ncbi.nlm.nih.gov/projects/gap/cgi-bin/study. cgi?study_id=phs000001.v3.p1

C.2 MIMIC-CXR

C.2.1 Dataset Description

The MIMIC-CXR dataset [27] is a comprehensive, publicly available collection of chest X-ray images, along with associated clinical data, from the larger MIMIC-IV [26] database. It includes over 370,000 chest X-ray images from more than 65,000 patients, annotated with both structured and unstructured clinical information, such as patient demographics, diagnoses, and other relevant clinical details. For our study, we utilize the MIMIC-CXR-JPG [27], a processed version of the MIMIC-CXR dataset, which provides images in JPG format derived from the original DICOM files. Additionally, we link this dataset to the broader MIMIC database to access patient demographic information, including age, sex, and race, which we incorporate as sensitive attributes in our fairness benchmarking framework.

C.2.2 TTE outcome construction

To construct TTE outcomes for chest X-ray images, we extract in-hospital mortality events from the MIMIC-IV database. For patients without a recorded date of mortality, the censoring dates are set as 1 year after their last recorded discharge date. We exclude any images that do not have a matching record in the MIMIC-IV patient table, images taken after the latest discharge date, and images taken after a recorded date of mortality. TTE is then calculated as the number of days from the image study date to either the date of mortality or the censoring date. This process results in a final dataset of 269,360 chest X-ray images, each paired with corresponding TTE information, enabling a robust analysis of the TTE prediction for in-hospital mortality.

C.2.3 Data Access

MIMIC-CXR, MIMIC-CXR-JPG, and MIMIC-IV are a publicly available datasets hosted by PhysioNet, which is a platform providing access to medical data. To access the dataset, researchers first need to create an account on PhysioNet and then complete the required training (CITI training). Once researchers have completed the CITI training, they will need to request access to the dataset at the following address:

MIMIC-IV: https://physionet.org/content/mimiciv/3.1/

MIMIC-CXR-JPG: https://physionet.org/content/mimic-cxr-jpg/2.1.0/

C.3 ADNI

C.3.1 Dataset Description

The Alzheimer's Disease Neuroimaging Initiative (ADNI) [49] is a large, longitudinal study aimed at identifying biomarkers for Alzheimer's disease (AD) and tracking the progression of the disease over time. Launched in 2004, ADNI is one of the most comprehensive datasets for studying Alzheimer's disease and other neurodegenerative disorders. It includes a wide range of data types, including clinical assessments, neuroimaging (MRI, PET), genetic data, and fluid biomarkers (e.g., cerebrospinal fluid and blood samples) from over 1,700 participants. These participants are categorized into different diagnostic groups, including cognitively normal individuals, those with mild cognitive impairment (MCI), and individuals with Alzheimer's disease. For our study, we focus on neuroimaging data, specifically MRI scans, and consider age and sex as sensitive attributes in our analysis.

C.3.2 TTE outcome construction

In our study, TTE outcomes for brain MRI scans are defined as the duration, measured in 6-month intervals, from the date an MRI scan was captured to the first recorded diagnosis of Alzheimer's disease. For participants without a recorded diagnosis of Alzheimer's disease, censoring dates are set to the time of their last imaging visit. To ensure that the model predicts the future risk of developing Alzheimer's disease, we excluded all MRI scans taken during the final visit, as this visit was used

solely for determining the TTE outcome. By removing the final visit, we ensured that no MRI scans were included for participants who had already been diagnosed with Alzheimer's disease at the time of acquisition. This process resulted in a final dataset of 2,227 brain MRI scans, each paired with corresponding TTE information, enabling a comprehensive analysis of TTE prediction for Alzheimer's disease progression.

C.3.3 Data Access

ADNI is a publicly available dataset hosted in the Image and Data Archive (IDA), a secure online resource for archiving, exploring and sharing neuroscience data. Access to the ADNI dataset requires that researchers register for an IDA account. Once the account is created and the ADNI Data Use Agreement is completed, they can access data at the following address: https://adni.loni.usc. edu

D Algorithms Details

D.1 TTE Prediction Models

D.1.1 PMF

PMF [40] is a discrete-time model designed for TTE prediction tasks. It represents the survival time PMF P ( t | x ) using a neural network f ( · ; θ ) : X → [0 , 1] L where θ denotes the model parameters and L represents the number of time intervals. The model parameters θ are estimated by minimizing the negative log-likelihood, averaged over the training data, as follows:

where n is the number of training samples, κ ( y i ) denotes the specific time interval (from 1 , 2 , · · · , L ) that corresponds to time y i , and f m represents the predicted probability that the TTE falls within the m -th interval.

D.1.2 DeepHit

DeepHit [43] extends the PMF model by incorporating a ranking loss function alongside the negative log-likelihood. Given a comparable set defined as E := { ( i, j ) ∈ [ n ] × [ n ] : δ i = 1 , y i < y j } , this ranking loss is calculated over the training data, as follows:

where n is the number of training samples, κ ( y i ) denotes the specific time interval (from 1 , 2 , · · · , L ) that corresponds to time y i , and f m represents the predicted probability that the TTE falls within the m -th interval.

D.1.3 Nnet-survival

Nnet-survival [15] is another discrete-time model designed for TTE prediction tasks. It represents the hazard function h ( ·| x ) using a neural network f ( · ; θ ) : X → R L where θ denotes the model parameters and L represents the number of time intervals. Specifically, Nnet-survival sets the hazard function equal to:

where f m ( x ; θ ) is the m -th output of the neural network. The model parameters θ are estimated by minimizing the negative log-likelihood, averaged over the training data, as follows:

where n is the number of training samples and κ ( y i ) denotes the specific time interval (from 1 , 2 , · · · , L ) that corresponds to time y i .

D.2 Fairness Algorithms

D.2.1 Distributional Group Optimization

Distributional Group Optimization [55, 22] is an optimization technique designed to enhance model robustness by minimizing the worst-case training loss across different groups. Instead of optimizing for the average performance, GroupDRO focuses on the group with the highest loss, ensuring that the model does not disproportionately underperform on any particular group. By incorporating increased regularization, this approach helps mitigate disparities in model performance across diverse subpopulations, making it particularly useful in settings where fairness and reliability across groups are critical.

D.2.2 Subgroup Rebalancing

This resampling method [31] addresses class imbalance by upsampling minority groups, ensuring that all groups have equal representation during training. By increasing the frequency of underrepresented samples, the model is exposed to a more balanced dataset, reducing bias and improving fairness. This approach helps prevent the model from being overly influenced by the majority group, leading to more equitable predictions across all groups.

D.2.3 Fair Representation Learning

Acommon approach to promoting fairness in machine learning is through fair representation learning, which aims to obfuscate sensitive group membership information in the learned representations. By ensuring that the model's latent features do not encode discriminatory patterns, this method helps mitigate bias in downstream predictions. Following [70], we incorporate fairness constraints into representation learning by leveraging kernel-based distribution matching via Maximum Mean Discrepancy. This technique enforces similarity in feature distributions across different groups, reducing disparities while preserving task-relevant information.

D.2.4 Domain Independence

The Domain Independence [71] is a method that trains separate classifiers for different groups while utilizing a shared encoder. This approach allows the model to capture group-specific patterns through distinct classifiers while maintaining a common feature representation across all groups. By leveraging a shared encoder, DomainInd enhances generalization and reduces the risk of overfitting to individual groups, ultimately improving the model's robustness and fairness in diverse domains.

D.2.5 Controlled for Sensitive Attribute

This approach [80, 51], similar to Domain Independence, involves training separate models for each group based on the values of a sensitive attribute. By doing so, the method captures group-specific patterns while maintaining model flexibility. During the prediction phase, the outputs of the fitted models are averaged across all groups in the population. This averaging process ensures that no single group disproportionately influences the predictions, promoting fairness and reducing bias in the model's outcomes.

D.3 Model Architecture Details

Each (fair) TTE prediction model comprises two key networks: an image encoder and a classifier. The image encoder transforms images into representation vectors, while the classifier predicts survival time intervals based on these learned representations. In our study, we employ a 2D EfficientNet [61] backbone as the image encoder for the AREDS and MIMIC-CXR datasets, and a 3D ResNet-18 backbone [65] for the ADNI dataset. To adapt these models for our task, we replace the original fully connected layers with new layers that map images into the representation space. The architectural details of our 2D and 3D image encoders, along with the classifier, are presented in Table A2.

Table A2: Architecture details of (fair) TTE prediction models. In our experiments, we set n_channel = 3 for both 2D and 3D images by duplicating grayscale chest X-ray and brain MRI images to obtain three-channel inputs. feature_dim is set to 64, and hidden_dim is set to 16. n_class is determined using the 'equidistant' discretization method, with values of 14 for AREDS, 28 for ADNI, and 128 for MIMIC-CXR.

E Evaluation Metrics

E.1 Performance Metrics

The performance metrics used in our study are defined based on [5].

E.1.1 Time-dependent concordance index

Harrell's concordance index (C-index) is one of the most widely used accuracy metrics in TTE prediction. It quantifies the fraction of data point pairs that are correctly ranked by a prediction model among those that can be unambiguously ranked. The C-index values range from 0 to 1, with 1 indicating perfect ranking accuracy. However, a notable limitation of the C-index is its dependence on a risk score function for ranking, which many TTE prediction models do not explicitly learn. To address this limitation, [1] introduced a time-dependent concordance index ( C td ), which leverages predicted survival functions, ˆ S ( ·| x ) , to assess model performance more effectively. The C td is computed as follows.

Definition 3. Suppose that we have a survival function estimate ˆ S ( ·| x ) for any x ∈ X . Then using the set of comparable pairs E := { ( i, j ) ∈ [ n ] × [ n ] : δ i = 1 , y i < y j } , we define the C td metric as:

which is between 0 and 1. Higher scores are better.

E.1.2 Time-dependent AUC

While the C-index and C td scores provide valuable single-number summaries of predictive accuracy in TTE prediction, they lack the ability to evaluate accuracy at a specific user-defined time, t . To address this limitation, [67] introduced time-dependent AUC scores ( AUC td ), which explicitly depend on the chosen time point t . The core idea behind AUC td is to frame a binary classification problem for a fixed time t , where the 'positive' class consists of data points that experienced the event no later than t , and the 'negative' class includes those that survived beyond t . The survival function ˆ S ( t |· ) : X → [0 , 1] serves as the probabilistic classifier, predicting survival probabilities. A lower predicted survival probability for a given point x implies a higher likelihood of belonging to the positive class. The AUC td score quantifies the classifier's ability to distinguish between these two classes at time t , offering a time-specific accuracy assessment of the model's predictions. The AUC td is computed as follows.

Definition 4. Suppose that we have a survival function estimate ˆ S ( ·| x ) for any x ∈ X . Then for any t > 0 , using the set of comparable pairs E ( t ) := { ( i, j ) ∈ [ n ] × [ n ] : δ i = 1 , y i ≤ t, y j > t } , we define the AUC td ( t ) (the AUC td at time t ) as:

where w 1 , w 2 , · · · , w n ∈ [0 , ∞ ) are inverse probability of censoring weights to be defined as w i := 1 / ( ˆ S censor ( y i ) ˆ S censor ( t )) , and ˆ S censor ( t ) is an estimation of S censor ( t ) := P ( C > t ) using Kaplan-Meier estimator [33]. AUC td ( t ) is between 0 and 1 and higher scores are better. Finally, we can get AUC td as

where t min and t min are user-specified lower and upper limits of integration.

E.1.3 Integrated Brier Score

The Integrated Brier Score ( IBS ) is a performance metric that directly evaluates the error of an estimated survival function ˆ S ( ·| x ) without relying on ranking. The IBS is calculated as follows.

Definition 5. Suppose that we have a survival function estimate ˆ S ( ·| x ) for any x ∈ X . Then for any t > 0 , we define the BS ( t ) (the IBS at time t ) as:

which is nonnegative. Lower scores are better. Finally, we can get IBS as

where t min and t min are user-specified lower and upper limits of integration.

E.2 Fairness Metrics

In this study, we define fairness metrics as the predictive performance gaps between groups. This kind of metric is used ensure that the model maintains equal predictive performance across different groups. In particular, given a performance metric Er for TTE prediction task, we define fairness metric F Er as follow:

where A is the set of groups considered in TTE prediction task, Er a and Er a ′ are the predictive performance metrics calculated from subsets containing data from groups a and a ′ , respectively. For each predictive performance metric defined above, we have a corresponding fairness metric as follows.

where C td a , AUC td a , IBS a are predictive performance metrics calculated from the subset containing data from group a , and C td a ′ , AUC td a ′ , IBS a ′ are predictive performance metrics calculated from the subset containing data from group a ′ .

E.3 Fairness-Utility Trade-Off Metric

The fairness metrics mentioned above do not capture the fairness-utility trade-off while in medical context, it is essential to balance fairness and utility to ensure that the model is not only fair but also accurate and effective for all groups. To handle this issue, we leverage the equity-scaling metric ( ES ) [44] that takes both utility and fairness into account for evaluation. Similar to fairness metric, for each predictive performance metric, we have a corresponding fairness-utility trade-off metric as follows.

The advantage of the equity-scaling metric lies in its intuitive interpretability. Specifically, a higher equity-scaling score indicates that the model is both more accurate and more equitable simultaneously.

F Experimental Setup Details

F.1 Implementation Details

F.1.1 Hardware Usage

The experiments were conducted at a supercomputing center utilizing multiple compute nodes. Each node was equipped with an NVIDIA Volta V100 GPU with 16 GB of memory, an Intel Xeon CPU, and 32 GB of RAM, ensuring the computational resources necessary for large-scale experiments. In total, we trained over 20,000 models, requiring approximately 4.56 GPU years of computational effort, highlighting the extensive scale of our study.

F.1.2 Package Usage

The FairTTE benchmark is implemented using Python 3, with PyTorch [47] serving as the framework for deep learning computations. The implementation of TTE models is built on the pycox [41]

package, while the evaluation metrics for TTE prediction leverage pycox, scikit-survival [50], and SurvivalEVAL [52]. Additionally, the training and evaluation pipeline for TTE prediction models is adapted from the demo code provided in [5], ensuring a robust and standardized framework for benchmarking.

F.2 Data Split and Pre-processing

Data Split. Each dataset in our study was divided into training, validation, and testing sets using a 60%:20%:20% split ratio. Models were trained on the training sets, evaluated on the testing sets, and the validation sets were used for model selection. Since a single patient may have multiple medical records, we took precautions to prevent data leakage during model training. Specifically, the data was split by patient, ensuring that no patient appearing in the testing set had any records in the training or validation sets. This approach maintains the integrity of the evaluation process and ensures that model performance is assessed on entirely unseen patient data.

Data Pre-processing. Before being fed into the TTE prediction models, chest X-ray and color fundus images are resized to 224 × 224 pixels, while brain MRI scans are resized to 128 × 128 × 96 . Additionally, all pixel values are normalized to a range of 0 to 1 to ensure stability during training and improve model performance.

We consider binary group setting in our experiment. These groups were constructed according to the following criteria:

Race: 'Non-White' (Group 0), 'White' (Group 1) · Sex: 'Female' (Group 0), 'Male' (Group 1) · Age: -MIMIC-CXR: ' ≤ 60 ' (Group 0), ' > 60 ' (Group 1) -AREDS: ' ≤ 70 ' (Group 0), ' > 70 ' (Group 1) -ADNI: ' ≤ 80 ' (Group 0), ' > 80 ' (Group 1)

F.3 Hyperparameter Search

To ensure a fair comparison, we perform a grid-based hyperparameter search using 10 random seeds. The details of the hyperparameter search for the methods used in our experiments are provided below.

•

•

TTE prediction models -Learning rate: 10 x where x ∼ Uniform ( -4 , -3) -Decay rate: 10 x where x ∼ Uniform ( -6 , -4) Fair TTE prediction models -η : 10 x where x ∼ Uniform ( -3 , -1) (DRO) x

-λ : 10 where x ∼ Uniform ( -5 , 2) (FRL)

For standard TTE prediction models, we select the best models based on their predictive performance metrics calculated on the validation sets. In contrast, for fair TTE prediction models, we prioritize fairness metrics when selecting the best models, allowing for up to a 5% reduction in accuracy compared to the baseline TTE models. This approach ensures a balanced trade-off between fairness and predictive performance.

F.4 Quantifying Source of Bias

In order to quantify the degree of bias sources in each dataset and sensitive attribute setting, we use several metrics as follows.

Disparity in mutual information between X Z and Y across groups. We quantify the disparity in mutual information between X Z (i.e., image representation generated from the vision backbones)

and Y across groups by computing the maximum difference in their normalized mutual information values across all groups, as defined below.

where I ( · , ·| A = a, ∆ = 1) represented the mutual information conditioned on A = a and ∆ = 1 and H ( ·| A = a ) denotes the entropy conditioned on A = a and ∆ = 1 .

Disparity in mutual information between X Z and ∆ across groups. Similarly, we quantify the disparity in mutual information between X Z (i.e., image representation generated from the vision backbones) and ∆ across groups by computing the maximum difference in their normalized mutual information values across all groups, as defined below.

Disparity in TTE distribution across groups. We measure the disparity in TTE distributions across groups by calculating the maximum Wasserstein distance [63], normalized by the range of TTE, between the TTE distributions of each group. This is defined as follows:

where W ( · , · ) denotes the Wasserstein-1 distance between the two distributions.

Disparity in image distribution across groups. We measure the disparity in image distributions across groups by calculating the maximum Wasserstein distance [63], normalized by the range of image feature values, between the image distributions of each group. This is defined as follows:

where W ( · , · ) denotes the Wasserstein-1 distance between the two distributions. Due to the high dimensionality of image representations, we implement sliced Wasserstein distance [3], a variant of the Wasserstein distance that approximates the full Wasserstein distance between high-dimensional distributions by projecting them onto one-dimensional subspaces and averaging the resulting 1D Wasserstein distances.

Disparity in censoring rate across groups. We quantify the disparity in censoring rates across groups by calculating the maximum normalized difference between the means of the censoring distributions for each group, as defined below.

F.5 Constructing Causal Distribution Shift

To construct distribution shift between training and testing data, we modify the training data by introducing correlations between the sensitive attribute and other RVs in the causal graph (Figure 2). This adjustment simulates real-world scenarios where biases in data collection or underlying relationships may lead to disparities across groups. The details of this process, including the specific modifications applied to establish these correlations, are outlined below.

Distribution shift on X : Images from disadvantaged groups are degraded using a Gaussian blur filter to simulate lower-quality data.

Distribution shift on Y : TTE labels for disadvantaged groups are corrupted by adding noise sampled from a uniform distribution.

Distribution shift on ∆ : To simulate biased censoring, we flip the censoring indicators for 90% of uncensored samples within disadvantaged groups.

We note that, although these distribution shifts are synthetic, they mimic real-world scenarios, as described below.

Adding noise to images mimics real-world scenarios such as patients in different geographic locations are scans with different equipment. This causes the medical image to appear systematically different for groups in each location.

Adding noise to TTE labels mimics real-world scenarios such as delayed or inaccurate event recording in EHR system.

Flipping censoring indices mimics real-world scenarios in which certain groups experience less consistent access to care due to financial or geographic. Thus, these groups are more likely to drop out of care, resulting in a higher censoring rate.

G Causal Graphs for Fairness in TTE Prediction

G.1 Causal Graphs for Biased and Unbiased Settings

Figure A1 presents the causal graphs for the unbiased and biased scenarios. In the unbiased scenario (Figure A1a), the sensitive attribute A is unrelated to the TTE outcome and influences only X A , with no effect on other variables in the graph. In contrast, in the biased scenarios (Figure A1b and Figure A1c), A also affects additional variables, resulting in dependencies between A and the TTE outcome. These causal pathways may be direct (Figure A1b), mediated through unobserved variables U (Figure A1c), or both.

、

X

A

A

△

1

X

A

I

A

△

I

1

X

A

1

A

△

X

X

X

U

X

Z

H

I

Z

T

Y

XZ

Z

T

Y

XZ)

Z

T

Y

(a)UnbiasedSetting

(c)Biased Setting (IndirectPathway)

G.2 Real-world Causal Graph Examples for Fairness in TTE Prediction

In this section, we present causal graphs illustrating real-world scenarios in time-to-event (TTE) prediction using medical imaging. Many of these examples are adapted from diagnostic settings in prior work [28]. We describe four scenarios in which the sensitive attribute A influences other variables in the causal graph-namely, the medical image X , the underlying condition Z , the time-toevent T , and the censoring time C -leading to disparities in group-specific data distributions. For each scenario, we include two examples: one where the causal pathway from A is valid (red arrows), appearing in both training and testing data, and one where the pathway is spurious (blue arrows), representing bias present only in the training data and absent in the testing data.

Brain

Age

Censoring

Censoring

Skin

Race

Censoring

Censoring

Volume

Time

Index

Pigmentation

Time

Index

Dermatoscopic

Accessto

BrainMRI

Alzheimer's

Image

Healthcare

Disease

Underlying

Observed

Lesion

Underlying

Observed

Feature

Condition

TTE

Time

Feature

Condition

TTE

Time

(a)

(b)

Non-

disease

Sex

Censoring

Censoring

Location

Censoring

Censoring

Time

Index

Artefact

Time

Index

Feature

Breast

Time to

Mammogram

Density

Ultrasound

referer

Disease

Underlying

Observed

Lesion

Underlying

Observed

Feature

Condition

TTE

Time

Feature

Condition

TTE

Time

(a)

(b)

Density

Bone

Age

Censoring

Time

Censoring

Index

Artefact

Location

Censoring

Time

Censoring

Index

Differential

Chest

Qualityof

CTScan

Diagnosis

X-ray

Care

Disease

Feature

Underlying

Condition

TTE

Observed

Time

Feature

Lesion

Underlying

Condition

TTE

Observed

Time

(a)

(b)

Non-

Non-

disease

Sex

Censoring

Censoring

disease

Race

Censoring

Censoring

Feature

Time

Index

Feature

Time

Index

Cardiac

Follow-up

Accessto

MRI

Pattern

BrainMRI

Healthcare

Disease

Underlying

Observed

Disease

Underlying

Observed

Feature

Condition

TTE

Time

Feature

Condition

TTE

Time

(a)

(b)

H Additional Results

H.1 Predictive Performance and Fairness in TTE Prediction Models

Figure A6 presents the complete per-group performance results of TTE prediction models-DeepHit, Nnet-Survival, and PMF-across all dataset, sensitive attribute, and metric combinations, while Figure A7 reports the corresponding significance tests using the two-sided Wilcoxon signed-rank test. As shown, performance gaps between groups are observed across all settings.

(a)

C

td

(b)

AUC

td

1.0

Group0

Group1

I%S6H

60

t

0.8-

0.7

0.6

Age

Sex

Age

Race

Sex

Age

Race

Sex

ADNI

AREDS

MIMIC-CXR

1.0

Group0

Group1

I %S6H

0.9

C

0.8

0.7

0.6

Age

Sex

Age

Race

Sex

Age

Race

Sex

ADNI

AREDS

MIMIC-CXR

0.4

Group0

Group1

I%S6H

0.3

S

B

0.2

0.1

0.0

Age

Sex

Age

Race

Sex

Age

Race

Sex

ADNI

AREDS

MIMIC-CXR

(c)

IBS

AUCtd

IBS

Ctd

p

Race

Race

Race

NA

2e-09

2e-09

NA

2e-09

4e-03

NA

2e-09

2e-09

-1e-02

-1e-04

1e-07

1e-08

2e-09

4e-09

2e-09

2e-09

Age

1e-03

2e-09

2e-09

-1e-06

xas

4e-05

1e-04

2e-09

xas

1e-04

2e-09

2e-03

xas

2e-03

2e-07

2e-09

-1e-08

ADNI

AREDS

MIMIC

ADNI

AREDS

I

MIMIC

ADNI

AREDS

MIMIC

H.2 Comparison between Pre-Training and Training from Scratch Strategies for TTE Prediction Models

H.2.1 Comparison in Predictive Performance

Figure A8 presents the complete per-group predictive performance gap between pre-training and training from scratch approaches across all dataset, sensitive attribute, and metric combinations. As shown, pre-training consistently improves the predictive performance of TTE models across most settings.

(a)

C

td

(b)

AUC

td

(c)

IBS

0.30

Group0

0.05

Group1

(ADNI)

(Others)

△Ctd(

0.15

0.02

△Ctd

0.00

0.00

Age

Sex

Age

Race

Sex

Age

Race

Sex

ADNI

AREDS

MIMIC-CXR

0.30

Group 0

0.05

Group1

AUCtd(ADNI)

AUCtd (others)

0.15

0.02

0.00

0.00

Age

Sex

Age

Race

Sex

Age

Race

Sex

ADNI

AREDS

MIMIC-CXR

0.01

1

(AII)

△IBS

Group 0

Group1

-0.10 J

Age

Sex

Age

Race

Sex

Age

Race

Sex

ADNI

AREDS

MIMIC-CXR

H.2.2 Comparison in Fairness

Figure A9 presents the significant differences in terms of fairness between pre-training and training from scratch strategies. As shown, we do not observe a significant improvement with pre-training compared to training from scratch. Specifically, the p-values from one-sided Wilcoxon signed-rank tests are larger than 0.05 in 18 out of 24 settings, suggesting that pre-training does not lead to more equitable predictions in most cases.

AUCtd

IBS

Crtd

P

Race

Race

Race

NA

9e-10

NA

9e-06

3e-01

NA

2e-09

- 0.8

3e-01

9e-03

1

- 0.6

- 0.4

Sex

Sex

Sex

9e-01

9e-01

1e-04

2e-08

9e-01

7e-02

4e-01

- 0.2

ADNI

AREDS

MIMIC

ADNI

AREDS

MIMIC

ADNI

AREDS

MIMIC

H.3 Comparison with Advanced Image Backbones and Medical Pre-training for TTE Prediction Models

To investigate whether more advanced vision backbones pretrained on medical imaging data can enhance the predictive performance and fairness of TTE prediction models, we conduct an additional experiment using the late AMD progression prediction task on the AREDS dataset. Specifically, we adopt RETFound-a widely recognized eye-specific foundation model based on the Vision Transformer (ViT) architecture [12] and pretrained on millions of retinal images-as the image backbone for our TTE models. As shown in Table A3, RETFound with ViT does not demonstrate any improvement over EfficientNet pretrained on ImageNet for this task, suggesting that general-purpose backbones may remain competitive despite the availability of domain-specific pretraining.

H.4 Fairness in Fair TTE Prediction Models

Table A4 and Figure A10 present the results of statistical significance testing for fair TTE prediction models, conducted using the Friedman test followed by the Nemenyi post-hoc test.

2.0

2.5

3.0

3.5

4.0

4.5

5.0

GroupDRO (1.9))

(5) DeepHit

Regularization (3.1)-

(4) Reweighting

(3.8) DomainlndAggregated

(a) Ctd

3.0

3.5

4.0

(b) AUCtd

3.5

(c) IBS

4.5

DomainlndAggregated (2.6))

Regularization (2.9)

Reweighting (3.2)

2.5

Regularization (2.5)

Reweighting (3)-

GroupDRO (3.4)-

3.0

4.0

4.5

(3.8) GroupDRO

(3.8) DeepHit

(4.6) DeepHit

(3.5) DomainlndAggregated

H.5 Fairness-Utility Trade-Off Results

Incorporating fairness shifts the objective from pure utility optimization to balancing utility and fairness. To assess this trade-off in fair TTE prediction methods, we compute equity scaling scores [44] across datasets and sensitive attributes under both in-distribution and distribution shift scenarios. As shown in Figures A11-A14, different methods exhibit varying fairness-utility trade-offs, with CSA achieving the most favorable balance in most settings.

80

70

60

DRO

CSA

DI

SR

FRL

50

AUCtd

IBS

Ctd

80

B

70

DRO

CSA

DI

SR

FRL

AUCtd

IBS

Ctd

80

1

70

DRO

CSA

DI

SR

FRL

AUCtd

IBS

Ctd

80

1

70

DRO

CSA

DI

SR

FRL

AUCtd

IBS

H.6 Additional Results for Predictive Performance and Fairness in Fair TTE Prediction Models under Distribution Shift

This section presents the complete results for fair TTE prediction under distribution shift scenarios. As illustrated in Figure 3, we define distribution shift as a setting where correlations between the sensitive attribute and other variables in the causal graph are present in the training data but absent in the testing data. To simulate such shifts, we intervene on one group (the intervened group) by corrupting specific aspects of the data-namely, the images ( X ), TTE labels ( Y ), or censoring indicators ( ∆ )-while leaving the other group unchanged. Detailed procedures for generating these shifts are provided in Appendix F.5.

H.6.1 Results for Distribution Shift in X

O

DI

CSA

DRO

DeepHit

FRL

.

SR

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

0040

.

0.700

.

0940

...

0.780

009°0

。

0.600

0.720

0.760

AUCtd

0.500

0.660

0.720

0.780

0.720

0.800

0.776

0.784

0.792

0.788

0.792

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.800

0.780

0.776

0.776

。

0000

0.768

0.774

0.768

0.600

0940

0.750

0.780

0.810

0.775

0.780

0.785

0.780

0.785

0.776

0.784

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

0.640

0.700

0.820

0180

0080

out-of-distribution

0.560

0.600

0.800

2

0.480

0.600

0.700

0.500

0.640

0.720

0.812

0.815

0.817

0.790

0.808

0.816

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

6

0.760

0940

0.750

·。

0.750

O

009°0

0.740

0.752

0.752

O

0.450

。

0.744

.

0.744

0.832

0.840

0.848

0.735

0.750

0.765

0.750

0.756

0.762

0.752

0.760

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

OSEO

0.180

0.165

F00E'0

.

0.250

.

0.250

0.160

8

0.150

.

O

0.200

0.180

0.240

0.200

0.250

0.300

0.145

0.150

0.155

0.148

0.152

0.156

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.175

.

0.225

0.220

0.220

0.150

0.210

O

0.210

。

0.210

.

O

0.125

0.200

0.200

0.115

0.120

0.125

0.210

0.225

0.200

0.208

0.216

0.195

0.210

In-distribution

DI

.

CSA

DRO

DeepHit

FRL

.

SR

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

0.100

.

0.160

0800

0500

0800

0.120

.

0.040

0.025

AUCtd

0.060

0.080

。

8

0.100

0.120

0.140

0.100

0.150

0.200

0.010

0.015

0.020

0.005

0.010

0.015

(e)

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.160

0.060

8

900

.

0.120

0.025

0.045

0.032

8

0.080

。

0200

O

0.030

O

0.028

0.105

0.120

0.010

0.020

0.052

0.056

0.030

0.036

0.042

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

0600

.

0.200

0.040

0.080

0.075

-of-distribution

0.150

0.020

F0900

0.040

0.100

(q)

0.050

0.060

0.080

0.120

0.020

0.025

0.004

0.008

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.024

0t00

0.120

0.048

O

0.018

S500

O

060°0

0.042

O

O

0.012

000

0.090

0.105

0.048

0.056

0.020

0.023

0.025

0.038

0.040

0.043

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

.

0.075

0.240

0900

.

0.200

090°0

0.150

p0.180

0.210

O

0.045

0.100

--○

0.045

(c)F1

0.120

0.160

0.200

0.030

0.045

0.060

0.105

0.120

0.040

0.044

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.180

.

0.150

0.016-

0E00

0.020

0.120

.

0.008

.

0.015

0100

0.000

.

0.096

0.104

0.012

0.018

0.024

0.010

0.020

0.000

0.008

0.016

In-distribution

DI

CSA

。

DRO

DeepHit

FRL

SR

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

F000

?

------O

o----o

0090

F0940

.

0.775

0090

AUCtd

0.500

.

.

0.540

8

。

0.720

0.680

0.750

0.725

0.660

0.720

0.780

0.600

0.660

0.720

0.768

0.776

0.780

0.785

0.790

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.800

0.768

.

O

0.756

0.765

0.700

0.760

.

0.752

0.762

0.752

0.600

.

0.759

。

0.750

0.780

0.810

0.756

0.762

0.750

0.755

0.760

0.762

0.768

0.774

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

-of-distribution

0.560

0.640

.

0090

0.500

.

0.760

0.800

0080

8

0.720

0840

0.480

0.400

0.600

0.700

0.560

0.640

0.720

0.796

0.800

0.800

0.808

0.816

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

Out-c

0.750

6

8

0.752

0.744

0.720

0.744

O

0090

990

0.710

0.736

0.450

.

0.832

0.840

0.710

0.720

0.736

0.744

0.752

0.732

0.738

0.744

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

00E'0

O

0.420

00E0

0.195

F090

0.240

0.250

0.180

0.180

0.165

O

8

0.240

0.320

0.400

0.200

0.250

0.210

0.225

0.164

0.168

0.172

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

.

0.222

0.216

0.175

0.216

0.205

O

O

8070

0.150

O

0.200

8

0.125

0.210

0.200

0.120

0.125

0.130

0.208

0.216

0.195

0.200

0.205

0.200

0.210

In-distribution

AUCtd

IBS

Ctd

p

Race

Race

Race

NA

8e-12

4e-06

NA

8e-12

9e-07

NA

8e-12

5e-08

0.8

Age

- 0.6

4e-11

1e-06

5e-02

2e-09

8e-02

1e-06

3e-08

2e-05

7e-08

0.4

xas

2e-10

6e-04

5e-10

xas

1e-06

1e-11

xas

1e-09

7e-06

1e-11

- 0.2

ADNI

AREDS

MIMIC

ADNI

AREDS

MIMIC

ADNI

AREDS

MIMIC

H.6.2 Results for Distribution Shift in Y

DI

CSA

。

DRO

DeepHit

FRL

.

SR

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

0.750

20

06L0

0.720

0.700

0.700

0.787

O

AUCtd

0.776

0.784

0.700

0.725

0.750

0.720

0.760

0.800

0.776

0.784

0.792

0.788

0.792

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.784

0080

0.780

。

0.786

0.783

0.776

0.760

0.776

0.780

0.768

0.750

0.780

0.810

0.775

0.780

0.785

0.780

0.785

0.776

0.784

ADNI-Age

ADNI - Sex

AREDS-Age

AREDS-Sex

0.700

.

0.720

0.820

O

0.816

O

ion

0.675

0.680

0.816

。

8080

-of-distributic

0.650

8

0.640

O

0.812

0.600

0.660

0.680

0.720

0.812

0.815

0.817

0.808

0.816

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

Out-

880

080

8

8

8

0.756

0.756

O

050

0.832

0.735

0540

。

.

0.744

0.832

0.840

0.848

0.735

0.750

0.765

0.750

0.756

0.762

0.752

0.760

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

0.270

0.275

0.155

0.152

-

0.240

8

0.250

O

0.150

。

0.148

0.225

0.145L

IBS

0.225

0.250

0.275

0.220

0.240

0.260

0.145

0.150

0.155

0.148

0.152

0.156

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.125

0.225

O

0.216

0.225

0.210

8

0.120

0.210

8

0.200

0.115

0.195

0.195

0.115

0.120

0.125

0.210

0.225

0.200

0.208

0.216

0.195

0.210

In-distribution

In-distribution

DI

CSA

DRO

。

DeepHit

FRL

.

SR

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

0.125

0.180

0.020

0.012

F00TO

0.150

0.016

AUCtd

0.075

.

0.120

O

0.012

0.008

0.004

。

0.100

0.120

0.140

0.100

0.150

0.200

0.010

0.015

0.020

0.005

0.010

0.015

(e)

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.120

.

0.024

0.042

0.056

?

0.090

0.105

0.016-

0.008

0.052

8

0.036

.：

?

0.105

0.120

0.010

0.020

0.052

0.056

0.030

0.036

0.042

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

O

0.125

0.025

0.075

8

6000

O

-of-distribution

0.050

.

O

0.100

0.020

8

0.006

.

。

0.075

0.015

000

(g)

0.050

0.060

0.080

0.120

0.020

0.025

0.004

0.008

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

Out-

0.100

0.060

0.055

の

O

0.024

0.042

。

0800

o50'0

0.020

6500

0.090

0.105

0.048

0.056

0.020

0.023

0.025

0.038

0.040

0.043

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

0.200

090

O

0.120

。

0.045

0.160

8

0.045

0.105

0.042

IBS

0.120

0000

0.039

0.120

0.160

0.200

0.030

0.045

0.060

0.105

0.120

0.040

0.044

()

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0000

0.102

0.024

0.016

0.096

O

O

0.016

..-

0.020

0.010

0.008

0.090

0.008

?

0.000

.

0.096

0.104

0.012

0.018

0.024

0.010

0.020

0.000

0.008

0.016

O

DI

CSA

。

DRO

DeepHit

FRL

SR

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

0.740

.

0.675

0.720

O

0.650

0.776

。

0.784

AUCtd

0.700

0.625

0.768

0840

0.700

0.720

0.740

0.630

0.660

0.768

0.776

0.780

0.785

0.790

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.774

0.765

0.800

0.760

8

0.760

O

0.768

0.760

0.755

0.756

0.762

O

0.752

0.750

0.780

0.810

0.756

0.762

0.750

0.755

0.760

0.762

0.768

0.774

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

0.725

0.640

5080

0.816

8

O

0.700

8080

o

f-distribution

O

0.600

0.800

0.675

0.560

0080

.

0.600

0.660

0.720

0.600

0.650

0.700

0.796

0.800

0.800

0.808

0.816

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

-no

0.840

O

0.720

0.744

80

0.832

0.710

。

0.736

0.732

.

0.824

0.700

0.832

0.840

0.710

0.720

0.736

0.744

0.752

0.732

0.738

0.744

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

FOSE'O

8

0.240

0.230

o--0

O

0.172

0.225

0.220

0.168

00E0

O

0.210

0.210

?

0.164

IBS

0.320

0.360

0.225

0.240

0.210

0.225

0.164

0.168

0.172

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.205

0.130

。

0.216

8

0.200

0.210

0.125

0.120

0.208

0.195

0.200

0.120

0.125

0.130

0.208

0.216

0.195

0.200

0.205

0.200

0.210

In-distribution

AUCtd

IBS

Ctd

p

Race

Race

Race

NA

4e-02

1

NA

5e-03

2e-05

NA

4e-02

1e-08

- 0.8

- 0.6

6e-02

1e-01

9e-01

3e-01

4e-01

7e-03

6e-02

3e-10

0.4

Sex

5e-03

3e-03

Sex

4e-02

2e-01

3e-04

xas

1e-02

3e-02

3e-05

- 0.2

ADNI

AREDS

MIMIC

ADNI

AREDS

MIMIC

ADNI

AREDS

MIMIC

H.6.3 Results for Distribution Shift in ∆

DI

CSA

。

DRO

DeepHit

.

FRL

.

SR

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

0.760

.

0.720

0.720

0.792

0.792

0890

8840

0.640

AUC

0.680

0.776

6

0.700

0.725

0.750

0.720

0.760

0.800

0.776

0.784

0.792

0.788

0.792

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.780

0.785

0.786

0.800

0.775

.

0.780

O

O

0.760

.

0.774

0.770

0.775

0.750

0.780

0.810

0.775

0.780

0.785

0.780

0.785

0.776

0.784

ADNI-Age

ADNI- Sex

AREDS-Age

AREDS-Sex

0.820

O

0.816

O

of-distribution

0.680

0.680

0.816

090

6

8080

0.640

0.812

0.600

0.660

0.680

0.720

0.812

0.815

0.817

0.808

0.816

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.762

0940

Out-

0.840

0.750

8.

0.756

8

0.752

0.832

0.750

.

。

.

0.832

0.840

0.848

0.735

0.750

0.765

0.750

0.756

0.762

0.752

0.760

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

00E0

a

0.280

0.155

0.152

0.270

0.260

0.150

--0

O

0.148

0.240

O

0.240

0.145

IBS

0.225

0.250

0.275

0.220

0.240

0.260

0.145

0.150

0.155

0.148

0.152

0.156

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.208

0.220

0.125

0.216

8

0.204

0.210

0.120

0.208

O

0.115

8

0.200

0.200

0.200

0.115

0.120

0.125

0.210

0.225

0.200

0.208

0.216

0.195

0.210

In-distribution

DI

CSA

DRO

。

DeepHit

FRL

.

SR

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

0.160

0.160

8100

.

0.012

0.120

0.120

0.012

O

O

0.006

(a) F AUCtd

0.080

0.000

0.100

0.120

0.140

0.100

0.150

0.200

0.010

0.015

0.020

0.005

0.010

0.015

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.105

0.024

0.056

0.042

0.016

900

060°0

0.008

O

0.052

0.105

0.120

0.010

0.020

0.052

0.056

0.030

0.036

0.042

ADNI-Age

ADNI - Sex

AREDS-Age

AREDS-Sex

6

0.024

600°0

一

090°0

8

Out-of-distribution

O

0.100-

0.020

0.006

0.040

0.075

0.016

000

(q)

0.050

0.060

0.080

0.120

0.020

0.025

0.004

0.008

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.105

0.060

z00

0600

0.054

0.024

：

0.075

0.048

0.021

0t00

0.038

0.090

0.105

0.048

0.056

0.020

0.023

0.025

0.038

0.040

0.043

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

0.250

0900

8

St00

0.200

0.105

O

0.040

0.042

O

0.090

(C)F1BS

0.150

.

00

0.120

0.160

0.200

0.030

0.045

0.060

0.105

0.120

0.040

0.044

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.030

0.018

0T00

O

O

0.020

0.012

S000

0.096

0.010

0.006

0.096

0.104

0.012

0.018

0.024

0.010

0.020

0.000

0.008

0.016

In-distribution

DI

CSA

。

DRO

DeepHit

FRL

.

SR

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

0.720

?

0.660

.

0.780

0.774

0.785

0640

0890

?

0.640

0.600

0.768

0840

-O

AUC

0.700

0.720

0.740

0.630

0.660

0.768

0.776

0.780

0.785

0.790

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.762

0.760

0080

0.756

0.755

0.765

0.760

0.750

.

0.750

0.760

0.750

0.780

0.810

0.756

0.762

0.750

0.755

0.760

0.762

0.768

0.774

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

0.720

0.625

0.805

0.816

-of-distribution

0.680

0.600

.

0.800

8080

0.640

。

0.575

O

。

0.795

0.800

0.600

0.660

0.720

0.600

0.650

0.700

0.796

0.800

0.800

0.808

0.816

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

-no

0.840

0.744

O

0.832

0.720

0.744

0.710

0.824

.

0.736

0.732

.

0.832

0.840

0.710

0.720

0.736

0.744

0.752

0.732

0.738

0.744

ADNI-Age

ADNI-Sex

AREDS-Age

AREDS-Sex

0.400

09E'0

8

a

0.240

0.225

0.210

0

0.172

0.168

0.320

0.220

.

0.164

IBS

0.320

0.360

0.225

0.240

0.210

0.225

0.164

0.168

0.172

AREDS-Race

MIMIC-CXR-Age

MIMIC-CXR-Sex

MIMIC-CXR-Race

0.130

0.224

0.216

0.201

.

0.125

0.216

O

O

0.198

0.120

0.208

0.200

0.120

0.125

0.130

0.208

0.216

0.195

0.200

0.205

0.200

0.210

In-distribution

AUCtd

IBS

Ctd

p

Race

Race

Race

NA

3e-04

3e-01

NA

1e-03

NA

9e-08

4e-11

- 0.8

Age

- 0.6

7e-04

2e-01

2e-06

3e-02

3e-02

4e-01

1e-09

0.4

Sex

3e-08

3e-02

2e-04

Sex

5e-03

xas

5e-05

8e-02

3e-09

- 0.2

ADNI

AREDS

MIMIC

ADNI

AREDS

MIMIC

ADNI

AREDS

MIMIC

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a framework for assessing fairness in time-to-event settings (in the setting of censored observations), where the prediction of event times is based on image data and possibly other covariates. The fairness criterion considered looks at the difference in the error rates between demographic groups. The paper's results include bounds for the fairness metric in terms of appropriate group separability quantifications, and a decomposition of differences between prediction distributions into various contributions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- (S1) The paper deals with an important, and timely topic of research,\n- (S2) The paper is well-written and clear,\n- (S3) Some parts of the analysis seems to be offering a novel approach,\n- (S4) This is visibly a high-effort submission, with extensive empirical analyses.\n\nWeaknesses:\n- (W1) Survival analysis for medical data is plagued with multiple serious challenges, with some of the most significant issues being informative censoring [1] and competing risks [2]. The methodology proposed in the paper does not even mention these issues, or attempts to address such problems, which are known to be key for state-of-the-art survival analysis in medical data. This is also reflected in the empirical data analyses. For instance, in the MIMIC-CXR dataset, patients are censored 1-year after their last discharge. However, hospital discharge is not treated as censoring, meaning that patients remain in the risk set, even though they are not really at risk. A core issue is that there is no information in the dataset about mortality outcomes outside the hospital. Furthermore, competing risks are an important consideration, since admission patterns differ significantly for racial minorities, and different groups have different competing risk after hospital discharge. Thus, it remains unclear if the proposed analysis is appropriate. Similarly, the setting considered for the ADNI dataset seems to include events that can only happen on patient visits, which is a setting where interval censoring is commonly considered. In summary, non-informative censoring, which is assumed throughout the paper, is a major limitation that appears to be glossed over.\n \n- (W2) Another point that may confound the reader are the claims about offering a causal perspective. Figures 2 and 3 seem to discuss the causal structure, but it remains unclear whether anything related to causal mechanisms is discussed. The figures specify DAGs that correspond to different settings that could be true or not, but these graphs are not used for any identification purposes (or specifying assumptions?). Furthermore, in the decomposition of Equation (4), the terms appearing seem to talk about mutual information, which are also not causal quantities, and the decomposition seems largely unrelated to different causal mechanisms (or at least this remains unclear). It seems from a distance that the analysis could rest on Bayesian networks entirely (relying only on notions of observational distributions), which makes the framing of a \"causal perspective\" somewhat unusual.\n\nReferences\n\n[1] Leung, Kwan-Moon, Robert M. Elashoff, and Abdelmonem A. Afifi. \"Censoring issues in survival analysis.\" Annual review of public health 18.1 (1997): 83-104.\n\n[2] Andersen, Per Kragh, et al. \"Competing risks in epidemiology: possibilities and pitfalls.\" International journal of epidemiology 41.3 (2012): 861-870.\n\nQUESTIONS:\n- (Q1) The decomposition is Equation (4) claims to decompose a disparity. Usually, statistical or causal disparities are quantified through contrasts, such as differences/ratios of conditional expectations/potential outcomes. Curiously, in Equation (4), the LHS $P(y, \\delta \\mid x, a)$ does not seem to be a contrast. Clarifying this would be helpful (e.g., are ratios considered for different values of $a$?).\n- (Q2) Is there a connection of the decomposition with causal decompositions appearing in the causal fairness literature, e.g. [3]?\n- (Q3) It would be interesting to understand whether there is a more fundamental argument for the proposed fairness criterion. In the text, there are arguments about the relation of distribution shift and fairness. What about the following simple example\n$$\n\\begin{align}\nA &= \\text{Bernoulli}(0.5) \\\\\\\\\nY &= A + (1+A) \\epsilon,\n\\end{align}\n$$\nwhere $\\epsilon$ follows an exponential distribution. In the above example (reflecting heteroscedastic noise across groups), why would equal error in prediction across groups be a useful criterion, if prediction is inherently more difficult for one group?\n\n[3] Zhang, Junzhe, and Elias Bareinboim. \"Fairness in decision-making—the causal explanation formula.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.\n\nLIMITATIONS:\nNo, the limitations are not adequately addressed (see Weakness (1)), and some of the paper's claims seem to be stronger than the results and methodology support.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FairTTE, a comprehensive causal framework for investigating and quantifying fairness in time-to-event (TTE) prediction within medical imaging. Prior fairness research in medical AI has focused largely on diagnostic tasks (classification, segmentation). This work targets prognosis tasks, with additional challenges due to censoring, lack of standard fairness metrics, and complex data generation processes. The paper combines a theoretical causal analysis to decompose sources of bias with empirical studies on three real-world datasets (MIMIC-CXR, ADNI, AREDS). The theoretical analysis shows how bias arises, and extensive experiments corroborate that bias is pervasive across demographic groups. The analysis shows that existing fairness algorithms only partially mitigate bias and distribution shifts pose further challenges.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1.\tPrognosis is potentially even more important than diagnosis in medical AI, and fairness in this area is substantially understudied compared to fair diagnosis. \n\n2.\tCompared to many empirical papers on fairness, this paper deposes the ways in which bias can arise using structural causal models, which provides a clearer way to understand and think about this important problem. \n\n3.\tEmpirical validation of state-of-the-art TTE and fairness methods is thorough, with good statistical analysis. I appreciate this good HPO protocol and statistical testing compared to many more ad-hoc papers in this area.\n\n4.\tGood that data and code are open for reproducibility. \n\n5.\tInsightful Findings: The theoretical analysis is helpful for thinking about and understanding the challenges of fairness. And the empirical study has interesting results such as the limited ability of existing fairness methods to ameliorate bias, a breakdown of the different sources of bias, and the fragility of fairness interventions under distribution shift.\n\n6.\tGenerally the paper is very well written, if a bit dense at times. Including the appendix, it’s very thorough, and comprehensive. \n\n7.\tI  really appreciate the practical examples of the SCMs in App G2. \n\n\n\nWeaknesses: \n\n1.\tMissed insight opportunity: It would have been great to have some suggestions about how to develop new algorithms or improve existing algorithms based on the insights of the SCM analysis.  Alternatively, some more concrete analysis on what is the limit of the best achievable fairness given the practical causal relations that mix the influence of conditions and sensitive attributes.\n\n2.\tMissed insight opportunity: In terms of the causal analysis of bias, it seems that some aspects are in common with standard diagnosis (eg: different image distribution across groups), and others are unique to prognosis (eg: Different TTE distribution across groups). It would have been nice to have an explicit compare & contrast to see what’s the unique part in this work that differs from standard SCM perspective on diagnosis. \n\n3.\tOn comparing images: (i) L106-110 claims that medical images are high-dimensional points that are hard to compare. But then as part of the bias analysis (Fig5, App F4), actually the paper directly compares images in embedding space using Wasserstein distance. (ii) In F4: Is comparing the images in embedding space what we should do? It seems a bit odd, because it depends on the feature representation that each deep network learns, which will be different for different prognosis and mitigation algorithms. \n\n4.\tEvaluation: The test-set bootstrapping for CI calculation (Fig4) seems like not the important source of variability. My experience is that especially for small-ish medical datasets (eg: AREDS, ADNI), sensitive fairness algorithms, and  automated HPO, then the huge source of variability is in seeds for the training phase.\n\nQUESTIONS:\n-\tCan the authors elaborate on how they envision FairTTE being used by practitioners—e.g., in regulatory audits, model certification, or dataset curation?\n\nLIMITATIONS:\nOK.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a a comprehensive framework, FairTTE for assessing and analyzing fairness in TTE prediction for medical imaging.  The FairTTE framework involved training / evaluating thousands of TTE models across 3 major medical imaging datasets, diverse fairness algorithms and TTE algorithms, and performing causal analysis to identify and quantify five primary sources of bias in TTE datasets (image feature distributions, TTE distributions, censoring rates, and in the MI between images and outcomes). \n\nThe authors report several key findings:\n- consistent performance gaps observed between different demographic groups across all tested imaging modalities and datasets. \n- current SOTA algorithms often fail to consistently mitigate bias, and any gains in fairness are frequently accompanied by a decrease in the model's predictive accuracy. \n- pre-training models on large datasets improves their accuracy but has a minimal impact on fairness. \n- maintaining fairness becomes increasingly difficult under distribution shifts\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- solid theoretical framework, large-scale comprehensive evaluation, clear presentation\n\nWeaknesses\n- nothing major, but could benefit from expanding to the scope of medical imaging to include pathology if time permits.\n\nQUESTIONS:\npotentially consider adding pathology as an additional medical imaging modality given recent rise in popularity and adoption of digital pathology, and earlier work investigating fairness in various digital pathology learning algorithms and pretrained foundation models. e.g. Vaidya, A., Chen, R.J., Williamson, D.F.K. et al. Demographic bias in misdiagnosis by computational pathology models. Nat Med\n\nThat said, I am more than impressed by the current scope of investigation by the authors.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a comprehensive framework, FairTTE, for evaluating and understanding fairness in medical image prognosis. They formalize a causal decomposition of bias sources in TTE prediction (e.g. disparities in image features, censoring, event labels, etc.) and use it to explain why fairness is hard to achieve in this setting. To support empirical study, the paper adopts three diverse medical imaging datasets with TTE outcomes: predicting in-hospital mortality from chest X-rays (MIMIC-CXR), late-stage AMD from retinal fundus images (AREDS), and Alzheimer’s progression from brain MRIs (ADNI). The experiments observe that current fairness interventions provide only limited mitigation. While some methods reduce group gaps, they often hurt overall accuracy, and none consistently outperform a baseline model in fairness. The paper also highlights that pre-training on large datasets improves overall TTE accuracy but does not consistently improve fairness, and that distribution shifts (changes between train/test data) further exacerbate unfairness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Medical prognostic model with fairness learning is an important yet underexplored direction.\n2. The paper provides a theoretical causal model and bias decomposition that rigorously characterize how sensitive attributes can influence TTE predictions. \n3. The authors collect three real-world publicly datasets spanning different imaging modalities and outcomes, each annotated with demographic attributes. This is significant to provide a benchmark for fairness in medical image prognosis.\n4. In addition, the evaluation is thorough. The authors train and evaluate a large number of models with varying sensitive attributes (race, sex, age), model types, and fairness algorithms.\n\nWeaknesses:\n1. The paper is primarily an analysis and benchmarking effort. It does not propose a novel algorithm to improve fairness in TTE predictions.\n2. The causal bias framework distinguishes “fair” vs “unfair” pathways. While conceptually sound, it relies on unobservable constructs (true health state Z, latent confounders U) and strong assumptions (e.g., whether data disparities reflect spurious correlations or true causal effects of A). In practice, it may be hard to definitively categorize which biases are “unfair” (e.g., if age genuinely affects disease progression vs. if it’s a proxy for unequal care).\n3. The authors simulate distribution shifts by adding noise or flipping censoring indicators for one group. While this stress-test is useful, it is somewhat artificial. The paper claims to base these on real medical scenarios, but the main text doesn’t elaborate much on how realistic these shifts are.\n\nQUESTIONS:\n1. Given that no existing methods work consistently, what do the authors envision as potential holistic approaches to address all bias sources?\n2. The paper adopts TTE models that assume discretized time intervals for event prediction. Could the authors clarify how the choice of discretization granularity (e.g., interval width or number of bins) impacts both fairness and predictive accuracy?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses a critical gap in fair AI research: while fairness in medical image diagnosis (e.g., classification, segmentation) has been studied, fairness in medical prognosis (predicting disease progression or outcomes over time) remains overlooked. Prognosis is often framed as time-to-event (TTE) prediction (e.g., time until death, disease recurrence). The authors argue that TTE prediction poses unique fairness challenges due to temporal dynamics, censored data (incomplete follow-ups), and the lack of standardized fairness metrics suited for survival analysis.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- This study evaluates the methods on three large-scale public datasets (AREDS, MIMIC-CXR, ADNI) spanning multiple imaging modalities (fundus, X-ray, MRI) and clinical outcomes (AMD progression, mortality, Alzheimer’s).\n\n- This study quantifies five distinct bias mechanisms (e.g., censoring disparities, feature distribution gaps) and shows their real-world prevalence.\n\n- This work demonstrates that while pre-training improves accuracy, it does not inherently improve fairness, a crucial insight for medical AI.\n\nWeaknesses\n\n- The authors' large-scale evaluation, involving over 20,000 trained models, reveals that while these methods can sometimes reduce performance gaps, the improvements are not guaranteed and can be accompanied by a decrease in predictive accuracy. Table 1 demonstrates this inconsistency; for instance, in the ADNI-Sex setting, nearly all fairness algorithms, including DRO, SR, FRL, and CSA, show a negative impact on the Integrated Brier Score (IBS) accuracy compared to the DeepHit baseline.\n\n- Similar to the point above, the paper notes that when sources of bias are small, applying these fairness methods can paradoxically worsen performance disparities. Statistical analysis confirms this, with Friedman and Nemenyi post-hoc tests showing that current algorithms do not offer a significant improvement in mitigating bias over the baseline DeepHit model. This indicates a fundamental gap in existing methodologies, which struggle to address multiple, co-existing sources of bias without a trade-off in utility.\n\n- The theoretical analysis (page 4) relies on performance metrics that satisfy triangle inequality and symmetry properties, which, while described as \"relatively mild\" , may not hold for all metrics used in practice, a limitation the authors acknowledge. More critically, the paper's framework for identifying sources of bias relies on being able to quantify disparities in data distributions, but it concedes that identifying the true causal pathways that create these disparities is a significant challenge that requires deep clinical insight. The study simulates distribution shifts by synthetically corrupting images, TTE labels, or censoring indicators for specific groups. While this provides a controlled environment for analysis, these artificial shifts, such as applying a Gaussian blur or flipping 90% of censoring indicators, may not fully represent the complex, subtle, and multifaceted nature of distribution shifts encountered in real-world clinical settings.\n\n- The study also finds that pre-training, a widely used technique for improving model performance, offers limited benefits for fairness. While pre-training was shown to improve predictive accuracy, especially for the smaller and more complex ADNI dataset , it had a \"minimal impact on fairness\". Statistical tests showed no significant improvement in fairness in 18 out of 24 experimental settings, suggesting pre-training alone is insufficient to produce equitable TTE predictions. This weakness implies that simply using larger, more general models is not a panacea for fairness issues and that dedicated fairness interventions are still necessary, even if they themselves are currently of limited effectiveness.\n\n- The selection of fairness metrics, while justified, narrows the scope of the evaluation. The authors focus on metrics that ensure equal predictive performance across groups, arguing that other categories of fairness metrics are less suitable for medical imaging. They dismiss metrics requiring a similarity measure as \"difficult to establish for medical images\" and deem metrics that enforce similar predicted outcomes as potentially \"nonsensical\" when a sensitive attribute like age is a strong risk factor. While these are reasonable justifications, this focus excludes a broader discussion of individual fairness or alternative group fairness definitions that might be relevant, thereby constraining the paper's conclusions to a specific type of fairness.\n\nQUESTIONS:\nIt would be better to add a paragraph to the discussion section that explores the challenges and potential consequences of using alternative fairness metrics in this context. For example, what would be the clinical and ethical implications of enforcing \"similar predicted outcomes\" (demographic parity) for a condition where a sensitive attribute like age is a known and legitimate risk factor? Briefly exploring this \"what if\" scenario would demonstrate a comprehensive understanding of the broader fairness landscape and proactively address potential questions from readers about why certain metrics were not chosen.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}