The Boundaries of Fair AI in Medical Image Prognosis: A Causal Perspective
Authors: Thai-Hoang Pham
Jiayuan Chen
Seungyeon Lee
Yuanlong Wang
Sayoko E. Moroi
Xueru Zhang
Ping Zhang
Venue: NeurIPS 2025 poster ()
Keywords: fairness
time-to-event
survival analysis
bias
distribution shift
causal
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: FairTTE is the first comprehensive framework for analyzing fairness in time-to-event prediction using medical imaging, revealing widespread bias across modalities and the limitations of existing fairness methods, especially under distribution shifts.

Abstract: As machine learning (ML) algorithms are increasingly used in medical image analysis, concerns have emerged about their potential biases against certain social groups. Although many approaches have been proposed to ensure the fairness of ML models, most existing works focus only on medical image diagnosis tasks, such as image classification and segmentation, and overlooked prognosis scenarios, which involve predicting the likely outcome or progression of a medical condition over time. To address this gap, we introduce FairTTE, the first comprehensive framework for assessing fairness in time-to-event (TTE) prediction in medical imaging. FairTTE encompasses a diverse range of imaging modalities and TTE outcomes, integrating cutting-edge TTE prediction and fairness algorithms to enable systematic and fine-grained analysis of fairness in medical image prognosis. Leveraging causal analysis techniques, FairTTE uncovers and quantifies distinct sources of bias embedded within medical imaging datasets. Our large-scale evaluation reveals that bias is pervasive across different imaging modalities and that current fairness methods offer limited mitigation. We further demonstrate a strong association between underlying bias sources and model disparities, emphasizing the need for holistic approaches that target all forms of bias. Notably, we find that fairness becomes increasingly difficult to maintain under distribution shifts, underscoring the limitations of existing solutions and the pressing need for more robust, equitable prognostic models.

The Boundaries of Fair in Medical Image Prognosis: A Causal Perspective
Thai-Hoang Pham 1,2, Jiayuan Chen Seungyeon Lee Yuanlong :ang!
Sayoko Moroi', Xueru Zhang', Ping Zhang Department of Computer Science and Engineering. The Ohio State University
Depaatment of Biomedical Informatics, The Ohio State University
Department of Ophthalmology and Visual Sciences, The Ohio State University {pham. 375, chen. 12930, lee. 10029, wang 16050) @osu.edu edu {moroi. 4, zhang. 12807, zhang 10631)@osu. edu
Abstract
As machine learning (ML) algorithms are increasingly used in medical image analysis, concerns have emerged about their potential biases against certain social groups. Although many approaches have been proposed to ensure the fairness of ML models, most existing works focus only medical image diagnosis tasks, such as image classification and segmentation, and overlooked prognosis scenarios, which involve predicting the likely outcome progression of medical condition over time. To address this gap, we introduce FairTTE, the first comprehensive framework for assessing fairness in time-to-event (TTE) prediction in medical imaging. FairTTE encompasses a diverse range of imaging modalities and TTE outcomes, integrating cutting-edge TTE prediction and fairness algorithms to en- able systematic and fine-gratd ane-grained analysis fairness in medical image prognosis. Leveraging causal analysis techniques, FairTTE uncovers and quantifies distinct sources of bias embedded within medical imaging datasets. Our large-scale eval- uation reveals that bias is pervasive across different imaging modalities and that current fairness methods offer limited mitigation. We further demonstrate a strong association between underlying bias sources and model disparities, emphasizing the need for holistic approaches that target all forms of bias. Notably, we find that fairness becomes increasingly difficult maintain under distribution shifts, underscoring the limitations of existing solutions and the pressing need for more robust, equitable prognostic models.
Introduction
Machine learning (ML. algorithms trained real-world medical images may inherently exhibit
bias, leading to discrimination against certain social groups [57 42 This is especially concerning in the medical field, where biased algorithms can result in inequitable treatment recommendations, misdiagnoses, or unequal access to care [46). Therefore, ensuring the fairness of ML models and identifying hidden biases within medical images is critical for advancing health equity [54).
Existing research on fairness in ML and medical image analysis has primarily focused on medical image diagnosis tasks, such as image classification and segmentation [22 64 These diagnostic tasks typically aim to determine the presence or absence of condition. However, prognosis scenarios, which involve predicting the likely outcome or progression of a medical condition over time, have been
Equual contribution authors. Corresponning author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Data
Algorithm
Evaluation
Image Modalities
TTE Outcomes
TTE Prediction
Fairness
Metrics
Pre-training
Cheal X-ray
Fundual braega
Brain MFN
In-hospital Mortality
DeepHit
Pre- procassing
Time-dependent
Distribution Shift
C-index
Stafistical Test
Late AMD
Nnet- Survival
In- processing
Integr ated Brier
Score
Causal Analysis
Alzheimer's Disesse
PMF
Post- processing
fime-dependent
Fairesss-ccom accuracy
2D
3D
AUC
Trade-off
Figure I: An overview of the FairTTE, a unified framework designed to investigate fairness in TTE prediction for medical image analysis.
largely overlooked. In ML, prognosis is framed as time-to-event (TTE) prediction or survival analysis, where the goal is to predict the time until critical event. Unlike classification, TTE prediction provides a richer and more dynamic approach to modeling medical outcomes by accounting not only for the presence absence of condition but also for the timing and progression of key health events. This makes especially valuable for predicting long-term outcomes such as survival or disease recurrence [21] Despite its importance, there remains significant gap in the literature regarding fairness in TTE prediction, particularly in the context of medical imaging (see Appendix (A).
Developing fairesss methods for TTE prediction in medical imaging presents several unique chal- lenges compared to tasks like image classification or segmentation. First, there is a lack of public datasets that pair medical images with TTE outcome information. While many public datasets include sensitive attributes, they primarily focus on diagnostic tasks and do not provide the temporal data necessary for training TTE prediction models. Second, there is limited understanding of how biases in medical images specifically affect the fairness TTE prediction models, complicating the development of fair algorithms. Third, the absence of a universally accepted fairness metric for TTE prediction poses a challenge in assessing and comparing fairness across studies 36 53 79). Existing fairness metrics are often not well-suited the complexities of medical applications (see Section 2.1 Furthermore, varying experimental designs and evaluation approaches in the literature hinder direct comparisons between fairness methods, leading to fragmentation in the field and impeding the establishment of best practices for fair TTE prediction in medical imaging.
To address the challenges of understanding fairness in TTE prediction for medical imaging, introduce FairTTE, the first comprehensive framework specifically designed for this task (Figure [] Our approach begins by applying causal reasoning develop general framework for understanding and quantifying bias in TTE prediction. This framework decomposes the data likelihood into distinct components, enabling fine-graines analysis of various sources of bias, including disparities in image features, censoring rates, TTE labels, mutual information between images and TTE, and mutual information between images and censoring indicators. This principled decomposition allows to explain why standard TTE prediction methods and fairness algorithms frequently fail to mitigate bias in practice (see Section 2.2 Using this framework, we characterize not only the presence but also the type and extent of bias across different datasets, offering a detailed understanding of fairness challenges in medical image prognosis. To complement the framework, of large-scale, publicly available medical image datasets tailored for fair TTE prediction. These datasets span a wide range of imaging modalities, including fundus images, chest X-rays, and brain MRIs, and cover diverse TTE outcomes, such as mortality, late age-related macular degeneration (AMD), and Alzheimer's disease. Finally, leveraging both the datasets and causal framework, we establish a realistic and comprehensive investigation study for fairness in TTE prediction. This investigation incorporates of-te-of-the-ar (SOTA) algorithms across various learning strategies and, to our knowledge, represents the first attempt to provide a fine- tne-grained analysis of fairness in TTE prediction, making a significant contribution to the growing field of fair ML in medical imaging.
construct a collection
Our work also explores fair TTE prediction methods across various settings, considering factors such as pre-training and distribution shifts that may influence model faimess. Building on causal framework and through the training of over 20,000 models, we derive several key insights to inform future research in this area, follows:
Bias is prevalent across TTE prediction models trained on different imaging modalities, with consistent performance disparities observed between demographie groups.
All medical image datasets examined in our study exhibit some sources of bias that adversely affect the fairesss of TTE prediction models.
2

---

Most SOTA fairness methods struggle to consistently mitigate Shas-wile they can improve fairness in some settings, these gains are often accompanied by reductions in predictive accuracy.
Pre-training improves model accuracy but has minimal impact on fairness.
Different types of distribution shifts affect fair TTE prediction in distinct ways, underscoring the growing difficulty of maintaining both fairness and utility under realistic clinical scenarios.
2 Unified Framework for Fair TTE Prediction
2.1 Fair TTE Prediction Setup
We first introduce the notations used throughout the paper and then formulate the fair TTE prediction task. In this study, we focus on analyzing methods designed for achieving group fairness in right- censored TTE prediction.
Notations, A TTE dataset contains observations for each individual along with their corresponding (right-censored) TTE outcomes. Specifically, each individual's data is represented by tuple of random variables (RVs) (X, Y, A, A), where X denotes set of features, Y represents an observed time, A is an event indicator, and A is sensitive attribute for the individual. If A = (indicating that the event has occurred), then Y represents true survival time; otherwise. Y is a censoring time. We denote T as a true (possibly unobserved) survival time associated with X, and C' as a true (possibly unobserved) censoring time associated with X. We use lowercase letters x, y, 5, a, t, to denote the respective realizations of these random variables. It is important to note that for each individual, we do not observe both T and C; instead, observe exactly one of them. More precisely, Y = min {T, C}
and A = 1 {T < C}, where 1 is an indicator function. Let S(t'(x) = -1 - fh P(t[x) represent the survival function time given the feature x. In fair TTE prediction, our goal is to accurately predict (t'((z) while adhering to fairness constraints.
Performance Metries for TTE Prediction. Consider TTE prediction model h X T where X' and T are the spaces of x and i, we denote the performance metric for this model on the dataset D = {X,, Y, A,, A,) A,)!21 Er(S, h, D) where / is the ground-truth labeling function (i.e.. f = )(t Note that this notation is sufficiently flexible to encompass censoring data and different metrics in TTE prediction. Furthermore, while prediction error (smaller values indicate better performance, e.g., Brier score [66] is used as the metric in our theoretical analysis, prediction accuracy (larger values indicate better performance, e.g., concordance index [19] also be applied.
Various fairness metrics have been proposed recently for TTE prediction, which can be roughly classified into three categories based on their objectives: (i) ensuring similar predicted TTE outcomes for similar data points [36] 53
Fairness Metries for TTE Prediction.
77 76 72), (ii) ensuring similar 80 and (iii) ensuring similar 79]. Notably, some of these metrics are less applicable in the context of medical imaging. For instance, metrics in the first category require a well-defined similarity measure between data points, which is difficult to establish for medical images. Metrics in the second category may also be inappropriate when sensitive attributes are strong risk factors for the TTE outcome. For example, age is often a key predictor of various TTE outcomes, such as mortality. In this case, asking for similar predicted survival times for young and elderly individuals would be nonsensical.
predicted outcomes for data points from different groups [36
53
predictive performance across different groups [TT 22
78
Therefore, in this study, we focus fairness metrics from the third category, which aim to ensure that the model maintains equal predictive performance across different groups. Specifically, given a performance metric Er for TTE prediction task, we define faimess metric FE. as follow:
F (h)
max Er h, Da) Er (far, h, Da:)
(1)
a, a.
where A is the set of groups considered in TTE prediction task, D. (X,, A, DD.l and D. = [X, Y,, A, A,|4, = DDrl are the subsets containing data from groups and a', and fa = P(I]x, a) and fa' = ((I)x,a)) are the ground-truth labeling functions for D, and Da', respectively. If the labeling function f is independent of the sensitive attribute, then f= fa = lar Note that the fairness metric in Eq. (T) is defined as the maximum performance gap between any two groups, and as such, it can be applied any performance metric used in TTE prediction.
3

---

2.2 Causal Structure for Fair TTE Prediction
To conduct a tineggriine analysis to understand fairness in TTE prediction, we leverage the structural causal model (SCM) [8] to represent the data generation process underlying TTE data. Specifically, we examine how the sensitive attribute influences the TTE prediction model, which aims to approximate the data likelihood P(t]z) using the SCM framework. To construct the causal graph, we introduce an unobserved underlying health condition the patient, denoted Z. In this context, the patient's s feature set X can be viewed noisy and partially observed information derived from Z. Following the approach [25] 669, we parti-
tion X into two components: Xz. represent- ing target-related features directly influenced by Z, and XA. representing features related to the sensitive attribute, directly influenced by A. By construction, X encodes sensitive informa- tion as it is predictive of the sensitive attribute A. Using these definitions, we construct causal graphs for TTE data under two settings: unbi- ased and biased scenarios, illustrated in Figure 2 In the unbiased scenario (Figure 2). the sensi- tive attribute A is irrelevant to the TTE outcome. It only influences XA, and not any other vari- able in the graph. Therefore, capturing the in- variant feature Xz across groups is sufficient for learning a fair model. Specifically, we have, P(1[zz)= P(,,x,,, Va A. In contrast, in the biased scenario (Figure the sensitive attribute A influences additional variables in the causal graph beyond XA. leading to dependency between A and TTE outcome. This results in the conditional distribution of the outcome given Xz varies across groups P(t|æ2,, a) + P(t] d'), a' E A and/or the distribution of X differs across groups P(x=la)= # P(zzla") These discrepancies reflect the presence of bias in the data and challenge the assumption that learning invariant features alone is sufficient for fair TTE prediction
Unbissed Setting
(b) Blased Setting
Figure Causal structure in TTE prediction. Gray circles represent unobserved RVs. (a) Unbiased setting, where the sensitive attribute A affects only XA. (b) Biased setting, where the sensitive at- tribute A may be correlated (red arrow) with other RVs in causal graph.
important to note that disparities in data distributions across groups, as reflected in the causal graph, do not necessarily indicate that the sensitive information captured during model training is spurious or inappropriate every context. In some cases, genuine biological dif- ferences between groups may exist, making the sensitive attribute relevant for disease prediction. In such situations, incorporating group-specific information can be beneficial, it allows the model to capture distinct disease mechanisms that reflect true underlying biological variation.
(R) Training Dataset
(b) Testing Dataset
Figure 3: Causal structure in TTE prediction under distribution shift. We illustrate scenario where an unfair causal pathways (blue arrows). induced by unobserved RV U. are present in train data (a) but absent in test data (b), leading to distribution shift. Bidirectional arrows indicate that the causal direc- tion may vary depending on the specific context. Fair causal pathways (appear in both train and test data) may exists but are omitted for simplicity.
However, observed biases in the training data may arise from spurious correlations that reflect historical inequities in healthcare access, diag- nosis, and treatment practices (illustrated by the blue arrows in Ba.. When models are trained on such data, they risk encoding and amplifying these biases, thereby perpetuating unfairness in clinical pecision-making From this perspective, fairness and distribution shift are intrinsically connected: achieving fairness requires models that generalize to a test distribution in which unfair causal pathways are absent and only fair causal pathways remain (Figure 3b) [37] 71 62
Group-specific Distribution Disparity and Fairness. As shown in Figure 2p. the effect of A on other RVs can induce distribution disparity across groups We now formalize how it affect fairness in Theorem below.
We acknowledge the possibility of unobserved RVs that may influence both sensitive attribute A and other RVs (i.e., X, 2, in causal graph. However, since our primary objective characterize disparities group- specific data distributions, we omit unobserved RVs from the causal graph for simplicity (see Appendix (.T). Real world medical examples of fair and unfair causal pathways provided in Appendix G.2

---

Theorem 1. Given a performance metric Er satisfying triangle inequality and symmetry properties,
i... [rrh,,'' h', D) Er(h, D) I Er(h', R², D) and Er(h, h², D) Er(h', h, D), then we have: (h) S max (n1H,, fa. Ja. ) + D(H, Da, Da))), vh E a,a'EA
(2)
with
nHH, far fa') min (Er(fas h', D.) Br(fa,,'' (D,)), W'EH
D(H, Day Da')=
max [Er(M, N², Da) Er(W, Dar)]
W.NEHH
where H. represents the hypothesis class of the TTE prediction model h, (HH, fas far) denotes the minimum joint prediction enror on D, and Dz', and (H. D., is the largest distance between two groups a and a', with respect to the hypothesis class H. Note that the requirements on symmetric relatively mild and can be met by many performance metrics commonly used in practice (see Appendix [B.
and triangle inequality properties
Discussion of Theorem 1 The term D(H, Da, Dar) aligns with the concept of subgroup sep- arability-the the ability to predict group membership from image features-in the group fairness literature [30]. Low subgroup separability implies that the image feature distributions are similar across groups, which corresponds to a small value of D(H, Da. Dz') ). Prior work has shown that unfairness tends to be less severe in datasets with low subgroup separability [30] [73]. Our results extend these findings by demonstrating that this relationship also holds in the context of TTE predic- tion, while further revealing that model fairness is influenced by additional factors beyond subgroup separability. We observe that while is often possible to reduce DHH, Da. Da') during training- example, by learning fair representations across groups-miniminn minimizing the term n)H, fa. fa') may remain challenging. In particular, when the labeling functions fo and fo' differ significantly across groups (Figure 2p), even the optimal choice of hypothesis class H may not ensure small 1). In such cases, the upper bound on fairness error becomes large, implying that FEx (h) may also be large. Conversely, when fa and fa' are similar-as as in Figure 2h or when the only causal pathway from A is to X in Figure 2p tte upper bound becomes small, indicating that (g)(h) can also be small. In these settings, fairness can be achieved by learning invariant representations across groups. We formally state this observation as follows.
Proposition 2. Let Assume the following conditions hold:
Z be mapping from the input space X to a representation space Z.
(i) Data distribution disparity across groups is a covariate shift, i.e., P(t|æ,, a) = P(f]z), A.
(ii) Representation Z is sufficient for an arbitrary group a E A, i.e., (2( Ia (X,,))), where Tak:;)) :) denotes the mutual information computed over the distribution Da-
Then, distribution shift across groups w.r.t representation Z is also a covariate shift, that is, P( a) = P(4|=), Va E A.
Discussion of Proposition 2 This proposition suggests that under covariate shift (i), fair TTE prediction can be achieved by learning a fair representation Z across groups (i.e.. P(zla) P(z), Va E A). More clearly, P(z]a) = P(z), Va E A guarantees small D(H, Dar) while P(t|z, a) P(t|2), Va E A indicates small (H, far far). We note that the sufficiency condition imposed on Z (ii) is practical: it only needs to hold for one group. and during training we have access to labeled data. Moreover, the dimension of T is often smaller than that of Z, making the sufficiency assumption more attainable in practice.
To further investigate the influence of sensitive attributes on the fairness of TTE prediction models, we utilize our proposed causal framework to decompose the group-specific labeling functions. Following Bayes' theorem, we rewrite the labeling function fa w.r.t. each group as:
P(t|æ, a) = P(xlt, a) P(tla) P(ra 1, a) P(æ=(t,a) P(tla) = P(æzlt,a) P(tla) (3) P(zzla)
P(rla)
=
P(zalæs,, a) P(æzla)
Note that in practical TTE prediction scenario, we observe Y and A rather than T. Therefore, the TTE prediction model estimates P(y, 8|z) instead. Replacing with y and 5 in Eq. 33) we obtain:
Ply. 6|2, a) = P(xzly, 5. P(y. 8la)
P(rzlu. 0,
P( (0,0) a)
P(y P(8)a)
(4)
P(zz(a)
P(xz)
P(rz a)
TTE
censoring
PMI(F.W)
PMI(Z,.6)
5

---

where the first and second terms represent the pointwise mutual information (PMI) between X2 and Y (conditioned on Á). and between X and A, respectively. The third term corresponds to the TTE distribution, while the fourth term accounts for the censoring rate. As shown in Figure 20. when A correlated with other random variables in the causal graph, alters these four terms, resulting in P(y. ,|z, a) * P(v, a').
Sources of Bias in Fair TTE Prediction. Based on Theorem D and the decomposition formula in Eq. 4 we can identify five primary sources of bias across groups in fair TTE prediction: 1) disparity in image feature distributions, 2) disparity in mutual information between Xz and Y. 3) disparity in mutual information between Xz and A, 4) disparity in the TTE distributions, and 5) disparity in the censoring rates. In practice, we observe that TTE datasets often contain multiple sources of bias rather than just one. These five cases represent the fundamental sources of bias and are crucial for understanding the complex biases present in real-world TTE data.
3 Experimental Setup for Fair TTE Prediction
We propose FaiTTTE, reproducible and user-friendly framework for evaluating fairness algorithms in TTE prediction within medical imaging. Our framework includes large-scale experiments conducted on three real-world medical image datasets, covering diverse imaging modalities and TTE outcomes, with up to three sensitive attributes considered for each dataset. Using these datasets, we evaluate three TTE prediction models and five fairness algorithms in the context of fair TTE prediction.
Datasets. FairTTE includes MIMIC-CXR [7 for predicting in-hospital mortality from chest X-ray images, ADNI 49 for predicting Alzheimer's 's disease from brain MRI images, and AREDS 14] for predicting late AMD from color fundus images. We selected these datasets for our benchmark based on several key criteria: the availability of temporal information to derive TTE outcomes, the diversity of medical imaging modalities, the presence of various potential sources of bias, the availability of sensitive attributes, and the range of dataset sizes. Table Al provides basic information about these datasets, with additional details on data access and TTE label construction provided in Appendix C
Algorithms. TTE prediction models: We employ DeepHit [3 Nnet-survival 15 and PMF [40] as the base models for TTE prediction to investigate the fairness issue in medical image prog- nosis. Fairness algorithms: We incorporate five SOTA fairness algorithms and adapt them to the TTE prediction setting to promote equitable predictions. These algorithms span a wide range of learning strategies and are categorized into three main groups: 1) pre-processing: subgroup rebal- ancing (SR) [31], 2) in- in-processing: domain independence (DI) [71). fair representation learning (FRL) 700, distributionally robust optimization (DRO) [22), and post-processing; controlling for sensitive attributes (CSA) [51]. To the best of our knowledge, this is the first study to compre- hensively evaluate such diverse set of fairness algorithms for TTE prediction in medical imaging. Detailed descriptions of each algorithm are provided in Appendix D
Evaluation Metries. Compared to classification tasks, evaluating TTE prediction, particularly in medical contexts, requires careful consideration. The primary challenge in assessing accuracy TTE prediction is censoring, which complicates the use of standard evaluation metrics. As result, no single evaluation metric is universally ideal for all TTE prediction scenarios.
To address these challenges, we adopt multiple evaluation metrics to ensure a comprehensive as- sessment of TTE predictive performance. These metries can be grouped into two categories: 1) ranking-based metrics which evaluate the ranking of patients in the dataset based on their correspond- ing survival times: time-dependent C-index (Ctd) M.. time-dependent AUC (AU 667 and 2) squared error which measures the error between estimated survival times and ground-truth values: Integrated Brier score (IBS) [66. For each performance metric, we consider a corresponding fairness metric. A detailed description of all metrics can be found in Appendix E
Model Selection. Prioritizing faimess often involves trade-off with utility, as the model's objective shifts from utility to balancing both utility and fairness. Consequently, model selection becomes critical in determining the optimal trade-off. In our setting, we perform a hyperparameter search
"In our context, sources of bias refer to disparities in the data probability distributions across groups, which can be quantified from observed data. However, identifying causal pathways between the sensitive attribute and other variables in the causal graph (b) remains challenging and requires deep clinical insight.

---

and select the best fair TTE prediction models based on faireess performance on the validation set, allowing for up to a 5% reduction in predictive performance compared to the base TTE models.
Implementation Details. We use 2D EfficientNet [61] backhone for the AREDS and MIMIC-CXR datasets, and a 3D ResNet-18 backbone [65] for the ADNI dataset. These lightweight backbones are chosen to mitigate overfitting. In addition to training models from scratch, we also explore pre-training using weights from models trained on the ImageNet [T0] and Kinetics [35] datasets. To
ensure stability across random initializations, we perform a hyperparameter search with 10 random seeds for each combination of dataset, sensitive attribute, algorithm, and evaluation metric. More implementation details can be found in Appendix E.2 and in our code repository.
Experiment and Result
Bias Across TTE Prediction Models in Di- verse Imaging Modalities and Outcomes. We first train TTE prediction models, including DeepHit, Nnet-survival, and PMF, on various datasets and sensitive attributes, then select the best models based on their performance on the validation sets. For each dataset and sensitive attribute pair, we report predictive performance using metrics including Otd AU Crddd and I IBS, along with fairness the performance gap be- tween the best and worst groups. As shown in Figure 4 these performance gaps (measured by (idd) are prevalent across all datasets and sensi- tive attributes. From this figure, we also observe that these gaps are more pronounced for age and race compared sex across all datasets. While bias in model predictions has been extensively discussed in the context of medical classification and segmentation, as well as TTE prediction for tabular data, has not been systematically quan- tified for TTE prediction in medical imaging. This study provides the first comprehensive analysis across a wide range of imaging modalities, TTE outcomes, and sensitive attributes.
1.0
Group 1
0.9
Age
Scx
Age
Race
Sox
Age
Race
SeA
AREDS
MIMIC-CXR
Figure Per-group performance (Ctd) of TTE pre- diction models across various datasets and sensi- tive attribute combinations. The visualized perfor- mances correspond to the best models determined by model selection conducted on validation sets. The 95% confidence intervals (CIs) are calculated using bootstrapping over test sets. Definitions of groups O and are provided in Appendix F.2
Statistical Tests We conduct statistical tests to verify the robustness of our findings against the variability in TTE prediction models and hyperparameters. Specifically, for each combination of dataset, sensitive attribute, and evaluation metric, we perform two-sided Wilcoxon signed-rank test 6 on the results across all TTE prediction mod- els and hyperparameters to identify significant differences in predictive performance between groups (p-value < 0.05). As illustrated in Fig- ure A7 our analysis reveals significant perfor- mance disparities between groups across all ex- perimental settings, highlighting the importance of considering fairness in TTE prediction tasks.
ADNI
Ago
Sex
Age
AREDS
Race
Sex
MIMIC-OXA
Age
Quantifying Sources of Bias. To better derstand the unfair behavior of TTE prediction models, quantify the degree of each source of bias in various datasets and sensitive attribute settings. Specifically, estimate disparities in PMI between RVs across groups, we compute normalized mutual information scores. To eval- uate disparities in image features and TTE out- comes across groups, we calculate the Wasser- stein distance [68] Lastly, disparities in censor-
Race
Sex
0.0
0.1
0.0
0.3 0.0
Cenaoring
TTE
Image
PM](a.a)
PM
Figure 5: Quantification of the degree of various sources of bias across all datasets and sensitive attributes. Bias degrees range from 0 to I, where indicates bias and represents maximum bias within the datasets.
http::///gthabbbcc////t11///////
7

---

ing rates are quantified by examining differences in the ratio of censored data across groups. Detailed methods for quantifying sources of bias can be found in Appendix F.4
Figure 5 illustrates the degree of bias across these five sources for each dataset and sensitive attribute setting. The results reveal correlation between the degree of bias and the fairnsss performance of TTE prediction models. For instance, in the AREDS dataset, the degree of bias in terms of disparities in censoring rates, PMI between Xz and Y, and in PMI between X and A is significantly higher when considering race as the sensitive attribute compared to sex or age. This finding aligns with Figure where the performance gap between racial groups is considerably larger than the gaps between sex or age groups. Additionally, we observe that settings with greater disparities in image features across groups exhibit larger performance gaps. This observation aligns with prior findings in medical image classification tasks, where higher subgroup separability is associated with greater performance disparities
30
The Role of Pre- Training, Pre-training on large datasets has been proven effective in many applica- tions [2]. In this study, we investigate the impact of pre-training on both the accuracy and fairness of TTE prediction models. As shown in Figure pre-training outperforms training models from scratch in terms of predictive performance. This improvement is particularly notable for the ADNI dataset, which is relatively small and contains more complex data structures (3D im- ages) compared to the AREDS and MIMIC- CXR datasets (2I) images). However, in terms of fairness, we do not observe a significant im- provement with pre-training compared to train- ing from scratch, as shown in Figure A9 Specif- ically, the p-values from one-sided Wilcoxon signed-rank tests are larger than 0.05 in 18 out of 24 settings, suggesting that pre-training does not lead to more equitable predictions in most cases. These results imply that while pre-training en- hances accuracy, combining it with fairness al- gorithms may be necessary to achieve both fairer and more accurate TTE prediction.
Grae
-005
Ac" (AONN)
-003
AC (cthers
6.15
Age
Sex
Age
Race
Sex
Age
Race
Sex
ADNI
AREDS
MIMIC CXR
Figure 6: Per-group average performance gap (AC')) for TTE prediction models using a pre- training strategy compared to training from scratch across various datasets and sensitive attribute com- binations. A positive A A indicates that the pre- training strategy enhances predictive performance relative to training from scratch.
Performance of Fairness Algorithms in TTE Prediction. We conduct experiment to evaluate the effectiveness of fairness algorithms in reducing disparities between groups in TTE prediction. In this experiment, we use DeepHit as the base TTE prediction model and integrate various fairness algorithms to achieve equitable predictions. To ensure a comprehensive evaluation, we select algorithms from pre- rrepprocsssing (i.e.. SR), in inooocessing (i.e., DI, FRL. DRO), and post-processing approaches (i.c., CSA). As shown in Table [ existing fairness methods reduce performance gaps between groups compared to DeepHit in settings where bias sources are substantial (e.g.. ADNI with age, AREDS with race, and MIMIC-CXR with age). However, when bias sources are small, enforcing fairness can sometimes exacerbate performance disparities. Moreover, no method consistently outperforms DeepHit across all settings. To verify the significance of our observations, we perform a Friedman test followed by a Nemenyi post-hoc test ( across all algorithms, datasets, and sensitive attribute settings. As shown in Figure A10 the results confirm that current fairness algorithms do not significantly outperform DeepHit mitigating bias. Additionally, we observe trade-offs in predictive performance in most cases where fairness interventions reduce group disparities: while certain sources of bias (e.g., disparities in image features) are mitigated, others often persist. These findings highlight the need for developing new fairness algorithms capable of addressing multiple sources of bias without compromising predictive performance.
Fair TTE Prediction Under Distribution Shift. Fairness algorithms are known to lose effectiveness Building on this insight, we investigate the impact of distribution shifts on fair TTE prediction. As illustrated in Figure 3 we define distribution shifts as scenarios where correlations between the sensitive attribute and other random variables in the causal graph exist in the training dataset but disappear in the testing dataset. To simulate such shifts, we manipulate the data generation process by corrupting images, TTE labels, or censoring indicators for one group while keeping the other group unchanged. The
under distribution shifts, partieularly in medical image diagnosis tasks 733 29
8

---

Table 1: Predictive and faimess performances of fairness algorithms across all dataset and sensitive attribute combinations. We report the actual predictive performance as percentages for DeepHit. For fair algorithms, relative changes in each metric compared to DeepHit are shown. The reported performances correspond to the best models selected via model selection conducted on the validation sets. Blue indicates positive changes, red indicates negative changes. Metries with & are better when lower, and with f when higher. Best fair algorithm performances are highlighted with gray cells
Model
Accuracy IBS
Paimess Fias
Accuracy 185
Faimess Fins
AUC
Cru
FADCI
Fou
AGC
Od
Fanc
Few
ADNI- Age
ADNI Sex
DeepHit
82.02
24,00
74,20
14.19
16.39
7.34
82.02
24,00
73.84
7.02
2.93
15.82
DRO
-6.40%
0.848
-6.611
-18.26%
-2.65%
-1:557%
-1.69%
23.569
-17.17%
-74,88%
-5.76%
0.90%
6.73%
-26.50%
-3.54%
1.99%
-1.67%
825
-71.74%
-24.78%
FRI.
-6.00%
18.63%
-2.55%
53.91%
60.6355
-35.8646
-1.75%
-12.58%
-1.98%
-2.95%
8.74%
-35.60%
DI
1.09
-14.88%
-2.19%
55.545
-37.51%
-66.2358
-15.81%
5.10%
-5.57%
14.09%
193.83%
7.49%
CSA
-.18%
-11.44%
-5.79%
-32.44%
-42.55%
-26.01
-3,89%
-4.90%
-10.38%
-0.51%
-50.43%
AREDS
Age
AREDS
Race
Despliät
78.41
15.37
$1.30
1.58
12.56
2,20
81.78
11.74
B4.53
14.00
10.14
11.09
DRO
0.88%
-1.37%
0.08%
19.19%
-20.08%
-5.03%
1.78%
-2.28%%
-1.41%
-34,89%
-10.33%
-37.21%
15.44%
-0.58%
-3.73%
11.97%
FRL
0.80%
-0.70%
0.20%
-25/11%
-8.21%
-35.01%
1.75%
4.41%
0.211%
-33,199
16.10%
-11.51%
DI
2.12%
-1.74%
0.73%
83.73%
-6.53%
9.891%
2,47%
-0.67%
-0.84%%
-8.5898
-9.6%%
-26.1198
CSA
0.89%
-5.44%
0.11%
-1.95%
29.56%
4.60%
1.37%
-1.32%
-0.231%
-10.74%
-8.14%
-21.97%
AREDS
MIMIC-CNE
Age
DoepHtin
79.08
15.36
81.77
0.76
3.84
1:32
78.61
20.38
76.21
3.06
1.49
5.93
DRO
1.21%
-3.62%
-0.54%
12.75%
6.525
995.7448
-1.5556
13,50
-1.61%%
-49.46%
58.14%
-17.81%
-0.07%
4.02%
-0.14%
47.46%
4.14%
-85.27%
-0.96%
2.75%
-0.37%
-82:53%
-8.31%
2.86%
FRI.
1.76%
-5.775
-0.54%
-5107%
-1.200%
23.24%
-4. 2.55
5.26%
-2.42%
-61.41%
-60.79%
-21.96%
DI
0.18%
-5844%
1.31%
-13. 82%
16.125%
-65.705
0.41%
5.29%
-0.63%%
-52.214
-44.27%
-10.24%
CSA
-0.18%
8.23%
0.46%
-28.46%
17.58%
-71.42%6
0.36%
-3.19%
-0.97%
-66.46%
.29.215
-13.72%
MIMIC-CXR Race
MIMIC-CXR- -Scs
DeepHit
78.61
21.10
76.21
2.89
0.87
4.03
78.61
20.38
76.21
4.77
1:23
2.09
DRO
-0.72%
3.13%
-0.78%
-5.69%
31.34%
-4.80%
-0.35%
4.51%
-1.20%
12.28%
92.27%
-9.61%
SR
-0.11%
-0.90%
-0.11%
10.28%
-71.377%
-0.30%
-0.12%
2.31%
-0.19%
17.02%
25.35%
-1.37%
FRL -0.1%%
2.44%
36.89%
-2.24%
0.21%
10.97%
DI -0.44%
0.00%
-0.52%
37,80%
30.77%
-0.43%
-1.94%%
1,21%
-0.34%
14.26%
-8.22%
-13776%
CSA
-0.129
4.61%
-0.96%
-14.83%
36.144%
0.445
-0.17%
-2.58%
-0.63%8
10.63%
-19.08%
-0.16%
real-world motivations and detailed descriptions of these causal distribution shifts are provided in
Appendices G.2 and F.5
Figure 7 compares the predictive performance of models under in-distribution and distribution shift learning scenarios for the MIMIC-CXL dataset, where sex is considered the sensitive at- tribute. The complete results for all settings are provided in Appendíx [.6] Our findings indicate that different types of distribution shifts impact model performance in distinct ways. Specifi- cally, adding noise to TTE labels (shift in Y) leads to significant decline in IBS, as the in-
Di
CSA
DRD
Deepltt
FRI
SR
Shifi X
0.760
0.776
0.220
0.10
0.760
0.780
0.765
6.750 0.756 0.762
0,200 0.200 0.166
Shin
Out-of dis-of-dittitttoon
0.216
0.786
0.208
0.783
5.750
0.200
creased label uncertainty makes accurate predic- tions more challenging. Flipping censoring in- dicators (shift in A), on the other hand, severely degrades ranking-based metrics such as OLd and Ald as it reduces the availability of com- parable pairs during training, making it harder for the model to learn an effective ranking func- tion. Additionally, introducing noise to medical images (shift in X) negatively affects all per- formance metries, as degraded image quality limits the model's 's ability to extract meaningful features. These observations align with our ex- iiirations-corrupptii TTE labels directly im- pacts metrics measuring error between predicted and ground-truth TTE, while censoring flips dis- rupt ranking-based evaluations by impairing the model's 's ability to differentiate survival times.
0.700
0.765
6,750
0.756
0.16
Shifi
0,762
0.785
5.756
0.204
0.780
0,700
0.785
0,750
8,356 0,762
0,200 0.200 0216
AUC
IB5
In-distrbution
Figure 7: Predictive performance comparison of fair TTE prediction models in ID vs. OOD set- tings on the MIMIC-CXE dataset with sex as the sensitive attribute. Results are averaged across all random seeds. Points on the dashed line indicate equal performance in both settings. For AU A and Cdd (resp. IBS), points below (resp. above) the line indicate degraded performance OOD,

---

Regarding faimess, our observations reveal mixed results. In some cases, distribution shifts negatively impact fairness, while others, they lead improvements. Notably, adding noise the data of one group generally degrades predictive performance on that group. However, we also observe that this noise can similarly affect the other performance gap between groups and unintentionally improving fairness. These findings underscore the complexities achieving fairness in the presence of distribution shifts and highlight the need for developing TTE algorithms that are both fair and robust in such challenging scenarios.
sometimes even more severely-therebe reducing the
5 Limitation and Broader Impact
While FairTTE represents a sigmificant step forward in advancing fairness research in medical image prognosis by providing the first comprehensive benchmark across multiple datasets and fairness algorithms, we acknowledge several limitations in the current study. Specifically, our analysis focuses on the standard TTE prediction setting, which assumes a single clinical risk and non-informative right censoring. In real- world clinical applications, however, patients often face multiple competing risks and may drop out of studies for reasons that introduce informative censoring, making the fairness landscape considerably more complex. Addressing fairness such settings remains an important direction for future research. Additionally, although adopt group faimess definitions (i.e.. minimizing performance gaps across subgroups), we recognize that strictly enforcing these criteria can, in some clinical scenarios, degrade overall utility harm performance for all groups. In such scenarios, alternative fairness notions may be more appropriate, depending on the clinical context and ethical objectives.
Selecting an appropriate fairness criterion in the medical domain requires careful consideration of the clinical context, ethical principles, and statistical validity. An effective fairness notion should align with the model's intended use, its potential impact on different patient subgroups, and real-world constraints. In our study, we focus on statistical group fairness (i.e., predictive performance gaps across subgroups) which is widely examined in the medical image analysis literature. This choice is grounded in the assumption that any causal pathway from sensitive attributes represents unfair influence in the causal graph and should be mitigated through fairness constraints. However, we acknowledge that in practical clinical scenarios, such causal pathways may reflect fair and clinically meaningful relationships. Enforcing group fairnsss in these cases may inadvertently remove relevant information and degrade predictive performance.
Moving forward, we suggest several directions for fairness research medical image analysis: (1) Identifying the causal nature of bias to distinguish between fair and unfair sources, enabling models to address specific pathways appropriately: (2) Developing fairness metrics and mitigation strategies that preserve clinically relevant (fair) pathways while minimizing the effect of unfair ones; and (3) Collaborating with clinicians and domain experts to define context-specific fairness objectives that aligned with both clinical utility and ethical standards.
6 Conclusion
We systematically investigate fair TTE prediction in medical imaging, introducing a unified frame- work to define and quantify bias sources and establishing comprehensive and realistic evaluation. Our framework includes three TTE prediction methods, five fairness algorithms, and three large-scale public datasets spanning diverse modalities and outcomes. Extensive experiments reveal critical insights, including the link between bias sources and model unfairness, the inconsistent effectiveness of existing fairness methods, the role of pre-training strategies, and the challenges of causal distribu- tion shifts. We hope our fine-grained analysis encourages more rigorous evaluations and drives the development of new fairness algorithms for TTE prediction.
Acknowledgements
This work was funded in part by the National Science Foundation under award number IIS-2145625 and by the National Institutes of Health under awards number RO1AI188576 and RO1CA301579.
10

---

References
[1] Laura Antolini, Patrizia Boracchi, and Elia Biganzoli. A time-dependent discrimination index for survival data, Statistics in medicine, 24224),9977-3944 2005.
[2] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXtv:2108.072558 2021.
[3] Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sliced and radon wasser- stein barycenters of measures. Journal of Mathematical Imaging and Vision, 51:22-45, 2015.
[4] Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney. Optimized pre-processing for discrimination prevention. Advances in neural information processing systems, 30, 2017.
[5] George H Chen. An introduction to deep survival analysis models for predicting time-to-event outcomes, arXiv preprint arXiv:2410.0106 2024.
[6] William Jay Conover. Practical nonparametric statistics, volume 350. john wiley sons, 1999.
[7] Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd acm sigkdd international conference knowledge discovery and data mining, pages 797-806, 2017.
[8] Matthew D Davis, Ronald E Gangnon, Li Yin Lee, Larry D Hubbard, BE Klein, Ronald Klein, Frederick Ferris. Susan B Bressler, Roy Milton, al. The age-related eye disease study severity scale for age-related macular degeneration: Areds report no. 17. Archives of ophthalmology (Chicago, III.: 1960), 123(11): 1498, 2005.
[9] Janez. Demšar. Statistical comparisons of classifiers multiple data sets. The Journal of Machine learning research. 7:1-30, 2006.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Iece, 2009.
[11] Hyungrok Do, Yuxin Chang, Yoon Sang Cho, Padhraic Smyth, and Judy Zhong. Fair survival time prediction via mutual information minimization. In Machine Learning for Healthcare Conference, pages 128 149. PMLR, 2023.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2024.
[13) Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214-226, 2012.
[14] Frederick L Ferris, Matthew Davis, Traci Clemons, Li-Yin Lee, Emily Y Chew, Anne Lindblad, Roy C Milton, Susan B Bressler, Ronald Klein, et al. A simplified severity scale for age-related macular degeneration: Areds report no. 18. Archives of ophthalmology (Chicago. Ill.: 1960), 123(1 13:111):55001157 2005.
[15] Michael F Gensheimer and Balasubramanian Narasimhan. scalable discrete-time survival model for neural networks. PeerJ, 7:e6257, 2019.
[16] Erika Graf, Claudia Schmoor, Willi Sauerbrei, and Martin Schumacher. Assessment and comparison of prognostic classification schemes for survival data. Statistics medicine, 18(17-18):2529-5555 1999.

---

[17] Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, and Omar Badri. Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1820-1828, 2021.
[18] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
[19] Frank E Harrell, Robert M Califf, David B Pryor, Kerry L Lee, and Robert A Rosati. Evaluating the yield of medical tests. Jama, 247(18))2543-22 2546, 1982.
[20] Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographies in repeated loss minimization. In International Conference on Machine Learning, pages 1929-1938. PMLR, 2018.
[21] Gregory Holste, Mingquan Lin, Ruiwen Zhou, Fei Wang, Lei Liu, Qi Yan, Sarah Van Tassel, Kyle Kovacs, Emily Y Chew, Zhiyong Lu, et al. Harnessing the power of longitudinal medical imaging for eye disease prognosis using transformer-basee sequence modeling. NPJ Digital Medicine, 7(1):216 2024.
[22] Shu Hu and George Chen. Fairness survival analysis with distributionally robust optimiza- tion. Journal of Machine Learning Research, 25(246):1-88 2024.
[23] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference artificial intelligence, volume 33, pages 590-597, 2019.
[24] Ray Jiang, Aldo Pacchiano, Tom Stepleton, Heinrich Jiang, and Silvia Chiappa. Wasserstein fair classification. In Uncertainty in artificial intelligence, pages 862-872. PMLR, 2020.
[25] Yibo Jiang and Victor Veitch. Invariant and transportable representations for anti-causal domain shifts. Advances in Neural Information Processing Systems, 35:20782-20794 2022.
[26] Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
[27] Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih- ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. Mimic-cxt-jpg, large publicly available database of labeled chest radiographs. arXiv preprint 19Xiv:1901.07042, 2019.
[28] Charles Jones, Daniel C. Castro, Fabio De Sousa Ribeiro, Ozan Oktay, Melissa McCradden, and Ben Glocker. A causal perspective on dataset bias in machine learning for medical imaging. Nature Machine Inielligence, 6: 6:18-146, 2024.
[29] Charles Jones, Fabio de Sousa Ribeiro, Mélanie Roschewitz, Daniel C Castro, and Ben Glocker. Rethinking fair representation learning for performaace-sensittve tasks. In The Thirteenth International Conference on Learning Representations, 2025.
[30] Charles Jones, Mélanie Roschewitz, and Ben Glocker. The role of subgroup separability in group-fair medical image classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 179-188. Springer, 2023.
[31] Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrimination. Knowledge and information systems, 33(1):1-33, 2012.
[32] Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairness-aware clas- sifier with prejudice remover regularizer. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 23, pages 35-50. Springer, 2012.
12

---

[33] Edward L Kaplan and Paul Meier. Nonparametric estimation from incomplete observations. Journal of the American statistical association, 53(282):457-481, 1958.
[34] Jared L Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger. Deepsurv: personalized treatment recommender system using a cox proportional hazards deep neural network. BMC medical research methodology, 18:1-12, 2018.
[35] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint /70.:06950. 2017.
[36] Kamrun Naher Keya, Rashidul Islam, Shimei Pan, Ian Stockwell, and James Foulds. Equitable allocation of healthcare resources with fair survival models. In Proceedings of the 2021 siam international conference on data mining (sdm). pages 190-198. SIAM, 2021.
[37] Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn: Training deep neural networks with biased data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9012-9020, 2019.
[38] Newton M Kinyanjui, Timothy Odonga, Celia Cintas, Noel CF Codella, Rameswar Panda, Prasanna Sattigeri, and Kush R Varshney. Fairness classifiers across skin tones dermatology. In International Conference on Medical Image Computing and Computer- Intervention, pages 320-329. Springer, 2020.
[39] Oleksandr Kovalyk, Juan Morales-Sánchens Rafael Verdúi-Monedero Inmaculada Sellés- Navarro, Ana Palazón-Caba Cabanes, and José-Luis Sancho-Gómez. Papila: Dataset with fundus images and clinical data of both eyes of the same patient for glaucoma assessment. Scientific Data, 91):2911 2022.
[40] Hävard Kvamme and Ørnulf Borgan. Continuous and discrete-time survival prediction with neural networks. Lifetime data analysis, 27(4):710-73) 2021.
[41] Håvard Kvamme, Ørnulf Borgan, and Ida Scheel. Time-to-event prediction with neural networks and cox regression. Journal of machine learning research, 20(129):1 1-11:9931333. 2019.
[42] Agostina Larrazabal, Nicolás Nieto, Victoria Peterson, Diego H Milone, and Enzo Ferrante. Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proceedings of the National Academy of Sciences, 117223):1252--1254 2020.
[43] Changhee Lee, William Zame, Jinsung Yoon, and Mihaela Van Der Schaar. Deephit: A deep learning approach to survival analysis with competing risks. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[44] Yan Luo, Yu Tian, Min Shi, Louis R Pasquale, Lucy Q Shen, Nazlee Zebardast, Tobias Elze, and Mengyu Wang. Harvard glaucoma fairness: retinal nerve disease dataset for fairness learning and fair identity normalization. IEEE Transactions on Medical Imaging, 2024.
[45] David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. In International Conference on Machine Learning, pages 3384-3393. PMLR, 2018.
[46] Trishan Panch, Heather Mattie, and Rifat Atun. Artificial intelligence and algorithmic bias: implications for health systems. Journal of global health, 9(2), 2019.
[47] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems. 32, 2019.
[48] Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CambridgeUniveesityPeess 19(2):3, 2000.
13

---

[49] Ronald Carl Petersen, Paul S Aisen, Laurel A Beckett, Michael C Donohue, Anthony Collins Gamst, Danielle J Harvey, CR Jack Jr. William J Jagust, Leslie M Shaw, Arthur W Toga. et al. Alzheimer's s disease neuroimaging initiative (adni) clinical characterization. Neurology, 74(3):201-203) 2010.
[50] Sebastian Pölsterl, scikit-survival: A library for time-to-event analysis built on top of scikit-learn. Journal of Machine Learning Research, 21(212):16, 2020.
[51] Devin G Pope and Justin R Sydnor. Implementing anti-diserimination policies in statistical profiling models. American Economic Journal: Economic Policy, 3(3):206-231, 2011.
[52] Shi-ang Qi, Weijie Sun, and Russell Greiner. SurvivaEEVAl: A comprehensive open-source python package for evaluating individual survival distributions. Proceedings of the AAAI Symposium Series, 2:453-4555 457, 01 2024.
[53] Md Mahmudur Rahman and Sanjay Purushotham. Fair and interpretable models for survival analysis. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1452-1462, 2022.
[54] María Agustina Ricci Lara, Rodrigo Echeveste, and Enzo Ferrante. Addressing fairness in artificial intelligence for medical imaging. nature communications, 13(1):4581, 2022.
[55] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2019.
[56] Laleh Seyyed- Kallntari, Guanxiong Liu, Matthew McDermott, Irene Y Chen, and Marzyeh Ghassemi. Chexclusion: Fairness gaps in deep chest x-ray classifiers. In BIOCOMPUTING 2021: proceedings of the Pacific symposium, pages 232 243. World Scientific, 2020.
[57] Laleh Ghassemi. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature medicine, 27(1)) 21822 2021.
Kalantari, Haoran Zhang, Matthew BA McDermott, Irene Chen, and Marzyeh
[58] Ammar Shaker and Carolin Lawrence. Multi-source survival domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 9752-9762, 2023.
[59] Saeed SharifiMMalvajerue Michael Kearns, and Aaron Roth. Average individual faimess: Algorithms, generalization and experiments. Advances in neural information processing systems, 32, 2019.
[60] Changjian Shui, Gezheng Xu, Qi Chen, Jiaqi Li, Charles X Ling, Tal Arbel, Boyu Wang, and Christian Gagné. On learning fairness and accuracy on multiple subgroups. Advances in Neural Information Processing Systems, 55:34121-34133 2022.
[61] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105
3105-61144 14. PMLR, 2019.
[62] Enzo Tartaglione, Carlo Alberto Barbano, and Marco Grangetto. End: Entangling and disentan- gling deep representations for bias correction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13508- 13517, 2021.
[63] Matthew Thorpe. Introduction to optimal transport. Notes of Course at University of Cambridge, 2018.
[64] Yu Tian, Min Shi, Yan Luo, Ava Kouhana, Tobias Elze, and Mengyu Wang. Fairseg: A large-scale medical image segmentation dataset for fairness learning using segment anything model with fair error-bound scaling. In The Twelfih International Conference on Learning Representations, 2023.
[65] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray. Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450-6455 2018.
14

---

[66] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
hammm00000 10000 dataset, large collection
[67] Hajime Uno, Tianxi Cai, Lu Tian, and Lee-Jen Wei. Evaluating prediction rules for t-year survivors with censored regression models. Journal of the American Statistical Association, 1027478):527-537, 2007. 2007.
[68] Cédric Villani and Cédric Villani. The wasserstein distances. Optimal transport: old and new, pages 93-111, 2009.
[69] Tan Wang. Zhongqi Yue, Jianqiang Huang, Qianru Sun, and Hanwang Zhang. Self-supervised learning disentangled group representation as feature. Advances in Neural Information Process- ing Systems, 34: 14:18225-18240, 2021.
[70] Yuyan Wang. Xuezhi Wang, Alex Beutel, Flavien Prost, Jilin Chen, and Ed H Chi. Understand- ing and improving ttirness-accurae tradc-offs in multi-task learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1748-1757, 2021.
[71] Zeyu Wang. Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness in visual recognition: Effective strategies for bias mitigation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8919-8928, 2020.
[72] Zichong Wang, Jocelyn Dzuong, Xiaoyong Yuan, Zhong Chen, Yanzhao Wu, Xin Yao, and Wenbin Zhang. Individual faimess with group awareness under uncertainty. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 89-106. Springer, 2024.
[73] Yuzhe Yang. Haoran Zhang, Judy W Gichoya, Dina Katabi, and Marzyeh Ghassemi. The limits of fair medical imaging ai in real-world generalization. Nature Medicine, 30(10))2888-2848, 2024.
[74] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. Fairness constraints: Mechanisms for fair classification. In Artificial intelligence and statistics, pages 962-970. PMLR, 2017.
[75] Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 335-340, 2018
[76] Wenbin Zhang, Tina Hernandez- Bernandez-Boussard. and Jeremy Weiss. Censored fairness through awareness. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 14611-14619, 2023.
[77] Wenbin Zhang, Zichong Wang. Juyong Kim, Cheng Cheng, Thomas Oommen, Pradeep Raviku- mar, and Jeremy Weiss. Individual fairness under uncertainty. In ECA/ 2023, pages 3042-3049, IOS Press, 2023.
[78] Wenbin Zhang and Jeremy C Weiss. Fair decision-making under uncertainty. 2021 IEEE international conference on data mining (ICDM), pages 386-895. IEEE, 2021.
[79] Wenbin Zhang and Jeremy C Weiss. Longitudinal faimess with censorship. In proceedings of the AAAI conference on artificial intelligence, volume 36, pages 12235-122433 2022.
[80] Zhouting Zhao and Tin Lok James Ng. Fairness-aware processing techniques in survival analysis: Promoting equitable predictions. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pages 460-476. Springer, 2023.
[81] Yuyin Zhou, Shih-Cheng Huang, Jason Alan Fries, Alaa Youssef, Timothy J Amrhein, Marcello Chang, Imon Banerjee, Daniel Rubin, Lei Xing, Nigam Shah, et al. Radfusion: Benchmarking performance and fairness for multimodal pulmonary embolism detection from et and ehr. arXiv preprint arXiv:2111.11665. 2021.
15

---

[82] Yongshuo Zong, Yongxin Yang, and Timothy M Hospedales, Medfair: Benchmarking faireess for medical imaging, In The Eleventh International Conference on Learning Representations. 2023.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims presented in the abstract and introduction accurately reflect the paper's contributions and scope. They are supported by both theoretical formalyy stated and proven and comprehensive experimental evaluations across multiple datasets, metrics, and fairness algorithms. We clearly distinguish between established findings and broader motivations, and explicitly state assumptions and limitations to ensure that the claims are appropriately contextualized and not overstated.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: explicitly discuss the assumptions made in our theoretical frame- work-such triangle inequality and symmetric properties for evaluation metric-and their potential violations in real-world scenarios. We also acknowledge limitations in generaliz- ability, experiments conducted finite set of publicly available medical imaging datasets. These limitations help contextualize our contributions and inform directions for future research.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
16

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed
penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The assumptions for Theorem and Proposition 2 are explicitly stated in Section 2.2, and complete, formal proofs are provided in Appendix B. All theoretical results are clearly numbered, rooss-reference, and presented with the necessary assumptions to ensure clarity and correctness. Where appropriate, we also include intuitive explanations in the main text to aid understanding.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Thcorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The experimental design and implementation details necessary reproduce the main results are thoroughly documented in Section and Appendix F. This includes dataset descriptions, preprocessing steps, model architectures, training protocols, hyperparameters. and evaluation metrics. We also describe our experimental setup across all dataset and sensitive attribute combinations. These details are sufficient for reproducing our findings, even without access to the code or data at submission time. Code and documentation will be released publicly to further facilitate reproducibility upon publication.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility also be provided via detailed
17

---

instructions for how to replicate the results, access to a hosted model (e.g., g. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide open access to all code, documentation, and scripts necessary to reproduce our main experimental results. Detailed instructions for setting up the environment, accessing and preprocessing the datasets (including MIMIC-CRR, ADNI, and AREDS), and running the experiments are included in Appendix F and will be made publicly available upon publication. For datasets with access restrictions, we provide clear guidance on how to obtain access through the appropriate data use agreements in Appendix includes scripts for training, evaluation, and statistical analysis, along with configuration files for all reported experiments to ensure faithful reproduction of our results.
Our released code
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
swinnooiccoyycc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
18

---

Answer: [Yes]
Justification: The experimental settings, including data splits, hyperparameter configurations, optimization strategies, and model selection procedures, are comprehensively detailed in Appendix F. These include descriptions of how hyperparameters were tuned, the choice of optimizer, training schedules, and evaluation protocols. This information is sufficient to understand and contextualize the reported results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report error bars using bootstrapping across multiple runs to capture variability due to model initialization, train/validation splits, and random seeds. Specifically, we calculate 95% confidence intervals based on bootstrapped samples to quantify uncertainty in model performance. Additionally, we assess statistical significance using well-established non paraamamctiicc tests, including the Wilcoxon signed-rank test for pairwise comparisons, and the Friedman test followed by the Nemenyi post-hoc test for comparing multiple methods across datasets and sensitive attributes.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of eтrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details on compute resources used for all experiments are provided Ap- pendix F.1. This includes the type of hardware, memory specifications, and estimated runtime for each experimental run. This information is sufficient to support reproducibility and assess the computational requirements of our work.
19

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wwe..ss..cuuuull//////aa
Answer: [Yes]
Justification: We have thoroughly reviewed the NeurlPS Code of Ethics and confirm that rescarch adheres to all applicable guidelines. Our study involves no human subjects, privacy risks, or potentially harmful applications, and we have taken appropriate steps to ensure transparency, reproducibility, and responsible use of data and models.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents foundational research on the methodological evaluation of fairness in TTE prediction using medical imaging. The work is primarily intended to inform academic research and improve scientific understanding bias in prognostic modeling. does not introduce new systems intended for deployment, nor does it involve the creation or use data or models with direct societal impact. As such, we consider the broader societal implications beyond the scope of this work and have not included a dedicated discussion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
20

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external assets used in this paper-including datasets (MIMIC-CXR, ADNI, and AREDS) and pretrained models (e.g., RETFound and EffiicintNeet- are properly cited with their corresponding publications. We use publicly available versions of these datasets and models in accordance with their respective licenses and terms of use. We include dataset description and access URLs in the Appendix C.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, wwwppsawitthteee thcode..ttccee.... com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
21

---

Justification: We release new assets part of this work, including benchmark code, evalua- tion scripts, and structured documentation for fair TTE prediction across multiple medical imaging datasets. These assets are well-documented and include detailed instructions for dataset preprocessing, model training, evaluation, and reproduction of results. All assets are anonymized for the review process and will be made publicly available upon publication. No personally identifiable information or private data included, and all released components comply with the data use agreements of the underlying datasets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This study does not involve crowdsourcing or research with human subjects; therefore, IRB approval not required.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard component; therefore, a declaration of LLM usage is not applicable.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
23

---

Appendix Contents
Related Works
25
B Missing Proof
26
Dataset Details
28
AREDS
28
C.2 MIMIC-XXR
29
ADNI
29
D Algorithms Details
30
D.I TTE Prediction Models
30
D.2 Fairness Algorithms
D.3 Model Architecture Details
32
Evaluation Metrics
32
Performance Metries
32
Fairness Metrics
34
E.3
Fairness- Ctiines-Utilityy Trade Off Metric
34
Experimental Setup Details
34
Implementation Details
34
F2
Data Split and Pre-processing
................................
35
Hyperparameter
................................
................................
35
Quantilying Source of Bias
35
F.5
Constructing Causal Distribution Shift
36
Causal Graphs for Fairness in TTE Prediction
37
G.1
Causal Graphs for Biased and Unbiased Settings
37
G.2
Real-world Causal Graph Examples for Fairness in TTE Prediction
37
Additional Results
40
H.
Predictive Performance and Fairness in TTE Prediction Models
40
H.2
Comparison between Pre- Training and Training from Scratch Strategies for TTE Prediction Models
H.3
Comparison with Advanced Image Backbones and Medical Pre training for TTE Prediction Models
43
H.4
Fairness in Fair TTH Prediction Models
44
H.5
Pairness-Utility Trade-Offf Results
46
H.6
Additional Results for Predictive Performance and Fairness in Fair TTE Prediction
Models under Distribution Shill
48
24

---

A Related Works
TTE Prediction. TTE prediction models can generally be classified into two categories: continuous- time and discrete-time models, each with distinct approaches for handling event timing. Continuous- time models treat time as a continuous variable and often extend traditional models like Cox re- gression. For example, DeepSurv [4]] extends the Cox regression by using deep neural network with non-linear activation functions hidden layers. Cox-Time [41] further builds on DeepSurv, introducing time-dependent predictors that allow for the estimation of time-varying effects. In con- trast, discrete-time models treat time as a series of distinct intervals and typically use classification techniques. DeepHit [43] learns survival times directly without assuming specific underlying stochastic process, parameterizing the discrete probability mass function. Another method, Nnet- survival [15). parametrizes the discrete hazard function using neural network and optimizes the negative log- likeiiili loss.
Fairness in Machine Learning. Faimess in machine learning has gained significant attention in recent years, with a focus on ensuring models are unbiased and equitable across individuals and groups. Fairness metrics. Fairness metrics can be broadly categorized into two types: group faimesss 13 [4 18] and individual fairness 13 (5). Group fairness ensures that models are fair across different demographic groups, while individual fairness emphasizes that similar individuals should be treated similarly. Fairness algorithms. To address fairness and bias issues, bias mitigation methods are generally classified into three approaches: pre-processing, which focuses modifying the input data before model training [3 4]: in-processing, which incorporates fairness constraints during model training
32 75
45
60
and post-processing, which adjusts model outputs to improve
fairness 18 24).
Fairness in Medical Imaging. In medical image analysis, machine learning (ML) models have been shown to exhibit systematic biases related to various attributes such as race, gender, and age [57 42
These biases are prevalent across different medical imaging modalities, including chest X-rays 566
CT scans [8]] and skin dermatology images [38. While several efforts have been made to benchmark
fairness algorithms on medical images, existing datasets [33 39 66 17 and benchmarks
82 64
primarily focus on diagnostic tasks like image classification and segmentation. Unfortunately, they often overlook the crucial domain of medical prognosis, which involves predicting TTE outcomes.
Fairness in TTE Prediction. Despite significant advances in TTE prediction, research on fair- ness in this area remains limited. Fairness metrics for TTE prediction. Fairness metrics for TTE prediction have only recently been defined. These metrics can be roughly classified into three categories based on their objectives: (i) ensuring similar predicted TTE outcomes for similar 72), (ii) ensuring similar predicted outcomes for data points from 80 and (iii) ensuring similar predictive performance across different 78 (9). Fairness algorithms for TTE prediction. Building on these metrics, several methods have been proposed to achieve fair TTE prediction. One approach incorporates fairness as regularization term during model training [36 [33 [1I, ensuring that the model accounts for fairness constraints throughout its optimization process. Another approach focuses on improving worst-group accuracy by leveraging distributionally robust optimization techniques [2 20 55), which aim to enhance performance for underrepresented or disadvantaged groups. In addition to these in processing methods, recent work has also explored pre- and post-processing strategies to address fairness in TTE prediction [80]. However, these efforts are limited to tabular data and fail to consider medical images, which essential and pervasive in medical prognosis tasks.
data points [36, 53
77
76
different groups 36
53
groups 22
25

---

B
Missing Proof
Proof for Theorem
For any a, a' E A, we have:
Er(fa, h, De:) s Er(fa h, Da) [Br(Sar h Do) Er(fa' , h Da:)
Er(Ja; h, D2) [Br(Ja h1 Da) Er(h, h;, D&)l
[Er(b, h², Da) Er(h, h² Dw:) + [Er(h, ht," Da.) Er(fw, h, Dar)
2 Er(fa; h, Da) Er(fa, h" Da) + D(H, Da. Dwr) ) + Er(far, h², Da')
@ Er(fa; h, Dz) + (HH, NH, fa' Do: Da))
where h- arg minnen (Er(fa, h1, D.) + Er(far. h1, Do.)). We have S by using inequality la (2)
bl S |a| |b|: S by using triangle inequality for Er metric and [Er(h, h", D.) Er(h, h", Da s
Maxx"MMER [Er(h, h², Da) Er(h', h", Dar)l D(H, Da, Da:); (3) because 1)(H, Say
(r([J,,", DD) + Er(Ja,, h," Da by definition. Subtracting Er(Ja, h, Da) from both sides and taking max operator, we have:
FE. (h) = max [Er (fa, Da) - Er (far, h, Dall S max (m, fa. fa: a.a'EA
far + Da, Dw
a.a'E.A E.A
Discussion on the assumption of performance metric. The proof of Theorem | relies on the assumption that the performance metrie Er satisfies the properties of triangle inequality and symmetry. This assumption is relatively mild and holds for many commonly used performance metrics. For instance, [58] introduced the symmetric discordance index (SDI), ranking- based metric that adheres to these properties, demonstrating the practical applicability of this assumption in practice.
Proof of Proposition 2
For a E A with (2,T) [X(, T), we have:
log P({|x,a) log (ʃ P(t, p(1,2)s,,)dk
log / [Plt)e,, a) P()/() a)dz
P(t|e,
logg (E)(s [P(t|2, a)))
E((z))) [log((((z,a)
(5)
(1) We have 2 by using Jensen's inequality. Val A, taking expectation w.r.t. P(x, Ma') over both sides, we have:
E E(( [log P(1 a) - Ep(t)) [log P({[=a))) a)]
= =fJ (log P(t|x, Ep(a))) [log P(t]2, P(x, dlad)
= // (log P(4pe, a) - Ep(()a) [log P(||2, a))) P P(r, 11a) P(z,t/a)
dxdl
P(x,
o (log ((1 Ep(e))) flog P(2|z, 4))) P(x, 4aa)
P t(a)
26

---

P(r,1aa))
max n.t
Ep(z,(a) [log P(t|ar, a) p(())) olog P(1]2,a)]]
P(z,t(a)
P(x, tla')
max
(EP(a)))) [log P(t|a, a)] (a(a)) [log P(t]z,a))
2,t
P(x, t/aa)
P(x, 1la')
max
(Ha(T, HT, 2))
P(x, t/a)
P(x, tla')
max
((Ha(T) Ha(T, Z)) (Ha(T) Ha(T,
P(x,
P(x,tla')
max
(J,(T, 2) ()(,
P(x,t|a)
(6)
(1) We have S because log P(t a) Ep(())) [log P(t|2, 2 0 according to Eq.
5
because
(T,, 2) = (T, X). Based on Eq. (6) we have:
Er(atta') [lo ((t|x, a)] Br(z,taa)) [E(())=) [log P(t]z,
Ep(c.eee)) [log P(1)2,))
(7)
also have:
Ep(c.sja) [log P(1|2, a')] - (T
IT 7) (T)(T)
(1)
S (T)
--Hop -Ha((
= Er(a,taa) [log P( a')l
(8)
We have < by using data processing inequality. Finally, we have:
Er(sla') [DKL (P(t)z,a') || P(t]2,))
(1) Ep(e)e)) [Dxz (P(t)z,a') ((t]2,))) Br(z)e') [Del (P(t)a, P(t|2, a))]
Ep(t.za)) [log P(t|z,a') - log P(t]2,, a)] Er(nyaa [log P(t]z, a') - log P(t]x, (Era,zta)) [log P(t)2, Er(n)a) [log P(t|r, d')])
+ (Ep()))))) [log P(t|p, a)] EP(.)))) (log (() a)l)
2 have because the shift between two domains w.r.t. input space X' covariate shift; by using Eq. 7 and Eq. (8) and the fact that KI KL-divergence is non-negative. Note that Eq. (9) implies that the shift between these two domains w.r.t. representation space 2 is also covariate shift (i.e., P(t|z, a) P(t]z, a') P(t Va, a' e A).
27

---

Table A1: Overview of medical image datasets for fair TTE prediction evaluation.
Dataset
Prediction Task
Modality
Subgroup
Attribute
images
Censoring rate
Mean TTE
Total
129708
83.9%
4.4 (years)
270
44224
83.1%
5.1
Age
>70
85484
84.4%
3,9
Retinal Fundus
AREDS
Late AMD
Female
71837
83.0%
4.4
Sex
Male
57871
85.1%
4.3
Non-white
4888
99.2%
3.1
Race
White
124820
83.3%
4.4
Total
269360
61.7%
488.6 (days)
2
103437
77.3%
503.3
Age
>60
165923
52.0%
484.2
In-hospital Mortality
Chest X-ray
MIMIC-CXR
Female
125742
63.9%
514.4
Sex
Male
143618
59.7%
468.4
Non-white
83234
66.7%
487.2
Race
White
186126
59.4%
489.1
Total
2227
63.2%
35.9 (months)
<80
1397
62.2%
37.1
Alzhcimer's Disease
Brain MRI
Age
ADNI
>80
630
65.7%
32.6
Female
986
64.7%
38.5
Sex
Male
1241
62.0%
33.9
C Dataset Details
C.1
AREDS
C.1.1 Dataset Description
The Age-Related Eye Disease Study (AREDS) [14 was clinical trial conducted between 1992 and 2001 across 1l retinal specialty clinics the United States. The primary objective was to study risk factors for age-related macular degeneration (AMD) and the impact of dietary supplements on AMD progression. The study followed 4,757 participants, aged 55 80 at enrollment, for a median of 6.5 years. Participants were selected with broad range of AMD severity, from AMD to late-stage AMD in one eyc. At each visit, certified technicians captured color fundus photography images using standardized imaging protocol, although adherence the protocol varied, leading to visits at irregular intervals. AMD severity scores were determined by expert graders at the University of Wisconsin Fundus Photograph Reading Center, with late AMD defined the presence of neovascular AMD or atrophic AMD with geographic atrophy (severity scores from 10 to 12 using severity scale { For this study, include images from both the left and right eyes of each participant and make predictions separately for each eye. In cases where multiple images were captured per eye during visit, we select only one image for analysis. Demographic information, including age, sex, and race, is available in the dataset and is used as sensitive attributes in our study.
C.1.2 TTE Outcome Construction
In our study, TTE outcomes for fundus images are defined as the duration, in years, from the date an image was captured to the first recorded diagnosis of late AMD in the corresponding eyc. For eyes without recorded diagnosis of late AMD, the censoring dates are visit. To ensure that the model forecasts the future risk of developing late AMD, all images taken during the final visit for given eye were excluded from the dataset, as this visit was used solely to determine the TTE outcome. By removing the final visit, we ensured that no images were included with late AMD already present the time of acquisition. This process resulted in final dataset of 708 fundus images, each paired with corresponding TTE information, enabling a robust analysis of the TTE prediction for AMD progression.
the time of the last imaging
C.1.3 Data Access
AREDS is publicly available dataset hosted in the National Center for Biotechnology Information (NCBI) database of Genotypes and Phenotypes (dbGAP) through controlled access. Researchers can request access to this dataset dbGAP. Once the application is approved, researchers access data
28

---

at the following address:
nlm. http://wwww.........oooo////////
htt.://www......oooo///////////a study
C.2 MIMIC-CXR
C.2.1 Dataset Description
The MIMIC-CXR dataset [27] is comprehensive, publicly available collection of chest X-ray images, along with associated clinical data, from the larger MIMIC-IV [26] database. includes over 370,000 chest X-ray images from more than 65,000 patients, annotated with both structured and unstructured clinical information, such as patient demographics, diagnoses, and other relevant clinical details. For our study, we utilize the MIMIC-CX--JG [27]. processed version of the MIMIC-CXR dataset, which provides images in JPG format derived from the original DICOM files. Additionally, link this dataset the broader MIMIC database to access patient demographic information, including age, sex, and race, which we incorporate as sensitive attributes in our fairness benchmarking framework.
C.2.2 TTE outcome construction
To construct TTE outcomes for chest X-ray images, we extract in-hospital mortality events from the MIMIC-IV database. For patients without recorded date of mortality, the censoring dates are set as I year after their last recorded discharge date. We exclude any images that do not have matching record in the MIMIC-IV patient table, images taken after the latest discharge date, and images taken after recorded date of mortality. TTE is then calculated as the number of days from the image study date to either the date of mortality or the censoring date. This process results in final dataset of 269,360 chest X-ray images, each paired with corresponding TTE information, enabling robust analysis of the TTE prediction for in-hospital mortality.
C.2.3 Data Access
MIMIC-CXR, MIMIC-CXR-JPG, and MIMIC-IV are a publicly available datasets hosted by Phys- ioNet, which is a platform providing access to medical data. To access the dataset, researchers first need to create an account PhysioNet and then complete the required training (CITI training). Once researchers have completed the CITI training, they will need to request access to the dataset at the following address:
MIMIC-IV: XMMNeC///Lrr///phsshhhhrrro/o/rr
MIMIC-CXR-JPG: https:://pyssioneeeeee wwtp://tttteeooottt//eiiiiiii..o cxt..cp...p...///////
C.3 ADNI
C.3.1 Dataset Description
The Alzheimer's Disease Neuroimaging Initiative (ADNI) 49] is large, longitudinal study aimed identifying biomarkers for Alzheimer's 's disease (AD) and tracking the progression of the disease over time. Launched in 2004, ADNI is one of the most comprehensive datasets for studying Alzheimer's disease and other neurodegenerative disorders. includes a wide range of data types, including clini- cal assessments, neuroimaging (MRI, PET), genetic data, and fluid biomarkers (e.g., cerebrospinal fluid and blood samples) from over 1, 1,700 participants. These participants are categorized into differ- diagnostic groups, including cognitively normal individuals, those with mild cognitive impairment (MCI), and individuals with Alzheimer's disease. For our study, we focus on neuroimaging data, specifically MRI scans, and consider age and sensitive attributes in our analysis.
C.3.2 TTE outcome construction
In our study, TTE outcomes for brain MRI scans are defined as the duration, measured in 6-month intervals, from the date an MRI scan was captured to the first recorded diagnosis of Alzheimer's disease. For participants without recorded diagnosis of Alzheimer's disease, censoring dates are set the time of their last imaging visit. ensure that the model predicts the future risk of developing Alzheimer's ' disease, excluded all MRI scans taken during the final visit, as this visit was used
29

---

solely for determining the TTE outcome. By removing the final visit, we ensured that no MRI seans were included for participants who had already been diagnosed with Alzheimer's disease at the time of acquisition. This process resulted in a final dataset of 2,227 brain MRI scans, each paired with corresponding TTE information, enabling a comprehensive analysis of TTE prediction for Alzheimer's disease progression.
C.3.3 Data Access
ADNI is a publicly available dataset hosted in the Image and Data Archive (IDA), secure online resource for archiving, exploring and sharing neuroscience data. Access the ADNI dataset requires that researchers register for an IDA account. Once the account is created and the ADNI Data Use Agreement completed, they can access data at the following address: https::/tt/at /aaps://add.... loni usc edu
D Algorithms Details
D.1
TTE Prediction Models
D.1.1
PMF
40 is discrete-time model designed for TTE prediction tasks. It represents the survival time PMF P(t using neural network 0) X - [0, 112 where 0 denotes the model parameters and represents the number of time intervals. The model parameters are estimated by minimizing the negative likelloo,,,d, averaged over the training data, follows:
PMF
(PMF
log (fx(u)(a;;)) 0)) + (1 - 6;) log
Em(as:0)
71
m=x(y)+1 +1
where is the number of training samples, k(у) denotes the specific time interval (from 1, .... that corresponds to time yı, and fm represents the predicted probability that the TTE falls within the th interval.
D.1.2 DeepHit
DeepHit [33 extends the PMF model by incorporating ranking loss function alongside the negative
log-likelihood. Given comparable set defined as & == ((i.g) [m] [n] de 1, ye < ys this ranking loss is calculated over the training data, as follows:
(EM(n) fm 0) Em(ys) fm (zzz
Crank it (0)
Σ exp
σ
(K.))E
where the number of training samples, k(y)) denotes the specific time interval (from 1,2, that corresponds to time y;, and fm represents the predicted probability that the TTE falls within the th interval.
D.1.3 Nnet-survival
Nnet-survival 15 is another discrete-time model designed for TTE prediction tasks. It represents the hazard function ((((1z) using a neural network f(::0) R² where denotes the model parameters and L represents the number of time intervals. Specifically, Nnet-survival sets the hazard function equal to:
h rx; 0 ==
1 fe(x:0) for fe [L], E X
30

---

where (z; 0) is the m th output of the neural network. The model parameters & are estimated by minimizing the negative log- likelihood, averaged over the training data, as follows:
L (0) - 1 2
Σ
6 log (1+ e (Acuuleeer)))) + (1 6,) log (1 elrovolert))
x(3=))
Σ log ( clmlerro))
+
where is the number of training samples and k(y;) denotes the specific time interval (from 1,2, 1. ,.. that corresponds to time y/:
D.2 Fairness Algorithms
D.2.1 Distributional Group Optimization
Distributional Group Optimization [55 22 is optimization technique designed to enhance model robustness by minimizing the worst-case training loss across different groups. Instead of optimizing for the average performance, GroupDRO focuses on the group with the highest loss, ensuring that the model does not disproportionately underperform on any particular group. By incorporating increased regularization, this approach helps mitigate disparities model performance across diverse subpopulations, making it particularly useful in settings where fairness and reliability across groups are critical.
D.2.2 Subgroup Rebalancing
This resampling method (31 addresses class imbalance by upsampling minority groups, ensuring that all groups have equal representation during training. By increasing the frequency of underrepresented samples, the model is exposed to more balanced dataset, reducing bias and improving fairness. This approach helps prevent the model from being overly influenced by the majority group, leading to more equitable predictions across all groups.
D.2.3 Fair Representation Learning
common approach promoting fairness in machine learning through fair representation learning, which aims to obfuscate sensitive group membership information in the learned representations. By ensuring that the model's latent features do not encode discriminatory patterns, this method helps mitigate bias in downstream predictions. Following [70], we incorporate fairness constraints into representation learning by leveraging kernel-based distribution matching via Maximum Mean Discrepancy. This technique enforces similarity in feature distributions across different groups, reducing disparities while preserving task-relevant information.
D.2.4 Domain Independence
The Domain Independence 771 is method that trains separate classifiers for different groups while utilizing shared encoder. This approach allows the model capture group-specific patterns through distinct classifiers while maintaining common feature representation across all groups By leveraging shared encoder, DomainInd enhances generalization and reduces the risk of overfitting to individual groups, ultimately improving the model's robustness and fairness in diverse domains.
D.2.5 Controlled for Sensitive Attribute
This approach 80 [51, similar to Domain Independence, involves training separate models for each group based on the values of sensitive attribute. By doing so, the method captures group-specific patterns while maintaining model flexibility. During the prediction phase, the outputs of the fitted models are averaged across all groups in the population. This averaging process ensures that no single group disproportionately influences the predictions, promoting fairness and reducing bias in the model's outcomes.
31

---

D.3 Model Architecture Details
Each (fair) TTE prediction model comprises two key networks: an image encoder and a classifier. The image encoder transforms images into representation vectors, while the classifier predicts survival time intervals based on these learned representations. In our study, we employ a 2D EfficientNet [61] backbone the image encoder for the AREDS and MIMIC-CXR datasets, and 3D ResNet- 18 backbone [65] for the ADNI dataset. To adapt these models for our task, we replace the original fully connected layers with new layers that map imgges into the representation space, The architectural details of our 2D and 3D image encoders, along with the classifier, are presented in Table A2
Table A2: Architecture details of (fair) TTE prediction models. In our experiments, we set n channel
=3 3 for both 2D and 3D images by duplicating grayscale chest X-ray and brain MRI images to obtain three-channel inputs. feature_ dim is set to 64, and hidden dim set to 16. n class is determined using the 'equidistant' discretization method, with values of 14 for AREDS, 28 for ADNI, and 128 for MIMIC-XRR.
Networks
Layers
Conv2d channel = n channel, output channel = 32, kernel
MBConv (input channel = 32, output channel = 16, kemel= 3)
MBConv6(input channel = 16, output channel = 24, kerne=
MBCCon66input channel 24, output channel 40, kemel=
MBConv6(input channel 40, output channel -80, kemel= =3)
MBConv6(input channel 80, output channel 112, kemel=
2D image
Image Encoder
MBConv6 (input channel 112, output channel 192, 55m *4
MBConv6 (input channel 192, output channel 320, kernel=3)
Conv2d channel = 320, output channel = 1280, kernel = 1)
adaptiveAvgPPoll2dooo
size 1)
Linear(input dim 1280, output dim feature dim) Dropout(p=0.5)
Conv3d channel __ channel. output channel = 64, kemel= 3x777)
Conv Conv3d (inputt channel = 64, output channel = 64, kernel = 3xx3x33*444
Convud channel = 64, output channel = 128, kernel =3x3x3)
Conv3d channel 128, output channel 256, =
3D image
Conv3d channel 256, output channel 512, =333c|=3x3x3)+
ddaapiveAvgPoolBJJouuu (i..=(,.1.1)
Linear(input dim 512, output dim feature_din Dropout(p=0.5)
Lincar(input dim feature dim, output dim hidden dim) Linear(input dim hidden dim, output dim c_aaassss)
Classifier
E
Evaluation Metrics
E.1
Performance Metries
The performance metrics used our study are defined based on 5
E.1.1 Time-dependent concordance index
Harrell's concordance index (C-index) is one of the most widely used accuracy metrics in TTE prediction. It quantifies the fraction of data point pairs that are correctly ranked by prediction model among those that can be unambiguously ranked. The C-index values range from 0 to I, with indicating perfect ranking accuracy. However, notable limitation of the C-index is its dependence on a risk score function for ranking, which many TTE prediction models do not explicitly learn. To address this limitation, [ introduced a time- dependentnt concordance index (Ctd), which leverages predicted survival functions, S(-(x), to assess model performance more effectively. The Cld is computed as follows.
32

---

Definition 3. Suppose that we have survival function estimate S(-[ar) for any 2 X' Then using the set of comparable pairs & = {(i,]) E [n]] x [n] :o = 1,14 1/4 < fj we define the Cl metric as:
1 Cld = & (G,))E
Σ 1 1 (S(uslzi) ((vilzz))
which is between 0 and 1. Higher scores are better.
E.1.2 Time-dependent AUC
While the C-index and Cdd scores provide valuable single-number summaries of predictive accuracy in TTE prediction, they lack the ability to evaluate accuracy specific user-defined time, To address this limitation, (67] introduced time-dependent AUC scores (AUC'd), which explicitly depend on the chosen time point The core idea behind AU is frame binary classification problem for fixed time t, where the "positive" class consists of data points that experienced the event no later than and the "negative" class includes those that survived beyond The survival function ((t)-)
[0, 1] serves as the probabilistic classifier, predicting survival probabilities. lower predicted survival probability for given point z implies higher likelihood of belonging to the positive class. The AU Ctd score quantifies the classifier's ability to distinguish between these two classes at time i, offering time-specific accuracy assessment of the model's predictions. The AU At is computed as follows.
Definition Suppose that we have survival function estimate S(-|z) for any X Then for any > O, using the set of comparable pairs E(t) == {(,,)) E [n] [m] 6, 1. Vi S t, > t}, we
define the (U) (the A at time t) as:
AUC Ea.nEs w.1 (((1)))) < 5(1
E(NEEE w/:
where 11, 22 Wn E [0, are inverse probability of censoring weights to be defined as w;; 1/($.en.on (v: S (I)), and Seensor (1) (1) is an estimation of Spensor (1) = P(C >> I) using Kaplan-Meier estimator 33 AU) is between O and and higher scores are better. Finally, we
can get A as
Amax
A
t (u)du
mmx
Iiin
Emin
where Emin and min are user-specified lower and upper limits of integration.
E.1.3 Integrated Brier Score
The Integrated Brier Score (IBS) performance metric that directly evaluates the error of an estimated survival function Š(-1z) without relying on ranking. The IBS calculated as follows.
Definition 5. Suppose that we have a survival function estimate ((-1x) for any x E X., Then for any t> 0, we define the BS(() (the IBS time t) as:
((tlx;) {uu St. (1 (1- S(1(2;)2) Eur
1 BS(() ==
Σ
Scensor
Scensor (1)
which is nonnegative. Lower scores are better. Finally, can get IBS
1 tyaa
IBS
BS(u)
tmax
tmin
where tmin and Emin user-specified lower and upper limits of integration.
33

---

E.2
Fairness Metries
In this study, we define faireess metrics the predictive performance gaps between groups. This kind of metric is used ensure that the model maintains equal predictive performance across different groups. In particular, given performance metric Er for TTE prediction task, define fairness metric FEr as follow:
FE)
max Er, Era' a.a' EA
where A is the set of groups considered in TTE prediction task, Er, and Era the predictive performance metrics calculated from subsets containing data from groups and a', respectively. For each predictive performance metric defined above, have corresponding fairness metric as follows.
Fou
mAN [Cad C.al
a.a'Ed EA
F FAUCU= mAx a.a'EA EA
AUC AUC
FIBS
max a.A'EAA
IBS - IBS
where Ctd. AUCA IBS, predictive performance metrics calculated from the subset containing data from group a, and Cld, AUC IBS are predictive performance metrics calculated from the subset containing data from group d'.
E.3 Pairness-Utility Trade-Off Metric
The fairness metrics mentioned above do not capture the fairness-utility trade-off while in medical context, is essential to balance fairness and utility ensure that the model is not only fair but also accurate and effective for all groups. To handle this issue, we leverage the equity-scaling metric (ES) [44 that takes both utility and fairness into account for evaluation. Similar to fairness metric, for each predictive performance metric, we have corresponding fairnes--utiltyy trade-off metric as follows.
ES
1+ E.EA
C
CH
AUCH AUUCH
ES ES AUCM 1+
Σ aEA
AAUC
1BSD
ESTas
1 ITBS IBSp
The advantage of the ecuity-scaling metric lies in its intuitive interpretability. Specifically, higher cquity-scaling score indicates that the model is both more accurate and more equitable simultane- ously.
F
Experimental Setup Details
F1
Implementation Details
F1.1 Hardware Usage
The experiments were conducted at supercomputing center utilizing multiple compute nodes. Each node was equipped with an NVIDIA Volta V100 GPU with 16 of memory, an Intel Xeon CPU, and 32 GB of RAM, ensuring the computational resources necessary for large-scale experiments. In total, we trained over 20,000 models, requiring approximately 4.56 GPU years of computational effort, highlighting the extensive scale of our study.
F.1.2 Package Usage
The FairTTE benchmark implemented using Python 3, with Py PyTorch 47 serving as the framework for deep learning computations. The implementation of TTE models is built on the pycox [4]
34

---

package, while the evaluation metrics for TTE prediction leverage pycox, scikit-survival [50, and SurvivalEVAL [52]. Additionally, the training and evaluation pipeline for TTE prediction models is adapted from the demo code provided in [ ensuring a robust and standardized framework for benchmarking.
F.2 Data Split and Pre-processing
Data Split. Each dataset in our study was divided into training, validation, and testing sets using a 60%:20%:20% split ratio. Models were trained on the training sets, evaluated on the testing sets, and the validation sets were used for model selection. Since a single patient may have multiple medical records, we took precautions to prevent data leakage during model training. Specifically, the data was split by patient, ensuring that no patient appearing in the testing set had any records in the training or validation sets. This approach maintains the integrity of the evaluation process and ensures that model performance assessed on entirely unseen patient data.
Data Pre-processing. Before being fed into the TTE prediction models, chest X-ray and color fundus images resized to 224 x 224 pixels, while brain MRI scans are resized to 128 x 128 x 96, Additionally, all pixel values are normalized a range of to I to ensure stability during training and improve model performance.
We consider binary group setting our experiment. These groups were constructed according the following criteria:
Race:
'Non- White' (Group O), 'White' (Group 1)
Sex: 'Female' (Group 0), 'Male' (Group 1)
Age:
MIMIC-CXR: ' 60' (Group 0), '> 60' (Group AREDS: < 70' (Group 0), 70' (Group 1)
ADNI: S 80' (Group 0), "> 80' (Group 1)
F3
Hyperparameter Search
To ensure fair comparison, we perform grid-based hyperparameter search using 10 random seeds. The details the hyperparameter search for the methods used in our experiments are provided below.
TTE prediction models
Learning rate: 10 where x ~ Unifform-44----
Decay rate: 10t where Uniform(-6i -4)
Fair TTE prediction models
17 10 where x ~ Uniform( -3,, -1) (DRO)
A: 10a where x ~ Uni Uniform(-55p (FRL)
For standard TTE prediction models, we select the best models based on their predictive performance metrics calculated on the validation sets. In contrast, for fair TTE prediction models, we prioritize fairness metrics when selecting the best models, allowing for up to 5% reduction in accuracy compared to the baseline TTE models. This approach ensures a balanced trade-off between fairness and predictive performance.
F.4 Quantifying Source of Bias
In order to quantify the degree of bias sources in each dataset and sensitive attribute setting, several metries as follows.
Disparity mutual information between X and across groups. We quantify the disparity in mutual information between Xz (i.c., image representation generated from the vision backbones)
35

---

and Y across groups by computing the maximum difference in their normalized mutual information values across all groups, as defined below.
max a.a' 'A H(Xz|A = -, - a
d', [[Xz|A=a', - A
where [(;-AA= a, == 1) represented the mutual information conditioned A and A 1 and H (-|A= = a.) denotes the entropy conditioned on A = a and A = 1.
Disparity in mutual information between Xz and A across groups. Similarly, we quantify the disparity in mutual information between Xz (i.e., image representation generated from the vision backbones) and A across groups by computing the maximum difference their normalized mutual information values across all groups, as defined below.
2r, 4/A = =a) H(X2 a) + H a H(X a') H H = a')
21(Xz z, A =
Bías () = max
e,''EA
Disparity in TTE distribution across groups. We measure the disparity in TTE distributions across groups by calculating the maximum Wasserstein distance [63], normalized by the range of TTE, between the TTE distributions of each group. This is defined as follows:
W A P(Y A
BiaSTTE max a.a' E.A
maxyey y
where W (.:-) denotes the Wasserstein-1 distance between the distributions.
Disparity in image distribution across groups. We measure the disparity in image distributions across groups by calculating the maximum Wasserstein distance [63] normalized by the range of image feature values, between the image distributions of each group. This defined as follows:
(V)(
a())
BiaSImage Image
max E.A
max y
where (..-) denotes the Wasserstein-1 distance between the two distributions. Due to the high dimensionality of image representations, implement sliced Wasserstein distance ( variant of the Wasserstein distance that approximates the full Wasserstein distance between high-dimensional distributions by projecting them onto one-dimensional subspaces and averaging the resulting ID Wasserstein distances.
Disparity in censoring rate across groups. We quantify the disparity in censoring rates across groups by calculating the maximum normalized difference between the means of the censoring distributions for each group, as defined below.
E EAA a) - -
BiasCensoring
max
a,a' E.A
E E
E.5
Constructing Causal Distribution Shift
To construct distribution shift between training and testing data, we modify the training data by intro- ducing correlations between the sensitive attribute and other RVs in the causal graph (Figure This adjustment simulates real-world scenarios where biases in data collection or underlying relationships may lead to disparities across groups. The details of this process, including the specific modifications applicd to establish these correlations, are outlined below.
Distribution shift X: Images from disadvantaged groups degraded using Gaussian blur filter to simulate lower-quality data.
Distribution shift on Y: TTE labels for disadvantaged groups are corrupted by adding noise sampled from a uniform distribution.
36

---

Distribution shift on A: To simulate biased censoring, we flip the censoring indicators for 90% of uncensored samples within disadvantaged groups.
We note that, although these distribution shifts are synthetic, they mimic real- woaldworld scenarios, as described below.
Adding noise to images mimics real-world scenarios such as patients in different geographic locations are scans with different equipment. This causes the medical image to appear systematically different for groups each location.
Adding noise to TTE labels mimics real-world scenarios such as delayed or inaccurate event recording in EHR system.
Flipping censoring indices mimics real-world scenarios in which certain groups experience less consistent access to care due to financial or geographic. Thus, these groups are more likely drop out of care, resulting in higher censoring rate.
G Causal Graphs for Fairness in TTE Prediction
G.1
Causal Graphs for Biased and Unbiased Settings
Figure Al presents the causal graphs for the unbiased and biased scenarios. In the unbiased scenario (Figure A la), the sensitive attribute A is unrelated to the TTE outcome and influences only XA. with no effect on other variables in the graph. In contrast, in the biased scenarios (Figure Alb and Tigure(ATb) A also affects additional variables, resulting in dependencies between A the TTE outcome. These causal pathways may be direct (Figure Al p). mediated through unobserved variables (Figure Al 1), or both.
x
A
A
XA
A
X
4
A
X
X
(a) Unbiased Setting
(b) Biased Setting (Direct Pathway)
(c) Biased Setting (Indirect Pathway)
Figure A1: Causal structure TTE prediction. Gray circles denote unobserved random variables. (a) Unbiased setting, where the sensitive attribute A influences only XA. (b) Biased setting with direct causal pathways, where is directly associated (red arrows) with other variables in the graph. (c) Biased setting with indirect causal pathways, where influences (red arrows) other variables through unobserved variables U.
G.2 Real-world Causal Graph Examples for Fairness in TTE Prediction
In this section, we present causal graphs illustrating real-world scenarios time-to-event (TTE) prediction using medical imaging. Many of these examples are adapted from diagnostic settings prior work (28). We describe four scenarios in which the sensitive attribute influences other variables in the causal graph namply, the medical image X, the underlying condition Z, the time-to- event T, and the censoring time C-leading to disparities in group-specific data distributions. For each scenario, we include two examples: one where the causal pathway from A is valid (red arrows), appearing both training and testing data, and one where pathway spurious (blue arrows), representing bias present only in the training data and absent in the testing data.
37

---

Volume
Age
Consering
Censering Index
Skin Pigmentrion
Censoring Time
Comsoring
Flakk of Athaimer's
Dematoscopic Image
Access Heathhare
Underlying Consition
Lesion Feature
Undortying
Feature
Time
Conditon
Time
(a)
(b)
Figure A2: Causal graphs illustrating scenarios where the sensitive attribute A affects the underlying
condition Z. (a) Valid pathway: age is known clinical risk factor for Alzheimer's disease. (b) Invalid pathway: race appears spuriously correlated with Z due to disparities in healthcare access.
disease Feature
Sex
Cansoring Time
Censoring Index
Scarner Artefact
Location
Consoring
Time
Cansoring
Index
Mammnggrm
Denaity
Ultrasound
Disease
Festure
Underlyina
Condition
TTE
Observed Time
Lesion Feature
Undertying Conditon
TTE
Observed Time
(a)
(b)
Figure A3: Causal graphs illustrating scenarios where the sensitive attribute A influences the medical
image X. (a) Valid pathway: breast cancer presents differently in men and women due to inherent differences in breast tissue. (b) Invalid pathway: spurious correlation arises when patients in different locations are imaged at varying disease stages due to inconsistent ultrasound referral policies.
Bone Density
Age
Censoring Timee
Censoring Indax
Scarneer Artefact
Location
Censoring
Time
Comsoning
Index
Orferential) Diagnasis
Quality
CT Scan
X-ray
Disease Feature
Underlying Concition
Observed Time
Leeion Feature
Undertying Condttio
TTE
Observed Time
(a)
(b)
Figure A4: Causal graphs illustrating scenarios where the sensitive attribute A influences the time- to-event outcome T. (a) Valid pathway: age contributes to differential diagnosis in epidemiology and legitimately affects disease progression. patients from different locations receive healthcare services of varying quality, impacting T non-causal manner.
(b) Invalid pathway: spurious correlation arises when
38

---

Non- Fenture
Censering
Censoring Index
Non- Foature
Censoring Time
Comsoring
Carclac
Follow-up Paman
Brain
Access Healhharn
Underlyin
Undortying
Feature
Consition
Time
Feature
Conditon
Time
(a)
(b)
Figure A5: Causal graphs illustrating scenarios where the sensitive attribute A affects the censoring
time C. (a) Valid pathway: Women may be less likely to receive aggressive follow-up diagnostic imaging for cardiac conditions, resulting in higher censoring for female patients. (b) Invalid pathway: race appears spuriously correlated with censoring time due disparities healthcare access.
39

---

Additional Results
H.1 Predictive Performance and Fairness in TTE Prediction Models
Figure A6 presents the complete per-group performance results of TTE prediction models models-DDeepHHi Nnet-Survival. and PMF-across all dataset, sensitive attribute, and metric combinations, while Figure A7 reports the corresponding significance tests using the two-sided Wilcoxon signed-rank test. As shown, performance gaps between groups observed across all settings.
Group 0
Group
95% CT
0.9
Ctd
0.8
0.6
Age
Sex
Race
Sex
Age
ADNI
AREDS
MIMIC-CXR
(a) Cr
1.0
Group Group
95%
0.9
AU
0.6
Age
Sex
Age
Race
Sex
Age
Race
Sex
ADNI
AREDS
MIMIC-CXR
(b) AU
Gruup
Group
IBS
0.7
0.0 Age
Sex
Age
Race
Sex
Age
Raoc
Sex
ADNI
AREDS
MIMIC-CRR
(c) IBS
Figure A6: Per-group predictive performances of TTE prediction models across various datasets and sensitive attribute combinations. The visualized performances correspond the best models determined by model selection conducted on the validation sets. The 95% confidence intervals (CIs) are calculated using bootstrapping over the test sets. a) Results measured by Ctdd, b) Results measured by Crdd, c) Results measured by IBS.
40

---

AUC
IBS
2o-09
2c-09
Race
NA
2c-09
4c-03
Race
2c-09
2c-09
10
Age
1e-07
1e-08
2e-09
Age
4e-
2e. 09
2e
Age
1e-03
2e-09
2e-09
16
1c-04
2c-03
ADNI
AREDS
MIMIC
ADNI
AREDS
MIMIC
ADNI
AREDS
MIMIC
Figure A7: P-values from the two-sided Wilcoxon signed-rank test computed across all TTE predic- tion models and random seeds. A p-value < 0.05 indicates that there significant differences in predictive performance between groups.
41

---

H.2 Comparison between Pre-Training and Training from Scratch Strategies for TTE Prediction Models
H.2.1
Comparison in Predictive Performance
Figure A8 presents the complete per-group predictive performance gap between pre-training and training from scratch approaches across all dataset, sensitive attribute, and metric combinations. As shown, pre-training consistently improves the predictive performance of TTE models across most
settings.
Group
Group
4 (ADNI)
AC (Othens)
0.15
0.02
0.00
0.00
Age
Race
Sex
ADNI
AREDS
MIMIC-CXR
(a) Cr
0.30
Grop Grrug
0.05
AUC! (ADNN)
AU (Otbers)
0.15
0.02
Sex
Age
Race
Sex
Race
Sex
ADNI
AREDS
MIMIC-CXR
(b) AU
AIBS (All)
Group
Group
00.10
Age
Age
Age
(c) IBS
Figure A8: Per-group average performance gap for TTE prediction models using a pre-training strategy compared training from scratch across various datasets sensitive attribute combinations. Positive ACdd and C and negative ATBS values indicate that the pre-training strategy enhances predictive performance relative training from scratch. a) Results measured by Crdd b) Results measured by Crd; c) Results measured by IBS.
42

---

H.2.2 Comparison in Fairness
Figure A9 presents the significant differences in terms of faimesss between pre-training and training from scratch strategies. As shown, do not observe a significant improvement with pre-training compared to training from scratch. Specifically, the p-values from one-sided Wilcoxon signed-rank tests are larger than 0.05 in 18 out settings, suggesting that pre-training does not lead to more equitable predictions in most cases.
AU
IBS
C
Ravco
Ruce
Rate
NA
9c-10
3c-01
Age
3c-01
9c-03
Age
-64
See
9c-01
9c-01
Io-04
2c-08
9c-01
7c-02
4c-01
ADNI
AREDS
MIMIC
ADNI
AREDS
MIMIC
ADNI
AREDS
MIMIC
Figure A9: P-values from the one-sided Wilcoxon signed-rank test computed across all TTE prediction models and random seeds. A p-value > 0.05 suggests that pre-training does not result in significantly more equitable predictions compared to training from scratch.
H.3 Comparison with Advanced Image Backbones and Medical Pre-training for TTE Prediction Models
To investigate whether more advanced vision backbones pretrained on medical imaging data can enhance the predictive performance and fairness of TTE prediction models, conduct an additional experiment using the late AMD progression prediction task on the AREDS dataset. Specifically, we adopt RETFound widely recognized eye-specific foundation model based on the Vision Transformer (ViT) architecture [12] and pretrained on millions of retinal images as the image backbone for our TTE models. As shown Table A3 RETFound with VIT does not demonstrate any improvement over EfficientNet pretrained on ImageNet for this task, suggesting that general-purpose backbones may remain competitive despite the availability of domain-specifie pretraining.
Predictive and faimess performances of DeepHit using EfficientNet and Vision Transformer as vision backbones on AREDS dataset. EfficientNet is pretrained on ImageNet dataset while Vision Transformer is initialized with pretrained weights provided by RETFound.
Table A3:
EfficientNet
Vision Transformer
Sensitive Attribute AU t
IBS 4
Ctd
AU 1
IBS 4
Ctd +
Age
78.41
15.37
81.30
78.04
15.22
80.65
Accuracy Sex
79.08
15.36
81.77
78.01
14.51
80.72
Race
81.78
11.74
84.53
80.99
11.99
83.91
Sensitive Attribute
FAUCua
FIBS
Feu
FAUCI
FIBS
Feu
Age
1.58
12.56
2.20
0.35
9.50
1.44
Fairness
Sex
0.76
3.84
1.32
0.85
4.26
0.44
Race
14.00
10.14
11.09
9.32
10.47
10.27
43

---

H.4 Fairness in Fair TTE Prediction Models
Table A4 and Figure A10 present the results of statistical significance testing for fair TTE prediction models, conducted using the Friedman test followed by the Nemenyi post-hoc test.
Table A4: P-values from the Friedman test followed by a Nemenyi post-hoc test computed across all
dataset and sensitive attribute combinations. A p-value < 0.05 indicates that the significant difference terms of fairness between two corresponding methods.
Metrics
Models
DI
CSA
DRO
DeepHit
FRL
SR
DI
1.000
0.995
0.684
0.420
1.000
0.9667
CSA
0.995
1.000
0.340
0.765
0.985
1.000
Ctd
DRO
0.684
0.340
1.000
0.011
0.765
0.206
DeepHit
0.420
0.765
0.011
1.000
0.340
0.894
FRL
1.000
0.985
0.765
0.340
1.000
0.937
SR
0.967
1.000
0.206
0.894
0.937
1.000
1.000
0.206
0.894
0.894
0.340
0.596
CSA
0.206
1.000
0.836
0.836
1.000
0.985
DRO
0.894
0.836
1.000
1.000
0.937
0.995
AUCad
DeepHit
0.894
0.836
1.000
1.000
0.937
0.995
FRL
0.340
1,000
0.937
0.937
1.000
0.999
SR
0.596
0.985
0.995
0.995
0.999
1.000
DI
1.000
0,995
0.985
0.985
0.596
0.894
CSA
0.995
1.000
1.000
0.836
0.894
0.995
DRO
0.985
1,000
1.000
0.765
0.937
0.999
IBS
DeepHit
0.985
0.836
0.765
1.000
0.206
0.507
FRL
0.596
0.894
0.937
0.206
1.000
0.995
SR
0.894
0.995
0.999
0.507
0.995
1.000
44

---

20
30
400
4.
5.0
GRo (1.9)
5) DeepHit
Regularization (3.1)
(4) Reweighting
DomainInd (3.2)
(3.8) DomainlndAggregaated
0
30
33
DomainlndAggregatee (2.6)
(4.8) DomainInd
Regularization (2.9)
(3.8) GroupDRO
Reweighting (3.2)
(3.8) DeepHit
(b) AUC
Regularization (2.5)
(4.6) DeepHit
Reweighting (3)
4) DomainInd
GroupDRO (3.4)
(3.5) DomainlndAggregated
(c) IBS
Figure A10: Critical Difference diagrams for all methods calculated from all dataset and sensitive attribute combinations. Although fairness algorithms are generally ranked higher than DeepHit in all settings, there is no significant difference in terms of fairness indicated by the connections between fairness algorithms and DeepHit in the diagrams. a) Diagram for Ctd. b) Diagram for AU c) Diagram for I IBS.
45

---

H.S Fairness-Utilit Trade-Off Results
Incorporating fairness shifts the objective from pure utility optimization to balancing utility and fair- ness. To assess this trade-off in fair TTE prediction methods, we compute equity scaling scores [44] across datasets and sensitive attributes under both in-distribution and distribution shift scenarios. As shown in Figures ( [ different methods exhibit varying cairness-utility trade-offs, with CSA achieving the most favorable balance in most settings.
80
70
E5 (%)
60
DRO
CSA
SR
FRL
50
AUC
IBS
Ctd
Figure A1l: Fairness-utility trade-off's of fairness algorithms for TTE prediction across various utility metrics. For each metric, compute the corresponding equity scaling score measure of the trade-off. The results for each fairness algorithm are aggregated across all dataset and sensitive attribute combinations.
80
ES (96)
70
DRO SR
CSA
FRL
AU
C
Figure A12: Fairness-utility trade-offs of fairness algorithms for TTE prediction across various utility metrics under distribution shift created by flipping censoring indices (shift on X). For each metric, we compute the corresponding equity scaling score as a measure of the trade-off. The results for each fairness algorithm are aggregated across all dataset and sensitive attribute combinations.
46

---

80
ES (%)
DRO
CSA
DI
70
SR
FRL
AUC
185
C
Figure A13: Fairness-utility trade-off's of fairness algorithms for TTE prediction across various utility metrics under distribution shift created by flipping censoring indices (shift Y). For each metric, we compute the corresponding equity scaling score as a measure of the trade-off. The results for each fairness algorithm aggregated across all dataset and sensitive attribute combinations.
80
ES (%)
70
AU
IBS
Ctd
Figure A14: Fairness- trade-offs of fairness algorithms for TTE prediction across various utility metrics under distribution shift created by flipping censoring indices (shift on A), For each metric, compute the corresponding equity scaling score as measure of the trade-off. The results for each fairness algorithm are aggregated across dataset and sensitive attribute combinations.
47

---

H.6 Additional Results for Predictive Performance and Fairness in Fair TTE Prediction Models under Distribution Shift
This section presents the complete results for fair TTE prediction under distribution shift scenarios. As illustrated in Figure 3 we define distribution shift as a setting where correlations between the sensitive attribute and other variables in the causal graph are present in the training data but absent in the testing data. To simulate such shifts, we intervene on one group (the intervened group) by corrupting specific aspects of the data data-namely, the images (X). TTE labels (Y), or censoring indicators (Á) while leaving the other group unchanged. Detailed procedures for generating these shifts are provided in Appendix E.5
H.6.1
Results for Distribution Shift in
CSA
FRL
ADNI
ADNI Sex
AREDS Age
ARFDS -Sx
0.700
2.786
3.000
0
AUC7a
0.992
0,720
0.700
0.:00
0.776
1.996
0.792
8/292
Age
0.000
0.778
0.776
8/888
0.7/4
0.600
81.9
0.775
0.750
0.785
0.700
5.785
0.776
0.781
ADNI Age
ADNI - Sex
AREDS Age
AREDS Sex
0.930
5.70
0.640
0,800
Out-of ait-o--distributiin
0.500
2800
0.400
2332
(b) C32
0.002
0.720
0.812
D:05
0/300
AREDS
MIKIC-CKR Age
MIMIC-CXR Sex
MIMIC-CXR Race
0.760
0.750
5.750
6.752
0.000
2746
0.450
0.744
0.832
0.040
0.546
0.255
0,750
0.763
0.750
0.735
0,762
0.752
0.790
Age
Age
AREDS
0.390
3.300
6::05
0.20
01000
0.50
(c) IBS
0/240
0.200
0.29
5.935
3144
ARFDS Race
MIMIC-CXR Age
MIMIC-CXR 5504
MIMIC-CXR Rare
0.1.9
3,223
0.200
0.150
5128
0.200
0:200
0218
In-distribution
Figure A15: Comparison of predictive performance for (fair) TTE prediction models in in-distribution vs. out-o--distribution (i.e., shift in X) learning scenarios, evaluated across dataset and sensitive attribute combinations. The displayed results represent the average performance across all random seeds. Points on the dashed line indicate equal performance in both scenarios. a) Results for AUC b) Results for Ctd, c) Results for IBS.
48

---

Di
CSA
DRO
Deephit
FRL
SR
ADNI Age
ADNI SSex
AREDS Age
AREDS Sex
0.300
0.30
2040
0.805
181F AD70*
0.080
500
0.100
0.20
0.000
5100
0.130
0,200
0,000
0.055
0.000
0,25
2010
0015
AREDS Race
MIMIC-CXR Agl
MIMIC-CXR Sex
MIMIC-CKR- Race
0.000
20155
0
0.020
0.000
p.300
0.305
0.200
0.010
0.000
0,252
0.008
0.020
0.006
0.012
ADNI Age
ADNI Sex
AREDS Age
AREDS
0.090
0.00
0.040
90000
0.073
Out-of distribbbbittoonn
0.000
(b) DM
0.050
0.100
0.000
0.04
AREDS
MIMIC-CXR Agl
MIMIC-CXR.
D.226
0320
048
5/018
0,05
0.000
0 042
0.030
AUNI
AUNI Sex
AMEDS
ARtUS
03200
0713
D1M0
0.045
(c)FF188
51000
11160
0,200
6.055
3.000
6.120
0.400
ARLDS Race
MIMIC-CXR Age
MIMIC-CAR Sex
MIMIC CXR Race
5/300
0.010
0.000
0.104
1994
0:.00
0.000
U.US
G.Ubb
In-distribution
Figure A16: Comparison of fairness for (fair) TTE prediction models in-distribution vs. out-of- distribution (i.e.. shift in X) learning scenarios, evaluated across all dataset and sensitive attribute combinations. The displayed results represent the average performance across all random seeds. Points on the dashed line indicate equal performance in both scenarios. a) Results for F FAUCH; b) Results for Fo e) Results for Fras-
49

---

ARFDS
ARFDS
0.705
0.502
AUCCe
AREDS
MIMIC-CXR-
MIMIC-CXA-
0.800
0.756
0.252
0.750
0.730
0.758
0.762
0.735
3,782
0.774
ADNI
AREDS
AREDS
5.000
0.700
0.560
0,500
Out-of-dllstribuue
0.760
0.482
C.00
0.560
0,048
0.500
0.800
0.866
Age
0.144
0.726
0.709
0.430
0.032
0.720
0.755
0.744
0.73
0730
2.74
Age
Age
0.422
0.300
5.200
0.390
0.250
00
0.240
0.300
0.400
3.300
0.200
0.225
0.184
0.122
[c]
AREDS
Age
0.205
0236
0.20
0.550
0,000
0.720
0.10
0.208
0216
0.14
0.00
0.00
In-distribution
Figure A17: Comparison of predictive performance on the intervened group for (fair) TTE prediction models in in-distribution vs. out-of-distribution (i.e., shift in X) learning scenarios across all dataset and sensitive attribute combinations. The displayed results represent the average performance across all random seeds. Points on the dashed line indicate equal performance in both scenarios. a) Results for AUC b) Results for Ctd, c) Results for / BS.
50

---

AUC
IBS
C
Race
NA
Be-12
46-66
NA
Rc-12
9c-07
Race
NA
Ro-12
Sc-08
0.
Age
Rc-02
lo-0n
Age
7c-08
3e-10
Ce-C4
56-10
Sex
Ic-05
1e-11
Sex
1c-09
7c-06
Ic-11
-02
ADNI
AREDS
MIMIC
ADNI
AREDS
MIMIC
ADNI
AREDS
MIMIC
Figure A18: P-values from the one-sided Wilcoxon signed-rank test computed across all fair TTE prediction models and random seeds. A p-value < 0.05 suggests distribution shift on X significantly degrades TTE predictive performance compared no distribution shift.
51

---

H.6.2 Results for Distribution Shift in Y
DeepHit
Age
Ago
6/750
0,792
0.725
0.754
737
0.792
5.700
734
c.700
0,725
0.750
0,720
0,760
0,000
0.716
0.754
6.792
0.788
0,792
Ago
Race
5.754
0.798
6.750
0.792
0.100
0 0.208
8,750
0.280
0,810
0.735
0,760
0.785
0.180
0.785
0.778
0.784
Age
Age
5.700
0
5.000
0.816
Out-of-distributoon
0.400
0.500
0.600
0.850
0.720
0.822
0.015
D.32
0.005
AREDS
MIMIC-CXR Age
MIMIC-CXR
Race
0.845
0.842
5.700
0.750
0.756
0.70
5.755
0.750
0.744
0,032
0.810
0.818
0.735
0.750
0.785
0.730
0.216
0.662
0.712
0.165
Age
Age
0.270
5.255
0:155
0.152
52200
0.22
0.230
0.255
2220
0.40
0,200
0332
0.100
0:55
0,14
0,522
0.35
ARFDS
MIMIC-CXR Age
MINIC-CXR
MIMIC-CXR Race
0.25
0.323
5.25
0.000
5,200
0.00
0.155
0.113
0.120
0.125
0.10
0225
0,200
0.208
2218
0.203
0210
In-distribution
Figure A19: Comparison of predictive performance for (fair) TTE prediction models in-distribution vs. out-of-distribution (i.e., shift in Y) learning scenarios, evaluated across all dataset and sensitive attribute combinations. The displayed results represent the average performance across all random seeds. Points the dashed line indicate equal performance in both scenarios. a) Results for AUC b) Results for Ctd, c) Results for IBS.
52

---

Di
CSA
DRO
Deephit
FRL
SR
ADNI Age
ADNI -Sex
AREDS Age
AREDS Sex
5.200
0.323
0.022
5/366
0.300
0.003
[a)FF AUCC
0.023
0.120
5/12
0.004
0.100
0.20
0.000
5100
0.130
0.000
0,000
0.15
0.000
0,25
2010
0.15
AREDS Race
MIMIC-CXR Age
MIMIC-CXR Sex
MIMIC-CKR- Race
0.22
0.012
0.001
0.034
0.000
5,52
0.000
0.305
0.120
0.010
0.000
0,052
0.008
0.020
0.006
0.012
Age
ADNI
AREDS Age
AREDS
0.125
0
0.075
9,220
Out-of distiiibbbittoo
0.053
0.966
5015
0.055
(b) F.
0.003
6.000
0.04
AREDS
Agl
MIMIC-CKR
0.042
0.039
0.003
AUNI
AUNI Sex
AMEES
ARtUS
0.200
0.362
A.L05
U.URZ
(c),FIp
00320
0.000
116
0,200
6.055
6.120
0.400
ARLDS Race
MIMIC-CXR Age
MIMIC-CUR Sex
MIMIC CXR Race
5/010
0.202
0.024
0.044
0.055
0.093
0
0.000
0.104
1994
0:000
0.000
0.055
G.Lb
In-distribution
Figure A20: Comparison of fairness for (fair) TTE prediction models in in-distribution vs. out-of- distribution (i.e., shift in Y) learning scenarios, evaluated across all dataset and sensitive attribute combinations. The displayed results represent the average performance across all random seeds. Points on the dashed line indicate equal performance in both scenarios. a) Results for F FAUCH; b) Results for Fou; c) Results for FIBS.
53

---

ARFDS
AREDS
0.740
6.755
t,650
0.779
0.900
0.780
AUC
AREDS
MIMIC-CXA-
5.765
0.8000
0.960
702
5.750
0.790
0:758
0.762
0.735
3,782
ADNI
ADNI
AREDS
AREDS
6.400
0.005
0.702
0.000
0,500
Out-of-dlisribbuee
0.872
0.800
0.000
1.700
0.600
0.700
0.500
0.800
0.866
AREDS
Age
MIMIC-CXI
0.842
5.730
0.144
0.732
6.700
0.032
0.720
0.755
0.73
0.30
2.748
Age
Age
0100
0.14
0,300
0.360
0.25
0.225
0.164
0.22
Age
0.203
00
5.210
00
5.208
0.195
8:200
0.100
0.208
0.16
0.145
0.70
0.00
In-distribution
Figure A21: Comparison of predictive performance on the intervened group for (fair) TTE prediction models in in-distribution vs. out-of-distribution (i.e., shift in Y) learning scenarios across all dataset and sensitive attribute combinations. The displayed results represent the average performance across all random seeds. Points on the dashed line indicate equal performance in both scenarios. a) Results for AUC b) Results for Ctd, c) Results for / BS.
54

---

AU
IBS
C
Race
NA 4e-02
NA
Sc-03
20-05
Race
NA
4p402
Ic-08
0.
Age
Age
Sex
Se-03
3e-03
Sex
4c-02
2c-01
3e-04
Sex
1e40
3o-02
3:-05
-02
ADNI
AREDS
MIMIC
ADNI
AREDS
MIMIC
ADNI
AREDS
MIMIC
Figure A22: P-values from the one-sided Wilcoxon signed-rank test computed across all fair TTE prediction models and random seeds. A p-value 0.05 suggests distribution shift on Y significantly degrades TTE predictive performance compared no distribution shift.
55

---

H.6.3 Results for Distribution Shift in A
DeepHit
Age
Ago
c,750
0,792
C700
0.784
738
5.800
AUC
0.776
c.700
0,725
0.750
0.720
0,760
0,000
0.776
0.754
6.792
0.785
0.792
Ago
Race
0.703
0.802
5.775
8,750
0.200
0,800
0.735
0,760
0.785
0.100
0.785
0.798
0.784
Age
ADNI
Ago
0
0.602
5.400
0.816
Out-of-distributen
0.640
0.802
(b) C
0.500
0.600
0.850
0.720
0.812
0.015
0.52
0.005
AREDS
Age
MIMIC-CXR
Race
1.100
0.700
0.842
0.832
E7755
0.024
0.764
0,032
0.810
0.815
0.735
0.750
0.785
0.730
0.216
0.662
0.752
0.165
Age
Age
0.300
0.152
5200
0.270
0.22
02200
0.255
2220
0.44
0,200
0.322
0.100
0.55
0,14
0,552
0.35
ARFDS
MIMIC-CRR Age
MIMIC-CXR
MIMIC-CXR Race
5.260
0,200
5.210
0.11
0.120
0.125
0.10
0:25
0,200
0.208
2216
0.203
0210
In-distribution
Figure A23: Comparison of predictive performance for (fair) TTE prediction models in-distribution vs. out-of-distribution (i.e., shift in A) learning scenarios, evaluated across all dataset and sensitive attribute combinations. The displayed results represent the average performance across all random seeds. Points the dashed line indicate equal performance in both scenarios. a) Results for AUC b) Results for Ctd, c) Results for IBS.
56

---

Di
CSA
DRO
DeepHit
FRL
SR
ADNI Age
ADNI -Sex
AREDS Age
AREDS Sex
0.013
2/218
0.222
(4) [a] ATOON
0.000
0.000
0.100
0.20
0.000
5100
0.130
0.200
0,000
0.15
0.000
0,25
20110
0.055
AREDS Race
MIMIC-CXR Age
MIMIC-CXR Sex
MIMIC-CKR- Race
U1042
0.305
2/030
0.016
0.000
0052
01008
0.305
0.320
0.010
0.000
0.052
0.008
0.030
0.005
0.012
ADNI Age
ADNI
AREDS Age
AREDS
0.000
0.000
Out-of distiiibbbbttoo
0.005
0.002
5/06
(b) F.
0.000
0.100
6.000
0.04
AREDS
Agl
MIMIC-CKR
0.001
2.000
0.090
0.073
0 038
0.058
AUNI
AUNI Sex
AMEDS
ARtUS
0.253
5:105
0.200
(c)F199 18.7
0.353
0.039
116
0,200
0.300
6.455
9.000
0.105
6.120
6.400
ARLDS Race
MIMIC-CXR Age
MIMIC-CXR Sex
MIMIC-CXR Race
0.305
0.020
0.104
1994
0:.00
0.000
ULMS
G.Lb
In-distribution
Figure A24: Comparison of fairness for (fair) TTE prediction models in-distribution vs. out-of- distribution (i.e.. shift in A) learning scenarios, evaluated across all dataset and sensitive attribute combinations. The displayed results represent the average performance across all random seeds. Points on the dashed line indicate equal performance in both scenarios. a) Results for F FAUCH; b) Results for For;; e) Results for Fras-
57

---

ARFDS
ARFDS
0,750
0.785
6.000
0.750
0.692
100
6.00
[a]
AREDS
Ago
MIMIG-CXR-
MIMIC-CXR-
0,710
5,759
0.790
0,750
0,730
0./90
0.758
0.762
0.735
3,782
ADNI
ADNI
AREDS
AREDS
0.810
0.690
Out-of Dut-of-ulistribution
6.600
5375
0.795
0.000
5.700
0.600
0.700
0.500
0.000
0.866
AREDS
Age
MIMIC-CXIR
0.812
0.944
5.740
0.764
5.710
0.033
0.720
0.755
0.744
0.73
0.30
2.748
5.240
1:8
0164
0,300
0.360
0.25
0.225
0.164
0.222
Age
0,201
6.216
0.208
000
0.200
0720
0110
0.94
0700
0.00
In-distribution
Figure A25: Comparison of predictive performance on the intervened group for (fair) TTE prediction models in in-distribution vs. out-of-distribution (i.e., shift in A) learning scenarios across all dataset and sensitive attribute combinations. The displayed results represent the average performance across all random seeds. Points on the dashed line indicate equal performance in both scenarios. a) Results for AUC b) Results for Ctd, c) Results for / BS.
58

---

AUC
IBS
Chli
Rice
NA
Ie-04
3e-01
NA
Ic-03
Rece
NA
9c-08
4c-11
0.
Age
3c402
Age
Se-08
3e-02
20-04
Sex
Sc-03
Sex
Se-05
Sc-02
3:-09
-02
ADNI
AREDS
MIMIC
ADNI
AREDS
MIMIC
ADNI
AREDS
MIMIC
Figure A26: P-values from the one-sided Wilcoxon signed-rank test computed across all fair TTE prediction models and random seeds. A p-value < 0.05 suggests distribution shift on A significantly degrades TTE predictive performance compared no distribution shift.
59


Reviews:
[{"review_id": "hELBRKUkui", "reviewer": "Reviewer_VK6y", "summary": "The paper addresses a critical gap in fair AI research: while fairness in medical image diagnosis (e.g., classification, segmentation) has been studied, fairness in medical prognosis (predicting disease progression or outcomes over time) remains overlooked. Prognosis is often framed as time-to-event (TTE) prediction (e.g., time until death, disease recurrence). The authors argue that TTE prediction poses unique fairness challenges due to temporal dynamics, censored data (incomplete follow-ups), and the lack of standardized fairness metrics suited for survival analysis.", "questions": "It would be better to add a paragraph to the discussion section that explores the challenges and potential consequences of using alternative fairness metrics in this context. For example, what would be the clinical and ethical implications of enforcing \"similar predicted outcomes\" (demographic parity) for a condition where a sensitive attribute like age is a known and legitimate risk factor? Briefly exploring this \"what if\" scenario would demonstrate a comprehensive understanding of the broader fairness landscape and proactively address potential questions from readers about why certain metrics were not chosen.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- This study evaluates the methods on three large-scale public datasets (AREDS, MIMIC-CXR, ADNI) spanning multiple imaging modalities (fundus, X-ray, MRI) and clinical outcomes (AMD progression, mortality, Alzheimer’s).\n\n- This study quantifies five distinct bias mechanisms (e.g., censoring disparities, feature distribution gaps) and shows their real-world prevalence.\n\n- This work demonstrates that while pre-training improves accuracy, it does not inherently improve fairness, a crucial insight for medical AI.\n\nWeaknesses\n\n- The authors' large-scale evaluation, involving over 20,000 trained models, reveals that while these methods can sometimes reduce performance gaps, the improvements are not guaranteed and can be accompanied by a decrease in predictive accuracy. Table 1 demonstrates this inconsistency; for instance, in the ADNI-Sex setting, nearly all fairness algorithms, including DRO, SR, FRL, and CSA, show a negative impact on the Integrated Brier Score (IBS) accuracy compared to the DeepHit baseline.\n\n- Similar to the point above, the paper notes that when sources of bias are small, applying these fairness methods can paradoxically worsen performance disparities. Statistical analysis confirms this, with Friedman and Nemenyi post-hoc tests showing that current algorithms do not offer a significant improvement in mitigating bias over the baseline DeepHit model. This indicates a fundamental gap in existing methodologies, which struggle to address multiple, co-existing sources of bias without a trade-off in utility.\n\n- The theoretical analysis (page 4) relies on performance metrics that satisfy triangle inequality and symmetry properties, which, while described as \"relatively mild\" , may not hold for all metrics used in practice, a limitation the authors acknowledge. More critically, the paper's framework for identifying sources of bias relies on being able to quantify disparities in data distributions, but it concedes that identifying the true causal pathways that create these disparities is a significant challenge that requires deep clinical insight. The study simulates distribution shifts by synthetically corrupting images, TTE labels, or censoring indicators for specific groups. While this provides a controlled environment for analysis, these artificial shifts, such as applying a Gaussian blur or flipping 90% of censoring indicators, may not fully represent the complex, subtle, and multifaceted nature of distribution shifts encountered in real-world clinical settings.\n\n- The study also finds that pre-training, a widely used technique for improving model performance, offers limited benefits for fairness. While pre-training was shown to improve predictive accuracy, especially for the smaller and more complex ADNI dataset , it had a \"minimal impact on fairness\". Statistical tests showed no significant improvement in fairness in 18 out of 24 experimental settings, suggesting pre-training alone is insufficient to produce equitable TTE predictions. This weakness implies that simply using larger, more general models is not a panacea for fairness issues and that dedicated fairness interventions are still necessary, even if they themselves are currently of limited effectiveness.\n\n- The selection of fairness metrics, while justified, narrows the scope of the evaluation. The authors focus on metrics that ensure equal predictive performance across groups, arguing that other categories of fairness metrics are less suitable for medical imaging. They dismiss metrics requiring a similarity measure as \"difficult to establish for medical images\" and deem metrics that enforce similar predicted outcomes as potentially \"nonsensical\" when a sensitive attribute like age is a strong risk factor. While these are reasonable justifications, this focus excludes a broader discussion of individual fairness or alternative group fairness definitions that might be relevant, thereby constraining the paper's conclusions to a specific type of fairness.", "was_revised": true, "final_justification": "The authors have addressed most of my comments. Due to the unclear complexity within TTE scenarios, I would retain my original rating.", "rebuttal": "Thank you for your thoughtful review. We appreciate your recognition that our work addresses a critical gap in fair AI research. Below, we address your remaining concerns:\n\n> **Q1. Regarding the limitations of existing fairness methods for TTE prediction.**\n\nWe would like to clarify that the observed limitations are not weaknesses of our work. Instead, our study is the first systematic investigation into the fairness behavior of existing (fair) TTE prediction algorithms. Through extensive evaluation, we identified several key limitations of current methods:\n\n- Fairness algorithms exhibit inconsistent performance across datasets, sensitive attributes, and fairness metrics.\n\n- Enforcing fairness may harms predictive performance, especially in case sources of bias are small, causing fairness-accuracy trade-off.\n\n- Pre-training improves model accuracy but has minimal impact on fairness.\n\nIn sum, these findings highlight shortcomings in existing methods that our framework helps to reveal, not deficiencies in our own contribution.\n\n> **Q2. Regarding triangle inequality and symmetric assumptions for TTE performance metrics.**\n\nWhile some metrics such as the concordance index do not satisfy the triangle inequality or symmetry assumptions, they can be modified to do so. For example, prior work [1] proposed the symmetric discordance index, a variant of the concordance index that satisfies both assumptions. Moreover, these assumptions were introduced solely to facilitate the proof of Theorem 1 and do not affect our other theoretical results, including Proposition 2 and the decomposition formula in Equation (4), or our empirical findings.\n\n> **Q3. Regarding causal pathway identification.**\n\nWe would like to clarify that causal pathway identification is not required for our work. Our goal is to measure bias from medical images which can be quantified from observed data. The causal graph serves as a conceptual tool to explain why disparities exist in data distributions across subgroups, not as a target for full identification.\n\n> **Q4. Regarding distribution shift settings.**\n\nWe would like to clarify that distribution shifts created by intervening causal graph in our study mimic real-world scenarios as follows.\n\n- Adding noise to images mimics real-world scenarios such as patients in different geographic locations are scans with different equipment. This causes the medical image to appear systematically different for groups in each location.\n\n- Adding noise to TTE labels mimics real-world scenarios such as delayed or inaccurate event recording in EHR system.\n\n- Flipping censoring indices mimics real-world scenarios in which certain groups experience less consistent access to care due to financial or geographic. Thus, these groups are more likely to drop out of care, resulting in a higher censoring rate.\n\nBy leveraging these data intervention mechanisms, we can analyze how different sources of bias and distribution shift types impact the fairness behavior of TTE prediction models. While studying real-world distribution shifts is important, it is often challenging to identify the exact sources of bias, which complicates the fairness analysis.\n\n> **Q5. Regarding fairness metrics for TTE prediction.**\n\nWe would like to clarify that performance gaps across subgroups are appropriate fairness metrics and not narrows the scope of our study. These metrics are widely used in medical imaging tasks [2,3,4] and are clinically meaningful. While other fairness metrics exist in the TTE literature, they may not be appropriate for the clinical domain. For example:\n\n- Enforcing similar predicted TTE outcomes across subgroups may compromise clinical relevance and accuracy. A case study in Alzheimer’s disease illustrates this point well: requiring young and elderly individuals to have similar predicted times to disease onset would not make sense. \n- Enforcing similarity between similar patients is impractical in medical imaging, where well-defined measures of semantic similarity between images are lacking. This could lead to forcing similarity between inherently different cases.\n\nBased on your suggestion, we will expand the discussion of fairness metrics for medical image prognosis in the revised manuscript.\n\n**References**\n\n[1] Shaker, Ammar, and Carolin Lawrence. \"Multi-source survival domain adaptation.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 8. 2023.\n\n[2] Seyyed-Kalantari, Laleh, et al. \"Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations.\" Nature medicine 27.12 (2021): 2176-2182.\n\n[3] Vaidya, Anurag, et al. \"Demographic bias in misdiagnosis by computational pathology models.\" Nature Medicine 30.4 (2024): 1174-1190.\n\n[4] Tian, Yu, et al. \"FairSeg: A Large-Scale Medical Image Segmentation Dataset for Fairness Learning Using Segment Anything Model with Fair Error-Bound Scaling.\" The Twelfth International Conference on Learning Representations (2024)."}, {"review_id": "uv4wwsGSTC", "reviewer": "Reviewer_g759", "summary": "The authors propose a comprehensive framework, FairTTE, for evaluating and understanding fairness in medical image prognosis. They formalize a causal decomposition of bias sources in TTE prediction (e.g. disparities in image features, censoring, event labels, etc.) and use it to explain why fairness is hard to achieve in this setting. To support empirical study, the paper adopts three diverse medical imaging datasets with TTE outcomes: predicting in-hospital mortality from chest X-rays (MIMIC-CXR), late-stage AMD from retinal fundus images (AREDS), and Alzheimer’s progression from brain MRIs (ADNI). The experiments observe that current fairness interventions provide only limited mitigation. While some methods reduce group gaps, they often hurt overall accuracy, and none consistently outperform a baseline model in fairness. The paper also highlights that pre-training on large datasets improves overall TTE accuracy but does not consistently improve fairness, and that distribution shifts (changes between train/test data) further exacerbate unfairness.", "questions": "1. Given that no existing methods work consistently, what do the authors envision as potential holistic approaches to address all bias sources?\n2. The paper adopts TTE models that assume discretized time intervals for event prediction. Could the authors clarify how the choice of discretization granularity (e.g., interval width or number of bins) impacts both fairness and predictive accuracy?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Medical prognostic model with fairness learning is an important yet underexplored direction.\n2. The paper provides a theoretical causal model and bias decomposition that rigorously characterize how sensitive attributes can influence TTE predictions. \n3. The authors collect three real-world publicly datasets spanning different imaging modalities and outcomes, each annotated with demographic attributes. This is significant to provide a benchmark for fairness in medical image prognosis.\n4. In addition, the evaluation is thorough. The authors train and evaluate a large number of models with varying sensitive attributes (race, sex, age), model types, and fairness algorithms.\n\nWeaknesses:\n1. The paper is primarily an analysis and benchmarking effort. It does not propose a novel algorithm to improve fairness in TTE predictions.\n2. The causal bias framework distinguishes “fair” vs “unfair” pathways. While conceptually sound, it relies on unobservable constructs (true health state Z, latent confounders U) and strong assumptions (e.g., whether data disparities reflect spurious correlations or true causal effects of A). In practice, it may be hard to definitively categorize which biases are “unfair” (e.g., if age genuinely affects disease progression vs. if it’s a proxy for unequal care).\n3. The authors simulate distribution shifts by adding noise or flipping censoring indicators for one group. While this stress-test is useful, it is somewhat artificial. The paper claims to base these on real medical scenarios, but the main text doesn’t elaborate much on how realistic these shifts are.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We appreciate your recognition that our work addresses an important yet underexplored problem and that our evaluation is thorough. We would now like to address your remaining concern below:\n\n> **Q1. Regarding the proposal of a new fairness method for TTE prediction.**\n\nWe would like to clarify that our primary contribution lies in providing a novel and systematic framework to analyze the fairness behavior of (fair) TTE prediction models in medical image prognosis—a critical yet underexplored area. To the best of our knowledge, this is the first comprehensive study that rigorously examines how existing (fair) TTE prediction methods perform across different datasets, sensitive attributes, and fairness metrics under clinically relevant distribution shifts. Our framework enables the identification of key limitations in current approaches, offering actionable insights for the development of more robust and equitable models in the future. Thus, our work fills an important methodological gap and lays the foundation for principled fairness evaluation in medical image prognosis.\n\n> **Q2. Regarding identification of fair and unfair pathways in the causal graph.**\n\nWe would like to clarify that the proposed causal graph serves as a conceptual tool to identify and explain potential sources of bias and to guide fairness analysis in a more structured manner. As stated in our main paper, our primary goal is to measure bias sources from medical images and TTE prediction models, which can be quantified using observed data; therefore, full identification of causal pathways is not required for our study. That said, we acknowledge your point that identifying causal pathways can be challenging in some medical contexts, and input from domain experts may be necessary to assess the fairness of specific causal pathways (as noted at the end of page 6 in our paper). Ultimately, fairness decisions should be informed by context-specific ethical and clinical considerations. Our framework is intended to support that reasoning—not to replace it.\n\n> **Q3. Regarding distribution shift settings.**\n\nWe would like to clarify that distribution shifts created by intervening causal graph in our study mimic real-world scenarios as follows.\n\n- Adding noise to images mimics real-world scenarios such as patients in different geographic locations are scans with different equipment. This causes the medical image to appear systematically different for groups in each location.\n\n- Adding noise to TTE labels mimics real-world scenarios such as delayed or inaccurate event recording in EHR system.\n\n- Flipping censoring indices mimics real-world scenarios in which certain groups experience less consistent access to care due to financial or geographic. Thus, these groups are more likely to drop out of care, resulting in a higher censoring rate.\n\nBy leveraging these data intervention mechanisms, we can analyze how different sources of bias impact the fairness behavior of TTE prediction models. While studying real-world distribution shifts is important, it is often challenging to identify the exact sources of bias and distribution shift mechanisms which complicates the fairness analysis. Based on your suggestion, we will add this discussion into our revised version.\n\n> **Q4. Regarding development of potential holistic approaches for fair TTE prediction.**\n\nWe would like to clarify that a holistic approach to fairness should span the entire AI development pipeline (i.e., from data collection and fairness metric selection to model architecture and evaluation) to effectively mitigate bias in real-world applications. In particular, we provide some suggestions for future research on fairness in medical image prognosis as below:\n\n- Identifying the causal nature of bias is essential for distinguishing between fair and unfair sources, enabling models to adjust for specific pathways rather than treating all disparities uniformly.\n\n- Developing appropriate bias mitigation strategies and fairness metrics that leverage fair causal pathways while minimizing the influence of unfair ones.\n\n- Collaborating with clinicians and domain experts to define application-specific fairness objectives ensures that fairness interventions align with clinical utility and ethical standards.\n\nFor example, if we identify an unfair causal pathway between a sensitive attribute and the underlying health condition during the data collection process, techniques such as fair representation learning may help remove spurious correlations. However, if the sensitive attribute is a clinically meaningful risk factor for the TTE outcome, the model should be encouraged to extract sensitive attribute-related features from the image to calibrate its predictions appropriately. In such cases, applying fairness methods that obscure this information like fair representation learning could inappropriately degrade performance by forcing the model to ignore clinically relevant signals.\n\n> **Q5. Regarding the impact of discretization granularity on both fairness and accuracy in TTE prediction models.**\n\nWe would like to clarify that the choice of discretization granularity in TTE prediction models can influence both predictive accuracy and fairness outcomes. However, based on our preliminary experiments, varying the discretization scheme did not alter the key empirical findings of our study. Therefore, we chose to fix the discretization scheme across all experiments to isolate the effects of fairness interventions and distribution shifts on both fairness and accuracy in TTE prediction.\n\nTo provide additional context, discretization granularity affects model behavior in several ways. Finer-grained discretization allows the model to capture more precise temporal information, potentially improving predictive accuracy. However, it can also increase model complexity and lead to data sparsity within individual bins, particularly for underrepresented subgroups, which may exacerbate disparities and fairness concerns. On the other hand, coarser discretization reduces data sparsity and simplifies the learning task, which can enhance model stability and, in some cases, mitigate fairness disparities. This benefit, however, comes at the cost of lower temporal resolution, which may limit the model’s ability to make accurate TTE predictions.\nWe will expand this discussion in the revised manuscript to better clarify the trade-offs involved."}, {"review_id": "IQCQMXc3qC", "reviewer": "Reviewer_Z8y7", "summary": "The authors propose a a comprehensive framework, FairTTE for assessing and analyzing fairness in TTE prediction for medical imaging.  The FairTTE framework involved training / evaluating thousands of TTE models across 3 major medical imaging datasets, diverse fairness algorithms and TTE algorithms, and performing causal analysis to identify and quantify five primary sources of bias in TTE datasets (image feature distributions, TTE distributions, censoring rates, and in the MI between images and outcomes). \n\nThe authors report several key findings:\n- consistent performance gaps observed between different demographic groups across all tested imaging modalities and datasets. \n- current SOTA algorithms often fail to consistently mitigate bias, and any gains in fairness are frequently accompanied by a decrease in the model's predictive accuracy. \n- pre-training models on large datasets improves their accuracy but has a minimal impact on fairness. \n- maintaining fairness becomes increasingly difficult under distribution shifts", "questions": "potentially consider adding pathology as an additional medical imaging modality given recent rise in popularity and adoption of digital pathology, and earlier work investigating fairness in various digital pathology learning algorithms and pretrained foundation models. e.g. Vaidya, A., Chen, R.J., Williamson, D.F.K. et al. Demographic bias in misdiagnosis by computational pathology models. Nat Med\n\nThat said, I am more than impressed by the current scope of investigation by the authors.", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n- solid theoretical framework, large-scale comprehensive evaluation, clear presentation\n\nWeaknesses\n- nothing major, but could benefit from expanding to the scope of medical imaging to include pathology if time permits.", "was_revised": true, "final_justification": "No change in assessment. The authors did perform additional experiments on pathology image data but I acknowledge the current scope is already sufficient.", "rebuttal": "Thank you for your thoughtful review. We appreciate your recognition of our work's solid theoretical framework, large-scale comprehensive evaluation, and clear presentation. We would like to address your remaining concern below:\n\n> **Q1. Regarding experiments with pathology images.**\n\nThank you for the suggestion. Pathology images indeed play a critical role in medicine by providing detailed visual information about tissue morphology, which is essential for accurate disease diagnosis, such as in cancer. However, to the best of our knowledge, existing publicly available pathology image datasets lack either sensitive attributes or longitudinal clinical outcomes—both of which are necessary to study fairness in TTE prediction. For example, the dataset referenced in [1] does not include longitudinal outcomes, making it unsuitable for TTE analysis. Additionally, due to the time constraints of the rebuttal phase and the complexity involved in accessing (e.g., controlled access) and processing pathology images (e.g., whole-slide image handling, linking images with clinical outcomes) from sources such as TCGA [2], conducting a large-scale analysis using these data is currently not feasible. We will continue to monitor developments in this area and plan to incorporate pathology image datasets into our future studies as soon as appropriate resources become available.\n\n**References**\n\n[1] Vaidya, Anurag, et al. \"Demographic bias in misdiagnosis by computational pathology models.\" Nature Medicine 30.4 (2024): 1174-1190.\n\n[2] https://www.cancer.gov/ccg/research/genome-sequencing/tcga"}, {"review_id": "oSX8mBRoc7", "reviewer": "Reviewer_jaCt", "summary": "This paper introduces FairTTE, a comprehensive causal framework for investigating and quantifying fairness in time-to-event (TTE) prediction within medical imaging. Prior fairness research in medical AI has focused largely on diagnostic tasks (classification, segmentation). This work targets prognosis tasks, with additional challenges due to censoring, lack of standard fairness metrics, and complex data generation processes. The paper combines a theoretical causal analysis to decompose sources of bias with empirical studies on three real-world datasets (MIMIC-CXR, ADNI, AREDS). The theoretical analysis shows how bias arises, and extensive experiments corroborate that bias is pervasive across demographic groups. The analysis shows that existing fairness algorithms only partially mitigate bias and distribution shifts pose further challenges.", "questions": "-\tCan the authors elaborate on how they envision FairTTE being used by practitioners—e.g., in regulatory audits, model certification, or dataset curation?", "limitations": "OK.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths: \n\n1.\tPrognosis is potentially even more important than diagnosis in medical AI, and fairness in this area is substantially understudied compared to fair diagnosis. \n\n2.\tCompared to many empirical papers on fairness, this paper deposes the ways in which bias can arise using structural causal models, which provides a clearer way to understand and think about this important problem. \n\n3.\tEmpirical validation of state-of-the-art TTE and fairness methods is thorough, with good statistical analysis. I appreciate this good HPO protocol and statistical testing compared to many more ad-hoc papers in this area.\n\n4.\tGood that data and code are open for reproducibility. \n\n5.\tInsightful Findings: The theoretical analysis is helpful for thinking about and understanding the challenges of fairness. And the empirical study has interesting results such as the limited ability of existing fairness methods to ameliorate bias, a breakdown of the different sources of bias, and the fragility of fairness interventions under distribution shift.\n\n6.\tGenerally the paper is very well written, if a bit dense at times. Including the appendix, it’s very thorough, and comprehensive. \n\n7.\tI  really appreciate the practical examples of the SCMs in App G2. \n\n\n\nWeaknesses: \n\n1.\tMissed insight opportunity: It would have been great to have some suggestions about how to develop new algorithms or improve existing algorithms based on the insights of the SCM analysis.  Alternatively, some more concrete analysis on what is the limit of the best achievable fairness given the practical causal relations that mix the influence of conditions and sensitive attributes.\n\n2.\tMissed insight opportunity: In terms of the causal analysis of bias, it seems that some aspects are in common with standard diagnosis (eg: different image distribution across groups), and others are unique to prognosis (eg: Different TTE distribution across groups). It would have been nice to have an explicit compare & contrast to see what’s the unique part in this work that differs from standard SCM perspective on diagnosis. \n\n3.\tOn comparing images: (i) L106-110 claims that medical images are high-dimensional points that are hard to compare. But then as part of the bias analysis (Fig5, App F4), actually the paper directly compares images in embedding space using Wasserstein distance. (ii) In F4: Is comparing the images in embedding space what we should do? It seems a bit odd, because it depends on the feature representation that each deep network learns, which will be different for different prognosis and mitigation algorithms. \n\n4.\tEvaluation: The test-set bootstrapping for CI calculation (Fig4) seems like not the important source of variability. My experience is that especially for small-ish medical datasets (eg: AREDS, ADNI), sensitive fairness algorithms, and  automated HPO, then the huge source of variability is in seeds for the training phase.", "was_revised": true, "final_justification": "I have read the reviews and feedback. \nI tend to think this paper brings some insight. While there are inevitably some weaknesses, none seem to be fatal. Therefore it could be accepted in my opinion.", "rebuttal": "Thank you for your thoughtful review. We appreciate your recognition that our work addresses an important problem in medical AI, presents a thorough evaluation, and offers insightful findings. We would now like to address your remaining concern below:\n\n> **Q1. Regarding suggestions to develop new fairness algorithm for TTE prediction based on the causal graph.**\n\nThank you for this valuable feedback. We offer several suggestions for future research on fairness in medical image prognosis:\n\n- Identifying the causal nature of bias is essential for distinguishing between fair and unfair sources, enabling models to adjust for specific pathways rather than treating all disparities uniformly.\n\n- Developing appropriate bias mitigation strategies and fairness metrics that leverage fair causal pathways while minimizing the influence of unfair ones.\n\n- Collaborating with clinicians and domain experts to define application-specific fairness objectives ensures that fairness interventions align with clinical utility and ethical standards.\n\nIn our opinion, a holistic approach to fairness should span the entire AI development pipeline (i.e., from data collection and fairness metric selection to model architecture and evaluation) to effectively mitigate bias in real-world applications.\n\nFor example, if we identify an unfair causal pathway between a sensitive attribute and the underlying health condition during the data collection process, techniques such as fair representation learning may help remove spurious correlations. However, if the sensitive attribute is a clinically meaningful risk factor for the TTE outcome, the model should be encouraged to extract sensitive attribute-related features from the image to calibrate its predictions appropriately. In such cases, applying fairness methods that obscure this information like fair representation learning could inappropriately degrade performance by forcing the model to ignore clinically relevant signals.\n\nWe will incorporate this discussion in the revised manuscript to highlight the potential impact of our framework beyond benchmarking.\n\n> **Q2. Regarding comparison with medical image diagnosis scenario.**\n\nWe clarify that there are both shared and distinct aspects in the causal analysis of bias between diagnosis and prognosis tasks. While causal analysis has been applied to study biases in diagnostic settings, prognostic tasks introduce unique challenges that require specialized consideration. Specifically, TTE prediction involves longitudinal outcomes and censoring, which fundamentally differ from the static labels used in diagnosis. These temporal aspects introduce additional causal pathways through which bias can arise, such as disparities in event timing or censoring mechanisms across subgroups.\n\nAmong the five sources of bias identified in our framework, disparities in image feature distributions have been explored in prior works on medical image diagnosis [1,2]. However, the remaining sources, particularly those related to TTE outcome and censoring, are unique to prognosis.\n\nWe will make these distinctions more explicit in the revised manuscript to highlight how our framework extends fairness analysis in prognostic modeling beyond the conventional diagnostic setting.\n\n> **Q3. Regarding image comparison.**\n\nWe would like to clarify that the image comparisons discussed in Section 2.1 (lines 106-116) and Appendix F.4 (lines 1418-1425) are conducted in different contexts. Specifically, in Section 2.1, the goal is to define a fairness metric that requires measuring distances between images in a clinically meaningful way, which is inherently a challenging task. In contrast, the analysis in Appendix F.4 involves comparing image features from a statistical perspective to quantify sources of bias. In this latter case, using a statistical distance measure such as the Wasserstein distance is sufficient and appropriate for the intended analysis.\n\n> **Q4. Regarding source of variability.**\n\nWe would like to clarify that, in addition to calculating confidence intervals using test-set bootstrapping, we also performed statistical significance testing across results from all random seeds in our experiments to verify the robustness of our findings to training variability induced by random initialization.\n\n> **Q5. Regarding the practical utility of FairTTE.**\n\nWe would like to clarify that FairTTE can be used as a versatile tool that can support practitioners in multiple stages of the ML pipeline for medical prognosis. Specifically:\n\n- FairTTE provides a standardized framework to evaluate fairness across subgroups under realistic clinical conditions, such as distribution shifts and censoring. Regulators could use FairTTE to assess whether a TTE prediction model exhibits consistent performance and equitable treatment across demographic groups, forming part of an evidence base for approval or deployment.\n\n- FairTTE also enables practitioners to systematically identify sources of bias, whether from image features, outcome disparities, labeling process, or censoring mechanisms. This can guide unbiased dataset curation process.\n\n- Developers can use FairTTE to benchmark the fairness behavior of candidate models or fairness-aware training strategies across multiple axes (datasets, attributes, metrics). This allows for more informed model selection and ensures that fairness-performance trade-offs are clearly understood before deployment.\n\nUltimately, we see FairTTE as a practical and extensible framework that can be integrated into the clinical ML development lifecycle.\n\n**References**\n\n[1] Jones, Charles, Mélanie Roschewitz, and Ben Glocker. \"The role of subgroup separability in group-fair medical image classification.\" International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2023.\n\n[2] Yang, Yuzhe, et al. \"The limits of fair medical imaging AI in real-world generalization.\" Nature Medicine 30.10 (2024): 2838-2848."}, {"review_id": "oQaUMN66YS", "reviewer": "Reviewer_PBKE", "summary": "The paper introduces a framework for assessing fairness in time-to-event settings (in the setting of censored observations), where the prediction of event times is based on image data and possibly other covariates. The fairness criterion considered looks at the difference in the error rates between demographic groups. The paper's results include bounds for the fairness metric in terms of appropriate group separability quantifications, and a decomposition of differences between prediction distributions into various contributions.", "questions": "- (Q1) The decomposition is Equation (4) claims to decompose a disparity. Usually, statistical or causal disparities are quantified through contrasts, such as differences/ratios of conditional expectations/potential outcomes. Curiously, in Equation (4), the LHS $P(y, \\delta \\mid x, a)$ does not seem to be a contrast. Clarifying this would be helpful (e.g., are ratios considered for different values of $a$?).\n- (Q2) Is there a connection of the decomposition with causal decompositions appearing in the causal fairness literature, e.g. [3]?\n- (Q3) It would be interesting to understand whether there is a more fundamental argument for the proposed fairness criterion. In the text, there are arguments about the relation of distribution shift and fairness. What about the following simple example\n$$\n\\begin{align}\nA &= \\text{Bernoulli}(0.5) \\\\\\\\\nY &= A + (1+A) \\epsilon,\n\\end{align}\n$$\nwhere $\\epsilon$ follows an exponential distribution. In the above example (reflecting heteroscedastic noise across groups), why would equal error in prediction across groups be a useful criterion, if prediction is inherently more difficult for one group?\n\n[3] Zhang, Junzhe, and Elias Bareinboim. \"Fairness in decision-making—the causal explanation formula.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.", "limitations": "No, the limitations are not adequately addressed (see Weakness (1)), and some of the paper's claims seem to be stronger than the results and methodology support.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- (S1) The paper deals with an important, and timely topic of research,\n- (S2) The paper is well-written and clear,\n- (S3) Some parts of the analysis seems to be offering a novel approach,\n- (S4) This is visibly a high-effort submission, with extensive empirical analyses.\n\nWeaknesses:\n- (W1) Survival analysis for medical data is plagued with multiple serious challenges, with some of the most significant issues being informative censoring [1] and competing risks [2]. The methodology proposed in the paper does not even mention these issues, or attempts to address such problems, which are known to be key for state-of-the-art survival analysis in medical data. This is also reflected in the empirical data analyses. For instance, in the MIMIC-CXR dataset, patients are censored 1-year after their last discharge. However, hospital discharge is not treated as censoring, meaning that patients remain in the risk set, even though they are not really at risk. A core issue is that there is no information in the dataset about mortality outcomes outside the hospital. Furthermore, competing risks are an important consideration, since admission patterns differ significantly for racial minorities, and different groups have different competing risk after hospital discharge. Thus, it remains unclear if the proposed analysis is appropriate. Similarly, the setting considered for the ADNI dataset seems to include events that can only happen on patient visits, which is a setting where interval censoring is commonly considered. In summary, non-informative censoring, which is assumed throughout the paper, is a major limitation that appears to be glossed over.\n \n- (W2) Another point that may confound the reader are the claims about offering a causal perspective. Figures 2 and 3 seem to discuss the causal structure, but it remains unclear whether anything related to causal mechanisms is discussed. The figures specify DAGs that correspond to different settings that could be true or not, but these graphs are not used for any identification purposes (or specifying assumptions?). Furthermore, in the decomposition of Equation (4), the terms appearing seem to talk about mutual information, which are also not causal quantities, and the decomposition seems largely unrelated to different causal mechanisms (or at least this remains unclear). It seems from a distance that the analysis could rest on Bayesian networks entirely (relying only on notions of observational distributions), which makes the framing of a \"causal perspective\" somewhat unusual.\n\nReferences\n\n[1] Leung, Kwan-Moon, Robert M. Elashoff, and Abdelmonem A. Afifi. \"Censoring issues in survival analysis.\" Annual review of public health 18.1 (1997): 83-104.\n\n[2] Andersen, Per Kragh, et al. \"Competing risks in epidemiology: possibilities and pitfalls.\" International journal of epidemiology 41.3 (2012): 861-870.", "was_revised": true, "final_justification": "Doing state-of-the-art medical survival analysis is impossible without considering informative censoring and competing risks, which the paper does not handle.\n\nHowever, the paper does move the current baseline significantly in the right direction, offering a first at-scale evaluation of fairness in TTE prediction.", "rebuttal": "Thank you for your thoughtful review. We appreciate your recognition that our work deals with an important and timely topic, offers novel approach, and provides extensive empirical analysis. We would now like to address your remaining concern below:\n\n> **Q1. Regarding the challenges of TTE prediction such as informative censoring and competing risks.**\n\nWe completely agree that informative censoring and competing risks present important challenges in TTE prediction. However, we would like to clarify that these challenges arise primarily from the perspective of TTE modeling itself, rather than from the fairness perspective, which is the main focus of our study.\n\nIn this work—the first comprehensive benchmark of fairness in medical image prognosis—we investigate the fairness behavior of data and models under the standard TTE prediction setting, which assumes a single risk and non-informative right censoring. This setting already presents significant experimental challenges, especially given the constraints of publicly available medical image datasets. Constructing datasets suitable for studying informative censoring or competing risks would require substantial effort during data collection and annotation, which we believe is currently not feasible at scale.\n\nThat said, we greatly appreciate your valuable suggestion and will consider exploring these directions in future work. We also emphasize that our proposed framework FairTTE is flexible and can incorporate a wide range of TTE models and datasets, including those designed for settings involving competing risks or informative censoring, when such data become available. We will include a discussion of these future directions in the revised version of the manuscript.\n\n> **Q2. Regarding causal perspective used in our study.**\n\nWe would like to clarify that the causal graphs proposed in our study serve as conceptual tools to illustrate how sources of bias may arise in medical image data—specifically, when there are causal pathways between sensitive attributes and other variables in the graph. In addition, our study focuses on group fairness notions, which are quantified using observed data distributions, rather than on counterfactual fairness, which requires causal path identification. As such, the causal graph is intended to aid in interpreting why disparities exist across subgroups, not to serve as a target for complete causal identification.\n\n> **Q3. Regarding decomposition formula.**\n\nAs mentioned in our response to Q2, our study focuses on group fairness notions, which are quantified based on observed data distributions. The decomposition formula is designed to break down disparities between group-specific data distributions into quantifiable terms, enabling us to systematically analyze group fairness in our experiments. In other words, the decomposition formula is not related to causal or counterfactual fairness concepts explored in works such as [1], but rather provides a descriptive tool grounded in observed data.\n\n> **Q4. Regarding the appropriateness of fairness notion in medical domain.**\n\nWe would like to clarify that there is no universal fairness notion suitable for all scenarios. Selecting an appropriate fairness criterion in the medical domain requires careful consideration of clinical context, ethical principles, and statistical validity. An effective fairness notion should align with the intended use of the model, its potential impact on different patient subgroups, and real-world constraints.\n\nIn our study, we adopt group fairness metrics, such as performance gaps across subgroups, because they are widely used in medical imaging tasks and offer clinically meaningful interpretations [2,3,4]. However, we acknowledge that this does not imply such metrics are appropriate for all medical applications. In contexts where prediction is inherently more challenging for one group, as you mentioned, minimizing performance gaps may inadvertently reduce performance for the advantaged group, which can be counterproductive. In such cases, maximizing subgroup-wise performance may be a more suitable objective.\n\nFurthermore, by examining fairness through the lens of the data generation process, as we do in our study, we offer a principled framework for understanding the sources of bias. This perspective can help guide the selection of fairness notions that are best suited to the specific characteristics and ethical goals of a given application.\n\n**References**\n\n[1] Zhang, Junzhe, and Elias Bareinboim. \"Fairness in decision-making—the causal explanation formula.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.\n\n[2] Seyyed-Kalantari, Laleh, et al. \"Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations.\" Nature medicine 27.12 (2021): 2176-2182.\n\n[3] Vaidya, Anurag, et al. \"Demographic bias in misdiagnosis by computational pathology models.\" Nature Medicine 30.4 (2024): 1174-1190.\n\n[4] Tian, Yu, et al. \"FairSeg: A Large-Scale Medical Image Segmentation Dataset for Fairness Learning Using Segment Anything Model with Fair Error-Bound Scaling.\" The Twelfth International Conference on Learning Representations (2024)"}]

Author rebuttal:

Meta-review: The paper introduces FairTTE, a unified framework for analyzing fairness in time-to-event prediction in medical imaging. The authors provide theoretical analyses from a causal perspective, and then run a large-scale study across multiple datasets, TTE models, and fairness interventions.

**Strengths**

Reviewers broadly agree this is an important, underexplored problem, and that the authors have conducted a thorough empirical evaluation. Writing and presentation are clear. and several reviewers view the work as impactful even without a new algorithm.

**Initial Weaknesses**

1. The most serious weakness is ignoring information censoring and competing risks (PBKE, VK6y).
2. The distribution-shift simulations may not reflect clinical reality (VK6y, g759).
3. The paper focuses on group fairness, which may not be appropriate in certain cases (PBKE, VK6y).

**Rebuttal Period**

The rebuttal provided many clarifications. The authors have acknowledged some of the limitations above, and promised to mention them in the revision. All reviewers had a positive assessment of the paper after the rebuttal period.

**Overall Evaluation**

I believe this is a comprehensive, principled framework that will be impactful to the field in shaping future proposed methods. While the survival analysis caveats are valid, the authors have acknowledged them in their rebuttals, and should include these limitations in the camera ready. I recommend acceptance.
Decision: Accept (poster)