FairPFN: A Tabular Foundation Model for Causal Fairness
Authors: Jake Robertson
Noah Hollmann
Samuel Müller
Noor Awad
Frank Hutter
Venue: ICML 2025 poster (2025)
Keywords: Causal Fairness
Foundation Models
In-Context-Learning
Prior-Fitted-Networks
Primary area: general_machine_learning->causality

Abstract: Machine learning (ML) systems are utilized in critical sectors such as healthcare, law enforcement, and finance, but often rely on historical data that contains demographic biases, leading to decisions that perpetuate or intensify existing inequalities. Causal and counterfactual fairness provide a transparent, human-in-the-loop framework to mitigate algorithmic discrimination, aligning closely with legal doctrines of direct and indirect discrimination. However, current causal fairness frameworks hold a key limitation in that they assume prior knowledge of the correct causal model, restricting their applicability in complex fairness scenarios where causal models are unknown or difficult to identify. To bridge this gap, we propose FairPFN, a tabular foundation model pre-trained on synthetic causal fairness data to identify and mitigate the causal effects of protected attributes in its predictions. FairPFN's key contribution is that it requires no knowledge of the causal model and demonstrates strong performance across a diverse set of hand-crafted and real-world causal scenarios relative to robust baseline methods. FairPFN paves the way for a promising direction for future research, making causal fairness more accessible to a wider variety of complex fairness problems.

FairPFN: A Tabular Foundation Model for Causal Fairness
Jake Robertson Noah Hollmann Samuel Müller 52 Noor Awad? 2 Frank Hutter 412
Abstract
aware ML community, critiques highlight the contextual lim- itations and lack of transferabilityy of current statistical fair- ness measures to practical legislative frameworks (Weerts et al., 2023). In response, the field of causal fairness has emerged, providing a transparent and human-in-the- the-loop causal framework for assessing and mitigating algorithmic bias with strong analogy to existing anti-discrimination legal doctrines (Plecko & Bareinboim, 2024).
Machine learning (ML) systems are utilized in critical sectors, such as healthcare, law enforce- ment, and finance. However, these systems are often trained on historical data that contains demo- graphic biases, leading ML decisions that per- petuate exacerbate existing social inequalities. Causal fairness provides transparent, human-in- the-loop framework to mitigate algorithmic dis- crimination, aligning closely with legal doctrines of direct and indirect discrimination. However, current causal fairness frameworks hold key lim- itation in that they assume prior knowledge of the correct causal model, restricting their applicabil- ity in complex faimess scenarios where causal models are unknown or difficult to identify. To bridge this gap, we propose FairPEN, a tabular foundation model pre-trained on synthetic causal fairness data to identify and mitigate the causal effects of protected attributes in its predictions. FairPEN's key contribution is that it requires no knowledge of the causal model and still demon- strates strong performance in identifying and re- moving protected causal effects across a diverse set of hand-crafted and real-world scenarios rel- ative to robust baseline methods. FairPFN paves the way for promising future research, making causal fairnsss more accessible to wider variety of complex fairness problems.
A recent review comparing outcome- based and causal fair- ness approaches (Castelnovo et al., 2022) argues that the non-identifiabilit of causal models from observational data (Pearl, 2009) limits the usage of current causal fairness frameworks in practical applications. In practice, users must provide full partial information about the underly- ing causal model, challenging task given the complexity of systemic inequalities. Furthermore, an incorrectly pre- sumed causal graph, such as one falsely assuming a variable is independent of a protected attribute, can invalidate causal fairness metrics (Ma et al., 2023; Binkyté-Sadauskien et al., 2022), resulting in fairwashing and fostering a false sense of security and trust.
This paper takes a bold new perspective on achieving causal fairness. Our key contribution is FairPFN, a tabular foun- dation model for causal fairness, pre-trained on synthetic causal fairness data to learn to identify and remove the causal effects of protected attributes in tabular classifica- tion settings. When used on new dataset, FairPPN does not rely user-specified causal model or graph, instead solely relying on the causally- during pre-training. We demonstrate through extensive ex- periments that FairPFN effectively and consistently miti- gates the causal impact of protected attributes across various hand-crafted and real-world scenarios, yielding causally fair predictions without user specifeed causal information. We summarize our various contributions:
data it has seen
1. Introduction
Algorithmic discrimination is among the most pressing Al- related risks of our time, manifesting when machine learning (ML) systems produce outcomes that disproportionately dis- advantage historically marginalized groups (Angwin et al., 2016). Despite significant advancements by the fairness-
1. PFNs for Causal Fairness We propose paradigm shift for algorithmic fairness, in which a transformer is pre-trained on synthetic causal faimness data.
Equua contribution 'ELLIS Institute Tübingen University of Freiburg 'Charité University Medicine Berlin "Prior Labs 1 Meta. Correspondence to: Jake Robertson <robertsje @cs.uni- frieburg.de>.
2. Causal Fairness Prior: We introduce synthetic causal data prior which offers a comprehensive rep- resentation for fairness datasets, modeling protected attributes as binary exogenous causes.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

FairPEN: A Tabular Foundation Model for Causal Fairness
3. Foundation Model:
We present FairPEN, a foundation model for causal fairesss which, given only observa- tional data, identifies and removes the causal effect of binary, exogenous protected attributes in predic- tions, and demonstrates strong performance in terms of both causal fairness and predictive accuracy on a combination of hand-crafted and real-world causal scenarios. We provide a prediction interface to eval- uate and assess our pre-trained model, well code to generate and visualize our pre-training data at htttt::///// :/gp::::///itt thub. com/ c /Faa IPFN.
3. Background
This section establishes the scientific foundation of FairrFNN, including terminology relevant to algorithmic faimesss causal ML, counterfactual fairness, and prior-data fitted networks (PFNs).
Algorithmic Fairness Algorithmic discrimination occurs when historical biases against demographic groups (e.g., ethnicity, sex) are reflected in the training data of ML al- gorithms, leading to the perpetuation and amplification of these biases in predictions (Barocas et al., 2023). Fair- ness research focuses on measuring algorithmic bias and developing fairness aawae ML models that produce non- discriminatory predictions. Practitioners have established over 20 fairesss metrics, which generally break down into group-level and individual-level metrics (Castelnovo et al., 2022). These metrics can be used to optimize predictive models, balancing the commonly observed trade-off be- tween fairness and predictive accuracy (Weerts et al., 2024).
2. Related Work
In recent years, causality has gained prominence in the field of algorithmic fairness. providing fairness researchers with structural framework reason about algorithmic diserim- ination. Unlike traditional faimess research (Kamishima et al., 2012; Agarwal et al., 2018; Hardt et al., 2016), which focuses primarily on optimizing statistical fairness measures, causal fairness frameworks concentrate on the structure of bias. This approach involves modeling causal relationships among protected attributes, observed variables, and out- comes, assessing the causal effects of protected attributes, and mitigating biases using causal methods, such optimal transport (Plecko& & Bareinboim, 2024) or latent variable estimation (Kusner et al., 2017; Ma et al., 2023; Bhaila et al., 2024).
Causal Machine Learning Causal ML is a developing field that leverages modern ML methods for causal reasoning (Pearl, 2009), facilitating advancements in causal discov- ery, causal inference, and causal reasoning (Peters et al., 2014). Causal mechanisms are often represented as Struc- tural Causal Models (SCMs), defined as .M- = (U, O, F), where U are unobservables, O are observables, and F is a set of structural equations. These equations are expressed as J3 X, = (y N2), indicating that an outcome vari- able F depends on its parent variables PA P and independent noise Ny. minelinaassties in the set of structural equations F influence data complexity and identifiability of causal quantities from observational data (Schölkopf et al., 2012). In an SCM, interventions can be made by setting X < I and propagating this value through the model M. posing the question of "what will happen if I do something?"" Coun- terfactuals expand upon the idea of interventions and are relevant when a value of X is already observed, instead posing the question of "what would have happened if some- thing had been different?" In addition to posing a slightly different question, counterfactuals require that exogenous noise terms are held constant, and thus classically require full knowledge of the causal model. In the context of algo- rithmic fairness, we are limited to level of counterfactuals as protected attributes are typically given and already observed.
Counterfactual fairness, introduced by Kusner et al. (2017), posits that predictive outcomes should remain invariant be- tween the actual world and counterfactual scenario in which protected attribute assumes an alternative value. This notion has spurred interest within the fairesss research community, resulting in developments like path-specific ex- tensions (Chiappa, 2019) and the application of Variational Autoencoders (VAEs) to create counterfactually fair latent representations (Ma et al., 2023).
The initial counterfactual fairness framework necessitates comprehensive knowledge of the causal model. In contrast, the Causal Fairness Analysis (CFA) framework (Plecko & Bareinboim, 2024) relaxes this requirement by orga- nizing variables within a Standard Fairness Model (SFM) for bias assessment and mitigation. Moreover, the CFA framework presents the Fairness Cookbook, which defines causal fairness
In causal reasoning frameworks, one major application of counterfactuals is the estimation of causal effects such as the individual and average treatment effects (TTE and ATE) which quantify the difference and expected difference be- tween outcomes under different values of X.
Direct-Effect, and Spurious- Salsss-ffffttteee that directly align with US legal doctrines of disparate impact and treatment. Furthermore, the CFA framework challenges Kusner et al. (2017)'s modeling of protected attributes as exogenous causes, permitting correla- tions between protected attributes and confounding variables that contribute to the legally admissible Spurious-Fing -ffectt.
ITE T Yx-z - YX-xx
(1)
ATE: F(T) = F[ -FVVxel)
(2)
2

---

FairPEN: A Tabular Foundation Model for Causal Fairness
a) Data generation: For each pre-training dataset, we generate an SCM and sample dataset D comprised of prolected altribute A. potentially biased observables Xp. and biased oulcome y. We also sample fair outcome y by removing the outgoing odges of
b) Transformer imput: The observational dataset partitioned into training and validation splits. Given in-context examples Devats the transformer makes es predictions on the inference set D out (A
Fair prediction: The transformer makes prodictions Y, the validation set, and the pre-training loss is calculated with respect the fair outcomes in the validation set. The transformer thus learns the m mapping X, -> Y
Real-world Inference
Structural Causal Model (SCM)
Observational Dataset
Pre-training Loss
U2
FairPEN
A
X1
X2
X3
Ya
y
Y
Ao
22
Yb
X3
Ka)) or p 100000
FairPFN Pre-training
Figure 1. FairPEN Overview: FairPEN is a foundation model for causal fairness, pre-trained synthetic datasets generated from sparse MLPs that represent SCMs with exogenous protected attributes (a). A biased dataset created for each MLP/SCM and supplied as context to the transformer (b), with loss computed based on fair outcomes obtained by excluding the causal influence of the protected attribute (c). In practice, (d) FairPEN takes in only an observational dataset to predict fair targets by integrating over the simplest causal explanations for the biased data.
Counterfactual Fairness a foundational notion of causal fairness introduced by Kusner et al. (2017), requiring that an individual's 's predictive outcome should match that in counterfactual scenario where they belong to differ- ent demographic group. This notion is formalized in the theorem below.
Theorem 3.3 can also be transformed into counterfactual fairness metric by quantifying the difference between natu- ral and counterfactual predictive distributions. In this study quantify counterfactual fairness as the distribution of the counterfactual absolute error (AE) between predictions in each distribution.
Theorem 3.1 (Unit-leve//rroaabiistt), Given an SCM M F) where O AU X, a predictor Y is coun- terfactually fair on the unit-level Vÿ E Y Vx, a, E A
Definition 3.3 (Absolute Error). Given an SCM M (U, O, F') where O A U X, the counterfactual absolute error of predictor Y is the distribution
PP (4) X, A = a) P(i (1) X A = 2,, x.a)
(()A-a(w) = a) P(OA (u))x, A, a.
We note that because the outcomes are condition on the same noise terms u our definition AE builds off of Theorem I. Intuitively, when the AE is skewed towards zero, then most individuals receive the same prediction in both the natural and counterfactual scenarios.
Kusner et al. (2017) notably choose to model protected at- tributes exogenous, which means that they may not be confounded by unobserved variables with respect to out- comes. We note that the definition of counterfactual fairness in Theorem 3.1 the unit-level probabilistic one as clarified by Plecko & Bareinboim (2024), because counterfactual outcomes are generated deterministically with fixed unob- servables U u. Theorem 3.1 can be applied the dataset level to form the population-level version also provided by Plecko & Bareinboim (2024) which measures the alignment of natural and counterfactual predictive distributions.
Kusner et al. (2017) present various implementations of Counterfactually Fair Prediction (CFP). The three levels of CFP can be achieved by fitting a predictive model Y to observable non-descendants any exist (Level-One). inferred values of an exogenous unobserved variable K (Level-Two), additive noise terms (Level-Three), Kusner et al. (2017) acknowledge that in practice, Level-One rarely occurs. Level-Two requires that the causal model be invert- ible, which allows the unobservable K to be inferred by abduction. Level-Three models the scenario as an Additive Noise Model, and thus is the strongest in terms of represen- tational capacity, allowing more degrees of freedom than in
Theorem 3.2 (Population- level). Given an SCM M (U,O,F) F) where = AU X, predictor Y is counterfac- tually fair on the population-level ij E Y, Vx, a, a' E A
P(jA-al X, A x, a) = P(AA-a X, = x. a)
3

---

FairPFN: A Tabular Foundation Model for Causal Fairness
Level-Two represent fair terms. The three levels of CFP depicted in Appendix Figure 22.
Algorithm 1 FairPEN Pre-training
Input:
Number of pre-training epochs E and steps S
Causal Fairness The Causal Fairness Analysis (CFA) framework (Plecko & Bareinboim, 2024) introduces the Standard Fairness Model (SFM), which classifies variables as protected attributes A, mediators Xmed, confounders Xcon/, and outcomes Y. This framework includes a Fair- ness Cookbook of causal fairness metries with a strong anal- ogy to the legal notions of direct and indirect discrimination and business necessity as illustrated in Appendix Figure 23. Plecko & Bareinboim (2024) refute the modeling choice of Kusner et al. (2017) by their inclusion of confounders Xconf in the SFM, arguing that these variables contribute to the legally admissible Spurious-Eiffect (SE).
Transformer M with weights 0
Hypothesis space of SCMs e + begin
for epoch = 1 to E do
for step = 1 to S do
Draw a random SCM @ from &
Sample Dbses = (A, Xbias Yosas) from e where {ao, is exogenous binary protected attribute
Sample yaur from Φ by performing dropout out- going edges of A if any exist
Partition Dosas and juir into train/ vval
Pass Derain into M as context
bias Pass Dias into M to generate Ypred
For simplicity of our experimental results, we follow the modeling of Kusner et al. (2017), and focus on the elim- ination of the Tofal-Effect (TE) of protected attributes as defined by Plecko & Bareinboim (2024), while noting in Section 6 the importance of relaxing this assumption in future extensions.
Calculate loss L = CY
pred: Y/ai
Update weights 0 w.r.t VoL
end for
end for Output: Transformer M: Xbbiss - Ysair
Prior-data Fitted Networks Prior-data Fitted Networks (PFNs) (Müller al., 2022) and TabPFN (Hollmann al., 2023; 2025) represent a paradigm shift from traditional ML with a causal motivation, namely that simple causal mod- els offer a quality explanation for real-world data. PFNs incorporate prior knowledge into transformer models by pre-training on datasets from a specific prior distribution (Müller et al., 2022). TabPFN, popular application of PFNs, applies these ideas to small tabular classification tasks by training transformer synthetic datasets derived from sparse Structural Causal Models (SCMs). As noted in Hollmann et al. (2023), a key advantage of TabPFN is its link to Bayesian Inference; where the transformer ap- proximates the Posterior Predictive Distribution (PPD), thus achieving state-of- of-te-of-thh-arr performance by integrating over simple causal explanations for the data.
steps below.
Data Generating Mechanisms FairPEN pre-training begins by creating synthetic datasets that capture the causal mech- anisms of bias in real-world data. Following the approach of Hollmann et al. (2023), we use Multi-I Perceptrons (MLPs) to model Structural Causal Models (SCMs) via the structural equation f z(P WT x + c). where W de- notes activation weights, represents Gaussian noise, P is a dropout mask sampled from a log-scale to promote sparsity, and is non-linearity. Figure illustrates the connection among sampled MLPs, their corresponding SCMs, and the resulting synthetic pre-training data generated. We note that independent noise terms are not visualized in Figure 1.
Biased Data Generation An MLP is randomly sampled and sparsity is induced through dropout on select edges. The protected attribute is defined as a binary exogenous variable A E {ao, a1) } at the input layer. We uniformly select m features X from the second hidden layer on- wards to capture rich representations of exogenous causes. The target variable Y is chosen from the output layer and discretized into a binary variable using a random thresh- old. A forward pass through the MLP produces a dataset D (A, Xtias Ybias) with n samples containing the causal influence of the protected attribute.
4. Methodology
In this section, we introduce FairPFN, foundation model for legally or ethically sensitive tabular classification prob- lems that draws inspiration from PFNs and principles of causal fairness. We introduce our pre-training scheme, syn- thetic data prior, and draw connections to Bayesian Infer- ence to explain the inner workings of FairPEN.
4.1. FairPEN Pre-Training
First, we present our pre-training scheme, where FairPFN is fit to a prior of synthetic causal faimess data to identify and remove the causal effects of protected attributes in practice from observational data alone. We provide pseudocode for our pre- training algorithm in Algorithm 2, and outline the
Fair Data Generation A second forward pass generates a fair dataset fa by applying dropout to the outgoing edges of the protected attribute A in the MLP, as shown by the red edges in Figure 1. This dropout, similar to that in TabPFN, masks the causal weight of A to zero, effectively reducing its
4

---

FairPEN: A Tabular Foundation Model for Causal Fairness
Algorithm 2 FairPEN Synthetic Data Generation
transformer M. yielding predictions Y ypred. We then com- pute the binary-cross-entropy (BCH) loss prep fair against the fair outcomes Yaar which do not contain effects of the protected attribute. Thus, the transformer M learns the mapping M Xbias + Yfurr.
Input:
Number of exogenous causes U
-umber of endogenous variables U H
Numbeer of features and samples M N
begin
Prior-Fitting The transformer is trained for approximately 3 days on an RTX-2080 GPU on approximately 1.5 million different synthetic data-generating mechanisms, in which we vary the MLP architecture, the number of features m, the sample size n, and the aon-liearities 2.
Define MLP & with depth H and width U
Initialize random weights W : (U x U x H
Sample sparsity masks with same dimensionality as weights
Sample H per-layer non-linearities 2
{Identity, ReLU, Tanh)
Real-World Inference During real-world inference, FairPEN requires no knowledge of causal mechanisms in the data, but instead only takes as input biased observational dataset and implicitly infers potential causal explanations for the data (Figure Id) based on the causally generated data it has seen during pre-training. Crucially, FairPEN pro- vided information regarding which variable is the protected attribute, which is represented in a protected attribute en- coder step in the transformer. A key advantage of FairPEN is its alignment with Bayesian Inference, transformers pre- trained in the PFN framework have been shown to approx- imate the Posterior Predictive Distribution (PPD) (Müller et al., 2022). FairPEN thus approximates a modified PPD, predicting a causally fair target ys given biased features X and a biased dataset D& by integrating over hypotheses for the SCM & E 4:
Initialize output matrix X (U H)
Sample location k of protected attribute in Xo
Sample locations of features Xbiased in X1:H-1, and outcome Ybins in X
Sample protected attribute threshold a and binary val- ues {ao,ai }1}
for 0 to N samples do
Sample values of exogenous causes Xo (U x 1) Sample values of additive noise terms e (U x H)
for i 0 to H - layers do
Pass intermediate representation through hidden layer Xi+1 (P; W?'x, X; +G) end for
Select prot. attr. A, features Xbins and outcome Ybias from Xo. 1 and XH
Binarize A E {ag, a1} } over threshold ae
Set input weights in row & of Wo to 0 for j = to H layers do
plyslab, D&) plyr|ao () (3)
Pass intermediate representation through hidden layer X,+1 (P; WJ'x, x cg) end for
This approach has two advantages: it reduces the neces- sity of precise causal model inference, thereby lowering the risk of fairwashing from incorrect models (Ma et al., 2023), and carries with it regularization-reltte performance im- provements observed in Hollmann al. (2023). We also emphasize that FairPFN is foundation model and thus does not need to be trained for new faimess problems in practice. Instead, FairPEN performs predictions in a single forward pass of the data through the transformer.
Select the fair outcome afr from XH end for
Binarize yfair E {0,1) 1} and yoias E {0,1) over ran-
domly sampled output threshold y/:
Output: Driae = (A, Xtiasi Ubias) and yfair
influence Gaussian noise c. This increases the influence of fair exogenous causes Un and U, and independent noise terms all over the MLP visualized in Figure 1. We note that is sampled from an arbitrary distribution A E {ao, 41 as opposed to A E fo, 1}. since both functions f = 0- wrr + and f = p- Our + yield equivalent outcomes. Only after generating the pre-training dataset is A converted to a binary variable for processing by the transformer.
5. Experiments
This section assesses FairPEN's performance on synthetic and real-world benchmarks, highlighting its capability to remove the causal influence of protected attributes with- out user-specified knowledge of the causal model, while maintaining high predictive accuracy.
5.1. Baselines
In-Context Learning After generating Doins and D Dairrs
we partition them into training and validation sets: Dérain Duel fair and Diaay- We pass Derain as context to the transformer to provide information about feature-target val into the relationships. To simulate inference, we input bias
We implement several baselines to compare FairPEN against a diverse set of traditional ML models, causal-fairness frameworks, and faimess-aware ML approaches. We sum- marize our baselines below, and provide a visualization of
5

---

FairPEN: A Tabular Foundation Model for Causal Fairness
1) Blased
2) Dffect-Effeet
3) Enfirect-fffect
Fair Observable
$) Fair Unobservable
6) Fair Additive Noise
ugg 1))
ut up Fx, fyy - Mis Movn)
Ce Ma.e).Mu.o MDuo)
Py Nguel Mpu A UCi Xy) N(Lo)
*ay fy Mpuo). Mare
Py - Morn), Mire)
Cy Mir. MG.. Mo..o))))
A 11)
- udt 10.00 NUA
A DCD0.1
Xs = mad?
My NG.e)
x; Myde 1
Xs madd
Xa maa molt oo
X, Mudd
wx,Aff my
M wxy wn
Wy, wast B1 2
10r F)
y UCT P)
y 10r R
y B2Y 2 F)
Prot. Am
Outcooe
Caglarr Olsernable
Ohovrvablee
Fav Lholoorrulle
Cause
Addine Notee
Now- dor-doonnd
Suen by FarPFi
Visualization and data generating processes of synthetic causal case studies, a handcrafted of benchmarks designed to evaluate FairPEN's ability to remove various sources of bias in causally generated data. For each group, 100 independent datasets are sampled, varying the number of samples, the standard deviation of noise terms and the base causal effect wa of the protected attribute.
Figure Causal Case Studies:
our baselines applied to the Fair Observable bench- mark in Appendix Figure 25.
5.2. Causal Case Studies
We first evaluate FairPEN using synthetic causal case stud- ies to establish an experimental setting where the data- generating processes and all causal quantities are known, presenting a series of causal case studies with increas- ing difficulty to evaluate FairPEN's capacity to remove various sourees of bias in causally generated data. The Gata-generating processes and structural equations are il- lustrated in Figure 2, following the notation: A for pro- tected attributes, X₁ for biased-observables, X, for fair- observables, U for fair-umobservables, <x for additive noise terms, and Y for the outcome, discretized as Y 1(Y 2 Y). We term a variable X "fair" iff & anc(X). The structural equations in Figure 2 contain exponen- tial linearirssssee ensure the direction of causality is identifiable (Peters et al., 2014), distinguishing the Fai Unobservable and Fair Additive Noise scenar- ios, with the former including unobservable yet identifi- able causal effect U.
Unfair: Fit the entire training set (X,AYY)
Unaware:
Fit to the entire training set (X, A, Y). In- ference returns the average of predictions on the orig- inal test set (X, A) and the test set with alternative protected attribute values (X, a').
Avg. Crft: Fit to the entire training set (X, Inference returns the average (avg.) of predictions on the original test set (X, A) and the counterfactual (entf) test set (XA-sa' A a'd
Constant: Always predicts the majority class
Random: Randomly predicts the target
CF?:
Combination of the three-levels of CFP as pro- posed in Kusner et al. (2017). to non-descendant ob- servables, unobservables, and independent noise terms (Xfairy fuir, ffirr Y).
For a robust evaluation, we generate 100 datasets per case study, varying causal weights of protected attributes wa, sample sizes m (100, 10000) (sampled on log-scale), and the standard deviation E (0, (log-scale) of addi- tive noise terms. We also create counterfactual versions of each dataset to assess FairPEN and its competitors across multiple causal and counterfactual fairness metrics, such as average treatment effect (ATE) and absolute error (AE) between predictions on observational and counterfactual datasets. We highlight that because our synthetic datasets are created from seratch, the fair causes, additive noise terms, counterfactual datasets, and ATE are ground truth. As a result, our baselines that have access to causal quan- tities are more precise in our causal case studies than in world scenarios where this causal information must be inferred.
EGR: Exponentiated Gradient Reduction (EGR) as pro- posed by (Agarwal al., 2018) fit to non-protected attributes (X,Y) with XGBoost (Chen & Guestrin, 2016) base model.
In the CFP, Unfair, Unaware, and Avg. Cntf. base- lines, we employ FairPEN with random noise term passed as a "protected attribute." We opt to use this UnfairPFN in- stead of TabPFN so as to not introduce any TabPFN-specific behavioral characteristics artifacts. We show in Appendix Figure 17 that this reverts FairPEN to normal tabular clas- sifier with competitive peformance to TabPFN. We also note that our Unaware baseline is not the standard approach of dropping the protected attribute. We opt for our own implementation of Unaware as it shows improved causal effect removal to the standard approach (Appendix Figure 17).
6

---

FairPFN: A Tabular Foundation Model for Causal Fairness
Binsed
2. Direct-Effect
3. Indirect-Effee
off competitive with CFP and Cntf. Avg., which use causal quantities from the true data-generating process. This is even the case in the Fair Unobservable and Fair Additive Noise benchmark groups, producing causally fair predictions using only observational variables that are either a protected attribute or causal ancestor of it. This indicates FairPEN's capacity to infer latent unobserv- ables, which we further investigate in Section 5.3. We also highlight how the Cntf. Avg. baseline achieves lower error than CFP. We believe that this is due to Cntf. Avg- having access to both the observational and counterfactual datasets, which implicitly contains causal weights and non- linearities, while CFP is given only fair unobservables and must infer this causal information. The fact that a PFN used as base model in Cntf. Avg. could further ex- plain this performance gain, as access more observable variables helps guide the PFN toward accurate predictions realistic for the data. We suggest that this Cnt Avg. as an alternative should be explored in future studies.
Eroo Sloor1l.)) (l:AlC)
0.30
0.50
0.50
8.40
1440
1140
0..0
0.30
0030
6.20
0.20
0.20
0:00
05
O 10 0.015 0.20
0.00
0.03
0.10 0.2
6.00 6dd G.I5 0.30 0.25
Observable
Fair Unolservable
Fair Additive Noise
Erroe (1-DC)
0.90
1.50
1.50
0.40
0.40
0.10
6:30
0.30
0.30
D.30
0:20
mar to i 0.200.25 Causal Effisct (ATE)
OUz Cuto 6.35 Causal Effect (ATE)
Cu00 CLIO (LI EL25
Causal Effect (ATE)
Unfair
Constant
EGR *
FairPPN
Unaware
Random
CFP
Cutf. Avg.
Figure 3. Fairness Accuracy Trade-Off' (Synthetic): Average Treatment Effect (ATE) of predictions, predictive error (I-AUC), and Pareto Front performance of FairPFN versus baselines in our causal case studies. Baselines which have access to causal information are indicated by light border. FairPEN is on the Pareto Front on 40% of synthetic datasets using only observational data, demonstrating competitive performance with the CFP and Cntf. Avg. baselines that utilize causal quantities from the true data-generen data-generating process.
Causal Effect Removal We evaluate FairPFN's s efficacy in causal effect removal by analyzing box plots depicting the median, interquartile range (IQR), and average treatment effect (ATE) of predictions, compared to baseline predictive models that also do not access causal information (Figure 4). We observe that FairPFN exhibits a smaller IQR than the state-of-the-art bias mitigation method EGR. In an average rank test across 600 synthetic datasets, FairPFN achieves an average rank of 1.88 out 4. We provide a comparison of FairPEN against all baselines in Figure 24. We note that our case studies crucially fit our prior assumptions about the causal representation of protected attributes. We show in Appendix Figure 13 that FairPFN reverts to normal classifier when, for example, the exogeneity assumption is violated.
Blased
Direct-Effect
3. Indirect-Effer
Cansal Edfect (ATS)
0.35
0.25
0.25
0.25
Fair Observable
5. Fuir Unolosrvable
6. Fair Additive Noise
Cunall Effeat (ATE)
0.5
as
825
0.25
0.25
4.5
Ablation Study We finally conduct an ablation study to evaluate FairPEN's performance in causal effect removal across synthetic datasets with varying size, noise levels, and base rates of causal effect. Results indicate that FairPFN maintains consistent performance across different noise lev- els and base rates, improving in causal effect removal as dataset size increases and causal effects become easier to distinguish from spurious correlations (Dai et al., 1997). We note that the variance of FairPEN, illustrated by box-plot outliers in Figure that extend to 0.2 and -0.2, is primar- ily arises from small datasets with fewer than 250 samples (Appendix Figure 11), limiting FairPEN's ability to iden- tify causal mechanisms. We also show in Appendix Figure 14 that FairPFN's fairness behavior remains consistent as graph complexity increases, though accuracy drops do to the combinatorially increasing problem complexity.
Avp. Rank (ATE)
TairPTN: 1.884
DGR: 2.11/4
Unaware: 2.16/4
Unfair: 3.42/4
Figure 4. Causal Fairness (Synthetic): Average Treatment Effect (ATE) of predictions of FairPEN compared to baselines which do not have access to causal information. FairPEN consistently removes the causal effect with a margin of error of (-0.2, 0.2) and achieves an average rank of 1.88 out of 4, only to outperformed on the Di rereerttffeee benchmark where Unaware is the op- timal strategy.
Fairness-Accuracy Trade-Off Figure 3 presents the fairnsss-accuracy trade-off for FairPEN and its baselines, displaying the mean absolute treatment effect (ATE) and mean predictive enror (I-AUC) observed across synthetic datasets, along with the Pareto Front of non-dominated solutions. FairPEN (which only uses observational data) attains Pareto Optimal performance in 40% of the 600 synthetic datasets, exhibiting a carness--accuran trade-
For a more in-depth analysis these results, we refer to Appendix B.
7

---

FairPFN: A Tabular Foundation Model for Causal Fairness
Law School Admissions
Adalt Census Income
Unfuir
Unuwure
Random
EGR
FairPTN
Rack
SEX
Law School Admissions
Adult Census Income
ELEAT
Poot, Atr
Cnscome
Ulgarr Olterrabble
Fair Gkabserrable
Caute
Additive Noive
Seac by Fao
0.0
0.2
11.
0L6
0.5 1.0
0.0
0.2
11.4
0L.
0.5
1.0
Absollate Error (AE)
Absolute Error (AL)
Figure 5. World Datasets: Assumed causal graphs of real-
Figure 7. Counterfactual Fairness of Absolute Error (AE) between predictive distributions on obser- vational and counterfactual datasets. Compared to baselines that do not have access to causal information, FairPEN achieves the lowest median and maximum AE on both datasets.
Distributions
world datasets Law School Admissions and Adult Census Income.
Law School Admissions
Adult Census Income
Unlair
0.50
Unoware
Constant
Random
B43
0.39
EGR
0:00
0.30
CFP F
0.38
0.25
0.35
0.00
0.20
0.02 0.05
Cutf. Avg.
CLAIRE
dataset level. Figure 7 illustrates the distribution of Absolute Error (AE) achieved by FairPEN and baselines that do not have access to causal information. FairPFN significantly reduces this error in both datasets, achieving maximum di- vergences of less than 0.05 on the Law School dataset and 0.2 on the Adult Census Income dataset. For a visual inter- pretation of the AE on our real-world datasets we refer to Appendix Figure 16.
Causal Ellect (ATE)
Causal Effect (ATE)
Figure 6. Pairness-Accuracy Trade-off Worll: Average Treatment Effect (ATE) of predictions, predictive error (I-AUC), and Pareto Front of the performance of FairPFN compared to our basclines each of 5 validation folds (light) and across all five folds (solid) of our real-world datasets. Baselines which have access causal information have a light border. FairPEN matches the performance of baselines which have access to inferred causal information with only access to observational data.
In contrast, EGR performs similarly to Random in terms of counterfactual divergence, confirming previous studies which show that optimmizing for group faimesss metrics does not optimize for individual level criteria (Robertson et al., 2024). Interestingly, in an evaluation of group fairness metric Statistical Parity (DSP) FairPEN outperforms EGR on both real-world data and causal case studies, baseline was specifically optimized for this metric (Appendix Figures 20 and 21).
5.3. Real-Worlld Data
This section evaluates FairPEN's causal effect removal, pre-
dictive error, and correlation with fair latent variables on two real-world datasets with established causal graphs (Fig- ure 5). For a description of our real-world datasets and the methods we use to obtain causal models, see Appendix A.
Trust & Interpretability In order to build trust in FairPFN and explain its internal workings, we first perform a feature correlation analysis of FairPFN and baseline mod- els using the Law School Admissions dataset. We measure the Kendall rank correlation between observable variables "LSAT" and "UGPA," and inferred noise terms ELSAT and CUGPA, with predicted admission probabilities FÝ
Fairness-Accurayy Trade-Off We evaluate FairPFN's ef- fectiveness real-world data in reducing the causal impact of protected attributes while maintaining strong predictive accuracy. Figure 6 shows the mean prediction average treat- ment effect (ATE) and predictive error (I-AUC) across 5 K-fold cross-validation iterations. FairPEN achieves a pre- diction ATE below 0.01 on both datasets and maintains accuracy comparable to Unfair. Furthermore, FairPFN exhibits lower variability in prediction ATE across folds compared to EGR, indicating stable causal effect removal'.
Figure 8 shows that despite only having access to observa- tional data, FairPFN's predictions correlate with fair noise terms similarly to CFP which was fit solely to these vari- ables. This result suggests FairPFN's ability to not only integrate over realistic causal explanations for the data, but also correctly remove the causal effect of the protected at- tribute such that its predictions are influenced only by fair exogenous causes. We note that while FairPFN mitigates the effect of "Race," it increases the correlation of "Sex" compared to the Unfair r and CFP baselines. We discuss how future versions of FairPEN can tackle the problem of in- tersectionality in Section 6. We also further investigate this
Counterfactual Fairness Next, we evaluate the counter- factual fairness of FairPEN on worldworld datasets as intro- duced in Section 3, noting that the following analysis is conducted at the individual sample level, rather than at the
We note that we also evaluate pre-trained version of CLAIRE (Ma et al., 2023) the Adult Census income dataset, but observe little improvement to RGR
8

---

FairPEN: A Tabular Foundation Model for Causal Fairness
Law School Admissions
accommodating multi-objective scenarios (Lin et al., 2019), regression problems (Hollmann al., 2025), and time se- ries (Hoo et al., 2025). Additionally, FairPFN can generate causally fair versions of previously unfair observables, im- proving prediction explainability. This enables practitioners to use FairPFN as a fairness preprocessing technique while employing their preferred predictive models in practical applications.
0.7
Corcelation clX. F.PA)
0.6
0.5
0.4
0.3
0.2
0.1
0.0
Race
Sex
UGPA Feature Name (X)
LSAT
RUGPP
$1SAT
PFNs for Causal ML FairPEN implicitly provides evidence for the efficacy of PFNs to perofm causal tasks, and we be- lieve that our methodology can be extended to more complex challenges both within and outside of algorithmic fairness. In algorithmic faimesss, one promising extension could be path-specific effect removal (Chiappa, 2019). For example, in medical diagnosis, distinguishing social effects of sex (e.g., sampling bias, male-focus of clinical studies) from biological effects (e.g.. symptom differences across sex) is essential for fair and individualized treatment and care. Be- yond fairness, we believe PFNs can predict interventional and counterfactual effects, with the latter potentially facili- tating FairPEN's evaluation in real-world contexts without relying on estimated causal models. Currently, FairPEN can also mitigate the influence of binary exogenous confounders, such as smoking, on the prediction of treatment success.
Unfair
CFP
FairPEN
Cotf. Avg.
Figure& 8. Feature Correlation (Law School): Kendall Tau rank correlation between feature values and the predictions FairPEN compared to our baseline models. FairPEN produces predictions that correlate with fair noise terms EUGPA and ELS to a similar extent as the CFP baseline, variables which it has never seen in context-or at inference.
result in Appendix Figure 12, which confirms that FairPEN does not remove the effect of additional protected attributes other than the one specified.
We also observe in Figure 3 and 6 the strong performance of our Cntff. Avg. baseline, which predicts the average outcome probability in the observational and counterfac- tual worlds. We thus carry out similarity test to Cnt Avg. in Appendix Tables 1 and 2, calculating for each other baseline the mean difference in predictions, the stan- dard deviation of this distribution, and the percentage of outliers. We find that FairPEN's predictions are among the closest to this target, with a mean error on synthetic datasets of 0.00+0.06 06 with 1.87% of samples falling outside of three standard deviations, and mean error on real-world datasets of 0.02+0.04 04 with 0.36% of outlying samples.
Alignment to Anti-Discrimination Law Future versions of FairPEN could also relax the assumption of exogenous protected attributes, enabling differentiation between legally admissible spurious effects and direct or indirect effects. An- other key concept proposed by (Plecko & Bareinboim, 2024) introduces "Business Necessity" (BN) variables that allow the impact of the protected attribute to indirectly contribute to outcomes to achieve specified business objectives, such as a research company hiring doctorate holders. In EU law, the analogous "objective justification" " concept necessitates a "proportionality test," asserting that justifiable indirect effects must persist only as necessary (Weerts et al., 2023). We contend that proportionality bears causal interpretation, akin to counterfactual explanations (Wachter et al., 2018).
6. Future Work & Discussion
This study introduces FairPEN, a tabular foundation model pretrained to minimize the causal influence of protected attributes in binary classification tasks using solely observa- tional data. FairPFN overcomes key limitation in causal fairness by eliminating the need for user-supplied knowl- edge of the true causal graph, facilitating its use in complex, unidentifiable causal scenarios. This approach enhances the applicability of causal faimess and opens new research avenues.
Impact Statement
This study attempts to overcome a current limitation in causal fairness, making what we believe is a useful frame- work for addressing algorithmic discrimination, more ac- cessible to wider variety of complex faimess problems. While the goal of this work is to have positive impact on problem we think is crucial, we acknowledge that we our perspective on fairness is limited in scope to align with EU/US legal doctrines of anti-discriminatio. These doc- trines are not representative of the world as a whole, and even within these systems, there are vastly different nor- mative viewpoints regarding what constitutes algorithmic fairness and justice.
Extended Problem Scope We limit our experimental scope to simple testable setting with single, binary protected at- tribute but believe that our prior and transformer architecture can be extended to handle multiple, non-binary protected attributes, addressing both their individual effects and in- tersectional interactions. We also suggest that FairPFN is capable of predicting not only fair binary target but also

---

FairPEN: A Tabular Foundation Model for Causal Fairness
Acknowledgements
Ding, E, Hardt, M., Miller, J., and Schmidt, L. Retiring adult: New datasets for fair machine learning. In Ranzato, M., Beygelzimer, A., Nguyen, K., Liang, P., Vaughan, J., and Dauphin, Y. (eds.), Proceedings of the 35th Inter- national Conference on Advances in Neural Information Processing Systems (NeurIPS'21), volume 34, pp. 6478- 6490, 2021.
The authors of this work would like to thank the reviewers, editors and organizers of ICML 25. We would like to addi- tionally thank the Zuse School ELIZA Master's Scholarship Program for their financial and professional support of our main author. We would finally like to thank Sai Prasanna, Magnus Bühler, and Prof. Dr. Thorsten Schmidt for their insights, feedback, and discussion.
Dua, D. and Graff, C. UCI machine learning repository, 2017.
References
Hardt, M., Price, E., and Srebro, N. Equality of opportunity in supervised learning. In Lee, D., Sugiyama, M., von Luxburg, U.. Guyon, I., and Garnett, R. (eds.), Proceed- ings of the 30th International Conference on Advances in Neural Information Processing Systems (NeurIPS' 16), pp. 3323-3331, 2016.
Agarwal, A., Beygelzimer, A., Dudík, M., Langford, J., and Wallach, H. A reductions approach to fair classification. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th In- ternational Conference on Machine Learning (ICML') 18), volume 80, pp. 60-69. Proceedings of Machine Learning Research, 2018
Hollmann, N., Müller, S.,, Eggensperger, K., and Hutter, F. TabPFN: transformer that solves small tabular classifi- cation problems in a second. International Conference on Learning Representations (ICLR'23) 2023. Published online: iclr.cc. cc.
Angwin, J., Larson, J., Mattu, S., and Kirchner, L. Machine bias. ProPublica, May, 23(2016):139 159, 2016.
Barocas, S., Hardt, M., and Narayanan, A. Fairness and Machine Learning: Limitations and opportunities. MIT Press, 2023.
Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A.,
Körfer, M., Hoo, S. B.. Schirmeister, R. T., and Hutter, F. Accurate predictions on small data with tabular foundation model. Nature, 63778045) 2025.
Bhaila, K., Van, M., Edemacu, K., Zhao, C.,, Chen, F., and Wu, X. Fair in-context learning via latent concept vari- ables, 2024.
Hoo, S. B., Müller, S., Salinas, D., and Hutter, F. The tabular foundation model tabpfn outperforms specialized time series forecasting models based on simple features, 2025.
Sinkyte-Sdduskients R., Makhlouf, K., Pinzón, C., Zhioua, S., and Palamidessi, C. Causal discovery for fairness, 2022.
Hoyer, P. O., Janzing, D., Mooij, J. M., Peters, J., and Schölkopf, Nonlinear causal discovery with additive noise models. In Platt, J. and Koller, D. (eds.), Proceed- ings of the 22 International Conference on Advances in Neural Information Processing Systems (NeurlPS'O8) 08), pp. 689-696, 2008
Castelnovo, A.. Crupi, R., Greco, G., Regoli, D., Penco, I. G., and Cosentini, A. C. A clarification of the nuances in the fairness metrics landscape. Scientific Reports, 12 (1), 2022.
Chen, T. and Guestrin, C. XGBoost: A scalable tree boost- ing system. In Krishnapuram, B., Shah, M., Smola, A., Aggarwal, C., Shen, D., and Rastogi, R. (eds.), Proceed- ings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining (KDD'16) 16), pp- 785-794, 2016.
Kamishima, T., Akaho, S., Asoh, H., and Sakuma, J. Fairness-aware classifier with prejudice remover regu- larizer. In Machine Learning and Knowledge Discovery Databases: European Conference, PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part 23, pp. 35-50. Springer, 2012.
Chiappa, S. Path-specific counterfactual fairness. In Henten- ryck, P. V. and Zhou, Z. (eds.), Proceedings of the Thirty- Third Conference on Artificial Intelligence (AAAI'19) volume 33, pp. 7801-7808. AAAI Press, 2019.
Kusner, M., Loftus, J., Russell, C.,, and Silva, R. Counterfac- tual faimess. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.). Proceedings of the 31st International Conference on Advances in Neural Information Processing Systems (NeurIPS' 17), pp. 4069-4079, 2017.
Dai, H., Korb, K. B., Wallace, C. S., and Wu, X. A study of causal discovery with weak links and small samples. In Pollack, E. (ed.), Proceedings the 15th International Joint Conference on Artificial Intelligence (IJCAI'95), 1997.
Lin, X., Zhen, H., Li, Z., Zhang, Q., and Kwong, S. Pareto multi-task learning. 2019.
10

---

FairPFN: A Tabular Foundation Model for Causal Fairness
Ma, J., Guo, R., Zhang, A., and Li, J. Learning for counter- factual fairesss from observational data. In Singh, A. K., Sun, Y., Akoglu, L., Gunopulos, D., Yan, X., Kumar, R., Ozcan, F., and Ye, J. (eds.), Proceedings of the 29th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KD'233) pp. 1620-1630, 2023.
Weerts, H., Pfisterer, F., Feurer, M., Eggensperger, K., Bergman, E., Awad, N., Vanschoren, J., Pechenizkiy, M., Bischl, B., and Hutter, F. Can faimess be automated? guidelines and opportunities for faimesss-awaee automl. Journal of Artificial Intelligence Research, 79:639-677, 2024.
Wightman, L. F. Lsac national longitudinal bar passage study. Isac research report series. 1998.
Müller, S., Hollmann, N., Arango, S., Grabocka, J., and Hutter, F. Transformers can do Bayesian inference. In Proceedings of the International Conference on Learn- ing Representations (ICLR'22), 22), 2022. Published online: iclr.ce cc.
Pearl, J. Causality: Models, Reasoning and Inference. Cam- bridge University Press, 2009.
Peters, J., Janzing, D., and Schölkopf, B. Causal inference on discrete data using additive noise models. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 33 (12):2436-2433) 2011.
Peters, J., Mooij, J. M., Janzing, D., and Schölkopf, B. Causal discovery with continuous additive noise models. Journal of Machine Learning Research, 15:2009-2053, 2014.
Plecko, D. and Bareinboim, E. Causal fairness analysis. Foundations and Trends Machine Learning, 17:304- 589, 2024.
Robertson, J.. Schmidt, T., Hutter, F,, and Awad, N. H. A human-in-the-loop fairness- model selection frame- work for complex fairness objective landscapes. In Das, S., Green, B. P., Varshney, K., Ganapini, M., and Renda, A. (eds.), Proceedings of the Seventh AAAI/ACM Confer- ence on AI, Ethics, and Society (AIES-24) Full Archival Papers, October 21-23, 2024, San Jose, California, USA Volume 1, pp. 1231-1242. AAAI Press, 2024.
Schölkopf, B., Janzing, D., Peters, J., Sgouritsa, E., Zhang, K., and Mooij, J. On causal and anticausal learning. In Langford, J. and Pineau, J. (eds.), Proceedings of the 29th International Conference on Machine Learning (2C Omnipress, 2012.
Sharma, A. and Kiciman, E. Dowhy: An end-to-end library for causal inference. arXiv:201 1. .042 16 [stut.ME], 2020.
Wachter, S., Mittelstadt, B., and Russell, Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harvard Journal of Law and Technology, 288842-87,200
Weerts, H., Xenidis, R., Tarissan, F., Olsen, H. P., and Pech- enizkiy, M. Algorithmic unfairness through the lens of eu non-discrimination law. In Proceedings of the 2023 ACM Conference on Fairess,, Accountability, and Trans- parency. pp. 805 816, 2023.

---

FairPEN: A Tabular Foundation Model for Causal Fairness
A. Real-World World Datasets
Law School Admissions The first dataset is the Law School Admissions dataset from the 1998 LSAC National Lon- gitudinal Bar Passage Study (Wightman, 1998), which includes admissions data fr approximately 30,000 US law school applicants, revealing disparities bar passage rates and first-year averages by ethnicity. We generate counterfactual data and measure causal effects using slightly different causal model than what originally proposed by Kusner et al. (2017), which additionally includes edges "UGPA" - "LSAT" and "LSAT" - "FYA". These edges have plausible temporal explanation, and create more realistic scenario where "Race" and "Sex" have both a direct and indirect effect on first year averages.
Causal Modeling with Do We use the causal graph in Figure 5 (left) and observational data as inputs for the dowhy. , gcm module (Sharma & Kiciman, 2020), employing an automated search using the dowhy. .gm.. auto, which selects the best predictive model in model zoo of non- non-linear tree-based models represent each edge, minimizing either the MSE or negative F1-score depending on the distribution of target following Hoyer et al. (2008) and Peters et al. (2011). 1), We apply each models generate counterfactual datasets, allowing for the estimation of the Average Treatment Effect (ATE) and absolute error (AE). We also use the compute moisee se function to estimate noise terms CGPA and CTS for our CFP baseline.
Adult Census Income The second dataset, derived from the 1994 US Census, is the Adult Census Income problem (Dua & Graff, 2017), containing demographic and income outcome data (LNC 2 50K) for nearly 50,000 individuals?. We fit causal model to assess the Average Treatment Effect (ATE) of the protected attribute RACE, generate a counterfactual dataset, and calculate noise term values e.
B. Ablation Study
To evaluate FairPEN's performance across datasets with varying characteristics, conduct ablation study comparing the prediction Average Treatment Effect (ATE) of FairPEN and Unfair under different noise levels, base rates of the protected attribute's causal effect, and dataset sizes.
Base Rate Causal Effect We analyze the distributions of prediction ATE from FairPEN and Unfair across five quintiles (QI-Q5) of base ATE (Figure 9). FairPEN's prediction ATE remains stable, while Unfair's prediction ATE increases linearly. In datasets within the Biased, Direct Effect, Lev,l-Two, and Level-Three benchmark groups, where the protected attribute has a high base ATE (Q5), FairPEN exhibits greater tendency for positive discrimination, resulting in negative prediction ATE values.
Dataset Noise Analyzing dataset noise, indicated by the standard deviation (STD) a of exogenous noise in the structural equations Figure 10 shows that FairPFN retains consistency across varying noise levels. Conversely, Unfair exhibits decreased and more peaked distributions of prediction ATE as noise increases from Q1 to Q5, suggests that noise terms may obscure causal effects and diminish their observed impact in the data.
Dataset Size Ablation studies on dataset size (Figure 11) show that FairPFN's prediction ATE displays tighter distribution with larger datasets, indicating improved performance in causal effect removal. This improvement arises from better identification causal mechanisms as data availability increases, enabling the transformer to distinguish noise from causal effects.
C. Future Extensions
In this section we expand upon our discussion of future extensions of FairPFN, in order to encourage the community to build upon and expand our approach.
We note that Adult has been heavily criticized in the faimess literature (Ding et al., 2021) due to evidence of sampling bias and an arbitrary chosen income threshold, but elect include it due its widely accepted causal model and appearance benchmark in other similar studies (Ma et al., 2023)
12

---

FairPEN: A Tabular Foundation Model for Causal Fairness
FairPFN
Unfair
1. Biased
2. Direct-Effed
3. Indirect-Effect
Pred. Causal Effeet (ATE)
1.0
0.8
0.6
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.2
-022
-004-0..0
0.000:22
00200.7
0.0-022
0.408.8
00.0.07
0.7-00117
0.170.36
0.26-045
0445-064
401-0001
001-004
0.4-0.11
0.1--225
0.500.83
Fair Observable
5. Fair Unobservable
6. Fair Additive Noise
Pred. Causal Effect (ATE)
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.0
0.0
-0.2
.0.1-0.05
0.5-0.11
11-02223
0:23-0.11
0.41-0.79
00-0.07
0.02-0.14
0.14.0.44
0.24-0399
a 0.19.0.72
.0.1-0:00
0.15-0.11
0.11-022
0.240.88
0.38 0.9
0
0.
Base Causal Effect (ATE)
Base Causal Effect (ATE)
Base Causal Effect (ATE)
Figure 9. Effeet of Base ATE (Synthetic): Distributions of prediction ATE produced by FairPEN and Unfair over quintiles (Q1-Q5)
of the protected attributes's base causal effect (base ATE). FairPEN remains consistent quintiles, sometimes over-correcting and producing negative prediction ATE in QS.
Regression Problems FairPFN can be pre-trained regression model with very little architectural changes by discretiz- ing continuous output distributions into piecewise intervals and calculating misclassification costs in order to reflect the natural ordering between categories (). Thoroughly evaluated in (Hollmann et al., 2025), such post-proccessing strategies have shown strong performance tabular regression problems and enable the effective use of classification architectures for continuous targets.
Protected Attributes in the Wild While we limit the scope of this study to binary classification tasks with single, binary protected attributes, lecnowledge that real-world fairness-aware MI problems often more complex than that. More precisely, protected attributes be not only binary, but continuous mulit-category, and discrimination may occur only with respect to individual protected attributes but with respect to multiple and the interactions between them. Our prior is currently extensible to handle multiple by changing the number of protected attributes that are sampled into each synthetic dataset, removing the outgoing edges of all protected attributes to generate ffarr, and informing the transformer about which variables are protected attributes. Changing the distribution of protected attributes also possible, and simply requires transporting the protected attribute into the distribution(s) of choice either before or after its natural continuous value is propagated through the MLP during pre-training.
13

---

FairPEN: A Tabular Foundation Model for Causal Fairness
FairPFN
Unfair
1. Biased
2. Direct-Effect
3. Indirect-EEfee
Pred. Causal Effect (ATE)
1.2
1,2
1.2
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
-02
-0.2
-0.2
-0.4
-044
-0.4
0.12-0.88
0.18-0227
027-04
0.4.0.9
0,59..888
0.38-0511
0.51-669
0.69.095
0.95-1.0
1.2
1.20-1.75
0.1240118
0.18-0.66
0.26-0.37
0.37-252
0552775
4. Fair Observable
5. Fair Unobservable
Fair Additive Noise
Pred, Causal Effect (ATB)
1.2
1.2
0.6
0.0
-22
-022
a4
004
0.400.44
0.54-0.72
0.730.97
097-1.3
1.31.75
03550667
06740.81
081-0
0.88-1199
0.8800511
0.51-0699
069-0.95
22911.75
1. 1.19.1
099-1.
Additive Noise (std.)
Additive Noise (std.)
Additive Noise (std.)
Figure 10. Effect of Dataset Noise (Synthetic): Distributions of prediction ATE produced by FairPEN and Unfair over quintiles
(Q1-Q5) of the standard deviation (std.) of exogenous noise the data. FairPFN remains consistent across quintiles, while increased noise decreases the prediction ATE of Unfair
D. Supplementary Results
14

---

FairPEN: A Tabular Foundation Model for Causal Fairness
1. Biased
2. Direct-Effect
3. Indirect-Effect
Pred. Causal Effect (ATE)
0.2
0.2
0.2
0.0
0.0
0.0
0.1
-0.1
0.2
-0.2
-0.2
98-250
2504530
630-1883
1583 398-
19988
98-250
2504330
630-1583
1583
1583-3981
9998
98-250
250-630
1583
3981
390 pong
3081
1583
3o0
630
188
4.
Fair Observable
5. Fair Unobservable
6. Fair Additive Noise
Pred. Causal Effect (ATE)
0.2
0.1
-0.1
-0.1
-0.1
-0.2
98-250
230-630
630-1587 1583
1581-3981
1998
98-250
250-630
630 1583
9998
98-250
250-630
630-1583
1883-098
3981
3981 9998
1882
3981
3081
Dataset Size
Dataset Size
Dataset Size
Figure 11. Effect of Dataset Size (Synthetic): Distributions of prediction ATE produced by FairPEN over quintiles (Q1-Q5) of dataset
sizes from 100-10,000 (log-scale). FairPEN becomes better its task of removing the causal effect of protected attributes when more data is available.
Multiple Protected Attributes
Multiple Prot. Attrs.
Unfair FairPEN
A1
0.6
Emro AUC)
Ao
Xb
EXp
0.4
0.2
Xf
Y
Ey
0.00
0.25
0.50
0.75
1.00
Causal Effect (ATE)
Figure 12. Multiple Protected Attributes (Synthetic): Distributions of prediction ATE and predictive accuracy produced by FairPEN vs the Unfair predictor when there multiple protected attributes. This violates FairPEN's prior assumptions and reverts to normal classifier.
15

---

FairPEN: A Tabular Foundation Model for Causal Fairness
Endogenous Prot. Attrs.
Endogenous Protected Attribute
0.7
Unfair
E Aa
0.6
FairPEN
A1
Ao
0.5
Error (1 AUC)
0.4
0.3
EY
0.2
1
Yb
0.1
0.0
0.1
0.2
0.3
0.4
Causal Effect (ATE)
Figure 13. Endogenous Protected Attributes (Synthetic): Distributions of prediction ATE and predictive accuracy produced by FairPEN the Unfair predictor when the protected attribute is endogenous. This violates FairPEN's prior assumptions and reverts it to normal classifier.
1.0
0.8
Metric
0.6
Statistical Parity (DSP)
Accuracy (AUC)
0.4
0.2
0.0
100
200
SCM Size (# Nodes)
Figure 14, Graph Complexity (Prior): Distributions of Statistical Parity and predictive accuracy produced by FairPEN on prior samples with graph complexity between 10 and 200 nodes. As graph complexity increases, accuracy drops fairness remains constant.
16

---

FairPEN: A Tabular Foundation Model for Causal Fairness
1. Biased
2. Direct-Effect
3. Indirect-Effect
Error (I-AUC)
0.75
0.75
0.75
0.5
0.5
0.5
0.25
0.25
0.25
0
4. Fair Observable
5. Fair Unobservable
6. Fair Additive Noise
0.75
0.75
0.75
Error (1-AUC)
0.5
0.5
0.25
0.25
0.25
Avg. Rank (1-AUC)
Unfair: 2.17
Cntf. Avg.: 3.62
Random: 6.67
Unaware: 2.62
CFP: 4.28
Constant: 6.75
FairPEN: 3.51
EGR: 5.18
Figure 15. Predictive Error (Synthetie): Predictive error (1-AUC) of FairPFN compared baselines. FairPEN maintains
competitive of predictive error with traditional MI. algorithms, achieving an average rank out
Law School Admissions
Unfair
Unaware
FairPFN
Real
Real
Real
Cntf.
Cntf.
Cntf.
Density
2 3
8.0
0.2
0.4
0.0
0.2
0.4
1.0
0.0
0.2
0.4
0.6
0.8
1.0
FÝA A
FÝA /
FÝ /
10
17.5
15.0
60
Density
12.5
50
10.0
40
4
7.5
30
5.0
2.5
10
0.0
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
|F
FÝAnosal
IF
FÝAg-al
|
FÝA,-sa
Figure 16. Counterfactual Distributions (Law School): Predictive distributions of Unfair, Unaware, and FairPEN on observational and counterfactual versions of the Lawschool Admissions dataset. FairPEN reduces the maximum pairwise difference between these distributions 0.05.
17

---

FairPEN: A Tabular Foundation Model for Causal Fairness
1. Biased
2. Direct- Effecc
3.
Cedirect-Effeed
0.40
0.40
0.40
Error (1-AUC)
0.35
0.35
0.35
0.30
0.30
0.25
0.25
0.20
0.20
0.20
0.15
0.15
0.15
0.0
0.2
0.3
0.0
0.1
0.2
4. Fair Observable
5.
Fair Unobservable
6.
Fair Additive Noise
0.40
0.40
0.40
Error (I-AUC)
0.35
0.35
0.35
0.30
0.30
0.30
0.25
0.25
0.25
0.20
0.20
0.15
0.15
0.15
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Causal Effect (ATE)
Causal Effect (ATE)
Causal Effect (ATE)
TabPFN (vl)
Unfair
Unaware
Fairness Through Unawarcness
Figure 17. Baseline Validation (Synthetic): Fairness- tradc-off achieved by our baselines Unfair and Unaware compared to alternative choices of TabPFN (vl) and "Fairness Through Unawareness." " Un achieves competitive performance with TabPFN (vl), while Unaware outperforms the standard strategy of dropping the protected attribute from the dataset.
Law School Admissions
Adult Census Income
0.20
0.355
Error (I-AUO
0.350
0.19
TabPFN (v1)
0.345
0.18
Unfair
0.340
0.17
Unaware
0.335
Fairness Through
0.16
0.330
Unawareness
0.325
0.15
0.10 0.15 0.20 0.25 0.30
0.04
0.06 0.08 0. 0.12
Causal Effect (ATE)
Causal Effect (ATE)
Figure 18. Baseline Validation (Real-Worldy Fairness- trade-off achieved by baselines Unfai and Unaware compared to alternative choices of TabPEN (v1) and "Fairess Through Unawareness." Our choices of baselines achieve competitive performance on the Law School Admissions problem, while alteraatve baselines perform slightly better the Adult Census Income problem.
18

---

FairPEN: A Tabular Foundation Model for Causal Fairness
Adult Census Income
Unfair
Unaware
FairPEN
Real Catf.
Real
Real
Cntf.
Cotf.
Density
8.0
0.2
0.4
0.8
1.0
8.0
0.2
0.4
0.6
0.8
1.0
8.0
0.2
0.6 INC
0.8
1.0
INC
INC
12
10
Density
0.0 0.1 0.2 0.3 0.4 0.5
0.0 0.1 0.2
0.3 0.4
0.5
0.0
0.1
0.2
0.3
0.4 0.5
IINCa INC
INC a-2d' INC
|ICC INC
Figure Aligning Counterfactual Distributions (Adult): Alignment observational and counterfactual predictive distributions Ý and Yataa the Adult Census Income problem. FairPEN best aligns the predictive distributions (top) and achieves the lowest mean (0.01) and maximum (0.75) absolute error.
1. Biased
2. Direct-Effect
3. Indirect-Effect
Statistical Parity (DSP) Statistical Parity (DSP)
0.75
0.75
0.75
0.5
0.25
0.25
0.25
4. Fair Observablc
5. Fair Unobservable
6. Fair Additive Noisc
0.75
0.75
0.75
0.5
0.5
0.5
0.25
0.25
0.25
Avg. Rank (ATE)
Constant: 1.0
Random: 4.16
EGR (Mitig.): 5.23
CFP (Ground): 2.96
Unaware: 4.52
Unfair: 6.15
FairPEN: 3.97
Figure 20. Statistical Parity (Synthetic): Statistical Parity (DSP) of FairPEN compared to our baselines. FairPEN achieves similar DSP
as Random baseline and outperforms EGR which was optimized specifically for this fairness metrie, achieving an average rank 3.97 out of
19

---

FairPEN: A Tabular Foundation Model for Causal Fairness
Law School Admissions
Adult Census Income
Unfair
Error (I-AUC)
0.50
0.50
0.20
Unaware
0.48
0.45
Constant
0.45
0.375
0.40
0.15
Random
0.43
0.35
1.00
0/2
EGR
0,40
0.30
CFP
0.38
0.25
FairPEN
0.35
0.20
CLAIRE
0.33
0.15
0.00
0.05
0.10
0.00
0.02 0.04 0.06 0.08
Cntf. Avg.
Statistical Parity (DSP)
Statistical Parity (DSP)
Figure 21. ArroupFaarnees--cccrra Trade-off (Real- :orld)) Statistical Parity (DSP), predictive error (1-AUC), and Pareto Front of the performance of FairPEN compared to our baselines on each of 5 validation folds (light) and across all five folds (solid) of our real-world datascts. FairPEN dominates EGR which specifically optimized for this group faimess metric.
1) Level-One
2) Level-Two
3) Level-Three
GP4
GP4
CePA
LS47
X
ESAT
RACE
RACE
FYA
FYA
FYA
Prot. Aitr
Outcome
Unfair Observable
Fair Unobservable
Cause
Additive Noise
Fair Observable
Seen by CFP
Figure 22. Counterfactually Fair Prediction (CFP): Three levels of counterfactually fair prediction (CFP) (Kusner et al., 2017), obtained by fitting a predictor 1) fair observables (if any exist; left), 2) the inferred values of fair exogenous variables (middle) and the inferred values of independent noise terms (right).
FaimessdFaimms Model (SFM)
Faimess Cookbook
X
Spurious Effect (SE)
Y
A Protected Attribuuess
Direct (DE)
Y
Attributes
Culcomes
Outcomes
V Mediators
Indrect Effect (IE)
medialors
Figure 23. Causal Fairness Analysis (CFA) Framework: Components of the CFA framework relevant to FairPEN's prior and evaluation. (Plecko & Bareinboim, 2024) Standard Fairness Model (left: SFM), which provides meta-model for causal fairness and heavily the design of our prior, and the Fairness Cookbook of causal fairness metrics (right).
20

---

FairPEN: A Tabular Foundation Model for Causal Fairness
1) Biased
Direct-Effect
3) Indirect-Effect
Unfair
-0,00 13 (3.05%)
0.0020.1 14 (0.00%)
-0.00=0. 12 (1.65%)
Unaware
-0.01=0.09 (2.60%)
0.00+0.00 (0.12%)
-0.01 (1.81%)
Constant
-0.3640.34 34 (0.00%)
-0.27=0.43 (0.00%)
-0.38 (0.00%)
Random
0.01+0.30 (0.01 %)
0.01=0.31 (0.01%)
0.00=0.30 (0.00%)
BGR
-0.05 46 (0.00%)
-0.07=0.42 (0.00%)
-0.06=0044 45 (0.00%)
CFP
-0.00=0.03 (1.31% %)
-0.01=0.03 (0.56%)
-0.01=0.07 (2.29%)
FairPEN
0.00+0.06 (2.03%)
-0.01 (1.29%)
-0.00=0.05 (2.22%)
4) Fair Observable
5) Fair Unobservable
6) Fair Additive Noise
Average
Unfair
0.00 14 (0.02%)
-0.02a0. 19 (0.00%)
-0.0000. 18 (0.00%)
0.0020.15 (0.79%)
Unaware
-0.00=0.05 (2.63%)
-0.00=0.09 (3.68%)
-0.00=0. (3.07%)
-0.00=0.07 (2.32%)
Constant
-0.49 18 (30.10%)
-0.38 (4.63%)
-0.37=0.33 (0.11%)
-0.38=0.32 (5.81% 1)
Random
0,0140.34 (0.00%)
0.08 (0.00%)
0.0620.37 (0.00%)
0.03+0033 (0.00%)
EGR
-0.09+0.38 (0.00%)
-0.0620.39 (0.00%)
-0.07+0.37 (0.00%)
-0.0720.41 (0.00%)
CFP
-0.02+0.1 14 (1.72%)
0.00 (1.02%)
-0.00=0.05 (1.00%)
-0.01=0.06 (1.32%)
FairPFN
-0.0100.77 (1.01%)
0.01 (2.20%)
0.01 (2.47%)
0.00=0.06 (1.87%)
Table Difference Cntf. Avg. (Synthetic): Mean, standard deviation and percentage outliers the predictions on our causal casestudies of FairPEN and our baseline models compared to the predictions of the Cntf. Avg. baseline, which shows strong performance causal effect removal and predictive error due access to both observational and counterfactual datasets. FairPFN achieves predictions with an average difference Cntf. Avg. of 0.00=0.066, with 1.87% of samples falling outside of three standard deviations.
1. Biased
2. Direct-Effect
3. Indirect-Effed
Causal Effect (ATE)
0.75
0.75
0.75
0.5
0.5
0.5
0.25
0.25
0.25
-0.25
-0.25
-0.25
-0.5
-0.5
-0.5
4. Fair Observable
Fair Unobservable
6.
Fair Additive Noise
Causal Effect (ATE)
0.75
0.75
0.75
0.5
0.5
0.5
0.25
0.25
0.25
0.25
-0.55
-0.5
-0.5
-0.5
-0.5
Avg. Rank (ATE)
Cntf. Avg.: 2.24
Random: 2.53
Unaware: 3.57
Constant: 2.24
FairPFN: 3.0
Unfair: 5.04
CFP: 2.24
EGR: 3.33
Figure 24, Causal Fairness (Synthetic-All Baselines): Average Treatment Effect (ATE) of predictions of FairPEN compared to all baselines. FairPEN consistently removes the causal effect with a margin of error (-0.2, 0.2) and achieves an average rank 3.0 of
21

---

FairPEN: A Tabular Foundation Model for Causal Fairness
Unfair
Unaware
Cntf. Avg.
Constunt
X X EXp
A
X
X, Ex,
Y
A X X
Cxp
Y
A
X
X
Cxp
Y
Fair Observable
X
X A
X
X
Causal effect removed
FairPFN
CFP
EGR
Random
A
X
X,
EXp
Y
A
X X, EX Y
X X, EXp
Y
A
X
X
Exp
Y
Training examples
Inference examples
Predictions
Accesses causal model
Figure 25, Baseline Models: Visualization of FairPEN and our baseline models on our Fair Observable benehmark group, in terms of which variables each model fit and performs inference on.
Law School Admissions
Adult Census Income
Average
Unfair
0.09+0.10 10 (0.00%)
0.05+0.06 (0.60%)
0.07=0.08 (0.30%)
Unaware
0.03+0.03 (0.00%)
0.02+0.04 (1.49%)
0.03=0.04 (0.75%
Constant
-0.40 (97.51%)
-0.1840. (15.69%)
-0.29 009 (56.60%)
Random
0.10+0.30 (0.00%)
0.3240.31 (0.30%)
0,21+0.31 (0.15%)
EGR
0.06+0.45 (0.00%)
0,01 (0.00%)
0,03=0.44 .00 (0.00%)
CFP
0.09=0.03 (49.21%)
0.05#0.066 (2. 13%)
0.07=0.05 (25.67%)
FairPEN
0.01+0003 (0.11%)
0.02+0.04 (0.60%
0.02=0.04 (0.36%)
Table Difference to Cntf. Avg. (Real): Mean, standard deviation and percentage of outliers of the predictions on our real-world
datasets of FairPEN and baseline models compared to the predictions of the Cntf. Avg. baseline, which shows strong performance causal effect removal and predictive error due to access to both observational and counterfactual data. FairPEN achieves predictions with average difference to Crt... Avg. of 0.02+0.04, with 0.36% of samples falling outside of three standard deviations.
22


Reviews:
[{"review_id": "pmSP18Z5Zm", "reviewer": "Reviewer_GboP", "summary": "The paper introduces FairPFN, a tabular foundation model for causal fairness in machine learning. Pre-trained on synthetic causal fairness data, it mitigates the influence of protected attributes without requiring prior causal knowledge. Experiments show FairPFN effectively removes causal bias while maintaining strong predictive accuracy. Key contributions include a novel pre-training strategy, a synthetic causal data prior, and a fairness-aware foundation model.\n\n## Update after rebuttal\n\nThanks for author's rebuttal, it solved some of my problems, I'll maintain my rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your constructive feedback! In response, we have: (1) clarified that FairPFN requires no retraining for new applications (2) added analysis on intersectional fairness through a synthetic case study; (3) clarified the availability of inference and pre-training data generation code; and (4) simplified Figure 1 while adding a supporting pseudocode algorithm detailing the pre-training data generation process. We welcome any additional guidance should you have further suggestions to enhance our work.\n\n> However, the pre-training process requires significant computational resources (3 days on an RTX-2080 GPU), which might limit accessibility for some researchers.\n\nWe apologize for the confusion! In fact, FairPFN is a pre-trained foundation model that requires no retraining for researchers wishing to reproduce our results or apply it to new fairness problems. We propose adding a sentence at the end of Section 4.1 (Real World Inference) stating: \"As a pre-trained foundation model, FairPFN can be directly applied to new fairness problems through a single forward pass of data through the transformer, eliminating any need for retraining.\" Our inference and synthetic data code is available at https://anonymous.4open.science/r/FairPFN\n\n> FairPFN increases the correlation of \"Sex\" with predictions in one analysis, suggesting potential issues with intersectional fairness that need further investigation.\n\nWe appreciate your concern regarding the increased correlation of “Sex” in Figure 8. The reason for this is that FairPFN is currently pre-trained for binary classification tasks with single, binary protected attributes, so it is simply not tasked to mitigate the effects of secondary protected attributes. \n\nIn order to further investigate this result, we have created a synthetic case study \"Multiple Protected Attributes\" demonstrating that while FairPFN removes the effect of the first protected attribute, the causal effect of secondary protected attributes resembles that of a non-fairness aware classifier. A visualization of our new \"Multiple Protected Attributes\" case study and an illustration of this result is linked below: \n\nhttps://anonymous.4open.science/r/FairPFN-supplementary/figures/multiple.pdf \n\nWe propose the following changes:\n1. Highlight this limitation in the results section\n2. Include the causal graph visualization of this new case study in the appendix, as well as the above figure illustrating how FairPFN reverts to a non-fairness aware classifier regarding secondary protected attributes\n3. Detail methodological changes in an appendix section titled “Future Extensions” for addressing the challenge of intersectionality. \n    - This would include sampling multiple protected attributes as exogenous causes in the synthetic pre-training data generation and informing the transformer via an encoding mask which of these multiple variables is protected. Then in order to generate the fair outcomes, dropout would need to be performed on the outgoing edges of all simulated protected attributes. \n\n> The authors might consider releasing the code and pre-trained models to facilitate reproducibility and further research in this area.\n\nWe appreciate your emphasis on reproducibility! We have actually already provided these resources in our initial submission, specifically an inference pipeline to access our pre-trained model and the code to generate our synthetic pre-training data. Our Anonymous GitHub Repository link is included at the end of Section 1: https://anonymous.4open.science/r/FairPFN\n\n> Some figures (e.g., Figure 1) are quite complex and might benefit from additional explanation or simplification for better understanding.\n\nThank you for this feedback regarding the clarity of our figures! We've modified Figure 1 to focus exclusively on visualizing the SCM from which our synthetic pre-training data is sampled. The updated and simplified version of Figure 1 is linked below:\n\nhttps://anonymous.4open.science/r/FairPFN-supplementary/figures/flowchart.pdf \n\nTo compensate for this simplification, we've developed a pseudocode algorithm explaining our synthetic data generation process, which we propose including in the main text:\n\nhttps://anonymous.4open.science/r/FairPFN-supplementary/figures/data_generation.pdf", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence, quantitative results showing improvements in causal fairness metrics (ATE) and predictive accuracy (1-AUC) across synthetic and real-world datasets. Meanwhile visual comparisons and ablation studies demonstrating the effectiveness of FairPFN in removing causal effects.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-suited for causal fairness in tabular data. Synthetic causal case studies enable controlled experiments with known ground truth, ensuring rigorous testing. Standard metrics like ATE and predictive error assess both fairness and accuracy. Comparisons with multiple baselines, from traditional ML models to causal fairness frameworks, further validate FairPFN’s effectiveness.", "theoretical_claims": "The theoretical claims about FairPFN's ability to approximate the Posterior Predictive Distribution (PPD) and integrate over causal explanations are plausible and align with previous work on Prior-data Fitted Networks (PFNs). The connection to Bayesian Inference is well-established, and the modification to focus on causally fair targets is logically consistent with the goals of the paper. No specific proofs are provided, but the conceptual framework is sound and builds on existing theoretical foundations in causal ML and PFNs.", "experimental_designs_or_analyses": "The author use a diverse set of synthetic causal case studies with varying complexity and known ground truth. Also evaluate on real-world datasets with established causal graphs. Additionally, they compare against multiple relevant baselines, including both traditional ML models and causal fairness methods.", "relation_to_broader_scientific_literature": "FairPFN addresses limitations in existing causal fairness frameworks that require prior knowledge of causal models. It builds on concepts from counterfactual fairness and the Causal Fairness Analysis (CFA) framework, while relaxing assumptions about the need for user-specified causal information. Also FairPFN extends the PFN paradigm to causal fairness, leveraging the success of models like TabPFN in small tabular classification tasks. It demonstrates how PFNs can be adapted for complex causal tasks, opening new research avenues in causal ML.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The model is tested on real-world datasets with established causal graphs, demonstrating its practical utility. Meanwhile, The paper is well-structured and clearly explains the methodology, experiments, and results, the figures and tables effectively support the text.\n\nHowever, the pre-training process requires significant computational resources (3 days on an RTX-2080 GPU), which might limit accessibility for some researchers. Meanwhile, the paper notes that FairPFN increases the correlation of \"Sex\" with predictions in one analysis, suggesting potential issues with intersectional fairness that need further investigation.", "other_comments_or_suggestions": "The authors might consider releasing the code and pre-trained models to facilitate reproducibility and further research in this area.\n\nSome figures (e.g., Figure 1) are quite complex and might benefit from additional explanation or simplification for better understanding.", "questions_for_authors": "See \"Other Comments Or Suggestions\" and \"Other Strengths And Weaknesses\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TsQ75MpThl", "reviewer": "Reviewer_1GNH", "summary": "Let $A$ denote (binary) protected attributes, $X$ features, and $Y$ a binary response variable.  A FairPFN is a transformer trained on synthetic data in such a way that when conditioned on $(A, X_{bias}, Y_{bias})$ it is encouraged to complete a query from the same distribution, denoted $X_{bias}^{val}$ with not $Y_{bias}^{val}$ but $Y_{fair}^{val}$. After extensive pretraining, the FairPFN can be used to predict in a casually fair manner. Specifically, given a new dataset $\\mathcal D$, we pass it to the FairPFN as context and then complete new queries in an ICL manner without any updates to the FairPFN.\n\n## update after rebuttal\n\nI maintain my score and positive impression of the paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments! Detailed responses below. Most importantly, we would like to clarify our perspective on the mathematical foundation of our method and how it relates to PPDs.\n\n> In Line 238, “FairPFN thus approximates a modified PPD,...,” pointing to the PPD in Equation 3. This is not mathematically true, or at least I fail to see how this could be true..\n\n> I don’t think this question needs to necessarily be answered in this paper, but the paper certainly should avoid any chance of being misleading on [whether it approximates a modified PPD]\n\nWe believe FairPFN indeed approximates the statement from Eq. (3), that is\n\n$$p(y_f|x_b,D_b) \\propto \\int_{\\Phi} p(y_f | x_b, \\phi)p(D_b | \\phi)p(\\phi)d\\phi.$$\n\nHere, $\\phi$ is the latent, which in our case is the whole SCM, its weights, noise terms, and activations. $\\phi$ is sampled per dataset during training. We would like to clarify that the fair outcomes $y_f$ come from the same data generating process $\\phi$ as $y_b$ with a deterministic mapping applied to the original SCM, namely the removal of outgoing edges from the protected attribute, with everything else remaining constant. In this way we believe that both SCMs, the original and intervened upon one, are encapsulated in $\\phi$ as the distribution shift is deterministic.\n\nWe would also be happy to introduce an additional term $\\phi'$ and a deterministic mapping from $\\phi$ in order to reflect the difference between the original and intervened upon SCM. If your criticism is about calling this a \"modified PPD\", we are open to recommendations on the exact wording. \n\nWe now outline why we believe that FairPFN approximates $p(y_f|x_b,D_b)$ (Eq. 3):\n\nDuring training we sample examples for our training set and our test set using the likelihood $p(x_b,y_b,y_f|\\phi)$, as detailed in the Algorithm at https://anonymous.4open.science/r/FairPFN-supplementary/figures/data_generation.pdf \nWe can use this distribution to sample our biased conditioning set $D_b=\\{(x^i_b,y^i_b) \\sim p(x_b,y_b|\\phi)\\}_{1 \\leq i \\leq n}$, as well as our hold-out examples with fair labels.\n\nOur training loss is defined as:\n\n$$E_{\\phi \\sim p(\\phi); (x_b,y_f) \\sim p(x_b,y_f|\\phi); D_b \\sim p(D_b|\\phi)}[-log(q_\\theta(y_f|x_b,D_b))],$$\n\nwhere we first sample our latent $\\phi$ and then conditioned on it both our test input-output pair and our conditioning set.\n\nWe can reformulate this loss as a KL-divergence with the \"modified PPD\" from above, similar to the derivation by Müller et al. (2022), as follows\n\\begin{align}\n&= E_{x_b,y_f,D_b \\sim p(x_b,y_f,D_b)}[-log(q_\\theta(y_f|x_b,D_b))]\\\\\\\\\n&= - \\int\\int\\int p(x_b,y_f,D_b)log(q_\\theta(y_f|x_b,D_b))dx_b dy_f dD_b\\\\\\\\\n&= - \\int\\int p(x_b,D_b) \\int p(y_f|x_b,D_b) log(q_\\theta(y_f|x_b,D_b))dy_f dx_b dD_b\\\\\\\\\n&= \\int\\int p(x_b,D_b) \\left[ \\text{KL}(p(y_f|x_b,D_b)|| q_\\theta(y_f|x_b,D_b)) + \\int H(p(y_f|x_b,D_b))\\right] dx_b dD_b\\\\\\\\\n&= E_{x_b,D_b \\sim p(x_b,D_b)}[\\text{KL}(p(y_f|x_b,D_b)|| q_\\theta(y_f|x_b,D_b))] + C,\n\\end{align}\n\nwhere $H$ is entropy, $KL$ is KL divergence and $C$ is a constant that does not depend on $\\theta$.\n\nSo minimizing the train loss is the same as minimizing the average KL div. to $p(y_f|x_b,D_b)$, our model $q_\\theta$ thus approximates it. Do you agree with this view? We propose to add this derivation to the paper.\n\n> Figure 1 is visually appealing, but I’d still prefer to understand the data generating mechanism through formal equations which is currently missing in the early parts of Section 4.1\n\nWe propose to provide an Algorithm 2 in Section 4.1, namely https://anonymous.4open.science/r/FairPFN-supplementary/figures/data_generation.pdf, detailing how synthetic datasets are sampled from our MLP implementation of Structural Causal Models (SCMs).\nFurther, our synthetic datasets can be sampled and visualized with https://anonymous.4open.science/r/FairPFN/prior_data_example.ipynb \n\n> Might be useful to define A in Algorithm 1\n\nWe will update line 173 to\n\n*Sample $D_{bias} = (A, X_{bias}, Y_{bias})$ from $\\Phi$*\n\nand define the terms very explicitly in our new Algorithm 2\n\n> Would you consider writing a few comments on extending the current methodology to regression settings, i.e., when Y is real-valued. Should we be considered that PFNs do not handle real-valued responses in a very natural way?\n\nThe recently released version of the TabPFN (Hollmann et al. 2025), whose predecessor we are building upon, integrates regression and one could follow their simple, but, according to their benchmarks, powerful setup of discretizing the continuous space. We will further add an Appendix, where we detail more future work possibilities like this.\n\n**References**\n\n1. Müller, S., et al. F. Transformers Can Do Bayesian Inference. ICLR, 2022. https://openreview.net/forum?id=KSugKcbNf9 \n\n2. Hollmann, N., et al. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319–326, 2025. https://www.nature.com/articles/s41586-024-08328-6", "claims_and_evidence": "I am uncomfortable with presenting the FairPFN as learning some sort of Bayesian posterior predictive distribution. I think this would be true if the pretraining loss employed was $L(Y_{pred}, Y_{bias}^{val})$. However the actual pretraining loss employed is $L(Y_{pred}, Y_{fair}^{val})$. \n\nIn Line 238, “FairPFN thus approximates a modified PPD,...,” pointing to the PPD in Equation 3. This is not mathematically true, or at least I fail to see how this could be true. The pretraining of PFNs no longer targets the underlying Bayes PPD when the query undergoes a distribution shift relative to the context.", "methods_and_evaluation_criteria": "The method here consists of synthetic data generation followed by FairPFN pre-training on the synthetic data. This part is solid. \n\nThe baselines, synthetic and real datasets considered in the Experiments also seem sound to me.", "theoretical_claims": "The only theoretical claim is that the FairPFN is approximating the PPD in (3) which I have doubts about, see comment under Claims and Evidence.", "experimental_designs_or_analyses": "The experimental design and analyses are sound. A wide array of synthetic and real world data sets are considered. The evaluation metrics are sensible and align with standard ones studied in the fairness literature.", "relation_to_broader_scientific_literature": "This work borrows existing notions of causal fairness and really pushes what can be done by leveraging the power of Transformers. Notably, no algorithm for learning the underlying DAG is needed, allowing us to bypass a key challenge in causal fairness.", "essential_references_not_discussed": "I think all relevant references are cited and discussed.", "other_strengths_and_weaknesses": "I really enjoyed reading this paper. I think it’s a big step forward for causal fairness research. I can envision many future papers inspired by this work, for instance where we use other notions of causal fairness to generate synthetic data. \n\nThe main weakness that I see is related, again, to the claims that FairPFN is approximating a “modified” Bayes posterior predictive density (PPD). If it is, what is the likelihood-prior pair that underlies said modified Bayes PPD? I don’t think this question needs to necessarily be answered in this paper, but the paper certainly should avoid any chance of being misleading on this matter.", "other_comments_or_suggestions": "* Figure 1 is visually appealing, but I’d still prefer to understand the data generating mechanism through formal equations which is currently missing in the early parts of Section 4.1 \n* Might be useful to define $A$ in Algorithm 1", "questions_for_authors": "1. Would you consider writing a few comments on extending the current methodology to regression settings, i.e., when $Y$ is real-valued. Should we be considered that PFNs do not handle real-valued responses in a very natural way?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UJyKzpHPX5", "reviewer": "Reviewer_b4g5", "summary": "FairPFN is a tabular foundation model designed to address algorithmic bias in machine learning without requiring prior knowledge of the underlying causal model. Existing causal fairness methods rely on predefined causal structures, limiting their applicability in complex real-world scenarios. FairPFN overcomes this by pre-training a transformer on synthetic causal fairness data, where protected attributes are modeled as binary exogenous causes. This foundation model allows FairPFN to identify and mitigate the causal effects of protected attributes using only observational data. The model demonstrates strong performance in both fairness and predictive accuracy across hand-crafted and real-world fairness scenarios, outperforming robust baseline methods. By removing reliance on manually specified causal models, FairPFN makes causal fairness more accessible and provides a scalable solution for mitigating algorithmic discrimination in critical applications.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your detailed response! We have outlined our clarifications and proposed changes below:\n\n>  When the author tries to extend counterfactual fairness to the dataset level, why is the sensitive attribute changed under X rather than Y^hat, as described in Theorem 3.1?\n\nWe apologize for the inconsistency in notation. We propose to adopt Plecko et al.'s (2023) notation for unit-level/probabilistic and population-level counterfactual fairness, clarifying that AE is a metric form of their population-level version.\n\n* **Unit-Level/Probablistic:** $P(y_{A\\rightarrow a}(u) | X = x, A=a) = P(y_{A\\rightarrow a'}(u) | X = x, A=a)$\n* **Population-level:** $P(Y_{A\\rightarrow a} | X = x, A=a) = P(Y_{A\\rightarrow a'} | X = x, A=a)$\n* **Absolute Error (AE)**  $$AE = |P(Y_{A\\rightarrow a} | X = x, A=a) - P(Y_{A\\rightarrow a'} | X = x, A=a) |$$\n\n> Counterfactuals evaluate the impact of interventions on outcome variables. Counterfactuals are not the same as interventions. \n\nWe appreciate your comment regarding the important distinction between interventions and counterfactuals. We propose to highlight this distinction in the Background section, also noting that counterfactuals crucially hold noise terms constant and thus require full knowledge of the causal model. We will also remove the term \"interventional datasets\" instead correctly referring to them as counterfactual.\n\n> The title claims 'Causal Fairness,' while the paper mainly focuses on counterfactual fairness. \n\nThank you for bringing up this question of framing. We understand counterfactual fairness as a specific causal fairness criteria. We believe that this work falls under the umbrella term of \"causal fairness\" for the following reasons:\n\n1. Our pre-training objective is to make predictions from a modified SCM with outgoing edges from the protected attributes removed. This deviates from the methodology proposed in the original counterfactual fairness paper (Kusner et. al. 2017)\n2. One of our evaluation measures is average treatment effect (ATE), which to your point is not a counterfactual fairness measure\n3. Our key contribution is that causal effect removal can be performed from observational data alone. This contribution is relevant not only to causal fairness, but to the point of other reviewers could be of great impact to causal ML as a whole.\n\n> How do we measure counterfactual fairness? The paper uses ATE, but ATE is not equivalent to counterfactual fairness.\n\nThank you for bringing up this point of confusion! We define the metric Absolute Error (AE) between predictions on observational and counterfactual datasets to measure counterfactual fairness, not ATE. AE is calculated as follows:\n\n1. Predict with natural demographics and features\n2. Predict with counterfactual demographics and counterfactual features\n3. Take the absolute difference\n\nThis process is repeated over the samples in a dataset and is visualized in distribution form as in Figure 7.\n\n> Feature Correlation can only show the association relationship rather than causal relationship.\n\nThis is a very valid point. To the best of our knowledge Figure 8 has not made a causal claim. However, we propose explicitly stating that this result should not be interpreted causally.\n\n> Causal case studies all focus on simple cases, but what about complex causal structures?\n\nThis comment poses an interesting research question! In order to explore FairPFN's performance on more complex causal structures, we have sampled increasingly complex SCMs (up to 200 nodes) from our prior, and evaluated our model's performance. In the figure linked below, we observe that as SCM size increases, accuracy drops while fairness remains relatively constant. This is an interesting insight that we would include in the appendix.\n\nhttps://anonymous.4open.science/r/FairPFN-supplementary/figures/complexity.png \n\n> How does the model handle cases where the real-world bias patterns differ from the assumed causal bias model?\n\nThis is an important question! We have created a new synthetic case study \"Endogenous Protected Attribute\" showing that when protected attributes are confounded by unobserved variables, FairPFN reverts to an unfair classifier.\n\nhttps://anonymous.4open.science/r/FairPFN-supplementary/figures/endogenous.pdf \n\nWe also refer to our response to reviewer GboP regarding an additional case study on intersectionality.\n\n> Can the proposed tabular foundation model work with continuous sensitive attributes?\n\nCurrently FairPFN handles binary protected attributes, but can be extended to continuous attributes by not binarizing values of $A$ during pre-training. We propose to including this in an appendix section \"Future Extensions\"\n\n**References**\n1. Plecko, D et. al. Causal fairness analysis. Foundations and Trends in Machine Learning, 17:304–589, 2024, https://arxiv.org/abs/2207.11385 \n2. Kusner, M., et. al. Counterfactual fairness. NeurIPS’17 pp. 4069–4079, 2017. https://arxiv.org/abs/1703.06856", "claims_and_evidence": "The paper presents FairPFN as a foundation model for causal fairness, claiming it effectively identifies and removes the causal effects of protected attributes without requiring prior causal knowledge. The empirical results likely support its strong performance in fairness and predictive accuracy across diverse scenarios.", "methods_and_evaluation_criteria": "Somehow, it makes sense.", "theoretical_claims": "1. Theorem 3.1 can be relaxed to form dataset-level counterfactual fairness metrics, such as the Absolute Error (AE) between predictive distributions on real and counterfactual datasets, Equation (2). When the author tries to extend counterfactual fairness to the dataset level, why is the sensitive attribute changed under X rather than Y^hat, as described in Theorem 3.1?\n\n2. [089-091] Counterfactuals evaluate the impact of interventions on outcome variables. Counterfactuals are not the same as interventions. Please specify these two definitions more carefully.", "experimental_designs_or_analyses": "1. How do we measure counterfactual fairness? The paper uses ATE, but ATE is not equivalent to counterfactual fairness. Feature Correlation can only show the assoication relationship rather than calusa relationship.\n\n2. Causal case studies all focus on sample cases, but what about complex causal structures?", "relation_to_broader_scientific_literature": "This work addresses a key limitation in existing causal fairness frameworks, namely, the requirement for prior knowledge of the causal graph.", "essential_references_not_discussed": "Probably not.", "other_strengths_and_weaknesses": "1. This paper is likely an incremental work that applies PFN-related ideas to causal fairness. However, it is well-presented and open-source.\n\n2. The title claims 'Causal Fairness,' while the paper mainly focuses on counterfactual fairness. Please be aware of the distinction between these two definitions.\n\n3. Intervention and counterfactual are two different levels; please be aware and do not mix them up.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. How does the model handle cases where the real-world bias patterns differ from the assumed causal bias model?\n\n2. Can the proposed tabular foundation model work with continuous sensitive attributes? (since it is a tabular foundation model)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper studies causal fairness in tabular data and proposes to pre-train a foundation model  (transformer) on synthetic causal fairness data, where protected attributes are modelled as binary exogenous causes. This allows mitigation of the causal effects of protected attributes using observational data without requiring prior knowledge of the underlying causal model.

The reviewers and AC find the task appealing and important to address, and the approach interesting and sound. In the initial reviews, there were some technical questions that the rebuttal was able to successfully address, as well as some minor issues (clarifications, need to discuss limitations, analysis as the complexity of the data-generating process increases etc). These should be incorporated in the final version.

After careful deliberation the decision was made to accept the paper. Congratulations to the authors! It is crucial, however, that all the clarifications and improvements promised in the rebuttal are implemented in the final version of the paper.
Decision: Accept (poster)