Towards Fair In-Context Learning with Tabular Foundation Models
Authors: Patrik Kenfack
Samira Ebrahimi Kahou
Ulrich Aïvodji
Venue: Accepted by TMLR ()

Abstract: Transformer-based tabular foundation models have recently demonstrated promising in-context learning (ICL) performance on structured data, emerging as competitive alternatives to gradient-boosted trees. However, the fairness implications of this new paradigm remain largely unexplored. We present the first investigation of fairness in tabular ICL, evaluating three recently proposed foundation models—TabPFNv2, TabICL, and TabDPT—on multiple benchmark datasets. To mitigate biases, we explore three pre-processing fairness-enhancing methods: correlation removal (decorrelating input features from the sensitive attribute), group-balanced sample selection (ensuring equal representation of protected groups in context examples), and uncertainty-based sample selection (prioritizing context examples with high sensitive-attribute prediction uncertainty). Our experiments show that the uncertainty-based strategy consistently improves group fairness metrics (e.g., demographic parity, equalized odds, and equal opportunity) with minimal impact on predictive accuracy. We release our code to facilitate reproducibility https://github.com/patrikken/Fair-TabICL.

Published in Transactions on Machine Learning Research (01/2026)
Towards Fair In-Context Learning with Tabular Foundation Models
Patrik Kenfack
ÉTS Montréal
Mila Quebec AI Institute
patrik-joslin. Mallin.kenface kenfack. 10ens etsmtl.ca
Samira Ebrahimi Kahou
University Calgary
Mila Quebec AI Institute, CIFAR
samira. aumiraaebrraiimlllabuudaggggggg
Ulrich Aïvodji
ÉTS Montréal
Mila Quebec AI Institute
ulrich. uLrich.aivodji@etee. www.dviieettttl.c....
Reviewed OpenReview: httpp:///ppaaaeeie...erer/rrootp
Abstract
consfforerrbbseed tabular foundation models have recently demonstrated promising in- context learning (TCL.) performance on structured data, emerging as competitive alternatives to gradient- gradient-boosted boosted trees. However, the fairness implications of this new paradigm remain largely unexplored. We present the first investigation of fairness in tabular ICL, evaluating three recently proposed foundation benchmark datasets. To mitigate biases, we explore three pre-processing fairness-enhancing methods: correlation removal (decorrelating input features from the sensitive attribute), group-balanced sample selection (ensuring equal representation of protected groups in context examples), and uncertainty- baaseaiitt--eeed sample selection (prioritizing context examples with high sensitive-attribute prediction uncertainty). Our experiments show that the uncertainty- stralegy consistently improves group fairness metrics (e.g., demographic parity, equalized odds, and equal opportunity) with minimal impact on predictive accuracy. We release our code to facilitate reproducibility (https:/ gtthhub.com/ppttiikken///ee
FaaellssTTaPN TabICI, and TabDPT on multiple
1 Introduction
Tabular data, represented in rows and columns, is data modality widely used for prediction tasks in domains such finance and healthcare (Asuncion al., 2007). bree-based models such XGboost (Chen et al., 2015) and Grostent-Boosted Trees (Ke et al., 2017) have shown the strongest generalization performance tabular data. Recently, with the emergence of foundation models, Deep Learning (DL) based models have challenged the dominance of tree-based models (Hollmann el al., 2025). Foundation models are models pretrained on vast datasets, without a specific task in mind, and they can be adapted across a wide range of downstream tasks. Large language models (LLMs) such as GPT-3 are common examples of foundation models, and they have demonstrated emerging capabilities such as in-context learning (ICL) with few labelled data (Brown et al., 2020). In-context learning (ICT.) has primarily been applied to natural language tasks using large

---

Published in Transactions on Machine Learning Research (01/2026)
language models (LLMs). For example, in text classification, labeled examples are formatted as textual demonstrations and provided as context to a language model, enabling it to predict the label of a new instance without any parameter updates or fine-tuning (Radford et al., 2019; Brown et al., 2020). More recently, efforts have extended ICL to tabular data by scrializing table rows into text or sentences (Hegsclmann et al., 2023). However, since LLMs are not pretrained to model the complex structural relationships inherent. in tabular data-such as interactions between rows and columns-thein performance on large-scale tabular tasks still lags behind base- methods (Hegselmann et al., 2023).
Alternatively, recent work has proposed foundation models explicitly tailored for tabular data, achieving competitive performance with tree-based models while reducing the need for extensive model selection and hyperparameter tuning. For instance, TabPEN (Hollmann et al., 2022) is a transformer-based model pretrained on synthetic datasets, and its successor, TabPFNv2 (Hollmann et al., 2025), extends support to larger datasets with up to 10k samples. Similarly, TabICL (Qu et al., 2025), also pretrained on synthetic data, scales to datasets with up to 500k samples. These models leverage synthctic pretraining to encode a wide range of statistical priors, enabling effective target. inference from in-context examples.
To better reflect the priors found in real-world datasets, other models incorporate real data during pretraining Tabular Discriminative Pre-trained Transformer (TabDPT) (Ma et al., 2024) is pretrained directly on real- world datasets, while Real-TabPFN (Garg et al., 2025) builds on synthetic pretraining with additional fine-tuning on real data. These approaches generally yield improved performance by aligning the learned representations more closely with real-world data distributions.
Given the strong performance and in-context learning capabilities of tabular foundation models, we will likely see widospread adoption in real-world docision-making tasks. This shift could mark a turning point in how tabular data problems are approached. However, the use of ICI-based models in high-stakes domains such as healthcare, finance, or criminal justice-raise important ethical concerns. In particular, it is critical to assess their potential to perpetuate or even amplify existing social biases. Traditional machine learning models have already been shown to replicate biases present in the data (Mehrabi et al., 2022), and recent studies indicate that LLM-based TCL, can also produce biased predictions (Hu et al., 2024; Bhaila et al., 2024). However, these studies rely on serialized representations of tabular data and therefore inherit the limitations of LLMs in handling tabular structures (Ma et al., 2024).
This paper investigates the fairness of TCL. prediction using transformer-based tabular foundation models. First, our study reveals, perhaps unsurprisingly, that while these models focus on improving prediction accuracy, they can also amplify bias. Motivated by recent studics on the scnsitivity of ICL performance- in terms of fairness and accuracy to demonst ration selection, we aim to address the following research question: What in-contert selection/trann /tranfformmtiion method can improve the fairness of ICL predictions?
In the fairness literature, several metrics have been proposed to mcasure fairness at the group individual levels (Dwork et al., 2012). In this work, we focus widely used group fairness notions, including demographic parity, equalized odds, and equal opportunity (Hardt et al., 2016). This group metrics mcasure the performance disparity across different demographic groups while we acknowledge that other fairness metries, beyond group fairness, such as individual or counterfactual fairness, could be used depending on the use case. To achieve these fairness notions, several fairness- eehaneeissenanninngggg methods have been proposed. They are generally grouped into three categorics: pro-processing, in in-processing, and post-proccssing (Mchrabi et al., 2022) methods. Projecting these categories into the ICL paradigm, pre-processing methods perform ratoont transformation or selection before predicting in context (Hu et al., 2024). In-processing methods would fine-tune or retrain the foundation model with fairness constraints (Robertson et al., 2024). Post-proccssing methods would alter the ICL predictions to improve a given fairness metric (Hardt et al., 2016). Pre- and Post-processing methods are more computationally friendly since they do not require model updates. This motivates our choice to focus on the pre- pre-processing techniques and leave post-processing interventions for future exploration. More specifically, we propose and investigate three pre-processing fairness interventions: (i) Correlation Remover (Feldman et al., 2015), method that alters each input feature to reduce their correlation with the sensitive attribute; (ii) group-balanee in-context selection, cnsures that the in-context set is groupebalanced; (iii) Uncertainly-base in-context selection, estimates the uncertainty of predicting the
Underlined represents the method's name throughout, the paper and the results.
2

---

Published in Transactions on Machine Learning Research (01/2026)
sensitive attribute of in-context samples and only selects samples with uncertain predictions. We performed intensive experiments on eight fairness benchmark datasets to investigate the effectiveness of each method in terms of fairness and accuracy. Our results reveal that the umcertainty-based method can provide better fairness performance across datascts, fairness metrics, and foundational models, with marginal impact on accuracy. Our contribution can be summarized as follows:
While most existing studies focus on fair ICL with serialized tabular data, we provide, to our knowledge, the first investigation into preprocessing methods for fair prediction ICL using transformer-basee tabular foundation models.
We propose and investigate three pre pre-psscs intervention methods to enforce fair ICL predictions. These methods aim to reduce the information about the sensitive attributes of in-context samples. We remenstrate that uncertainly-based in-context sample selection can significantly improve the fairness of ICL predictions with slight drop in utility, e.g., accuracy.
We perform extensive experiments on broad range start-of-the-arl fairness benchmarks and provide insights into contexts where a given fairness intervention performs best in terms of fairness-utillity tradcoff.
We release the code to ease reproduction of the results and help researchers and practitioners integrate the proposed methods.
2 Related works
Fairness. Numerous methods have been developed to enforce group fairness in classical machine learning models (Mehrabi et al., 2022; Mbiazi et al., 2023; Kenfack et al., 2024a). These methods are often categorized as pre-processing, in-processing, or post-processing approaches. Model-agnostic methods in the pre-processing category typically modify or reweight the input data to reduce information correlated with sensitive at- tributes (Madras et al., 2018; Creager et al., 2019; Kamiran & Calders, 2012; Celis el al., 2020; Balunović el al., 2021; Feldman et al., 2015). In contrast, post- post-processing techniques adjust the model's prediction outcomes after training to satisfy fairness constraints (Hardt et al., 2016; Petersen et al., 2021). Finally, in in-processing approaches embed fairness constraints directly into the training objective (Agarwal el. al., 2018; Zhang el al., 2018; Roh et al., 2020). Unlike prior work, our approach focuses on pre-processing interventions applied in in-context learning (ICL) settings, where downstream predictions are made by foundation models without any model updates. We emphasize that model-agnostic methods-partionn
pre- and post-processing are especially suitable in ICI. because they do not rely on access to or retraining of the model. However, their effectiveness in this setting, especially for tabular foundation models, remains largely unexplored and is the locus of our evaluation.
Tabular Foundation Models. In-context learning with tabular foundation models presents notable advantage over traditional machine learning approaches by enabling models to adapt dynamically to new data without the need for retraining (Hollmann et al., 2022; Qu et al., 2025; Hollmann et al., 2025). Conventional ML methods typically depend on predefined training datasets, meaning that any alteration in the data or task necessitates line-consuming and resource-intensive retraining process. In contrast, tabular foundation models utilize in-context learning to execute tasks based on the specific context of the data provided at inference time. This allows these models to interpret and process new tabular data with minimal prior preparation, facilitating more flexible and efficient pocision-maaing (Hollmann et al., 2022). The advantages of this approach are particularly apparent in scenarios where data distributions change over time or when models must quickly adjust to various data tasks without undergoing retraining. Thus, as in-context learning emerges as powerful tool for real-time, adaptive predictions in complex and dynamic environments, assessing and mitigating biases in the prediction can make its use more socially acceptable. Existing models are pre-trained using synthetic (Qu et al., 2025; Hollmann et al., 2022) or real-world data (Ma et al., 2024). Pre-training on real-world data often provides competitive or better performance, and the performance of synthetic pre-trained tabular foundation models can be boasted with continued pre-training on real-world data (Garg et al., 2025). In this work, we consider tabular foundation models pretrained on synthetic (Hollmann et. al.,
3

---

Published in Transactions on Machine Learning Research (01/2026)
2025; Qu et al., 2025) and real-world data (Ma et al., 2024) and assess the fairness implications of each pretraining strategy.
Fairness in ICL. Fairness in in-context learning has primarily been studied in the context of large language models (LLMs) applied to labular data serialized as texi(Bhaila et al., 2024; Hu et al., 2024; Ma et al., 2023). For instance, Hu et al. (2024) explore group-aware sampling strategies, finding that prioritizing minority group demonstrations improves fairness outcomes. Similarly, Bhaila et al. (2024) propose a data augmentation technique that guides demonstration selection reduce bias during inference. While related in spirit, our approach differs in two key ways: (1) we focus on numerical tabular foundation models (rather than LLMs), and (2) our ancertainty-based selection method aims to reduce model reliance on sensitive attributes rather than optimize informativenoss per sc. Prior uncertainty-driven methods (Mavromatis et al., 2023; Kung et al., 2023) focus on selecting informative examples under a labeling budget, whereas we use uncertainty to guide fair demonstration selection. It is also important to highlight that LLMs, though flexible, are not optimized for tabular data and often perform worse than specialized numeric models, such as wradient-boosted trees (Hegselmann al., 2023). As such, our work fills key gap by investigating fairness interventions tailored to numeric tabular foundation models. To our knowledge, this is the first study evaluate pre-processing fairness methods in this emerging model class.
Pairness-Award Tabular Foundation Models. Recent work by Robertson et al. (2024) introduced FairPEN, TabPFN-like model trained to suppress the causal effect of sensitive attributes during pretraining. Their approach seeks counterfactual fairness (Kusner et al., 2017), ensuring that model predictions remain invariant when sensitive altributes are counterfactually changed. Our work is conceptually distinct in two respects. First, we do not require model retraining and rely on model-agnostic pre-processing methods, making our approach broadly applicable to any pretrained tabular foundation model. Second, we aim for group fairness, focusing on improving performance disparities between subgroups, rather than enforcing counterfactual invariance the individual level.
3 Methodology
Problem Setup We consider a classification task with the given training data D {(i,, J where 2; is an imput feature vector, yi is the corresponding class label, and the corresponding demographic group. The goal is to obtain a classifier f, via ICL, to accurately predict the larget given sample æ while being fair w.r.t, demographic information s. Several metrics have been proposed to measure fairness at the group or individual levels (Dwork et al., 2012). In this work, we focus on group fairness notions, mcasuring the performance disparity across different demographic groups, i.., demographic parity, equalized odds, and equal opportunity (Hardt al., 2016). A detailed description of these fairness metrics be found in Appendix B.1.
This section presents three processig techniques proposed in this work to ensure fairer ICL inference on tabular data. In particular, we consider correlation remover (Feldman et al., 2015), group-balanced demonstration selection, and uncertainty-based demonstration selection.
3.1 In-context Samples Transformation
Correlation remover (Feldman et al., 2015; Bird al., 2020) is a preprocessing method that reduces the correlation between the sensitive and non-sensitive attributes before fitting the model. More specifically, linear transformation is applied to each non-sensitive feature to reduce its correlation with the sensitive feature. We use the correlation remover as a preprocessing step over the training (in-context example) and testing sets before performing in-context prediction. Ultimately, transforming input features to reduce their linear dependency on the sensitive feature can reduce the reliance on sensitive features in the downstream models. However, as we will see Section 4.5, nonlincar and complex downstrcam models can still infer the nonlinear dependencies over the sensitive feature and provide unfair results. more detailed description of correlation remover be found in Appendix B.2.
4

---

Published in Transactions on Machine Learning Research (01/2026)
3.2 In-context Samples Selection
In this work, we posit that in-context sample selection can have significant impact on the fairness of ICL prediction. We analyze two demonstration selection methods that can improve the fairness of ICL predictions without model update.
3.2.1 Group-balanced demonstration set selection.
Representation bias is common source of bias in machine learning models (Mchrabi et al., 2022). It occurs when the collected training data does not reflect the demographic diversity of the population. As a result, some demographic subgroups are under-represented, if not represented at all. Recent studies have demonstrated the benefits of group-balanced training data on the fairness properties of the downstream model. Several methods have been proposed to mitigate representation bias in the data, including subsampling the majority group or reweighting the training data based on group proportions (Kamiran & Calders, 2012; Celis al., 2020). In this paper, we focus on subsampling since current tabular foundation models do not handle sample weights (Hollmann et al., 2025; Qu et al., 2025). We perform ICL with group-balanced demonstration set sampling from each group uniformly at random. When the demonstration set size does allow equal group reprosentation, we subsample the majority group at random. A similar strategy is employed by (Hu et al., 2024) to select demonstrations for few-shot ICL prediction with LLMs. In this paper, we evaluate the effectiveness of this fairness intervention with tabular foundation models instead of using LLMs on serialized tabular data.
3.2.2 Uncertainty-baser demonstration set selection.
Kenfack et al. (2024b) demonstrated that models trained without fairness constraints can have better fairness properties when the training data consists of samples with uncertain sensitive altributes. We hypothesize that the uncertainty of the sensitive affribute prediction can be good mensure select demonstrations that improve the fairness of in-context predictions. To validate this, we measure the uncertainty of predicting the sensitive attribute in the demonstration set and use samples with high uncertainty for in-context learning. We focus conformal prediction (Shafer & Vovk, 2008; Vovk et al., 2005) as uncertainty measure since it provides strong theoretical guarantecs for the coverage. Instead of returning single label, a conformal predictor returns a prediction set containing the true label with a probability of at least 1 -,,, e, with e being a user-defined coverage parameter the conformal prediction (Angelopoulos et al., 2023). For example, setting = 0.1 ensures the prediction set contains the true sensitive attribute value with at least 90% probability. Specifically, samples with prediction sets containing more than one value are uncertain. Intuitively, the coverage parameter controls the Mairness-utiliy ility tradeoff, with s 1 meaning no fairness intervention where all the datapoints are used and 0 meaning maximal fairness intervention where only uncertain samples are included in the in-context examples. We show in Section 4.2.1 how the coverage parameter c of the conformal predictor consistently controls the tradeoff between accuracy and fairness across fairness metrics and downstream foundational models.
Since conformal prediction is model agnostic, we considered both classical methods, c.g., Logistic Regression (LR), and foundation models, e.g., TabPFN for training the sensitive attribute classifier to measure the prediction uncertainties. Note that the method could be applied using other uncertainty measures, such as Monte Carlo dropout and confidence interval (Kenfack et al., 2024b). We focus on conformal prediction due to its rigorous theoretical guarantees, and it does not require a hyperparameter to threshold the level from which a prediction considered uncertain (Angelopoulos et al., 2023). More details about uncertainty measurement with conformal prediction can be found Appendix B.3.
3.3 In-Context Prediction
After performing demonstration selection or transformation using the fairness intervention methods presented previously, we pass them through the tabular foundation model as in-context examples for predicting class labels on the test set. In this paper, we consider three tabular foundation models: TabDPT (Ma al., 2024), TabICI. (Qu et al., 2025), and TabPFNv2 (Hollmann et al., 2025). While all these models Transformer
5

---

Published in Transactions on Machine Learning Research (01/2026)
architectures their backbones and are pretrained large amount of datasets, TabICL and TabPFN use only synthetic data generated from structured causal networks, and TabDPT uses real-world data. The current version of TabPFN can handle a maximum of 10k samples with 500 features, and TabICL can handle up to 500K samples. We randomly subsample in-context examples when the context set size exceeds the maximum number of samples allowed by the foundation model being evaluated.
4 Experiments
We describe the experimental setup and perform intensive experiments to answer the following research questions:
(RI) Does group-balanced demonstration selection and correlation remover effectively reduce infor- mation about the sensitive attribute and improve the fairness of ICL?
(R2) What fairness intervention provides a better fairness-utility tradeoff?
(R3) What foundation tabular model can provide a better fairness-utility tradcoff?
4.1 Experimental Setup
Datasets We experiment on tasks from folktables (Ding et al., 2022), which contains data extracted from the American Community Survey (ACS) Public Use Microdata Sample (PUMS). More specifically, we experi- ment with the following ACS PUMS tasks: ACSIncome, ACSMobility, ACSTravelTime, ACSEmploy ACSPublicCoverage. These tasks reflect a range of real-world predictive challenges with fairness concerns. We use the data of the year 2018 from the state of Alabama (AL), which is one of the states with the largest fairness violation (Ding al., 2022). A limitation of the ACS PUMS datasets is that they US-centric; we diversify the experimental setup by including he tasks and datasets. Specifically, we also experiment on the other tabular datasets and tasks including: Diabetes (Gardner et al., 2023), German Credit (Frank, 2010), and CelebA (Liu et al., 2018). More details about each dataset, including the sensitive attributes used, the number of samples, and the number of features, can be found the Appendix A.
We measure the utility of the model using accuracy, and measure fairness using three group fairness metrics, i.e,, Demographic Parity (ADP), Equal Opportunity (AEOP), and Equalized Odds (AEOD). More details about group fairness metrics can be found in B.1.
Metrics
Baselines
We evaluate ICL under four in-context selection strategies, comparing both fairness and accuracy:
Vanilla: randomly selects in-context examples from the training set without fairness considcrations.
Group Balanced: samples in-context examples to maintain equal group ratios, uniformly downsam- pling the majority group across runs.
Correlation Remover (CR): applies correlation-removall (Feldman et al., 2015) to reduce dependence between sensitive and non-sensitive features in both in-context and test examples. We use the fairlearn implementation (Bird et. al., 2020) with varying values of a, where = enforces maximal fairness.
Uncertain: selects in-contexi examples with high uncertainty in sensitive attribute prediction. We estimate uncertainty using conformal prediction (Cordier et al., 2023) as implemented in Mapie (Taquet al., 2022), varying the coverage paramoter to control the hhircess-utility tradcoff. We study two variants of sensitive attribute classifier: one with a Logistic Regression classifier (Uncertain+LR) and with foundation model (Incertain+TabpFX),

---

Published in Transactions on Machine Learning Research (01/2026)
Evaluation For evaluation, held out 20% of the data to train the sensitive attribute classifier used in the uncertainty-based baseline. The remaining data were split into 80% for training and 20% for testing. We applied the fairness interventions to the training set to derive in-context samples, and reported fairness and accuracy on the test set, averaged over three random seeds. This setup provides robustness, as each data point can appear as a lest sample or an in-context sample across seeds. As noted above, we use TabICL, TabDPT, and TabPFN with their default parameters.
4.2 Results and Discussion
We evaluate fairness in ICL predictions with tabular foundation models along several dimensions. First, we compare the baseline methods in terms of fairness and accuracy. For interventions with a tunable fairness-ute utility tradeoff, we vary the corresponding hyperparameter and compare their Pareto fronts. While accuracy the main utility metric, we verify in the Appendix that results consistent when using ROC AUC. Next, we compare foundation models under the best-performing intervention. Finally, conduct an ablation on the effect of in-context set size and analyze the failure case of the correlation-removal method.
4.2.1 Fairness-Utilit Tradeoff
Figure/ shows the fairness utility Pareto fronts on ACSIncome, ACSMobility, and ACSTravelTime datasets using TabPFN the foundation model. We compare the Vanilla baseline, Group Balanced, Correlation Remover, and the two variants of our uncertainty-based method (Uncertain+LR and Uncertaiin+TaPFF) For Correlation Remover and Uncertain, we vary the tradeoff parameters a and over [0, IJ, with each shaded point corresponding to a parameter setting. The main findings, consistent across datasets and models, are as follows:
(R1) The Group Balanced method yields only marginal fairness gains over Vanilla ICL, with similar utility. This suggests that representation bias alone does not explain observed disparities; even with group-balanced contexts, subgroup performance gaps remain, likely due to other sources of bias such as historical, algorithmic, or measurement bias (Mehrabi et al., 2022).
(R1, CR failure) Surprisingly; Correlation Remover often exacerbates unfairness. For example, on ACSIncome, demographic parity worsens from 0.14% to 0.26% at = 1 (see Figure 1). We attribute this to bias amplification due to sensitive attribute leakage, as discussed in Section 4.5.
(R2) In contrast, both variants of Uncertain consistently achieve Parcto-dominant points. They offer stable control over the fairness accuracy tradeoff across values, enabling significant. fairness improvements with modest accuracy loss. The reduced accuracy stems mainly from smaller in-context. sets, since excluding low-uncertainty samples decreases the size of the in-context set, which can reduce accuracy as we discuss in Section 4.3).
(R2, Uncertainty variants) Among the two uncertainty-based methods with different model classes, Uncertain+TabPFN typically yields stronger Pareto fronts, indicating that TabPFN provides more reliable conformal predictions than logistic regression, leading to better fairness-are aware sample selection.
We observe similar patterns on other datasets. Additional results are reported in the Appendix: Figure for other datasets with TabPFN; Figures and for TabICL; and Figures and 10 for TabDPT.
4.2.2 Accuracy of sensitive attribute reconstruction
We assess whether foundation models can reconstruct the sensitive attribute after different fairness interven- tions. Specifically, we perform ICL to predict the sensitive attribute and measure test accuracy. Effective debiasing should lower this accuracy, ideally approaching random guessing (50%).
Table shows that the Group Balanced intervention only marginally reduces reconstruction accuracy for TabPFN and TabICL on ACS datasets. This limited effect is expected, since ACS nearly group-balancee for gender (Ding al., 2022), e.g,, for the state of Alabama, the gender distribution is 51.96% females,
7

---

Published in Transactions on Machine Learning Research (01/2026)
Vanilla
Group Balanced
Correlation R.
Uacertain+111
U
Demographic Parity
Equal Opportunity
Equalized Odds
82
82
82
80
Accuracy (%)
70
70
70
15
20
10
15
Fairness Violation (%)
Fairness Violation (%6
Fairness Violation (%)
(a) ACSIncome
Demographic Parity
Equal Opportunity
Equalized Odds
78
78
78
Accuracy (%6
77
77
77
76
76
76
75
75
73
73
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(b) ACSMobility
Demographic Parity
Equal Opportunity
Equalized Odds
70.0
70.0
70.0
67.5
67.5
67.5
Accuracy (96)
65.0
60.0
60.0
60.0
57.5
57.5
57.5
0.0
2.5
7.5
10.0
0.0
2.5
7.5
10.0
0.0
2.5
5.0
10.0
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(c) ACSTrave/Time
Figure 1: Comparing the fairness-utility Pareto-front of different fairness interventions using TabPFN on the ACSIncome, ACSMobility, and ACSTravelTime datasets.
48.04% males. In contrast, the Diabetes dataset is highly imbalanced by race, and group balancing reduces reconstruction accuracy from 80.2% for the vanilla baseline to 66.2%.
For Correlation Remover, we observe the opposite effect: reconstruction accuracy rises to nearly 100%. We identify this as the source of bias amplification. Transformer- based foundation models, pretrained on synthetic data with diverse structural priors, appear detect and exploit the transformations applied by Correlation Remover. Because each non-sensitive feature is modified using the sensitive attribute, sensitive information leaks into the transformed features. The models leverage this leakage, relying hidden sensitive

---

Published in Transactions on Machine Learning Research (01/2026)
Dataset
ICL Method
TabPFN
TabICL
Accuracy
Fl Score
Accuracy
F1 Score
Vanilla
77.2=0.5
78. 78.4
75.040.5
76.240.3
Group Balanced
27.1
77.
75.0
75.510.1
ACSIncome
Correlation R.
100.040.0
100.04u.0
99.940.0
99.940.0
Uncertain+LR
74.7=1.3
76.7
74.940.5
76.040.4
Uncertain+TTbpFN
51.3312.3
66.1 14.4
71.7
73.610.6
Vanilla
75.9
77.5
72.8+0.4
73.840.5
Group Balanced
75.9
77.0 77.0=0.5
72.5+u.6
72.64v.6
ACSTravelTime
Correlation R.
100.0+0.0
100.0+0.0
100.0
100.0+0.0
Uncertain+LR
75.9
77.8
72.8+0.5
74.3+0.5
Uacbrrain~TabPFN
74.6#111
75.7
67.4+1.6
66.2
Vanilla
91.4
88.9
91.5.0.1
89.210.2
Group Balanced
91.1-0.4
89.0
90.940.3
88.9
ACSPublicCoverage
Correlation R.
100.040.0
100.040.0
100.0.0.0
100.040.0
Uncertain+LR
16.4_10.0
55.3 55.314.0 14.0
57.7111.9
60.0
Uncertain+TTbPFN
42.540.9
58.7
42.7
58.640.6
Vanilla
64.0=o.4
62.0+1.8
65.040.3
62.2+1.4
Group Balanced
64.0=0.5
65.0+1.0
64.8+0.4
65.3+1.1
ACSEmployment
Correlation R.
100.040.0
100.04u.0
100.0
100.0.0.0
Uncertain+LR
64.3=0.4
62.4+2.0
64.940.3
62.8
Uncertain+TabPFF
57.543.3
47.048.6
61.1
53.946.1
Vanilla
68.320.8
67.8+1.0
67.6+1.0
67.4+1.2
Group Balanced
68.1
67.9
67.6+1.1
67.641.5
ACSMobility
Correlation R.
100.010.0
100.0
100.0
100.040.0
Uncertain+LR
68.1-0.9
67.8-0.9
67.440.8
66.940.8
Uncertain+TabPFN
68.1
67.7
67.2
66.8
Vanilla
72.5
70.3
71.843.1
71.043.0
Group Balanced
72.7
70.7
71.943.0
71.242.6
German Credit
Correlation R.
100.040.0
100.040.0
100.0+0.0
100.0+0.0
Uncertain4LR
64.6
62.3
68.4
67.8
Uncertain+TabPFF
60.445..5
51.34426.5
63.843.99
60.8
Vanilla
80.2
89.0+o.1
80.44o.1
89.040.1
Group Balanced
66.2
75.9+0.8
65.1+0.4
74.640.4
Diabetes
Correlation R.
100.0+0.0
100.0+0.0
100.0+0.0
100.0+0.0
Uncertain+LR
70.4
77.1
80.3
89.0
Uncertain-TabPPM
74.9=10.0
84.8+8.0
80.2+0.1
89.0+0.1
Vanilla
84.7-0.2
83.2
85.040.
83.24u.3
Group Balanced
84.6
83.2 83.2+0.3
84.940.
83.340.3
CelebA
Correlation R. 100.010.0
100.010.0
100.0
100.0
Uncertain+LR 72.4
61.242117
84.94u02
83.140.3
Uncertain+TTabPNN
74.5=a.4
70.
81.247.2
77.2
Table 1: ICL prediction performance of sensitive attributes after applying different fairness interventions. Smaller accuracy is better since it indicates how well the foundation model can reconstruct the sensitive attribute after the pre-processing fairness interventions. Uncertain methods yield the smallest accuracy, which justifics the improved fairness performance.

---

Published in Transactions on Machine Learning Research (01/2026)
signals to predict the target variable and thereby amplifying unfairness. We provide further analysis in Section 4.5, showing that applying correlation removal only to the training set while keeping the test set. unchanged improves fairness compared to the Vanilla baseline.
Finally, the Uncertain methods yield the lowest reconstruction accuracy across datasets. This suggests that the selected in-context samples carry little information about the sensitive attribute, limiting the model's ability to exploit and thereby improving fairness.
4.2.3
Comparison of Foundation Models: TabICL vs. TabDPT vs. TabPFN
Demographic Parity
Equal Opportunity
Equalized Odds
80
80
Acouracy
78
78
76
74
74
72
72
72
Vantila TabDPT
TabDPT
70
70
70
0.0
2.5
5.0
7.5
10.0
12.5
10
15
5
10
15
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(a) ACSIncome
Demographic Parity
Equal Opportunity
Equalized Odds
77.5
Accuracy
75.0
75.0
75.0
72.5
72.5
72.5
70.0
70.0
70.0
67.5
67.5
67.5
65.0
Vanilis DID
Patr TI
2
2
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(b) ACSMobility
Demographic Parity
Equal Opportunity
Equalized Odds
72.5
72.5
70.0
70.0
70.0
Accuracy
67.5
67.5
65.0
63.0
65.0
62.5
62.5
Vanille TabiCI
Fair TabiCt
60.0
TabeFN
Par TuaPEN
60.0
60.0
Vanilla TREOPT
Tabner
57.5
57.5
10
2.5 Fairnsss Violation (%)
7.5
10.0
0.0
5.0
10.0
12.5
Fairness Violation (%)
Fairness Violation (%)
(c) ACSTravelTime
Figure 2: Comparing the fairness-utility tradeoffs of tabular foundation models (TabICL, TabDPT, and TabPFN) under uncertainty-bant based in-context sample selection (Uncertain+TabPFN) for different coverage (c controlling the tradeoff). Results with other datasets can be found in the Appendix (Figure 5).
In the previous experiment, we compared the Correlation Remover and Uncertain methods in terms of their fairness-acot accuracy tradeoffs by varying their respective control parameters. The results showed that the Uncertain methods often achieved better Pareto-dominant points compared to both Vanilla
10

---

Published in Transactions on Machine Learning Research (01/2026)
Vanilla
Group Correlation Remover
Uncertain+TabPEN
Accuracy
Demographic Parity
Equal Opportanity
Equalized Odds
30-
30
80
25
25-
20
15-
15-
76-
10
10-
10-
74
5-
1,000
3,000 4,000 5,000
1,000 2,000
4,000 5,000
1,000
4, 4,000
1,000
4,000
It-Contlent Size
Io-tonlext Size
In-Conlext Size
In-Conlend Size
(a) ACSIncome
Accuracy
Demographic Parily
Equal Opportunity
Equalized Odds
15
76-
10
72
0
1,000
2,000
3,000
4,000
5,000
0
1,000
2,000
3,000 4,000 5,000
1,000 2,000
3,000 4, 4,000 5,000
0
1,000 2,000
3,000 4,000 5,000
In-context Size
Io-context Size
In-context Size
In-context Size
(b) ACSMobility
Accuracy
Demographic Parity 4
Equal Opportunity 4
Equalized Odds L
20
67.5
15-
65.0
10
10
62.5
5
60.0
37.5
-5
1,000 2,000 3,000 4,000 5,000
0
1,000 2,000 3,000 4,000
5,000
1,000
2,000
3,000 4,000 5,000
1,000 2,000 3,000 4,000 5,000
In-context Size
Inscontext Size
In-context Size
In: co-contest Size
(c) ACSTrave/Time
Figure 3: Ablation on the in-context example set size. Analyzing the impact of the in-context set size the fairness and accuracy of ICL prediction with TabPFN.
ICL and Correlation Remover. Notably, using TabPFN for uncortainty estimation (Incertain+TabpFr) outperformed the variant that relies on logistic regression (Incertain+LR).
In this experiment, we extend the comparison to different foundation models by applying the same setup using our best-performing fairness intervention: Uncertain+TabPFN. Figures 2a and 2b show the fairness accuracy Pareto fronts on the ACSIncome and ACSMobility datasets, respectively.
Without any fairness intervention (i.e., using the vanilla ICL approach), there no clear winner among the foundation models in terms of fairness; the results vary across datasets. While one might expect TabDPT-pretee pretrained real-world data-to encode more real-world biases and thus exhibit poorer fairness performance, this is not consistently observed. In fact, TabDPT often performs comparably to or better than foundation models pretrained on synthetic data (R3).
Under fairness intervention with Uncertain+TabPFN, all three foundation models exhibit similar fairness performance, demonstrating that the Uncertain method can consistently control the fairness- accuuacy trade-off regardless of the underlying foundation model. However, both TabPFN and TabDPT consistently achieve higher accuracy across datasets (R3). These findings consistent with previous studies: the
11

---

while Published in Transactions on Machine Learning Research (01/2026)
benchmark Erickson et al. (2025) reported strong performance for TabPFN (Qu et al., 2025), while the authors of TabDPT highlighted its competitive performance relative to TabPFN. We observe similar patterns across additional datasets, and results are provided in the Appendix (see Figure 5).
4.3 Ablation on Impact In-context Sample Size
In all previous experiments, we used the full training set as the in-context example set whenever possible. For example, the current version of TabPFN is limited to handling maximum of 10,000 samples (Hollmann et al., 2025). To understand the impact of in-context set size, conduct ablation study by varying the number of in-context examples across the range keeping the evaluation setup identical described above.
Figure 3 shows that accuracy increases substantially with larger in-context sets, particularly in the lower range. However, unfairness shows a slight increase initially and then stabilizes once the in-context set exceeds approximately 700 examples. Across all in-context sizes and datasets, the Correlation Remover method consistently exhibits the highest level of unfairness, reinforcing earlier observations of its bias amplification. In contrast, the Uncertain+TabPFl method consistently achieves the lowest fairness violation, demonstrating robustness to the size of the in-context set. This indicates that even when only subset of training data be used, Uncertain+TabPFl remains effective at miligating bias without overly compromising accuracy. We observe a similar trend when using TabICL model, and results are provided in Appendix 13.
In Appendix C, we perform a computational analysis comparing running and inference time of different fainress interventions and loundation models. This analysis shows that fairness interventions themselves are computationally inexpensive. The runtime dominated by the choice of the tabular foundation model. TabPFN is the most efficient, benefiting from its compact architecture. The uncortainty-based variant with LR is fastest, while using it with TabPFN itself for uncertainty estimation adds moderate overhead..
4.4 Comparison with LLM-based ICL Methods
Method
Foundation model
Accuracy
APP
A BоP
Vanilla
TabPFN
85.72+0.3
16.99+1.0
8.80
Vanilla
LLaMA-2-18B
76.00
14.00
11.00
Hu et al. (2024)
GPT-3.5-turbo
77.93
6.64
10.9
Bhaila et al. (2024)
LLaMA-2-13B
75.72-1.6
8.00
3.00+3.0
Uncertain+TabPFN
TabPFN
82.0541.0
5.51:1.4
3.86
Table 2: Comparison of accuracy and fairness metrics (ADp, ΔBоP) on the ADULT dataset.
Despite LLMs' in-context learning capabilities, they often lag behind classical machine learning models large-scale tabular datasets (Hegselmann et al., 2023). To validate the effectiveness of our uncertainty- demonstration selection method, we compare it against fairness-aware LLM-based fair ICL approaches proposed by Hu et al. (2024) and Bhaila et al. (2024) on the widely studied ADULT dataset (Asuncion et al., 2007). Table presents the results. Vanilla TabPFN achieves an accuracy of 85. 85.72% significantly outperforming the I.L.MA---33B bascline, which reaches only 76.00%. This 10-point gap underscores the performance advantage of specialized models for tabular data, consistent with prior findings that LLMs struggle such inputs when used naively (Hegselmann et al., 2023).
While these comparisons rely on LLMs that are not the most recent generation, and performance may have improved with newer models, the results nonetheless highlight. the cont strength of tabular foundation models like TabPFN in this setting. Moreover, when fairness interventions are applied, our uncertainty-bart selection strategy consistently preserves high prodictive performance while achieving comparable or better fairness outcomes than the based fair ICL approaches-ddmonstraeee that principled sample selection improve fairness without sacrifieing accuracy.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Dataset
Fairness Intervention
TabPFN
TabICL
Accuracy 4
F1 Score
Accuracy +
F1 Score
None
77.2
78.4
75.040.5
76.18
ACSIncome
Correlation R. (S1)
100.040.0
100.040.0
99.940.0
99.93
Correlation R. (S2)
53.8
67.0+0.9
52.9
68.9
None
10.4
17.5
72.8
73.7510.5
ACSTravelTine
Correlation (S1)
100.040.0
100.040.0
100.040.0
100.00
Correlation R. (S2)
53.641.4
54.2+11.5
51.8+1.4
66.4
None
91.4
88.9
91.510.1
89.23
ACSPublicCoverage
Correlation (S1)
100.0+0.0
100.040.0
100.0+0.0
100.00
Correlation R. (S2)
57.8
0.1
56.5+1.0
0.1
None
64.020.4
62.0
65.0+0.3
62.23
ACSEmployment
Correlation (S1)
100.0+0.0
100.0+0.0
100.0+0.0
100.00
Correlation R. (S2)
52.610.6
27.7
53.715.1
53.0+10.6
None
68.320.8
67.8+1.0
67. 67.641.0
67.40+1.2
ACSMobility
Correlation (S1)
100.0+0.0
100.040.0
100.0+0.0
100.00
Correlation R. (S2)
49.2 11.5
49.2 113.2
49.2 10.8
40.2 L31.2
None
80.2
89.0
80.4
89.04
Diabetes
Correlation R. (S1)
100.010.0
100.040.0
100.0.0.0
100.00
Correlation R. (S2)
67.8 113.8
78.9
79.3.0.3
88.4
None
12.5
70.3
71.813.1
71.02
German Credit
Correlation R. (S1)
100.040.0
100.040.0
100.0+0.0
100.00
Correlation R. (S2)
37.8
44.0+11.5
46.3
38.8+14.6
None
84.7
83.2
85.0
83.1610.3
CelebA
Correlation R. (S1)
100.0+0.0
100.0+0.0
100.0+0.0
100.00
Correlation (S2)
54.8
1.0
45.2
62.2+0.2
Table 3: Accuracy ICL prediction of sensitive attribute after applying Correlation Remover on the training and testing datasets (S1) or only to train dataset (S2). Applying the transformation only to the train datasct significantly reduces the accuracy of predicting the sensitive altribute.
4.5 On the Failure of Correlation Remover
In our previous experiments, we observed that applying the CR to both the training and test data can exacerbate unfairness in ICL predictions. We hypothesized that the foundation model infers the sensitive attribute from the línear transformation applied to each non-sensitive feature, which inadvertently leaks sensitive information and increases unfairness. To validate this hypothesis, we conducted correlation removal prediction of the sensitive attribute after applying fairness interventions. As shown in Table 1, the ICL prediction the sensitive attribute achieves 100% accuracy following the application of CR. This indicates that the foundation model continues rely heavily on the sensitive attribute even after correlation removal is performed.
For further verification, we tested variant of correlation removal where the feature transformation (see Eq. 10 in Appendix) is applied only to the training data, leaving the test data unchanged (referred to variant S2). Table 3 shows that this variant significantly reduces the ICL prediction accuracy of the sensitive altribute. This demonstrates that the foundation model exploits the transformation applied to the test set in the original correlation removal method (variant S1) as a proxy to fully reconstruct sensitive attributes.
This observation aligns with prior discussions by Aïvodji et al. (2021) regarding scenarios where data transformations reduce correlation with sensitive attributes. Besides the classical correlation removal setting
13

---

Published in Transactions on Machine Learning Research (01/2026)
Vanilla TabPFN
Correlation Remover (S1)
Correlation Remover (S2)
Demographic Parity
Equal Opportunity
Equalized Odds
81.5
81.5
81.5
81.0
81.0
81.0
Accuracy
80.5
80.5
80.5
80.0
80.0
80.0
79.5
79.5
79.5
79.0
79.0
15
20
25
10
30
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(a) TabPFN
Vanilla TabICL
Correlation Remover (S1)
Correlation Remover (S2)
Demographic Parity
Equal Opportunity
Equalized Odds
81.5
81.5
81.5
81.0
81.0
Accuracy
80.5
80.5
80.5
80.0
80.0
80.0
79.5
79.5
79.5
79.0
79.0
79.0
78.5
78.5
10
15
20
25
5
10
15
20
25
5
10
15
20
25
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(b) TabICL
Figure 4: Evaluating variants of the correlation remover on the ACSIncome dataset with TabPFN and TabICL. Applying correlation remover to the training and lesting data exacerbates unfairness, while applying the transformation only to the training set improves fairness.
(S1), where transformations are applied to both training and test sets, we evaluated the fairness accursscccccyy trade-off when applying the transformation only to the training (S2).
As illustrated in Figure 4, variant S2 of CR substantially improves fairness compared to S1. This confirms the foundation models' capacity to reconstruct sensitive information when the test set is transformed, leading to bias amplification. And the drop in reconstruction accuracy of the sensitive attribute directly supports the observed improvement in lairness when using variant. S2
Based on these findings, we recommend practitioners apply the S2 variant of Correlation Re- mover-transinn transfornnnn only the training data-when using correlation removal in-context learning frameworks, to avoid sensitive attribute leakage and mitigate bias amplification during testing.
5 Conclusion and future works
In this study, proposed and analyzed the effectiveness of three preprocessing methods to enhance the fairness of in-context lcarning (ICL) predictions. Our empirical results, performed on cight fairness benchmarks, posit the uncertainty-
in-context selection method as a strong baseline for improving the fairness of tabular ICL. The key advantages of this method are threefold: (1) it does not require finetuning or retraining the foundation model enforce the desired fairness metrics; (2) it can consistently improve three widcly used group fairness metrics; (3) it offers a parameter control the fairness-utility, tradeoff. To our knowledge, this is the first work that explores pre-processing fairness intervention tabular foundation models. We hope
14

---

Published in Transactions on Machine Learning Research (01/2026)
this work will trigger more investigations into fair tabular ICL, since in-context learning new learning paradigm will be increasingly adopted into decision-making tools. Interesting future research directions include investigating in-processing and post-processing methods and analyzing the effect of distribution shift between in-context and test examples on fairness and accuracy.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Ethics Statement
This paper explores ways reduce unfairness in tabular foundation models, emphasizing fair treatment for various groups. We recognize the significance of fairness in machine learning, especially regarding sensitive attributes like race, gender, and socio-cconomic status. Our research seeks to uncover and tackle potential biases in these models, thereby enhancing transparency, accountability, and inclusivity. While the proposed method uses a sensitive attributes predictor, which could be unlawful in some countries, we emphasized that predicted sensitive values are not used either for training or measuring unfairness. We use the attribute classifier only quantify uncertainty, and emphasize that this method should not be used for any purpose other than bias measuring or mitigation.
References
Alekh Agarwal, Alina Beygekimer, Miroslav Dudík, John Langford, and Hanna Wallach. A Reductions Approach to Fair Classification, July 2018. URL http://arxiv, wrgg/bs//80...../// 02455 arXiv:1803.0245:
[cs].
UIrich Aivodji, François Bidet, Sébastien Gambs, Rosin Claude Ngueveu, and Alain Tapp. Local data
debiasing for fairness based on generative adversarial training. Algorithms, 14(3):87, 2021.
Anastasios N Angelopoulos, Stephen Bates, et al. Conformal prediction: A gentle introduction. Foundations and Trendol in Machine Learning, 16(4):494-55) 591, 2023.
Arthur Asuncion, David Newman, et al. Uci machine learning repository, 2007.
Mislav Balunović, Anian Ruoss, and Martin Vechev, Fair normalizing flows. arXiv preprint ar (rXi::2066.0033,, 2021.
Karuna Bhaila, Minh-Hao Van, Kennedy Edemacu, Chen Zhao, Feng Chen, and Xintao Wu. Fair in-context. learning via latent concept variables. arXiv preprint arXi::241.006711 2024.
Sarah Bird, Miro Dudik, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, and Kathleen Walker. Fairlearn: A toolkit for assessing and improving fairness in ai. Microsoft, Tech. Rep. AASR-TR-20200 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1999 1901, 2020.
L Elisa Celis, Vijay Keswani, and Nisheeth Vishnoi. Data preprocessing to mitigate bias: A maximum entropy based approach. In International conference on machine learning, pp. 1349-1359. PMLR, 2020.
Tianqi Chen, Tong He, Michael Benesty, Vadim Khotilovich, Yuan Tang, Hyunsu Cho, Kailong Chen, Rory Mitchell, Ignacio Cano, Tianyi Zhou, et al. Xgboost: extreme gradient boosting. package version 0.4-2, 1(4):1 4, 2015.
Thibault Cordier, Vincent Blot, Louis Lacombe, Thomas Morzadee, Arnaud Capitaine, and Nicolas Brunel. Flexible and Systematic Uncertainty Estimation with Conformal Prediction via the MAPIE library. In Conformal and Probabilistic Prediction with Applications, 2023.
Elliot Creager, David Madras, Jörn-Henrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly Fair Representation Learning by Disentanglement, June 2019. URL http: //arxiv.org//aaa wrg./bs/////..oooo 02589. Number: arXiv:1906.02555 0025999 arXiv:1906.0255) [cs, stat].
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring Adult: New Datasets for Fair Machine Learning, January 2022. URL wtt.aaawwww..raa/////a///// 04884. arXiv:2108.0488. (es, stat].
16

---

Published in Transactions on Machine Learning Research (01/2026)
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd Inmovations in Theoretical Computer Science Conference on ITCS '12, pp. 214 226, Cambridge, Massachusetts, 2012. ACM Press. ISBN 978-1-4503- 1-15--1-11111... doi: N0.1145/2090266.200000
URL http://ål. acm. org/citation cfm7doid 2090255.
Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, and Frank Hutter. TabArena: A Living Benchmark for Machine Learning on Tabular Data, June 2025. URL btttp://arxiv: arg/abs/2506. 16791. arXiv:2506. 16791 [es].
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasntbramanian. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 259 268, 2015.
Andrew Frank. Uci machine learning repository. /archve ics. uci. edu/ml, 2010.
Josh Gardner, Zoran Popovic, and Ludwig Schmidt. Benchmarking distribution shift in tabular data with tableshift. Advances in Neural Information Processing Systems, 2023.
Anurag Garg, Muhammad Ali, Noah Hollmann, Lennart Purucker, Samuel Müller, and Frank Hutter. Real- tabpfn: Improving tabular foundation modcls via continued pre-training with real-world data. arXiv preprint urXiv22077.3977, 2025.
Moritz Hardt, Eric Price, and Nathan Srebro. Equality of Opportunity in Supervised Learning, October 2016. URL. aattp://arxte http:///aavvvvo.oaaabaaaa/e wttpg/b//1610.00000
Number: arXiv:1610.02415 arXiv:1610.02413 [es].
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models. Tn International Conference Artificial Intelligence and Statistics, pp. 5549 5581. PMLR, 2023.
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXic:2207. Xii:2200700848, 2022.
Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Ti- bor Schirrmeister, and Frank Hutter. Accurate predictions on small data with tabular foundation model. Natwre, 637(8045):319 326, 2025.
Jingyu Hu, Weiru Liu, and Mengnan Du. Strategic demonstration selection for improved fairness in Ilm in-context learning. arXiv preprint arXiv:2 09757, 2024.
Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrimination. Knowledge and Information Systems, 33(1): 33, October 2012. ISSN 0219-1377, 0219-3116. doi: 10.1007/ s10115-011-0463-.. URL http://linke springer.rprnggr.... wwwwcc..cc..//occ 1007 http:/00..0000/11111--00----0000
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
Patrik Joslin Kenfack, Samira Ebrahimi Kahou, and Ulrich Aivodji. A survey on fairness without demographics. Transactions on Machine Learning Research, 2024a. ISSN 2835-8856. URL https://openrevierr wtw.tpeerrrieeeeeeeeeeeeee net// forum?id=3HE4PPMIH.
Patrik Joslin Kenfack, Samira Ebrahimi Kahou, and Ulrich Aivodji. Fairness under demographic scarce regime. Transactions on Machine Learning Research, 2024b. ISSN 2835-8856. URL attps://openrent
nett/forrmiiid--11166oolll.
Po-Nien Kung, Fan Yin, Di Wu, Kai-Wei Chang, and Nanyun Peng. Active instruction tuning: Improving cross-lask generalization by training prompt sensitive tasks. arXiv preprinl ar wxXim:2311.00288 2023.
Matt Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Advances in neural information processing systems, 30, 2017.
17

---

Published in Transactions on Machine Learning Research (01/2026)
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018):11, 2018.
Huan Ma, Changqing Zhang, Yatao Bian, Lemao Liu, Zhirui Zhang, Peilin Zhao, Shu Zhang, Huazhu Fu, Qinghua Hu, and Bingzhe Wu. Fairness-guided few-shot prompting for large language models. Advances in Newral Information Processing Systems, 36:43136-4331 2023.
Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse C Cresswell, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, and Anthony Caterini. Tabdpt: Scaling tabular foundation models. arXiv preprint ar orXic22400.188 1011116,, 2024.
David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning Adversarially Fair and Transferable Representations, October 2018. URL http://arxias http:/axxxvvorr/eeeee./
06309. Number:
arXiv:1802.0633 arXiv:1802.0638 [cs, stat].
Costas Mavromatis, Balasubramaniam Srinivasan, Zhengyuan Shen, Jiani Zhang, Huzefa Rangwala, Christos Paloutsos, and George Karypis. Which examples to annotate for in-context learning? towards effective and efficient selection. arXiv preprint arXiv:2310.20046. 2023.
Dave Mbiazi, Meghana Bhange, Maryam Babaei, Ivaxi Sheth, and Patrik Joslin Kenfack. Survey ai ethics: A socio-technical perspective. arXiv preprint arXiv:2311.17228, 2023.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A Survey on Bias and Fairness in Machine Learning, Jamuary 2022. URL http://arxiv. org/obg httg///b//990....e. 09635. Number: arXiv:1908.0966 09635 arXiv:1908.088. 09635 [cs].
Felix Petersen, Debarghya Mukherjee, Yuekai Sun, and Mikhail Yurochkin. Post-processing for individual fairness. Advances Neural Information Processing Systems, 34:25944-25955, 2021.
Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. Tabiel: A tabular foundation model for in-context learning on large data. arXiv preprint arXiv:2502.0556 2025.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAT 7 blog, 188):9, 2019.
Jake Robertson, Noah Hollmann, Noor Awad, and Frank Hutter. Fairpfn: Transformers can do counterfactual fairness. Xiii preprint (77::22407.05792, 2024.
Yuji Roh, Kangwook Lee, Steven Fuijong Whang, and Changho Suh. Fairbatch: Batch selection for model fairness. Xiv preprint orXiw:2012.01696 2020.
Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008.
Vianney Taquet, Vincent Blot, Thomas Morzadec, Louis Lacombe, and Nicolas Brunel. Mapie: an open-source library for distribution-free uncertainty quantification. arXic preprint 12X7:2207.12274, 2022.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world, volume 29. Springer, 2005.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. Proceedings of the 2018 AAAT/ACM Conference on AT, Ethics, and Society, pp- 335-33 340, 2018.
18

---

Published in Transactions on Machine Learning Research (01/2026)
Appendix
A
Datasets
We experiment on tasks from the recently proposed folktables (Ding et al., 2022), which contains data extracted from the American Community Survey (ACS) Public Use Mierodata Sample (PUMS) (Ding et al., 2022). More specifically, we experiment with the following ACS PUMS tasks:
The task involves predicting whether an individual's income exceeds $50,000. The dataset is filtered to include only individuals over the age of 16 who reported working at least 1 hour per week during the past year and earned a minimum of $100.
ACSIncome:
ACSMobility: This task involves predicting whether individual had the same residential address one year ago. The dataset is filtered to include individuals aged between 18 and 35. This filtering increases the difficulty of the task, as more than 90% of the general population tends to stay at the same address ycar-to-ycar.
This task predicts whether an individual has a commute longer than 20 minutes. The dataset is filtered to include only employed individuals above the age of 16. The 20-minute threshold corresponds to the median commute time the US, according to the 2018 ACS PUMS data.
ACSTravelTime:
mey::::oyment: The objective is predict whether an individual employed, using a dataset. filtered to include individuals aged between 16 and 90.
The goal is to predict whether individual has public health insurance. The dataset is filtered to include individuals under 65 years of age and those with income below $30,000, focusing on low-income individuals who are ineligible for Medicare.
ACSPublicCoverage:
These tasks were selected to reflect a range of real-world predictive challenges with fairness concerns. A limitation of the ACS PUMS datasets that they are US-centric; we diversify the experimental sctup by including other tasks and datasets. Specifically, we also evaluate on the following tabular datasets and tasks:
Diabetes (Gardner et al., 2023): The diabetes prediction task uses features related to physical health, lifestyle factors, and chronic conditions, derived from the BRFSS questionnaires. Demographic attributes like race, Scx, state, and income also included. The target is binary indicator of whether the respondent has ever been diagnosed with diabetes.
German Credit (Frank, 2010): The German Credit dataset contains 20 attributes of 1,000 individuals. We create the task of classifying people according to whether they have good or bad credit risk using age (over or below 25 years old) as the sensitive altribute.
CelebA (Liu et al., 2018): The dataset contains 202,599 samples described with 40 facial attributes of human annotated images. We create the task of predicting attractiveness with facial attributes using gender as the sensitive attribute (Kenfack al., 2024b). Note that we do not train the model with images and consider this task diversify the experimental tasks.
B Background
B.1 Fairness Metrics
In this work, we focus on group fairness notions that measure the performance disparity across different demographic groups. More specifically, consider the following three widely used group fairness metries:
19

---

Published in Transactions on Machine Learning Research (01/2026)
Table 4: Summary datasets used in our experiments. For each dataset, we report the number of features (including the sensitive attribute), the number of samples available, and the sensitive attribute used for fairness evaluation.
Dataset
# Features
# Samples
Sensitive Feature
Prediction Task
ACSIncome
10
22,268
Gender
Income 2 $50,000
ACSEmployment
16
47,777
Gender
Employment status
ACSTravelTime
16
19,492
Gender
Commute time over 20 minutes
ACSMobility
21
8,625
Gender
Residential mobility
ACSpublieCoverrae
19
18,525
Gender
Public health insurance coverage
CelebA
39
202,599
Gender
Attractiveness
Diabetes
183
38,575
Race
Prior diabetes diagnose
German
58
990
Age
Credit risk
Demographic parity (DP): DP enforces equal positive outcome rate for different groups (Dwork et al., 2012) and defined as follows:
P(f(X) = 1|8 = P(f())
P(f(X) = 1)
(1)
Equalized Odds (EOD): EOdds is satisfied when the model makes correct and incorrect predictions at the same rate for different demographic groups (Hardt et al., 2016). The metric enforces equal true positive and false positive rates across groups and is mcasured as follows;
P(S(X) 1/5 0, = y) -((X) 1,y
-, = v), Vy (0.1)
=
(2)
Equalized Opportunity (EOP): In some settings, one can care more about assessing unfairness when the model makes correct predictions. EOP enforces equal true positive rates across groups, i.e., we only consider in Eq. i.e.
P(f(X) = 1|5 0, = P(f(X) 1,Y
33)
Empirically, we measure each fairness considered, i.e., Demographic Parity (ADP), Equal Opportunity (ABOP), and Equalized Odds (AEOD) as follows.
ADP
E
[[f(())
= 111
E [(f()) 1H
(4)
x A=0
Where II(-) the indicator function.
AEOD og 01
(5)
ΑEOP = 01
(6)
Where and on measure the difference between the false positive and the true positive rates across groups, respectively, and are empirically measured as follows.
Where on and ag measure the difference between the false positive and the true positive rates across groups, respectively, and empirically measured as follows.
20

---

Published in Transactions on Machine Learning Research (01/2026)
0j =| [[[())
E
E =A=LY=) == - 11] jE (0,
[[(x)=
(7)
Since the disparities can be below 0.1 on some datasets, we scaled the fairness values reported throughout the paper by 100 to make it easier to read.
B.2 Correlation Remover
The Correlation Remover (Feldman et al., 2015) is a preprocessing technique designed to eliminate linear corrclations between sensitive attributes and non-scnsitive features in datasot. This method is particularly useful in mitigaling biases that may arise due to such correlations, especially when employing linear models.
Considering classification task with the given training data D {(x,, V4, s=));=1 where æ; is input feature vector, is the corresponding class label, and 8; the corresponding demographic group.
To apply Correlation Remover, we assume the training data is formulated as follows:
X E gexd represents the training data matrix containing sensitive and non-sensitive features.
S E Rexm, matrix of the sensitive feat ures. For simplicity, we assumed in this work m, 1, which corresponds to a single binary sensitive attribute.
Z E R=xm= a matrix of non-senstive feat ures such that X = (S 2]
The goal of Correlation Remover is to transform Z into Z" such that z is uncorrelated with S, while retaining as much information from the original Z as possible.
For each non-sensitive feature vector z² E E (the j-th column of Z), the algorithm solves the following least squares problem:
min 2 - (S - -1,5 11,,s)
W,
where:
s S1 ,i,,s,,..,.... Sm,] is the mean vector of the sensitive features, i.e., Sy is the mean of j-th the sensitive feature.
1, is an ne-dimnnsional column vector of ones.
w R the weight vector that projects the centered sensitive fcatures onto Zj.
After computing the optimal weight vectors w for all j E (l....,m.k they are assembled into a weight matrix W = (w....., w w The transformed non-sensitive features are then obtained by:
Z' = Z - (S - 1,, ))
(9)
This operation effectively removes the linear correlations between S and resulting in Zz that is uncorrelated with the sensitive features.
Correlation Remover introduces tunable parameter E [0, 1] that controls the extent of correlation removal, ie., (i) a == corresponds to full removal of linear correlations, thus best possible fairness; (ii) a = 0 corresponds no transformation, the original data is used; (iii) 0 < a < 1 corresponds to partial removal, balancing between the original and transformed data, thus controlling the fairness accuracy tradeoff. More specifically, the final transformed dataset X' computed as:
='=a2e aZ* + (1 (l
(10)
Note that X' is derived using Z' since S is dropped after transformation. The convex combination 10 allows practitioners to adjust the fairness accuracy tradeoff based on specific requirements of their application.
Equation 8 is optimized on the training datasct, and the optimal weight vectors w are used to apply the transformation 10 the test dataset.
21

---

Published in Transactions on Machine Learning Research (01/2026)
B.3 Uncertainty measure with conformal prediction
While any uncertainly measurement method could be used in our method, we employ conformal prediction due to its strong coverage guarantees. Specifically, we used Split Conformal Prediction (Angelopoulos et al., 2023), which is a distribution-froe and model-agnostic method that provides prediction sets for classification tasks, ensuring a user-specified coverage level e with no assumptions beyond data exchangeability.
Given labeled data Pirain {I 9;) s;));== with binary sensitive attribute s; E {0,1). we split the data into proper training set (Di) and calibration set (Dz). In the experiments, we did 50%-50% split of the dataset. with sensitive attribute to obtain D1 and D2.
We then train a probabilistic classifier f: f X + [0, 1], on D1, yiclding predictions:
i. = f(x)) = P(S = 1 X = =)))-
(11)
In our sctup, f could be Logistic Regression (Uncertain+LL) or TabPFN (Uncortain+TabPPE) The starting point for conformal prediction is what is called nonconformity measure, a real-valued function that measures how prediction is different from any possible class label.
Nonconformity Scores After training I on the proper training set, use the calibration dataset to compute the nonconformity scores, which measure how far the prediction is from the true label. More specifically, for each calibration point (a,, si) E D2, we considered the nonconformity score is defined as:
==
(12)
We then compute the quantile Chreshold based on the user-defined larget coverage e E [0, 1J, e.g., 0.05 mean 95% coverage. Specifically, 7 is defined based on - quantile of the nonconformity scores.
= Quantile ((c))))))
(13)
Prediction Set The quantile threshold is used to build the prediction set data points from the test set. More specifically, for a test sample Fiest, we compute the prediction probability Prest = f((rost) and derive its prediction set as follows:
[[ = {s E 10, :11::=1 :8 Prest ST..
(14)
When the prediction set only contains {0} or {1}, then the prediction is confident with at least 1 - probability, while when the prediction set contains both labels, i.e. f0, 1}, the prediction is considered uncertain. Our uncertainty-based demonstration selection method uses only samples whose prediction set contains two values. The coverage guarantee of conformal prediction holds under the assumption that the calibration and test data are eachangeable. We use the open-source implementation of split conformal prediction provided by the MAPIE Python package.
Example. Consider a simplified task consisting of two non-sensitive features (f' and f²), one binary target (V), and a binary sensitive attribute (s).
To estimate the uncertainties, we first train sensitive attribute classifier, using a fraction of the dataset (20% in our experiments). We train the sensitive attribute classifier (Logistic Regression TabPFN) using f and f² to predict s. A conformal predictor returns for given (test) sample z prediction set l'(z) that contains the true sensitive attribute with probability of at least --c. c. For example, setting = 0.05 mcans 95% of the data will contain the true label in their prediction sets. The size of the prediction set,, therefore, provides information about the prediction uncertainty, i..., [[())] = 1 means the prediction is confident with at least 1 probability, and [(x)) = 2 means the prediction is uncertain. The intuition of our method is that using samples with uncertain predictions makes challenging for the foundation model to infer and rely the sensitive attributes to make predictions on the target.
22

---

Published in Transactions on Machine Learning Research (01/2026)
We therefore include training example in the context set if [x)) = and filter it out otherwise. helps to control the fairness-utility tradeoff since smaller corresponds to filtering out more demonstraaion examples with higher confidence sensitive attributes prediction, which can better improve fairness while impacting accuracy as the size of the in-context set is reduced.
C Runtime Comparison of Fairness Interventions
In addition to fairness -ttiiess tradeoffs, we provide a detailed comparison of the computational cost of the fairness interventions across tabular foundation models. While prior sections focused on predictive performance, practical deployment also requires understanding efficiency, particularly in large-scale or resource-constrained settings.
Sctup. We measured wall-clock running time an NVIDIA A100 GPU (20 GB memory). Each includes preprocessing, fairness intervention, and in-context learning inference using the full training set and test set. Results are averaged over three seeds, with standard deviations reported. The largest dataset in our experiments, CelebA, was used to provide representative estimates of runtime under high load. Table 5 summarizes runtimes across methods and models. We make the following observations: (i) TabPFN is the most efficient (8 478), benefiting from its compact architecture. The uncertainty- variant with LR is fastest, while using TabPEN itsolf for uncertainty estimation adds moderate overhead. (ii) TabICL shows moderate cost (57 97s), with relatively small variance across interventions. (iii) TabDPT is substantially slower ( 595s), reflecting the higher computational demands of its transformer- based architecture.
Across models, uncertainty-basee with Logistic Regression is consistently the fastest intervention, while correlation removal and group balancing introduce negligible overhead compared to vanilla inference. Recall TabPEN uses at most 10K samples, which reduces the context size compared to other models, thereby explaining the faster inference. Likewise, the uncertainty-based intervention reduces the context size comparod other interventions. Running time on the ACSIncome dataset (Table 6) confirms that fairness interventions themselves computationally inexpensive; the runtime is dominated by the choice of tabular foundation model. For practitioners, TabPEN with uncertainty- based sampling (LR) offers the best balance of fairness gains and efficiency, while TabDPT provides stronger model capacity at substantially higher cost.
Model
Vanilla
Group Balanced Correlation Remover
Uncertain+ Uncertain
TabDPT
596.04 + 69.12
594.98 + 68.35
596.83 70.10
595.58 65.73
316.86 + 50.17
TabICL
69.50 + 0.75
67.84 + 0.85
70.11 + 1.26
57.05 + 1.24
96.65 1.06
TabPFN
10.71 + 1.27
10.84 1.16
13.94 + 1.35
8.45 + 0.82
47.18 0.65
Table 5: Average running time (s) + standard deviation on CclebA dataset across fairness interventions and tabular foundation models.
Model
Vanilla
Group Balanced Correlation Remover
Uncertain LR
TabDPT
154.35 + 53.13
153.23 + 53.06
151.75 +: 52.88
155.09 + 53.41
303.63 + 52.21
TabICL
45.05 + 0.62
45.23 + 0.04
44.63 4 0.62
45.09 + 0.29
45.43 + 1.40
TabPFN
1.85 0.87
1.96 +: 1.07
3.22 + 0.32
1.87 + 0.91
3.65 0.76
Table 6: Average inference time (s) + standard deviation on ACSIncome dataset across fairness interventions and tabular foundation models.
D Supplementary resutls
23

---

Published in Transactions on Machine Learning Research (01/2026)
Demographic Parity
Equal Opportunity
Equalized Odds
70.0
70.0
Accuracy
67.5
67.5
67.5
63.0
65.0
65.0
62.5
62.5
62.5
Vhaiia Thaict.
60.0
Wasila TuarsN
N
60.0
60.0
Vhaila TharPT
Thir 7aao77
57.5
57.5
57.5
2.5
7.3
10.0
0.0
5.0
10.0
Pairmess Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(a) ACSTravelTime
Demographic Parity
Equal Opportunity
Equalized Odds
82
82-
82-
Accuracy
80-
80
70
78-
76
TabiCC
76-
74
3
2
6
8
8
10
Fairness Violation (%)
Fairness Violation (%)
Faimesss Violation (%)
(b) ACSEmployyent
Demographic Parity
Equal Opportunity
Equalized Odds
8.5
85
80-
90-
Accurscy
75
75-
70
70
70
65
63
60
Werdis DAPPN
Vandia TRaICL
Pair Trte
Pair Tabe
60
355
Vendll DaNPT
Fstr IAOFT
55
53-
10
2.5
12.5
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(c) ACSPublicCoverage
Demographic Parity
Equal Opportunity
Equalized Odds
65.0
65.0
65.0
Accuracy
62.5
62.5
62.5
60.0
60.0
60.0
57.5
57.5
57.5
55.0
55.0
55.0
52.54
Werfia TRROFT
52.5
52.5
6
8
2
6
Fairness Violation (%)
Pairness Violation (%)
Fairness Violation (%)
(d) Diabetes
Demographic Parity
Equal Opportunity
Equalized Odds
80
80
Accuraty
78
78
76
76
76
74-
74
72
Vanila Tabici
Pür Tubtei
72
72
Vanila Tabtee Vooills Tea
Pür TubreN
70
Fair ThatiT
70
10
15
20
10
20
30
Fairness Violation (96)
Fairness Violation (9)
Faimess Violation (%)
(e) CelebA
fairness-aceuracy Figure 5: TabPFN vs. TabICL vs. TabDPT. Comparing the fairness- accuracy tradeoffs of tabular foundation models under uncertainty-assed fairness interventions. TabPFN generally provides better fairness accuracy tradcoffs.
24

---

Published in Transactions on Machine Learning Research (01/2026)
Vanilla
Group Balanced
Correlation R.
Uncartain LR
Uncertain + TabPFN
Demographic Parity
Equal Opportunity
Equalized Odds
82.5
82.5
82.5
Accuracy (%)
02.0
82.0
82.0
81.5
81.5
81.5
81.0
81.0
80.5
80.0
2
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(a) ACSEmployment
Demographic Parity
Equal Opportunity
Equalized Odds
80
80
80
Accuracy (%)
70
70
60-
40
40
20
10
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(b) ACSPublicCoverage
Demographic Parity
Equal Opportunity
Equalized Odds
65
Accuracy (%)
60
60
60-
55-
55-
55-
50
45
45
45-
10
15
10
15
10
15
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(c) Diabetes
Demographic Parity
Equal Opportunity
Equalized Odds
80
80-
80
78
Accuracy (%)
76-
76-
76-
74
74
72-
72
72-
70
70
15 Fairness Violation (%)
10
20
10
20
Fairness Violation (%))
Fairness Violation (%)
(d) CelebA
Figure 6: Fairness-accuracy Pareto-front of different fairness interventions with TabPFN.
25

---

Published in Transactions on Machine Learning Research (01/2026)
Vanilla
Group Balanced
Correlation
URC
Uncertain PicertainTTbbPFy
Demographic Parity
Equal Opportunity
Equalized Odds
81
81
81
80
80-
Acouracy (%)
79-
78
77-
76-
76
75
75
75
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(a) ACSIncome
Demographic Parity
Equal Opportunity
Equalized Odds
79
79
79-
Accurscy (%)
77
77
76-
76
76
75-
75
75
74
74
74
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(b) ACSMobility
Demographic Parity
Equal Opportunity
Equalized Odds
66
Accuracy (%)
64-
62
10
15
10
15
10
15
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(c) ACSTravelTime
Demographic Parity
Equal Opportunity
Equalized Odds
83
83
83
82
82-
82
Accuracy (%)
81
79
79
79-
78
78
78
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(d) ACSEmployment
Figure 7: Fairness-accuract tradeoffs the ACS datasets with TabICL foundation model
26

---

Published in Transactions on Machine Learning Research (01/2026)
Vanilla
Group Balanced
Correlation
URC
Uncertain MmorraiinTabPPNN
Demographic Parity
Equal Opportunity
Equalized Odds
81
81
Acouracy (%)
79
79
79
77
77
77-
76
75
75
75
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(a) ACSIncome
Demographic Parity
Equal Opportunity
Equalized Odds
78-
78
78
Accuracy (%)
76-
76
76
10
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(b) ACSMobility
Demographic Parity
Equal Opportunity
Equalized Odds
68
68
68
Acouracy (%)
66
66
66
64
64
62
62
60
60-
60
58
58
0
10
10
15
5
10
15
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(c) ACSTravelTime
Demographic Parity
Equal Opportunity
Equalized Odds
82
82
62
Acouracy (%)
81
8.1
611
80
80
80
78
78
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(d) ACSEEmplyyent
Figure 8: Pairness-accuracy tradeoffs on the ACS datasets with TabDPT as foundation model
27

---

Published in Transactions on Machine Learning Research (01/2026)
Group Balanced
Correlation
Uncertain-LR
Demographic Parity
Equal Opportunity
Equalized Odds
66
66
66
64
64
64
Accuracy (%)
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(a) Diabetes
Demographic Parity
Equal Opportunity
Equalized Odds
81
81
81
80
80-
80
Accuracy (96)
79-
79-
76
76
75
75
75
74
74
74
15
10
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(b) CelebA
Figure 9: Fairness-accuracy tradeoffs on the Diabetes and CelebA using TabICL as foundation model
28

---

Published in Transactions on Machine Learning Research (01/2026)
Group Balanced
Correlation
Uncertain-LR
Demographic Parity
Equal Opportunity
Equalized Odds
64
62
62
Accuracy (%)
60
60
60
58
58
58
56-
56-
56-
54
54
54
Fairness Violation (%)
Fairness Violation (%6
Fairness Violation (%)
(a) Diabetes
Demographic Parity
Equal Opportunity
Equalized Odds
Accuracy (%)
78
78
78
76-
74
74
10
20
40
10
30
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(b) CelebA
Figure 10: aacuracy tradeoffs on the Diabetes and CelebA using TabDPT as foundation model
29

---

Published in Transactions on Machine Learning Research (01/2026)
Vanalla
Group Balanced
Correlation R.
Uncertain+LR
Uncertain+
Demographic Parity
Equal Opportunity
Equalized Odds
86
86
86-
ROC AUC (%)
80
80
80
Fairness Violation (%)
Fairness Violation (%)
Fairness Vlolation (%)
TabICL
Vanilla
Group Balanced
Cormiation
Uncertain+L.R
Encertain+TabbPE
Demographic Parity
Equal Opportunity
Equalized Odds
88
88
88
ROC AUC (9)
86
86
84
82
80
10
20
10
5
10
15
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(b) TabDPT
Vanalla
Group Balanced
Correlation R
Uncertain+LR
Uncertain TabPPNN
Demographic Parity
Equal Opportunity
Equalized Odds
85
85
85
ROC AUC (%)
75
75
75
70
70
70-
10
10
10
20
Fairness Violation (%)
Fairness Violation (%))
Fairness Violation (%)
(c) TabPFN
Figure 11: utility tradeoff on the ACSIncome with utility measured by ROC-AUC.
30

---

Published in Transactions on Machine Learning Research (01/2026)
Demographic Parity
Equal Opportunity
Equalized Odds
ROC AUC (%6
80
80
80
Vanilla TabiCL
Vanilla TabPFN
75-
Vanilla TabDPT
Fair TabICL
70
Fair TabPFN
70
70
Fair TabDPT
65
65
65
10 Fairness Violation (%)
15
10
10
15
Pairness Violation (%)
Fairness Violation (%)
(a) ACSIncome
Demographic Parity
Equal Opportunity
Equalized Odds
75
75
ROC AUC (%)
70
70
70
65
Vanilla TabiCL
65
65
Vanilla TabPFN
60
Vanilla TabDPT
60
60
Fair TabICL
Fair N
55
55
55-
Fair TabDPT
50
50
10
0.0
2.5
5.0
7.5
10.0
0.0
2.5
7.5
10.0
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(b) ACSTravelTime
Demographic Parity
Equal Opportunity
Equalized Odds
ROC AUC (%6
85
85
85
80
Vanilla TabiCL
TabPFN
Vanilla TabDPT
Fair TabICL
Fair TabPFN
Fair TabDPT
0
2
8
10
0
10
15
Fairness Violation (%)
Fairness Violation (%)
Fairness Violation (%)
(c) ACSPublicCoverage
Demographic Parity
Equal Opportunity
Equalized Odds
75
ROC AUC (%)
70-
Vanilla Tab[CL
70
Vanila TabPFN
65
Vanilla TdbDPT
65
Fair TabCCL
Fair TabPFN
60
TabDPT
60-
4 Fairness Violation (%)
3
Fairness Violation (%)
Fairness Violation (%)
(d) ACSMobility
Figure 12: Comparing the fairness-utility (ROC AUC) tradeoff of different foundation models under Uncertain+TabPEN fairness intervention. Mcasuring the utility with the ROC AUC score shows sim- ilar trend with accuracy.
31

---

Published in Transactions on Machine Learning Research (01/2026)
Vanilla
Group Balanced
Correlation Remover
Tabertain+TabPEN
Accuracy
Demographic Parity 4
Equal Opportunity
Equalized Odds 4
35
35
80
25
30-
78
20
25
20
20-
15
76-
15
15
10
10
74
20
5-
0-
1,000
2,000
3,000
4,000
1,000
2,000
3,000
4,000
1,000
2,000
3,000
4,000
5,000
1,000
2,000
3,000
4,000
5,000
In-context
In-context
(a) ACSIncome
Accuracy
Demographic Parity
Equal Opportunity
Equalized Odds
78
12.5
10.0
2-
7.5
5.0
74-
-1
-2
4,000
4,000
5,000
In-context Size
In-context Size
In-conbext Size
In-context Size
(b) ACSMobility
Accuracy
Demographic Parity
Equal Opportunity
Equalized Odds
065
50
50
40
40-
64
30
30-
30-
20
20
20
60
58
56
1,000
2,000
3,000
4,000
5,000
1,000 2 2,000
3,000
4,000
5,000
1,000
2,000
3,000 4,000 5,000
0
1,000
2,000
3,000
4,000
5,000
In-con con-oot Size
In-context Size
In-context Size
In-context Size
(c) ACSTravelTime
Figure 13: Ablation on the in-context example set size with TabICL.
32


Reviews:
[{"review_id": "rpNhLAq7XI", "reviewer": "Reviewer_e8tx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work conduct a systematic study of fairness mitigation methods for ICL under the context of tabular foundation model. Extensive experiments are performed to study the fairness utility tradeoff between multiple fairness metrics, datasets, and methods. The authors observe that correlation remover could exacerbates the unfairness of ICL.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are well supported. The experiments are extensive.", "audience_yn": "Yes", "audience_explanation": "The area of ICL using tabular foundation model is interesting and relatively under explored. I think such systematic study is important and is interesting to the community", "requested_changes": "- While I like Section 4.2, I think the comparison in Figure 1 is not apple-to-apple comparison and a bit unnecessary. For the following reason: 1. given that we already have the pareto fairness utility tradeoff, it's not clear to me what is the point of selecting a single point from the curve and compare. 2. The authors mention Figure 1 is obtained by fixing $\\alpha$ and $\\varepsilon$. However, these are hyperparameters for different methods. How are these directly comparable? I recommend removing this section as (correct me if I'm wrong) the information is a direct subset of Section 4.2 and could be confusing.\n- The authors report accuracy as utility. However, for tasks such as ACS, one cares about metrics such as precision and recall, could the authors also report these metrics and the pareto frontier between these metrics and fairness?\n- The authors should provide more motivation on why group fairness is the main fairness notion rather than other fairness notions such as individual fairness?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ED0Q7HCNq4", "reviewer": "Reviewer_DFr8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates fairness in in-context learning using transformer-based tabular foundation models, evaluating three fairness-enhancing methods—correlation removal, group-balanced sample selection, and uncertainty-based sample selection—on multiple benchmark datasets. The study finds that the uncertainty-based strategy consistently improves group fairness metrics with minimal impact on predictive accuracy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The submission makes several important claims about improving fairness in in-context learning with tabular foundation models, and the evidence provided is generally accurate and clear.", "audience_yn": "Yes", "audience_explanation": "Yes, the findings on improving fairness in in-context learning with tabular foundation models would likely interest TMLR’s audience", "requested_changes": "1. The correlation remover method unexpectedly amplifies bias. The authors should provide a deeper analysis of why this happens and suggest ways to mitigate it.\n2. While the uncertainty-based method shows promise, the paper lacks a detailed explanation of how the uncertainty is calculated and how it specifically reduces bias.\n3. The study uses a limited set of datasets. How well do these methods generalize to other types of tabular data, especially those with different distributions or larger scales?\n4. The paper does not discuss the computational cost of the proposed methods. How do they scale with the size of the dataset and the complexity of the model?\n5. The ablation study on the impact of in-context set size is insightful.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EUttEykVeU", "reviewer": "Reviewer_qoEa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work focus on fairness in machine learning for tabular data. In particular, the focus is on tabular foundation models which are pretrained on large amounts of tabular data, and are capable of in-context learning (i.e. making predictions based on new labeled data at inference time, without updating the weights). The goal of this paper is to ensure fairness with respect to a sensitive feature corresponding to e.g. a demographic group (using a few classic “fair ML” performance measurements).\n\nThe paper presents three relatively simple methods: “correlation remover”, which involves transforming features to reduce correlation between sensitive and non-sensitive features, “uncertain”, which consists in selecting in-context demonstrations which have high uncertainty, and “balanced”, which consists in selecting in-context demonstrations to have a balanced population wrt the sensitive feature.\n\nIn their experiments, the authors find that correlation remover performs remarkably poorly (although a variant performs better). The other methods perform reasonably well. The authors also consider the number of demonstration samples, and find that generally, accuracy continues to increase with large demonstration sets, whereas fairness properties actually get slightly worse but plateau quickly. They also compare to regular LLMs (as opposed to tabular foundation models) and find that performance remains competitive, although not always better.\n\nFor correlation remover and “uncertain”, there is also a “knob” that can be tuned to trade off fairness and accuracy. There is then a question of what the Pareto frontier between accuracy and fairness looks like. The authors do these experiments, also considering different variations on the \"uncertain\" method, and different foundation models; unsurprisingly, since “correlation remover” performed so poorly overall, it is Pareto dominated by other methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Everything in the paper is described very clearly. The choices of fairness metrics and datasets are totally standard and defensible. The experiments are thorough and convincing, and answer a number of interesting questions about the relative performance of these methods.", "audience_yn": "Yes", "audience_explanation": "Fair ML is of interest to some people in TMLR's audience, as are tabular foundation models, so the combination of the two is also of interest. Anyone who needs to ensure fairness properties are satisfied when using ICL with tabular foundation models will get something out of this paper.", "requested_changes": "I don't have any substantive requested changes. The paper does what it sets out to do.\n\nSection 2 has one typo that I saw (\"Tabular Fondation Models\").", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "While TMLR does not consider novelty, this work is very closely adjacent to some other work. For example there are other papers looking at preprocessing for fairness in ICL using some of the same techniques (just with regular LLMs, not with tabular foundation models). This is all cited, and it's not a problem per se, but I would have preferred to see the distinctions and similarities emphasized more clearly.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)