Fairness via Independence: A General Regularization Framework for Machine Learning
Authors: Yezi Liu
Hanning Chen
Wenjun Huang
Yang Ni
Mohsen Imani
Venue: ICLR 2026 Poster (2026)
Keywords: Bias Mitigation
Statistical Independence
Fairness in Machine Learning
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We introduce a general framework to promote fairness in machine learning by reducing the dependence between model predictions and sensitive attributes.

Abstract: Fairness in machine learning has emerged as a central concern, as predictive models frequently inherit or even amplify biases present in training data. Such biases often manifest as unintended correlations between model outcomes and sensitive attributes, leading to systematic disparities across demographic groups. Existing approaches to fair learning largely fall into two directions: incorporating fairness constraints tailored to specific definitions, which limits their generalizability, or reducing the statistical dependence between predictions and sensitive attributes, which is more flexible but highly sensitive to the choice of distance measure. The latter strategy in particular raises the challenge of finding a principled and reliable measure of dependence that can perform consistently across tasks. In this work, we present a general and model-agnostic approach to address this challenge. The method is based on encouraging independence between predictions and sensitive features through an optimization framework that leverages the Cauchy–Schwarz (CS) Divergence as a principled measure of dependence. Prior studies suggest that CS Divergence provides a tighter theoretical bound compared to alternative distance measures used in earlier fairness methods, offering a stronger foundation for fairness-oriented optimization. Our framework, therefore, unifies prior efforts under a simple yet effective principle and highlights the value of carefully chosen statistical measures in fair learning. Through extensive empirical evaluation on four tabular datasets and one image dataset, we show that our approach consistently improves multiple fairness metrics while maintaining competitive accuracy.

Published as a conference paper at ICLR 2026
FAIRNESS VIA INDEPENDENCE: A GENERAL REGULAR- IZATION FRAMEWORK FOR MACHINE LEARNING
Yezi Liu & Hanning Chen & Wenjun Huang
Yang Ni
Department of Electrical Engineering and Computer Science University of California, Irvine Irvine, CA, USA {yezil3, hanninge, wenjunh3
Purdue University Northwest Hammond, IN, USA Langni@purdue. .dd
Mohsen Imani
Department of Electrical Engineering and Computer Science University of California, Irvine
Irvine, CA, USA m. iimaaai@ui @uciiiuucciiiii
ABSTRACT
Fairness in machine learning has emerged as a central concern, as predictive models frequently inherit even amplify biases present in training data. Such biases often manifest as unintended correlations between model outcomes and sensitive attributes, leading systematic disparities across demographic groups. Existing approaches to fair learning largely fall into two directions: incorporating fairness constraints tailored to specific definitions, which limits their generalizability. or reducing the statistical dependence between predictions and sensitive attributes, which more flexible but highly sensitive to choice of distance measure. The latter strategy in particular raises the challenge of finding principled and reliable measure of dependence that can perform consistently across tasks. In this work, we present general and model-agnostie approach to address this challenge. The method based on encouraging independence between predictions and sensitive features through optimization framework that leverages the Cauchy-Schwar (CS) Divergence as a principled measure of dependence. Prior studies suggest that CS Divergence provides tighter theoretical bound compared to alternative distance measures used in earlier fairness methods, offering a stronger foundation for
optimization. Our framework, therefore, unifies prior efforts under a simple yet effective principle and highlights the value of carefully chosen statistical measures in fair learning. Through extensive empirical evaluation on four tabular datasets and one image dataset, we show that our approach consistently improves multiple fairness metrics while maintaining competitive accuracy.
INTRODUCTION
Fairness in machine learning has garnered growing concern, as machine learning (ML) models are playing key roles in many high-stakes decision- makingg scenarios, such as credit scoring (Leal, 2022), the job market (Hu & Chen, 2018), healthcare (Grote & Keeling, 2022; Xu et al., 2025), and education (Bøyum, 2014; Kizilcec & Lee, 2022). Among the various fairness notions, group fairness one of the most extensively studied ones it addresses the prediction disparities across demographic groups, including gender, age, skin color, and region (Mehrabi et al., 2021; Dwork al., 2012; Barocas al., 2017). While many group fairness ML algorithms are proposed, they have challenges in their applications, especially the generalizability, i.c., their adaptation to different fairness notions, and robustness, i.e., the stability of the fairness when they encounter a slight change of model parameter.
"Corresponding author.

---

Published as a conference paper at ICLR 2026
0/1
Aqpm20..2
Gender
2.0
Gender
Fomale
Fomale
penaa
Made
Density
1,5
Male
0.0
0.5
1.0
Prediction
Prediction
Figure I: From left to right: (1) Prediction distribution of all classes; (2) T-SNE plot of embeddings for samples from all classes; (3) Prediction distribution of class 1; (4) T-SNE plot of embeddings for samples from Adu t, and the sensitive attribute is gender. The blue points represent samples with sensitive attribute 0, while the red points represent samples with sensitive attribute 1.
Figure 2: Fairness loss landscapes evaluated using three functions, presented from left right: Kullback-Leibler (KI.) divergence, Hilbert-Schmin Independence Criterion (HSIC), and Cauchy- Schwarz (CS) divergence. smaller inner circle indicates greater robustness. Among these methods, the CS divergence achieves the smallest inner circle, ranging from - to 1, while the inner circles of KL and HSIC divergences both span from -2 2.
Existing group fairness approaches can be intrinsically categorized into two main approaches based on their debiasing objectives: i) directly integrate the fairness notion into the training objective, ii) minimizing the correlation between predictions and sensitive attributes. Methods belongs to i) such as a demographic parity (DP) regularizer, and an equality of opportunity (EO) regularizer. The benefit of this approach is that the model trained by the target fairness objective perform well on specific fairness notions. For example, the machine learning model trained at demographic parity has high possibility of achieving good demographic parity in testing. However, such methods limited their generalizability to other fairness notions (shown in Figure 1). Method ii) solves from a more fundamental way that deal with generalizability. including using information theory, adversarial approach to minimize the correlation between the prediction and the sensitive attribute. The most straightforward way is use distance measurement that assesses the relationship between the sensitive attribute and the prediction, thus minimizing this distance during the training. This enables the generalizability, but ascribes the pressure the fairness performance to the quality the distance measurement.
Existing fairness regularizers mainly assessed this correlation using gap parity, Maximum Mean Discrepancy (MMD) (Gretton et al., 2012), Kullback-Leibbe (KI.) divergence, and the Hilbert- Schmidt Independence Criterion (HSIC). However, the current fairness regularizers are sensitive to the model parameter change, making them less robust in maintaining fairness, responding to small change the model parameters (shown in Figure 2). Theoretical studies have shown that the Cauchy- -chwwaz divergence provides tighter bound compared to the Kullback-Leblerr divergence and gap parity, suggesting its potential to improve fairness machine learning models. Motivated result in
by this, we would like to know if using CS divergence consistent utility-farresss trade-off across hyperparameter settings than standard regularizers due fairness
generalizable and more
the benefit of tighter upper bound of CS divergence. In light of this, propose regularizer based on the Cauchy-Schie Schwarz divergence for fair machine learning. To evaluate the generalizability, we tested the fairness under wide range of fairness notions proposed by previous studies, and to evaluate the robustness, we visualize whether small change in the learned model parameters can influence the fairness. We summarize our contributions as follows:
We introduce the Cauchy- Schwarz divergence to fair machine learning and present novel regular- ization method.
We elucidate the relationships between the Cauchy-Sch Schwarz regularizer and other fairness regular- izers, emphasizing its superior effectiveness in debiasing.
2

---

Published as a conference paper at ICLR 2026
Our experimental results, obtained from four tabular datasets and one image dataset, validate the efficacy of the proposed Cauchy-Schwart regularizer in achieving faimesss across multiple fairness notions simultancously.
This develops CS divergence in ttirness-speciffe setting that is complementary to prior work on CS- uS-based density estimation, clustering, and representation learning.
2
PRELIMINARIES
In this section, establish the foundational concepts for our study. We start by exploring the notion of fairness in machine learning, including the relevant notations. Next, we provide an overview of general fairness- aware machine learning methods. Finally, we introduce the Cauchy- Sehwwz divergence and discuss its benefits in reducing bias.
Problem scope. This paper focuses in procese group faimesss in the context of binary classification with binary-sensitive attributes. In contrast, group fairness seeks to ensure that machine learning models treat different demographic groups equitably, where groups attributes such as gender, race, and age (Feldman et al., 2015; Zemel et al., 2013).
defined based sensitive
Notations. Under this setting, we consider a dataset D {(x:, Wa. s;) where M is the number of samples, X; E Rd represents the features excluding the sensitive attribute, y/ E {0, 1] is the label of the downstream task, and 8; E (0, 1} is the sensitive attribute of the i-th sample. The predicted probability for the i-th sample is denoted
= 1]. computed by the machine learning model as 2, = f(x,, 8.) Rdd > [0, 1]. The binary prediction is represented as úi E {0, 1}. defined by
d: 1 1 t[z]], where 1 2 t(-) is the indicator function that evaluates whether its input greater
than or equal to the threshold t. Finally, X, Y. S, and Y denote random variables corresponding to X Yi. 8;, and gi. respectively.
Problem Formulation. Generally, the faimess objective be summarized as follows: min f
Lutility + XL Affaasree fairnese:
where the term Curititt denotes the loss function that measures the utility of the model, often a binary entropy loss, for our binary classification problem, while & fairneas (also shown in Figure 2) indicates the fairness constraint applied the model. The parameter À is used to control the trade-off between utility and fairness.
2.1 GROUP FAIRNESS
There are many ways to define and measure the group fairness. Each definition focuses on distinct statistical measures aimed at achieving balance among subgroups within the data. Among these, De- mographic Parity and Equal Opportunity are the most popular ones, other popular faimess definitions are summarized in Appendix C.2.
Demographic Parity (DP). Demographic Parity (Zafar et al., 2017; Feldman et al., 2015; Dwork et al., 2012; Liu et al., 2026) mandates that the predicted outcome Y be independent of the sensitive attribute S, expressed mathematically as Ý 1. S. Most of the existing literature primarily addresses binary classification and binary attributes, where Y E {0, 1} and {0, 1}. Similar to the concept of equal opportunity, the metric evaluating the DP fairness is defined by:
App P 0) P('SS 1)}-
A lower value of AP signifies a fairer classifier.
Equal Opportunity (EO). Equal Opportunity (Hardt al., 2016) mandates that a classifier achieves equal true positive rates across various subgroups, striving towards the ideal of a perfect classifier. The corresponding fairness measurement for EO can be articulated follows:
A = P(P = 1, S.
p(-|Y 1,5 = 1)].
A low go indicates that the difference in the probability of instance in the positive class being assigned a positive outcome is relatively small for both subgroup members. Both DP and EO can be effectively extended to problems involving multi-class classifications and multiple sensitive attribute
3

---

Published as a conference paper at ICLR 2026
categories, Note that in the binary classification task, ADP and Ago are sometimes calculated after
binarizing P(Y). Specifically, Apr is defined Apr IP(Y 1|5 0) P(ý 1|S = =1)1 and Ag defined as Æ0 = IP(Ý 1|Y =1 - 1,5 0) - P(Y 1| 1,5 1)] (Beutel et al.,
2017: Dai & Wang, 2021; Dong et al., 2022).
2.2 CAUCHY-SCHWARZ DIVERGENCE
Motivated by the well-known Cauchy-Schwarz (CS) inequality for square-integrable functions' which holds with equality if and only if p(x) and g(x) are linearly dependent, we can define measure of the distance between p(x) and ((x). This measure is referred to as the CS divergence (Principe
et al., 2000; Yu et al., 2023), given by:
(J ((xx))((x)d)) Des(p: q) = log (J((x)d [ g(x))dz
(4)
The CS divergence, denoted as Des. is symmetric for any two probability density functions (PDFs)
and 9. satisfying 0 S Des < co. The minimum divergence is achieved if and only if p(x) = (()..
WHAT MAKES A GOOD FAIRNESS REGULARIZER?
Current fair machine learning algorithms adopt a variety of approaches, prompting us explore the essential properties that make for effective fairness regularizers. By categoriz- ing these algorithms into three types and con- ducting preliminary experimentss we aim to identify the key characteristics that contribute to mitigating bias in machine learning models.
Reg.
Faireess Objective (Crarnness)
EEY|S 0) B(YSS 1)1
EO
E(Y|Y= = 1,5 -B)=0 1,5 = 1)
MMD
DAMD (Zs=o, Zz-1)
HSIC
Dusic (S) Dra(Y,
PR
Table 1: Faimess regularizers (Reg.) and objectives.
3.1 BALANCING THE PREDICTION ACROSS DIFFERENT SENSITIVE GROUPS
The first to directly integrate the faimesss notions, such DP and EO, into the faimesss objective. where
Ltarmess [TP, Q)
P( 1 0). (0 P
S 5= 1) for DP,
(5)
P- Y--,,-) Q=
P for EO.
Calculating the distance between P and Q has many ways by using difference distance measurement D, and the most used one is to calculate the absolute distance between the mean empirical estimations:
Lfairnes = [E(P) B(Q)..
(6)
where the expected values are calculated as the mean of summation since PP and Q are discrete distributions.
Previous fair machine learning studies have shown that the fairness loss of the testing can be upper bounded by the loss of training. Therefore, a distance function having a tighter generalization error bound used in training will lead to better fairness guarantee for testing.
Lfairness = | S = 0) E = 1)
(7)
We can see that the basic idea is to balance the prediction distribution between two sensitive groups. Therefore, for this type of approach, we can also use other distance measurements: Therefore, except for the absolute distance between the two prediction distributions.
3.2 BALANCING THE LATENT REPRESENTATION ACROSS DIFFERENT SENSITIVE GROUPS
Distance measures and Lemmiiizttinnn
L Lfairness D(Zsoo, Zs=1), Zs=1))
(8)
where Z is the latent representation from the neural networks, and Zs=o and 75=1 are the represen- tation when the sensitive attribute is 1 or O. The distance metric DD-) here can be Mean Maximum Discrepancy (Louizos et al., 2016).
(S ((x)g(() dr)² < p(x)² dx J q(x)² dx
4

---

Published as a conference paper at ICLR 2026
Extension to multiple sensitive attributes. While theoretical development is presented for a single sensitive attribute S for notational simplicity, the proposed CS- bS-dd regularizer naturally extends to multiple sensitive attributes. Let S (S Sk) denote vector of sensitive at- tributes. One can either (i) treat S as a j.joint variable and apply the same CS divergence to (Y,S). i.., penalize Des Pris: Ps Ps), or (ii) sum the CS divergences over individual attributes, e-g.. Σ Dcs(Pris. Ps.), depending on whether joint or per-attribute control is desired. Both variants use exactly the same empirical estimator as in (Equation (13)) and do not require any algorithmic changes; only the definition of 5 in the mini-batch needs to be updated.
3.3 MINIMIZING THE RELATIONSHIP BETWEEN PREDICTIONS AND SENSITIVE ATTRIBUTES
The goal of this category is to ensure that a fair machine learning algorithm's predictions retain minimal sensitive information.
fairness D(Y,S),
(9)
The HSIC and PR in Table belong to this category. Specifically, the DpR (Y, .S is defined as:
p(ù. 8)
DPr ()=
-==))>=> ((),)) log p(ū)p(s)
(10)
BEY-ES
Note that, adversarial debiasing methods (Zhang et al., 2018) fall into this category because they em- ploy discriminators to predict sensitive group membership from the learned encoded representations. These methods aim to make sensitive attributes difficult to deduce from the encoded representations.
4 CAUCHY-SCHWARD FAIRNESS REGULARIZER
In this section, we first introduce three prominent fairness regularizers that assess distribution distance using different metrics: Mean Maximum Discrepancy, Kullback-Leibler divergence, and Hilbert- Schmidt Independence Criterion (HSIC), For each metric, we explore its relationship with CS divergence. Subsequently, we explain how CS divergence can be utilized to achieve faimess.
4.1 How CAN THE CAUCHY-SCHWARE DIVERGENCE BE APPLIED TO MITIGATE BIAS?
Given samples {x" J and {x?]__ drawn independently and identically distributed (i.id.) from p(x) and q(x) respectively, performed using the kernel density estimator (KDE) as described (Parzen, 1962) and follows the empirical estimator formula in (Jenssen al., 2006).
estimate the empirical CS divergence. This estimation can be
Proposition 4.1. Given two sets of observations {x and {x] let p and q denote the distributions of two groups. The empirical estimator of the CS divergence Des(p; 9) is then given by:
N, Ti-1
1
Des(p: =log
c(x², x²)
log
<(x², xx)
N²
N2
4,1=1
(11)
No
1
2og
<xx,,, xf))
N
1=
The proof of this proposition is detailed in Appendix B.1. where k represents a kernel function, such the Gaussian kernel defined Ka (Ir, exp(-I<a x' 2'1l// 11///222) In the following sections, will explore the relationship between this kernel function and the existing faimess regularizer.
As mentioned earlier, the goal of fairness to ensure an equal distribution of predictions across sensitive attributes. To achieve this, faimess-aware algorithms focus on minimizing the dependency of predictions these sensitive attributes. Therefore, effectively modeling the relationship between the outcome variable Y and the sensitive attribute S becomes crucial. The prediction distribution over the sensitive attribute S is defined follows:
P = P( S = 0); Q= |S
(12)
5

---

Published as a conference paper at ICLR 2026
By substituting the distribution of predictions over the sensitive attribute into Equation (20), where p = P and = Q we can define the objective we aim to solve as follows:
min LBC + aDes (P, 2 + 1012
(13)
where LECE is the binary cross-entropy loss, which measures the classifier's accuracy. It is defined as:
M
LACP
1 M
Σ
Y log Yi
(14)
where Y; is the predicted output obtained from the training model parameterized by 0. This model can be a Multi-Layer Perceptron for tabular data or a ResNet for image data. Additionally, |@]l serves an L2 regularizer.
Computational complexity. Naively computing the CS divergence on all n training examples has (n?²) cost, since it involves pairwise interactions between samples. In practice, as standard for kemnel-based regularizers such as MMD and HSIC, we evaluate the CS-based fairness loss on mini-batches. Given a batch of size B (typically < n), the additional cost per optimization step is O(B²) and is implemented using vectorized matrix operations that reuse the same mini- batches as the prediction loss. Under typical batch sizes used in our experiments, this overhead modest and comparable to that of existing kernel- based fairness methods, making the CS regularizer computationally practical for both tabular and image models.
4.2
WHY IS THE CS DIVERGENCE MORE EFFECTIVE FOR ENSURING FAIRNESS?
The CS Divergence is particularly well-suited for promoting faimess due several key reasons:
(1) Closed-form solution for the mixture of Gaussians. The CS divergence has several advantageous properties, of which that provides closed-form solution for the mixture of Gaussions (Kampa et al., 2011). This particular property has facilitated its successful application in various tasks, including deep clustering (Trosten et al., 2021), disentangled representation learning (Tran al., 2022), and point-set registration (Sanchez Giraldo et al., 2017).
(2) CS Divergence has a tighter error bound than the KL divergence.
Proposition 4.2. For any d-variate Gaussian distributions p N(up. Ex) and q (( Eg), where En and Eg are positive definite, the following inequality holds:
Des(p: 9) DKL (p: 9) and Dcs(p: q) S DKL(9;p).
(15)
The proof can be found in Appendix B.3. It is important to note that the divergences are being compared under the same model parameter 0.
Remark on the Gaussian setting. Proposition 4.2 should be interpreted as a stylized comparison that carried out under Gaussian model solely for analytical convenience. The Gaussian assumption allows us to derive closed-form expressions for both the CS and KI. divergences and to make their relationship explicit. It is not required by our training procedure, and it is not assumed any in the empirical evaluation; in particular, the CS-based regularizer we optimize is defined for arbitrary distributions of predictions and sensitive attributes.
(3) CS divergence can provide tighter bounds than MMD and DP when the distributions are far apart or when the scale of the embeddings varies significantly. Based on Remark A. we know that CS divergence employs cosine distance, while MMD relies on Euclidean distance. In addition, DP Equation (5) utilizes mean disparity, which is Manhattan distance for the mean estimations of two distributions. CS divergence measures the angle between two distributions in the feature space, focusing on the difference direction rather than magnitude. cases where the distributions have significantly different variances or scales, MMD and DP may yield a large distance even if the distributions are aligned in the feature space. In contrast, CS divergence normalizes this comparison, resulting in more accurate measure of similarity and thereby providing tighter generalization bound. This normalization enhances the robustness of CS divergence, preventing MMD and DP from overestimating the discrepancy due to their reliance
unnormalized distance measure.

---

Published as a conference paper at ICLR 2026
Utility
Fairnsss
Methods
ACC (%)
t
AUC (%)
ADd (%6
AEO (%)
MLP
85.63
90.8220.23
116.5220.91
8. 8.43
DDP
82.42 0.
-3.75%
86.91 10.80
-4.31%
1.29
92.19%
20. 15:11.1
-139.03%
Gender
MMD3
81. 20.68
-4.36%
85.27 10.52
-6.11%
2.47
1.052
85.05%
17.53
-107.95%
HSIC
82.89
2.
-3.20%
$7.25.m 40.41
-3.93%
2.6644055
83,90%
18.47 #1.22
-119.10%
Adult
23
$1.81#0.64
-4.46%
85.38.40.22
-5.99%
0.714
95.70%
12.45
422388
-47.69%
CS
83.31
-2.71%
90. 90.15=0.49
-0.74%
2.42+
85.35%
2 2.27 11.04
73.07%
MLP
84.4220.31
90.15
13.47
9.25 86
DP
83.64 10.78
-0.92%
88.45
-1.89%
22.45
81.81
2.16
76.65%
Race
MMD
83. 03.12
-1.54%
00.06
-1.99%
2.58 2.58
80.85%
3.33
64.00%
HSIC
84. 44.99840.17
0.66%
90. 100.00
0.83%
7.90
41.35%
2. 11 11-1.18
77.19%
PR
82. 1.16
-2.71%
47..33
-3.01%
10.83
88.64%
0.86 0.86_0.60
90.70%
Cs
83.53
-1.05%
90.26
0.12%
2.16+0.61
83.96%
0, 10.12
95.24%
MLP
66.85
72.10
133.22
11.41 15.83
DP
64. 24.20
-3.96%
70.64 1.00
-2.02%
5.78 +0.33
56.28%
6. 6.78
40.58%
Gender
MMD
64.82 +1.02
-3.04%
70.72+0.92
-1.91%
3.094
76.63%
3. 45.3774.7
72.39%
HSIC
63. 17 23.46
-5.50%
71.17
-1.29%
+0.43
86.08%
2.60
77.21%
COMPAS
23
64.9 64.95
-2.84%
72. 20.77
0.03%
3.85+0.60
70.88%
3.91= 1.02
65.73%
CS
60.97
-3.89%
40.61
-0.79%
1.30
90.17%
0.44 10.13
96.14%
MLP
66..991105
12.46
17.24
19.44
D2
64. (4.9843.77
-3.00%
72. +.us
0.51%
8..70+1.12
49.54%
7.04
63.79%
Race
MMD
64..04
-3.85%
1.883
0.50%
4.42+
74.36%
5.60=1.25
71.19%
HSIC
64.52-22.0
-3.69%
16..64
0.41%
2.21 +0.68
87.18%
2.72
86.01%
PR
67. 20.00
0.34%
82.8620.87
-0.55%
1.10
67.52%
6.52
66.46%
cs
65.62 1.24
-2.05%
72.70
0.33%
1.79
89.62%
1. 1.48 11.84
92.39%
ML?
02
90.1640.11
10.26
2.13
DP
81
-0.88%
89.33
-0.92%
0..6
90.64%
5.37 20.32
-152.11%
Gender
MMI
20.93
-1.35%
88, 88.4411.71
-1.91%
2.45+0.66
76.12%
4.91 +1.4
-130.52%
HSIC
81.40=o.12
-0.78%
89. 80.1320.10
-0.70%
1.54:10.18
84.99%
4.95=2.39
-132,39%
ACS-I
P3
80.03
-2.45%
88. 10
-2.28%
0.35
96.59%
4.54
-113.15%
Cs
81.86 11.86
-0.22%
89.15
-1.12%
0.77
92.5%
0.90
57.75%
MLP
90.16
10.06
7.42
DP
81.25
81.25
0.02%
89.45
-0.79%
0.56 0..6
94.43%
4.53
38.95%
Race
MMD
80, 1..22
-1.24%
88, 08,42+1..3
1.63
-1.93%
1.45
85.59%
4. =..1
45.96%
HSIC
81. $1.4110.1
0.22%
89.67
-0.54%
1.04
89.66%
2.77 20.35
62.67%
PR
80.27 =0
-1.18%
88, 15.121
-1.90%
0.37
96.32%
1. 1.25 49
42.72%
CS
80. 00.78
-0.55%
89.14
-1.13%
0.81 1.0.8
91.95%
1.35 20.84
81.81%
Table 2: Fairness performance of existing fair models on the tabular datasets, considering race and gender as sensitive attributes. indicates accuracy improvement compared to MLP, with higher accuracy reflecting better performance, and J. denotes fairness improvement compared to MLP, where lower values indicate better fairness. Green values denote better than MLP on the corresponding metric (ACC/AUC 1: App/A (o RO.), while red denote worse. All results are based on 10 runs for each method. The best results for each metric and dataset are highlighted in bold text.
Distribution-free applicability. We also emphasize that the optimization framework of Section is distribution-free. The empirical CS divergence used as fairesss loss is implemented via kernel-based estimator that only depends on samples from the joint distribution of predictions and sensitive attributes and does not impose any parametric form on this distribution. Consequently, the same regularizer can be applied to both tabular and image models. Our experiments in Section 5, which span four tabular datasets and one image dataset with clearly non-Gaussian distributions, show that the CS-based regularizer consistently improves group-lairness metrics while maintaining competitive utility, supporting the practical relevance of the theoretical insight obtained from the Gaussian comparison in Proposition 4.2.
Choice of kernel. Throughout this work, we instantiate the empirical CS divergence with Gaussian (RBF) kernel and choose its bandwidth using the median heuristic, following common practice in kernel-based dependence measures such MMD and HSIC (Gretton et al., 2012; 2005). This choice is kept fixed across datasets and models to ensure a controlled comparison between regularizers. In principle, framework is not restricted to a particular kernel family: alternative kernels (e.g..
7

---

Published as a conference paper at ICLR 2026
DP
MMD
HSIC
PR
CS
Adult
COMPAS
ACS-I
14.0
9.0
4.0
Ape
6.0
2.0
3.0
0.0
0.0
76.0
79.0
82.0
85.0
58.0
61.0
64.0
67.0
77.0
79.0
81.0
Accuracy (%)
Accuracy (%)
Accuracy (%)
Figure 3: Faimess-accurag trade-off curves on the test sets for (left) Adult, t. (middle) COMPAS, and (bottom) ACS Ideally, results should be positioned in the bottom-right corner.
Laplacian or polynomial) can be plugged into the same estimator without changing the overall training objective algorithm (Schölkopf & Smola, 2002; Shawe-Taylor & Cristianini, 2004). A more systematic investigation of how different kernel families and bandwidth-selection strategies affect the utility-farreess trade-off is an interesting direction for future work.
EXPERIMENTS
In this section, evaluate the effectiveness of the CS fairness regularizer from several perspectives:
(I) utility and fairness performance, (2) the tradeoff between utility and fairness, (3) prediction distributions across different sensitive groups, (4) T-SNE plots for these sensitive groups, and (5)
the sensitivity of parameters in Equation (13). Our evaluation encompasses five datasets with diverse sensitive attributes, including four tabular datasets: Adu 1 COMPAS, ACS-I, and ACS-T, as well as one image dataset, Ce CbAAA Utility performance is assessed based on accuracy and the area under the curve (AUC), while fairness performance is measured using DP Equation (2) and A go Equation (3). Detailed information about the datasets and baselines can be found in the Appendices and We denote an observation drawn from the results as Obs..
5.1 FAIRNESS AND UTILITY PERFORMANCE
We conducted experiments on five datasets along with their corresponding baselines, previously mentioned. For each dataset, we performed 10 different splits to ensure robustness in our results. We calculated the mean and standard deviation for each metric across these splits. The accuracy and fairness performance of the downstream tasks is in Table 2. Our observations are as follows:
Obs. 1: CS achieves the hest near-best Ag most datasets and is competitive Apr, while maintaining high utility. Notably, CS demonstrates exceptional fairness performance on the image dataset, Ce leleeb- Ap reduction of 99.7730 97.36% and a Æg reduction of 98.58%. Furthermore, in the Adult t and ACS I datasets, which include gender groups, traditional methods such DP, MMD, HSIC C, and do effectively optimize for EO faimess. In contrast, the proposed CS achieves significant reductions in go by 72.12% and 63.85%, respectively, compared to MLP.
where the disparity in the 'Young' and 'Non- 'Non-Young" groups sees
Obs. 2: CS achieves good fairness performance with a small sacrifice in utility, Specifically, CS exhibits a decrease of less than 3.1% in accuracy and less than 2.2% AUC. The only exception observed with COMPAS AS when gender is treated as a sensitive attribute, resulting in slightly higher accuracy loss of 3.6%. Notably, CS demonstrates either equivalent or improved AUC performance, with increases of 0.02% and 0.58% Adult for the gender and race groups, respectively, as a 0.35% increase on COMP AS for the race group. Among the baselines, HSIC ranks highest in utility, achieving the best performance on ACS-I for the race group and ACS-T -T for both the gender and race groups. This is followed by PR, which shows the best utility on COMPAS for both the gender and race groups, well as Ce ebA-A A for the gender group.
well
8

---

Published as a conference paper at ICLR 2026
DP
MMD
ISIC
PR
Gender
Seea, Alt
Seaa, Docads
Alte
Prodiction
Preduction
Prediction
Preducion
Prrdiction
Aew 20
Groder
Semss Altr.
Sees, Alte
Seats Atto
Sees, AIr
Hale
Mae
Nat
2.3
0.0
0.0
Prediction
0.0
Prediction
0.0
Prodictio
Prodictien
3.0
0.0
Prediction
0.4
Figure 4: Prediction distributions for female and male groups in the Adult dataset. The top row shows kernel density estimates of the raw predictions Y for all target labels, grouped by gender, while the bottom row shows the prediction densities for the positive class, Y 1, for the two gender groups. Each column corresponds to a different fairness regularizer. A larger overlap between the blue and red curves indicates better group faireess, and the reported values above each panel give the corresponding gaps in APP (top row) and Ago (bottom row).
5.2 How Do ACCURACY AND FAIRNESS TRADE-OFF IN BASELINE MODELS AND CS?
We evaluate the trade-off between accuracy and Apr for the baselines by varying the fairness hyperparameters (Yao et al., 2023; Deka & Sutherland, 2023). The results are presented in Figure 3, where the x-axis represents the target accuracy, while the y-axis shows the average Demographic Parity (DP) across both positive and negative target classes. It is important to note that the figure in the bottom right corner represents the optimal result.
Obs. 3: At the same utility level, CS is the most effective method in promoting fairness. Analyzing the results, find that CS consistently achieves the lowest Apr across most accuracy levels, with this effect becoming more pronounced at higher accuracy levels. This is evidenced by the significant gap in ADP between CS and other baselines. It is important note that while all baselines can demonstrate good fairness when the optimization prioritizes fairness over task objectives, the task objective remains critical for the practical application of these models. Obs. 4: High accuracy can sometimes lead to worse fairness compared to MLP, as the fairness objective becomes more challenging to optimize when there is a stronger focus on task-speeific objectives. As shown in Table 2, the DP for MMD is over 14.0, which is greater than the average AP of 13.22 for MI.. However, these fairness regularizers generally prove effective controlling bias in representations, especially when more emphasis is placed on the task-specific objective. Notably, some datasets with particularly sensitive attributes pose greater challenges for achieving fairness. For instance, the COMP AS dataset, which includes gender as a sensitive attribute, demonstrates this difficulty. One possible explanation is the relatively small sample size of COMPAS, which contains only 6, 172 samples, significantly fewer than other datasets where faimess is easier to achieve. Obs. 5: CS displays significant increase in DP at a slower rate than other baselines as accuracy increases. We analyze the slope of the lines representing the increase in App with rising accuracy. Many methods, such as PR and DP, demonstrate strong fairness performance at low accuracy levels; however, they quickly lose control over fairness as accuracy begins to increase. This is evident from the abrupt rise in 4 pr observed at around 82.0% on Adult, 63.0% on COMP AS, and 81.0% on ACS-I. I. In contrast, CS only exhibits a sudden increase at 85.0%, 65.5%, and 81.5% for the same datasets, respectively.
5.3 How CAN THE CS FAIRNESS REGULARIZER PERFORM WELL ON BOTH DP AND EO?
We visualize the kernel density estimate plot? of the predictions y across different sensitive groups to analyze how CS achieves better balance of various faimess definitions compared to other baselines. The first row displays the predictions for all target classes, specifically Y and Y = 1, grouped by sensitive attributes. In this row, the blue areas represent the prediction density for S 0. while the red areas indicate the prediction density for S = 1. The second row illustrates the prediction
Phttps:/ htts//ebbbrrrro. httppps///eaboompppttattttaeeee httppppe///eooooo.ppttteeeeeeeee wwtpeeeeeeeeeeeeeeeeeeeeeeeeeeee kdeplot html

---

Published as a conference paper at ICLR 2026
density for the positive target class, Y = 1, across two different sensitive groups. Figure 4 presents the results for Adult based on gender and race groups, with additional results for other datasets available in Appendix D.2.
Obs. 6: Cs effectively optimizes the prediction distributions for the two sensitive groups, specifically 81S- = O and YIS = Additionally, it optimizes the prediction distributions for these groups within the positive target group, ie., Y|S = O,Y = 1 and YIS = L. = 1. Achieving DP and EO fairness requires different objectives. For instance, DP directly optimizes the Apr, which results in reduced effectiveness for achieving EO fairness. This is evident across all datasets, as DP ranks among the worst, achieving 7/10 of the lowest EO fairness scores on Ægo when tested on five datasets with two types of sensitive attributes. The distribution plots for DP further illustrate this, showing a generally larger gap between the two sensitive groups in the EO plots compared to other methods. In contrast, CS consistently minimizes the prediction density gap between the two sensitive groups
Accuracy
Āgr
5.4 PARAMETER SENSITIVITY ANALYSIS
3.001
$45
$4.4
542
322
124
124
16.0
000
12
0.5
46
2.4
0.01
664
572
535
811
6:2
5111
0.0
76
6.2
63
54
52
2.4
71
For all models, we tune the hyperparameters us- ing cross- cooss-alliattion on the training set. The hy- perparameters for these variants are determined through grid search during cross-validation. Specifically, we vary the parameters a and & in Equation (13) across the ranges (le 6, 150) and (le - 3, 10), respectively. In this experi- ment, specifically visualize the values of a in the range (1e 4, le 1) for CS.
3.1
44
611
537
815
813
514
0.1
73
42
61
52
51
43
05
457
45
K14
817
813
92.6
0.5
54
54
54
40
15
66
R22
32
624
021
903
56
48
51
23
14
52
623
928
112
01-
55
59
912
91.
80.1
745 732
52
25
85
1o.4
5c4
1o.3
5e
5-2
le1
1o.4
1o-3
Se
16 5e 1p-1
Figure 5: Parameter sensitivity of the CS regu- larizer ADULT: heatmaps show test accuracy (left) and App (right) as the faimess weight a and l2 weight B vary over the cross-validated ranges. Overall, CS exhibits a smooth urility-fairness trade-off, remaining stable over broad range of I and becoming noticeably more sensitive only when a is very large.
The heatmap in Figure 5 illustrates the accuracy and Ar across various combinations of a and values for the Adult. In the accuracy plots, darker colors indicate higher values, which are preferable, while lighter colors in the DP plots represent better fairness performance.
Obs. 7: The highest accuracy is achieved when is set to its smallest value, le 4, while the best fairness is obtained with a 5e 2. Notably, fairness drops significantly when increases from 5e 2 to 1. Generally, smaller values of a can still yield satisfactory fairness performance when paired with an appropriate range of 3, specifically around 5 10. Obs. 8: The fairness performance more sensitive to changes in a than in 3. For instance, adjusting 3 from le to 10, which represents 10, x00 increase, results in only slight decrease in DP from 7.2 to 4.2. In contrast, increasing a from le - 2 to 5e 2, a 5x change, leads to a significant drop in ADP from 6.7 to 2.8, when keeping 3 fixed at le 3.
6 CONCLUSION
In this paper, we introduce novel fair machine learning method called the Cauchy-Schwarz (CS) fairness regularizer. Empirically, our approach achieves a more consistent dulity-fairness trade-off across hyperparameter settings than standard regularizers, and yields more generalizable fairness by minimizing the Cauchy-Schwarz divergence between the prediction distribution and the sensitive attributes. We demonstrate that the CS divergence provides tighter bound compared both the Kullback-Leibler divergence and the Maximum Mean Discrepancy, as well as the mean disparity used in Demographic Parity regularization. This superiority is particularly evident when the distributions are significantly different or when there is substantial variation in the scale of the embeddings. As a result, our CS fairness regularizer delivers improved fairness performance in practical scenarios. While our work currently only evaluates on general machine learning tasks, and thus leaves future work to other tasks such as graph learning.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported in part by Nthe DARPA Young Faculty Award, the National Science Foun- dation (NSF) under Grants #2431561, #2127780, #2319198, #2321840, #2312517, and #2235472. the Semiconductor Research Corporation (SRC), the Office of Naval Research through the Young Investigator Program Award, Grants EN00014-21-15 1-2-2-2-22 and #N00014-22 1--22-1-2067. Army Research Office Grant #W91 NNF2410360, and the National Defense Science & Engineering Graduate (ND- SEG) Fellowship Program. Additionally, support was provided by the Air Force Office of Scientific Research under Award #FA9550- 22-1-0253, along with generous gifts from Xilinx and Cisco.
REFERENCES
Tameem Adel, Isabel Valera, Zoubin Ghahramani, and Adrian Weller. One-network adversarial fairness. In Proceedings of the AAAI, pp. 2412-2420, 2019.
Chirag Agarwal, Himabindu Lakkaraju, and Marinka Zitnik. Towards a unified framework for fair and stable graph representation learning. UAI 2021: Uncertainty in Artificial Intelligence 2021.
Wael Alghamdi, Hsiang Hsu, Haewon Jeong, Hao Wang, Peter Michalak, Shahab Asoodeh, and Flavio Calmon. Beyond adult and compas: Fair multi-class prediction via information projection. Advances in Neural Information Processing Systems, 35:38747-38760, 2022.
Sina Baharlouei, Maher Nouiehed, Ahmad Beirami, and Meisam Razaviyayn. Rényi fair inference.
In International Conference on Learning Representations, 2020. URL https: /oweeeveeeeeeeeeeeeeeeeeeeeeeeee net ftt:/forummtmm d=Hk
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness in machine learning. NeurIPS tutorial, 1:2017, 2017.
Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H Chi. Data decisions and theoretical implications when adversarially learning fair representations. arXiv preprint arXiv:1707.00075 2017.
Steinar Bøyum. Fairness in education-a normative analysis of oecd policy documents. Journal of Education Policy, (9(6))856-877) 2014.
Maarten Buyl and Tijl De Bie. Optimal transport of classifiers to fairness. In Advances in Neural Information Processing Systems, 2022.
Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney. Optimized pre-processing for discrimination prevention. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017a. URL https: //proceedings neurips cc ct/ccc///////eee /ppperrpr__. ff /ttt:///pee/22////11le//////// 2tt.:0/07//111e//4942220020 a483fae4be7e344
.df..
Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush Varshney. Optimized pre-processing for discrimination prevention. NeurlPS, 30, 2017b.
Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, ((2):153-163 2017.
Ching-Yao Yao Chuang and Youssef Mrouch. Fair mixup: Fairness via interpolation. ICLR, 2020.
Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
Enyan Dai and Suhang Wang. Say no to the discrimination: Learning fair graph neural networks with limited sensitive attribute information. In Proceedings of the 14th ACM WSDM, pp. 680-688, 2021.
Namrata Deka and Danica J Sutherland. Mmi-b-fair: Learning fair representations with statistical testing. In International Conference on Artificial Intelligence and Statistics, pp. 9564-9576 PMLR, 2023.
11

---

Published as a conference paper at ICLR 2026
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. NeurIPS, 2021.
Yushun Dong, Ninghao Liu, Brian Jalaian, and Jundong Li. Edits: Modeling and mitigating data bias graph neural networks. Proceedings of the ACM web conference 2022, pp. 1259-1269, 2022.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL httttttt///chhtt ics. www...e.uuee................eeee
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Conference on Innovations in Theoretical Computer Science (ITCS), 2012.
Harrison Edwards and Amos Storkey. Censoring representations with an adversary. arXiv preprint arXiv:1511.05897 2015.
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubra- manian. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 259-268, 2015.
Hans Gebelein. Das statistische problem der korrelation als variations-und eigenwertproblem und sein zusammenhang mit der ausgleichsrechnung. ZAMM- JouMnJlurnal of Applied Mathematics and Mechanics/Zeitschrift für Angewandte Mathematik und Mechanik, 21(6)3364-377 1941.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In International conference on algorithmic learning theory, pp. 63 77. Springer, 2005.
Arthur Gretton, Kenji Fukumizu, Choon Teo, Le Song, Bernhard Schölkopf, and Alex Smola. A kernel statistical test of independence. Advances in neural information processing systems, 20, 2007.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(1):723-773, 2012.
Thomas Grote and Geoff Keeling, Enabling faimess in healthcare through machine learning. Ethics and Information Technology, 24(3):39, 2022.
Xiaotian Han, Zhimeng Jiang, Hongye Jin, Zirui Liu, Na Zou, Qifan Wang, and Xia Hu. Retiring Si Si,delta (DP)S New distribution-leve metrics for demographie parity. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: hatle::///pppeeeeee wttp///peeee.eeeeeeeeee orum?id-Ljjee IWWVVa.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning, Advances in neural information processing systems, 29, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, June 2016.
Hermann Hirschfeld, connection between correlation and contingency. In Mathematical pro- ceedings of the cambridge philosophical society, volume 31, pp. 520-524. Cambridge University Press, 1935.
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
Lily Hu and Yiling Chen. A short-term intervention for long-term faimess in the labor market. In WwW, pp. 1389-1398, 2018.
Robert Jenssen, Jose C Principe, Deniz Erdogmus, and Torbjørn Eltoft. The cauchy-schwarz divergence and parzen windowing: Connections to graph theory and mercer kernels. Journal of the Franklin Institute, 343(6):614-29, 2006.
Ray Jiang, Aldo Pacchiano, Tom Stepleton. Heinrich Jiang, and Silvia Chiappa. Wasserstein fair classification. In Uncertainty in Artificial Intelligence, pp. 862-872. PMLR, 2020.
12

---

Published as a conference paper at ICLR 2026
Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrimi- nation. Knowledge and information systems, 33(1):1-33, 2012.
Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairness-aware classifier with prejudice remover regularizer. In ECML PKDD. Springer, 2012.
Kittipat Kampa, Erion Hasanbelliu, and Jose C Principe. Closed-form cauchy-schwarz pdf divergence for mixture of gaussians. In IJCNN, pp. 2578-2585. IEEE, 2011.
Jian Kang, Jingrui He, Ross Maciejewski, and Hanghang Tong. Inform: Individual faimess on graph mining. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 379-389, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
René F Kizilcec and Hansol Lee. Algorithmic fairness in education. In The ethics of artificial intelligence in education, pp. 174-202. Routledge, 2022.
Jon Kleinberg. Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determi- nation of risk scores. arXiv preprint arXiv: 160.0660790007,,. 2016.
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual faimess. NeurIPS, 30, 2017.
Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. Propublica compas analysis data and analysis for 'machine bias'. https: / /g1 thub. 2016. Accessed: 2023-03-13.
wwwpchwww..cc...eccooooo..eeeeet www.cooo////oooo propubb i / compas s- ana 1 ys issss
Tai Le Quy, Arjun Roy, Vasileios Iosifidis, Wenbin Zhang, and Eirini Ntoutsi. A survey on datasets for fairness-aware machine learning. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 12(3):e452, 2022.
Ana Alves Leal. Algorithms, creditworthiness, and lending decisions. International Conference on Autonomous Systems and the Law, pp. 321-353. Springer, 2022.
Peizhao Li, Yifei Wang, Han Zhao, Pengyu Hong, and Hongfu Liu. On dyadic fairness: Exploring and mitigating bias in graph connections. In International Conference on Learning Representations, 2020.
Zhu Li, Adrian Perez-Suay, Gustau Camps-Vall and Dino Sejdinovic. Kernel dependence regularizers and gaussian processes with applications to algorithmic fairness. arXiv preprint arXiv:1911.004 2019.
Hongyi Ling, Zhimeng Jiang, Youzhi Luo, Shuiwang Ji, and Na Zou. Learning fair graph representa- tions via automated data augmentations. In ICLR, 2023.
Yezi Liu. Fairgraph: Automated graph debiasing with gradient matching. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 4135-4133 2023.
Yezi Liu and Yanning Shen. Enabling group fairness in machine unlearning via distribution correction. In CIKM, pp. 1925-1935, 2025.
Yezi Liu, Hanning Chen, and Mohsen Imani. Promoting fairness in link prediction with graph enhancement. Frontiers in Big Data, 7: 7:1489306, 2024.
Yezi Liu, Hanning Chen, Wenjun Huang, Yang Ni, and Mohsen Imani. Lune: Efficient Ilm unlearning via lora fine-tuning with negative examples. arXiv preprint arXiv:2512.07375, 2025a.
Yezi Liu, Hanning Chen, Wenjun Huang, Yang Ni, and Mohsen Imani. secover-to-foreet Gradient reconstruction from lora for efficient IIm unlearning. arXiv preprint arXiv:25/2.07374, 2025h..
13

---

Published as a conference paper at ICLR 2026
Yezi Liu, William Youngwoo Chung, Hanning Chen, Calvin Yeung, and Mohsen Imani Are hypervec- tors enough? single-call Ilm reasoning over knowledge graphs. arXiv preprint arXi::2512.09369 2025c.
Yezi Liu, Wenjun Huang. Yang Ni, Hanning Chen, and Mohsen Imani. White admitted by stanford, black got rejections: Exploring racial stereotypes in text-to-image generation from a college admissions lens. In Companion Proceedings of the ACM Web Conference 2025, pp. 1138-1142, 2025d.
Yezi Liu, Prathyush Poduval, Wenjun Huang, Yang Ni, Hanning Chen, and Mohsen Imani. Enabling group faimesss in graph unlearning via bi-level debiasing, arXiv preprint arXiv:2505.09702, 2025e.
Yezi Liu, Hanning Chen, Yanning Shen, and Mohsen Imani. Debias once for all: A data-centric strategy for fair machine learning. In Proceedings of the Nineteenth ACM International Conference on Web Search and Data Mining, pp. 447 457, 2026.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard S Zemel. The variational fair autoencoder. International Conference on Learning Representations (ICLR), 2016.
Gilles Louppe, Michael Kagan, and Kyle Cranmer. Learning to pivot with adversarial networks. NeurlPS, 30, 2017.
David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. International Conference on Machine Learning, 2018.
Ninarch Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6)11333 2021.
Anay Mehrotra and Nisheeth Vishnoi. Fair ranking with noisy protected attributes. Advances in Neural Information Processing Systems, 35:31711-31722 2022.
Debarghya Mukherjee, Mikhail Yurochkin, Moulinath Banerjee, and Yuekai Sun. Two simple ways to learn individual faimess metrics from data. In ICML, pp. 7097-7107. PMLR, 2020.
Debarghya Mukherjee, Felix Petersen, Mikhail Yurochkin, and Yuekai Sun. Domain adaptation meets individual fairness. and they get along. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https:/ / opwnreview net forum? inmeeidddie XSNEXG9HBAu.
Emanuel Parzen. On estimation of a probability density function and mode. The annals of mathemat- ical statistics, 3363):006 1962.
JC Principe. Dongxin Xu, Fisher, and S Haykin. Information theoretic learning. unsupervised adaptive filtering. Unsupervised Adapt Filter, 1, 2000.
Alfréd Rényi. On measures of dependence. Acta mathematica hungarica, 10(3-4):441-44 1959.
Luis G Sanchez Giraldo, Erion Hasanbelliu, Murali Rao, and Jose C Principe. Group-wise point-set registration based on renyi's second order entropy. In CVPR, pp. 6693-6701, 2017.
Bernhard Schölkopf and Alexander Smola. Learning with kernels: support vector machines, regularization, oplimization, and beyond. MIT press, 2002.
John Shawe-Taylor and Nello Cristianini. Kernel methods for pattern analysis. Cambridge university press, 2004.
Changjian Shui, Gezheng Xu, Qi Chen, Jiaqi Li, Charles X Ling, Tal Arbel, Boyu Wang, and Christian Gagné. On learning fairness and accuracy on multiple subgroups. Advances in Neural Information Processing Systems, 35:34121-3413: 2022.
Gábor Szěkely, Maria L Rizzo, and Nail K Bakirov. Measuring and testing dependence by correlation of distances. 2007.
14

---

Published as a conference paper at ICLR 2026
Linh Tran, Maja Pantic, and Mare Peter Deisenroth. Cauchy-schwarz schwarz regularized autoencoder. Journal of Machine Learning Research, 23(115)1133,, 2022.
Daniel J Trosten, Sigurd Lokse, Robert Jenssen, and Michael Kampffmeyer. Reconsidering represen- tation alignment for multi-view clustering. In CVPR, pp. 1255-1265, 2021.
Ioannis Tsaousis and Mohammed H Alghamdi. Examining academic performance across gender differently: Measurement invariance and latent mean differences using bias-corrected bootstrap confidence intervals. Frontiers in Psychology, 13:896638, 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911),, 2008.
Sahil Verma and Julia Rubin. Fairness definitions explained. In 2018 ieee/acm international workshop on software fairness (fairware), pp. 1-7. IEEE, 2018.
Mingyang Wan, Daochen Zha, Ninghao Liu, and Na Zou. In-processing modeling techniques for machine learning fairness: A survey, ACM Transactions on Knowledge Discovery from Data, 17 (3):1-27, 2023.
Xinyu Wang, Linrui Ma, Jerry Huang, Peng Lu, Prasanna Parthasarathi, Xiao-Wer Chang, Boxing Chen, and Yufei Cui. Resona: Improving context copying in linear recurrence models with retrieval. arXiv preprint arXiv:2503.22 229113, 2025.
Xinyu Wang, Mingze Li, Peng Lu, Xiao-Wen Chang, Lifeng Shang. Jinping Li, Fei Mi, prasanna Parthasarathi, and Yufei Cui. Infmem: Learning system-2 memory control for long-context agent. arXiv preprint ar arXiv:2602.02704, 2026a.
Xinyu Wang, Sicheng Lyu, Yu Gu, Jerry Huang, Peng Lu, Yufei Cui, and Xiao-Wen Chang. Beyond hard writes and rigid preservation: Soft recursive least-squares for lifelong IIm editing. arXiv preprint arXiv:2601. 15686, 2026b.
Gelei Xu, Xueyang Li, Yixiong Chen, Yuying Duan, Shuqing Wu, Alexander Yu, Ching-Hao Chiu, Juntong Ni, Ningzhi Tang, Toby Jia-Jun Li, et al. A comprehensive survey of agentic ai in healthcare. Authorea Preprints, 2025.
Yao Yao, Qihang Lin, and Tianbao Yang. Stochastic methods for auc optimization subject to auc- based fairness constraints. In International Conference on Artificial Intelligence and Statistics, pp. 10324-10342. PMLR, 2023.
Wenzhe Yin, Shujian Yu, Yicong Lin, Jie Liu, Jan-Jakob Sonke, and Efstratios Gavves. Domain adaptation with cauchy-schward divergence. arXiv preprint arXiv:2405. 19978, 2024.
Shujian Yu, Hongming Li. Sigurd Løkse, Robert Jenssen, and José C Principe. The conditional cauchy-schwarz divergence with applications to time-series data and sequential decision making. arXiv preprint (=XXi:2301.08977 2023.
Shujian Yu, Xi Yu, Sigurd Løkse, Rohert Jenssen, and Jose C Principe. Cauchy-schwarz divergence information bottleneck for regression. arXiv preprint arXiv:2404.19551, 2024.
Shujian Yu, Hongming Li. Sigurd Løkse, Robert Jenssen, and José C Principe. The conditional cauchy-schwarz divergence with applications to time-series data and sequential decision making. IEEE Transactions Pattern Analysis and Machine Intelligence, 2025.
Mikhail Yurochkin and Yuekai Sun. Sensei: Sensitive set invariance for enforcing individual fairness.
arXiv preprint arXiv:2006. /4/68, 2020.
Mikhail Yurochkin, Amanda Bower, and Yuekai Sun. Training individually fair ml models with sensitive subspace robustness. arXiv preprint arXiv:1907.00020 2019.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P. Gummadi. Fair- ness constraints: Mechanisms for fair classification. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2017.
15

---

Published as a conference paper at ICLR 2026
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International conference on machine learning, pp. 325-333. PMLR, 2013.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In AAAI/ACM Conference on AI, Ethics, and Society (ATES), 2018.
Guanhua Zhang, Yihua Zhang, Yang Zhang, Wenqi Fan, Qing Li, Sijia Liu, and Shiyu Chang. Fairness reprogramming. arXiv preprint 2rXiv:2209.00222, 2022.
Ji Zhao and Deyu Meng. Fastmmd: Ensemble of circular discrepancy for efficient two-sample test. Neural computation, 27(6):1345-133 2015.
Aoqi Zuo, Susan Wei, Tongliang Liu, Bo Han, Kun Zhang, and Mingming Gong. Counterfactual fairness with partially known causal graph. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: hppps///peerevieee net/ / forum? idd 9aLbnt Hz 10q.
16

---

Published as a conference paper at ICLR 2026
A WHAT Is THE RELATIONSHIP BETWEEN CS DIVERGENCE AND EXISTING DISTRIBUTION DISTANCE MEASURES?
To illustrate the advantages of the CS fairness regularizer, we begin by summarizing the com- monly used distribution distance metrics: Maximum Mcan Discrepancy (MMD), Kullback-Leibler divergence (KI.), and Hilbert-Schmide Independence Criterion (HSIC).
Demographic Parity Regularizer. The demographic parity regularizer is widely utilized in fairesss- focused machine learning studies (Chuang & Mroueh, 2020). It aims to optimize the mean disparity between two prediction distributions. This regularizer can be formally expressed as:
1 N2
DP(p: q) = Ni Σ (xx)
N2 Σ g(,)))+
(16)
where x; and x, are data points from S = and S = 1, in the context of fairness. In the following, we represent x, with distribution p and x, with distribution q as x. and x for simplicity. However, only optimizing on the mean disparity of two distributions cannot always generate an optimized DP or EO, as the Equation (16) equals is necessary but not sufficient condition for achieving DP and EO.
Mean Maximum Discrepancy. One of the most widely used distance metrics is the Mean Maximum Discrepancy (MMD) (Gretton et al., 2012). In the context of faimesss, previous studies have employed MMD as a regularizer to enforce statistical parity among the embeddings of different sensitive groups within machine learning model (Deka & Sutherland, 2023; Louizos et al., 2016). This approach aims to facilitate fair representation learning.
N₂ R
1 N?
MMD (p:9)=
Σ 1
xx
+
x(x,.x] x3)
N2
(17)
N2
N₂
2
k(x², xf).
N
By comparing with Equation (20), we observe that the CS divergence introduces logarithmic term for each component of the MMD. Through simple transformations, we can deduce the following: Remark A.1. CS divergence measures the cosine distance between empirical mean embedding in Reproducing Kernel Hilbert Space, while
L Hp E f(x²) and M4 N.
-i=1
MMD utilizes Fuclidean distance.
Kullback-Leibler Divergence. Kullback-Leible (KL) Divergence is a key concept in information bottleneck theory, where it is used to quantify the mutual information between two probability distributions. This metric has gained popularity across various domains, including fair machine learning (Kamishima et al., 2012).
p(x) g(x)
DKL = p(x) log
(18)
Hilbert-Schmidt Independence Criterion (HSIC). Let K and L denote the Gram matrices for the variables and y, respectively. Specifically, K is defined such that Kij =(xi, x5), and L is defined
Laj <(y:: y5), where is the Gaussian kernel function given by K. exp 1-|2 The Hilbert-
Schmidt Independence Criterion (HSIC) can be estimated using the following expression (Gretton al., 2007):
HSIC(p: q) =
KiyLi
Σ KijLor
N
N
$.....
(19)
2
Σ
KijLin
tr(KHLH).
N
NZ
$,1.9
17

---

Published as a conference paper at ICLR 2026
where H I N 11" represents centering matrix of size N N. In this expression, is the identity matrix, is vector of ones, and N 11 computes the average across the columns, effectively centering the data by subtracting the mean from each entry.
Compared Equation (17), The HSIC can be interpreted the MMD between the joint distribution p(x,y) y) and the product of their marginal distributions p(x)p(y).
B DETAILS ON THE RELATION OF CS AND EXISTING FAIRNESS REGULARIZERS
B. PROOF PROPOSITION 4.1
Proposition 1. Given two sets of observations {x N1 and {xq}}ı 1395 let and denote the distributions of two groups. The empirical estimator of the CS divergence Des (pr q) is given by:
N1
N2 Σ
D) = =log
k
<(x,,
x²)
+ log
K (xx] x3)
N7
N₂
1,==1
s,j=1
(20)
N1
N2
2og
n(x², x5)
N1 im
1=1
Proof. The CS divergence is defined as:
(ʃ (x)) dx)²
Des(pig) -llog
[ (p(x)2dx dx g(xx)2 dx
p(xx)²
(21)
where p(x) 3 En Ko (x - x²) and x(x) -tr Eim (x - xf) are kernel density estimation.
Then we can obtain:
1 M M
/ (x) =
k V2n (xn -x9).
(22)
M
By similar approach,
/ (1()) dx
1
/20 (x] x9),
(23)
N22
i=1 j=1
and
M N
/ f(x)((x)dx
Σ
k V20 (x - x9).
(24)
MN
Substituting (Equation (22))-EEqnatio (24)) into Eq. (21), we obtain:
M
Dcs(p; = log
1
k /20 (x" - xx9)
+ log
Σ
k V20 (x - x3))
M
N2
i,=1
i===1
(25)
M Σ
N
log
K 1 -x9)
MN
Y=1 1=1
B.2 PROOF OF REMARK A.1
Remark CS divergence measures the cosine distance between Ht, and fyg in Reproducing Kernel Hilbert Space, while MMD utilizes Euclidean distance.
18

---

Published as a conference paper at ICLR 2026
Proof. Let H. be Reproducing Kernel Hilbert Space (RKHS) associated with kernel x(x,,x x)) (f(x"), fxx)) (Yu al., 2024). The mean embeddings of two distributions p and q in H. denoted by Mp N Σm f(xx") and My No شه f(x3) in H. respectively. The CS divergence defined by Equation (20) can thus be written as:
(pyps May) -2 log Dcos( 14g
Des(p: 9) -o
|
Here, () denotes the inner product in the RKHS, and - || represents the norm induced by the inner product. The mean embeddings μ4 and Ma are elements H. Thus, the CS divergence is computed based on the cosine distance Dcos between Mp and p..
Similarly, the Maximum Mean Discrepancy (MMD) between distributions and q defined in Equa- tion (17) can be written as:
MMD g) = -11pp- - -,yll3 (pu Hg).
Thus, the MMD measures the Euclidean distance between the mean embeddings of and in the RKHS H. i.e., the up and 1lg.
B.3 PROOF PROPOSITION 4.2
Proposition 2. For any d-variate Gaussian distributions N(pp: Enp) and N( Eng) with positive definite Mp and Ex. the following inequality holds:
Des(p; Dr.. (p: q) and Dcs(p: DKI(ggp).
(26)
Proof. The KL divergence for p and q is given by:
1 Drl(p:g) = ( dd (H4 Hp) pp)) 12g (mo Mp) + log -pp)
Egg
(27)
Expl
The CS divergence is expressed as (Kampa et al., 2011):
1 Dcs(p: = log(day) + log(dzz) log(dyo),
(28)
2
where:
(29)
exp (- (14 M4g (gggT( (Ep Eg)
- (Mp M₂))
dpg
(30)
((2T)d dd Ep + Egg
dpp
(22)d dgg = (22T)dd222.
(31)
We simplify:
1 Des(p: [(m, Mp) (Ep + En)- Mp) log
E + Egl
(32)
2dd / |
When the mean vectors differ, based on the property (Horn & Johnson, 2012), En - -EEp + Egg)
positive semi-definite given Exp = Exg, we have:
2(Dcs(p: 9) - (p)
(My - -Mp) (Ep + Eng 1 11g
T())
(Hg pp))
(33)
-(u, mp)) TT (M, - Up)) S 0.
19

---

Published as a conference paper at ICLR 2026
When the covariance matrices differ, let I be the d-dimensional identity matrix (Yin al., 2024):
Ep + Egg
2(Dcs(1:1) (PI g)) = log
2a V 12 120
[g -
-log
tr (xp) +
(34)
=d 2 + log (12g 21p 1|)
log (|Eg Epl) Ixpl
tr 1 --1 Ep) + d.
2
We have 1 I (2 ( and 1ل 11 s (1+ 1/2 xx))
Thus, based
on Equation (34), we can obtain:
2(Dcs(p: (KL(p)))
< - dog 2 + dlog (1 _ σ ir( 1r(27'
(35)
+ log (2 x tr tr(2 Ep) + d.
The combined Equation (33) and Equation (35), we can obtain:
2(Dcs( () (p: g)) 00
(36)
Similarly, we can obtain 2(Des(pp)) DKL p)) - In conclusion, we conclude:
Des(1;4) S (2I,(z5)) and Des(p: q) S (xI.(g:).
(37)
L
C RELATED WORK
In this section, we first review relevant prior studies, beginning with an overview of algorithmic fairness in machine learning. We then narrow our focus regularization--aael in-prccessing methods, which are central to our approach.
C.I ALGORITIMIC FAIRNESS IN MACHINE LEARNING
The importance of fairness in machine learning has grown significantly as the demand for unbiased decision-making models for individuals and groups increases (Wan et al., 2023; Le Quy et al., 2022: Liu et al., 2025d). This especially critical in high-stakes applications where the consequences of biased decisions be severe. Fairness is commonly categorized into three main types: Individual fairness (Yurochkin et al., 2019; Mukherjee et al., 2020; Yurochkin & Sun, 2020; Kang et al., 2020; Mukherjee al., 2022), which aims ensure that similar individuals fairness (Hardt et al., 6016; Verma & Rubin, 2018; Li et al., 2020; Ling et al., 2023), which focuses on achieving fairness across predefined subgroups, often defined by sensitive attributes such as gender or race; Counterfactual fairness (Kusner et al., 2017; Agarwal et al., 2021; Zuo et al., 2022), which seeks ensure fairness by considering how decisions would hold under alternative scenarios. Given the widespread adoption of group faimess metrics in real-world applications and the increasing development in ir-precessing techniques for deep neural network models (Liu, 2023; Liu & Shen, 2025; Liu et al., 2025e; 2024), we focus on benchmarking these methods to ensure group fairness in neural networks, particularly for tabular and image data.
treated similarly; Group
Various techniques for mitigating bias in machine learning models can be categorized into three main approaches: pre-processing, in-processing, and post-processing. Pre-processing methods focus on addressing biases present in the dataset itself ensure that the trained model exhibits fairness (Kamiran & Calders, 2012; Calmon et al., 2017a). For instance, these techniques may involve rebalancing the dataset modifying the data collection process (Calmon al., 2017b). In-processing methods, on the other hand, adjust the training objectives by incorporating fairness constraints directly into the learning process (Kamishima al., 2012; Zhang al., 2018; Madras
20

---

Published as a conference paper at ICLR 2026
et al., 2018; Zhang et al., 2022; Buyl & De Bie, 2022; Alghamdi et al., 2022; Shui et al., 2022; Mehrotra & Vishnoi, 2022). This approach aims to ensure that the model learns fair representations during training. Finally, post-poreeee protessnnsssnng methods modify the predictions made by classifiers after the model has been trained, with the goal of promoting fairness across different groups (Hardt et al., 2016; Jiang et al., 2020; Tsaousis & Alghamdi, 2022). By categorizing these techniques, we can better understand the different strategies available for mitigating bias in machine learning systems.
C.2 REGULARIZATION-BASEDD IN- IN-PROCESSING METHODS
In this paper, we explore three types of regularization -aaaad in-rroesssing methods. First, Gap Regularization (Chuang & Mrouch, 2020) streamlines the optimization process by offering smooth approximation of real-world loss functions, which are typically non-convex and difficult to optimize directly. This category includes methods such as DP, EO, and EOD. Second, the Independence approach integrates fairness constraints into the optimization, aiming to mitigate the influence of protected attributes model predictions while maintaining overall performance. Notable examples of this approach include PR (Kamishima et al., 2012) and HSIC (Li et al., 2019). Lastly, adversarial debiasing seeks to minimize utility loss while hindering an adversary's 's ability to accurately prediet the protected attributes. This approach encompasses methods like ADV (Zhang et al., 2018; Louppe al., 2017: Beutel et al., 2017; Edwards & Storkey, 2015; Adel et al., 2019) and LAFTR (Madras et al., 2018).
Beyond the algorithmic-faine fairness literature, recent studies have highlighted several important develop- ments in LLM research, including eermenel- recurrent models for long-context processing, memory-controlie long-context agents, lifelong model editing (Wang et al., 2025; 2026a;b), efficient LLM unlearning (Liu et al., 2025a;b), and mowledge-intensive LLM reasoning (Liu et al., 2025c). While these works are not designed for fairness, they reflect the growing importance of controllability and stability in modern LLM systems. This broader trend suggests a promising direction for adapting fairness-award aware regularization to more complex LLM settings.
C.3 CAUCHY-SCHWARZ DIVERGENCE IN OTHER MI. SETTINGS
Cauchy-Schward Schwarz (CS) divergence has also been studied and applied in several machine-learning problems outside of algorithmic fairness. Early work used CS divergence together with Parzen window density estimates to build ieoomatiion-hhorette criteria for clustering and graph-based learning, and to relate CS divergence to Mercer kernels and graph cuts (Jenssen et al., 2006). More recent studies have employed CS divergence as a training objective for representation learning and deep models, for example, in information-bottlene 2024), domain adaptation (Yin et al., 2024), and CS-regularized autoencoders that improve density estimation and clustering performance (Tran et al., 2022). Conditional variants of CS divergence have further been developed for time-series analysis and sequential decision making (Yu et al., 2025). These works demonstrate that CS divergence is versatile discrepancy measure for density estimation and representation learning; our contribution is complementary, as we systematically develop and evaluate CS divergence as an in-processing fairness regularizer with dedicated theoretical analysis and extensive experiments in the Algorithmic-faimeet setting.
formulations for regression (Yu et al.,
In contrast these applications, which primarily target density estimation, clustering, or represen- tation learning objectives, our focus is on algorithmic fairness. To the best of our knowledge, our work is the first to systematically develop regularizer, with theoretical analysis tailored to groue-fairness notions and an extensive empirical study of the resulting fillit--fiinnss trade-offs.
Schwarz divergence as an in in-processing fairness
C.4 IS THE REPRESENTATION LEARNED BY APPLYING CS VIEWED AS FAIR?
To further validate that CS can learn fair representations, visualize the T-SNE embeddings of the latent space from the last layer before the prediction layer (Van der Maaten & Hinton, 2008)². Figure 6 displays the representations learned from the last embedding layer on the Adult, COMPAS,
httpps htt//siiiittttetete learn. http:/aaabbemmmduieeegggeeereeee /kt http:/kkaaannnrrraeeeeee \pena Ityl mani dold. html
21

---

Published as a conference paper at ICLR 2026
Gender
Race
Gender
Race
Gender
Race
Adult
COMPAS
ACS-I
Figure T-SNE visualizations of the latent representations on Adult, COMPAS, and ACS-I, colored by the target attribute (top) and the sensitive attribute (bottom).
and ACS-I datasets, while Figure 12 presents the results for ACS-P and CelebA-A. A. Based on these visualizations, we make the following observations:
The CS can learn representations that are indistinguishable between sensitive groups. This observation validates the effectiveness of CS in learning fair representations. Specifically, the plots in the first row of Figure 6 illustrate the embedding visualization of two sensitive groups: blue for S=0 and red for =. Overall, the points are uniformly dispersed, with no clear clusters of nodes sharing the same color. This indicates that the embeddings are learned independently of the sensitive attribute. Although some groups have greater number of data points-suu succh as the Adult t dataset with the sensitive attribute race, where the ratio of 5 is 1:9.20, and in the COMPAS dataset with gender, where the ratio is 1:4.7 (as shown in Table 4)-the the distribution points in both colors remains even.
The CS can learn distinguishable representations for different target attributes. Observing the second row of Figure 6, points across different locations in the plot. Among these, the embedding for ACS-I exhibits the clearest pattern, followed by Adult. This observation is consistent with the utility results presented in Table 2, which show a decrease in accuracy and AUC as the degree of negativity increases, particularly evident the f columns compared the MLP. contrast, COMPAS presents a greater challenge in ensuring utility while considering fairness, as indicated by the less distinct pattern in the learned embeddings, corroborated by the most significant utility drops in Table 2.
identify distinct pattern in the distribution of the blue and red
D MORE EXPERIMENTAL RESULTS
D.1 EXPERIMENTS ON IMAGE DATASET
In this section, we present the experimental results on the CelebA-A image dataset. The CelebA-A face attributes dataset (Liu et al., 2015) contains over 200, 000 face images, where each image has 40 human-labeled attributes. Among the attributes, we select 'Attractive' as a binary classification task and consider 'Gender' and 'Young' as sensitive attributes. The results are presented in Table 3.
The results show similar finding with the tabular dataset, demonstrating that 1) DP method always achieves lower App but a relatively high A 2) HSIC is more promising fair model to achieve equal opportunity.
D.2 MORE PREDICTION DISTRIBUTIONS OVER THE SENSITIVE GROUPS
As described in Section 5.3, the kernel density plots in this subsection visualize how the prediction distributions vary across sensitive groups, and how different fairness regularizers affect this alignment. In general, a larger overlap between the distributions of different sensitive groups indicates better group fairness. For each figure in this subsection, the top shows the distributions of the raw
22

---

Published as a conference paper at ICLR 2026
Utility
Faimess
Methods
ACC (%)
t
AUC (%)
ADP (%)
A (%)
MLP
66.21
28
8.32
55.11
DP
69.38
-1.25%
72. 200.08
-1.87%
0.29 +0.15
96.51%
1.83:1..26
64.19%
Gender
MMD
61.28
-2.61%
72.92 20.31
-1.17%
1.22
85.34%
2. 2.144
58.71%
HSTC
66.01
-0.30%
73. 73.1610.32
-0.84%
0.98 2.22
88.22%
1.00 +0.28
80.43%
PR
62.72+1.01
-5.27%
69.36
-5.99%
+0.80
90.63%
1. 40.36
79.06%
NCS -2
Cs
60.40
-0.77%
72.83
-1.29%
0.1 20.00
97.96%
0.75 2.222
85.32%
MLP
20.42
73.69
9.28
6.21
DP
20.29
-2.14%
71.86=a.23
-2.48%
0.82 +0.34
91.16%
1.30
79.07%
Race
MMD
65.71 10.65 20.65
-1.01%
20.57
-4.23%
+0.97
57.22%
1.55
75.04%
HSIC
65.81 10.24
-0.86%
72.92
9..25....
-1.04%
1..5
81.14%
0.43
20.23
93.08%
PR
64.25 20.87
-3.21%
70.25
-4.67%
1..6
83.19%
1.21
40.74
80.52%
cs
65.16
-1.84%
72.56
-1.41%
0.55 20.19
94.07%
1.38
77.78%
RN
18.14
86.58
51.66
35.67
DP
42.4224.79
-20.12%
66.86 19
-22.78%
0.46
99.11%
4.84
86.43%
Gender
MMD
62.54
-19.96%
66.47 #3.80
-23.23%
+o.04
97.31%
5. 5.89
83.49%
HSIC
43.69
-18.88%
63.33
-19.92%
2. 40.3
95.66%
3.83 +2.22
89.26%
CelebA-
PR
65.5
65.5143.68
-16.16%
71. 20.
-17.19%
4.00
92.26%
5.05
85.84%
CS
65.05
65.
43.30
43.80
-16.75%
71.42 4-2.
-17.51%
0.98 6..62
98.10%
1.53 11.05
95.71%
RN
78.14
86.67
41.74
18.33
DP
66.78.3.61
-14.54%
73.99 73.95.3..44
-14.68%
2.43
94.18%
0.91
95.04%
Young
MMD
63.82
-15.77%
72.84
-15.96%
3.49 10.83
91.64%
1.60
91.28%
66.04
-15.49%
73.
73.08
-15.68%
1.99
10.55
95.23%
1.04
94.33%
PR
62.98 4.689
-19.40%
69.63
-19.66%
1.32
96.84%
1.82
90.08%
CS
4.223
-16.39%
73. 75.15 15
-15.60%
1.28 0.24:
96.93%
0.30 00.12
98.37%
Table 3: The fairness performance on the tabular dataset for existing fair models, and we consider race and gender as sensitive attributes. A higher accuracy metric indicates better performance. represents the accuracy improvement compared to MLP. lower fairness metric indicates better fairness. J. represents the improvement of fairness compared to MLP. Green values denote better than MLP on the corresponding metric (ACC/AUC 1: App/A while red values denote worse. The results are based on 10 runs for all methods.
predictions for all target labels, grouped by a given sensitive attribute, while the bottom row focuses on the positive class.
For example, in Figure 7: the first row plots the prediction densities for Race: the blue shaded area corresponds to Y Race Black and the red shaded area corresponds to Y Race = White; the second row then plots the prediction densities for the positive class, i.e., Y 1, again conditioned on the two race groups (blue for Y Race Black and red for Y Race White).
The other figures are interpreted analogously for their respective sensitive attributes. Since the degree of overlap sometimes be difficult judge by eye, we also print the corresponding group-fairness. metric in each subfigure: the first row reports App (demographic parity gap) and the second row reports ArO (equalized odds gap).
23

---

Published as a conference paper at ICLR 2026
DP
MMD
HSIC
PR
CS
Sena. Aliz.
Abtc
Demplts
Mille
wake
Dequett
Whdee
Demetty
Nalt
Prodiction
Prediction
Prodiction
Prediction
Prediction
Sent, Attr.
Race
Sento, Altr.
Sens, Atts
Sinne Atte- Milie
Whlts
White
1.00
1.0
0.5
0.0
0.6
0.8
Prediction
Prediction
Prodiction
Prodiction
Prediction
Figure 7: Prediction distributions for black and white groups in the Adult t dataset.
HSIC
1000
Beart 27
Deiudity
Gender
Fiesice
Gonder Fimaa
Sens. Alts = Presie
Desetry
Sent. Att:
Femaala
Dossaity
10.0
Sens,
Freale
0.0
0.5
1.0
0.25
0.50
8.75
1,00
0.2
0.8
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
6.0
0.4
0.6
0.8
3.0
Gender
Gender
Sema, Aftr.
Sens.
154
Swtta Attr.
Denaity
Madle
Nais
Diuuity
Mala
Maas
Danuttt
20.
Stale
1.00
%
1.0
1.0
Prediction
Prediction
Prediction
Prediction
Prodiction
(a) Prediction distributions for female and male groups in the COMPAS dataset.
MMD
Reamd 7%
Aua
Race Connutton
Race
Race Cennaston
Race Gisrentie
Counctten
Cuereiles
0.00 0.35 0.50 8.75 1.40
0.4
0.6
0.8
1.0
0.25
0.50
6.75 1.00
0.25
0.50
0.75
1.00
044
8.5
0.6
0,7
2
Race
Race Otter
Race Oaker
Race
15
Race other
Dureiit
other
Dessiity
Prepoitt
0.25 0.75
1.00
0.8
1.0
675
1.00
0.25
0.50 0.75
1.00
6.4
0.5
0.6
0.7
Prediction
Prediction
Prediction
Prediction
Prediction
(b) Prediction distributions for Caucasian and (all) other groups in the COMPAS dataset.
Figure 8: Prediction distribution over gender and race in the COMPAS dataset.
24

---

Published as a conference paper at ICLR 2026
DP
MMD
HSIC
PR
CS
Scus. 2.0
Scite
Alts
Mait
Naae
1.9
Mab
Dequaitt
Nale
Muk
1.
2.0
0.5
Prediction
Prediction
Prodiction
Prediction
Prodiction Ooynt
Sone. Atte:
Attr. Fessle
Sone. Attr:
Sens, Foeal
Deality
Deeale
_ Eneale
- Docale
0.0
0.5
1.0
0.5
1.0
0.5
1.0
0.0
0.3
1.0
0.00
0.25
6.50
0.75 1.00
Prodiction
(a) Prediction distributions for female and male groups in the ACS-I datasct.
2.0
Sedae Attr.
2.0
Sunt. Attr.
Sent. Altr.
1.5
Seats Altr.
Seate Attr:
Diaalty
Widte
Deanlly
0.0
0.0
0.5
1.0
0.0
0.0
0.5
1.0
0.0
0.0
1
0.0
0.0
0.5
1.0
0.0
0.0
0.6
Prediction
Scata, Altr,,
Seas.
Sena, Whtt
Deraatt
Wisee
Dervdtyy
White
Made
Whil
Predictson
Prodiction
Prodiction
Prodiction
Prodiction
(b) Prediction distributions for black and white groups in the ACS-I dataset.
Figure 9: Accuracy and A DP trade-off on ACS-I with sensitive attribute gender and race. Results located in the bottom-right corner are preferable.
25

---

Published as a conference paper at ICLR 2026
1.5-
Presule
Poreate
Presate
Femako
Dorrcht
Tresic
0,00
0.25
0.75
1.0
0.6
02
Prediction
Prediction
Prediction
Prediction
Prediction
Seus, Alle Made
Sens, Allr.
Seus, Alte.
Seto Alli,
Sest. Allr-
Deroiit
Danplit
Made
Made
Dusiity
Mai
Male
Prediction
Prediction
Prodiction
Prediction
Prediction
(a) Prediction distributions female male groups the ACS dataset.
DP
MMD
HSIC Bar-iit
PR
Aar
Sens. Arti: Marck
Sens, Attr.
Sono. Attr.
Seno. Attr.
Sens, Art
LG
5.5
8.00 0.23 0.30
0.75 1,00
0.00
0.23
0.50
0.75
1.00
0.0
0.0
0.5
1.0
0.0
0.9
0.5
1.0
0.0
0.5
1.0
Afti:
Seas,
Sens, Mit
Sens. Alte
Seos, Atter
Whiee
Prosally
White
Makn
Prediction
Prodiction
Prediction
Prodiciio
Prvdiction
(b) Prediction distributions black and white groups the ACS dataset.
Figure 10: Accuracy and AP trade-off ACS-T with sensitive attribute gender and race. Results located in the bottom-right corner are preferable.
26

---

Published as a conference paper at ICLR 2026
Demilty
Mode
Nake
Prenuie
Muls
Deuiity
Mal
Deuulty
Male
0.60
0.35
Prediction
Prediction
Prediction
Prediction
Prediction
Ave or
Ru
Au-11.1
Femal
Exmule
Pemult
Urauity
Make
Apphnn
Nals
Nals
1.00
1.00
Prediction
Prediction
Prediction
Prediction
Prediction
(a) Prediction distributions for female and male groups in the CelebA-A datasct.
DP
MMD
HSIC
Awe
Bap-115
Net
Not
Not
Nat
Yearo
Deauity
Yexend
Yeeno
Thuns
Tourss
Prodiction
Prediction
Predictton
Prediction
Prediction
Aup
Aup
Aupit
Raped
Not
Young
Not Yearg
Net
Not
Yeaug
Yeurg
Tounss
10.00
0.5
1.0
0.0
Prediction
Prediction
Prediction
Prodiction
Prediction
(b) Prediction distributions for young and non-yong groups in the CelebA-A dataset.
Figure 11: Accuracy and ADP trade-off on CelebA-A with sensitive attribute gender and race. Results located in the bottom-right corner are preferable.
27

---

Published as a conference paper at ICLR 2026
D.3 MORE T-SNE PLOTS
Gender
Race
Gender
Young
ACS-T
Celeba-A
Figure 12: Accuracy and AD trade-off on ACS-T and CelebA-2 Results located in the bottom- right corner are preferable.
In addition to the T-SNE plots shown in Figure 6, which show the results on three datasets, also include the T-SNE plots on the two remaining datasets ACS-T T and CelebA-A in Figure 12.
E DATASET DESCRIPTIONS AND DETAILS
We conducted experiments on five datasets, including four tabular datasets and one image dataset. The introduction of these datasets is as follows:
Adult 4 (Dua & Graff, 2017) The Adult dataset includes data from 45, 222 individuals based on the 1994 US Census. The primary task is to predict whether an individual's income exceeds $50k USD, using various personal attributes. In this analysis, we focus on gender and race as sensitive attributes.
COMPAS* (Larson al., 2016) The COMPAS dataset contains records of criminal defendants and is designed to predict the likelihood of recidivism within two years. It encompasses various attributes related to the defendants, including their criminal history, gender, and race.
ACS-I and ACS-T" (Ding et al., 2021) The ACS dataset is derived from the American Community Survey (ACS) Public Use Microdata Sample and encompasses several prediction tasks. These tasks include predicting whether an individual's 's income exceeds $50k and whether an individual is employed, with features such race, gender, and other relevant characteristies tailored each task.
CelebA-A" (Liu al., 2015) The CelebFaces Attributes dataset comprises 20, 000 face images of 10, 000 distinct celebrities. Each image is annotated with 40 binary labels representing various facial attributes, including gender, hair color, and age. In this study, we focus the 'attractive' label for a binary classification task, while considering 'young' and 'gender' as sensitive attributes.
The detailed statistics for the aforementioned datasets are summarized as follows:
/ ies. ics.uci............o......o....u /ttp.//ciiee//////aattaaa//////a http:ttee///...o..ceccc/oooooooa
htm1
28

---

Published as a conference paper at ICLR 2026
Dataset
Task
Sen. Attr. (S)
#Samples
#Feat.
Class Y 0:1
1st S 0:1
2nd
0:11
Adult
Income
Gender, Race
45, 222
101
1:0.33
1:2.08
1:9.20
COMPAS
Credit
Gender, Race
6, 172
405
1:0.83
1:4.17
1:0.52
ACS-I
Income
Gender, Race
195, 665
908
1:0.70
1:0.89
1:1.62
ACS-T
Travel Time
Gender, Race
172, 508
1, 567
1:0.94
1:0.89
1:1.61
CelebA-A Attractive
Gender, Young 202, 599
48 x 48
1:0.95
1:0.71
1:3.45
Table 4: The table presents the statistics of the datasets. #Feat. refers to the total number of features after preprocessing". The ratio 0: 1 represents the proportion between the two categories the target label or sensitive attributes.
F BASELINES DETAILS
We consider four widely used fairness methods: DP, MMD, HSIC, and PR. Specifically, DP and HSIC minimize the demographic parity and Sehbert-Sehmidt Independence Criterion, respectively, MMD learns classifier that optimizes the Mean Maximum Discrepancy. We also include base models MLP and RN for tabular data and image data, respectively.
DP: It is a gap regularization method for demographic parity (Chuang & Mroueh, 2020), As these fairness definitions cannot be optimized directly, gap regularization is differentiable and can be optimized using gradient descent.
MMD: The Maximum Mean Discrepancy (MMD) (Gretton et al., 2012) is metric used to measure the distance between probability distributions. Previous research has leveraged MMD to enhance fairness in machine learning models, specifically in variational autoencoders (Louizos al., 2016) and MLPs (Deka & Sutherland, 2023). In this paper, we build on the methodologies from earlier works (Zhao Meng, 2015) to compute the MMD baseline.
HSTC: minimizes the Hilbert-Schie Schhidt Independence Criterion between the prediction accuracy and the sensitive attributes (Gretton et al., 2005; Baharlouei et al., 2020; Li al., 2019). Prejudice Remover (PR) (Kamishima al., 2012) (Prejudice Remover) minimizes the prejudice index, which is the mutual information between the prediction accuracy and the sensitive attributes.
G MORE FAIRNESS DEFINITIONS
H DETAILS OF THE GROUP FAIRNESS
In this section, we provide the details of the group faimess. We first introduce the definition of group fairness. Then, we introduce the existing group fairness metrics and algorithms.
DP (Demographic Parity Statistical Parity) (Zemel et al., 2013). A classifier satisfies de- mographic parity if the predicted outcome Y is independent of the sensitive attribute S, i.e., P(Y 5 0) P(Ý S 1).
prule (Zafar et al., 2017). A classifier satisfies p%--rle if the ratio between the probability of subjects having a certain sensitive attribute value assigned the positive decision outcome and the probability of subjects not having that value also assigned the positive outcome should be no less than p/100, i.e., P(Ý 1 = 1)/P(Ý = 1| S 0)l S p/100.
EOpp (Equality of Opportunity) (Hardt et al., 2016). A classifier satisfies equalized opportunity if the predicted outcome Y is independent of the sensitive attribute S when the label Y 1, i.e.
P(ý S 0, Y 1) P(Ý
1,Y = 1).
EOdd (Equalized Odds) (Hardt et al., 2016). A classifier satisfies equalized odds if the predicted outcome Y is independent of the sensitive attribute S conditioned on the label Y, i.c., P(Ý
S 0, y) P(ý 1, y), E (0,
ACC (Accuracy Parity). A classifier satisfies accuracy parity if the error rates of different sensitive attribute values are the same, i.e., P( Y S = 0) = P(Й Y S 1).y (0, 1}. aucp (ROC AUC Parity). A classifier satisfies ROC AUC parity its area under the receiver operating characteristic curve with w.r.t. different sensitive attribute values is the same.
29

---

Published as a conference paper at ICLR 2026
ppv (Predictive Parity Value Parity) A classifier satisfies predietive parity value parity if the probability of a subject with a positive predictive value belonging to the positive class w.r.t different sensitive attribute values are the same, i.e., P(Y = Y 0) = P(Y = 1
=
bnegc (Balance for Negative Class). A classifier satisfies balance for the negative class the average predicted probability of subject belonging to the negative class is the same w.r.t different sensitive attribute values, i..., (x) Y = 0, = 0J = [ Y = 0,5 1].
bposc (Balance for Positive Class). A classifier satisfies balance for the negative class the average predicted probability of a subject belonging to the positive class is the same w.r.t different sensitive attribute values, i.e., E[j(() = =, 0] = ([f(X) Y 1,S = 1].
abcc (Area Between Cumulative density function Curves) (Han et al., 2023) is proposed to precisely measure the violation of demographic parity at the distribution level. The new fairness metrics directly measure the difference between the distributions of the prediction probability for different demographic groups
I ADDITION EXPERIMENTS
1.1 ADDITION EXPERIMENTS ON MORE FAIRNESS METRICS
We provide additional results comparing framework with baselines under the following fairness tions: Predictive Parity (PPV) (Chouldechova, 2017), p&e-Ruee (PRULE) (Zafar et al., 2017), Balance for Positive Class (BFP) (Kleinberg al., 2016), and Balance for Negative Class (BFN) (Kleinberg et al., 2016). The dataset is Adult, using gender as the sensitive attribute. All other experimental settings are consistent with Table the paper.
Method
APPV (1)
PRULE (t)
ABFP (1)
ΑBFN (4)
DP
27.35 5.64
81.21
11.25
2.75
5.15 + 0.44
MMD
35.19 + 6.33
7.15
18.32 3.74
3.49 + 0.25
HSIC
37.25 3.19
9.11
46.47
4.04 + 0.32
PR
25.46 3.17
7.357
21.45 + 2.37
3.46 = 0.28
CS
31.59 4.35
07.75+3.24
15.25 2.58
3.18 + 0.36
Table 5: Fairness performance comparison the Adult dataset, with gender the sensitive attribute under Appv, PRULE, ABFP, and ABFN.
We observe the following:
CS generally achieves the best fairness trade-off performance across the four tested fairness notions.
On the Adult. t, BFN is generally minimized more effectively than BFP.
Since BFN is related to EO, the ranking of ABEN aligns with AçO in Table 1 of the paper. Note that, stated in previous studies (Kleinberg et al., 2016), there is an inherent trade-off between BFP and BFN in practice.
I.2 ADDITIONAL EXPERIMENTS ON COMBINING MULTIPLE REGULARIZER TERMS SIMULTANEOUSLY
We conducted additional experiments where we combined both KI. divergence and CS divergence as regularizers. The experiments were performed on the Adult, with gender as the sensitive attribute.
Actually, combining multiple faimess objectives has several drawbacks, which is why most existing studies avoid using multiple regularizers. Instead, they often choose to add simple constraint terms. The key drawbacks of combining fairness regularizers are summarized follows:
*We adopt the preprocessing in previous studies (lee Quy al., 2022: Mehrabi et al., 2021) involving identifying the target lahels and sensitive attributes, and then selecting the relevant features for the analysis.
30

---

Published as a conference paper at ICLR 2026
Method
Apr (4)
A (4)
DP
1.29 + 0.95
20.15 = 1.13
CS
2.42 + 0.85
2.27 + 1.04
KI
2.77 + 0.86
10.42 + 4.34
CS+KL
2.46 1.25
13.42 = 6.12
CSK0.
2.25 + 1.14
9.33 + 6.36
Table 6: The fairness performance on Adult (gender).
The CS divergence is upper-bounded by the KL divergence. Therefore, adding KL as an additional faimess objective is theoretically redundant and will not provide further benefits.
Adding KI. or other faireess metries increases computational complexity, making the optimization process more challenging.
These experimental results further show the significance of our contribution: identifying suitable, tighter- boulded fairness regularizer that balances effectiveness and computational efficiency.
1.3 ADDITIONAL EXPERIMENTAL ON EODD AND EOPP
In this subsection, we provide additional experimental results on EOdd and EOpp. defined in Ap- pendix H. Each line represents the results for one of the hyperparameter values a = 0.2/0.8/1..4 denoted as 'lam' in the legend. We observe that both EOdd and EOpp regularizers perform well on the EO metrics but do not perform well the DP metrics. We summarize the results as follows:
Table 7: Fairness performance comparison on the ACS-I (gender) on additional metrics.
Method
Aprv (4)
PRULE (t)
ABFr (4)
ABEN (4)
Apr (1)
Ago (L)
DP
11.86 6.54
97.14 12.41
4.83 + 4.35
3.97
4.02
0.96 = 0.22
5.37 + 0.32
FOdd
8.38 1.28
84.75 + 3.88
0.43 + 0.63
0.31
0.73
5.76 + 1.42
0.64
1.25
Eopp
8.00 + 1.65
83.36 4.41
0.34 + 1.00
1.24
1.43
6.32 + 0.81
0.52
1.29
CS
7.00 + 8.35
96.90 + 8.35
4.39 + 1.32
2.77 1.83
0.77 + 0.38
0.90 + 0.46
The observations from the results align with our claims, and the CS regularizer demonstrates signifi- cant effectiveness:
As shown in Table 2 of our paper, the MLP achieves A of 2.13 3.64, whereas the DP regularizer gets a higher A EO of 3.97 4.02. This indicates that the DP regularizer does not effectively optimize and may even negatively affect EO faimess..
EO-based methods (EOdd and EOpp) show the worst performance terms of DP fairness, even compared to other baselines such MMD, HSIC, and PR (as reported in Table 2 in the paper). In particular, EOpp reaches approximately 15 DP on the Adult dataset, shown in the Appendix K. This high AD is consistently observed across different hyperparameters (a 0.2, 0.8, 1.4, referred to as 'lam' in the figure).
1.4 ADDITIONAL EXPERIMENTS ON PRE-PROCESSING AND POST-PROCESSING BASELINES
We have added post-proccee processing method, PostEO (Hardt et al., 2016) on the Adult dataset (with gender as the sensitive attribute).
The PostPro method is specifically designed to optimize for EO (Hardt al., 2016), which explains its lower Ago.
However, both pre-processing and procesproessing methods share common limitation: they result in lower utility (ACC or AUC). Considering the need for a balanced trade-off between faimesss and utility, CS emerges as the most favorable option in comparison.
31

---

Published as a conference paper at ICLR 2026
Method
ACC (1)
AUC (1)
ADP (4)
Å no (4)
DP
82.42 = 0.39
86.91 0.80
1.29 0.95
20.15 = 1.13
CS
83.31 .0.47
90.15.1049
2.42 0.85
2.27 1.04
PR
81.81 + 0.52
85.38 + 0.82
0.71 0.40
12.45 + 2.38
PostPro
80.25 + 0.83
84.35 + 0.98
5.75 + 1.67
2.12 + 1.44
Table 8: Comparison of methods on various metrics.
1.5 ADDITIONAL. BASELINES WITH DEPENDENCE MEASURES
To further contextualize our CS-based fairness regularizer. we additionally compare against sev- eral classical dependence measures between the model prediction Y and the sensitive attribute S Hirschfeld - Rényi maximal correlation (HGR), mutual information (MI), and (distance) co- variance (dCov) or empirical distance covariance. In all cases, we regularize the model by penalizing the corresponding dependence between Y and S.
HGR maximal correlation (HGR). The Hirrchfeld-GGeellinn--ee tion (Hirschfeld, 1935; Gebelein, 1941; Rényi, 1959) between two random variables X and S defined as:
maximal correla-
S)) (o s.t. [[f(()) =1
f.g
(38)
where the supremum is taken over square-integrable functions f and g. We instantiate an HGR-based fairness regularizer by penalizing PHOR (Y, S) using neural estimator.
Mutual information (MT). Mutual information (Cover, 1999) between and is
T(X /ʃ px,s(s, log
(x, 8) dr ds,
(39)
px px(r)ps(s)
or, for discrete variables,
(X;8)= E[ px,((,), log px,s(2,s)
(40)
px(x)
Equivalently, MI can be written as a Kullback-Leibh divergence
((x:)) DKL(Px,s PxPs).
(41)
Our MI-based baseline regularizes the mutual information between the prediction and the sensitive attribute, I(Y: S), using a differentiable estimator.
Distance covariance (dCov) and empirical distance covariance. Distance covariance (Székely al., 2007) between X E RP and S R24 defined via their joint and marginal characteristic functions xx,ss 4xx and sps
S (xx,s(t,,) px (t) ps()) w(t, u) dt du,
dCov² (X,
(42)
for suitable weight function wr(,,u) u). In practice, we the standard empirical distance covariance
estimator. Given samples {Tr,, st) define pairwise distances dij = ||x;--;;)| and bi = ||8; (19-88/1 ||.
and their double-centered versions:
Ag = a.j ō, ā.j + a... Byj bij b, b, b... where a; and a. denote row and column means, and a. the grand mean (and analogously for ii). The empirical distance covariance is then:
(43)
(44)
doo (X = 1
Ais Bij-
(45)
n2
i=1 ==
- 2
Our dCov-based regularizer penalizes dCov (Ý,S). S).
32

---

Published as a conference paper at ICLR 2026
Experimental setup. To assess how CS-based regularizer compares with other classical dependence measures, we conduct an additional experiment on the ADULT dataset using gender as the sensitive attribute. We keep all settings identical to the main experiment on ADULT in Section 5: the same data split and preprocessing, the same MLP classifier architecture, optimizer, batch size, learning rate, number of epochs, and the same protocol for tuning the fairness-regularization weight. The only change is the choice of the fairness loss term Liair.
Concretely, for each additional baseline, we replace the CS-based fairness loss in Equation (13) with the corresponding dependence measure between the model prediction Y and the sensitive attribute Frrschfeld-Geeeleiin- -ényy maximal correlation PHGR (Y, .) S: (i) the HGR baseline penalizes the Hirschfeld Gebelein
using a neural cstimator; (ii) the MI baseline penalizes the mutual information I(Y; 5) between prediction and sensitive attribute, estimated with a differentiable MI estimator; and (iii) the dCov
baseline penalizes the empirical distance covariance dCov (Y .S) defined before. In all cases, the overall training objective retains the same form as Equation (1), and we evaluate the resulting models on accuracy, AUC, App, and AgO using the same test split as in the main experiments. The results are summarized in Table 9.
Method
ACC (1)
AUC (t)
ADP (4)
Ag (4)
HGR
80.13 + 1.35
84.20 + 1.27
3.82 + 0.84
6.82 3.77
dCov
82.31 + 0.62
85.39 + 0.89
4.75 + 1.67
12.41 + 1.44
PR
81.81 + 0.52
85.38 + 0.82
0.71 + 0.40
12.45 + 2.38
CS
83.31 + 0.47
90.15 + 0.49
2.42 + 0.85
2.27 + 1.04
Table 9: Additional comparison of with HGR and dCov.
From the results in Table 9, we observe that the proposed CS-based regularizer achieves the best overall performance among all dependence-baue baselines: it attains the highest ACC and AUC while keeping both Apr and Ago low, confirming that CS offers robust utility-fairness trade-off. More specifically, (i) HGR is theoretically very strong dependence measure, but in practice it requires a neural estimator of the maximal correlation, which makes optimization noisy and sensitive to hyperparameters; this is reflected in its relatively low ACC/AUC and larger standard deviations, although its fairness metrics are still competitive, indicating that can reduce dependence when the optimization succeeds. (ii) dCov is a beemel-based or distance- bastance-based statistic with closed-form empirical estimator, so it is easier to optimize and leads to higher ACC/AUC and smaller variance than HGR; however, its fairness performance is weaker, suggesting that penalizing average pairwise distances between prediction and sensitive-feature embeddings is less aligned with group-rate gaps than the density-ratio style CS divergence, which yields tighter control over the discrepancies that drive ADP and ΔEO. (iii) PR (an MI-based regularizer) achieves very small App, consistent with its design of directly reducing mutual information between Y and S and thereby aligning the marginal prediction rates across groups, but its AςO remains large and its utility moderate, as MI does not explicitly constrain the conditional error rates P( Y, S) that underlie equalized odds. Overall, these observations support CS as the most balanced choice among the considered dependence measures.
Discussion:
Relation between MI and our KI. and PR-based regularizers. For completeness, we briefly discuss how the generic MI baseline above relates to the KL-bbaed fairness losses and the Prejudice Remover (PR) baseline used in our main experiments. By definition, mutual information is a Kullback-Leibler divergence between the joint distribution and the product of the marginals:
I(X; Dra(px.s pxps)
(46)
Thus, MI and KI. belong to the same family of information-theorret discrepancy measures: MI uses KL to quantify any deviation of px.s from independence, while many fairness regularizers based on KL (including the KL term used in our fairness-loss landscape in Figure 2) penalize Kullback- -Libler divergences between conditional distributions, such as DKL (pyls=o Pr)s=1). These conditional KI. penalties are closely related to MI but not identical: if all py coincide,
then both the conditional KL and I [Ý;; S) vanish, yet vanishing conditional KL for one pair of groups does not necessarily minimize the full KI. between py, s and Py-S
33

---

Published as a conference paper at ICLR 2026
The Prejudice Remover (PR) (Kamishima al., 2012) can be viewed explicit MI-based regularizer. PR minimizes the prejudice index, which is defined as the mutual information between the prediction and the sensitive attribute, T(Y; under log-linear model of the conditional odds. In this sense, PR instantiates the generic MI regularization principle with particular parametric form and optimization scheme.
In summary, the generic MI baseline, the KL-based fairness penalties, and PR all enforce indepen- dence between Y and S using KL divergence in different guises: MI regularizes the KL divergence between Py,s and Py: Our KL-based fairness loss penalizes KL divergences between group- conditional predictions distributions; and PR minimizes a parametric approximation of 5)) Our CS-based regularizer complements this family by replacing the KL-based dependence measure with the Cauchy-Schwarz divergence, which enjoys closed-forn kernel estimators and the tighter bounds analyzed in the main text.
1.6 ADVERSARIAL METHODS EXPERIMENTS
We conducted additional experiments using Adversarial Debiasing (Louppe et al., 2017), which refer to as ADV below.
Method
ACC (f)
AUC (t)
ADP (L)
AκO (4)
DP
82.42 + 0.39
86,91 + 0.80
1.29 +: 0.95
20.15 +: 1.13
CS
83.04 1: 0.51
90.84 1 0.35
2.13 1 0.89
2.35 + 1.15
ADV
81.58 1.26
83.08 1 0.75
16.3 1 7.5
14.2 8.6
Table 10: Additional experiment on the faimess performance of ADV on the Adult dataset (gender attribute).
From Table 10, we observe that:
The ADV method exhibits lower utility (in terms of ACC and AUC) and higher ADP compared to both the DP and CS fairness regularizers. It also performs worse than the CS regularizer in terms of Aoo.
ADV also shows a higher variance in accuracy, likely due to the greater difficulty of optimizing adversarial objectives compared the DP and CS regularization approaches.
MORE EXPERIMENTAL DETAILS
In this section, we describe the details the experimental setup. In this work, we adopted straightforward stopping strategy. We employ a linear decay strategy for the learning rate, halving every training step. The model training is stopped when the learning rate decreases to a value below 1e-5. Across all datasets, we use weight decay of 0.0, StepLR with step size of 50 and a gamma value of 0.1, and train for 150 epochs using the Adam Optimizer (Kingma & Ba, 2014) the dataset, with specific values provided below.
The batch size and learning rate vary depending Additionally, Table 11 lists the range of the control hyperparameter 3 each fairness approach. The experiments were executed using NVIDIA RTX A4000 GPUs with 16GB GDDR6 Memory.
J.1 HYPERPARAMETER SETTINGS
1. Training Hy Hyperparameters:
Tabular data (Adult, COMPAS, ACS-I, and ACS-T):
Learning rate: 1e
Weight decay: 0.0
-Step ReppR__ae R_step 50
StepLL -SteLLRggammaa 0.1
Training epochs: 150
34

---

Published as a conference paper at ICLR 2026
Batch sizes: 1, 024 on Adult, 32 COMPAS, 4, 096 ACS-I 4, 096 ACS-I
Image data (Ce (CelebA-A) AA):
Learning rate: 1e" 3
Weight decay: 0.0
StepLR step: 50
StepLR gamma: 0.1
Training epochs: 150
Batch sizes: 256.
Architecture Hyperparameters:
Multilayer perceptron:
Number of layers:
Number of hidden neurons: {512, 256, 64)
ResNet-18 (He et al., 2016):
Model: https:
http:///ww...tttooo/o//////////t m//moelss/resnet.py
J.2 HYPERPARAMETER SELECTION
To implement CS and the baseline methods, we adjust the hyperparameter 3 by tuning it within specified range. The details of the hyperparameter selection process and the specific range for / are provided below:
Method
Fairness Control Hyperparameter 3
DP
0.5, 1.0. 1.2, 1.4, 1.6, 1.8, 2.0, 3.5,
HSIC
0.1, 1, 5, 10, 50, 100, 200, 300, 400, 500, 600, 700, 800, 1,0,,10000
PR
0.05, 0.2, 0.3, 0.40, 0.50, 0.7, 1.0
ADV
0.5, 1.0, 1.2, 1.4, 1.6, 1.8, 3,00
CS
1e-0 le le
le
le 2e-2 1.55- 0.5.1,05,, 1.0, 50, 150
Table 11: The selections of fairness control hyperparameter blue,3.
35

---

Published as a conference paper at ICLR 2026
K CURVES
In this section, we show some important curves we recorded.
Note:
The line represents the mean values, and the shaded area indicates the variation across all
runs.
K.
EODD ADDITIONAL RESULTS
[Adult (Gender)
Figure 13: Other metrics (Arrv (4), PRULE(T), ABPP (4), ABNN (4) ) ADP (Shown as
'dpe", and eodde.
Adult (Gender)
uod
codeeent per
Figure 14: ACC, AUC and Ago (Shown as "eoppe")
36

---

Published as a conference paper at ICLR 2026
ACS-I (Gender)
Figure 15: Other metrics (Arrv (1), PRULE(1), ABPP (4), ABEN (4) ) ADP (Shown as 'dpe', ) and eodde.
ACS-I (Gender)
catse
Afficii
Indidd's
Sadagge
Figure 16: ACC, AUC and A (Shown "coppe") '))
37

---

Published as a conference paper at ICLR 2026
K.2
EOPP ADDITIONAL RESULTS
Adult (Gender)
tevilee
taaloodee
Figure 17: Other metrics (Appv (4), PRULF(t), ABPP (4), ABEN (4) ) ADP (Shown as
'dpe", and eodde.
Adult (Gender)
Chhoc
Figure 18: ACC, AUC and A50 (Shown as "coppe").
38

---

Published as a conference paper at ICLR 2026
ACS-I (Gender) ACS-I (Gender)
WOM
Sea
Figure 19: Other metrics (Appv (4), PRULE(1), ABPP (4), ABEN (4) ) ApP (Shown as "dpe"), and eodde.
cdooe teyit senatee
Figure 20: ACC, AUC and A (Shown "eoppe")
Dataset: Adult
Sensitive Attribute: Gender (represented as "Sex")
Hyperparameters:
Learning rate 1 x 10 10-2, 5 x 10 batch size 1024, and 3 = 1.0 (kept consistent across all regularizers).
39

---

Published as a conference paper at ICLR 2026
Training Fairness loss: HSIC, CS, and DP
3.000
Figure 21: Training loss.
Training Fairness Loss: HSIC and CS
3.3
Figure 22: Training loss (excluding DP regularizer line) more clearly observe the gap between HSIC and CS regularizers.
40

---

Published as a conference paper at ICLR 2026
Test ADP and Ago:
testippe
testoogp:
dissatt Isut, target ant rnoosss sendiian, you sou moceshele dateee claik, laset, sersiine clifip
dateerr ada Sirpit, 201 hoome unsttee atto modit hil dausettoouul, All altisss mode: slFlyp
302
122
180
200 123 140
Figure 23: Test Apr and AEO-
L ADDITIONAL ABLATIONS AND EXPERIMENTS WITH MULTIPLE SENSITIVE ATTRIBUTES
L.I ABLATION OVER KERNEL FAMILIES (GAUSSIAN/ LAPLACIANPOLYYOOMIAJ
Beyond the bandwidth study, we also compare different kernel families used inside the CS regularizer. We consider three standard choices:
Gaussian (RBF): (u, v) exp (- ||u-
20
Laplacian: (u, v) = exp ||u-v|l2)
Polynomial (degree 2): Kpoly (u, = (vu 1/0.
On the Adult-Income task (sensitive attribute Sex), we keep the model architecture, optimizer, training schedule, and regularization weight À identical to the main experiment, and only change the kernel family used the CS loss. For fair comparison, we use the same bandwidth 0₂ Fy Ceross 1
for all three kernels. We summarize the final results from Figure 26 into Table 12.
Figure 24: Ablation study: Accuracy and faimess of different kernel functions (Gaussian/LLapla- cian/polynomial) on Adult (sex).
Kernel type
AUC (%)
Ace (%) f
ABO
ADp 4
Laplacian
84.2
83.5
0.75
3.3
Gaussian RBF
85.8
84.7
2.5
5.6
Polynomial
86.3
85.1
3.3
6.5
Table 12: Ablation study: performance different kernel functions (Gaussian/ Lataaiaaa/LLppaaiiappoooooo mial)
From Table 12, all three kernlls achieve strong predictive performance (AUC 84-86%), but they trace out different points the fciuracy-fairness trade-off curve:
41

---

Published as a conference paper at ICLR 2026
The Laplacian kernel yields the smallest EO and DP gaps, i.e., the best group fairness, at the cost of small drop in AUC/aceuracy.
The polynomial kernel attains slightly higher AUC/aceuracy, but with noticeably larger EO/DP gaps.
The Gaussian (RBF) kernel lies between the two, offering a balanced trade-off: it preserves most of the performance benefits of the polynomial kemee while significantly improving fairness compared to polynomial and remaining closer to Laplacian.
These results confirm that CS is compatible with multiple kernel families and that the choice of kernel can be used to tune the faimess-utility trade-off. In the main paper, we adopt the Gaussian kernel as default because it provides a stable, middle-ground trade-off and is widely used in dependence measures (MMD/HSIC), making our comparison to existing divergences more direct.
L.2 ABLATION OVER KERNEL BANDWIDTH Ø
To study the effect of the kemel bandwidth in the CS regularizer, we fix the model and training setup used on Adult-Income (sensitive attribute Sex) and vary the kernel bandwidths while keeping all other hyperparameters fixed (same optimizer, learning rate, batch size, and a as in the main Adult experiments).
We use an RBF kernel hig (u, v) = exp uuv 3 and set 0₂ = 1 for the prediction output, while 20
varying 02 = By = Cevos E {0.5, 1, 2, 5, 10, 15, 20} for the sensitive attribute and cross terms in the CS loss. For cach configuration we record: (i) the AEO ("test/eoppe') ) (ii) the ADP ("test/dpe'), (iii) test accuracy, and (iv) test AUC at the final epoch.
25-25 DIT 1ORE
141
147 4010 2003 24 4500 oesrc signa meceit
14015 4110-22 MND0
1 LE1T
14113 No60 2O Wokl
1914 Ateeri U...A. deseer
Figure 25: Ablation study: Ablation over kernel bandwidth of CS on Adult (sex). How to read these figures: in each subtigure, the lines are ordered in the legend from top to bottom according to their values, from highest to lowest.
From Table 13, we two observations:
Extremely small bandwidths (o < 1) Here the RBF kernel becomes extremely peaked. The CS loss forces almost pointwise independence, which makes optimization unstable: accuracy collapses
42

---

Published as a conference paper at ICLR 2026
dy Oz
CruOs
Ago +
App +
ACC (%)
AUC (%)
0.5
1.52
4.34
24.8
84.5
1
0.75
3.39
24.8
84.9
2
7.76
12.80
76.9
90.4
5
6.39
22.15
82.1
90.8
10
5.64
20.27
85.5
91.2
15
3.68
16.37
84.5
90.8
20
8.08
19.40
85.0
90.7
Table 13: Ablation study: Kernel bandwidth on the proposed CS fairness regularizer.
25% and AUC drops = 84-85%. The very small DP/EO gaps in this regime are therefore misleading-thhy correspond to a nearly random classifier.
Moderate bandwidths (a 20)) In this regime, the classifier maintains high utility (AUC ~ 90 91%, accuracy between 77% and 85.5%) Fairness varies smoothly with a:
Ao generally improves when moving from too-local kernels (a = 2) to more moderate ones; 15 achieves the smallest Ago among the stable runs.
2, but this comes with noticeably lower accuracy, For {10, 15, 20J. both DP and EO are within a similar, reasonable range while utility is highest.
ADP best
Overall, the results show that CS is not hypersensitive to the exact kernel bandwidth: once a is chosen in a reasonable range, the method achieves consistently high AUC with stable fairness-utie utility trade-off. In our main experiments we therefore use moderate bandwidth (c.g..o = 10) that lies in this stable region, balancing strong accuracy (AUC ze 91%) with substantially reduced Apr/AgO-
L.3 EXPERIMENTS WITH MULTIPLE SENSITIVE ATTRIBUTES
To evaluate CS on attribute intersectional attribute combining and race. We construct four groups S {0, 1,2,3) as White-Male, White-Female, Non- Non-White-Moe White- and Non- White-Female. For each group
fairness, we extend the Adult setting from a single sensitive
we define: DPg- = P(Ý = -1 S = g). EO,- = P(Ý = 1 l = S = g), Acc, P( = S = g). We then report the intersectional demographic-parity gap ADpineer max, DP, min, DPg. the intersectional equal-opportunity gap intrr max, EO, min, EOg. and the worst-group accuracy Accmin = min, Accg. The code snippet of these metrics are in Appendix N Using the same MI.P architecture and ('lam' in the figure) 0.5, compare CS with three representative dependence-based regularizers (diftDP, HSIC, diftEOOpp) The results are summarized
below:
Figure 26: Evaluation of CS on the Adult (sex) as the sensitive attribute.
From Table 14, we observe that CS attains the smallest intersectional /AFO while also achieving the highest worst-group accuracy, indicating that the proposed Cauchy-Schward regularizer remains effective even when faimess is evaluated over four intersectional subgroups rather than single sensitive attribute.
In particular, CS improves both ADpinter and AnOrrter over HSIC and DP regularizer, and slightly improves Acemin compared to all baselines. These results suggest that the tighter dependence
43

---

Published as a conference paper at ICLR 2026
Method diftDP
ADpiiler
AEO
Acemin
0,255
0.112
0.792
HSIC
0,262
0.152
0.791
diffFOpp 0.268
0.183
0.790
CS (ours) 0.241
0.108
0.803
Table 14: Evaluation of CS on the Adult (sex) as the sensitive attribute.
control provided by CS translates into more balanced treatment across intersectional groups without sacrificing worst-case utilityy
M DISCUSSION: WHEN AND WHY IS CS EXPECTED TO OUTPERFORM OTHER DIVERGENCES?
To address the question of when CS is practically preferable to other dependence measures, we summarize the regimes where CS theoretically and empirically advantageous:
Heavy-tailed or skewed group distributions. When one group exhibits heavier prediction tails (e.g.. more extreme probabilities), KI. and DP penalties can be dominated by these tails. In contrast, the CS divergence, through its Lgzoormalization of group embeddings, limits the influence such extremes and yields more stable fairness penalty.
Scale-mismatched representations. When latent embeddings for different groups differ markedly in variance or norm (a common scenario deep models), Euclidean- based MMD can report large distances even when the group embeddings are well aligned in direction. CS compares normalized embeddings and therefore provides a tighter and more meaningful notion of "closeness" for fairness.
Imbalanced group sizes. In highly imbalanced datasets, group-condition condiiiooi densities are estimated with very different effective sample sizes. In such cases, KL and HSIC can fluctuate considerably with the minority group's empirical variance, whereas the cosine-style normalization implicit in CS makes the fairness loss less sensitive this sampling noise.
These regimes are not hypothetical: the datasets Section (Adult, COMPAS, ACS, CelebA-A) all exhibit least one of these characteristics. This helps explain why CS often achieves lower Apr and Ago at comparable or better utility our experiments.
N CODE SNIPPET
1 def itc.uuaatt_inttrsstiiooaiio.oooe
(y_pred, y_target, sensitive):
2 if isinstance ly_pred, torch.
y-pred y_pred, (). cpu nnmmy
if isinstance (y_target,, torch.Tensor.
5 y_target y_target. () cpuc numpy ()
6
if isinstance (sensitive, torch. torch.Tensors Tensor):
7
sensitive sensitive.detach cpp () numpy
8
9
pred y fradeelatten
10
y_target - y_target. flarget..latten.I/
11
sensitive
flatttee
13
yipred_binary = (y-pred > 0.5) asstpe (int)
14
15
groups [0, 2, 3]
16
rates {
18
for g in groups:
19
mask = sensitive == g
20
if mask. > 0:
21
rates[g] __pred_b' nary [mask] mean
44

---

Published as a conference paper at ICLR 2026
22
else:
23
rates[g]
24
25
dp_gap max (rates, values ()) min (rates. values ()
26
27
tprs {)
28
for g in groups:
29
mask = (sensitive == g) & (y_target == 1)
30
if mask.sum > 0:
31
tprsigl y_pred binary mask] mean else:
32
33
tprs[g] 0.0
34
35
ga_ max vortts..valuss
values ()) min min(sprs.values values ())
36
37
accs {
38
for g in groups:
39
mask sensitive ==
40
if mask.sum > 0:
41
accs[g]
y_target [mask]) mean
42
else:
43
accs [g] 0.0
44
worst gros__groupace = min laacss.vauee )))
45
46
return
47
"intereectionaa_____ap"" dp_gap,
48
Lntersectiooaa__o__ep eo_gap,
49
"xorst_group_ace"; worst_group_ace,
50
Listing 1: Calculation of Intersectional Fairness Metries
45


Reviews:
[{"review_id": "sTSu7SH3Q7", "reviewer": "Reviewer_LT9c", "summary": "It argues that CS divergence offers a tighter bound compared to existing measures like KL divergence and MMD, leading to more robust and generalizable fairness. The authors present empirical results on several datasets demonstrating improved fairness metrics while maintaining competitive accuracy. The paper is well-written, technically sound, and addresses an important problem in the field of fair machine learning.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The application of CS divergence to fairness regularization is a novel contribution. The paper provides a good theoretical justification for why CS divergence might be superior to other measures.\n2. The paper is generally well-written and easy to follow, clearly explaining the proposed method and its advantages. The inclusion of algorithm implementation details enhances reproducibility.\n3. The empirical evaluation is comprehensive, covering multiple datasets, fairness metrics, and comparisons to existing methods. The analysis of the trade-off between accuracy and fairness is particularly insightful. The paper takes considerable care in describing the dataset parameters and the rationales that underpinned their selection.", "weaknesses": "1. While the paper highlights the advantages of CS divergence, a more thorough discussion of its limitations would be beneficial. For example, are there specific types of datasets or scenarios where CS divergence might not perform well? Are there computational costs associated with using CS divergence compared to simpler measures?\n2. While the paper compares against several standard fairness methods, including more recent and sophisticated techniques from the fairness literature, it could further strengthen the empirical evaluation.\n3. There are nine figures in the paper. These help provide greater clarity on the data and results. It would also be helpful if the visualisations were briefly discussed in the text of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "feaxRs9MqB", "reviewer": "Reviewer_ESYX", "summary": "In this paper, the authors explore fairness via independence. The method is based on encouraging independence between predictions and sensitive features through an optimization framework that leverages the Cauchy–Schwarz (CS) Divergence as a principled measure of dependence.", "questions": "- This paper gives a comparison between cs divergence and kl divergence under Gaussian distribution. In fact, Gaussian distribution is very special, I suggest the authors discuss more advantages and provide comparisons with other regularizers, hgr, MI, dc or empirical dc for instance. \n\n- Line 084, you should add citations for gap parity and MMD.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The authors introduce Cauchy–Schwarz (CS) Divergence as a principled measure of dependence.\n- The authors give three tubular datasets to validate.", "weaknesses": "- This paper gives a comparison between cs divergence and kl divergence under Gaussian distribution. In fact, Gaussian distribution is very special, I suggest the authors discuss more advantages and provide comparisons with other regularizers, hgr, MI, dc or empirical dc for instance. \n-", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pH4aly5aWk", "reviewer": "Reviewer_Pqrj", "summary": "This paper modifies the typical \"performance+ fairness regularizer\" framework by using the Cauchy-Schwarz divergence as the fairness regularizer.", "questions": "- Is the proposed method applicable to scenarios involving multiple sensitive attributes?\n- What is the impact of using different kernel functions on the method’s performance?\n- In Section 5.4, the parameter sensitivity analysis is conducted only on $\\alpha$ and $\\beta$, not including the $\\sigma$ used in the Gaussian kernel. How sensitive is the performance to the kernel parameters?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- The paper offers a new viewpoint on measuring fairness and thereby is applicable to existing fairness optimization frameworks.\n\n- Using Cauchy-Schwarz divergence as a fairness regularizer offers a mathematically simple and interpretable alternative to traditional measures like KL divergence or MMD and the empirical estimation of CS divergence is often more stable than KL divergence under small or skewed samples, reducing the risk of gradient explosion or extreme behaviors during fairness optimization.", "weaknesses": "- The overall contribution is rather incremental, the main framework simply replaces existing discrepancy measures (KL, MMD, HSIC, f-divergences) with the Cauchy–Schwarz divergence. This substitution alone does not introduce a new conceptual insight, making the work less exciting. Moreover, CS divergence has already been widely adopted in areas such as representation learning, domain adaptation, and clustering and this paper mainly transfers an existing metric into a new application area (fairness), rather than introducing a fundamentally new fairness paradigm.\n\n- The empirical estimation of the CS divergence relies on kernels (Eq. 11). However, the theoretical analysis (e.g., Proposition 4.2) only holds under the Gaussian distribution assumption. Could the authors clarify whether this assumption is necessary for the empirical version as well? If not, why did the paper not explore or compare other kernel functions (e.g., Laplacian, polynomial), or analyze the impact of kernel choice on fairness performance in the experiment section? Since the kernel implicitly defines a density estimator, the fairness performance could be highly sensitive to kernel type and bandwidth. A kernel ablation or sensitivity analysis would strengthen the empirical validity of the claim.\n\n- It is unclear whether the choice of CS divergence is practically applicable or scalable, the complexity of using CS is $O(n^2)$, however, the paper does not include any runtime or complexity comparison, nor any discussion of stability during optimization. Without such analysis, the claimed robustness of CS divergence remains questionable in practice. \n\n- Some experimental claims appear not well supported. For instance, in obs 1 (page 7), the authors state that \"...CS consistently achieves the best $\\Delta_{EO}$ and ranks among the top four for $\\Delta_{DP}$...” However, the baselines do not include any method that explicitly optimizes EO as a fairness objective, making this comparison incomplete. Moreover, the evaluation omits several recent works that use f-divergence or information-theoretic measures for fairness regularization, which weakens the empirical significance of the conclusions.\n\n- Table 2 is overly compact and hard to read. The authors could consider presenting results in a clearer format such as $85.63_{\\pm 0.34}$ instead of the current layout. (minor)\n\n- Appendix C.3, all the figures are left without any explanations. The authors could provide some brief explanations of what each figure illustrates and conclude how they relate to the main fairness claims. (minor)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3k7yDiHUti", "reviewer": "Reviewer_JuMp", "summary": "The paper proposes a novel, general, and model-agnostic regularization framework based on the Cauchy-Schwarz (CS) Divergence. The core idea is to encourage independence between model predictions and sensitive attributes by using CS divergence as the fairness loss term $\\mathcal{L}_{fairness}$. The authors motivate this choice by arguing that CS divergence provides a tighter theoretical bound compared to other measures like Kullback-Leibler (KL) divergence and gap parity, which they posit leads to improved generalizability and robustness.", "questions": "see above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper is well structured and easy to follow.\n\n2. The proposed methods are well presented and evaluated with extensive experiments.\n\n3. The choice of the Cauchy-Schwarz divergence is well-motivated. It is justified by theoretical properties, and the paper highlights its advantages, which provide a strong foundation for the method.", "weaknesses": "1. The paper claims the CS regularizer is more \"robust\", which is partially supported by the smoother trade-off curves (Obs. 5 ). However, robustness to model parameters is also claimed. Figure 2 is presented as evidence for this, but its caption is vague, and it is not well-integrated into the main text. The parameter sensitivity analysis in Figure 5 is good for the CS method itself, but it doesn't include a comparative analysis against the baselines.\n\n2. The authors explicitly state that they exclude adversarial debiasing methods to avoid complexity. While this is an understandable simplification, adversarial methods (e.g., Zhang et al., 2018 ) are a very common, powerful, and closely related family of in-processing techniques for achieving independence. Including at least one such baseline would have provided a more complete picture of the method's performance relative to the state-of-the-art.\n\n3. The paper introduces CS divergence as a novel regularizer for fair machine learning. While this application appears novel, the paper itself notes that CS divergence is used in other ML domains like deep clustering and representation learning. The \"Related Work\" section (in the appendix ) is quite general.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)