Group Fair Federated Learning via Stochastic Kernel Regularization
Authors: Huzaifa Arif
Pin-Yu Chen
Keerthiram Murugesan
Alex Gittens
Venue: Accepted by TMLR ()

Abstract: Ensuring \textbf{group fairness} in federated learning (FL) presents unique challenges due to data heterogeneity and communication constraints. We propose Kernel Fair Federated Learning (\texttt{KFFL}), a novel framework that incorporates group fairness into FL models using the Kernel Hilbert-Schmidt Independence Criterion (KHSIC) as a fairness regularizer. To address scalability, \texttt{KFFL} approximates KHSIC with Random Feature Maps (RFMs), significantly reducing computational and communication overhead while  achieving \textit{group fairness}.

To address the resulting non-convex optimization problem, we propose \texttt{FedProxGrad}, a federated proximal gradient algorithm that guarantees convergence. Through experiments on standard benchmark datasets across both IID and Non-IID settings for regression and classification tasks, \texttt{KFFL} demonstrates its ability to balance accuracy and fairness effectively, outperforming existing methods by comprehensively exploring the Pareto Frontier. Furthermore, we introduce \texttt{KFFL-TD}, a time-delayed variant that further reduces communication rounds, enhancing efficiency in decentralized environments.

Published in Transactions on Machine Learning Research (Apr/2025)
Group Fair Federated Learning via Stochastic Kernel Regu- larization
Huzaifa Arif
arifh edu
Rensselaer Polytechnic Institute, Troy, NY, United States
Keerthiram Murugesan
Keerthiram. www.rruruugggabmmmmmmmmmmm
IBM Research, Yorktown Heights, NY, United States
Pin-Ye Yu Chen
pin-gn.chenniibm.com
IBM Research, Yorktown Heights, NY, United States
Alex Gittens Rensseluer Polytochnic Insfibute, Trog NY, United States
gittea@rpi.eed edu
Reviewed on OpenReview: https: openreview. net/ forum? id- k8m4/uVIsI
Abstract
Ensuring group fairness in federated learning (FT) presents unique challenges due to data heterogeneity and communication constraints. We propose Kernel Fair Federated Learning (KFFL), novel algorithmic framework that incorporates group fairness into FL models using the Kernel Hilbert-Schmidt Independence Criterion (KHSIC) as fairness regularizer. To address scalability, KFFL approximates the KHSIC with random fcature maps, significantly reducing computational and communication overhead while achieving group fairness.
To address the resulting non-convex composite optimization problem, we propose FedProxGrad, a federated proximal gradient algorithm that guarantees convergence. Through experiments on standard benchmark datasets across both IID and Non-IID set- lings for regression and classification tasks, KFFL demonstrates its ability to balance accuracy and fairness effectively, outperforiing existing methods by comprehensively exploring the accuracy fairness trade-offs. Furthermore, introduce KFFL-TD, time-delayed variant that further reduces communication rounds, enbancing efficiency in decentralized environ- ments. Code available at ggithub. wwtp:/ccooocaaaaaaaaaaaa
1 Introduction
Unintended unfairness in machine learning models poses significant challenges, particularly in decision- making processes that impact specific population groups @Doork et al. 2012a Agarwal et al. 2019b Jalal et al. 2021). For instance, the COMPAS software, used in judicial decision-making for criminal offenses, has been shown yield unjust outcomes disproportionately affecting the African American community [rrsssel & Farid 2018 Barenstein 2019). Such findings underscore the need for model outputs to be fair with respect to protected demographic attributes like gender and race. Ensuring demographic fairness has therefore emerged as critical challenge in machine learning, driving efforts to develop robust solutions for mitigating bias and ensuring equitable model deployment.
The ure on fair federated learning often uses the term fairness ambiguously to refer to either client fairness or group fairness. However, these concepts address distinct objectives. Works such as Chaudhury ct al. (2022 : Li al. (2019); Donahue & Kleinberg (2021): ; Cui al. (2021); Du al. (2021) focus on client fairness federated learning, aiming ensure that the model performs equitably across clients' data (Mohri et al. 2019), thereby mitigating disparities arising from data heterogeneity among clients. In contrast. group

---

Published in Transactions on Machine Learning Research (Apr/202)
fairness (Ezzeldin et al. 2023 Papadaki et al. 2022) seeks to achieve fairness across different demographic groups. This involves esta esta blishing performance guarantees to ensure the global model is fair with respect to sensitive attributes, such as race or gender. More discussion on the related works is available in the Appendix This work addresses group fairness in the trained global model.
A.1
Recent advances bias mitigation algorithms (Jalal et al. 2021 Correa et al., 2021 Agarwal et al. 2019b;
Memarrast et al., 2023) largely depend on fairness regularizers incorporated into the training objective, which typically requires centralized access to data. However, in federated learning (FL) [Li ct al. 2020), where data is distributed across clients, privacy regulations and bandwidth constraints often prohibit raw data sharing, making centralized approaches to group fairness impractical. Furthermore, implementing fairnoss regularizers in the FL setting presents additional challenges, including communication overhead, computational costs, and data heterogeneity, all of which complicate training a globally fair model.
Discieikass
X
Fecking
pern)
Dann les Disenniuutee
Figure 1: FEDAVG can result in models whose predictions biased with respect to sensitive attributes such as race gender. KFFL is a principled approach designed specifically to mitigate demographic bias when training a model in a distributed setting.
Consoquently, prior efforts to achieve group fairness in foderated settings have primarily focused on aligning local and global fairness metrics (Ezzeldin et al., 2023 Papadaki et al. 2022). often avoiding the direct in- corporation of regularizer terms ensure statistical group fairness. However, this approach faces challenges, applying local debiasing mechanisms at individual clients alone is inadequate to ensure group fairness in the globally trained model.
We propose a novel approach to fair federated learning that integrates the Kernel Hilbert-Schmidt Inde- pendence Criterion (KHSIC) (Gretton el al. 2005b) as fairness regularizer into FL. KHSIC is a powerful measure of statistical dependence capable of capturing complex, non-linear relationships between variables, making it well-suited for enforcing group fairness in statistically principled manner.
The Hilbert-Schmidt Independence Criterion (HSIC) is effective fairness regularizer in centralized regres- sion models, as demonstrated in Pérez-Suay et al. (2017). KHSIC, the kernelized version of HSTC, captures complex, nonlinear dependencies among random variables through the use of kernel functions. Importantly, KHSIC provably quantifies statistical parity: a low KHSIC value between model outputs and sensitive at- tributes ensures approximate statistical parity (Kim & Gittens 2027 Compared to the closely related Rényi correlation (Baharlouci et al., 2019b), KHSIC more practical for mcasuring dependence bocause its em- pirical compulation reduces to BLAS Level 3 matrix calculations. This makes KHSIC not only theoretically robust but also computationally efficient in practical applications.
However, directly applying KHSIC in federated learning (FL) setup presents challenges, as it is compu- tationally expensive and communication intensive due to the necessity of computing and exchanging large kernel matrices. To address these limitations, we leverage Random Feature Maps (RFMs) (Rahimi & Recht 2007) to approximate kernel functions. This approximation significantly roduces both computational and communication costs, making the integration of KHSIC as a fairness regularizer more feasible and efficient in the FT. setting.
2

---

Published in Transactions on Machine Learning Research (Apr/2025)
Furthermore, to efficiently solve the distributed optimization problem incorporating the fairness regularizer, we introduce FEDPROXGRAD, a federated proximal gradient algorithm. This method ensures convergence for the non-convex composite optimization problems that arise from the integration of the fairness term.
Our contributions are as follows:
We propose KFFL, a novel federated learning algorithm that incorporates group fairness using KHSIC as a fairness regularizer. To the best of our knowledge, this the first work to adapt KHSIC for use in federated learning, addressing the unique challenges of the lederated setting.
To reduce the computational and communication overhead associated with KHSIC, we develop a communictioon-ffcienn approxination to the KHSIC using Random Feature Maps. This allows to avoid transmitting large kernel matrices, reducing communication costs by orders of magnitude.
We introduce FEDPROXGRAD, federated proximal gradient algorithm that provides convergence guarantees for non-convex composite optimization problems. FEDPRoXGRAD allows both terms of the composite objective to be non-convex, in contrast to prior works on federated composite
optimization (Wang & Li 2023 Bao et al. 2022 Yuan et al. 2021 Tran Dinh et al. 2021
We analyze the communication overhead of the resulting KHSIC-rogularieed fair federated learn- ing algorithm, KFFL, and introduce time-delayed variant, KFFI,-TD, which further reduces communication rounds while maintaining performance. This makes the method more attractive in real-world FL applications where communication must be minimized.
We conduct extensive experiments standard benchmark datasets under both IID and Non-IID data distributions on both classification and regression tasks. Our results demonstrate that KFFL effectively balances the trade-off between accuracy and fairness, outperforming existing baselines and exploring the accuracy fairness trade-offs more comprehensively. We recommend that practi- tioners use binary search to choose the fairness hyperparameter À KFFL and KFFL- TD to achieve their desired accuracy-faiin fairness trade-off.
2
Preliminaries
The goal of fair learning is to ensure that the model's output exhibits no undesirable dependencies on sensitive attributes.
We assume the observations are sampled i.i.d. from a joint distribution P(X, S) to obtain training data {x,, s. w Here, x, contains the non-sensitive covariates, s; contains sensitive covariates (which may be binary scalar 8; multi-dimensional vector s,), and y is the ground truth label for the i-th sample. This dataset is employed to train classifier f(x;; w), where denotes the model parameters) [
As an example, consider the task of training a binary classifier for making hiring decisions. Here x consists of features that are ethically and legally allowable for use in making hiring decisions, represents binary sensitive features such as the individual's sex or marital status, and the ground truth decisions are yi E (0. The classifier makes predictions 0s f(x;: w) E (0, 1}, where signifies decision to hire.
2.1 Metrics for Group Fairness
We employ two canonical group group-fairness fairness cotions-statisticn parity and equality of opportunity-boot both in their classic binary form and in a more general multi-group form needed for intersectional or many-valued sensitive attributes.
We follow the notation convention where boldface lowercase letters (e.g., v) denote vectors, non-bold lowercase letters (o.g., v) represent scalars, boldface uppercase letters (e.g., M) signify matrices. For example, the equation Mu + and vectors, M is matrix, and is scalar.
3

---

Published in Transactions on Machine Learning Research (Apr/202)
2.1.1 Binary Classification Metrics
Statistical-Parity Difference (SPD). A classifier attains statistical parity when the prediction is
independent of the sensilive attribute, 9 I1. 8 (Dwork al. 2012b). For binary attribute 8 (0, measure the gap
SPD Pr(g 1 = Pr(j 0).
(1)
Equality of Opportunity (EO). Equality of opportunity requires identical truc-positive rates
across sensitive groups (Hardt et al. 2016 similarly quantify EO by measuring the gap
RO = Pr(j y = 1, 1) Pr(O 1 11y 1, ==
(2)
2.1.2 -uoui Extension
When the sensitive attribute admits more than two categories several attributes combined (e.g. race and sex), we enforce the fairness constraints across all resulting groups. Let S = 21, KK with K23.
.......K
Definition (Multi-group Stalistical Parity (woork et al. 2012b)). Define rs Pr(ů 8). The multi-group SPD gap is
mule:
max f #ES
min r. #ES
max s.s'ES
T. rel.
(3)
Definition 2 (Multi-gooup Equality of Opportunity 1, 8).
Hardt et al. 2016 Let TPR( = Pr(ý 1
EO = max
TPR T (1)
(4)
Lower values of SPDmulti and FOmulii indicate greater fairness; both reduce to Eqs.
1 when =
If several sensitive attributes are recorded (e.g. race and sex) one may: (i) concatenate them into an inter- sectional label S (race, sex) and apply Definitions I 2 to the resulting K = II K2 groups, or (ii) compute the same gaps separately for each attribute and treat the model fair only if fairness with respect to each attribute meets the chosen tolerance. The choice between joint and per-attribute constraints depends on the policy requirements of the application.
Although our algorithm is designed to target SPD, as is common in the literature we use both SPD and EO to evaluate classifier fairness empirically Section 6 We report both the binary metrics (Eqs. and their multi-group counterparts (Eqs.
2.1.3 Fairness Metrics for Regression
Fair regression has been extensively studied in centralized settings (Chzhen et. al.] 2020a Agarwal el al. 2019a; Chzhen et al. 2020b). However, the training of fair regression models in the distributed settings remains underexplored. Fairness metrics like EO and SPD that are designed for classifiers are not applicable in this context. Instead, the Kolmogorov-Smirnov (KS) distance has been employed to evaluate regression fairness (Chzhen et al. 2020a).. This metric captures the maximum disparity between the distributions of the model's predictions for different sensitive groups.
To formally define the KS distance, we denote the set of indices of samples belonging a sensitive group S by
I' =- (1, : = 8}.
The empirical cumulative distribution function (CDF) of the model's predictions for group is
F (t:
1[f(x;w) St),
(5)
I T
where f(x;:) is the model's prediction for input X,-
4

---

Published in Transactions on Machine Learning Research (Apr/2025)
The KS distance between the predictions for two distinct sensitive groups and s' is
KS(w)= =
max sup (t w) - E (t: S ER
(6)
A smaller KS distance indicates lower disparity between the regression results across the different sensitive groups. In fact, the KS distance is commonly used in the KS test, statistical test for the equality of one-dimensional probability distributions.
Appendix provides experimental results on using the KHSIC regularizer for rogularization tasks, where the resulting fairness quantified using the KS distance.
2.2 The Kernel Schhedt Independence Criterion (KHSIC)
The Kernel Hilbert-Schmidt Independence Criterion (KHSIC) Gretton et al. (2005a) is key to our approach to ensuring statistical parity. The KHSIC is predicated on the observation that and s are independent if and only if every function of 9 is uncorrelated with every function of
Given two Reproducing Kernel Hilbert Spaces F and G, the (population) KHSIC is the Hilbert-Schmidt norm of the centered cross-covariance operator Cjs of I and (Gretton et al. 2005a):
4p. lICo >Σ Cov (f
Here ( and [g are orthonormal bases for FF and 9, respectively. It follows from the bilinearity of the covariance operator that the KHSIC quantifies the dependence of ò and s by bounding the covariance of every function of 9 in F with every function of in G.
Under some regularity conditions F and G (universality), the population KHSIC is zero if and only if g and 8 are independent. Thus when the KHSIC is zero,
SPD = Pr(ů = = - Pr(j = 1 = Pr(j = - Pr( 1) 0,
the classifier achieves statistical parity. More generally, the population KHSIC gives an upper bound the total variation distance between the joint distribution Py,, and the product of the marginals Py & P., and thus quantifies the dependence of ý and (Kim & Gittens 2021
The KHSIC is more practically useful as a measure of dependence than the closely related Rényi correlation: the latter is defined in terms of the probability density functions (PDFs) of ÿ and s. This approach does not scale to high figh-dimensional inputs because accurately estimating the PDFs of ÿ and s requires an exponential number of observations relative to their dimensionality. By comparison, empirical estimation of the KHSIC reduces to simple and scalable linear algebraic computations. The empirical KHSIC is derived in Gretton
et al. 2005a):
1 Vemp (8,8) (n 1)2
Tr HH,,HKKHH),
where
Ky = [K (3..0)))) 0y)) K; [ng(s sy)lis=1 = - 11 n
The functions Ky and Kg are the kernels associated with the RKHSes F and G, respectively, and the matrices Ky and Kg are n x n kernel matrices evaluated on the predictions and the corresponding sensitive attributes. Gretton et al. (2005a) show that Vemp (ý, s) is a - -onsiisten estimator of (pp s).
We propose to measure the fairness (in the sense of statistical parity) of models by using the empirical KHSIC:
X,S)
Tr (KK,HKyH))
(n -
Forr notational brevity, notation refer two potentially different kernel functions the features and the sensitive variables.
5

---

Published in Transactions on Machine Learning Research (Apr/202)
The notation (w X) emphasizes that the fairness measure depends on through the predictions ý ((;:)).
2.3
The Fair Learning Objective
To learn fair model w. by using (w:; X, as a regularizer, we solve the optimization problem
argmin
Evn, f(x;; Xb;
(10)
The first term in the objective function represents the loss E(Ve, [xx; W)), which quantifies the discrepancy between the model's predictions and the ground truth labels. This loss function may be instantiated the cross-entropy loss for classification tasks or the mean squared error (MSE) for regression lasks. The second term acts as a fairness regularizer by measuring the dependence between the model's predictions and the sensitive attributes using the KHSIC criterion. The fairness paramcter À controls the tradc-off between optimizing predictive performance and enforcing fairness.
Solving (10) in a centralized setting is conceptually straightforward but computationally challenging because, at each iteration, it involves computing forward and backward passes (for backpropagation) through matrices Ky(w) of size X The computational burden is compounded in the federated learning setting by the additional communication burden: clients need to communicate these n x matricos to compute their local contributions to the gradients. Hence, a straightforward adaptation of this centralized approach to a distributed setting is prohibitive from the perspectives both computation and communication. We address this challenge in Section
In addition to the challenges posed by the specific form of v, the composite structure of optimization problem 11) is itself challenge, both terms in the objective are non-convex. We introduce FedProxCrad in the next section (Section 30. to address this challenge.
3 Composite Optimization in the Federated Setting
Employing the fair ML formulation of equation (T)] in a federated setting requires solving federated composite optimization (PCO) problem that can be written in the standard form
argmin Σ (w) + (bw).
where the Ø are data-fitting terms local each elient, and &o is the global fairness regularizer. For brevity, we have absorbed the regularization constant À into v. We denote the sum of local-data fitting terms with E(w) Em (w) and the composite objective with F(w) E(W) + v(w). In Equation 110 the fairness term 16 is non-convex and the data fitting term, e, may also be non-convex. This section provides an algorithm for solving the resulting non-convex FCO problem.
Several existing works provide algorithms for the FCO problem- Gaablem-Wangga & Li 2023 Bao et al. 2022 3 Yuan et al. 2202)) develop algorithms that. require the & and de to be convex, while the algorithm of Tran Dinh ct al. 2021 requires only sa to be convex-but, to our knowledge, no extant optimization algorithms for FCO guarantees convergence for problems where the & and are both non-convex. To fill this gap, we introduce FedProxGrad, federated proximal gradient descent algorithm. Unlike prior FCO algorithms, FedProxCrad imposes no convexity requirements.
The FedProxGrad algorithm, described in Algorithm[I [ exlends the stochastic proximal gradient algorithm in a straighttorwwad manner from the centralized setting to the federated setting. We note that the FedProxGrad algorithm differs significantly from the FedProx algorithm: FedProx is a federated prozimal point algorithm designed minimize a single objective, while FedProxGrad is a federated prozimal gradient algorithum de- signed to minimize composite objective.

---

Published in Transactions on Machine Learning Research (Apr/2025)
Algorithm Federated Proximal Gradient Descent (FedProxCrad)
1: Input: Wu, T.
2: for t 0
do
Server computes gi, stochastic gradient estimator for Val(wi), and computes w/41/2 w. ag:
4: Server sends we+1/2 to clients
Each client for E (I,...,, m) computes
wit argmin (' (w) + _ -| w111/2l2
1
2xx
6: Each client returns wi+1 back to the server
7: Server aggregates the device models to form
We++
wH41-
m
1=1
8: end for
Nonetheless, the analysis of the convergence FedProxGrad follows closely that of the convergence rate of FedProx, so we introduce the same notions used in its convergence (Li et al. 2020).
1 ||w w/21/2 (see Algorithm for the definition of Definition 3 (y-suboptimalitt). Let (w)) (w)) + 2a
W1+1/2). Given 7 E [0, IJ, a point @ is a 7-suboptimal solution of argmin (w) if V [b) s (w>)]] Smaller y correspond to higher accuracy.
This definition of y-suboptimality slightly differs from that used in Li et al. 2020 to facilitate the analysis of a composite objective. This condition on the local solvers ensures that the local solution for the (t + 1)th iterate a factor of y less suboptimal than the tth iterate. This condition is agnostic to the particular solvers employed, and can be achieved using deterministic or randomized solvers that use full gradients or stochastic gradients.
Definition 4 ((G, B)-Bounded Dissimilarity (Definition A1 Karimireddy et al. 2020 )) The local func- tions e are (G, boundedyy dissimilar at w if VV + Ve(w) 1" (F(w) G
This condition is standard Li et al. (2020): Karimireddy et al. 22200), and ensures that local progress on the individual clients can be translated to global progress. The bounded dissimilarity condition mirrors common optimization assumptions (e.g., bounded Tipschitz or bounded Hessian assumptions): when violated by taking B + co, the allowed step size shrinks, and the convergence rate degrade
Theorem 1. Assume that the functions E', tb, and F are L-smooth; the functions are L L_-weakly convez; F is bounded below by a constant F*; the bounded dissimilarity condition (Definition and that the stochastic gradient estimate, for the fairness regularizer satisfies Elg, wi] Vib(we) and
4 holds with = 0;
Ellg: - V 0²
for all w. If the local solvers on each client ensure 7 suboptimal solutions (Defonition with porameter I 1 and the global stepsize is chosen to satisfy s(B+1)
a min min
20 2L 120L + +1) 5LB²
then the sequence of iterates generated by FedPronGrad satisfies
I E [[I < F(Wo) F
aT 402
T
3 reviewer astutely pointed that the bounded dissimilarity assumption may possibly avoided of reduced convergence by using the approach of (Yuan Li 2022

---

Published in Transactions on Machine Learning Research (Apr/202)
This result shows that, for appropriate choice of hyperparameters, the FedProxGrad algorithm converges at rate of + up to the noise level of the stochastic gradient. A proof is provided in the Appendix A.2 We note that only recently has a nonasymptotic rate of convergence of stochastic proximal gradient algorithms for fully nonconvex composite optimization problems been established in the shared memory setting (Xu jet al. 2019). Theorem shows that a natural generalization of the stochastic proximal gradient algorithm to the fully nonconvex foderating sctting also converges at a sublinear rate.
4 Communication-Efficee Kernel Regularized Fair Learning
In the centralized setting, kernel methods pose a computational challenge due to their inherent complexity, roquiring optimization with a kernel matrix incurring a computational cost of up to O(n)). One line of research for reducing this burden, starting with the seminal work of |Raiiii Recht (2007), uses random feature maps (RFMs). random feature map for a infartiivaaiann kernel function Φ RD - RD that is constructed to satisfy x(x,y) [()(x), o(y)) for any two vectors and y, where the expectation is laken over the randomness in Ø.
random function
RFMs enable the efficient formation of randomized low-rank approximations to kernel matrices. In particular, if the rows of Zg E R= consist of the application of an RFM ø to the s;, and the rows of Z y(w) E R likewise consist of the application an RFM to the observed x,, then
Ks = 25 and K K(()) F nw)
(11)
and the variance of the approximations go down as the number of random features D increases, 80 ZsZ and )(w 2T,w are principled randomized low-rank approximations to the corresponding kernel matrices. A
substantial body of work has demonstrated that these approximations exhibit theoretically and empirically
similar performance to full kernel matrices (Hamid ct al. 2014, Rahimi & Recht 2007 Yu ct al. 2016 More details on the ruction of randomized feature maps is given in Appendix
We utilize these randomized low-rank approximations to efficiently compute principled approximations to the regularizer (bw) in Equation 10 Note that Z f(w) and Zg have dimensions x D where D < n. Specifically, we utilize the Orthogonal Random Feature Maps (ORFM) of Yuu et al. 2016
The first crucial observation is that by using RF,s, one need only communictte a D matrix to approx- imate (ww) and its gradient g(w), rather than communicating two n x kernel matrices.
Theorem 2. Let I(w) and Zs be the x D matrices constructed using RFMs, corresponding respeclively to the prediction kernel K K(()) and the sensitive attribute kernel Ks. Then ((w) E [[]G(w)) where G(w)= = 2 ((w) - (w) DDDD Here, p2 the mean over the rows of Zs and μ is the meon over the rows of Zz(w)-
Corollary 1. g(w) = G)) is an unbiased slochastic estimate of V)(().
Proofs of these results, consisting of basic linear algebraic manipulations, are supplied in Appendices A.3 and A respectively. In the remainder of this section, we detail how G(w), and consequently the gradient estimate g(w), can be computed efficiently in the federated setting.
The next important observation is that in the federated setting, the RFMs Zs(w) and Zs are naturally partitioned across the clientss:
Zg,1 :
f((x)=
and Zg
(12)
Z Z(()).m
Zs
where Z Z((x),: Zs, E D Here n, is the number of data points on client i, so n Σm n4. This observation allows each client to, independently from the other clients, efficiently compute its contribution to the feature interaction matrix G(w))
8

---

Published in Transactions on Machine Learning Research (Apr/200) 2025)
Lemma 1. The global feature interaction matrix G(w) con be partitioned into local interactions
m G(w) = Σ 25x),
m
Σ
m
n
7
where Msa and My are the row averages of Zs, and zx()),, respectively.
As a consequence of this result, the server can compute the global interaction term G(w) after each client transmits its local feature interaction matrix s f(x), E RDD along with its local average feature vectors Ms Ms. E RRD to the server. This approach facilitates the computation of ((w) and g(w) in a manner that does not require communication between the clients.
The unbiased estimate of the gradient of the fairness regularizer from Corollary be partitioned into local interactions:
Lemma 2.
m
g(w) Σ Jn. (T)TT(())
where Ri(w) = Zz,Z - n.Mss and Jn, (w) is the Jacobian of S2 with respect to the model paramelers w.
This result states that once the server computes the global interaction matrix G(w) and distributes it to the clients, each client can then independently compute its contribution to a stochastic estimate of the gradient of the fairness regularizer (bw). Proofs of these lemmata are provided in Appendices A.4 and A.6
Therefore, by using random feature maps, instead of transmitting n X n matrices, only D D matrices need be sent to compute unbiased approximations to v and Vy. This results in substan- tial reduction in communication costs. For instance, consider training fair model on the ADULT dataset (n 32K). Choosing D = 1024 (the dimensionality of the RFMs used in our experimental evaluations), computing (b(w exactly requires communicating 32K x 32K matrices, compared to 1K x KK matrices when RFMs are employed to estimate (ww). In this example, the communication cost is reduced by three orders of magnitude!
The use of randomized projections to reduce the cost communication is well-established. We note particular (Han et al. 20024): this work uses randomized projections to help in their aim of achieving perfor- mance fairness (see Definition 1 of (Han al., (024)), ), by minimizing the variance in the client objectives; this concept of fairness differs from that of group fairness, which is our focus. Our contribution is in applying random projections to achieve group fair federated learning in a statistically principled manner.
5
The KFFL algorithm for rerulle-egularried Fair Federated Learning
We now introduce the KFFL algorithm, which uses the FedProxGrad method to implement kernel-regularized fair Icarning using the approximation to & introducod the previous section. A time-dclayod variant that stale fairness gradient information to incur less round of communication per iteration, KFFL-TD, is
presented in Appendix C
The KFFL algorithm is detailed in Amgorithm 2 which gives the client-side procedure, and Algorithm 3 which gives the server-side process. Equations
13 through 16 are referenced in these algorithms. At a high level, KFFL uses federated composite optimization to fit a fair model in three rounds:
FAIR1 At the start of the (t 1)-th iteration, the clients use the RFMs to compute the local terms ('(w) needed to compute the interaction matrix G(w,) for the current global model wi, and communicate them to the server. Clients also store the local interaction term for next round (ww,). The server combines the local terms to compute the interaction matrix Gww,) on the current global model, and returns these to the clients.
FAIR2 The clients use (t)))
from the server and the local interaction term ((,,,) previously computed to compute their contribution to the stochastic estimate of the fairness gradient and communicate these to the server.
9

---

Published in Transactions on Machine Learning Research
KFFL
SERVER
ran Sewnellpshiii
CLUINT
CIIENT 3
CLIENTS
Figure 2: The communication pattern for KFFL. Different colors correspond to the FAIR1, FAIR2 and Local Update sub-rounds of KFFL. The direction of the arrows indicate an uplink or downlink communication and the width of each arrowhead highlights the communication cost in each sub-round. Thicker lines indicate (large) communication costs on the order of the size of w, while thinner lines represent communication costs on the order of D
Update The rest of the iteration implements FedProxGrad: the server computes the gradient estimate for & and sends W = w, - gr to the clients. The clients locally update their models wH41 and send them back the server, which then computes the next global model W(+1-
Figure f graphically depicts the rounds of communication in each iteration of the KFFL algorithm. The following expressions are used in Algorithms 3 and 2
Equations used in KFFL/KFFL- TD
G(w) z Zy(w) m1 (w)
(13)
M' (w) = 2, z,,(w)
(14)
Ω, M'(w) mHHeH
(15)
g' (w) Jnz (w) G(w) m nilhes
(16)
μ,
(17)
m nip
11
(18)
ZZ.; 74 Mg
(19)
1
54 n4
Zi,i
(20)
W+11 argmin
filwe)) + 2ac
|| wi
(21)
Here Update is written as short the Local Update Blag
10

---

Published in Transactions on Machine Learning Research
Algorithm 2 KFFL Client Side
Input: (ROUND.-.)
if ROUND FAIR1 then
2: Clients compute M' M'(w) (we) (see Equation 14 using shared random seed Ç to generale their RFM, (,,(() using
Equation and (t) using Equation
3: Combine terms Ø (w:) {M'(w) (we), M 1.a
Return: & (w:)
5:
else if ROUND FAIR2 then
Client compute local interaction for gradients using Equation gel 52 (wi)
7: The clients then compute the local gradient g' (wx) using Equation 16 from (ww))
Return: g (we)
9: else if ROUND = Local Update then
10: Clients do a local update on Wc+1/2 following Equation to get w
11: Return: wiy1
12: end
Algorithm 3 KFFL Server Side
1: (This is the initial model]
2-40
3: while not converged do
4: for all 1.....m in parallel do
Generation of random seed
6: (w(,)) = Client
end for 8: ((n)) = (0)()))
From ((w:) compute G(wr) using Equation
13 (t,(t) using Equation and (t using Equation
10: for all l...., m in parallel do
A(w:) (FAIR2, (t)),,
g" (w:)) Client Update(A(wz))
13: end for
Wr+1/2 t w: Em" gw())
15: for all = 1..... in parallel do
wi+1 Client Update(Local Update,
17:
end for
18: W1+1 + average (wi+1)
19:
end while
The following section evaluates the empirical performance of KFFL and KFFL-TD against baseline fed- erated learning algorithus designed to mitigate demographic bias, utilizing the fairness metrics introduced in Section Furthermore, we examine the communication costs of KFFL and KFFL-TD relative to these basclines.
6 Experimental Evaluation
In this section, we evaluate the performance of our methods, KFFL and KFFL-TD, at achieving group fairness the federated setting. Fairness assessed using statistical parity, ① and equal opportunity, ( for classification models and the KS distance, (6). for regression models. While most of the work fair federated learning has explored fairness a algorithms that aim to achieve client fairness (i.e., consistent per- formance across clients), as in Cui et al. (2021) and Du et al. (2021), such algorithms do not directly target statistical group fairness. Thus, those algorithms are not suitable benchmarks for our approach, as we focus explicitly on addressing statistical group fairness in the federated learning.
We compare KFFL against three baseline methods: FedAvg McMahan et al. 2017): ; the MinMax algorithm of Pappdakk et al. (2022), which aims optimize model performance for the worst-performing demographic group: the FairFed algorithm of Ezzeldin al. (2023). where clients convey localized fairness metrics and the server optimizes weighting coefficients by minimizing the contribution of the poorest-performigg client
11

---

Published in Transactions on Machine Learning Research (Apr/2023)
for chosen fairness metric. The framework of FairFed allows for different local bias mitigation algorithms. We use FairFed with the best performing local bias mitigation algorithm, FairBatch (Roh et al. 2020). The latter two baselines were chosen as these methods are explicitly designed to mitigate demographic bias in federated learning. To highlight the importance of global communication of fairness information, we also consider FairFed-Kernel, which uses FedAvg with each local client. implementing a local bias mitigation algorithm (similar to (0 Finally, KFFL is compared against time delay variant KFFL-TD.
The selection of the hyperparameters for KFFL and its variants is discussed in Appendix B.1 This section includes a discussion of both common hyperparameters (e.g., batch size, learning rate, local epochs, global rounds) and algorithm-specific hyperparametors, such as the feature map size D used in KFFL and KFFL- TD.
We evaluate the performance in two different federated learning settings: IID (independent and identically distributed) and Non-IID. In the IID sctting, each client is provided with an equal number of samples and a shared local data distribution Li et al. (2020). In the Non-IID setting, each client has different distribution of the protected attribute. Specifically, since the protected group A is binary with attributes Ao and A half of the clients have 90% of Ao and 10% of AI, while the other half has 90% of A and 10% of Ao Li ct al. 2020).
For classification tasks, we used five datasets commonly encountered in group fairness research (Han et al. 2023): Adult, COMPAS, Bank, ACS, and German. Logistic regression and neural networks are used to evaluate the fairness accuracy trade-off of KFFL (Han et al. 2023). Additional results and details can be found in Appendix [
When the underlying task is regression, incorporate additional datasets into our evaluation. Beyond the Adult dataset, we also consider the Law School and Communities and Crime datasets, as utilized in the work Agarwal et al. (2019a). For more details on the datasets refer to Appendix B.2
A note on the scope of the experimental evaluation. Our primary focus in this work the development and evaluation of a novel kernel-based approach to ensuring group fairness in a principled manner. Similar prior works in federated learning that do not impose privacy constraints (Crawshaw & Liul 2024 Cho trivacy-presevvig mechanisms in this study. The integration of privacy-preserving protocols such as DP-SGD (Chua el al. 2024) or secure aggregation
et al. 2023 Gu et al.] 2021 Malinovsky et al.) 2023 Eichner et al. 2019), use
(Bonawitz et al. 2017 is left as future work. Furthermore, we consider full client participation, consistent with approaches such as FairFed (Ezzeldin et al.) 2023), the MinMax algorithm proposed by Papadaki et al. 2022), and other works in federated learning (Zhang et al., 2023 Li et al. 2023a by Zakerinia et al. 2023 Huang et al. (022)). Finally, as wall-clock time differences may sensitive difierences in the implementation efficiency of our methods versus the baseline methods, our experimental
evaluation focuses communication costs rather than wall-clock time. This focus is justified by the fact
that the performance of federated learning techniques is typically conmunication-oounde
6.1 Performance Evaluation of KFFL
Table D compares the communication rounds required by our proposed methods, KFFL and time-delay variant KFFL-TD see (), with baseline methods FedAvg and FairFed Ezzeldin al. (2023). While KFFL
Table 1: Communication measured in terms of the number of rounds required for one global update using KFFL, KFFL-TD, FedAvg, and FairFed. Algorithms that incorporate fairness, such as FairFed, require a similar number of communication rounds as our methods KFFL and KFFL-TD.
Method
Rounds of Communication
KFFI.
3
KFFL-TD (Time Delay Variant)
2
FairFed Ezzeldin et al.
2023
3
FedAvg McMahan et al
2017
1
12

---

Published in Transactions on Machine Learning Research (Apr/2025)
Det RITMV
Demographi
Eoo lnd Opporiiity
Essal 203 Corerbinty
Eeoslzed Opeartuiity
14
Figure 3: Accuracy versus Demographic Parity (DP) and Equalized Oppurtunity (EO) for KFFL and the basclines under IID and Non-IID conditions on the ADULT test dataset. Each point represents a different fairness weight À ranging from 20.00 1000.00 for both KFFL and KFFL-TD. The blue region denotes higher levels of group fairness (low SPD and EO), while the gray region indicates lower levels of group fairness (high SPD and E.O.
Oececied hocee face
Derrggaphie Party
Demagraphic Patly
Deroggastic Party
Soun Ind Opparauity
Equal Oypertunty
Equakeed Oppontrtty
Figure 4: Accuracy versus Demographic Parity (DP) and Equal Opportunity (EO) for KFFL and its baselines under IID and Non-IID conditions on the COMPAS test dataset. Each point represents a different fairness weight X ranging 0.01 to 123.16 for both KFFL and KFFL-TD. The blue region denotes higher levels of group fairness (low SPD and EO disparities), while the gray region indicates lower levels of group fairness (high SPD and EO disparities)
requires a similar number of communication rounds as FairFed, KFFL-TD reduces the number of rounds needed for a global model update by using stale fairness gradients, thereby lowering the total communication rounds needed per update.
Existing fair classifiers and regressors that balance fairness and accuracy typically optimize for a single point the trade-off curve (see Figure 4 and Figure 3 whereas KFFL and KFFL-TD explore the trade-offs between accuracy and group fairness more thoroughly as the fairness hyperparameter À is swept. This suggests that the baseline algorithms may lead to suboptimal outcomes when practitioners employ them to achieve specific trade-offss between accuracy and group fairness, while KFFL and KFFL-TD are more suitable for achieving more control over the achieved trade-off.
For the classification tasks, we evaluate the models' test accuracy and fairness metrics, focusing on SPD, 1 and RO, (2) across five datasets: BANK, ACS, COMPAS, ADULT, and GERMAN. Each dataset. considers a single
13

---

Published in Transactions on Machine Learning Research (Apr/2025)
protected binary sensitive attribute. For example, in COMPAS, the protected attribute is race (black/whitee while in ADULT, it is sex (male/female). This subsection presents results for the logistic regression model, with the Appendix D providing similar results on the performance of the neural network model.
KFFL and its baselines are compared under both IID and Non-IID conditions on the COMPAS test dataset in Figure 4 The blue region, referred to as the 'Low SPD Region' and 'Low EO Region,' represents low statistical parity discrepancy and low equalized opportunity discrepancy. These regions correspond to higher levels of group fairness. In contrast, the gray region, labeled as the 'High SPD Region' and 'High EO Region," indicates lower levels of group fairness.
Points labeled "completely fair" indicate trade-off points where the model achieves no statistical parity or equalized opportunities discrepancies. In contrast, points labeled "completely unfair" represent the perfor- mance of the standard FedAvg model, which trained without any fairness objective (i.e., = in the distributed sctting of (I0)). Each point reflects different fairness weight X, ranging from 20.00 to 1000.00 for both KFFL and KFFL-TD using the ADULT datasel and from 0.01 to 123.16 for the COMPAS dataset. More details on the choice of À for the other datasets is given in Appendix [35] As À increases beyond this range, observe non-optimal points to the right of the "completely unfair" point.
It can be seen that other baselines, such as FairFed and MinMax, do not produce a smooth trade-off between accuracy and fairness. For FairFed (Ezzeldin et al. 2023), the tradeoff is controlled by a parameter , called the "fairness budge", which varies from 0.1 to 5, based on the recommendations in Ezzeldin et al. (2023) Optimal performance of the FairFed baseline requires local debiasing mechanism, for which we use the FairBatch algorithm (Roh et al. 2020
For MinMax, a "global adversary rate" Papadaki et al. 2022) is used to control the reduction of expected loss for the worst-performing demographic. To explore the accuracy-fairnes tradeoff, we varied this parameter from 0.001 to 0.1 based on the settings in that work.
In the right column, the Centralized method refers to a non-distributed data setting, corresponding to Equation 10 where the full kernel used as a regularizer. Similarly, Figure 3 provides an evaluation for the ADULT test dataset. Each point represents different fairness weight X, ranging from 0 to 0.01 for both KFFL and KFFL-TD.
The iied version for each dataset illustrates that using the full kernel allows for a clear tradeoff between accuracy and fairness. The federated KFFL algorithm preserves this trade-off in Figures 4 and 3 Specifically, KFFL and KFFL-TD maintain competitive accuracy (approximately 0.85 to 0.78) on the ADULT dataset and between 0.6 to 0.7 on the COMPAS dataset, across different fairness regions. contrast, FairFed and MinMax perform similarly to our methods regions with lower levels of group fairness but significantly underperform in regions with higher levels of group fairness, often failing provide any trade-off points in the areas identified as high fairness regions.
The smooth exploration of the achievable accuracy-fairnes trade-offs by KFFL and KFFL-TD persists even in the more challenging Non-IID setting. Unlike FairFed and MinMax, which require extensive tuning for specific evaluation metrics, KFFL and KFFL-TD perform well without metric-specific hyper- parameter optimization. However, the choice of the fairness weight hyperparameter À is crucial for effectively utilizing KFFL and KFFL-TD. Inercasing the fairness weight beyond certain threshold can push the model's performance towards lower levels of group fairness, as shown in Figures 4 and 3
Figures 3 and also illustrate the tradeoff achieved when KHSIC is applied as local fairness regularizer in the method -airFed-Kornel. In these methods, each client independently solves (10) without global communication of fairness gradients. The results clearly show that local debiasing methods fail to achieve a
tradeoff comparable to the centralized approach, where distributional information from across all clients is
taken into consideration using (10). These findings underscore the importance the principled strategies employed by KFFL and KFFL-TD.
14

---

Published in Transactions on Machine Learning Research (Apr/2025)
Table 2: Communication costs of KFFL and KFFL-TD, relative to FedAvg. Here, é 2 D2 D2/Ww]. The KFFL and KFFL-TD algorithms incur additional cost due to the exchange of fairness information, however, the use of RFM ensures that this additional cost is vanishing when the model size |w| is large.
Method
Uplink
Downlink
KFFL
2
2
KFFL-TD (Time Delay Variant)
2 2
2+
Fair Fed (Ezzeldin al.
2023
FedAvg (McMahan et al.
2017
6.2 The communication cost of KFFL
KFFL and KFFL-TD incur additional communication overhead over FedAvg, due to the exchange of the parameters needed to compute the fairness gradient. Table - compares the relative communication costs (per client, per iteration) for the FedAvg and KFFL algorithms. Let |w] denote the size of the model. In each iteration, FedAvg incurs uplink (client-to-server) and downlink (server-to-client) communication costs of w|, the number of parameters in the model. In contrast, KFFL requires the client to transmit w (wi), and (wwa) (sce Algorithm 21. resulting in an uplink cost of 2/w| + D2, and the server to transmit wt A(w)), and W/+1/2, incurring a similar downlink cost of 2/w] + D2 (see Algorithm 3). Notably, D2 is oflen smaller than [w), particularly for large models. KFFL-TD (Appendix [ incurs comparable communication costs but reduces the number of communication rounds per iteration (sce Figure 7
[w] where D is the dimension of the RFMs. This results in total uplink and downlink communication costs of 2 + per iteration for KFFL, compared to I for nonfair methods like FedAvg. Since e is relatively small when D << [w], the additional communication cost becomes negligible in practice for large models. Consequently, the overall communication costs of KFFL and KFFL- TD remains approximately two times that of FedAvg, due to the additional exchange of model parameters, while providing the benefits of ensuring learning a model that is fair with respect to different demographic groups.
To quantify this overhead, Table defines E _ D²
FairFed also exchanges local and global accuracy and fairness metrics during each global update, but we exclude these scalar quantities from the comparison for simplicity in Table
6.3 Additional examples of KFFL trade-offs
Table TableBhigghtn highlights selected trade-off points for additional datasets, including BANK, ACS, and GERMAN, alongside the previously analyzed COMPAS dataset. The models were trained under the Non-IID setting, and the trade- off points were computed based on performance on the test dataset averaged over three runs.
To showcase the benefits of approach, selected specific trade-off points. For the COMPAS dataset, if the desired accuracy is around 60%, similar to the performance of FedAvg, KFFL, and KFFI.-TD achieve a small SPD gap of 0.06. In contrast, the baselines achieve a similar SPD gap but with an approximately 5% drop in accuracy. similar trend is evident in the ACS dataset, where KFFL-TD provides a trade-off point of relatively high 80% accuracy for low SPD gap. For the baselines, a comparable fairness point results 22% drop in accuracy.
Additionally, KFFL is robust across different evaluation metrics. On the BANK dataset, KFFL and KFFL- TD offer trade-off point of 90% accuracy with SPD gaps of 0.05 and 0.04, respectively. In comparison, FairFed achieves 0.00 SPD gap with 88% accuracy but as consequence exhibits a high EO gap of 0.16, as all methods were optimized for reducing SPD in our experiments. This underscores the sensitivily of the FairFed method to the choice of the evaluation metric. Additionally, MinMax shows a high SPD gap but low EO gap, indicating that depending on the dataset one evaluation metric may be favored over the other with this approach. In contrast, KFFL and KFFL-TD provide trade-off point that reduces unfairness across both evaluation metrics for this dataset.
15

---

Published in Transactions on Machine Learning Research (Apr/202)
Method
COMPAS
BANKK
AC3
GERMAN
FedAvg
61.13 1.25
91:21 = 1.10
51.14
1:20
12.50
1.15
XPFL
60.30 = 1.30
90.23
1.25
T9.12
1.35
72.50
1.10
Acc. (1)
XFFL--TD
59.51 = 1.45
90.78
1.20
51.12 +: 1.05
72.50
1.40
FoirPad/FairBatch
88.05
1.25
Fed-KKree
44.13 + 1.55
88.77
1.15
68.77 + 1.06
70.00
1.35
NinMax
36.88
1.40
1.20 64.93
58.77
1.25
70.00
1.10
FodAvg
XPFL
0.06 0.01
0.05
0.03
0.01 = 0,02
0.00 + 0.04
SPD (L)
RPFL-ID
0.05 :: 0.00
0.06 0.01
0.00 1 0.03
0,00 1 0.02
FeinFed/FsirBatch
PairFad-Kernal
0.08 + 0.02
0.00 0.03
0.00 + 0.02
0.00 + 0.01
NLnMax
0.03 0.01
0.21 0.02
0.03
FedAvg
0.23
0.16 0.01
0.02
XPFL
0.07 1. 0.01
0.04 0.02
0.02 1. 0.00
0.00 1 0.02
BO (3)
REFL-ID
0.08 = 0.002
0.03 0.01
0,03 0.02
0,00 1 0.03
FairPad/FairBatch
0.09 0.02
0.16 0.03
0.00
0.00 1
FairPed-Keese Kereeel
0.05 = 0.03
0.00 + 0.02
0.00 = 0.01
0.00 + 0.03
Table 3: Selected fairness accuracy trade-off points for the BANK, ACS, GERMAN, and COMPAS datasets non- IID settings. The results are averaged over three runs on the test sets. Lower SPD and EO gaps indicate higher fairness. KFFL and KFFL-TD provide superior trade-off points compared to basclines.
6.4 KFFL provides fair trade-off points for regression tasks
Fairness hyperparameter
RMSE
KS distance &
0.00
0.49088
0.009294
0.38293 0.016433
0.01
0.49101 0.009303
0.39744 0.027059
0.10
0.49053
0.009130
0.37377
0.030239
1.00
0.49071 0.009039
0.38429
0.053008
500.00
40.32616 48.938383
0.27088 03939738
Table 4: RMSE and the KS distance with standard deviations for 5 runs KFFL for the Adult dataset in the IID case. Points with lower RMSE and lower KS distance are preferable.
Fairness hyperparameter
RMSE
KS distance
0.00
0.810188 + 0.002067
0.439208 + 0.175972
0.01
0.810068 0.002152
0.424060 + 0.164314
0.10
0.809940 0.002189
0.487244 + 0.139673
1.00
0.810052 + 0.002201
0.200248 + 0.170934
5.00
0.810100 0.002022
0.212672 + 0.065094
100.00
31.159116 19.366019
0.184708 + 0.081729
Table 5: RMSE and the KS distance with standard deviations for runs of KFFL for the Law School dataset in the Non-lID case. Points with lower RMSE and lower KS distance are preferable.
We evaluate KFFI, with a linear regression model, measuring predictive accuracy by the root mean square error (RMSE) and group fairness by the KS distance (Eq. 6 lower values are better for both metrics.
Table reports performance under increasing fairness weights À in the IID setting the Adult dataset. As À grows, the fairness term gains influence. For small to moderate weights (A < 1) the KS distance falls by roughly 15%, while RMSE is statistically unchanged. However, once the weight is large enough (A> 1) the expected accuracy fairness trade-off does materialize: RMSE increases, yet KS continues to decline.
An analogous pattern appears in the Non-lID setting on the Law School dataset (Table 5).
and 9 extend these findings the Law School, Communities and Crime, and Adult datasets T These results demonstrate the effectiveness
Additional results in the Appendix- Tables
16

---

Published in Transactions on Machine Learning Research (Apr/2023)
Cerlorr App
Figure 5: Intersectional (Race x Sex) fairness on COMPAS. Accuracy under KFFL versus the multi- attribute SPD and multi-attribute EO gaps. Markers sweep the fairness weight λ (0, 1000, 1100..... 2000}; larger pushes the model from the low fairness region towards the high fairness region. Groups are defined by the Cartesian product (African-Amerionn binary indicator)x { (Male, Female), yielding categorics. The two left pancls show the trado-offs between the SPD gap and accuracy in the IID and Non- IID settings, while the right half shows the trade-ofls with the EO gap under the IID and Non-IID seltings.
intersectional
Figure 6: Multi-racial fairness on COMPAS. These panels follow the same layout as in Figure 5 but with the sensitive attribute "race" taking values over the full set AAfrican-Amercan Caucasian, Hispanic, Asian, Native American, Other) (K = 6). The transition from the low fairness region the high fairness region as the fairness hyperparameter À increases illustrates the trade-off between accuracy and multi-group SPD and multi-group EO gaps.
of KFFL in balancing accuracy and fairness in regression tasks by appropriately tuning the fairness hyper- parameter X.
6.5 KFFL yields favourable trade-offs in settings with multiple sensitive attributes
We demonstrate the capability of KFFL to achieve fairness-accuracy trade-offs when there are multiple sensitive attributes by considering (i) an intersectional attribute race x ser that takes K 4 values, and (ii) the full racial spectrum recorded in COMPAS, which takes K = 6 values Fairness is mcasured by the max min gaps SPDmmuti and E0mult (13
Figures 5 and 0 plot the model accuracy in these two experimental setups, versus the fairness gaps while sweeping the fairness hyperparameter Xe (0, 1000, 1100, 2000} under IID and non-IID foderated settings. In every case the results trace a smooth trado-off between fairness and accuracy: increasing moves the operating point from the high-gap zone into the low-gap zone while keeping the accuracy in the range 0.60,0.72)
We note that when A= 1600 in the IID setting of the intersectional experiment, the model attains SPD < 0.05 and F.0mutti < 0.05 with an accuracy of around 0.65. This is a substantial improvement over the unregularised model (A = 0), in which SPDmulti ss 0.30 and E0 &e 0.40. A comparable pattern emerges for the multiple-race experiment, showing that KFFL scales gracefully from binary to genuinely multi-group fairness.
The orthogonal random-feature (ORF) Yu ct al. 2016) map utilized in our implementation of KFFL requires D> 2k: where k is the number of one-hot columns, to isly the orthogonality condition. The earlier binary- attribute experiments used D=10, but the richer group structure in these multiple sensitive attribute settings demands slightly larger D: we set D-16 for the intersoctional sctting (k=7) and D-24 for the six-race setting (k=11). Further increases to produced no significant changes in either accuracy or fairness, confirming that simple rule of thumb of Dz is adequate even the multiple sensitive attribute regime.
Inttersectional groups: (AA M), (AA 1,F), (AA O, M), (AA 0, F); racial groups: African- American, Caucasian, Hispanic, Asian, Native American, Other.
17

---

Published in Transactions on Machine Learning Research (Apr/202)
7 Conclusions
This work introduces systematic approach for training group-fair machine learning models in a federated setting by leveraging KHSIC as a fairness regularizer to capture complex, non-linear dependencies between model outputs and sensitive attributes. The proposed method, KFFL, significantly reduces the communi- cation and computation costs of a naive implementation by employing random feature maps and a novel lederated proximal gradient algorithm, FedProxGrad, which accommodales the non-convexity of both the data-fitting term and the fairness regularizer.
Experimental results demonstrate that KFFL performs robustly across diverse client data distributions and standard datasets commonly used to evaluate fair learning methods. It achieves strong performance in both regression and classification tasks by more thoroughly exploring the trade-off's between fairness and accuracy compared to existing baselines. In exchange for this flexibility, KFFL and KFFL-TD incur about twice the communication cost of FedAvg, and respectively require two and one more round of communication per iteration in comparison to FedAvg. We leave up to practitioners determine, based on the requirements of their use case, whether the ability better explore the accuracy-fairnes trade-offs justify this additional cost.
Limitations and Future Work
KFFL effectively enforces statistical parity in a principled manner. Expanding the method to address other notions of group fairness- e.g. by utilizing conditional variants of the KHSIC to ensure equalized odds- would enhance versatility. Also, the current framework assumes full client participation, making it less suitable for scenarios with partial participation or privacy constraints, and requires two additional rounds of communication over FedAvg to compute the fairness gradient in order to guarantee convergence. To address these challenges, future work incorporate differentially private gradient estimation techniques and develop modifications to KFFL and its analyses to support partial participation and to justify the use of stale gradient information in KFFL-TD, which requires only one additional round of communication over FedAvg. Additionally, the convergence rate of FEDPROXGRAD is influenced by the variance a² of the stochastic gradient estimate for the fairness regularizer, which imposes a fixed floor on the convergence the method to stationary point of the fairness-regulariized learning objective, and the use of the dissimilarity condition. Future work overcome these challenges by exploring variance reduction techniques to mitigate the impact of x² on convergence rates and alternative analysis strategies, such the approach taken in Yuan & Li 2022), to eliminate the need for the bounded dissimilarity assumption.
References
pyrfm: A library for random feature maps in python. URL
feature html
Alekh Agarwal, Miroslav Dudik, and Zhiwei Steven Wu. Fair regression: Quantitative definitions and reduction-based algorithms. In International Conference on Machine Learning, pp. 120 129. PMLR, 2019a.
Alekh Agarwal, Miroslav Dudik, and Zhiwei Steven Wu. Fair regression: Quantitative definitions and seduction-basel algorithms. In International Conference on Machine Learning, pp. 120 129. PMLR, 2019b.
Sina Baharlouci, Maher Nouiched, Ahmad Beirami, and Mcisam Razaviyayn. R 'enyi fair inference. arXiv preprinl Xix::11906. 12005, 2019a.
Sina Baharlouei, Maher Nouiehed, Ahmad Beirami, and Meisam Razaviyayn. R\ ''eyyi fair inference. arXiv preprint Xix::19006 12005, 2019b.
Yajie Bao, Michael Crawshaw, Shan Luo, and Mingrui Liu. Fast composite optimization and statistical recovery in federated learning. In Proceedings of the 39th International Conference on Machine Learning. volume 162, pp. 1508-150 1536. PMLRR, 2022.
18

---

Published in Transactions on Machine Learning Research AArr2223)
Matias Barenstein. Propublica's compas data revisited. arXiv preprint arXt::1906.04 04001, 2019.
Barry Becker and Ronny Kohavi. attps://doi.no 10.24432/ 24.2432//55X22.. C5.
Adult. UCI Machine
Learning Repository, 1996.
DOI:
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for learning. proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1175-1191, 2017.
machine
Hongyan Chang and Reza Shokri. Bias propagation in federated learning. In The Eleventh International Conference on Learning Representations, 2023. URL
Bhaskar Ray Chaudhury, Linyi Li, Mintong Kang, Bo Li, and Ruta Mehta. Fairness in federated learning via core-stability. arXiv preprint arXiv:2211.0009 2022.
Yae Jee Cho, Pranay Sharma, Gauri Joshi, Zheng Xu, Satyen Kale, and Tong Zhang. On the convergence of federated averaging with cyclic client participation. In International Conference on Artificial Intelligence and Stalistics, pp. 10351 10375, 2023. URL https: httppp///aaaxiioo//aa/////////// wwwp/b////a...a.oaa////////// 03109
Lynn Chua, Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Mamurangsi, Amer Sinha, and Chiyuan Zhang. How private is dp-sgd? arXiv preprint orXiv:2403. 7673, 2024.
Evgenii Chzben, Christophe Denis, Mohamed Hebiri, Luca Oneto, and Massimiliano Pontil. Fair regression with wasserstein baryconters. Advances in Neural Information Processing Systems, 33:7321-7331, 2020a.
Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, Luca Oneto, and Massimiliano Pontil. Fair regres- sion via plug-in estimator and recalibration with statistical guarantees. Advances in Neural Information Processing Systems, 33:19137-19144 2020b.
Jose Correa, Andres Cristi, Paul Duetting, and Ashkan Norouzi-Fard. Fairness and bias in online selection. In International conference on machine learning, pp. 2112-2121. PMLR, 2021.
Michael Crawshaw and Mingrui Liu. Federated learning under periodic client participation and heterogeneous data: A new communication -effiiiiiaato--eeiiiittt algorithm and analysis. arXiv preprint orX:::2410.23131, 2024. URL
hattps://ariiz http://///axivvorr/a/bbeeee/ aat////aa22211000 2410/
Sen Cui, Weishen Pan, Jian Liang, Changshui Zhang, and Fei Wang. Addressing algorithmic disparity and performance inconsistency in federated learning. Advances in Neural Information Processing Systems, 34: 26091-26102, 2021.
Kate Donahue and Jon Kleinberg. Models of fairness in federated learning. arXiu preprint or orXin:2112.008 2021.
Julia Dressel and Hany Farid. The accuracy, fairness, and limits predicting rocidivism. Science advances, 4(1 (())eaa05550, 2018.
Wei Du, Depeng Xu, Xintao Wu, and Hanghang Tong. Pairness-aware agnostic federated learning. In Proceedings of the 2021 SIAM International Conference Data Mining (SDM), pp. 181-189. SIAM, 2021.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovalions in theorelical computer science conference, pp. 214-226, 2012a.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pp. 214-226 2012b.
19

---

Published in Transactions on Machine Learning Research (Apr/202)
Hubert Eichner, Tomer Koren, Brendan McMahan, Nathan Srebro, and Kunal Talwar. Semi-cyclic stochastic gradient descent. In International Conference on Machine Learning, pp. 1764-1773, 2019. URL. https: //arxiv. seaxiv..rg//aeee httggabb/19904..0000
Yahya H Eazeldin, Shen Yan, Chaoyang He, Emilio Ferrara, and A Salman Avestimehr. Fairfed: Enabling group fairness in federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 7494 7502, 2023.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Mcasuring statistical depen- dence with hilbert-schmidt norms. In International conference on algorithmic learning theory, pp. 63-77. Springer, 2005a.
Arthur Gretton, Ralf Herbrich, Alexander Smola, Olivier Bousquet, Bernhard Schölkopf, et al. Kernel methods for measuring independence. 2005b.
Xinran Gu, Kaixuan Huang, Jingzhao Zhang, and Longbo Huang. Fast federated learning in the presence of arbitrary device unavailability. Advances in Neural Information Processing Systems, 34:12052- 120644, 2021. URL https://arxiv. org.obgg///o....o............ arggabs/2106.0700/
Raffay Hamid, Ying Xiao, Alex Gittens, and Dennis DeCoste. Compact random feature maps. In Interna- tional conference on machine learning, pp. 19 27. PMLR, 2014.
Xiaotian Han, Jianfeng Chi, Yu Chen, Qifan Wang, Han Zhao, Na Zou, and Xia Hu. Ffb: A fair fairness benchmark for in-processing group fairness methods. arXiv preprint arXiv:2306.094 2023.
Yuze Han, Xiang Li, Shiyun Lin, and Zhihua Zhang. A random projection approach to personalized federated lcarning: Enhancing communication efficiency, robustness, and fairness. Journal of Machine Learning Research, 25(380):1-88, 2024.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
Shengyuan Hu, Steven Wu, and Virginia Smith. Fair federated learning via bounded group loss.
Kaixuan Huang, Xinran Gu, Jingzhao Zhang, and Longbo Huang. Federated stochastic primal-dual learning with differential privacy. arXio preprint arXiv:2204.128 12284, 2022. URL https://arxiv org/ ab2/
12284
Ajil Jalal, Sushrut Karmalkar, Jessica Hoffmann, Alex Dimakis, and Eric Price. Fairness for image generation with uncertain sensitive attributes. In International Conference on Machine Learning, pp. 4721 17322 PMLR, 2021.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pp. 5132-5143. PMLR, 2020.
Kevin Kim and Alex Gittens. Learning fair canonical polyadical decompositions using kernel independence criterion. Xéve preprint wxXiv22144.1300 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.
ar preprint
arXiv:1412.6900 2014.
Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in federated learning. arXiv preprint arXic:1905.1044 10497, 2019.
Tian Li, Anit Kumar Sahu, Manzil Zaheor, Maziar Sanjabi, Amect Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429 450, 2020.
Wei Li, Jian Wang, Xiaoyang Zhang, Yifan Zhang, and Yanfeng Wang. Privacy protection bottom-up hierarchical federated learning. In International Conference on Artificial Intelligence and Security, pp. 25 25-36, 2023a. URL.
20

---

Published in Transactions on Machine Learning Research (Apr/202)
Wei Li, Jian Wang, Xiaoyang Zhang, Yifan Zhang, and Yanfeng Wang. Anofel: Supporting anonymity for privacy-preserving federated learning. arxic preprint arXiv:2306.0682) 2023b. URL. org/abs/2306.06822
Grigory Malinovsky, Samuel Horváth, Konstantin Burlachenko, and Peter Richtárik. Improving accelrrated federated learning with compression and importance sampling. Federated Learning and Analylics in Prac- tice: Algorithms, Systems, Applications, and Opportunities, 2023. URL https://arxiv org/abs/2302 03662
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication--lfieeet learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 2273-1282. PMLR, 2017.
Ninarch Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6):1 35, 2021.
Ninarch Mchrabi, Cyprien de Lichy, John McKay, Cynthia Hc, and William Campbell. Towards multi- objective statistically fair federated learning. arXie preprint orXiv:2201.0900 2022.
Omid Memarrast, Linh Vu, and Brian D Ziebart. Superhuman fairness. In International Conference Machine Learning, pp. 24420-24435. PMLR, 2023.
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In International Conference on Machine Learning, pp. 4615-4625 PMLR, 2019.
Afroditi Papadaki, Natalia Martinez, Martin Bertran, Guillermo Sapiro, and Miguel Rodrigues. Minimax demographic group fairness in federated learning. In 2022 ACM Conference on Fairness, Accountability, and Transporency, pp. 142-159, 2022.
Sikha Pentyala, Nicola Neophytou, Anderson Nascimento, Martine De Cock, and Golnoosh Farnadi. Priv- fairfl: Privacy-preserving group fairness in federated learning. ar prepriul or 2xXx:2205.11584, 2022.
Adrián Pérez-Suay, Valero Laparra, Gonzalo Mateo-Gareía, Jordi Muñoz-Marí, Luis Gómez-Chova, and Gustau Camps-Vall, Valls. Fair kernel learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 339 355. Springer, 2017.
Ali Rahimi and Benjamin Recht. Random features for largo-scale kernel machines. Advances in neural information processing systems, 20, 2007.
Michael Redmond and Alok Baveja. A data-driven software tool for enabling cooperative information sharing among police departments. European Journal of Operational Research, 141 131(3):600-77) 2002.
Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh. Fairbatch: Batch selection for model fairness. arXiv preprint arXiv:2012.0 01696, 2020.
Teresa Salazar, Miguel Fernandes, Helder Araujo, and Pedro Henriques Abreu. Fair-fate: Fair federated learning with momentum. arXie preprint xxXX:2220011678, 2022.
Teresa. Salazar, Helder Araújo, Alberto Cano, and Pedro Henriques Abreu. A survey group fairness in federated learning: Challenges, taxonomy of solutions and directions for future research. arXie preprint arXiv:24 10.000..00 03855, 2024.
Quoe Tran Dinh, Nhan Pham, Dzung Phan, and Lam Nguyen. FedDR: Randomized Douglas Rachllss--ahoorrd ford splitting algorithms for nonconvex federated composite optimization. In Advances in Neural Information Processing Systems, volune 34, pp. 30326 30338. Curran Associates, Inc., 2021.
Ganghua Wang, Ali Payani, Myungjin Lee, and Ramana Kompella. Mitigating group bias in federated learning: Beyond local fairness. Xix preprint XXw:2305.09031, 2023.
21

---

Published in Transactions on Machine Learning Research (Apr/202)
Jia Wang and Jueyou Li. Federated dual averaging learning algorithm with delayed gradients for composite optimization. Avoilable al SSRN 4507875, 2023.
Linda F Wightman. Lsac national longitudinal bar passage study. Isac research report series. 1998.
Yi Xu, Rong Jin, and Tianbao Yang. Non-asymplotic analysis of stochastic methods for non-smooth non- convex regularized problems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Felix Xinnan X Yu, Ananda. Theertha Suresh, Krzysztof M Choromanski, Daniel N Holtmann-Ric, and Sanjiv Kumar. Orthogonal random features. Advances in newrol information processing systems, 29, 2016.
Honglin Yuan, Manzil Zaheer, and Sashank Reddi. Federated composite optimization. In Proceedings of the 38th International Conference Machine Learning, volume 139, pp. 17253-122666 PMLR, 2021.
Xiaotong Yuan and Ping Li. On convergence of fedprox: Local dissimilarity invariant bounds, non- smoothnesss and beyond. Advunces Neural Information Processing Systems, 35:10752 35:10752-10765, 2022.
Hossein Zakerinia, Shayan Talaci, Giorgi Nadiradze, and Dan Alistarh. Blockchain-based optimized client selection and privacy preserved framework for federated learning. arXiv preprinl arxi::2908.04: 04442, 2023. URL
Yuchen Zeng, Hongxu Chen, and Kangwook Lee. Improving fairness via federated learning. arXiv preprint arXiv:2110.15544 2021.
Yuxin Zhang, Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Advancing personalized federated learning: Group privacy, fairness, and beyond. SN Computer Science, 4(5):1-13, 2023. URL
https://link.springee cww. ottp://pw......//tt..//o/o///.//
22

---

Published in Transactions on Machine Learning Research
A Appendix
A.1 Related Works
Methods for ensuring fairness within centralized machine learning are typically categorized into three distinct groups: pre-processing, in-processing, and post-processing Mehrabi et al. (2021). In federated learning, bias mitigation methods predominantly fall into in-processing approaches, although some work has also been done in post-processiigg methods.
For in-processing methods of bias mitigation in federated learning, EEzeldin et al. 2023 is notable for its versatility and compatibility with various local bias mitigation techniques.In this approach, clients convey their localized fairness metrics lo the server, which then optimizes weighting coefficients to minimize the client with respect to a chosen fairness metric. Papadaki et al. (2022)
contribution of the goorest-performin optimize model performance on the worst-performing demographic by adopting a minimax optimization framework. Salazar et al. (2022) propose a lairness-aware momentun-baseel method to address bias in federated learning. The approach in Mehrabi et al. (2022) strives for fair federated learning but requires the server maintain validation dataset. ZZng et al. (2021) address the challenge of bias mitigation in federated learning through bi-level optimization problem; their analysis predominantly pertains to specific loss functions. |Pentyala et al. 2022) consider post-processing and pre-processing approaches to ensuring fairness. Cui et al. (2021) require the clients to achieve Pareto optimality with respect to both fairness and accuracy.
Other variants of group fairness have been explored in the federated setting. Hu et al.] introduce the concept of bounded group loss as facet of group fairness in federated learning, although their work does not specifically develop algorithms targeting bias mitigation. Chang Shokri (022) analyze how bias within participating clients can propagate during the training process but do not propose methods aimed at explicitly addressing group fairness.A comprehensive summary of additional approaches in group fairness federated learning is provided in Table of Salazar et al. (2024). For baseline comparisons, we limit our focus to most published works in group fairness federated learning except the recent Wang et al. 2023).
Significant related works from the centralized setting that aim to ensure fair models include Pérez-Suay et al. (2017), which leverages the (non-kernel) Hilbert-Schmidt Independence Criterion (HSIC) to promote the learning of fair kernel machines, and [aaharlouii et al. (2019a), which incorporates the Rényi correlation as regularization term to achieve statistically fair models.
A.2 Proof of Theorem !
In the following, B+[-] = E[- [w]] denotes the expectation conditioned on all sources of randomness in the algorithm up to and including the calculation of w,, and E.[-] N Einll denotes the average of quantity over the clients.
The computations are complicated by the presence of composite objective and stochasticity in our estimate of Vyk(w,, but the concept.ull outline of the proof of the convergence rate follows that of the proof of the convergence rate for FedProx in Li et al. (2020). Namely,
First, we establish that the distance between consecutive iterates, [10++--Well is upper bounded by the quantity E, (. + Vul(w))lz, and use the bounded dissimilarity condition to upper bound the latter by a multiple of the composite objective gradient at wi, [[:F(, plus a term due to noise.
Next, we use this result and the smoothness of the composite objective establish that after one round of the algorithm, the composite objective satisfies E,F(w141) I F(w)) [V + (vanishing terms) + (noise).
23

---

Published in Transactions on Machine Learning Research (Apr/202)
We conclude that if the step-size is chosen appropriately, then the objective decreases in expectation at each iteration, up to the noise level. A standard argument with Jensen's 's inequality and telescoping sums delivers the claimed convergence rate.
Proof. We flesh out the preceeding outline.
Iterate proximity To bound the iterate proximity, begin by introducing the local exact minimizer,
1 _ 2a
6/41 = argmin, = ('(w +
س w1+1/211.
The data-fitting term e is weekly convex and the quadratic regularizer is =tstonggyy convex, the local objective &j is μ strongly convex for μ - L..
The j-strong convexity of e implies that the iterate distance between w, and wi+1 can be estimated using the size of the gradient of f at those models:
| - well2 s | 0/talla + ||,,- - 02+1lla
1 [IVe + [[](())))
#
Employing the g-suboptimality of wf+ to estimate the size of ( I| refines this estimate to
1 ? [V(((.)))l
| well2 S
We note that
Ve(())) Ve((w)) + _(w, w1+1/2) = V (w)))) +g
and consequently
| well2 5 1 1+ [Iv(())) + VV(w)))22 Vvb(s)) Ball2
Using Jensen's inequality delivers
E w. < E, E, [w ++1
wr1l
+ ?
2
E,
E&IlVe + Vob(w)) VWW(:)) + Vib(w,) - 1g
2
1+?
B3 + + +o2
2
(22)
The last inequality holds because of the bounded dissimilarity condition and the upper bound the variance of gr. Similarly,
E, [1+1 - wall2 S E, E - welle
< 1 ++1 E, [e) Vo + VIVbbl..) - 8B -g+
/
1 +
<
BI
(23)
24

---

Published in Transactions on Machine Learning Research (Apr/2025)
Objective Decrease The L-smoothness of the composite objective implies that
we+1 wi) _ Elwe++ - w E,F(w)+1) E, ((w.) + (VF(w:), ((F)))((V((())))))
(24)
=F(w)) - 0el all 12 + Eg (VF(w)), Wi+1 (wo a (V
L + 2 E1 111l -
- w1
Equation 22 establishes that, up noise terms, the term E, [|++1111 we scales like a² F because 1 2 is on the order of a². Now we develop series of estimates to establish that the quantity (e(F(w)), A.)
also scales like (2])) up to noise terms.
We begin by using the 7-suboptimality of wh+1 to find a useful expression for A. In particular, suboptimality implics that
1 1 (wit)) w14122 [E + g: + a (w//+) we)
(e((w))) a
1 (Ve((w(+)) Ve((w)) (Vu(wr) gi) + - Ws)
=c,-1+
where ||cttal22 S vl Consequently,
Wi+1 w. = - [ wi] = aB [ei+1 - (VE( Ve(wa)) + (Vi(wi)) g:)]
and, adding and subtracting terms judiciously yields
= aE, [ei+1 -
(VE(w(+1) - Vib(w:)) + V)(((41) - Velli+1)
+ (Vb(w)))
-VF((wLL) aE, ( (w) (+1) v VF
aB, (vb(w)) Vrb (a g:) + aB,e
It follows that
A W141 w. + aVF(w.)
= = (V[(w++1) VF(W.) -a E. (VE + Vi(((+11) V [((++))
=ta
E, [vb(w)) VIlWE ta (Vb(w,) gt) ta E,
=ts
t4
tg
Consider the quantity E, (VF A.)]
E. [.((()))) A S aE, VF((w)) (Itt + ||tell2 + ||tall2 + +|ttll2 + [[tol12)
Observe that because the composite objective is L-smooth,
Er, [||t S LE, [|| wrl12] < LIE, [,, ||wiy1 wella]
Similarly, the estimate for t3 uses the L-smoothness of the regularizer:
E, [||ts|/2] & E, [R4 Vi)(w.))
V I.E, [R, Il: w641l12]
25

---

Published in Transactions on Machine Learning Research (Apr/202)
The t2 term can also be bounded in terms of the iterate distance:
E. |||t|||]] s E, E. Ve (w)))) + ( - (1(
E
+ Vel(w)+) (e((w)+1) (ob(w))))))))
where the last equality holds because ve(we++) = E, (W((+))) We use the triangle incquality and the L-smoothness of vo and the functions f to continue our estimation:
2 E: | w [
2LE, ||++ well2 + 2LE, [E ||w w1+1ll2l
- 4 ER ||w: - -1+1
Thus we find that
Ec /FF(w)lla (l/ta/l2 + ||t2|12 + [|ta|12)
< 6L/|VF((.)ll E. [B || [111l
+1
BB + /VF(w))) (G+ o)]
61(1+7)
(B+1)IVV(())] + + G²).
1
The last two inequalities are justified by equation 23 and the fact that abl ]aa² +b) for any real numbers and b.
The noise term t4 is controlled by the variance of the stochastic gradient estimate
E. [1(4lla]]] < v E. (VV)( grl Sa.
To control the ts term, recall that ||e| > (w:)) 1|2- This implies that
E, [s/2]] = E, ||, - E, E, ||c1+1|111
S SB [E (w: + 8:l12)
$ B, EE (w,) + vo(w:)ll2 + [VV( - -gr g1112]
< ( (F G
The last inequality holds because of Jensen's inequality, the bounded dissimilarity condition, and the bound on the variance of the stochastic gradient estimate.
From these last two estimates, we find that
Ec [VF(w))l (|ltal22 + [ttll1)
[ + (1
VF(W:))
1 (BB
and therefore we conclude that
E, [VF((,,) A,)] < 6L(1+ (B 1))])/((,)] G² +
(25)
1
VF(W))) + -02 + a (BB 1)] G2
2
G.B.B (I|FF(w:) 11 G² +
where, for convenience, we defined
+ + 1) 6L(1 + (B + 1)
C, =
2
26

---

Published in Transactions on Machine Learning Research (Apr/2025)
Finally, consider the squared iterate distance in equation
24 In particular, equation 22 implies that
_
[Ellwe+1-- well
L
1
11 1 B (VI o²
Using this estimate and equation 25 in equation
gives that
L Be [| wel2
E, [F(w++) & F(w)) - VF + E, (VF(W)), A.)
+ 2 < F(w)) aal)) La /.l.B...a///(LV.V (F((w)) C. (G² + +0²),
(26)
where, for convenience, we define
1 (++?) 2
C, = a. G.L. LB².
1 =a ( + +
6L + (B 11)
LB2. (1+7*
1
Because 2
1
20,
"
aT.1
and consequently
C, Sa a + ( + 1) 12aL (1+ + LB2 + 7)
We also choose y < 8(B41) and y < 20, which further implics that
1
C, Sa a IBaL(B+1) (11) + 5a LB?
5
and because a < min [20L(B+T) 3LB : 20
in fact
3
4 5
Cy,.Ba S a + a²
Thus, we conclude that the expected decrease in the composite objective at each iteration satisfies
E, [F((++))) < F(w,) a
[VF(w))) + 4a
(G2 +
(27)
5
Convergence rate In the remainder of this proof, [E|-] denotes the expectation with respect to all sources of randomness and we take G 0. Using the tower rule of expectations and summing over the first T iterations, we find that
4aT
T
<
[[F(wr) - F(w(-1)
5
= F(wo) - E [F( F(Wo) F*.
Rearranging terms yields the claimed result:
T-1
F(Wo) F
E
(1))) <
aT + 402
T A=
27

---

Published in Transactions on Machine Learning Research (Apr/202)
A.3 Proof of Theoram 2
Proof. We know from the definition of KHSIC that
()(w) = 1
Tr(K, HK, HH)
(n 1)2
1
E
[r(Z, Zg H H)
(n 1)²
Tr(Z, Hz, 2, HZ.)
(n
1)²
||,,, HZ [
(n 1)²
The first equality is the definition of the KHISC, and the second holds because the outer products of the random feature maps are the kernel matrices, in expectation. The third holds due the cyclicity of the trace operator, and the final holds by the definition of the Frobenius norm. The rest of the proof follows Lemma 1
A.4 Proof of Lemma 1
Consider the following:
Proof. The reduced size fairness interaction matrix can be computed efficiently, by noting that
z HZ = Z HHZ,
Thus,
H. H Z!
I [-- 11 11-
Z 11,1%
and
71
I -
HZ,
Z, = 2 1p
n
Putting these identities together and using the local paritition of the random feature matrices and their means, we have that
Z HZ, = - 1 14, M,,T 1 2 1 mp
= =Z,ZZ, 14
m
m
Σ ZJ, Z,5 n
Σ
nilliis
(28)
12
1=1
A.5 Proof of Corollary 1
Proof. We use the linearity of expectation to obtain an unbiased approximation of the gradient of the fairness regularizer:
(w)((w) (w)
E E G
(n - 1)2
28

---

Published in Transactions on Machine Learning Research (Apr/202)
A.6 Proof of Lemma 2
The stochastic gradient can then be computed in terms of the Jacobians of the local interaction terms Z2 by a simple application of chain rule:
(n - 1)3 V
G(w))
= (n- - 1)2
(n
ja(w G(w)
2
Ja, (w)
(n 1)
B Details of the experimental Evaluation
B.1 Choice of Hyperparameters
To ensure a fair comparison, all evaluated algorithms utilize a consistent set of common hyperparameters. The batch size is uniformly set to 64. Each algorithm undergoes a total of 10 global training rounds, with each round comprising local epochs on every client. This choice of local epochs ensured that the global model converges within rounds or fewer for all datasots and distribution, allowing to limit the number of global rounds to 10. The experiments are conducted with clients, a configuration determined based on recent studies. Ezzeldin et al [2023 Papadaki et al. (0022). The learning rate
Common Hyperparameters:
set to 0.01 and Adamn Optimizer Kingma & Ba 2014 was used for optimization
A Algorithm Specific Hyperparameters: Some hyperparameters are specific to the algorithm being used. For example, the KFFL and KFFL-TD algorithms roly on feature maps D to estimate the kernel regular- izer. In our experiments, we use the Pyrfm librarypyyrJ to generate random feature maps based on Orthogonal Random Features Yu et al. (0166). The dimensionality of the feature maps used for kernel approximation, denoted D, is set to 10. While higher dimensions also yielded good results, selected the smallest feature map size that ensured KFFL performed effectively..
For other baselines such as Ezzeldin et al. (023), tradeoff parameter called "fairness budget" 3 is used to control the effect of reweighing. This tradeoff parameter determines the balance between model accuracy and a specific evaluation metric, by varying 33 from 0.1 to based on the suggestions provided in the paper. However, it should be noted that for the best performance of the Ezzeldin et al. (2023) baseline, a local debiasing mechanism is required. Based on the results from the paper, we used the Roh et al. (2020) algorithm as local demographic bias mitigation algorithm to compare with method. |Papadaki et al.) (2022) use "global adversary rate' to control how the expected loss over the worst- norst-performing demographic is reduced. To consider an accuracy-fairness tradeoff, we varied this parameter from 0.001 to 0.1.
To enable fine-grained control over the tradeoff between fairness and other performance metrics, our methods incorporate a controllable fairness weight A. This weight can be fine-tundd based on the desired tradeoff. More on this in Section B.5
B.2 Dataset
Datasets for Classification Task
ADULT: Becker & Kohavi [996] is binary classification dataset that contains up to 14 attributes used in predicting whether an individual would carn an income 2 50K or < 50K. The features used in the prediction include continuous attributes such as age, hours per week worked, etc, and discrete attributes inchuding relationship, race, sex, and education. For the purpose of our experimental evaluation, we train a Logistic Regression model Mohri et al. (2019 on this dataset and we consider sex as the protected sensitive variable.
COMPAS Barenstein (2019) used for predicting criminal recidivism for individuals. The number of samples considered 6,172 samples and the number of predictive features used in determining
29

---

Published in Transactions on Machine Learning Research (Apr/2025)
recidivism is 52 including race, age, and previous criminal offenses. For the purpose of our experi- mental evaluation on this dataset, we consider race as the protected sensitive variable evaluated on a Logistic Regression model.
BANK:Daaa from a Portuguese bank utilized forecast client subscriptions term deposits Han et al. (2023 Here consider age the sensitive attribute and Loan Approval the target attribute. There are 64 predictive features (including the sensitive) and 41188 target samples.
ACS: From the American Communily Survey, utilized for various prediction tasks including income and employment Han al. (202)). There are 910 (including the sensitive) predictive features and 195665 samples for this dataset. We consider income the target variable with sex as the protected attribute.
GERMAN:Datail credit applicants from a German bank used for predicting credit risk ratings Han (202)) where the sensitive attribute we consider is sex and the larget attribule is Credit risk rating. There are 60 (including the sensitive) predictive features and 1000 target samples.
Datasets for Regression Task
Sourced from the Law School Admissions Council's National Longitudinal Bar Passage Study Wightman (1998), this dataset contains 20,649 examples. The task to predict a student's GPA- normalized to the range [0, 1s-uning squared loss minimization. Race serves as the protected attribute, categorized as white versus non-white.
Law School:
Communities and Crime: This dataset comprises socio-economic, law enforcement, and crime statis- lies from various U.S. communities Redmond & Baveja (2002), lotaling 1,994 examples. The objec- tive to predict the number of violent crimes per 100,000 inhabitants, normalized to [O, 1], through squared loss minimization. The protected attribute is race, defined by whether the community's majority population is white.By including these datasets, we aim to thoroughly evaluate the fair regression estimator's performance across different contexts where fairness with respect to sensitive attributes like race is crucial.
B.3 Data Distribution on Clients
We explore both the IID (Independent and Identically Distributed) and Non-IID (Non-Independent and Non-Identically Distributed) settings in our evaluation:
IID: In this setting, each client is provided with an equal number of samples and a consistent data distribution for local training Li et al. 2020))
Non-IID; In this setting, each of the clients has different distribution of the protected attribute. Particularly, in our case, since the protected group A is binary with attributes being An and A half of the clients have 90 % of An and 10 % of Ai while the other half has 90 % of A and 10 % of Ao Li et al. 2020
B.4 Models
We consider two distinct types of models for classification and linear model for regression, which clients use for local training. In the fairness literature, Logistic Regression is commonly employed for fair classification tasks Ezzeldin et al. (2023), while linear models are the standard choice for fair regression Chzhen et al.] (2020a). Beyond these, we extend our evaluation to include a more complex, non-convex model for classification.
Logistic Regression: This involves a binary logistic regression model with a sigmoid activation function Han et. al. (2023)
30

---

Published in Transactions on Machine Learning Research (Apr/202)
Neural Network:
We also examine the performance of algorithm using neural network configuration. This neural network consists of single hidden layer with 100 neurons and employs ReLU activation, culminating in an output layer.
B.5
Fairness weights
ADULT: 2000,,0.00, 71.58, 123.16, 174.74, 226.32, 277.89, 329.47, 381.05, 432.63, 484.21, 535.79, 587.37, 690.53, 742.11, 793.68, 845.26, 896.84, 948.42, 1000.00
COMPAS: 0.00, 0.01, 0.10, 20.00, 71.58, 123.16
BANK: 0.00, 0.01, 0.10, 1.00
GERMAN: 0.00, 0.01, 0.10, 1.00
ACS: 0.00, 0.01, 0.10, 1.00
XFPL
1277-12 12
SERVER
SERVER T-beettyyle
Figure 7: KFFL and KFFL-TD are illustrated the figure. Different colors correspond to various segments of communication associated with either the FAIR1, FAIR2 or Local Update sub-rounds (sce the KFFL Algorithm for detail on these flags). For KFFL-TD the relevant flags are FAIR and Local Update (see see the KFFL-TD Algorithm for detail on these flags) The direction of the arrows indicate uplink downlink communication and the width of each arrowhead highlights the communication cost in each sub-round. Thicker lines indicate higher commmication overhead, while thinner lines represent smaller overhead.
C KFFL-TD Kernel Regualarized Fair Learning with time delay
The KFFL-TD variant optimizes communication efficiency by incorporating delayed information for the fair term. Assuming the training begins at round t with t2 1; the server transmits the current model Algorithm We and all global information from the preceding time step (t 1), Hy 1), G(w--1) to the client for local gradient computation. This set of downlink information denoted T(w,). this case, the server also shares a common soed Ç to control the randomness the generation of random fcature maps generated. The client also receives a FAIR flag, indicating that no data-fitting operation required. The clients leverage this information to calculate the fair gradient g' g((w:1) (if the global round is not zero) and relevant details ('(wxz) (see Algorithm 4h, contributing to the computation of the global interaction G(w:), (t,(),, and Hy (t) which will be used in the subsequent round for the fair update (see Algorithm 5 All of this information is
(w:) sent by each client.
With these outdated fair gradients, the server updates the global model see Algorithm 5 and compute the global fairness interaction terms ((w)).+..t) be used by the clients in the next round. To steer the model towards the data-fit direction, the server sends Local Update flag instructing the clients perform
31

---

Published in Transactions on Machine Learning Research
Algorithm 4 KFFL-TD Client Update
Input: (ROUND...)
1: if ROUND = FAIR then
2: if t +
0 then
3:
Clients compute local interaction for gradients using Equation 15 at W,-1 to get l2(w;-1)
4: The clients then compute the local gradient g' (wi-1) using Equation 16
end if
Clients compute (w,) using Equation 14 with random seed ( and RFMs (such ORFMs Yu et al.
(2016)) ) (t) using Equation 18 and Mx (t) using 17
D' (w())) (M: (t, (0)
if o then
V (wi) {g' (wi-1),
10: clse
12: end (we)) (we)
11:
13: Return: (w:))
14: clsc ROUND Local Update then
15: Clients do local updale Wt+1/2 following Equation 21 to gel w
16: Return: WH+1
17: end if
Algorithm KFFL-TD Server Side
we (This is the initial model) ++0
while w not converge do for all i 1, ... in parallel do
5: Generation of random seed Ç
if + 0 then
u, 7: I(w:) = (u (), G(we-1),
else
9:
I(w:) (it, end if
10:
11: ((w:)) Client Update(T(we), FAIR)
12:
end for
13: if o then
14: W2+1/2 W. Em g (w:-1)
15:
end if
16:
From ((w)) clients get ((w:) ( [
17:
From ((w:) compute G(w:) using Equation
13 14, (1) using Equation 17
and (t)()) using Equation
I8
18:
for all i= 1 m in parallel do
19:
if t + then
20:
w = Client Update(w(41/2 Local Update)
21:
else
22:
w Client Updatc(w,, Local Update)
23:
end if
24:
end for
25:
Wi+1 + FedAvg(w(11)
26:
end while
local optimization (sce Equation 21 and communicate the updated copy to the server the server conducts Fed-Avg after receiving these locally updated models, resulting in the generation of the updated model. The process continues till the model w, converges.
32

---

Published in Transactions on Machine Learning Research (Apr/2025)
D Additional Empirical Evaluation of Classification
Table and show the results of KFFL and its baselines on Logistic Regression and Neural Network across different datasets.KFFT KFFL is most robust across datasets in the Non-IID setting and provides greater tradeoffs for accuracy and fairness.
Method
Adult
COMPAS
BANK
ACS
GERMAN
Fed.vg
84.30 +
61.13
91.21
1.10
81.14
1,20
72.50
1.15
KFFL
83.14 4 0.42
60.3
1,30
90:23
4: 1.25
79.12
1.35
72.50
+: 1.10
KFFL-TD
83.97 1 1.12
59.51
1.45
90.78
1.20
81.12
1.05
72.50
1 1.40
Padr Bati
76.34 0.1
55.47
88.05
+: 1.30
58.77
1.20
30.00
1.25
KIISIC-Loca
63.40 + 0.03
44.13
1.55
88.77
1:15
58.77
1.05
70.00 + 1.35
M.nMax
76.34
64.93
1.20
PedAAg
0.18 :: 0.06
0.16 2: 0.02
0.23
0:::2:0000 0.01
0.08 :: 0.04
0.19 1: 0.03
KPPI.
0.14++0.02
0.05
KFFL-TD
0 0.03
0.05 + 0.02
0.06 # 0.01
0.00 + 0.03
0.00 0.02
DairFed/FaiBattis
0,001 0.001
0.05
0.00
0,00 1 0.01
0.00 0.03
CHSSC-LLoad
0.31 1: 0.00
0.08 1: 0.02
0.00 1: 0.03
0,00 1: 0.02
0.00 0.01
MinMax
0.004 + 0.001
0.03 0.01
0.21 + 0.02
0.00 + 0.03
0.00 0.02
PedAvg
0.03
0.02
0.16
0.02
0.02 0.03
KFFL.
0,12 0.03
0.07 0.01
0.04 # 0.02
0.02 + 0.03
0.00 0.02
KHPE-TT
0.04
0.05
FairRe//FttBaaei
0.013 d: 0.001
0.00 # 0.02
0.16 + 0.03
0.00 + 0.01
0.00 0.02
FairTed-Kenaa
0.12 0.00
0.05
0.00
0,00 1 0.01
0.00 0.03
MEnMax
0.013 0.001
0.03 1: 0.02
0.02 1: 0.01
0.00 1 0.03
0.00 0.02
Table 6: Comparison of Methods in the Non-IID environment with Logistic Regression for 3 seperate runs. Similar to the results in Table under Non-IID conditions we have improved tradeoff points us- ing KFFL
Adult
COMPAS
DANK
ACS
GERMAN
Method
PedArg
KFFL
83.10 1: 0.45
02.35 :: 1.35
92:20
1: 1.50
81.15 2: 1.40
75.0 0.01
KFPE-TD
8336
90:75
1:25
Fadr BairBath
76:30 + 0.15
55.50 # 1.55
$8.10
+: 1.35
58.75
4: 1.25
37.50 + 1.30
63.45 4: 0.05
41.15 1.60
$8.80
1.20
58,80
1.10
70.05
1.40
MEnMax
76.30 1: 0.15
56.70 1: 1.45
61.95
1: 1.25
58.75
1.30
70.05
1 1.15
PedArg
0.17 0.15
0.15 0.05
0.22
0.02
0.09 1: 0.06
0.20 0.04
0.03
0.07 0.02
0.06
0.03 0.03
0.01 0.05
KFFL-TD
0.17 0.04
0.06 + 0.03
0.07 = 0.02
0.01 + 0.04
0.01 0.03
BatiB
0.01
Pair Fed-Kerell
0.35 0.01
0.00 + 0.03
0.01 + 0.04
0.01 + 0.03
0.01 0.02
MaMMax
0.005 0.002
0.02
0.20 4:
0.04
0.01 0.03
FedArg
0.23 0.04
0.24 :: 0.03
0.17 1: 0.02
0.05 1: 0.03
0.03 0.04
KFFL
0.13 0.81
0.08 0.02
0.05 0.03
0.03 1 0.01
0.01 0.03
IFFLL-T
0.85 0.11
0.00 0.03
0.06 2: 0.02
0.04 + 0.03
0.01 0.04
Patr RaddRed///aaaae FattBaath
0.014 + 0.002
0.10 + 0.03
0.17 + 0.04
0.01 + 0.02
0.01 0.03
Fed-eed-e
0.04
0.01
MinMax
0.014 1: 0.002
0.01 1: 0.03
0.03 1 0.02
0.01 1 0.04
0.01 1: 0.08
Table 7: Comparison of Methods in the Non-IID environment with Neural Network for 3 seperate runs.
Similar to the results in Table g under Non-IID conditions we have improved tradeoff using KFFL
E
Additional Experiments for Regression Task
Tables
15
33
and 9 show how KFFL performs under various regression tasks.
F
Fair Regression
The KHSIC method facilitates the training of predictive models for regression tasks while ensuring fairness
across various sensitive groups. Fair regression has been extensively explored in contralized settings, with
significant contributions from studies such as Chzhen et al. (020aa) Agarwal et al. (2019a): Chzhen et al.
(2020b).
33

---

Published in Transactions on Machine Learning Research (Apr/202)
Fairness weight
RMSE +
KS Diffcrence 4
0.0
0.322400 + 0.015698
0.462100 + 0.152637
1.0
0.326067 + 0.018071
0.386633 + 0.163709
50.0
0.311633 + 0.024243
0.410500 = 0.106058
100.0
0.322400 + 0.016542
0.277200 + 0.114275
Table 8: RMSE and KS Difference with standard deviations for 5 runs KFFL-TD with IID for the Com- munities and Crime Dataset
Fairness weight
RMSE 1
KS Difference +
0.0
0.317528 + 0.024826
0.671308 = 0.202138
1.0
0.317952 + 0.025052
0.620508 + 0.251276
5.0
0.316936 + 0.025075
0.662032 = 0.159920
50.0
0.317576 + 0.024314
0.444808 = 0.163534
100.0
0.317592 + 0.025111
0.395280 + 0.141982
Table 9: RMSE and KS Difference with standard deviations for 5 runs KFFL with Non-IID for the Com- munities and Crime Dataset
In our framework, we employ the Root Mean Squared Error (RMSE) as the primary evaluation metric to assess the accuracy of the regression model. The objective function is defined as:
fly, ((x = (f(x;: w) y.)² + Aeb(w)
where x; E Rd denotes the input features, Y: E is the target variable, and f(x;; w) represents the model's prediction parameterized by w. Consistent with the rest of this paper, we exclude the sensitive attribute s; from the training process of the fair regressor.
Consider regression task on dataset D {(x;, S; ve)) where x; is the input feature vector, S; S is the sensitive attribute (e.g., gender, race), and y: is the target variable (e.g., GPA, income). For each sensitive group E S, we define the corresponding subset of data as:
D (Tx
To evaluate fairness in regression tasks, we utilize the Kolmogorov-Smirrov (KS) distance Chzhen et al. (2020a), which measures the distributional differences between the model's predictions for different sensitive groups s S. The KS distance a widely adopted fairness metric in regression, enabling the assessment of disparitics between groups. For instance, normalized GPA prediction task where the sensitive attributo S represents gender (e.g., male and female), the KS distance quantifies the difference in GPA predictions between these groups.
The KS distance between predictions for any two groups and s' is defined as:
KS(f(x,s)) max sup F*(t) p ( (t)
S. iER
where Ft) denotes the empirical cumulative distribution function (CDF) of the model's predictions for group 8, calculated as:
F*(1) = D
Σ
((((xs)s1)
(
34

---

Published in Transactions on Machine Learning Research (Apr/202))
Fairness weight
RMSE +
KS Diffcrence
0.0
0.320332 + 0.024803
0.604128 + 0.166035
5.0
0.320936 + 0.024753
0.618280 + 0.186275
10.0
0.321824 + 0.024351
0.551812 + 0.179296
50.0
0.314788 + 0.026521
0.391712 + 0.157367
100.0
0.318976 + 0.023506
0.290872 + 0.099638
250.0
1.034776 + 0.015979
0.774960 = 0.043567
Table 10: RMSE and KS Difference with standard deviations for 5 runs with KFFL with IID for the Com- munitics and Crime Dataset. Optimal points are those with lower RMSE (for accuracy) and KS (for fairness)
Fairness weight
RMSE 4
KS Difference &
0.00
0.800667 + 0.002491
0.450867 + 0.065047
0.01
0.800367 + 0.002579
0.398433 + 0.224212
0.10
0.800300 + 0.002524
0.412733 + 0.303714
1.00
0.799533 + 0.001940
0.148967 + 0.009340
5.00
0.820100 + 0.023256
0.348280 0.030390
Table 11: RMSE and KS Difference with standard deviations for 5 runs KFFL-TD with IID for the Law School Dataset
This formulation ensures that the regression model maintains fairness by minimizing the KS distance across all sensitive groups, thereby promoting equitable outcomes.
35

---

Published in Transactions on Machine Learning Research
Fairness weight
RMSE 1
KS Difference |
0.00
0.799293 + 0.002132
0.514920 + 0. 0.124809
0.01
0.799253 + 0.002306
0.452160 + 0.163044
0.10
0.799187 + 0.002101
0.467133 = 0. 0.144954
1.00
0.798800 + 0.002080
0.140173 + 0.032955
50.00
186.105140 + 120.592723
0.195000 =0.066063
Table 12: RMSE and KS Difference with standard deviations for 5 runs KFFL with IID for the Law School Dataset. Oitisal points are those with lower RMSE (for accuracy) and KS (for fairness)
Fairness weight
RMSE 1
KS Difference
0.00
0.485100 + 0.005092
0.410600 = 0.031681
0.01
0.485100 + 0.005260
0.410833 = 0.026668
0.10
0.484967 + 0.005424
0.408967 = 0.013403
1.00
0.484767 + 0.004964
0.381000 = 0.009752
5.00
0.483967 + 0.004944
0.282967 + 0.010901
Table 13: RMSE and KS Difference with standard deviations for runs KFFL-TD with IID for the Adult
Dataset
Fairness weight
RMSE 1
KS Difference
0.00
0.491523 + 0.006695
0.345782 = 0.082513
0.01
0.493383 + 0.008737
0.350317 = 0.111856
0.10
0.493500 + 0.008616
0.361175 = 0.080166
1.00
0.4934492 + 0.008715
0.371725 = 0.040675
5.00
0.493467 + 0.009102
0.348567 + 0.034269
50.00
0.494740 + 0.009975
0.130180 = 0.050417
100.00
0.498000 + 0.011371
0.118580 = 0.100111
200.00
0.589080 + 0.194744
0.286860 + 0.115417
500.00
4.510640 + 5.096420
0.351200 = 0.110467
Table 14: RMSE and KS Difference with standard deviations for 5 runs KFFL with Non-IID for the Adult
Datasct
Fairness weight
RMSE
KS Difference
0.00
0.49088 + 0.009294
0.38293 + 0.016433
0.01
0.49101 + 0.009303
0.39744 = 0.027059
0.10
0.49053 + 0.009130
0.37377 + 0.030239
1.00
0.49071 + 0.009039
0.38429 + 0.053008
5.00
0.49068 + 0.009474
0.26067 + 0.027564
50.00
0.52160 + 0.013804
0.06012 + 0.018358
100.00
0.83521 + 0.015799
0.34421 + 0.018229
120.00
1.10238 + 0.014253
0.43006 + 0.019893
500.00
40.32616 + 48. 18.938383
0.27088 + 0.139738
Table 15: RMSE and KS Difference with standard deviations for 5 runs KFFL with IID for the Adult
Dataset
36

---

Published in Transactions on Machine Learning Research (Apr/2025)
G Communication-Efficie Kernel Regularized Fair Learning
In this section, we address the computational and communication challenges of incorporating kernel-based fairness regularizers into federated learning. Specifically, we leverage Random Feature Maps (RFMs) to approximate kernel functions efficiently, reducing both computational complexity and communication over- head.
G.1 Random Feature Maps for Kernel Approximation
Kernel methods are powerful tools in machine learning but often sufeer from high computational complexity, especially when dealing with large datasets. Computing kernel matrices requires O(n²) memory and Onn)) computational time, which impractical for large n.
To overcome this, Rahimi & Recht (2007) introduced Random Feature Maps (RFMs) approximate shift- invariant kernel functions. A shift-invariant kernel x(x,y) x(x sxx-yy) y) can be represented using the Fourier transform via Bochner's theorem. Specifically, the kernel can be expressed as:
k(x,y) = J p(w) (xw))os"(<->i>>> (x-v)d
(29)
where p(w) is the spectral density function of the kernel K.
Constructing Random Feature Maps To approximate x(x,y), we draw random samples {w from p(w) and define the random feature map & TRd + RDD as:
VB 2
((x)
cos bı), cos(w bD)
(30)
where (bx are drawn uniformly from [0, 2m]
With this mapping, the kernel function can be approximated as:
x(x, R o(x) (ly).
(31)
Frequency of Drawing Random Features In our implementation, the random features are drawn once at the beginning global training round and are fixed throughout the optimization process. This ensures consistency across iterations and clients, and avoids the overhead regenerating random features at each iteration.
G.2 Computing Z, and Z
the context of our fairness regularizer, we need to compute feature maps for both the sensitive attributes S and the model outputs (X). Specifically:
Sensitive Attributes Feature Map (Zg): For each data point i, compute d(st), where s, is the sensitive attribute of the i-th sample. The matrix 2, E B has rows Φ(s;)
Model Outputs Feature Map (Zy): For each data point i, we compute o(fw (xi)), where fw (x4, is the model output (c.g., logits) for the i-th sample. The matrix Z E B has rows (x,)();))
(x,)),)))
Example: Gaussian Kernel Approximation As an example, consider the Gaussian (RBF) kernel:
x(x,y) = exp (- ||x-x0 y
(32)
20
37

---

Published in Transactions on Machine Learning Research (Apr/202)
The spectral density of the Gaussian kernel is p(w) = (w; 0 Therefore, to approximate the Gaussian kernel, we draw WP ~ (0, 2)) and compute the feature maps as:
2
((x)
[wos x b1), co + bb)]
(33)
Orthogonal Random Features To improve the quality of the approximation and reduce variance, we employ Orthogonal Random Features (ORF) as proposed by Yu al. (2016). Instead of sampling we independently, construct them to be orthogonal, which lead to better kernel approximations with fewer features.
38


Reviews:
[{"review_id": "4ThoagFBtV", "reviewer": "Reviewer_KtdY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper proposes Kernel Fair Federated Learning (KFFL), incorporating the Kernel Hilbert-Schmidt Independence Criterion (KHSIC) as a fairness regularizer. This is one of the first applications of KHSIC to federated learning (FL), providing a novel approach to enforcing group fairness in FL.\n- The authors propose using Random Feature Maps (RFM) to address the high computational and communication costs associated with kernel-based methods.\n- The paper presents comprehensive experiments on both IID and Non-IID federated settings, using multiple datasets.\n- The authors also present the Pareto frontier, which helps in understanding the trade-offs between fairness and performance.\n\nWeakness:\n- To derive equation 8, the sensitive information $s$ needs to be clearly defined. However, In image-based tasks or unstructured data, sensitive attributes are not explicitly labeled. For instance, the skin tone and gender of a person in the image could be the sensitive information in image-based tasks. This makes KFFL hard to apply in real-world cases where fairness-sensitive features are implicit.\n- As the authors mentioned, $s_i$ contains sensitive covariates (which may be a binary scalar $s_i$ or multi-dimensional vector $s_i$). However, if I understand correctly, $s_i$ is always chosen as a binary vector in the experiments, even if it could actually be both race and sex in the ADULT dataset. Could the authors explain the reasoning behind this choice? \n- In Sections 6.2 and 6.3, the authors select specific trade-off points to showcase the performance of KFFL. However, if I understand correctly, FairFed and other methods do not explore the Pareto frontier. This may lead to an unfair comparison, where KFFL appears superior simply because other methods were not evaluated across the full Pareto frontier.\n- Typically, adding fairness constraints reduces accuracy, but Tables 4 & 5 show no clear drop in accuracy when increasing fairness regularization, especially considering the standard deviation. This may indicate that the dataset does not contain strong predictive features tied to the sensitive attribute. I would appreciate it if the authors could explain why accuracy remains almost stable, as this is important for understanding the real impact of fairness constraints.\n- In practical applications, companies often have a predefined fairness constraint (e.g., \"We need SPD ≤ 0.05\"), rather than needing to explore the entire Pareto frontier. Therefore, while Pareto frontier analysis is useful for research, it may not be necessary for practical applications. Could the authors elaborate on this? Additionally, the paper could discuss how to efficiently select a fairness constraint without requiring full Pareto exploration.\n- As the authors themselves mentioned, they only consider full client participation. I am wondering if this choice is purely due to computational cost and/or time constraints.\n- In Figures 3 and 4, how are the curves for KFFL and KFFL-TD drawn? It appears to be an interpolation between points—please correct me if I’m wrong. Additionally, it seems that FairFed outperforms KFFL-TD in some parts of the Low SPD Region and Low EOD Region. If the fairness budget were chosen differently, would FairFed consistently outperform KFFL-TD?\n\nMinor:\n- In Figure 1 (bottom right), the scale still shows that one weighs more than another even though the text shows Does not discriminate against gender.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Kernel Fair Federated Learning (KFFL), a framework for ensuring group fairness in federated learning (FL). The authors use the Kernel Hilbert-Schmidt Independence Criterion (KHSIC) as a fairness regularizer to minimize statistical dependence between model predictions and sensitive attributes (e.g., race, gender). To address the high computational and communication costs of kernel-based methods, the authors approximate KHSIC using Random Feature Maps (RFM). They also introduce FedProxGrad, a federated proximal gradient algorithm, to handle the resulting non-convex optimization problem while ensuring convergence.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZojJLadFit", "reviewer": "Reviewer_c3nu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. Addresses an Important Challenge in FL\n    * The paper tackles the well-known problem of ensuring fairness in federated learning while maintaining efficiency.\n2. Theoretical Soundness and Convergence Analysis\n    * Using KHSIC as a fairness measure is a strong theoretical choice, capturing non-linear dependencies effectively.\n    * The mathematical formulation is solid, and the convergence analysis is well presented.\n3. Communication Efficiency through Random Feature Maps\n    * Leveraging RFMs to approximate KHSIC significantly reduces communication costs, making it feasible for FL.\n    * The communication efficiency analysis is well explained and highlights the advantages of KFFL over naive kernel methods.\n4. Comprehensive Empirical Evaluation\n    * The experiments compare KFFL against strong baselines (FedAvg, FairFed, MinMax) on classification and regression tasks.\n5. Generally Well-Written Paper\n    * The paper is mostly well-structured and readable, though some citations are missing, and certain word choices create confusion (see Weaknesses).\n\n\n**Weaknesses**\n1. Limited Novelty in Algorithm Design\n    * The algorithm is not fundamentally new—it is essentially FedProx applied to a regularized problem.\n    * While the combination of KHSIC regularization and RFMs is interesting, it does not introduce a fundamentally novel optimization framework.\n2. Misrepresentation of Definition 2 and Convergence Rate Issues\n    * The (G, B)-Bounded Dissimilarity condition is not exactly the one used by Li et al. (2020) and Karimireddy et al. (2020). The authors should cite the correct work and clarify the assumptions.\n    * Theorem 1 states a $1/T$ convergence rate, but the standard worst-case rate for FedProx is $1/\\sqrt{T}$ (see On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness, and Beyond, arXiv:2206.05187). Could the author explain the difference?\n    * The assumption that $B$ is finite is not always valid. The authors should provide a more systematic convergence rate analysis for different values of $B$, especially when $B=\\infty$.\n3. Lack of Discussion on Related Work in Random Feature Approximations\n    * The idea of using random features to reduce communication costs is not new. \n    * A relevant paper, A Random Projection Approach to Personalized Federated Learning: Enhancing Communication Efficiency, Robustness, and Fairness (JMLR 2023), also uses a random matrix and studies fairness but is not cited in this paper. A discussion with it is preferred.\n4. Computational Overhead Not Analyzed\n    * While KFFL reduces communication costs, it still adds computational overhead due to KHSIC and its gradient computation.\n    * The paper lacks a runtime analysis comparing the wall-clock time of KFFL to baseline methods.\n5. Hyperparameter Sensitivity Not Addressed\n    * The choice of the fairness weight $\\lambda$ is crucial, but its impact is not well-analyzed.\n    * How should practitioners select $\\lambda$ in practice? A sensitivity analysis would be valuable.\n6. Unclear Definition and Analysis of \"Pareto Frontier\"\n    * The term “Pareto frontier” is mentioned multiple times in Section 6.1, but its exact definition is unclear.\n    * Is it simply a curve in the plot of group fairness vs. accuracy?\n    * Figure 3 is somewhat hard to read, and its analysis is far away from the figure.\n    * The meaning of the solid curves is unclear—are these the so-called Pareto frontiers?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Kernel Fair Federated Learning (KFFL), a framework aimed at ensuring group fairness in federated learning (FL). \n\nThe key innovation is the use of the Kernel Hilbert-Schmidt Independence Criterion (KHSIC) as a fairness regularizer.\n\n Since computing KHSIC directly is computationally expensive and communication-heavy in FL, the authors propose using Random Feature Maps (RFMs) to approximate KHSIC efficiently. T\n\no handle the resulting non-convex optimization problem, they introduce FedProxGrad, a federated proximal gradient algorithm that provides convergence guarantees.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical (Necessary for Acceptance)**\n1. Clarify Algorithmic Novelty\n    * Explain what is new about FedProxGrad or conform that it is simply applying FedProx to a regularized problem.\n2. Correct Definition 2 and Convergence Analysis\n    * Properly cite the correct (G, B)-Bounded Dissimilarity definition.\n    * Provide a convergence rate analysis for different values of $B$, particularly when $B = \\infty$.\n3. Include a Hyperparameter Sensitivity Analysis\n    * Conduct experiments analyzing the impact of $\\lambda$ on fairness-accuracy tradeoffs.\n4. Add Related Work Discussion on Random Feature Methods\n    * Discuss the similarities and differences between KFFL and A Random Projection Approach to Personalized Federated Learning.\n5. Clarify and Improve the Explanation of the Pareto Frontier\n    * Clearly define what the Pareto frontier represents in the context of FL fairness.\n    * Improve the readability of Figure 3, clarify the meaning of the solid curves, and ensure that the figure is analyzed in the appropriate section.\n\n**Recommended (Would Strengthen the Paper)**\n\n6. Include a Runtime Comparison to Baselines\n    * Report training times for KFFL and FedProx across different FL settings.\n7. Explain Why the Model Uses Only Non-Sensitive Covariates ($x$) as Inputs\n    * It is unclear why $s$ is excluded—a common belief is that more features improve performance.\n8. Clarify Why Traditional Fairness Metrics Are Not Directly Applicable in FL\n    * The paper states that Equalized Odds (EOD) and Statistical Parity Difference (SPD) are not directly applicable in FL (Page 4).\n    * Why? What challenges does FL introduce that make these fairness measures difficult to use?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "B2lBUXN9EV", "reviewer": "Reviewer_JkPu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n1. The paper is overall well-written and well-structured. The basic concepts related to group fairness in FL have been appropriately introduced and are easy to follow.\n2.Experimental results seem solid and convincing, and various settings and datasets are considered to provide experimental validation.\n\n**Weaknesses**:\n1. I think the overall motivation for applying KHSIC for group fairness is not very convincing. This is because we know that KHSIC may lead to extra communication and computation costs (which the authors address by using random feature maps). Therefore, I find the motivation not particularly compelling.\n2. Figure 2 seems somewhat unclear. Based on Algorithm 2, it appears that the FAIR1, FAIR2, and DATA rounds do not occur in the same round. However, in Figure 2, it looks as if they happen simultaneously. Moreover, why do the authors choose DATA as a component in both the algorithm and the figure? I think this may lead to some misunderstanding, namely that clients need to send data to the server in certain rounds, which could violate a key constraint in FL.\n3. Table 2 seems confusing. Why are the rounds of communication different for each method?\n4. Can the authors also provide an exact computation time comparison for the baselines?\n5. Regarding Table 6, based on which rule is bold formatting applied? For example, for the SPD metric, it seems that KFFL and KFFL-TD are neither the largest nor the smallest SPD scores.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies group fairness in federated learning by introducing the Kernel Hilbert-Schmidt Independence Criterion (KHSIC) as a fairness regularizer in the FL framework. To further reduce computational and communication overheads, Kernel Fair Federated Learning (KFFL) is achieved using random feature map approximation. The authors provide theoretical analysis, and experimental results on both IID and non-IID settings demonstrates the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I suggest that the authors address the weaknesses if necessary.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission has received expert reviews from the domain who provided detailed and technical comments for the authors to further improve the submission. Most of the comments from Reviewer c3nu and JkPu have been properly addressed by the authors during the rebuttal period, including clarification on the discrepancy of convergence rate with existing work, as well as citation/discussion of related work in using random Fourier features to reduce the communication costs. 

However, there are several comments from KtdY, mostly regarding experiments, have not been well addressed by the authors. In particular, despite the claim of KHSIC being able to work in multi-group or multiple sensitive attributes setting, the experiments in the paper only include binary groups. Furthermore, I also concur with the reviewer's comments on the presented Pareto-front: at least in the large $\lambda$ regime, I would anticipate a drop in accuracy to compensate for fairness. The authors need to address the above two points in their revision:

- Add at least one more dataset in the experiment where there are multiple sensitive attributes (gender and race for example) or a single sensitive attribute with more than 2 groups. 

- Show a more wholistic view of the Pareto-front by also including large values of $\lambda$.

In terms of novelty, from a technical perspective, there is a consensus from the reviewers that novelty may not be enough to warrant a journal-to-conference track presentation -- the use of KHSIC to enforce statistical independence is not new in centralized setting; the use of random feature maps to avoid the explicit computation of kernel matrices is also not new; That being said, the overall presentation of the paper is clear and sound, hence the overall decision of accept with minor revision but no journal-to-conference presentation.
Decision: Accept (poster)