Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairness
Authors: Mojtaba Kolahdouzi
Hatice Gunes
Ali Etemad
Venue: NeurIPS 2025 spotlight ()
Keywords: Group fairness
Optimization
Stochastic differential equations
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We show that adaptive optimizers like RMSProp lead to fairer minima than SGD, both theoretically and empirically.

Abstract: We study whether and how the choice of optimization algorithm can impact group fairness in deep neural networks. Through stochastic differential equation analysis of optimization dynamics in an analytically tractable setup, we demonstrate that the choice of optimization algorithm indeed influences fairness outcomes, particularly under severe imbalance.  Furthermore, we show that when comparing two categories of optimizers, adaptive methods and stochastic methods, RMSProp (from the adaptive category) has a higher likelihood of converging to fairer minima than SGD (from the stochastic category). Building on this insight, we derive two new theoretical guarantees showing that, under appropriate conditions, RMSProp exhibits fairer parameter updates and improved fairness in a single optimization step compared to SGD. We then validate these findings through extensive experiments on three publicly available datasets, namely CelebA, FairFace, and MS-COCO, across different tasks as facial expression recognition, gender classification, and multi-label classification, using various backbones. Considering multiple fairness definitions including equalized odds, equal opportunity, and demographic parity, adaptive optimizers like RMSProp and Adam consistently outperform SGD in terms of group fairness, while maintaining comparable predictive accuracy. Our results highlight the role of adaptive updates as a crucial yet overlooked mechanism for promoting fair outcomes. We release the source code at: https://github.com/Mkolahdoozi/Some-Optimizers-Are-More-Equal.

Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairness
Mojtaba Kolahdouzi', Hatice Gunes", Ali Etemad!
Department of Electrical and Computer Engineering, Queen's University, Canada
Department of Computer Science and Technology. University of Cambridge, United Kingdom ww..olhhoozii@uueeeeeeee @queessu. ca, hatice. geeeeeungggse......ee...e......e cam. ac uk, ali. etteeeedduuueensu.u....
Abstract
We study whether and how the choice of optimization algorithm can impact group fairness in deep neural networks. Through stochastic differential equation analysis of optimization dynamics in an analytically tractable setup, demonstrate that the choice of optimization algorithm indeed influences fairness outcomes, par- ticularly under severe imbalance. Furthermore, we show that when comparing two categories of optimizers, adaptive methods and stochastic methods, RMSProp (from the adaptive category) has higher likelihood of converging fairer minima than SGD (from the stochastic category). Building on this insight, we derive two new theoretical guarantees showing that, under appropriate conditions, RMSProp exhibits fairer parameter updates and improved fairness in a single optimization step compared to SGD. We then validate these findings through extensive periments on three publicly available datasets, namely CelebA, FairFace, and MS-COCO, across different tasks as facial expression recognition, gender clas- sification, and multi-label classification, using various backbones. Considering multiple fairness definitions including equalized odds, equal opportunity, and demographic parity, adaptive optimizers like RMSProp and Adam consistently outperform SGD terms of group fairness, while maintaining comparable predic- tive accuracy. Our results highlight the role of adaptive updates as a crucial yet overlooked mechanism for promoting fair outcomes. We release the source code at:
Introduction
Machine learning is increasingly applied to a wide variety of domains, many of which have significant social implications 5 43) These include, but not limited to, decision-making systems, risk assessment models, recommendation engines, and personalized health interventions [2] 46). These systems influence people's lives and lead to unintended biases disparities (38), for instance by treating different groups of people differently. To address this, researchers have developed various techniques ensure group fairness in machine learning algorithms 20 13)
Prior work has demonstrated that a variety of factors such as training data imbalance 45 and model hyper-parameters [10] can impact the performance of deep neural networks in terms of group faimess and bias. In this paper, we pose the following question for the first time:
Does the choice of optimization algorithm impact group fairness in deep neural networks? and how?
The limited theoretical and empirical analysis in this area leaves major gap in our understanding of how optimizers impact the inherent performance of models in terms of bias. Consequently, we believe that this question can have significant implications in developing models that perform consistently across various groups, as optimization a fundamental component of any deep learning model.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this paper, we bridge this gap by first analyzing the impact of optimization in group faimess in an analytically tractable setup. This initial study confirms our hypothesis: that indeed the choice of optimizer impacts group fairness in downstream applications, particularly in the presence of severe bias in the training set. Next, we delve deep into the problem by proposing and proving two new theorems that demonstrate that among the two primary families of optimization algorithms, adaptive gradient methods such as Adam and RMSProp, and stochastic gradient methods such as SGD, the former is more likely to exhibit better group faimess.. Next, we validate our theoretical findings by conducting extensive experiments on three publicly available datasets-MS-COCE and FairFace [16] across three different tasks: multi-label classification, facial expression recogni- tion, and gender classification. We carefully evaluate fairness using three widely adopted metrics: Equalized Odds, Equal Opportunity, and Demographic Parity. The results are strongly aligned with our theoretical analysis, consistently demonstrating that adaptive gradient methods achieve better fairness scores compared to stochastic gradient methods.
[27], CelebA [II,
Our contributions in this paper are summarized as follows: (1) First, we provide a detailed mathemat- ical analysis to study the impact of optimization algorithms fairness in an analytically tractable setting. Here, we demonstrate that adaptive gradient algorithms like RMSProp have a higher proba- bility of converging to fairer minima. (2) Next, we establish and prove two theorems showing that under appropriate conditions: (a) RMSProp provides fairer updates than SGD, and (b) the worst-case bias introduced by RMSProp in single optimization step has upper bound equal to that of SGD. These theoretical insights highlight key properties of optimization algorithms that contribute to their advantages in terms of fairness. (3) Through extensive experiments on multiple publicly available datasets and tasks, we empirically validate our theoretical findings. We systematically evaluate the fairness of models optimized using adaptive gradient methods and compare the outcomes to those obtained from stochastic gradient methods. Our results consistently show that adaptive gradient methods lead to fairer minima. (4) To enable rapid reproducibility and contribute to the area, we make our code publicly available at:
2 Related work
Group fairness. Bias in machine learning models Ieads to discriminatory outcomes, and these models are increasingly being deployed in real-life applications, such biases can impose risks to society B8. This risk is particularly important in high-stakes applications, such as hiring [17), healthcare (2) etc. When models yield biased outcomes, they favor certain demographic groups, which reinforces social inequalities. As an example, facial expression recognition models show higher error rates for darker skin tones (55). Such biases undermine trust in machine learning models and pose ethical and legal challenges. To analyze fairness in machine learning, researchers in this area often distinguish the notion of "group fairesss" and "individual fairness" [38] Group fairness focuses on the notion that machine learning models treat different demographic groups, such as those defined by race, the same. However, individual fairness dictates that similar individuals should be treated similarly. In this paper, we focus on group fairness, which is more widely recognized in practical applications 77].
One of the main sources of bias in machine learning models is data imbalance 9. When certain demographic subgroups are underrepresented. models prioritize patterns from the majority group, leading biased decisions. To tackle this, a variety of methods have been proposed that can be categorized into 3 main groups [6: pre-processing, in-processing, and post-processing. Pre- processing methods often apply some type of transformation to data make the demographic information less recognizable to the model. As an example, [48) augments the training set with synthetically generated samples, thereby ensuring an equal distribution for demographic subgroups. In-processing methods directly modify the learning algorithm of the model. As an example, [20] proposes new loss function based on maximum mean discrepancy statistical distance enhance fairness. Post- Post-processing methods intervene after training is completed. For instance, [4]] introduces a post-processing approach using graph Laplacian regularization to enforce individual fairness constraints while preserving accuracy. Nevertheless, in practice, these fairness-enhancinn are rarely used in real-world pipelines [9. One major limitation is their computational overhead or interference with the training phase. As an example, pre-processing methods require extensive data augmentation or transformations. Considering these challenges, an alternative approach to enhance faimess is to use components that are inherently present in every training pipeline. One of
methods
2

---

the fundamental building blocks of modern deep learning is optimization [2 Understanding the role of different optimization algorithms in fairness is crucial for practitioners seeking to develop fairer models without additional specifc modifications. The work in [29] provides a key theoretical analysis, showing that standard and unconstrained machine learning implicitly favors one fairness criterion over others. The authors prove that model's deviation from calibration is bounded by its excess risk. This is related to our work as it also investigates the fairnsss properties that emerge from standard training process. Furthermore, [56] focuses on in-processing fairess methods that use surrogate functions to ensure fairness. Their theory also highlights that balanced datasets are beneficial for achieving tighter fairness and stability guarantees. This is relevant our findings, as we also identify data imbalance as a critical factor through stochastic differential equation analysis.
Optimization algorithms. Optimization algorithms in deep learning generally fall into two main categories: SGD and adaptive gradient methods (e.g.. Adam [88 and RMSProp (4 SGD updates
model parameters using constant learning rate [5]. while adaptive methods dynamically adjust per-parameter learning rates based on past gradients, often leading to faster convergence [5] The impact of these two types of optimization algorithms Nonetheless, several studies have explored their effects in other areas. For instance, [34] empirically demonstrates that models trained with SGD are more robust to input perturbations than those trained with adaptive methods. Fairness and robustness are often competing objectives in deep learning [3). This hints that adaptive optimizers may yield better faimess compared to SGD. However, this relationship remains largely unexplored.
group faimess remains not well understood.
3 Method
3.1 Preliminaries
Optimization algorithms. Stochastic gradient methods like SGD and adaptive gradient methods such as RMSProp and Adam are widely used for optimizing deep neural networks [58]. Given a loss function L(w), SGD updates the model parameters at each iteration k as:
wk++1 = w/k 17 [(wx),
(1)
where 1) denotes the learning rate. Adaptive gradient methods improve the speed of convergence by applying coordinate- wise scaling to the gradient AT) Specifically, RMSProp normalizes the gradient with an exponentially decaying average of its squared values:
Uk- = - + (1 7) ((w)) Wk+1
v [(Ww), V +
where in is the moving average of squared gradients, y is the decay factor, and c is a small constant added for numerical stability. On the other hand, Adam works by considering both the first and the second moments of gradients:
my = -Bmm--1 + (1 B1 ((ww),
(3)
0/
32 (1 10-B22) V
(4)
17
Uk
my
0g
(5)
3
35
11
Wk+1
mp.
(6)
V 0k + e
where 31 and 32 the first and second moment decay factors, respectively.
Stochastic Differential Equations (SDE). Given the interval [0, 7] where 7 0, we have
dX,- = (t), t) b)d((),
where Xe E Rdd is a stochastic process (the solution of the SDE), a Rd x [0, + Rd is the drift term, b Rdd x [0, T + R the diffusion term, and B()) E R is s-dimensional Brownian motion. Intuitively, SDEs extend ordinary differential equations (ODEs) by incorporating stochastic noise. Analyzing functions involving stochastic processes requires tools beyond classical calculus. fundamental theorem in stochastic analysis is Itô's 's lemma, extending the chain rule to the functions
3

---

of stochastic processes M. Please see the Appendix A for Itô's lemma. Unlike ODEs, the solution of an SDE is a stochastic process X,. Formally, the solution can be represented as:
(aaaaaaxx,,, X = Xo + L. a((x,,) + S. (Xx,
(8)
The second integral, also known as the Itô's integral, captures the stochastic contribution driven by Brownian motion. It is defined as:
L b(x,,))dB = lim
Σ 7k- -Bru-1:
n=000
where πt (n) denotes a sequence of n partitions in [0, f), and lim shows convergence in probability. In many cases, explicitly computing X, is intractable. Instead, it is often more practical to analyze the probability distribution of X,. The Fokker-Planc' equation, also known as the forward Kolmogorov equation, descríbes the time evolution of the probability density function p(æ, t) of X 21 as:
ap(a, t) - (a(x, ()p(z, f)) + 1 v (V (b(a, t)b(,, ôt
(10)
where V. represents the divergence operator. For completeness, we provide the derivation of the Fokker-Planck equation from Eq. in Appendix B
SDE approximations for optimization algorithms. SDEs provide a powerful framework for analyz- ing the dynamics of discrete optimization algorithms such as SGD and RMSProp in continuous setup. While direct analysis of discrete updates often challenging [26). SDE approximations enable the study of optimization trajectories. In the context of optimization, the sources of stochasticity include mini-batch gradient noise which are variations due data heterogeneity (e.g., differences in mini batchesteeee drawn from different subpopulations) among others. To formalize this, we adopt the Noisy Gradient Oracle with Scale Parameter (NGOS) [37,, which models stochastic gradients as ((w) = V ((w) + 22, where θ is noise scale parameter, and 2 is a random variable that follows a distribution with covariance matrix E(w) and mean of zero. NGOS formally models the gradients used in mini-batch training. It describes the gradient from small data batch as the "true" gradient (which would come from the full dataset) plus random noise introduced by the sampling process. A well-behaved NGOS defined as V((w) being Lipschitz and / (w) being bounded. See Appendix C for detailed description of well-behaved NGOS. Note that these assumptions are common and standard in prior works that study optimization using SDEs 225
Under well- behaved NGOS, SGD can be approximated by the following SDE 25
dW.- = -VV, + ()E(())))?dd(())
(11)
where W, represents an approximation of te at discrete time steps kn. More recently, SDE approxi- mation for RMSPPop has been proposed as a coupled system of equations 337:
dW, PP- ( (VL(W.)dl On +0 1/3 (V.))((.))))))))
du,
1 1-1 (((.)) 1/2
(12)
where Pt = ④ n diag(u;) + I and Uy is defined as S. Note that these SDE approximations converge uy in the weak sense, meaning that while individual trajectories of the discrete and continuous processes may differ, their probability distributions remain close. Please see Appendix D for a formal formulation of the convergence.
3.2 SDE analysis of optimizers
To illustrate the different behaviors of SGD and RMSProp in terms of group fairness, let's consider warm-up example. this example, assume training set consisting of two equally-sized subgroups, denoted subgroups 0 and 1. That is, each group constitutes 50% of the total population. Please note this example can be generalized over more than 2 subgroups. For simplicity, assume each subgroup has its own loss function, denoted as:
Co(w) ] (w- 1)2, (w) 7(u (w+ + 1)2.
(13)

---

The objective is to minimize a population- -lvell loss function defined as the weighted sum of the subgroup losses as:
Spop(u) (w) 0.5.o (w) 0.5C, (w.),
(14)
where the weights correspond the proportion of each subgroup in the population. In this setup, the optimal solution for subgroup 0 is wg = 1, while the optimal solution for subgroup 1 is wi = 1. However, The population's 's optimal solution is obtained by minimizing Ppop (w), leading to Whop 0.
The following lemma shows that under the demographic parity definition of group fairness (38 Wppp is the fairest minima.
Lemma 1. Let = and (i = 1)2 be the loss functions for subgroups
O and 1, respectively. Define the population loss as Lpop (w) = 0.5Co (w) + 0.521 (w). Under the demographic parity definition of fairness, the fairest minimizer of Spop (w) is Wpup 0.
Proof. See Appendix
In practice, we do not have access to the full population above, rather we have access to the set Q of N individual samples, sampled uniformly from the population. Additionally, set Q may not contain an equal number of samples from subgroups O and 1, thus introducing bias. We assume that the probability of subgroup in f is po while the probability of subgroup is P1 = 1 Po- For given set S2, the empirical minimization problem used in practice is defined as: Lemp(w)
I Ereo La. (w), where 4r E {0. 1 corresponds to samples from subgroups O and 1. Note that as N + 00, Lemp -> pop In min-batch training with batch size of 1, at each iteration, the optimizer processes single individual from subgroup 0 with probability of po and from subgroup 1 with a probability pt. In other words, with both SGD and RMSProp, the model receives VLo (w) with probability of po and (w) with probability of pl. updating according to Eqs. and 2 respectively. po > pt, the optimization trajectory is biased towards the group-specific minimum at w 1, and = - otherwise. This demonstrates that disproportionate sampling of subgroups leads to unfair minima, which is particularly relevant in practical settings. The following theorem establishes that when the sampling bias lpo Pı| exceeds certain threshold, RMSProp has higher probability of converging to Wpop compared to SGD.
1 be the subgroup sampling probabilities for the loss functions Lo (w) = (1) and (i(w) = = (w++1)² Suppose we optimize the empirical objective Lemp(w) (w) = N. Eres L.g. (w), where each sample qr E {0, 1} is drawn i.i.d. with probability po for subgroup O pı for subgroup 1. Consider mini-batch gradient updates size 1 (i.e., one sample per iteration) using SGD and RMSProp optimization algorithms. Then there exists a constant
Theorem 1. Let po: P1 E 1) with po + P1
A (ppiP.n) > such that, whenever lpo - Ppl > A(p1P2, 1), we have: Prom (w
> 1, where ( wpop) are the probabilities of RMSProp and SGD converging to fair minima oop = 0, respectively.
Pead(woop)
Prms (wpp and paga
Proof. As mentioned earlier, the gradient of Lamp is (u with a probability po and VL, (w) with probability of pt. Thus, the covariance is E(W,) = 4pop1- This means E(W, is independent of W.. When convergence occurs for RMSProp, U, defined in Eq. 12 converges to (W,) Thus, ue 4pop1. With this in mind, let's derive the SDE for R.MPProp:
dW.- = n22O/PoPT + c) (V (V (W.)dt + 20n / PPo).dB.).
(15)
Here, is added for numerical stability, but it can be ignored as it is commonly done so in prior work [37]. Next, let's derive the SDE for SGD:
dW. = - Lpop
+ P/T/Pi
(16)
For analyzing the two SDEs above, we use the Fokker-Planck equations, presented in Eq. 10 convergence occurs, 3p(w.)) at = 0. Thus, Fokker-Planck for RMSProp can be written as:
When
i (p(w) V Cpppp(u) (w)
1 1P
0.
(17)
Dw
2110 2110~ /PoPI
Ow
5

---

Substituting VL poo = 2 (w- - +)+ + (w + 1) in the equation above, we obtain:
0 (p(w) po - 1) + pi 1
119((w) O.
+ Ow?
(18)
ow
4rjO/PoP1
Similarly, the Fokker-Plann equation for SGD can written as:
(p(w) x Po 1) + pi(w 1) + 2mpop1 ppp(w) Ow?
= 0.
(19)
dw
2
Eq. i8 an analytically solvable ordinary differential equation, which can be written as:
Prms(w) V ni eap (-r(w (po - -P)))))
(20)
where / As sanity check, can see that the expectation of the distribution in Eq. 20 equal to Po P1. This means in the case of po = Pr, ie. no sampling bias, the weight converges to w" = 0, the unbiased minimum. Similarly, we can write the analytical solution of Eq. 19
as:
Paya w) = V erp 1(w (Po - --1))))
(21)
where Supopt we use batch size of we can set Q inequality holds. With this mind, by substituting Wpop = 0 in Eqs. and 21
Then, >= 2/PoP1
we can conclude
Prems (1r
holds if and only if [po pıl A(p1p,,)) where A (p1P2. 7)
In
Paga (1rpp)
Now, let's perform quick simulation verify
100
Po = 0.1
100
Po =
0.3
our findings. We use the loss functions de-
Proportion near Map
SGD RMSProp
SGD
fined in Eq. [3 and set po 0.1 and pı 0.9
RMSProp
60
60
to induce severe sampling bias. We fix the
40
40
learning rate 1) 0.1 and apply SGD and
20
20
RMSProp for 100 epochs. Each experiment repeated 1000 times, and at every epoch we record the fraction of runs converging to the neighborhood of the fair optimum Wpop- de- fined by |w- ww- topl 0.2. The result is shown in Fig. [ (eet). As it is evident from this figure, approximately 10% of the runs under RMSProp lie in the neighborhood of pop. whereas SGD rarely converges to this fair region. Now, let's induce milder bias by setting Po 0.3 and p1 0.7 and run the same experiment. The results are illustrated in Fig. (right). As shown in this figure, in the milder bias, the two optimizers behave more similarly, yet RMSProp attains a slightly higher fraction of solutions near pop. These observations align with our theoretical findings in Theorem To show that our findings are true for a wider range 1 and alternative definitions of the fair neighborhood. we refer the reader to Appendix I for additional experiments.
0
100
0
100
Epoch
Epoch
Percentage of 1000 runs converging within the fair neighborhood for SGD and RM- SProp, under severe bias (po 0.1) and mild bias (po 0.3).
Figure 1:
3.3 Group fairness analysis
The detailed SDE analysis carried out in Subsection 3.2 highlights the distinct behaviors of SGD and RMSProp concerning group fairness. However, extending this analysis higher dimensions, particularly through the Lokker-Planck, Eq.[0] introduces significant complexity. To address this challenge, we present two theorems that leverage alternative mathematical tools beyond SDE, to study the impact of SGD and RMSProp on group fairness. First, we illustrate how adaptive optimizers like RMSProp suppress subgroup disparities at the level of parameter updates. The second theorem quantifies the consequence, showing that this mechanism translates into provably smaller fairness violations in terms of demographic parity.
Theorem 2. Consider population that consists of two subgroups with subgroup-specific loss functions Lo(w) and Li (w), sampled with probabilities pa and pv. respectively. Suppose a stochastic (online) training regime, in which each parameter update is computed from sample drawn from

---

one of the two subgroups. Suppose the gradients V Lo(wx) and V Li (w0k) are well well-behaved isotropic NGOs, namely N (Ho. I and [(,, III,), respectively. Then, the difference in parameter updates between subgroups O and under RMSProp has upper bound given by the corresponding difference under SGD. Consequently, RMSProp offers fairer updates across subgroups.
Proof. For SGD, the difference between the parameter updates across subgroups can be quantified as || Lo Vill However, as the RMSProp scales the updates using exponentially decaying average. computing the difference more complex. Let's compute the expected value of dy defined in Eq. 2
E E(ux] = k vo + (1 - -) > VV
(22)
Given that & (loss function) is a discrete mixture of Gaussians, LBV [
can be calculated as:
E [ po (Mū + 0 + Pı (+] 011).
(23)
where 1 denotes a vector of all Is. Substituting the equation above in Eq. 22 yields:
Elva] = 60 + (1 - Mpo (nă + 0 + Pi (0 + 1
(24)
The equation above demonstrates that after enough steps, given that y < 1, [[vx] converges to Eq. 23 Next, let's analyze the variance of the jth coordinate of Uk. Uky, see its spread. To this end, we compute the variance of VL3 using the law of total variance as:
var V 233 E [var ( V v2, [2]] + var(EIV22, 12])
(25)
where Z is random variable denoting the subgroup. Finally, assuming that (a) O and θ are larger than S and pr, as shown experimentally in prior work (37. and (b) there is bias in the training set (i.e., PoP1 is small), the variance var("k,) ) can be computed as O((1 ? (e + +f,)). For large % typical in practical training, var();,) becomes negligible, and thus 2y + E[vx]. Using this, we can approximate the difference between the parameter updates across subgroups for RMSProp as [ID(VCC VLi)ll, where D is a diagonal matrix, whose jjth element can be computed as: D3
As O > n12 and in stochastic (online) training setup, we
V (vē (vă, 16, + (ni, (1i,+ +0,,)+
can set O² + 1, then Djj < 1. In other words, the eigenvalues of matrix D is less than 1, thus proving the theorem.
Theorem 2 shows that RMSProp, unlike SGD, shrinks disparities between subgroups in the updates. In other words, RMSProp prevents large gradient disparities from dominating the training dynamics. Consequently, the final model is less likely be pulled toward the subgroup with the larger gradients. Over many iterations, this can help prevent large performance gaps between subgroups. While this does not establish a universal fairness guarantee, outlines why RMSProp's 's dynamics yield fairer minima across subgroups. We relax the isotropic noise assumption in Theorem 2 and generalize it to the anisotropic case in Appendix G The following theorem analyzes fairness through the lens of demographic parity and establishes that, under appropriate conditions in a single iteration, RMSProp's 's worst-case increase in the gap of demographic parity has an upper- no greater than that of SGD.
Theorem 3. Consider population that consists of two subgroups with subgroup-specific loss functions Co(w) and L1 (w), sampled with probabilities po and pr. respectively. Suppose a stochastic (online) training regime in which each parameter update computed from a sample drawn from one of the two subgroups. Suppose that the gradients Lo(w)) and V VL1 (wk) are well-behaved isotropic NGOs, namely N (Ho, T) and N(n, ORT), respectively, with Jn > po- Then, in expectation, the worst-case increase in the demographic parity gap after one iteration of RMSProp has an upper-bound no greater than the corresponding increase under SGD.
Proof. See Appendix H
Theorem Jeseemmisss that the adaptive learning rate of RMSProp, which scales updates based on past squared gradients, helps mitigate demographic parity gaps that can occur during training. whereas SGD does not offer such a mechanism. We relax the isotropic noise assumption in Theorem 3 and generalize the result to the anisotropic case in Appendix [
7

---

4 Experiments
Setup. We use three public datasets that widely used in this area: CelebA [1,, FairFace [16], and MS-COCO [27] CelebA and FairFace are datasets commonly used for group faimess analysis, while we also use general-purpose dataset, MS-COCO, for further completeness. The details about these datasets are presented in Appendix !J The training protocol and backbones used in our experiments are described in Appendix [K] We report the performance across different tasks using accuracy and F1. We evaluate group fairness using three widely recognized fairness criteria [38] equalized odds, equal opportunity, and demographic parity. A predictor fair under equalized odds definition if the true positive and false positive rates are equal across demographic subgroups. We measure the violation
of this criterion as:
p(ğ c. = ci,
plù = calu + + cc, p(ù c: 2ng plù c:ly # Cs 2ng)
FEOD min mind $.4.
min[min(
)
(26)
In the equation above, e demonstrates class i. A classifier satisfies equal opportunity if the true positive rate is equal across demographic subgroups. The related fairness measure is defined as:
=0=1 TPR
FEOP = min (J
(27)
ΣC JP
where TPR., denotes the true positive rate for the 7th subgroup on class Finally, predictor is said to be fair from a demographic parity point of view if the predicted label distribution is independent of sensitive attributes. The corresponding measure is defined as:
FDPA = min p C2 [zz)
(28)
9.99 p = C2 kzg)
For all the above fairness metrics, higher values indicate better (fairer) outcomes.
Implementation details. We use PyTorch (42 and up to four Nvidia A100 GPUs. The hyperparame- ter values are provided in Appendix D which have been optimized using Weights & Biases 4
Results and analysis. We first calculate the fairness metries for the ViT backbone across different datasets and different sensitive attributes (gender: G, race: R, and age: A). Fig. presents the results for SGD, RMSProp, and Adam. Considering F Adam consistently outperforms SGD across all scenarios, mostly by substantial margins. For instance, on CelebA dataset with gender as the sensitive attribute, Adam achieves an 8% improvement over SGD. Furthermore, RMSProp outperforms SGI) in out of 5 scenarios, and they both achieve zero FEOD for MS-COCO dataset. Given that MS- COCO contains some extremely underrepresented object categories, this result suggests that the disparity captured by FEOD is largely driven by rare categories receiving no correct classifications. The largest gap between RMSPPrp and SGD is observed on the FairFace dataset with race as the sensitive attribute, where RMSProp outperforms SGD by 9%. Another point to mention is that Adam and RMSProp yield comparable FEOD values, expected. Considering FEOP, both Adam and RMSProp outperform SGD in all 5 scenarios. To shed more light, the largest difference between the two is 6% in the case of FairFace dataset with age the sensitive attribute. Additionally, unlike SGD, both RMSProp and Adam achieve near-perfect FEOD in the case of CelebA dataset with gender the sensitive attribute, indicating their effectiveness in reaching fairer minima.
100
100
60-
75
a0-
Fooe1%)
50-
SGD
20-
RMSP
80-
RMSProp
25
RMSProp
Adam
Adam
G
PairFece-R
Falrficedd
GSCOCO.G
G
PulrPace- A
FairFased
R
MSCOCO.G
G
Pairfamn CelabAd
Fuirfeced
MSCOCO.G
Figure 2: Fairness for ViT across different datasets, attributes (G: gender, A: age, race), and metrics.
Regarding FDPA, Adam consistently outperforms SGD, while RMSProp surpasses SGD out of 5 scenarios. Both optimizers achieve zero FDPA MS-COCO with gender the sensitive
8

---

attribute. Interestingly, the largest discrepancy between RMSProp and SGD occurs on the FairFace dataset with age as the sensitive attribute. This aligns with our theoretical insights from Theorem [ Specifically, FairFace exhibits extreme imbalance, with minority-to-ai ratio of approximately 0.9%, whereas CelebA with gender as the sensitive attribute has substantially higher ratio of 42%. These findings highlight that as dataset imbalance intensifies, the gap between adaptive optimizers and SGD becomes more evident. We refer the reader to Appendix M for the results on fairness metrics for other backbones across CelebA, FairFace, and MS- datasets with different sensitive attributes. The results are well-aligned with the results above. To further validate our findings across different variants of optimizers, conduct additional experiments with SGD w/ momentum [30]. AdamW [32. and AdaBound
33] optimizers. The results are provided in Appendix N
Beyond tuning the learning rate, we further inves- tigate the influence of additional hyperparameters on fairness outcomes. Specifically, we extend the hyperparameter tuning using Bayesian Optimiza- tion over the learning rate, decay rate, and batch size. This comprehensive search is performed using the VIT backbone the CelebA dataset, considering gender and age as sensitive attributes. The results are reported in Table [T As shown, adaptive optimizers like Adam still achieve better fairness compared to SGD. This confirm that the faimess trends observed in our main study remain consistent even when multiple hyperparameters jointly tuned.
Table 1: Fairness comparison across optimizers with tuning learning rate, decay rate, and batch size.
Gender
RMSProp
Adam
RMSProp
Fbon
65.21
65.18
62.66
72.34
71.99
68.
Pbor
99.90
99.91
96.60
10.10
99.10
97.77
Fbes
73.50
73.68
60.80
98.20
96.20
95.10
Table 2: Comparison of accuracy and F1 across different optimizers and datasets.
To demonstrate that our training is successful and the models are well-behaved, we evaluate the classification performance of ViT on the three datasets for facial expression recognition, multi- label classification, and gender classification, re- spectively. The results are presented in Table 2 The performance of the other models are pre- sented in Appendix O Here, can confirm that in most cases, all three optimizers exhibit comparable accuracy and F1 scores across all tasks. For instance, on the CelebA dataset, SGD and RMSProp achieve an accuracy of 91%, while Adam attains 92%. This trend is consistent with previous findings in the prior work [34]. As evident, the improved fairness observed with adaptive gradient algorithms cannot be solely attributed to their performance. For instance, on the CelebA dataset, the Fl score for SGD is higher than that of RMSProp, whereas on FairFace, RMSProp achieves a higher Fl score than SGD. However, in both cases. RMSProp demonstrates better fairness performance, as shown in Fig. 2 This further supports our theoretical conclusion that RMSProp fairer than SGD, regardless of overall performance.
Accuracy
Fl score
Dataset
SGD RMSProp Adam SGD
RMSProp Adam
CelebA
91.23
91.54
92.08 92.12
91.17
92.09
MS-COCO 89.62
89.71
90.03
68.35
71.03
74.10
FairFace
89.41
91.37
92,20 91.13
92.07
92.17
To rigorously assess the significance of fairness improvement of Adam and R.MPPrpp over SGD, we conduct the following experiment. We train ViT on CelebA dataset 10 times using SGD, RM- SProp, and Adam, while recording fairness met- rics for both gender and age attributes in each run. To evaluate statistical significance, we per- form paired Wilcoxon signed-rank test [3] on the fairness metrics and report the corresponding p-values in Table As shown in the table, most scenarios yield v-values in the order of 0.001, in- dicating a statistically significant improvement in fairness for adaptive gradient optimizers compared to SGD. The worst p-vaue in the table corresponds to SGD vs. RMSProp on CelebA with age as the sensitive attribute, However, even this value remains statistically significant under the conventional threshold of 0.05 [14 40).
Table 3: p-values from the Wilcoxon test com- paring SGD vs. RMSProp and SGD vs. Adam optimizers on the CelebA dataset for gender and age attributes.
Gender
Age
Metric
SGD-RMSProp SGD-Adam SGD-RMSProp SGD-Adam
FROD FROP FDPA
1 10 1 10 a
10-3
1 10
1x 10
1x 10
1 10 3
5x 10
10-3
1x
7x 10
3
Theorem T provides theoretical insight that under the demographic parity fairness criterion, RMSProp is more likely to converge to a fairer minimum than SGD in highly imbalanced datasets. Moreover. it predicts that as the dataset becomes more balanced, the fairness advantage of RMSProp over SGD

---

Table 4: Comparison of fairness metrics across optimizers, with and without Sairness-enhances methods. Lower values indicate better fairesss.
Gap in Equal Opportunity
Gap in Equalized Odds
Gap in Demographic Parity
Dataset
Adam
RMSProp
SGD
Adam
RMSProp
SGD
Adam
RMSProp
SGD
With fearffaiiinenniiie enhancing
ProPublica COMPAS
0.45
0.48
0.71
2.79
2.78
0.86
0.86
AduliCensus
3.15
3.16
3,38
2.39
2.41
4.28
7,00
6.80
11.79
Without fuirnes--nnhoncing
ProPublicu COMPAS
13,99
13.90
15.19
13.99
13.95
14.98
11.49
11:45
11.80
AduliCeasus 21.04
20.91
21.29
20.90
20.91
21.14
12.19
12.23
12:42
gradually diminishes. To empirically validate this behavior, conduct experiment where we ran- domly downsample the 'male' subgroup in the CelebA dataset from its native male-to-all ratio of 42% down 22% and 2%. We then train model on each downsampled version using both RMSProp and SGD. We repeat this process five times while recording the values of FDPA and FEOD- Fig. 3 shows the results, illustrating the absolute difference in P and FEOD between SGD and MMProp across different male-to-all ratios. As expected, the gap in demographic parity is maximum at the most severe imbalance level (2% male-to-all ratio). As the male-to-all ratio increases (indicating less imbalance), the absolute difference between the two optimizers gradually decreases, a trend that also holds for equalized odds. This empirical finding strongly aligns with the theoretical predictions in Theorem
Theorems and
general with respect to the loss function, which suggests their fairness benefits should be complementary to existing fairness-enhancing methods that add regularization terms to the primary loss. To val- idate this hypothesis, we conduct an experiment where we test the impact of optimizers in addition to the fairness- enhancing method [7] on two tabular datasets (ProPublica COMPAS (2 and AdultCensus (99) with gender as the sensitive attribute. The results are presented in Table 4 for SGD, Adam, and RMSProp. Following [47) we report the average gap in equal opportunity, gap in equalized odds, and gap in demographic parity (lower is better) over 10 runs. Regarding the fairness enhancing method, across all metrics on both datasets, Adam and RMSProp achieve substantially lower (better) fairness gaps than SGD. This experiment demonstrates that the faimess benefits of adaptive optimizers are not limited to standard training but are complementary to and can amplify the effectiveness of existing fairness interventions. Additionally, Tsole that in the absence of any faimess--nhancing method, adaptive optimizers yield better fairness outcomes than SGD. This finding further indicates that the fairness advantage of adaptive optimizers extends beyond computer vision applications.
Equallacd Odie Demegraphic
Abzolute Diffrrence Mhi
22%
dbt35ct
Figure 3: Difference of RMSProp and SGD's ' fairness in different malc-to-all ratios for CelebA dataset with gender as sensitive attribute.
5
Conclusion
This paper provided both theoretical and empirical evidence that the choice of optimizer can sig- nificantly affect the group fairness of resulting deep learning models. Through establishing an analytically tractable setup and analyzing it via stochastic differential equations, we showed that SGD and RMSProp can converge different minima with regards to fairness, with RMSProp more frequently reaching fairer optima. Next, we provided two theorems that showed I) adaptive gradient methods (c.g., RMSProp) shrink subgroup-speeific gradients more effectively, thereby reducing unfair updates compared to their SGD counterparts; 2) In a single optimization step, the worst case demographic disparity of RMSProp has the upper bound given by that of SGD. We validated these theoretical findings on three diverse datasets, namely CelebA, FairFace, and MS-COCO, and on multiple tasks. Across a range of group fairness criteria, i.e, equalized odds, equal opportunity, and demographic parity, adaptive optimizers consistently led to fairer solutions. We hope our work will encourage broader adoption of adaptive optimizers in fairness-critical domains, and we envision this research step toward developing more equitable deep learning models.
10

---

Acknowledgment
This work partially funded by the Natural Sciences and Engineering Research Council of Canada (NSERC), and was enabled in part by the support provided by the Digital Research Alliance of Canada.
References
[1] Jing An, Lexing Ying, and Yuhua Zhu. Why resampling outperforms reweighting for cor- recting sampling bias with stochastic gradients. In International Conference on Learning Representations, 2020.
[2] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias: There's 's software used across the country to predict future criminals. and its biased against blacks. ProPublica. 2016.
[3] Alessio Benavoli, Giorgio Corani, Francesca Mangili, Marco Zaffalon, and Fabrizio Ruggeri. A bayesian wilcoxon signed-rank test based the dirichlet process. In International Conference on Machine Learning, pages 1026-1034, 2014.
[4] Lukas Biewald. Experiment tracking with weights and biases, 2020. URL hattps:/wwkt wandb.com. Software available from wandb.com.
[5] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in com- mercial gender classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency, volume 81, pages 77-91, 2018.
[6] Junyi Chai and Xiaoqian Wang. Faimess with adaptive weights. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pages 2853-2866, 2022.
[7] Eunice Chan, Zhining Liu, Ruizhong Qiu, Yuheng Zhang, Ross Maciejewski, and Hanghang Tong. Group fairness via group consensus. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, page 1788-1808, 2024.
[8] Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos 7afeiriou. Retinaface: Single-shot multi-level face localisation in the wild. In Proceedings of the JEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 5203-5212, 2020.
[9] Zhun Deng, Jiayao Zhang, Linjun Zhang, Ting Ye, Yates Coley, Weijie Su, and James Zou. Fifa: Making fairness more generalizable in classifiers trained on imbalanced data. In International Conference Learning Representations, 2023.
[10] Samuel Dooley, Rhea Sukthanker, John Dickerson, Colin White, Frank Hutter, and Micah Goldblum. Rethinking bias mitigation: Fairer architectures make for fairer face recognition. In Advances in Neural Information Processing Systems, volume 36, pages 74366-74393, 2023,
[11) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[12] Jeevith Hegde and Børge Rokseth. Applications of machine learning methods for engineering risk assessment review. Safety Science, 122:104492, 2020.
[13] Mohammad Mehdi Hosseini, Ali Pourramezan Fard, and Mohammad H Mahoor. Faces of fairness: Examining bias in facial expression recognition datasets and models. arXiv preprint arXiv:2502.1 1049, 2025.
[14] Arthur Juliani and Jordan T. Ash. A study of plasticity loss in on-policy deep reinforcement learning. Advances in Neural Information Processing Systems, volume 37, pages 113884- 1 113910, 2024.
[15] Dayal Singh Kalra and Maissam Barkeshli. Why warmup the learning rate? underlying mechanisms and improvements. Advances in Neural Information Processing Systems, volume 37, pages 111760-111801, 2024.

---

[16] Kimmo Karkkainen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1548-1558, 2021.
[17] Aislinn -elty-Lyth. Challenging biased hiring algorithms. Oxford Journal of Legal Studies, 41: 899-928, 2021.
[18] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[19] Ron Kohavi et al. Scaling up the accuracy of baive-bayer classifiers: decision-tree hybrid. In Kdd, 1996.
[20] Mojtaba Kolahdouzi and Ali Etemad. Toward fair facial expression recognition with improved distribution alignment. In Proceedings of the 25th International Conference Multimodal Interaction, page 574-583, 2023.
[21] Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Ermon. Fp-diffusion: Improving score-based diffusion models by enforcing the underlying score fokker-planck equation. In International Conference on Machine Learning, pages 18365-18398, 2023.
[22] Elle Lett and William La Cava, Translating intersectionality to fair machine learning in health sciences. Nature Machine Intelligence, 5:476-479, 2023.
[23] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hy- perband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18:1 2018.
[24] Peizhao Li and Hongfu Liu. Achieving fairness at utility cost via data reweighing with influence. International Conference on Machine Learning, pages 12930, 2022.
[25] Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pages 2101-2110 2017.
[26] Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora. On the validity of modeling sgd with stochastic differential equations (sdes). In Advances in Neural Information Processing Systems, volume 34, pages 12712-12722 2021.
[27] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference Computer Vision, pages 740-755, 2014.
[28] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 2980-2988, 2017.
[29] Lydia T Liu, Max Simchowitz, and Moritz Hardt. The implicit fairness criterion of unconstrained learning. In International Conference on Machine Learning, 2019.
[30] Yanli Liu, Yuan Gao, and Wotao Yin. An improved analysis of stochastic gradient descent with momentum. Advances in Neural Information Processing Systems, 2020.
[31] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision, 2015.
[32] Ulya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019.
[33] Liangchen I Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. arXiv preprint arXiv: 09840,00993..... 2019.
12

---

[34] Avery Ma. Yangchen Pan, and Amir-massout Farahmand. Understanding the robustness difference between stochastic gradient descent and adaptive gradient methods. Transactions on Machine Learning Research, 2023.
[35] Chao Ma, Lei Wu, and E Weinan. A qualitative study of the dynamic behavior for adaptive gradient algorithms. In Mathematical and Scientific Machine Learning, pages 671-692, 2022.
[36] Xinsong Ma, Zekai Wang, and Weiwei Liu. On the tradeoff between robustness and faimesss In Advances in Neural Information Processing Systems, volume 35, pages 26230-26241, 2022.
[37] Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the sdes and scaling rules for adaptive gradient algorithms. In Advances in Neural Information Processing Systems, volume 35, pages 7697-7711, 2022.
[38] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys, 54, 2021.
[39] Oluseun Olulana, Kathleen Cachel, Fabricio Murai, and Elke Rundensteiner. Hidden or inferred: Fair learning-to-rant with unknown demographies. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 7:1088-1099, 2024.
[40] Rohan Paleja, Muyleng Ghuy, Nadun Ranawaka Arachchige, Reed Jensen, and Matthew Gombolay. The utility of explainable ai in ad hoc human-machine teaming. Advances in Neural Information Processing Systems, 34:610-623. 2021.
[41] Yan Pan and Yuanzhi Li. Toward understanding why adam converges faster than sgd for transformers. In OPT 2022: Optimization for Machine Learning Workshop, Advances in Neural Information Processing Systems, 2022.
[42] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
[43] Dana Pessach and Erez Shmueli. A review on fairesss in machine learning. ACM Computing Surveys, 55, 2022.
[44] Felix Petersen, Debarghya Mukherjee, Yuekai Sun, and Mikhail Yurochkin. Post-proce fost-processing for individual fairness. In Advances in Neural Information Processing Systems, volume 34, pages 25944-25955, 2021.
[45] Esther Puyol-Antón, Bram Ruijsink, Stefan K. Piechnik, Stefan Neubauer, Steffen E. Petersen, Reza Razavi, and Andrew P. King. Fairness in cardiac mr image analysis: An investigation of bias due to data imbalance in deep learning based segmentation. In Medical Image Computing and Computer Assisted Intervention, pages 413 413-423, 2021.
[46] Samira Abhasgholizadeh Rahimi, Mojtaba Kolahdoozi, Arka Mitra, Jose L Salmeron, Amir Mo- hammad Navali, Alireza Sadeghpour, and Seyed Amir Mir Mohammadi. Quantum-inspired interpretable ai-empowered decision support system for detection of early-stage rheumatoid arthritis in primary care using scarce dataset. Mathematics, 10:496, 2022.
[47] Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh. Fairbatch: Batch selection for model faimess. arXiv preprint 20X1:20012.01960 2020.
[48] Shubham Sharma, Yunfeng Zhang, Jesús M. Ríos Aliaga, Djallel Bouneffouf, Vinod Muthusamy, and Kush R. Varshney. Data augmentation for discrimination prevention and bias disambiguation. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, page 358 364, 2020.
[49] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1rrXi::149.111 1409. 2014.
[50] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, volume 25, 2012.
13

---

[51] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[52] Mathieu Tanneau and Pascal Van Hentenryck. Dual lagrangian learning for conic optimization. In Advances in Neural Information Processing Systems, volume 37, pages 55538 55538-55561, 2024.
[53] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59:64-73, 2016.
[54] Kan Wu, Jinnian Zhang, Houwen Peng, Mengehen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European Conference on Computer Vision, pages 68 08-85,22 2022.
[55] Tian Xu, Jennifer White, Sinan Kalkan, and Hatice Gunes. Investigating bias and fairness in facial expression recognition. In European Conference on Computer Vision Workshops, pages 506-523, 2020.
[56] Wei Yao, Zhanke Zhou, Zhicong Li, Bo Han, and Yong Liu. Understanding fairness surrogate functions in algorithmic fairness. arXiv preprint arXiv:2310.12211, 2023.
[57] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2979-2989, 2017.
[58] Yingxue Zhou, Belhal Karimi, Jinxing Yu, Zhiqiang Xu, and Ping Li. Towards better general- ization of adaptive gradient methods. In Advances in Neural Information Processing Systems, volume 33, pages 810-821, 2020.
14

---

Appendix
A
Itô's Lemma
Itö's lemma extends the chain rule of classical calculus to stochastic calculus, thereby accommodating functions that depend on Brownian motion (t). Suppose that the following stochastic differential equation (SDE) is given:
(t) (t. X(()) + (t)
(29)
In the equation above, X(O) is a stochastic process, MO and NO are the functions of (t) and t. Let's define stochastic process (t) by:
((t) (t)),
(30)
where, Q( is function of stochastic process (t) and Itô's lemma states that we can compute dY (L) by using the following:
d x11 dx
d
Q(t,)) M(t, + 0.5 & de
(t)
dt+ ((,,))
(31)
d
(tt,x) N(t, dB(t).
dx
In the above equation, as the term 0.5 d (t,, does not appear in classic calculus, it often referred to as Itô's correction term.
da
B Fokker-Planck equation
In this section, for completeness, we provide the derivation of fokker-planck equation from given SDE. This is well-known equation in the analysis SDEs. Suppose the following SDE is given:
dx((1) (t,, X()))dt + (t,,, (())dB((t..
(32)
Also, suppose that ((z) is function which is outside of a bounded interval. For simplicity, X instead X (t). By Itô's lemma, we have:
dQ(x) (M, X)Q'(X)))) + (,, (x))(x) dt + (tt, (X)B(((()
(33)
Let's integrate the equation above from t and then take the expected value from both sides:
[IQ(X) Q(0) = J."
0.55
(34)
J B[N(T, x)Q((x))
Suppose X is independent of B, then as B[)B(r)) = 0, the second integral above is zero. Next, let's take the derivative with regards from both sides:
dt d [[Q( X)Q'(x) (x) O.5
(t,
(35)
To make the distribution of appear in the equations, let's use the definition R[.) the equation above:
f t. ((()))((x)=
+ L p(t, x (t, (36)
p(t. (tt,
In the equation above, p(t, 2) is the distribution of stochastic process X. By applying integration by part the integrals in the Eq. 36 and re-arranging the terms, we get:
[ @2
a
Or (p(t, Mt, 0.5 (p(t. x) (t, Q(x)dr= =
(37)
d.
As the equation above holds for all Q(z), then the other factor must equal to zero. Thus, we have:
i
0²
/
p(t,2x) dr (p(t, (p(t, 0.5
2.d ( =
(38)
The equation above is known as Mokker-Planch equation.
15

---

C
Well-behaved NGOS
Noisy Gradient Oracle with Scale Parameter (NGOS) models stochastic gradients as g(w) (Z(w)+ 2 where θ is a noise scale parameter, and is random variable that follows a distribution with covariance matrix [(w) and mean of zero. An NGOS is well-behaved if the conditions below are satisfied [37]:
V (w) is Lipschitz.
Square root of covariance matrix is Lipschitz and bounded.
3. Both VL(w) and (ww) are differentiable and their partial derivatives up to 4-th order have polynomial growth.
Noise distribution the NGOS must have low skewness. other words, there must exist a function of polynomial growth such that [[[==]] is equal or less than that function divided by O.
Noise distribution must have bounded moments. In other words, there must exist a constant x such that |||||||2*] 2 x(1 [|w]])).
Gaussian noise a well-hehaved NGOS. However, the noise in NGOS not restricted to Gaussian. Particularly, if the noise distribution is not heavy-tailed, then it satisfies requirements 4 and 5. Whether or not the noise distribution in real-world applications heavy-tailed is still an open problem. Nevertheless, several experimental results denote that the conditions described above not too strong 37
D Approximation Quality of SDE for SGD
The dynamics of SGD can be approximated by the following SDE 25
dW. VL(W))d (TI(W:))\>dB(()
(39)
where We represents an approximation of at discrete time steps k and E is the covariance matrix of the underlying NGOS. The noises that make the individual paths for the above SDE and the SGD) are independent processes. Hence, SGD and its SDE approximation do not share the same sample paths; rather, they are close to each other in "distribution" or "weak sense" 255 To formalize this mathematically. suppose E\t be the set of functions, f (ar), of polynomial growth, i.e. Ea, b E R such that [f(] < a(1 + |=]²). Then, SDE defined in Eq. 39 is the order weak approximation such that for all k I...N, .. [E[((W,)) ).,]] - (ww)) < Cr²". This is a well-known result in analyzing optimization algorithms using SDEs, first time proposed in 25 Similar theorems for RMSProp and Adam are available in 137
of SGD if for each function f(x) in E, there exists constant
E Proof of Lemma 1
The demographic parity criterion for fairness between two subgroups is defined as:
P = 0) = P(j = 1),
(40)
where z denotes the subgroup, and ý represents the output of model. The gap in demographic parity serves a fairness measure [38]. A common approach to quantifying this gap is through the zero-one loss difference 244: [ (w) z = 0j ((o)(()) = 1]-. As established in prior work 24], the zero-one loss can be approximated by the training loss, leading to the following fairesss measure F(w) = [[[w]]2 = 0] - [[(w]] = 1].. Substituting the given loss functions Lo(w) and L (w). obtain F(w) w. Since F(w) achieves its minimum at 0, this proves that 0 is the fairest minimizer.
F Additional Results for Theorem
In this section, we extend our empirical analysis of Theorem by evaluating our findings with broader choices of learning rates and alternative definitions of the fair neighborhood.
16

---

Table Hyperparameter configurations for the experimental scenarios.
Configuration
PPoP
MSGD
Faimess Threshold
0.01
0.1
0.2
2
0.1
0.2
0.2
3
0.01
0.1
0.1
4
0.01
0.1
0,4
5
0.1
0.2
0.1
6
0.1
0.2
0.4
Po 0.1
Po 0.3
100
100
Proportion near Wooe
SGD
SGD
80
80
RMSProp
RMSProp
60
60
40
40
20
20
0
50
100
0
50 Epoch
100
Epoch
Figure 4: Convergence rates of RMSProp (1) 0.01) and SGD (1) = 0.1) to the fair neighbourhood, defined by threshold of 0.2.
Theorem establishes that for subgroup sampling probabilities po. Pr E (0, 1) with pa + P1 1, and loss functions (ow)) = =" 1)² and Li(w) (w) { - 1)², the optimization of the empirical loss
function:
1 Σ (w),
Cempl (w) =
(41)
N
rESt
where each sample E 0, is drawn i.i.d. with probability Pa for subgroup 0 or p1 for subgroup leads an increased likelihood of RMSProp converging the fair optimum Woop 0 as the imbalance between po and pı grows. To empirically validate these theoretical insights, we conducted simulations using fixed learning rate of f 0.1, training for 1000 independent runs per experiment. The fraction of runs converging to the neighborhood of the fair optimum, defined as | w < 0.2, was recorded each epoch. We call 0.2 the fairness threshold. To assess the robustness of our findings, we extend our analysis by examining a broader range of values and explore more restrictive and relaxed fairness thresholds. experiments. Fig.s to 9 illustrate the convergence behavior of RMSProp and SGD under the specified configurations in the table.
the hyperparameter configurations used in our
The figures reveal a consistent trend in the convergence behavior of RMSProp and SGD. In the severely biased setting (Po 0.1), RMSProp exhibits a higher likelihood of converging to the fair neighborhood, regardless of the learning rate. As the bias decreases (po = 0.3), the behavior of RMSProp and SGD becomes more similar. These experimental findings in line with theoretical findings in Theorem I Additionally, a more restrictive faimess threshold reduces the probability of convergence to the fair neighborhood. whereas a relaxed threshold increases This expected, as stricter definitions impose tighter constraints on acceptable fair solutions, while relaxed criteria naturally allow higher fraction of runs to qualify as fair.
G Extension of Theorem 2 to Anisotropic Noise
Theorem 4. Consider population that consists of two subgroups with subgroup-specifte loss functions Lo(w) and Li (w), sampled with probabilities pa and pv. respectively. Suppose a stochastic (online) training regime, in which each parameter update is computed from a sample drawn from one of the two subgroups. Suppose the grudients V Lo(ww) and V. (wn) are well-behaved anisotropic NGOs. namely Nuo, Eo) and N Ex), respectively. Then, the difference in parameter updates
17

---

Po
Po 0.3
100
100
Proportion near Waap
SGD
SGD
80
RMSProp
RMSProp
60
60
40
40
20
20
50
100
0
50
100
Epoch
Epoch
Figure 5: Convergence rates of RMSPPop (1 0.1) and SGD (17 0.2) to the fair neighbourhood,
defined by threshold of 0.2.
100
Po
0.1
100
Po
0.3
Proportion near Wjap
SGD
SGD
80
80
RMSProp
RMSProp
60
60
40
40
20
20
0
50
100
0
50 Epoch
100
Epoch
Figure 6: Convergence rates of RMSProp (1) = 0.01) and SGD (1) = 0.1) to the fair neighbourhood, defined by threshold of 0.1.
between subgroups 0 and under RMSProp has an upper bound given by the corresponding difference under SGD. Consequently. RMSProp offers fairer updates across subgroups.
Proof. The proof is similar to Theorem For SGD, the difference between the parameter updates across subgroups is VL,- VL1ll. Regarding RMSProp. let's compute the expected value and the variance of Un:
Bluk] uo + (1 - 111>*>pp (nă diag(Eo)) P1 (Rf + diag(Ex))).
(42)
Where diag( Eo) and (iag(Ei) are the vectors with the diagonal elements of Do and Σ,, respectively, Similar the the isotropic case, we can show that the variance of uk negligible and thus 1 - ETvx). Hence, the difference between parameter updates are (VD(VCo VLi)ll, where D is diagonal matrix, whose jjth element can computed as:
(43)
po ( + 0b,, Pı (ni, + +oî +
Prior work (37] has experimentally shown that o² > μ²a Additionally, in stochastic (online) training setup: @ 1. Thus, Djj < 1. This completes the proof.
H Proof of Theorem
Proof. As described in the proof of Lemma the following can be used to approximate the gap in [[(w)] 2 1]]. Let's define the function (wwk)
demographic parity: F(w) = [[[[w]]
(o(w)) L. (wx). Also, define ((wx)) = Ln(ws) (wx) Let's expand F(wx) around wn using Taylor series:
Flw 5) F(w)) VF(w))''.
(44)
We can compute VF(ww)) as:
VF(wx) (
(45)
18

---

100
Po
100
Po 0.3
Proportion near Wagp
SGD
SGD
80
RMSProp
RMSProp
60
60
40
40
20
20
50
100
50 Epoch
100
Epoch
Figure 7: Convergence rates of RMSProp (17 0.01) and SGD
0.1) the fair neighbourhood,
defined by threshold of 0.4.
100
Pg
0.1
100
Po 0.3
Proportion near Waa
SGD
SGD
80
80
RMSProp
RMSProp
60
60
40
40
20
20
50
100
0
50 Epoch
100
Epoch
Figure 8: Convergence rates of RMSProp (1 0.1) and SGD (17 0.2) to the fair neighbourhood, defined by threshold of 0.1.
If in Eq. 44
we set 8 V L(wx), given the updates of SGD in Eq. 1
we can re-write Eq. 44
F(wk+11.0)) F(wx) - - up VV
(46)
where F (ggga is the value of function F() after one iteration using SGD. Using the proof of Theorem we can perform similar operation for RMSProp:
F(uk+Lrm.) F(wx) (m)((k)) VW
(47)
In the equation above, F(WA-1,m. is the value F() after one iteration using RMSProp. and matrix D is diagonal matrix defined the proof of Theorem 2 As po fr, then the worst case increase in the gap of demographic parity for SGD occurs when V (ww) aligns with VL. Mathematically, the increase can be modeled as:
[F(W++1.ga) F(wx)) (/v(wx) (V < V (wx) | ||
(48)
Similarity, for RMSProp, have:
F(wx+1.m.) F(wx)) (V( DVL < (V)DVV()>)]] |V
(49)
1/ (wx)) - |VCC]]
In the equation above, as diagonal elements of D are strictly less than 1, we have strict incquality. Comparing Eqs. 48 and 49 we conclude the proof.
I Extension of Theorem
3 to Anisotropic Noise
Theorem 5. Consider population that consists of two subgroups with subgroup-specipy loss functions Co(w) and Li (w), sampled with probabilities pu and pr. respectively. Suppose stochastic (online) training regime in which each parameter update is computed from sample drawn from one of the two subgroups. Suppose that the gradients Lo( Lo(w)) and L, (wx) are well-behaved anisotropic NGOs, namely (po. Eo) and N(1,; En), respectively, with pa pn- Then, in expectation, the worst- case increase the demographic parity gap after one iteration of RMSProp has an upper-bound no greater than the corresponding increase under SGD.
IndN(ua
19

---

Po
Po 0.3
100
100
Proportion near Wagg
SGD
SGD
80
RMSProp
RMSProp
60
60
40
40
20
20
50
100
50
100
Epoch
Epoch
Figure 9: Convergence rates of RMSProp (n 0.1) and SGD (17 0.2) to the fair neighbourhood, defined by threshold of 0.4.
Proof. The proof follows directly as the proof of this Theorem relies on the conclusion of the extension of the Theorem 2 which is already proved in Theorem
Datasets
CelebA [31] is a large-scale and real-world dataset, which include 202,599 facial images from over 10,000 individuals. Each image is annotated with 40 distinct attributes, and the dataset is officially partitioned into training, validation, and test sets. In this study, we use gender (male vs. female) and age (young old) sensitive attributes, while performing facial expression recognition based on the "happy" attribute. Notably, the dataset exhibits an imbalance in demographic distribution, with the probability of an image depicting male being 0.42 and the probability of depicting young individual being 0.77. This skewed distribution indicates the presence of bias in the training set.
FairFace [16 is large-scale face attribute dataset, mainly designed to mitigate racial bias in facial analysis tasks. It includes 108,501 facial images sourced primarily from the VFCC-100M Flickr dataset [53]. Each image is annotated for gender, age (9 sub- groups). and race (7 sub- groups).. In this study, we leverage FairFace for gender classification, treating age and race as sensitive attributes. Given its broad demographic representation, FairFace serves robust benchmark in fairness related tasks.
MS-COCO [2 is large-scale, real-worke benchmark dataset widely used for multi- classifi- cation, object detection, and image captioning. consists of over 330,000 images, including more than 200,000 labeled images spanning 80 object categories. In this study, we leverage MS-COCO for multi-label classification. Notably, dataset does not provide explicit gender annotations. To infer gender labels, we follow a protocol established in previous works [57]. utilizing the five associated captions per image. Specifically, we retain images where at least one caption contains either the term "man" or "woman". Additionally, objects are defined as those appearing at least 100 times alongside
remove object categories that are not strongly associated
with humans. Human-assonn either "man" "woman" in the captions, resulting in reduced of 75 object categories. Fig. TO provides sample captions and their corresponding inferred gender labels. is worth mentioning that the probability of an image depicting male is 0.71.
K
Training Protocol
For facial expression recognition, we employ RetinaFace g to detect and crop faces from images. The images in the FairFace dataset are already cropped by default. As part of the preprocessing pipeline for all tasks, we normalize the images and resize them to 224 x 224. Data augmentation consists of random horizontal flipping and random rotation. Facial expression recognition and gender classification tasks are trained using the cross-entropy loss Loe, defined as:
1
Lee
My log(ū,) + (1 3/)) log(1 (gg(1-1 -5)]
(50)
20

---

(a) woman prepares several sub sandwiches a deli counter.
(b) 1. man in yellow shirt holding white dog and making a face
a person behind a display glass preparing food
2. a man holding hot dog bun beside his puppy.
lady behind sneeze guard making sub sandwiches.
a man putting hot dog on a puppy and pretending to eat
4. a woman behind a deli counter making sub sandwiches.
4. a man holding a white puppy and a red leash.
5. there is woman that is making sandwiches a deli
5. man who pretending bite puppy.
Figure 10: Two example images from MS-COCO dataset along with their captions. The inferred gender labels from the captions for the left and right images are "woman" and "man". respectively.
For the multi-label classification task, given the class imbalance in MS-COCO, we employ the focal loss 28 mitigate overfitting majority classes. It is formulated as:
Gsocal
E (1 0,)" log(ú,) + (1 E) y y5) log(1 ú,)1. N
1
(51)
where & is the balancing factor and is the focusing parameter. To optimize the learning rate during training, we leverage Bayesian optimization (50) combined with the Hyperband algorithm [23]. The best-performing models are trained twice, and report the average performance across these runs.
We experiment with multiple backbone architectures, including ResNet-18 18 and ResNet-50 [I], VGG-16 [49], and recently proposed vision transformer (tiny (54) These architectures have been widely used in studies analyzing the impact of optimization methods on different learning paradigms 34 as well as in fairness-focused works (55). ResNet models leverage residual connec- tions facilitate deep representation learning, while VGG-16 provides structured CNN design with uniform filter sizes. The tiny ViT, transformer-based model, processes images as patch embeddings, and showed state-of-the-an art performance (54). This diverse selection allows us to investigate fairness effects across both convolutional and transformer-baat architectures.
L Hyperparameters
We employ Bayesian Optimization [50] with Hyperband 23 algorithm to systematically tune the learning rate across different optimizers. Hyperband efficiently allocates training resources to more promising setups and progressively eliminate underperforming ones. evaluates training at specific checkpoints, called "brackets". We set Hyperband brackets to [10, 20, 40), allowing enough time for each run to demonstrate its potential. Additionally, the learning rate sampled using a log-uniform distribution within the following search spaces:
SGD: 11003110-1
RMSProp: [10-5 10
Adam: [10-5 10-2
All models are trained for maximum 60 epochs with batch size of and weight decay 10-4 We use Pytorch default values for optimizer specific hyperparameters:
21

---

100
75
Foor
90-
Fon
Faoo
40
50
RMSProp Adam
80
RMSProp
RMSProp
Adam
Adam
ColebA-G
Caleba-d
PatrFaco-R
RatFace-A
MSCOCO-G
CalebA-G
Calebd-A
FutrFace
FatFFace-A
MSCOCO-G
CelebA
CalebA-A
PairFace-R
FarrFice-A
MSCOCO-6
Figure 11: Fairness metrics for ResNet- across different datasets and sensitive attributes.
RMSProp: 0.9
Adam: 31 = 0.9, .32 0.999
Additionally, we use "Reduce-on-Platean" scheduler of Pytorch to decrease the learning rate during the training. if improvement is observed for epochs, then the learning rate multiplied by 0.5.
M Additional Fairness Results across Different Backbones and Datasets
In this section, we present fairness metries for SGD, RMSProp. and Adam across ResNet-18, ResNet- 50, and VGG-16 16 backbones, evaluated multiple datasets with different sensitive attributes.
ResNet-18: Fig. [ illustrates fairness metrics for the ResNet-18 backbone. Considering FEOD. both Adam and RMSProp consistently outperform SGD across all datasets and sensitive attributes. The only exception the MS-COCO dataset, where both RMSProp and SGD yield an FFOD of zero. This is due to MS-COCO's 75 object classes, some of which are highly underrepresented, leading an effective fairness score of zero. The largest difference in FEOD observed on the CelebA dataset with age as the sensitive attribute, where RMSProp significantly surpasses SGD. Adam and RMSProp exhibit comparable performance, as expected.
For the FEOP metric, both RMSProp and Adam outperform SGD across datasets. For example, in the FairFace dataset with age as the sensitive attribute, Adam achieves FEOP of 80%, compared to 72% with SGD. Regarding FDPA. Adam surpasses SGD in four out of five cases. The only exception CelebA with age as the sensitive attribute, where both optimizers yield similar FPPA. This aligns with Theorem [ as CelebA not sufficiently biased for adaptive gradient methods to demonstrate a significant fairness advantage.
ResNet-50:: Fig. 12 presents results for the ResNet-50 backbone. For FEOD, Adam consistently outperforms SGD, while RMSProp surpasses SGD in four out of five cases, with MS-COCO again being the exception. The difference between Adam and SGD reaches approximately 15% in CelebA when using age the sensitive attribute. Similar trends emerge for FEOP, where both Adam and RMSProp outperform SGD across datasets. Finally, for FDPA- more biased datasets, such as FairFace with age the sensitive attribute, show a higher gap between adaptive gradient methods and SGD.
VGG-16: Fig. 13 shows fairness metrics for the VGG-16 backbone. As with the previous archi- tectures, Adam outperforms SGD cases for FEOD. For FEOP, both Adam and RMSProp demonstrate superior fairness compared to SGD. In FairFace with age as the sensitive attribute, the difference between Adam and SGD reaches approximately 10%. For EDPA. the most significant difference is again observed in FairFace with age the sensitive attribute, which reinforces our theoretical findings.
Across all three faimess metrics, we observe consistent trends across multiple architectures, datasets, and problems, including gender classification, facial expression recognition, and multi-label clas- sification. These extensive experiments provide strong empirical evidence that adaptive gradient methods, such as Adam and RMSProp, converge to fairer minima with higher probability compared to SGD.
22

---

100-
100
75-
Faoo (h))
40
Foor
90
Faue
RMSProp
80
RMSProp
25-
RMSProp Adam
Adam
Adam
CelobA-G
Caleba-d
Patrfaco-R
Fatraace-A
MSCOCO-G
CalebA-G
Calabd-A
FairFace
PaiFaace4
MSCOC0-G
CalabA-G
Caleba-A
FairFace-R
FatrFace-
MSCOCO-6
Figure 12: Fairness metrics for ResNet-50 across different datasets and sensitive attributes.
100-
60
Fooe!%))
Feor(%)
90-
Fooe
SGD
80-
SGD
SGD
20
RMSProp
RMSProp
RMSProp
Adam
70-
CelebA.G
Calebied
RairaaceR
FaisFace-A
MSCOCO-0
Celeb4
RaisFace-R
FaieFaced
MSCOCO.G
Gelebt.G
FairPace CelebA
FarFare-A
R
MSC000-0
Figure 13: Fairness metries for VGG-16 across different datasets and sensitive attributes.
N Additional Fairness Results across Different Variants of Optimizers
Our theoretical analysis identifies gradient normalization via second moment the key mechanism for improving group fairness. This provides a clear framework for predicting the behavior of other optimizers with regards fairness. To validate these predictions, we have conducted new set of experiments with SGD w/ momentum [30, AdamW [32], and AdaBound [33 variants. Following the protocol from our main experiments, we use the CelebA dataset with a ViT backbone, treating
gender and age as sensitive attributes. We tuned the learning rate for each optimizer using Bayesian optimization with Hyperband and report the average fairness scores Table 6 As shown in table, AdamW consistently achieves the best fairness scores. This is expected, AdamW retains the normalization mechanism of Adam and RMSProp via the second moment. contrast, SGD w/momentum, lacking the normalization via the second moment, performs more biased. The performance of AdaBound is comparable to SGD w/momentum. The reason is that the AdaBound algorithm deliberately clips the second moments make it act like SGD, which in turn limits the fairness advantage.
Additional Classification Results across Different Backbones and Datasets
Table reports the accuracy and Fl scores of SGD, RMSProp, and Adam optimizers across ResNet- 18, ResNet-50, and ViT backbones. We evaluate these optimizers on CelebA (facial expression recognition), MS-COCO (multi-label classification), and FairFace (gender classification) datasets. For CelebA, the performance of SGD, RMSProp, and Adam is comparable, with only minor variations. For instance, using the VGG-16 16 backbone, SGD achieves an accuracy of 92. 22.22%. while RMSProp and Adam yield 92.37% and 91.42%. respectively. Similarly, for MS-COCO and FairFace, the performance differences across the optimizers remain limited. The largest accuracy gap between SGD
Table 6: Fairness comparison of optimizer variants.
Gender
Age
Adam W
AdaBound
SGD w/ mom.
Adam
AdaBound
SGD w/ mom.
FROD
64.90
61.10
61.12 95.07
71.11
67.20
67.25
FEOP
99,90
96.10
98.95
96.60
96.55
FDPA
73.11
70.01
69.50
98.11
95.39
95.39
23

---

Table 7: Performance of different backbones across datasets. Each cell reports three numbers corresponding to SGD, RMSProp, and Adam, respectively.
Rac hacksoness
Datuset
R8sNer18
ResNet50
VGG16
Acc.
Acc.
F1
Acc.
CelchA
91,12, 92.21, 92.19
91.41, 92.76, 92.80
91,19, 92.60, 92.39
91.33, 92.09, 92.25
92.22, 92. 91.42
92.20, 92.28, 92.50 66.47, 72,04, 73.20 90.59, 91.66, 92.69
MS-COCO
88.31, 91.10, 90.42
66.96, 76.26, 69.37
89.15,
91.89 91.70
67.68, 70.48, 70.16
89.48, 90.11. 91.57
FaiFace
86.16, 90,36, 90.31
$7.15, 88.71. 90.30
88.48, 90,58, 92.41
89.67, 90.77, 91.57
91.70, 91.11 92.09
and adaptive optimizers occurs MS-COCO with the VGG- 16 backbone, reaching approximately 5.5%. However, the highest disparity in fairness metrics does not occur on MS-COCO but rather on FairFace dataset. These findings indicate that the differences between SGD and adaptive optimization algorithms cannot be solely attributed to their performance. Instead, as demonstrated in Theorems 3 the normalization behavior of these optimizers plays crucial role in shaping their fairness properties.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly articulate the main research question, and the contributions are explicitly listed in a numbered format.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Each theorem provided in the paper clearly states its assumptions.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
25

---

Each theorem is preceded by a clear statement of assumptions, followed by complete and rigorous proof.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all relevant information, including details on open-source datasets, hyperparameter values, and implementation. Additionally, the source code is made publicly available.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
26

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We publicly available datasets and have released the source code with sufficient instructions to reproduce results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc.
Soblic//uiddes/oddeubbbmsss
caapssionoolyyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccppplgguuuuieeeeae
Stteeee//ooSSSSeisSSioeaa www.ssssiicgcyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper includes relevant experimental details, including hyperparameter values, optimizer types, and other training configurations.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report statistical significance using Wilcoxon signed-rank tests, as shown
in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The implementation details section includes information on the compute resources used.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't fidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have thoroughly reviewed the NeurIPS Code of Ethics and confirm that our research adheres its principles.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper provides practical insights to practitioners into how optimizer choice affects group fairness. We discuss the implications in the paper.
Guidelines:
28

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not involve the release of high-risk models or data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets and external tools used in the paper publicly available and properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release the source code used in our experiments, accompanied by README file that includes the guidelines.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve any crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve research with human subjects and therefore does not require IRB approval
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
30

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used as part of the core methodology or any original component of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurl) hett::///earrrppccc//oofeeeeeeee /c///onfeeeeeees/2225///... for what should or should not be described.
31


Reviews:
[{"review_id": "lDKQRdkJNq", "reviewer": "Reviewer_htpx", "summary": "In machine learning, optimization algorithms adjust the parameters to minimize the loss function. What hasn’t been examined yet is what effect optimizers have on common group fairness criteria. This paper sets out to do so by first examining the role of optimizers in an analytically tractable setup. The authors focus on two categories of optimization algorithms, namely adaptive ones and stochastic ones. In their theoretical examination, they focus on RMSProp (an adaptive optimization algorithm) and SGD (which is stochastic). They use stochastic differential equation analysis (SDE) to find that RMSProp leads to fairer updates than SGD (specifically, the difference in parameter updates between the demographic groups is smaller). The authors then confirm this result experimentally using three datasets and three group fairness definitions (demographic parity, equality of opportunity and equalized odds). The better performance of adaptive methods on group fairness holds not only when the adaptive methods lead to better accuracy. Adaptive optimizers can thus be a simple and cost-effective way to improve group fairness criteria (before potentially using other methods to improve fairness further).", "questions": "- Why do you use demographic parity as a fairness metric for a gender classification system? In gender classification, we care about correct labels, not whether the likelihood of getting a certain label is the same for all demographic groups. If one group is underrepresented in the dataset, there’s no reason (at least from my point of view) to expect demographic parity (that’s different for cases where, in an ideal world, we would expect different groups to have the same distribution for the ground truth label).\n- Since simply changing the optimizer method does not give us any fairness guarantees, do you see this change in optimizer to be used with other methods (e.g., post-processing methods)?\n- Out of curiosity (so not saying it’s missing): Have you considered examining other group fairness criteria, such as sufficiency (predictive parity)? If so, why did you decide against it?\n- Fig. 1: Why is SGD performing so much better early on at about 20 epochs?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths labeled with [S], weaknesses with [W], general thoughts/comments with [T]\n\n# Major\n## Introduction\n- [S] Well-written introduction that gets the point across\n## Related work\n- [S] My expertise is in fairness and not in theoretical ML/optimizers, but I thought the fairness section of related work was well-written and the section on optimizers helped me remember some basic stuff\n- [T] l.74f.: The reason why you focus on group fairness is plausible, although I wonder to what extend individual fairness would even be implementable for the facial recognition dataset\n## Method\n- [S] I appreciate the section on preliminaries to create a shared understanding\n- [W] However, building some more intuition would have really helped me get the paper content faster. The explanations are still quite theoretical and based on formulas, which didn’t help me in getting a good intuition for what’s happening. For something like the NGOs, for example, explaining in basic terms what it does would have helped me a lot rather than jumping straight to the formula.\n- [S] I appreciate that theoretical results were followed by empirical tests as sanity checks and a more robust verification\n\n## Experiments\n- [W] What does fairness in the context of gender classification mean? Is a fair classification simply a correct one? If so, why are you using demographic parity, equalized odds and equal opportunity over just accuracy? In other words: The experimental results from these datasets are interesting if you see the dataset as just a stand-in for a dataset with sensitive attributes, but using these fairness metrics for these datasets implicitly implies that it makes sense to measure fairness this way for real-world gender prediction – which I think is not the case, so this should be contextualized.\n\n# Minor\n## Related work\n- l.83: typo in “learning algorithm”\n- l.88f.: “One major limitation is their computational overhead or interference 89 with the training phase.” → Post-processing, when implemented as a simple threshold rule, for example, can be fairly cheap and easy to implement. However, your point still stands about the ease of changing the optimizer.\n- l.123: add “(ODE)” abbreviation after “ordinary differential equations” as you use the abbreviation in line 126 without introducing it\n## Experiments\n- Fig. 2: Please don’t use a line chart here as it implies a continuous or ordered relationship between data points when the CelebA-G, CelebA-A, etc. datapoints don’t have a continuous relationship.\n- l.307: The fairness metrics (F_{EOD} and the others) are only defined in the appendix. I don’t think that’s ideal since it’s an important part of the evaluation, but I can live with it. However, it seems important to at least clarify the naming convention (F_{EOD} = equalized odds, F_{EOP} = equality of opportunity and F_{DPA} = demographic parity) in the main text once.\n- As you don’t explain how F_{EOD} and the other fairness criteria are calculated in the main text, it would be helpful to at least mention whether you want these values to be high or low. This helps the reader interpret Fig. 2 more quickly.\n## Conclusion\n- l. 389: typo in “demographic disparity”", "was_revised": true, "final_justification": "The rebuttal answered my open questions and the paper makes an interesting and novel contribution, so I'll maintain my positive score.", "rebuttal": "We thank the reviewer for their valuable comments. Below we provide a careful point-by-point response to each comment. We would be happy to provide additional discussions/information in the author-reviewer discussion period.\n\n> **Individual fairness feasibility in facial recognition**\n\nIndividual fairness implies that the model should treat similar individuals, similarly. Given that each individual is different in FR datasets, this would not be very meaningful. However, one could define some metric or parameter, according to which different individuals are similar. This metric could then be utilized to ensure fairness for two individuals who are close in terms of that metric. Having said this, this approach is not quite common in FR tasks.\n\n> **Intuition about noisy gradient oracle with scale parameter (NGOS)**\n \nNGOS is simply a formal way to model the gradients used in mini-batch training. Because gradients are computed on a small batch of data rather than the entire dataset, the gradient at each step is not the ``true'' gradient, rather it is the true gradient plus some random noise from the sampling process. NGOS captures this by representing the stochastic gradient as the sum of the true gradient and a noise term. We will add this description to the text of the paper.\n\n> **Why demographic parity for gender classification?**\n\nPlease kindly note that in the gender classification experiment, while the prediction target (output class) is gender, the *sensitive attributes* we evaluate for fairness are race and age, not gender itself (please see l. 665). Thus, demographic parity in our work measures whether the model predicts a specific gender with the same probability across different racial or age groups. As an example, with regards to demographic parity, we are interested in the fact that the probability of being \"female\" is the same across \"young\" and \"old\" subgroups.\n\n> **Typos**\n\nWe thank the reviewer for catching these typos. We will correct all the noted typos and perform a thorough proofreading to prevent recurrences in the final version.\n\n> **Clarifying ``computational overhead'' of fairness methods vs. optimizer switch**\n\nWe agree that some fairness interventions, such as simple post-processing rules, can be computationally inexpensive. However, as the reviewer correctly noted, our intention with the statement on l.88 was to highlight a contrast. Specifically, we aimed to compare the introduction of a separate fairness-enhancing module with the inherent simplicity of selecting a different optimizer. This distinction is important for practitioners who are looking for a straightforward way to improve fairness in their pipeline. We will revise l.88 to reflect this.\n\n> **Usage of line charts to represent the results**\n\nWe will replace the line chart in Figure 2 with a grouped bar chart to more accurately represent the data. We will also apply this correction to Figures 11, 12, and 13 in the appendix to ensure consistency.\n\n> **Placement and clarification of fairness metrics**\n\nAs the reviewer correctly points out, the definitions of fairness metrics are currently in Appendix G, and the clarification that higher values are better is in l.629. We agree that moving them to the main text would help the readers to more easily understand the figures. We will do so in the final version, using the extra page allowed for the final version.\n\n> **Change in optimizer to be used with other fairness-enhancing methods**\n\nOur Theorems 2 and 3 are general with respect to the loss function, which suggests their fairness benefits should be complementary to existing fairness-enhancing methods that add regularization terms to the primary loss. To validate this hypothesis, and inspired by feedback from this reviewer and reviewer ``fC4c'', we have now conducted a new experiment where we test the impact of optimizers in addition to the fairness-enhancing method [1] on two new datasets (ProPublica COMPAS [2] and AdultCensus [3]) with gender as the sensitive attribute. The results are presented in the table below for SGD, Adam, and RMSProp. Following [1], we report the average gap in equal opportunity, equalized odds, and demographic parity below (lower is better)vover 10 runs.\n\n\n|             |        | **Equal Opportunity** |        |   |        | **Equalized Odds** |        |   |        | **Demographic Parity** |        |\n|:------------|:------:|:---------------------:|:------:|:-:|:------:|:------------------:|:------:|:-:|:------:|:----------------------:|:------:|\n| **Dataset** | Adam   | RMSProp               | SGD    | │ | Adam   | RMSProp            | SGD    | │ | Adam   | RMSProp                | SGD    |\n| &#8203;     | &nbsp; | &nbsp;                | &nbsp; | &#8203; | &nbsp; | **with fairness enhancing** | &nbsp; | &#8203; | &nbsp; | &nbsp; | &nbsp; |\n| ProPublica COMPAS  | 0.45 | 0.48 | 0.71 | │ | 2.79 | 2.78 | 2.90 | │ | 0.86 | 0.86 | 2.60 |\n| AdultCensus       | 3.15 | 3.16 | 3.38 | │ | 2.39 | 2.41 | 4.28 | │ | 7.00 | 6.80 | 11.79 |\n| &#8203;     | &nbsp; | &nbsp;                | &nbsp; | &#8203; | &nbsp; | **without fairness-enhancing** | &nbsp; | &#8203; | &nbsp; | &nbsp; | &nbsp; |\n| ProPublica COMPAS  | 13.99 | 13.90 | 15.19 | │ | 13.99 | 13.95 | 14.98 | │ | 11.49 | 11.45 | 11.80 |\n| AdultCensus       | 21.04 | 20.91 | 21.29 | │ | 20.90 | 20.91 | 21.14 | │ | 12.19 | 12.23 | 12.42 |\n\nRegarding the fairness enhancing method, across all metrics on both datasets, Adam and RMSProp achieve substantially lower (better) fairness gaps than SGD. This experiment demonstrates that the fairness benefits of adaptive optimizers are not limited to standard training but are complementary to and can amplify the effectiveness of existing fairness interventions. This finding has considerable impact, and we will add this new experiment and analysis to the final version of the paper. We also report the case of not using the fairness enhancing method, in which, as expected, Adam and RMSProp achieve better fairness.\n\n[1] Roh, Yuji, et al. \"FairBatch: Batch Selection for Model Fairness.\" ICML (2021).\n\n[2] Julia Angwin, et al. \"Machine bias: There’s software used\nacross the country to predict future criminals. And its biased against blacks.\" ProPublica (2016).\n\n[3] Ron Kohavi. \"Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid.\" KDD (1996).\n\n\n\n> **Have you considered examining other group fairness criteria, such as sufficiency (predictive parity)?**\n\nAcross recent work, the most frequently used group fairness criteria are equalized odds, equal opportunity, and demographic parity, whereas sufficiency (predictive parity/calibration) is reported less often [1]. We therefore adopted these prevalent criteria as our primary objectives.\n\n[1] Liu, Mingxuan, et al. \"A scoping review and evidence gap analysis of clinical AI fairness.\" npj Digital Medicine 8.1 (2025): 360.\n\n> **In Fig. 1, why is SGD performing so much better early on at about 20 epochs?**\n\nFig. 1 shows the fraction of 1000 independent runs whose iterate lies inside a fair neighborhood (near the fair minima of $w$=0) in the presence of both mild and severe imbalance. The apparent early lead of SGD at first 20 epochs is a transient crossing effect, not stable convergence. Because SGD has larger effective drift and diffusion terms (please see the stochastic differential equation of SGD in Eq. 11) many trajectories pass through the fair neighborhood early on as they move toward the minima that happens to be less fair. This inflates the early within-fair-neighborhood counts, but those trajectories do not remain there. In contrast, RMSProp’s diagonal normalization slows entry but retains more mass near the fair neighborhood. We will add this discussion to the final version of the paper."}, {"review_id": "QfXzYAcCJM", "reviewer": "Reviewer_qYpZ", "summary": "This paper tackles a crucial yet under-explored question in fair machine learning: How does the choice of optimization algorithm affect group fairness in deep neural networks?  It develops an SDE-based theoretical framework, proves two theorems bounding fairness gaps, and empirically evaluates on three vision datasets across multiple fairness metrics. The main claim is that adaptive optimizers consistently find fairer minima without sacrificing accuracy. \n\nThe main contributions of this paper include:\n\n1) It proposed a solid SDE-based approach, which uses SDEs and Fokker-Planck equation to connect optimizer dynamics to fairness outcomes. It includes two concrete theorems and closed-form distributions.\n2) Practical impact. No need to change model architecture or loss, simply choosing RMSProp/Adam over SGD can improve fairness, which lowers the barrier for real-world adoption .\n3) Open-source release for reproducibility. The authors publicly released the code and training pipelines at an anonymized repository, so that the reported experimental results can be reproduced.", "questions": "How well do isotropic NGOS assumptions match actual gradient noise in large models?\n\nAs to \"We hope our work will encourage broader adoption of adaptive optimizers in fairness-critical domains\" , since all experiments are conducted using vision datasets, what is the chance of the conclusions can be applied to other modalities?\n\nDoes the fairness conclusion depend on the choices of learning rates, decay rates, and batch sizes? If yes, how to choose proper hyperparemeters for this study?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The main strengths are \n1) Novel theoretical analysis. The paper provides the first SDE-based analysis that connects optimizer noise dynamics directly to fairness.\n2) Provable fairness bounds: The two theorems offer sound and general guarantees about optimizer-induced fairness effects, which is beyond empirical observations .\n3) Solid experimental results. Experiments use three datasets, three tasks, multiple architectures, three fairness metrics, and include statistical testing and imbalance-controlled ablations, solidifying the practical relevance.\n\nThe main weaknesses are\n1) The core SDE analysis is based on a one-dimensional quadratic loss with two classes. Deep networks are often high-dimensional and for multi-class, so it is unclear how these results extend. \n2) Comparisons are only between SGD and RMSProp/Adam. There are other popular variants such as SGD with momentum, AdaBound, AdamW, etc. that are not considered. \n3) Theorems 2 and 3 assume isotropic, well-behaved NGOS noise, which is an ideal condition that lacks empirical justification.", "was_revised": true, "final_justification": "The detailed rebuttal is appreciated. The additional analysis of various SGD variants provides meaningful insights, and the extension of Theorems 2 and 3 is a noteworthy contribution. The revised version represents a substantial improvement over the original submission, and so I have updated my review accordingly.", "rebuttal": "We thank the reviewer for their valuable comments. Below we provide a careful point-by-point response to each comment. We would be happy to provide additional discussions/information in the author-reviewer discussion period.\n\n> **SDE analysis (Theorem 1) based on a one-dimensional quadratic loss**\n\nWe deliberately designed a one-dimensional quadratic case and studied it as a warm‑up setup for 2 main purposes: *1.*  The local dynamics around any sufficiently smooth minimum can be approximated by a quadratic function via Taylor expansion. Hence, our SDE analysis of the fair minima provides crucial insights into the local convergence behavior of the optimizers on more general cases. *2.* To provide a clear and unambiguous *intuition* about the fairness behavior of adaptive and SGD optimizers that would be obscured in complex settings.\n\nAlso, please kindly note that we build upon the intuition from this tractable setup with more general, high-dimensional theorems. Our main theoretical results in Section 3.3 (Theorems 2 and 3) **are not limited to the 1D quadratic case.**\n\n> **Other variants such as SGD with momentum, AdamW, and AdaBound**\n\nOur theoretical analysis (Theorems 1-3) identifies gradient normalization via second moment as the key mechanism for improving group fairness. This provides a clear framework for predicting the behavior of other optimizers. To validate these predictions, we have conducted a new set of experiments with SGD w/momentum, AdamW, and AdaBound. Following the protocol from our main experiments, we use the CelebA dataset with a ViT backbone, treating gender and age as sensitive attributes. We tuned the learning rate for each optimizer using Bayesian optimization with Hyperband and report the average fairness scores below (higher is better).\n\n|               |        | **Gender** |        |   |        | **Age** |        |\n|:--------------|:------:|:----------:|:------:|:-:|:------:|:-------:|:------:|\n|  | AdamW  | AdaBound   | SGD w/mom.    | │ | AdamW  | AdaBound| SGD w/mom.    |\n| $F_{EOD}$         |   64.90    | 61.10          | 61.12      | │ | 71.11      |    67.20    | 67.25      |\n| $F_{EOP}$         | 99.90      | 96.10          | 95.07      | │ | 98.95      | 96.60       | 96.55      |\n| $F_{DPA}$         | 73.11      | 70.01          | 69.50      | │ | 98.11      | 95.39       | 95.39      |\n\n\nAs shown in the table, AdamW consistently achieves the best fairness scores. This is expected, as AdamW retains the normalization mechanism of Adam and RMSProp via the second moment. In contrast, SGD w/momentum, lacking the normalization via the second moment, performs more biased. The performance of AdaBound is comparable to SGD w/momentum. The reason is that the AdaBound algorithm deliberately clips the second moments to make it act like SGD, which in turn limits the fairness advantage.\n\n\n> **Assumption of isotropic noisy gradient oracle with scale parameter (NGOS) in Theorems 2 and 3**\n\nPlease kindly note that the isotropic assumption is common in the literature for analyzing optimizer dynamics as it yields clean and interpretable results. For instance, recent work such as [1] also employs an isotropic noise assumption to illustrate why decaying learning rate during the training is an effective mechanism.\n\nHowever, inspired by the reviewer's comment, **we now relax the conditions in Theorems 2 and 3 to non-isotropic (anisotropic) noise.** We show that the results are correct even in the anisotropic case.\n\n**Theorem 2 for anisotropic noise**: Consider a population that consists of two subgroups with subgroup-specific loss functions $\\mathcal{L}_0(w)$ and $\\mathcal{L}_1(w)$, sampled with probabilities $p_0$ and $p_1$, respectively. Suppose a stochastic (online) training regime, in which each parameter update is computed from a sample drawn from one of the two subgroups. Suppose the gradients $\\nabla \\mathcal{L}_0(w_k)$ and $\\nabla \\mathcal{L}_1(w_k)$ are well-behaved anisotropic NGOs, namely $\\mathcal{N}(\\mu_0, \\Sigma_0)$ and $\\mathcal{N}(\\mu_1, \\Sigma_1)$, respectively.  \nThen, the difference in parameter updates between subgroups 0 and 1 under RMSProp has an upper bound given by the corresponding difference under SGD.\nConsequently, RMSProp offers fairer updates across subgroups.\n\n*Proof*: For SGD, the difference between the parameter updates across subgroups is $||\\nabla \\mathcal{L}_0 - \\nabla \\mathcal{L}_1||$. Regarding RMSProp, let's compute the expected value and the variance of $v_k$:\n\n$$\n\\mathbb{E}[v_k] = \\gamma^k v_0 + (1-\\gamma^k) [ p_0 \\left( \\mu_0 ^2 + diag(\\Sigma_0) \\right)+ p_1 \\left( \\mu_1^2 + diag(\\Sigma_1) \\right)].\n$$\n\nWhere $diag(\\Sigma_0)$ and $diag(\\Sigma_1)$ are the vectors with the diagonal elements of $\\Sigma_0$ and $\\Sigma_1$, respectively. Similar to the the isotropic case, we can show that the variance of $v_k$ is negligible and thus $v_k \\rightarrow E[v_k]$. Hence, the difference between parameter updates are $|| D (\\nabla \\mathcal{L}_0 - \\nabla \\mathcal{L}_1) ||$, where $D$ is a diagonal matrix, whose $jj$th element can be computed as:\n\n$$\nD_{jj} =  \\frac{1}{\\sqrt{p_0 \\left( \\mu_{0_j} ^2 + \\sigma_{0_j}^2 \\right)+ p_1 \\left( \\mu_{1_j}^2 + \\sigma_{1_j}^2  \\right) + \\epsilon}}.\n$$\n\nPrior work [2] has experimentally shown that $\\sigma^2 > \\mu^2$. Additionally, in stochastic (online) training setup: $\\Theta^2 \\rightarrow1$. Thus, $D_{jj}<1$. This completes the proof.\n\n\nThe extension of Theorem 3 follows directly as the proof of Theorem 3 relies on the conclusion of Theorem 2.\n\nWe will add this to the appendix of the final version of our paper.\n\n[1] Shi, Bin, et al. \"On learning rates and Schrödinger operators.\" Journal of Machine Learning Research (2023).\n\n[2] Malladi, Sadhika, et al. \"On the SDEs and scaling rules for adaptive gradient algorithms.\" Advances in Neural Information Processing Systems (2022).\n\n> **What is the chance that the conclusions can be applied to other modalities beyond vision?**\n\nPlease kindly note that our claim is not modality-specific. Theorems 1, 2, and 3 reason at the level of optimizer dynamics under subgroup imbalance. They conclude that adaptive optimizer's help enhance the group fairness regardless of the input modality. These arguments depend on the geometry of gradient updates, not on the input.\n\nHowever, as per the reviewer's comment, we now perform a quick experiment using a new modality: **tabular data**. We use ProPublica COMPAS [1] and AdultCensus [2] datasets with gender as the sensitive attribute. We do classification using logistic regression, while optimizing the learning rate using Bayesian optimization. We repeat the process 10 times and following [3], we report the average gap in equal opportunity, equalized odds, and demographic parity below (lower is better).\n\n|             |        | **Equal Opportunity** |        |   |        | **Equalized Odds** |        |   |        | **Demographic Parity** |        |\n|:------------|:------:|:---------------------:|:------:|:-:|:------:|:------------------:|:------:|:-:|:------:|:----------------------:|:------:|\n| **Dataset** | Adam   | RMSProp               | SGD    | │ | Adam   | RMSProp            | SGD    | │ | Adam   | RMSProp                | SGD    |\n| ProPublica COMPAS  | 13.99 | 13.90 | 15.19 | │ | 13.99 | 13.95 | 14.98 | │ | 11.49 | 11.45 | 11.80 |\n| AdultCensus       | 21.04 | 20.91 | 21.29 | │ | 20.90 | 20.91 | 21.14 | │ | 12.19 | 12.23 | 12.42 |\n\n\nAs shown, adaptive optimizers offer better fairness over SGD in all cases. This confirms that our results are not limited to the vision application.\n\n[1] Julia Angwin, et al. \"Machine bias: There’s software used\nacross the country to predict future criminals. And its biased against blacks.\" ProPublica (2016).\n\n[2] Ron Kohavi. \"Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid.\" KDD (1996).\n\n[3] Roh, Yuji, et al. \"FairBatch: Batch Selection for Model Fairness.\" ICML (2021).\n\n\n> **Fairness conclusion depends on the choices of learning rates and other hyperparameters**\n\nIndeed the fairness outcomes can be influenced by hyperparameter choices, which is why we adopted a highly rigorous approach to mitigate this potential influence. Our primary method for ensuring a fair comparison was a robust and automated search for the most critical hyperparameter: the learning rate. As detailed in our Appendix J (l. 679) and F (l. 600), we did not manually select these values. Instead, we employed Bayesian Optimization with Hyperband algorithm to systematically find the best learning rate for each optimizer independently. We then run each optimizer 2 times with their tuned learning rate and report the average results. For other hyperparameters, we use the standard PyTorch default values. However, to further address the reviewer's comment, we conduct an additional hyperparameter search on the CelebA dataset with gender and age as the sensitive attributes using ViT backbone. This new set of experiments goes beyond only tuning the learning rate and uses Bayesian Optimization with Hyperband to jointly optimize learning rate, decay rate, and batch size.\n\n|               |        | **Gender** |        |   |        | **Age** |        |\n|:--------------|:------:|:----------:|:------:|:-:|:------:|:-------:|:------:|\n|  | Adam  | RMSProp   | SGD    | │ | Adam  | RMSProp| SGD    |\n| $F_{EOD}$         |   65.21   | 65.18          | 62.66      | │ | 72.34      |    71.99    | 68.40      |\n| $F_{EOP}$         | 99.90      | 99.91          | 96.60      | │ | 99.10      | 99.10       | 97.77      |\n| $F_{DPA}$         | 73.50      | 73.68          | 60.80      | │ | 98.20      | 98.20       | 95.40      |\n\n\nThe pattern in the above table is consistent with our initial findings. In all the scenarios, adaptive optimizers offer better fairness."}, {"review_id": "JpyxXFbGUD", "reviewer": "Reviewer_WGcQ", "summary": "The paper studies the relationship between optimizers and fairness of the resulted deep learning model. By approximating SGD and RMSProp as stochastic different equations, the paper proves that RMSProp provides fairer updates than SGD and the worst-case disparities introduced in a single step by RMSProp is less than SGD. The conclusion are validated through extensive experiments across different tasks.", "questions": "1. Could the SDE analysis provide some insights about how RMSProp can be modified to perform even better in terms of demographic parity?\n\n2. In what extent that Theorem 2 and Theorem 3 can be extended to the fairness metrics EOP and EOD?\n\n3. For the proof of Lemma 1, could the relationship between Eq 43 and zero-one loss difference be elaborated more?\n\n4. In Theorem 2, what is the definition of parameter updates between subgroups 0 and 1? Do you mean the expectation updates on the subgroups?", "limitations": "The paper does not have a limitation section. However, before the Theorems showed to be effective regarding to other fairness metrics, there should be a reminder for using the conclusion.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n* It is novel to analyze how the choice of optimizers will affect demographic parity through SDE. \n\n* The theoretical analysis is solid and the proofs are not depend on a specific network architecture.\n\n**Weakness**:\n\n* The paper compared the behavior between SGD and adaptive gradient methods. However, adaptive gradients methods, such as Adam, has already been widely used because of some other advantages, such as the converge speed. If the theoretical analysis cannot guide to find better optimizers regarding to fairness, the motivation seems to be weakened.\n\n* Some of assumptions in Theorem 2 and Theorem 3 are not justified. For example, why can we assume the gradients are well-behaved isotropic NGOs.\n\n* EOD, EOP and DPA are used in the experiments, but lack of theoretical discussion, especially when it seems like the advantage over SGD on DPA is less obvious than EOP and EOD from the experiments.", "was_revised": true, "final_justification": "The discussion period addressed most of my concerns. The new proof the author provides relax the strong assumption on isotropic noise. And it is more clear about the relationship between gradient disparities and fairness notions now.\n\nEven though I still feel the relationship can be studied and discussed further, the work, in my opinion is novel and valuable to be accept. So I maintain my score.", "rebuttal": "We thank the reviewer for their valuable comments. Below we provide a careful point-by-point response to each comment. We would be happy to provide additional discussions/information in the author-reviewer discussion period.\n\n> **Adaptive methods like Adam offer advantages like speed**\n\nWe agree Adam/RMSProp are popular for their speed. Our contribution is orthogonal: we explain and predict their fairness behavior. Our theory shows that per‑coordinate second‑moment scaling contracts subgroup‑specific update gaps under imbalance. This yields actionable intuition: the theory predicts that if one wants a higher fairness, they would increase the effective second‑moment strength (i.e., by tuning $\\beta_2$ hyperparameter). This intuition is novel and it is not related to the well-known advantages of the adaptive methods.\n\n\n> **Assumption of isotropic noise in Theorems 2 and 3**\n\nPlease kindly note that the assumption of isotropic noise is not uncommon in the literature. For instance, recent work such as [1] uses an isotropic noise assumption to illustrate why decaying learning rate during training is an effective mechanism for getting to a better minima.\n\nHowever, inspired by the reviewer's comment, **we now relax the conditions in Theorems 2 and 3 to non-isotropic (anisotropic) noise**. We show that the results are correct even in the anisotropic noise case. \n\n**Theorem 2 for anisotropic noise**: Consider a population that consists of two subgroups with subgroup-specific loss functions $\\mathcal{L}_0(w)$ and $\\mathcal{L}_1(w)$, sampled with probabilities $p_0$ and $p_1$, respectively. Suppose a stochastic (online) training regime, in which each parameter update is computed from a sample drawn from one of the two subgroups. Suppose the gradients $\\nabla \\mathcal{L}_0(w_k)$ and $\\nabla \\mathcal{L}_1(w_k)$ are well-behaved anisotropic NGOs, namely $\\mathcal{N}(\\mu_0, \\Sigma_0)$ and $\\mathcal{N}(\\mu_1, \\Sigma_1)$, respectively.  \nThen, the difference in parameter updates between subgroups 0 and 1 under RMSProp has an upper bound given by the corresponding difference under SGD.\nConsequently, RMSProp offers fairer updates across subgroups.\n\n*Proof*: For SGD, the difference between the parameter updates across subgroups is $||\\nabla \\mathcal{L}_0 - \\nabla \\mathcal{L}_1||$. Regarding RMSProp, let's compute the expected value and the variance of $v_k$:\n\n$$\n    \\mathbb{E}[v_k] = \\gamma^k v_0 + (1-\\gamma^k) [ p_0 \\left( \\mu_0 ^2 + diag(\\Sigma_0) \\right)+ p_1 \\left( \\mu_1^2 + diag(\\Sigma_1) \\right)].\n$$\n\nWhere $diag(\\Sigma_0)$ and $diag(\\Sigma_1)$ are the vectors with the diagonal elements of $\\Sigma_0$ and $\\Sigma_1$, respectively. Similar to the the isotropic case, we can show that the variance of $v_k$ is negligible and thus $v_k \\rightarrow E[v_k]$. Hence, the difference between parameter updates are $|| D (\\nabla \\mathcal{L}_0 - \\nabla \\mathcal{L}_1) ||$, where $D$ is a diagonal matrix, whose $jj$th element can be computed as:\n\n$$\n     D_{jj} =  \\frac{1}{\\sqrt{p_0 \\left( \\mu_{0_j} ^2 + \\sigma_{0_j}^2 \\right)+ p_1 \\left( \\mu_{1_j}^2 + \\sigma_{1_j}^2  \\right) + \\epsilon}}.\n$$\n\nPrior work [2] has experimentally shown that $\\sigma^2 > \\mu^2$. Additionally, in stochastic (online) training setup: $\\Theta^2 \\rightarrow1$. Thus, $D_{jj}<1$. This completes the proof.\n\n\nThe extension of Theorem 3 follows directly as the proof of Theorem 3 relies on the conclusion of Theorem 2.\n\nWe will add this to the appendix of the final version of our paper.\n\n[1] Shi, Bin, et al. \"On learning rates and Schrödinger operators.\" Journal of Machine Learning Research (2023).\n\n[2] Malladi, Sadhika, et al. \"On the SDEs and scaling rules for adaptive gradient algorithms.\" Advances in Neural Information Processing Systems (2022).\n\n\n> **Advantage over SGD on DPA is less obvious than EOP and EOD from the experiments.** \n\nFig. 2 shows that the advantage of adaptive optimizers over SGD on DPA is less obvious. This empirical result actually provides the confirmation of our Theorem 1. This theorem predicts that the DPA advantage of RMSProp is most significant under severe data imbalance. In datasets with only moderate imbalance (e.g., CelebA-Gender, with a 42\\% male-to-all ratio), the DPA gap is smaller. However, the DPA advantage is more obvious in the case of FairFace with race sensitive attribute, as it exhibits extreme imbalance, with a minority-to-all ratio of approximately 0.9\\%.\n\n> **How can RMSProp be modified to perform even better in terms of demographic parity?**\n\nOur SDE analysis provides a path for some modifications in RMSProp to further enhance its fairness. Our analysis (culminating in Eq. 15 and 20) pinpoints that the final model's bias stems from the SDE's drift term, which is driven by the imbalanced sampling probabilities ($p_0$ and $p_1$) of the data. Additionally, RMSProp's relative advantage comes from how it scales the updates: The denominator of the update involves the term $u_t$, which helps to avoid the domination by the majority sub-group. Based on these insights, a direct path to improving RMSProp's fairness is to correct the biased drift term while retaining the benefits of its adaptive normalization. The SDE model suggests that if we could modify the algorithm to optimize a debiased loss function, the stationary distribution of our SDE would shift its center to the truly fair minimum. This modification would use a debiased gradient in each step, computed by re-weighting subgroups within a mini-batch to simulate a balanced population. Our SDE framework predicts this change would directly correct the drift term, shifting the stationary distribution to be centered at the fair optimum.\n\n> **In what extent that Theorem 2 and Theorem 3 can be extended to the fairness metrics EOP and EOD?**\n\nPlease kindly note that Theorem 2 is *agnostic to the fairness metrics*. Theorem 2 is broadly applicable and provides the core intuition for the improvements in EOD and EOP. Its proof focuses on the fundamental mechanism of the optimizer that RMSProp shrinks the disparities between subgroup gradients ($||\\nabla\\mathcal{L}_0 - \\nabla\\mathcal{L}_1||$). By preventing the model from being disproportionately pulled toward one subgroup's minimum, it promotes more balanced error rates (e.g., true positive and false positive rates) across all groups. Since EOD and EOP are measures of error rate equality, the general mechanism established in Theorem 2 directly explains the strong empirical improvements we observe for these metrics.\n\nTheorem 3 relies on the DPA gap.  Direct formal extension to EOD or EOP is highly challenging, as it requires analyzing a fairness gap conditioned on the true label. This conditioning complicates the gradient analysis significantly and is beyond the current scope of the paper.\n\n\n> **For the proof of Lemma 1, could the relationship between Eq 43 and zero-one loss difference be elaborated more?**\n\nEq. 43 shows the definition of the demographic parity. Directly optimizing the non-differentiable demographic parity gap is intractable, thus it is a standard practice to replace it with a tractable setup [1]. Following prior works, we replace this with the gap in the loss functions for the sub-groups. The principle is that if a model yields a similar loss distribution for two sub groups, it is being penalized similarly for its mistakes on both, which encourages its output behavior to align.\n\n[1] Li et al. \"Achieving fairness at no utility cost via data reweighing with influence.\" ICML (2022).\n\n> **The definition of parameter updates in Theorem. 2**\n\nIn Theorem 2, the \"difference in parameter updates\" refers to a difference in two updates: one that is occurred by drawing the samples from sub group 0 and the one by drawing samples from sub group 1. We analyze the difference that would occur if the sample drawn at iteration $k$ came from subgroup 0 versus if it came from subgroup 1. As an example, For SGD, the update rule is $w_{k+1} = w_{k} - \\eta \\nabla\\mathcal{L}$; thus if a sample is drawn from subgroup say 0, the update relative to $w_k$ is $\\nabla\\mathcal{L}_0$. If there is imbalance in the dataset (e.g., the chance of drawing one sub group is substantially higher), the gradients (updates) from the majority would dominate the optimization process."}, {"review_id": "C7WmR2Dtqz", "reviewer": "Reviewer_fC4c", "summary": "This paper investigates the often-overlooked influence of optimization algorithms on group fairness in deep neural networks, particularly under severe data imbalance. Through stochastic differential equation analysis and extensive experiments on CelebA, FairFace, and MS-COCO datasets, the study reveals that adaptive optimizers like RMSProp and Adam consistently lead to fairer outcomes than stochastic methods like SGD, while maintaining comparable predictive accuracy. The research provides theoretical guarantees showing RMSProp's tendency towards fairer parameter updates and improved fairness in a single optimization step compared to SGD. Ultimately, these findings highlight the crucial role of adaptive updates as a mechanism for promoting equitable results in deep learning.", "questions": "1. Could the authors elaborate on how their findings complement or offer distinct advantages over existing fairness-aware approaches?\n2. Could the authors either provide a stronger justification for their inclusion in the main body or consider relocating them to the Appendix for improved readability?", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n* To my knowledge, the paper presents **novel findings** concerning the inherent fairness properties of various optimizers, especially in the absence of explicit fairness constraints. This is a significant and underexplored area.\n* The research effectively bolsters its core idea with a compelling blend of **intriguing theoretical analysis and extensive experimental validation**.\n\n---\n\n### Weaknesses\n\n* **Proof Placement:** I recommend moving some of the theoretical proofs to the Appendix. Specifically, the proofs for Theorem 1, Theorem 2, and Theorem 3, which together occupy over a full page in the main paper, could be relocated. Given that these theorems largely rely on toy examples, such as Gaussian distributions and simplified loss functions, the proof techniques don't strike me as particularly groundbreaking. Perhaps the authors could explicitly highlight the significance of these proof skills or justify their necessity for inclusion in the main body rather than the Appendix.\n\n* **Real-World Significance:** While the findings are undoubtedly interesting, their practical significance in real-world scenarios remains somewhat unclear. As the \"Related Work\" section acknowledges, numerous fairness-aware algorithms already exist, offering well-established trade-offs between fairness and accuracy. I suggest the authors elaborate further on how their findings complement or offer distinct advantages over these existing fairness-aware approaches, clarifying the unique contribution of this work to real-world applications.\n\n* **Missing Related Work:** Several relevant papers appear to be missing from the citations. For instance, references \\[1-2] also theoretically underscore the importance of debiased datasets, which aligns with some of the paper's underlying themes.\n\n* **Extended Experimental Comparisons:** The experimental section could be expanded to include comparisons with more in-processing fairness-aware algorithms, such as those in references \\[3-4]. While adding any comparative experiments would be beneficial, I am particularly keen to see whether the phenomena observed in this paper extend to these fairness-aware algorithms. For example, investigating how different optimizers influence the outcomes of a fairness-aware algorithm like the one in \\[3] could significantly enhance the paper's impact within the fairness research community.\n\n---\n\n[1] The Implicit Fairness Criterion of Unconstrained Learning. ICML 2019.\n\n[2] Understanding Fairness Surrogate Functions in Algorithmic Fairness. TMLR 2024.\n\n[3] FairBatch: Batch Selection for Model Fairness. ICLR 2021.\n\n[4] A Reductions Approach to Fair Classification. ICML 2018.", "was_revised": true, "final_justification": "My concern has been properly addressed. However, I am still not sure whether this paper meets the standard for this venue because it does not excite and impress me particularly. Therefore, I personally maintain the score, and both acceptance and rejection of this paper is OK for me.", "rebuttal": "We thank the reviewer for their valuable comments. Below we provide a careful point-by-point response to each comment. We would be happy to provide additional discussions/information in the author-reviewer discussion period.\n\n> **Proof replacement**\n\nWe agree that relocating the detailed proofs can improve readability and we will revise the camera ready version of the paper accordingly. Specifically, we will move the proofs of Theorems 1, 2, and 3 to the Appendix.\n\nWhile we will make this change, we want to take this opportunity to clarify a key distinction and the significance of our theoretical approach. We wish to clarify that Theorems 2 and 3 are not based on the analytically tractable setup of Theorem 1. Theorems 2 and 3 establish more general results under the much broader assumption of well-behaved isotropic noisy gradient oracle with scale parameter (NGOs). Additionally, we now **relax the isotropic assumption** and establish a new theorem based on non-isotropic (anisotropic) noise. Please see our new theorem and its detailed proof in our response to the reviewer ``qYpZ''.\n\nFurthermore, the analytically tractable quadratic setup in Theorem 1 was deliberately chosen. A quadratic function serves as a powerful local approximation for any sufficiently smooth loss function near a minimum (via a second-order Taylor expansion). Therefore, our analysis in this setting offers valuable insight into the local convergence behavior of optimizers in the complex settings.\n\n> **Real-world significance and extended comparisons over fairness-aware algorithms**\n\nOur goal is not to replace these established fairness-aware methods, but rather to demonstrate that the choice of optimizer is a fundamental, orthogonal, and complementary mechanism for improving fairness. However, based on your comment, to elaborate more on the real world significance of our findings, **we have conducted a new experiment**. We test the impact of optimizers in addition to the fairness-enhancing method [1] on two new datasets (ProPublica COMPAS [2] and AdultCensus [3]) with gender as the sensitive attribute. The results are presented in the table below for SGD, Adam, and RMSProp. We repeat the process 10 times and following [3], we report the average gap in equal opportunity, equalized odds, and demographic parity below (lower is better).\n\n\n|             |        | **Equal Opportunity** |        |   |        | **Equalized Odds** |        |   |        | **Demographic Parity** |        |\n|:------------|:------:|:---------------------:|:------:|:-:|:------:|:------------------:|:------:|:-:|:------:|:----------------------:|:------:|\n| **Dataset** | Adam   | RMSProp               | SGD    | │ | Adam   | RMSProp            | SGD    | │ | Adam   | RMSProp                | SGD    |\n| &#8203;     | &nbsp; | &nbsp;                | &nbsp; | &#8203; | &nbsp; | **with fairness enhancing** | &nbsp; | &#8203; | &nbsp; | &nbsp; | &nbsp; |\n| ProPublica COMPAS  | 0.45 | 0.48 | 0.71 | │ | 2.79 | 2.78 | 2.90 | │ | 0.86 | 0.86 | 2.60 |\n| AdultCensus       | 3.15 | 3.16 | 3.38 | │ | 2.39 | 2.41 | 4.28 | │ | 7.00 | 6.80 | 11.79 |\n| &#8203;     | &nbsp; | &nbsp;                | &nbsp; | &#8203; | &nbsp; | **without fairness-enhancing** | &nbsp; | &#8203; | &nbsp; | &nbsp; | &nbsp; |\n| ProPublica COMPAS  | 13.99 | 13.90 | 15.19 | │ | 13.99 | 13.95 | 14.98 | │ | 11.49 | 11.45 | 11.80 |\n| AdultCensus       | 21.04 | 20.91 | 21.29 | │ | 20.90 | 20.91 | 21.14 | │ | 12.19 | 12.23 | 12.42 \n\nAs the results show, adaptive optimizers outperform SGD both with and without the explicit fairness-enhancing method. This directly leads to the contribution and practical significance of our work: choosing an adaptive optimizer is not an alternative to existing fairness algorithms, but rather a **complementary** strategy that amplifies their effectiveness. A practitioner can gain almost ``free'' and significant boost in fairness by simply selecting Adam or RMSProp as the optimizer for their chosen fairness-aware training algorithm.\n\n[1] Roh, Yuji, et al. \"FairBatch: Batch Selection for Model Fairness.\" ICML (2021).\n\n[2] Julia Angwin, et al. \"Machine bias: There’s software used\nacross the country to predict future criminals. And its biased against blacks.\" ProPublica (2016).\n\n[3] Ron Kohavi. \"Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid.\" KDD (1996).\n\n\n> **Missing related work**\n\nWe will add the mentioned papers to the related work section:\n\nThe work by Liu et al. [1] provides a key theoretical analysis showing that standard and unconstrained machine learning implicitly favors one fairness criterion over others. The authors prove that a model's deviation from calibration is bounded by its excess risk. This is related to our work as it also investigates the fairness properties that emerge from a standard training process. \n\nThe paper by Yao et al. [2] focuses on in-processing fairness methods that use surrogate functions to ensure fairness. Their theory also highlights that balanced datasets are beneficial for achieving tighter fairness and stability guarantees. This is relevant to our findings, as we also identify data imbalance as a critical factor through SDE analysis.\n\n[1] Liu et al., \"The Implicit Fairness Criterion of Unconstrained Learning.\" ICML (2019).\n\n[2] Yao et al. \"Understanding Fairness Surrogate Functions in Algorithmic Fairness.\" TMLR (2024).\n\n\n> **Elaboration on how findings complement or offer distinct advantages over existing fairness-aware approaches**\n\nOur core theoretical results (Theorems 2 and 3) do not make strong assumptions about the specific form of the loss function. This generality is crucial because many in-processing fairness methods operate by adding a fairness regularizer to the standard loss. Our theory suggests that even with such a modified objective, the inherent mechanics of an adaptive optimizer like Adam should still offer benefits by normalizing subgroup-specific gradient disparities more effectively than SGD.\n\nTo validate this complementary effect, and motivated by your comment, we conducted **the new experiment** presented in the table in our previous response. Those results show that adaptive optimizers achieve superior fairness compared to SGD, both in a standard setup and when used as the optimization backbone for an existing fairness-enhancing algorithm. This clarifies the distinct and practical contribution of our work: the choice of an adaptive optimizer is not a substitute for dedicated fairness algorithms but is a complementary decision that can amplify their impact.\n\n> **Provide a stronger justification for inclusion of proofs in the main body or consider relocating them to the Appendix for improved readability**\n\nWe agree that relocating the detailed proofs can improve the readability. For a more detailed discussion on this, please see our response to the **Proof replacement** title."}]

Author rebuttal:

Meta-review: This paper investigates the under-explored relationship between optimization algorithms and group fairness in deep neural networks, demonstrating through stochastic differential equation analysis that adaptive optimizers like RMSProp and Adam consistently achieve fairer outcomes than SGD while maintaining comparable accuracy. The reviewers recognized the work's significant contributions, particularly its novel theoretical framework connecting optimizer dynamics to fairness outcomes and the practical insight that simply changing the optimizer can improve fairness without modifying model architecture or loss functions. During the rebuttal period, the authors effectively addressed all reviewer concerns by providing generalized theorems with relaxed assumptions and conducting experiments on tabular data to confirm generalizability beyond vision. While some reviewers initially questioned practical significance given existing fairness-aware algorithms, the authors convincingly demonstrated that optimizer choice serves as a fundamental, orthogonal mechanism that amplifies the effectiveness of existing fairness interventions rather than replacing them. Given the paper's novel theoretical insights into an overlooked aspect of fairness, strong empirical validation, and thorough responses addressing all reviewer concerns, I recommend accepting this paper for NeurIPS 2025.
Decision: Accept (spotlight)