Title: Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairness
Venue: NeurIPS 2025
Accepted: yes
Keywords: Group fairness

Optimization

Stochastic differential equations
Abstract: We study whether and how the choice of optimization algorithm can impact group fairness in deep neural networks. Through stochastic differential equation analysis of optimization dynamics in an analytically tractable setup, we demonstrate that the choice of optimization algorithm indeed influences fairness outcomes, particularly under severe imbalance.  Furthermore, we show that when comparing two categories of optimizers, adaptive methods and stochastic methods, RMSProp (from the adaptive category) has a higher likelihood of converging to fairer minima than SGD (from the stochastic category). Building on this insight, we derive two new theoretical guarantees showing that, under appropriate conditions, RMSProp exhibits fairer parameter updates and improved fairness in a single optimization step compared to SGD. We then validate these findings through extensive experiments on three publicly available datasets, namely CelebA, FairFace, and MS-COCO, across different tasks as facial expression recognition, gender classification, and multi-label classification, using various backbones. Considering multiple fairness definitions including equalized odds, equal opportunity, and demographic parity, adaptive optimizers like RMSProp and Adam consistently outperform SGD in terms of group fairness, while maintaining comparable predictive accuracy. Our results highlight the role of adaptive updates as a crucial yet overlooked mechanism for promoting fair outcomes. We release the source code at: https://github.com/Mkolahdoozi/Some-Optimizers-Are-More-Equal.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the often-overlooked influence of optimization algorithms on group fairness in deep neural networks, particularly under severe data imbalance. Through stochastic differential equation analysis and extensive experiments on CelebA, FairFace, and MS-COCO datasets, the study reveals that adaptive optimizers like RMSProp and Adam consistently lead to fairer outcomes than stochastic methods like SGD, while maintaining comparable predictive accuracy. The research provides theoretical guarantees showing RMSProp's tendency towards fairer parameter updates and improved fairness in a single optimization step compared to SGD. Ultimately, these findings highlight the crucial role of adaptive updates as a mechanism for promoting equitable results in deep learning.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n* To my knowledge, the paper presents **novel findings** concerning the inherent fairness properties of various optimizers, especially in the absence of explicit fairness constraints. This is a significant and underexplored area.\n* The research effectively bolsters its core idea with a compelling blend of **intriguing theoretical analysis and extensive experimental validation**.\n\n---\n\n### Weaknesses\n\n* **Proof Placement:** I recommend moving some of the theoretical proofs to the Appendix. Specifically, the proofs for Theorem 1, Theorem 2, and Theorem 3, which together occupy over a full page in the main paper, could be relocated. Given that these theorems largely rely on toy examples, such as Gaussian distributions and simplified loss functions, the proof techniques don't strike me as particularly groundbreaking. Perhaps the authors could explicitly highlight the significance of these proof skills or justify their necessity for inclusion in the main body rather than the Appendix.\n\n* **Real-World Significance:** While the findings are undoubtedly interesting, their practical significance in real-world scenarios remains somewhat unclear. As the \"Related Work\" section acknowledges, numerous fairness-aware algorithms already exist, offering well-established trade-offs between fairness and accuracy. I suggest the authors elaborate further on how their findings complement or offer distinct advantages over these existing fairness-aware approaches, clarifying the unique contribution of this work to real-world applications.\n\n* **Missing Related Work:** Several relevant papers appear to be missing from the citations. For instance, references \\[1-2] also theoretically underscore the importance of debiased datasets, which aligns with some of the paper's underlying themes.\n\n* **Extended Experimental Comparisons:** The experimental section could be expanded to include comparisons with more in-processing fairness-aware algorithms, such as those in references \\[3-4]. While adding any comparative experiments would be beneficial, I am particularly keen to see whether the phenomena observed in this paper extend to these fairness-aware algorithms. For example, investigating how different optimizers influence the outcomes of a fairness-aware algorithm like the one in \\[3] could significantly enhance the paper's impact within the fairness research community.\n\n---\n\n[1] The Implicit Fairness Criterion of Unconstrained Learning. ICML 2019.\n\n[2] Understanding Fairness Surrogate Functions in Algorithmic Fairness. TMLR 2024.\n\n[3] FairBatch: Batch Selection for Model Fairness. ICLR 2021.\n\n[4] A Reductions Approach to Fair Classification. ICML 2018.\n\nQUESTIONS:\n1. Could the authors elaborate on how their findings complement or offer distinct advantages over existing fairness-aware approaches?\n2. Could the authors either provide a stronger justification for their inclusion in the main body or consider relocating them to the Appendix for improved readability?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the relationship between optimizers and fairness of the resulted deep learning model. By approximating SGD and RMSProp as stochastic different equations, the paper proves that RMSProp provides fairer updates than SGD and the worst-case disparities introduced in a single step by RMSProp is less than SGD. The conclusion are validated through extensive experiments across different tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n* It is novel to analyze how the choice of optimizers will affect demographic parity through SDE. \n\n* The theoretical analysis is solid and the proofs are not depend on a specific network architecture.\n\n**Weakness**:\n\n* The paper compared the behavior between SGD and adaptive gradient methods. However, adaptive gradients methods, such as Adam, has already been widely used because of some other advantages, such as the converge speed. If the theoretical analysis cannot guide to find better optimizers regarding to fairness, the motivation seems to be weakened.\n\n* Some of assumptions in Theorem 2 and Theorem 3 are not justified. For example, why can we assume the gradients are well-behaved isotropic NGOs.\n\n* EOD, EOP and DPA are used in the experiments, but lack of theoretical discussion, especially when it seems like the advantage over SGD on DPA is less obvious than EOP and EOD from the experiments.\n\nQUESTIONS:\n1. Could the SDE analysis provide some insights about how RMSProp can be modified to perform even better in terms of demographic parity?\n\n2. In what extent that Theorem 2 and Theorem 3 can be extended to the fairness metrics EOP and EOD?\n\n3. For the proof of Lemma 1, could the relationship between Eq 43 and zero-one loss difference be elaborated more?\n\n4. In Theorem 2, what is the definition of parameter updates between subgroups 0 and 1? Do you mean the expectation updates on the subgroups?\n\nLIMITATIONS:\nThe paper does not have a limitation section. However, before the Theorems showed to be effective regarding to other fairness metrics, there should be a reminder for using the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles a crucial yet under-explored question in fair machine learning: How does the choice of optimization algorithm affect group fairness in deep neural networks?  It develops an SDE-based theoretical framework, proves two theorems bounding fairness gaps, and empirically evaluates on three vision datasets across multiple fairness metrics. The main claim is that adaptive optimizers consistently find fairer minima without sacrificing accuracy. \n\nThe main contributions of this paper include:\n\n1) It proposed a solid SDE-based approach, which uses SDEs and Fokker-Planck equation to connect optimizer dynamics to fairness outcomes. It includes two concrete theorems and closed-form distributions.\n2) Practical impact. No need to change model architecture or loss, simply choosing RMSProp/Adam over SGD can improve fairness, which lowers the barrier for real-world adoption .\n3) Open-source release for reproducibility. The authors publicly released the code and training pipelines at an anonymized repository, so that the reported experimental results can be reproduced.\n\nSTRENGTHS AND WEAKNESSES:\nThe main strengths are \n1) Novel theoretical analysis. The paper provides the first SDE-based analysis that connects optimizer noise dynamics directly to fairness.\n2) Provable fairness bounds: The two theorems offer sound and general guarantees about optimizer-induced fairness effects, which is beyond empirical observations .\n3) Solid experimental results. Experiments use three datasets, three tasks, multiple architectures, three fairness metrics, and include statistical testing and imbalance-controlled ablations, solidifying the practical relevance.\n\nThe main weaknesses are\n1) The core SDE analysis is based on a one-dimensional quadratic loss with two classes. Deep networks are often high-dimensional and for multi-class, so it is unclear how these results extend. \n2) Comparisons are only between SGD and RMSProp/Adam. There are other popular variants such as SGD with momentum, AdaBound, AdamW, etc. that are not considered. \n3) Theorems 2 and 3 assume isotropic, well-behaved NGOS noise, which is an ideal condition that lacks empirical justification.\n\nQUESTIONS:\nHow well do isotropic NGOS assumptions match actual gradient noise in large models?\n\nAs to \"We hope our work will encourage broader adoption of adaptive optimizers in fairness-critical domains\" , since all experiments are conducted using vision datasets, what is the chance of the conclusions can be applied to other modalities?\n\nDoes the fairness conclusion depend on the choices of learning rates, decay rates, and batch sizes? If yes, how to choose proper hyperparemeters for this study?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn machine learning, optimization algorithms adjust the parameters to minimize the loss function. What hasn’t been examined yet is what effect optimizers have on common group fairness criteria. This paper sets out to do so by first examining the role of optimizers in an analytically tractable setup. The authors focus on two categories of optimization algorithms, namely adaptive ones and stochastic ones. In their theoretical examination, they focus on RMSProp (an adaptive optimization algorithm) and SGD (which is stochastic). They use stochastic differential equation analysis (SDE) to find that RMSProp leads to fairer updates than SGD (specifically, the difference in parameter updates between the demographic groups is smaller). The authors then confirm this result experimentally using three datasets and three group fairness definitions (demographic parity, equality of opportunity and equalized odds). The better performance of adaptive methods on group fairness holds not only when the adaptive methods lead to better accuracy. Adaptive optimizers can thus be a simple and cost-effective way to improve group fairness criteria (before potentially using other methods to improve fairness further).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths labeled with [S], weaknesses with [W], general thoughts/comments with [T]\n\n# Major\n## Introduction\n- [S] Well-written introduction that gets the point across\n## Related work\n- [S] My expertise is in fairness and not in theoretical ML/optimizers, but I thought the fairness section of related work was well-written and the section on optimizers helped me remember some basic stuff\n- [T] l.74f.: The reason why you focus on group fairness is plausible, although I wonder to what extend individual fairness would even be implementable for the facial recognition dataset\n## Method\n- [S] I appreciate the section on preliminaries to create a shared understanding\n- [W] However, building some more intuition would have really helped me get the paper content faster. The explanations are still quite theoretical and based on formulas, which didn’t help me in getting a good intuition for what’s happening. For something like the NGOs, for example, explaining in basic terms what it does would have helped me a lot rather than jumping straight to the formula.\n- [S] I appreciate that theoretical results were followed by empirical tests as sanity checks and a more robust verification\n\n## Experiments\n- [W] What does fairness in the context of gender classification mean? Is a fair classification simply a correct one? If so, why are you using demographic parity, equalized odds and equal opportunity over just accuracy? In other words: The experimental results from these datasets are interesting if you see the dataset as just a stand-in for a dataset with sensitive attributes, but using these fairness metrics for these datasets implicitly implies that it makes sense to measure fairness this way for real-world gender prediction – which I think is not the case, so this should be contextualized.\n\n# Minor\n## Related work\n- l.83: typo in “learning algorithm”\n- l.88f.: “One major limitation is their computational overhead or interference 89 with the training phase.” → Post-processing, when implemented as a simple threshold rule, for example, can be fairly cheap and easy to implement. However, your point still stands about the ease of changing the optimizer.\n- l.123: add “(ODE)” abbreviation after “ordinary differential equations” as you use the abbreviation in line 126 without introducing it\n## Experiments\n- Fig. 2: Please don’t use a line chart here as it implies a continuous or ordered relationship between data points when the CelebA-G, CelebA-A, etc. datapoints don’t have a continuous relationship.\n- l.307: The fairness metrics (F_{EOD} and the others) are only defined in the appendix. I don’t think that’s ideal since it’s an important part of the evaluation, but I can live with it. However, it seems important to at least clarify the naming convention (F_{EOD} = equalized odds, F_{EOP} = equality of opportunity and F_{DPA} = demographic parity) in the main text once.\n- As you don’t explain how F_{EOD} and the other fairness criteria are calculated in the main text, it would be helpful to at least mention whether you want these values to be high or low. This helps the reader interpret Fig. 2 more quickly.\n## Conclusion\n- l. 389: typo in “demographic disparity”\n\nQUESTIONS:\n- Why do you use demographic parity as a fairness metric for a gender classification system? In gender classification, we care about correct labels, not whether the likelihood of getting a certain label is the same for all demographic groups. If one group is underrepresented in the dataset, there’s no reason (at least from my point of view) to expect demographic parity (that’s different for cases where, in an ideal world, we would expect different groups to have the same distribution for the ground truth label).\n- Since simply changing the optimizer method does not give us any fairness guarantees, do you see this change in optimizer to be used with other methods (e.g., post-processing methods)?\n- Out of curiosity (so not saying it’s missing): Have you considered examining other group fairness criteria, such as sufficiency (predictive parity)? If so, why did you decide against it?\n- Fig. 1: Why is SGD performing so much better early on at about 20 epochs?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}