Title: Efficient Adaptive Federated Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: federated optimization

communication- and memory-efficiency

adaptivity
Abstract: Adaptive optimization is critical in federated learning, where enabling adaptivity on both the server and client sides has proven essential for achieving optimal performance. However, the scalability of such jointly adaptive systems is often hindered by resource limitations in communication and memory. In this paper, we introduce a class of efficient adaptive algorithms, named $FedAda^2$ and its enhanced version $FedAda^2$++, designed specifically for large-scale, cross-device federated environments. $FedAda^2$ optimizes communication efficiency by avoiding the transfer of preconditioners between the server and clients. Additionally, $FedAda^2$++ extends this approach by incorporating memory-efficient adaptive optimizers on the client side, further reducing on-device memory usage. Theoretically, we demonstrate that $FedAda^2$ and $FedAda^2$++ achieve the same convergence rates for general, non-convex objectives as its more resource-intensive counterparts that directly integrate joint adaptivity. Extensive empirical evaluations on image and text datasets demonstrate both the advantages of joint adaptivity and the effectiveness  and efficiency of $FedAda^2$/$FedAda^2$++.


Peer reviews:
comments: SUMMARY:
This paper proposes FedAda 2 and FedAda 2++, two communication- and memory-efficient adaptive algorithms for federated learning that retain the benefits of joint adaptivity while achieving the same convergence rates as more resource-intensive methods, with strong empirical performance across image and text tasks.

STRENGTHS AND WEAKNESSES:
### Strengths:
The paper demonstrates that communicating preconditioners in adaptive optimization is unnecessary. This is an interesting finding, as it enables both memory and communication efficiency. Moreover, the approach does not compromise convergence performance, making it a promising direction for future research.


### Weaknesses
1. Without an aggregated or communicated preconditioner, the server must compute the preconditioner independently. This can introduce additional computational and memory overhead, especially since some prior work often assumes that the server performs only minimal computations.
  
2. Although the authors note that the bounded gradients assumption has been used in prior work, it remains a strong assumption given the potential variability of local objectives. It would be more exciting to explore results under weaker or more realistic assumptions.

QUESTIONS:
Since the proposed methods eliminate the need for communicating preconditioners, one would expect a significant reduction in communication cost, as reflected in Figure 2. However, this improvement is not clearly observed—could the authors clarify the reason for this discrepancy?

LIMITATIONS:
No significant limitation.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an adaptive federated optimizer featuring local preconditioner re‑initialization, global preconditioner updates, and memory‑efficient client optimizers, matching prior convergence rates and validated on image/text benchmarks. They further demonstrate that this design balances adaptivity with communication and memory efficiency, yielding performance on par with baselines under resource constraints.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- Provides rigorous convergence proofs for non‑convex jointly adaptive federated training under minimal optimizer conditions.\n- Reduces communication overhead by initializing local preconditioners to zero without degrading performance.\n- Integrates memory‑efficient optimizers (e.g., SM3) to dramatically reduce client‑side memory.\n\nWeaknesses\n\n- Thm 4.1 is not introduced well, and is not very intuitive.\n- The zero‑cost re‑initialization trick is conceptually obvious once you view the server preconditioner as an aggregator.\n- Memory‑efficient optimizers (SM3) are basically plugged in.\n- The theoretical insights beyond standard bounds are limited; proofs largely follow existing templates.\n- Empirical gains are modest and confined to standard benchmarks, with no demonstration on truly large‑scale or heterogeneous deployments.\n- Convergence rates match prior Ada‑style FL methods.\n- Experiments lack ablation on key hyperparameters (e.g., smoothing ε, delay z) in main text.\n- Scalability evaluation omits wall‑clock time or system‑level benchmarks in true cross‑device settings.\n- Privacy study is limited to logistic regression on StackOverflow.\n- Dataset diversity is modest (only four benchmarks), with no evaluation on non‑vision/text tasks or personalization objectives.\n- Ablation of memory vs. accuracy trade‑offs for varying model sizes is not provided.\n- Experiments with heterogeneous client data distributions do not control the degree of heterogeneity.\n\nQUESTIONS:\n./.\n\nLIMITATIONS:\n- The authors list some technical limitations. Broader societal impacts (e.g. misapplication in high‑stakes domains) are not well discussed, but risks appear negligible.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper presents efficient adaptive federated optimization algorithms FedAda2 and FedAda2++. Specifically, these two algorithms avoid the transfer of preconditioners between the server and clients, and also incorporates memory-efficient adaptive optimizers on the client side. They shows the advantage both theoretically and empirically.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The methods are theoretically well-founded. The paper first introduces a general algorithm, proves its convergence, and then develops a practical empirical version.
2. The writing is clear and easy to follow.

Weaknesses:
1. The empirical results, as shown in Figure 1, do not demonstrate a clear improvement over the baselines.
2. The scale of the experiments is limited. Given the current state of the field in 2025, more large-scale evaluations would be expected.
3. It would be better to have more insight on the advantage of certain joint adaptivity.

QUESTIONS:
1. What is the key new insight in the theoretical analysis, considering that the algorithm is fairly general? For instance, prior work such as [1] also provides theoretical guarantees for a general algorithmic framework. How does this work differ or advance the theory?


Ref:
1. Fednar: Federated optimization with normalized annealing regularization

LIMITATIONS:
It's better to have experiments in larger scale.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces less resource-intensive adaptive algorithms, FedAda and FedAda++, for cross-device FL, and explores joint server and client-side adaptivity.  FedAda avoids the transfer of preconditioners between server and clients, and FedAda++ uses memory-efficient adaptive optimizers on clients. The authors establish similar convergence guarantees to those of the resource-intensive counterparts like FedAdam for general non-convex objectives with a coordinate-wise bounded gradient assumption.  \n\nPS: Not sure if you really need a superscript $.^2$ in the algorithm's name. What exactly will be difficulty if that is avoided?\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The FedAda framework permits the use of different optimizers at the client and server sides. \n\n**Weaknesses:**\n\n2. The proofs presented are difficult to follow through and seem to introduce a lot of abstractions without sufficient explanation. A clearer explanation of the steps involved in high-level ideas at some points may aid the reader's understanding of the proofs. Eg., proof (ii) of Proposition A.3. is quite dense and should be clearly explained for better understanding. \n\n3. Line 620 is unclear. It is challenging to assess the validity of the main theoretical results. \n\n4. The experiments are conducted on an array of datasets and demonstrate that FedAda and FedAda++ performs better than other existing popular adaptive federated learning algorithms and are at par with costly joint adaptivity. \n\n4. Overall, the readability of the paper can be substantially improved in the main body and proofs.\n\nQUESTIONS:\nIn addition to addressing the weaknesses mentioned above, see the following:\n\n1. Table 1 does not contain the statistics for FedAdam. Can the authors explain why? It would have been insightful to compare the efficiency and performance of FedAda algorithms against FedOpt algorithms in [2]. \n\n2. The authors state that they aim to increase communication efficiency by minimizing the transfer of local preconditioners, as opposed to other works that use model compression like FedCAMS, FedAMS [1], etc. Can the authors explain or compare the efficiency metrics \nbetween these two approaches? And which one to use in specific scenarios? \n\n[1] Yujia Wang, Lu Lin, and Jinghui Chen. Communication-efficient adaptive federated learning. In International Conference on Machine Learning, pages 22802–22838. PMLR, 2022. \n\n[2] Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konecný, Sanjiv Kumar, and Brendan McMahan. Adaptive federated optimization. International Conference on Learning Representations, 2021.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-11"}