FedHyper: A Universal and Robust Learning Rate Scheduler for Federated Learning with Hypergradient Descent
Authors: Ziyao Wang
Jianyu Wang
Ang Li
Venue: ICLR 2024 poster (2024)
Keywords: Federate Learning
Learning rate
Hyperparameter
Hypergradient
Primary area: general machine learning (i.e., none of the above)

Abstract: The theoretical landscape of federated learning (FL) undergoes rapid evolution, but its practical application encounters a series of intricate challenges, and hyperparameter optimization is one of these critical challenges. Amongst the diverse adjustments in hyperparameters, the adaptation of the learning rate emerges as a crucial component, holding the promise of significantly enhancing the efficacy of FL systems. In response to this critical need, this paper presents FedHyper, a novel hypergradient-based learning rate adaptation algorithm specifically designed for FL. FedHyper serves as a universal learning rate scheduler that can adapt both global and local rates as the training progresses. In addition, FedHyper not only showcases unparalleled robustness to a spectrum of initial learning rate configurations but also significantly alleviates the necessity for laborious empirical learning rate adjustments. We provide a comprehensive theoretical analysis of FedHyper’s convergence rate and conduct extensive experiments on vision and language benchmark datasets. The results demonstrate that FEDHYPER consistently converges 1.1-3× faster than FedAvg and the competing baselines while achieving superior final accuracy. Moreover, FEDHYPER catalyzes a remarkable surge in accuracy, augmenting it by up to 15% compared to FedAvg under suboptimal initial learning rate settings.

Published as a conference paper at ICLR 2024
FEDHYPER: A UNIVERSAL AND ROBUST LEARNING RATE SCHEDULER FOR FEDERATED LEARNING WITH HYPERGRADIENT DESCENT
Ziyao Wang', Jianyu Wang", Ang Li"
'University of Maryland. College Park, Apple
{ziyaow, angliece (umdd edu, {jianywwrrg)espplee.com
ABSTRACT
The theoretical landscape of federated learning (FL) undergoes rapid evolution, but its practical application encounters a series of intricate challenges, and hy- perparameter optimization is one of these critical challenges. Amongst the di- verse adjustments in hyperparameters. the adaptation of the learning rate emerges as crucial component, holding the promise of significantly enhancing the effi- cacy of FL systems. In response to this critical need, this paper presents FED- HYPER, novel bypergradient-basel learning rate scheduling algorithm for FL. FEDHyPER serves as a universal learning rate scheduler that can adapt both global and local learning rates the training progresses. In addition, FEDHY- PER not only showcases unparalleled robustness to a spectrum initial learning rate configurations but also significantly alleviates the necessity for laborious em- pirical learning rate adjustments. We provide a comprehensive theoretical anal- ysis of FEDHYPER's convergence sion and language benchmark datasets. The results demonstrate that FEDHYper consistently converges 1.1-3x faster than FEDAVG and the competing baselines while achieving superior final accuracy. Moreover, FEDHYPER catalyzes a re- markable surge in accuracy, augmenting it by up 15% compared to FEDAVG under suboptimal initial learning rate settings. Our source code is released at https:// /tte:://qituuss com ATP com//A/11111111//eeeeeerrr
and conduct extensive experiments vi-
INTRODUCTION
Federated Learning (FL) (Zhang et al. 2021; McMahan et al., 2017 Reddi et al. 2020) has emerged as popular distributed machine learning paradigm in which server orchestrates the collaborative training of a machine learning model across massive distributed devices. The clients only need to communicate local model updates with the server without explicitly sharing the private data. FL has been widely applied in numerous applications. such as the next-word prediction a virtual keyboard on the smartphone (Hard et al. (218)) and hospitalization prediction for patients (Brisimi et al., 2018). However, FL. presents challenges compared to conventional distributed training, including communication bottlenecks, data heterogeneity, privacy concern,, etc (Kairouz et al. (021).
Difficulty of Scheduling Learning Rates in FL. In each communication round of FL, the selected clients usually perform several epochs of local stochastic gradient descent (SGD) to optimize their local models before communicating with the server in FEDAVG. The server then updates the global model by the aggregated client updates Li et al. (2019: Karimireddy et al.) (2020). According to this process, FL involves two types of learming rates, I.c., global learning rate on the server to update the global model and local learning rate for optimizing the local objectives on clients. They both signif- icantly impact the model accuracy and convergence speed (Jhunjhunwala et al.) (023), while their optimal values can be influenced by various factors such as the dataset, model structure, and train- ing stage (Reddi et al. 2020). Suboptimal learning rates can hinder model convergence, i.e., small learning rates slow down convergence while aggressive learning rates prohibit convergence (Barzilai & Borwein 1988). Therefore, scheduling the learning rates is crucial for both speeding up conver- gence and improving the model performance in FL, as observed previously in centralized machine learning (You et al. 2019), Nevertheless, one of the critical challenges in FL is the data heterogene-

---

Published as a conference paper at ICLR 2024
ity across clients Rush al. (002), the local optimization objectives might significantly diverge from the global one. Therefore, the server and clients need to cooperatively schedule learning rates to foster their synergy. This makes the scheduler in FL very complicated. Merely applying the opti- mizers in traditional machine learning, such as Adam (Kingma & Ba 2014) and Adagrad (Lydia & Francis 2019), to FL. cannot consistently work well across different FL. tasks.
In addition to the aforementioned complexities, configuring appropriate initial learning rates is also challenging for FL due heterogeneous optimization objectives across clients. It is impractical to explore different initial learning rates due to expensive communication and computation costs. Therefore, it necessary to design a scheduler that is robust against random initial learning rates.
Previous Works. Recent studies in speeding up FL have focused on adjusting the learning rate through optimizers or schedulers. primarily emphasizing the global model update. For instance, weight decay (Yan et al. [022) is used to decrease the global learning rate as training progresses but exhibits much slower convergence when its initial learning rate is below the optimal value. Al- ternative methods, such (Reddi et al., (220),, harness popular optimizers in centralized machine learning, i.e., Adam and Adagrad, for global updates. However, a very recent work, FEDEXP (Jhun- jhunwala et al. 2023), reveals that these methods are not efficient enough in FL. FEDEXP adjusts the global learning rate based on local updates and outperforms the methods aforementioned. How- ever, it exhibits inconsistent performance across tasks and can result in unstable accuracy in the late training stages. More importantly, these approaches mainly consider the server's 's perspective on global learning rate adjustments. We argue for a scheduler that integrates both global and local rates, enhancing server-client collaboration. Additionally, current solutions generally fail to ensure robustness against variations in the initial learning rate, with the exception of FEDEXP, which only maintains resilience against the initial global learning rate.
Therefore, we seek to answer a compelling research question: How can we schedule both global and local learning rates according to the training progress and the heterogeneous local optimization objectives while ensuring robustness against the random initial learning rate?
Our Solution. To tackle the question above, we design FEDHYPER, a universal and robust learn- ing rate scheduler for FL. Specifically, FEDHYPER functions as a versatile toolbox, facilitating the scheduling of learning rates both on the server and among the clients. It encompasses (1) global scheduler that schedules the global learning rate on the server; (2) a server-side local scheduler that tunes the local learning rate on the server; (3) client-side local scheduler that adjusts the local learning rate between local epochs on the clients. These schedulers can be either independently or cooperatively applied to FL tasks, aligning with user specifications. Moreover, FEDHYPER is also seamlessly compatible with current FL optimization algorithms, thereby further improving their efficacy, such as server momentum (Liu et al. 2020) and FEDADAM (Reddi et al. 2020).
The key idea of FEDHYPER, inspired by (Chen et al. 2022 Baydin et al.) 2017). is to utilize the hypergradient (Maclaurin et al.) (2015) of the learning rate to guide the scheduling. We begin by calling the theoretical analysis of the relationship between the optimization objective and the learn- ing rates. The analysis reveals that the hypergradient of the learning rate is influenced by the model gradients from the current and previous training steps. Based on this insight, we formally define the hypergradients of local and global learning rates in FL by exploiting the current and historical model updates. Consequently, we design FEDHYPER, consisting of the three aforementioned schedulers. As the hypergradient of the learning rate is based on the model updates, it can precisely capture the dynamics of the training progress. Therefore, FEDHYPER can effectively schedule random initial learning rates to optimal values, making it robust against different initial learning rates. Through experiments spanning computer vision and natural language tasks, we demonstrate that FEDHYPER effectively fosters convergence, facilitates superior accuracy, and enhances the robustness against random initial learning rates. Our contributions are summarized as follows:
We conduct meticulous theoretical analysis for the hypergradients of local and global learning rates with regard to the gradients of local and global models.
Based on the analysis described above, we designed FEDHYPER. a universal and robust learning rate scheduling algorithm. which can adjust both global and local learning rates and enhance the robustness against random initial learning rates.
We propose a novel theoretical framework of FL by considering the dynamic local and global learning rates, showing that FEDHYPER is guaranteed converge in FL scenarios.
2

---

Published as a conference paper at ICLR 2024
We conduct extensive experiments vision and language tasks. The results demonstrate not only the efficacy of FEDHYPER to improve both convergence speed and accuracy, but also its robustness under various initial learning rate settings. FEDHYPER convergences 1.1-3x faster than FEDAVG and most competing baselines, and increases accuracy by up to 15% in various initial learning rate setrings. Moreover, plugging FEDHyper into current FL optimization methods also exhibits additional performance improvement.
RELATED WORK
Hypergradient Descent of Learning Rates. Hypergradient descent is commonly used to optimize the learning rate due to its integral role in the gradient descent process. Baydin et al. introduced HD (Baydin al. 2017). , an algorithm that approximates the current learning rate's hypergradient using the gradients from the last two epochs. Building on this, the Differentiable Self-Adaptive (DSA) learning rate (Chen al. (022) was proposed, which precomputes the next epoch's 's gradient and uses it, along with the current epoch's s gradient, to determine the hypergradient. While DSA offers enhanced accuracy over HD, it requires greater computational resources. Moreover, studies like (Jie et al. 2022) have explored optimizing the meta-learning rate in hypergradient descent to further improve HD efficiency.
Hyperparameter Optimization in FL. Hyperparameter optimization is crucial in FL. While Wang etal. (2022) provides benchmark, other studies like Flora (Zhou et al. (202T) and (Zhou et al. 2022)
focus on hyperparameter initialization using single-shot methods. (Agrawal et al., 2021) clusters
clients by data distribution to adjust hyperparameters, and FedEx (Khodak et al. (020) employs
weight sharing for streamlined optimization. Some research, such as (Kan) 2022). targets specific
hyperparameters, while (Shi et al., 2022) dynamically changes the batch size during FL training.
Despite these efforts, comprehensive tool for hyperparameter optimization in FL lacking. Our work addresses this gap, offering enhanced versatility and practicality.
PROPOSED METHOD: FEDHYPER
Preliminaries on Hypergradient. In general machine learning, we aim at minimizing an em- pirical risk function F(w)= = + Σッ fi 8;), where w E Rd denotes the model weight and 8; denotes the i-th sample in the training dataset. The most common way to minimize this loss function to use gradient- based methods (Bottou 2010), the update rule of which is given as follows:
(1+1) (() ( FF
(1)
where (9) a time-varying scalar learning rate. The choice of v) significantly influences the training procedure. A greedy approach to set 77 to select the value that can minimize the updated loss (i.e., min, (1) 1/V F/ (t)()))) However, this is infeasible due to the need to evaluate countless possible values. Hypergradient follows this idea and allows the learning rate to be updated by gradient as well. Specifically, when taking the derivation, can get (F(w(t+))
Oww - 1 ((w((())))
0m(() =VF((((t+)))
(2)
On((1)
(+1) (())
(3)
Ideally, in order get (2) the current learning rate (1(-1) should be updated by -VF(w((+1)
((w) But this requires future knowledge (F(w)(+))) By assuming the optimal values of do not change much between consecutive iterations, (f-1)) is updated follows:
(+) - A
(4)
= (t- V (w (9) (t(w((-))).
(5)
(F(w(t-11) is defined as the hypergradient. This learning rate update rule is very intuitive, allowing adaptive updates based on the training dynamics. If the inner product (ww ()). (F(w(t-1) is positive, i.c., the last two steps' gradients have similar directions, then the learning rate will increase further speedup the good progress (see Figure [(a)). If the inner
where A ()))
product is negative, i.e., two consecutive gradients point to drastically diverged directions, then the learning rate will decrease mitigate oscillations (see Figure T(b).
3

---

Published as a conference paper at ICLR 2024
F(w)
p 1f p (F(WW())F(( p (-11)
F(w)
11 (1111) + VF(W 7 7 (-
VF 1) 17((11))
DF(w(t-1))-pp"
VF(W()) (m7(())
DF(W(())
Wop
w((+1)
w/ w((-1)
w
wopt w((+1) w(t-1)
(a) V(ww (F
(b) (" () <0
Figure 1: Two situations in hypergradient descent. Hypergradient descent adjusts the learning rate according to the inner product of gradients in the last two iterations.
Applying Hypergradient FL. In FL, the goal is to optimize a global objective function F(w) which is defined as weighted average over large set of local objectives:
M
1 M
F(w)
Σ
Fm
(w),
(6)
m=1
No. En fm (::: sn)) the local loss function of the m-th client. Due to privacy concerns, the server cannot access any client data and clients can neither share data with others. FEDAVG (McMahan et al. (017)) is the most popular algorithm to solve this problem. allows clients to perform local training on their own local dataset. central coordinating server only aggregates the model weight changes to update the server model:
where M is the number of clients and Fm (w) = 1
un (++1) (t)
(0
1
Σ
M
m
where (*) is defined the server learning rate, and A m is the local model changes. Moreover, the local update rule on each client is:
(t)
w (t,k+
i (0)
9m (((( un (1.0) (t.r) ( w m
w (1,8)
m
g(t.k)
(8)
(t,r)
m
where ((t.k) is the local learning rate, win (t,h) denote the local model at m-th client at the i-th global round and k-th local step, and wm (t.0) = (() is the initial model at each round.
Comparing the update rules of FEDAVG with vanilla gradient descent, the learning rate scheduling for FedAvg can be way more complicated. When applying the vanilla hypergradient idea on the server learning rate a, one will find that the global gradient F is not available at all; when applying the idea on the client learning rate, the derivative with respect g(t,k) nearly intractable.
In order to address these challenges, we develop FEDHYPER, a learning rate scheduler based on hypergradient descent specifically designed for FL. FEDHyPER comprises a global scheduler FEDHYpER--G G, server-side local scheduler FEDHYPER-SS SL, and client-side local scheduler FEDHYPER- CL. The general learning rates update rules are given below:
FEDHYPER-G: (1) = at (t--) A (t-1)
(10)
(t-1) FEDHYPER-SL:: g(t.0) = g(t--,0)_ -1-0
(11)
3,8
FEDHYPER-CL: ((t.)) = B(1,k-1)
A (1)
(12)
B,c
In the following subsections, we will specify the exact expressions of each hypergradient.
3.1 FEDHYPER-G: USING HYPERGRADIENT FOR THE GLOBAL LEARNING RATE
We first provide FEDHYPER- G to adjust the global learning rate a based on the hypergradient descent. According Eq. (1), the global learning rate for the /th round (!) should be computed by
4

---

Published as a conference paper at ICLR 2024
the last global learning rate () and global gradients in the two preceding rounds. Nevertheless, in accordance with the global model updating strategy in Eq. (7), the global model is updated by the aggregation of local model updates rather than gradients. Therefore, in FEDHYPER-G. we treat the aggregation of the local model updates the pscudo global gradient A
M
A
(4)
(
(13)
M
Then, we replace the gradients in Eq. (5) with global model updates and obtain the A (t-1) in Eq. (10) for the global scheduler as follows:
A (t-1) A (4) (t-1), (t-1) + A (9) (t-1)
(14)
(15)
In addition, ensure convergence and prevent gradient exploration, we clip (1) by preset bound value la in each round:
1 - }.7}
a = miinmmaxfor
(16)
Ya
Through experiments, we have found that Ya 3 yields consistent and stable performance across various tasks and configurations. Therefore, we generally do not need to further adjust /a-
3.2
FEDHYPER-SL. & FEDHYPER-CI. USING HYPERGRADIENT FOR THE CLIENT LEARNING RATE
FedHyper-SL. The server-side local scheduler adopts the same hypergradient as the global sched- uler, i.e., A (1-1) ((-1)) Thus, the local learning rates are updated by the server as follows:
g(t.0) (t00 A (t-1).
(17)
It adjusts all the selected local learning rates from the server synchronously. These local learning rates are also clipped by a preset bound 7/3:
1
g(t.0) = min (max
}. 788
(18)
The updated local learning rates g(+0) are sent to clients after t-th round and are used by clients in the f + 1-th round. Like FEDHYPER-G, we have experimentally set 79 to 10. This bound will also be used for g(t.k) in FEDHYPER-CL.
FedHyper-CL. Adjusting the local learning rate from the clients more fine-grained schedul- ing strategy since it adjusts learning rates for each local batch, while the server only performs such adjustments between communication rounds. This client-centric approach not only enhances con- vergence speed but also outperforms both FEDHYPER-C G and FEDHYPER SL in terms of accuracy. According to the hypergradient descent in Eq. (5) and local model update rule Eq. (8), the local learning rate for the k-th epoch client can be updated follows:
A
(t,k-1) 9.c = -
(m (w(t.k))
9m (w(f.k (t.k-))))
(19)
Nevertheless, directly applying Eq. (19) to local learning rates can lead to an imbalance in learning rates across clients, which may lead to convergence difficulty. To address this issue, and draw inspi- ration from our global scheduler, we incorporate global model updates to regulate the local learning rate optimization. Specifically, we augment A (f.k-1) by adding the component * 99m (w ( A where A t-1 is the global model update in the last round and is the number local SGD steps:
B.e
g(t.k) = g(t.k-1) + gm (mww (w(t,))) gm (w (t.k11)) + K 9m (m((w((k)) (t.(k))) (t-1)
(20)
The term Gm (w wise it is negative. The coefficient & acts to normalize the magnitude of gm gm (wol(k)) ().k))) (1-1) such
(w (2.k)) A (t-1) will be positive if gm (w (t.k)) and A have similar direction, other-
5

---

Published as a conference paper at ICLR 2024
Patala 22 18-1
Fle)
A11-1
Flv)
A0-1)
F0o
yFu
Beith (8
Rulle/ 21 11
(a) X > 0; Y > 0
(b) X < 0; Y <
(c) X 0;
Y
(d)
X < 0; Y >
Figure 2: Four situations in client-side local learning rate scheduling.
that it aligns with the scale of Im (wm (ww(h+)) 9m (wm (f.k ). This normalization is essential given that while Im (wnx is the gradient of single SGD step, A is the average over K local SGD steps. By integrating this term, we increase g(0,k) when the local model update is directionally consistent with the global model update; conversely, we decrease it when they exhibit significant discrepan- cies. It is pivotal to prevent certain clients from adopting excessively high learning rates that could potentially impact convergence.
Here we define X = Im (un (1))))) Im (Wm
K Im (wm (1.K)) A (t-1), The local learning
rate in the following four situations will be adjusted as: (1) X 0;
in Figure 2(a) ((4,k)
will increase; (2) X < Y;Y < 0 as in Figure 2(b) g(t.k) will decrease; (3) X 0; Y
< 0 in
Figure 2(c) g(r,A) will increase X YY; otherwise, it will decrease; (4) X < Y; 0 as in
Figure 2(d) g(t,A) will increase if X < |Y|; otherwise, it will undergo decrease.
The client-side local scheduler integrates insights from both global and local training dynamics. By leveraging global model updates, it steers the local learning rate optimization effectively. The full algorithm of FEDHYPER obtained Appendix C
CONVERGENCE GUARANTEE
In this section, analyze the convergence of FEDHYPER considering dynamic global and local learning rates. Our analysis demonstrate that FEDHYPER can ensure convergence given a sufficient number training epochs under some standard assumptions (Wang al. 2020b).
According to the unified convergence analysis in FEDNOVA (Wang et al. 2020b). the convergence of FL framework based the objective in Eq. (6) can be proved with some basic assumptions, which we list in Appendix A
In addition, we have the following upper and lower bounds for global and local learning rates:
Constraint 1 The learning rates satisfiy 7
(+) I %a and 70
((rR) 78. where Yas Y/ 1.
Theorem Under Assumption 1 to 3 in Appendix A and constraint above, deduce the opti- mization error bound of FEDHypER from the convergence theorem of FEDNOVA.
Q KT
min E V (ww(")) <0
+
(21)
[T
VMKT
where quantities P. Q are defined as follows:
Y002
P Σ
1)Tas
(22)
M
M Σ
1
k
[kk - 1>1 + 112
δ +
[kk [k- 1)73 +1}2
(23)
73
Where a and p are constants that satisfy 0², 0²,p² 2 0. The bound defined by Eq. (21), (22), and (23) indicates that when the number of rounds tends towards infinity, the expectation V ((w(()) tends towards O. We further provide comprehensive proof of Theorem in Appendix A

---

Published as a conference paper at ICLR 2024
5 EXPERIMENTS
We evaluate the performance of FEDHYPER on three benchmark datasets, including FMNIST (Xiao (ct al.} 2017) and CIFARIO Krizhevsky et al.) 2009) for image classification, and Shake- speare (McMahan al. 2017) for the next word prediction task. To evaluate the efficacy of adapting the global learning rate, we compare FEDHYPER
against four baselines: FEDADAGRAD (Reddi et al. 2020), FEDADAM (Reddi et al. 2020), FEDEXP (Jhunjhunwala et al., 2023) and global learn- ing rate decay with a 0.995 decay factor. We employ the across-round local learning rate step decay with also a 0.995 decay factor the baseline to compare with the server-side local scheduler FEDHYPER-ST As for client-side local scheduler FEDHYPER-CAL we opt for FEDAVG paired with local SGD and local Adam as the comparative baselines.
Experimental Setup. We partition FMNIST and CIFARIO dataset into 100 clients following a Dirichlet distribution with a 0.5 as presented in (n(ang et 2020a). We directly partition the inherently non-iid Shakespeare dataset to 100 clients as well. In each communication round, the server randomly selects 10 clients to perform local SGD and update the global model. For FMNIST, we utilize CNN model, while for CIFARIO, we deploy a ResNet-18 (He et al.) 2016). For the next- word prediction task on Shakespeare, we implement bidirectional LSTM model (Graves et al.) 2005). We set our global bound parameter To at 3 and the local bound 7/ at 10 in all experiments. In order to achieve global model convergence, we conduct 50, 200, and 600 communication rounds for FMNIST, CIFARIO, and Shakespeare, respectively.
5.1 FEDHYPER COMPREHENSSVVELY OUTPERFORMS FEDAVG AND BASELINES
Performance of FedHyper-G. As Figure 3(a) illustrates, the results underscore the superiority of FEDHYPER-G over the current global optimizers and schedulers in FL. Across all three datasets, FEDHYPER-6 G achieves faster convergence rates in the early training stage (e.g., rounds to 25 in FMNIST, rounds 0 to 80 in CIFARIO, and rounds 0 to 300 in Shakespeare) than all baselines. Moreover, FEDHYPER- G also maintains an equal or superior accuracy than baselines.
Such results corroborate our intuition in Section FEDHYPER can expedite the training process when the model remains far from the optimum and enhance the accuracy. Additionally, FEDHYPER- G increases accuracy by 0.45% 0.58%, and 0.66% compared to the best baseline in FMNIST, CIFARIO, and Shakespeare, respectively. This underscores the consistent performance improve- ment of FEDHYPER-C across varied tasks. Although FEDEXP shows comparable performance as FEDHYPER- in FMNIST and CIFARIO for image classification, FEDHYPER-G outperforms FEDEXP on Shakespeare dataset.
Performance of FedHyper--LL. We compare FEDHYPER-SL with FEDAVG and across-round local leamning rate decay, i.e., Decay-SL. As Figure 3(b) shows, FEDHYPER-ShP can potentially lead to comparable, if not superior, convergence speed and accuracy when contrasted with FEDHYPER-C G. This observation is particularly pronounced in the CIFAR10 and Shakespeare datasets. For in- stance, in the Shakespeare dataset, FEDHYPER-SS achieves 50% test accuracy approximately by the 150th round, whereas FEDHYPER- reaches similar accuracy near the 250th round. A plau- sible explanation for this might be that the local learning rate impacts every local SGD iteration, in contrast to the global scheduler, which only impacts the aggregation process once per round.
Performance of FedHHype--CL Figure 3(C) exhibits the efficacy of the client-side local scheduler. Not only does FEDHYPER-CCE CL outperform the baselines, but it also manifests superior performance compared our other two schedulers, as analyzed in Section 3 Specifically. in FMNIST and CI- FAR10, it outperforms FEDAVG with SGD and Adam, with improvements of 0.44% and 1.18%, respectively. Regarding the Shakespeare dataset, FEDHYPER--C cal SGD by 1.33%, though its performance aligns closely with that of FEDAVG with local Adam. Overall, FEDHYPER-CL consistently demonstrates commendable results across diverse tasks.
excels over FEDAVG with lo-
Additionally, FEDHYPER-CL also shows better performance compared to FEDHYPER-C and FEDHYPER-SL most notably in terms of convergence speed. For example, FEDHYPER-CL achieves 52.08% accuracy, converging approximately by the 100th round. In contrast, FEDRY G and FEDHYPER-SS -SL yield accuracies of 5 51.08% and 5 1.700 respectively, with a protracted con- vergence around the 300th round. It is noteworthy that FEDHYPER- -C. does incur a higher computa-
7

---

Published as a conference paper at ICLR 2024
FedHyper-B (ours)
FadAug
Decay-G
FedAdagrad
TMNIST
C3PAR1O
Shakespeare
Accuracy
67
40
04
86
40
Number of Rounds
Number
Rounds
(a) Performance of FedH-yper-G
FedHyper-SL (ours)
FedAvg
Decay-SL
Shakesspare
100
60
52
Test Accuracy (%)
04
54
88
46
0
10
20
30
40
50
80
120
160
200
100
200
300
400
600
600
Number of Rounds
Number of Rounds
Number of Rounds
(b) Performance of FedHyper-Sh
Fedityper-CL CL (ours)
Fed/wg-SGD SGD
Fedhugg-Adam
Shakespeare
100
80
25
Test Acsuracy (%)
40
Number
Number
Number Rounds
(c) Performance of FedHyper-CL.
Figure 3: Comparison of FedHyper with baselines. FEDHYPER consistently gives faster conver- gence compared baselines with superior performance.
tional overhead compared to the other two methods in FEDHYPER. Consequently, comprehensive discussion regarding the selection of three schedulers will be presented in the Appendix B
5.2 DISCUSSION AND ABLATIONS OF FEDHYPER
Robustness of F FEDHYPER Against Initial Learning Rates. As discussed in Section I identifying the optimal initial learning rates FL presents a significant challenge. FEDHYPER facilitates faster scheduling of the learning rates, showing its versatility to attain desired accuracy even in scenarios characterized by suboptimal initial learning rates, especially when contrasted with FEDAVG.
We first conduct experiments on CIFAR10 and Shakespeare to evaluate the robustness of FEDHY- PER against varied initial learning rates. Specifically, we employ five distinct initial global learning rates: 0.5, 0.75, 1, 1.5, and 2. Additionally, we also consider five separate initial local learning rates: 0.001, 0.005, 0.01, 0.05, and O. We first train CIFAR1O and Shakespeare using FEDAVG by apply- ing the aforementioned learning rates. The final accuracy is depicted in Figure 4(a) and Figure 4(c) The darker red shades represent higher final test accuracy.
Following the initial experiments, apply FEDHYPER in conjunction with both the global sched- uler and the client-side local scheduler, denoted as FEDH YPER-G + FEDHYPER- CL,, employing the identical initial learning rates as mentioned previously. The results are illustrated in Figure 4(b) and (dd) the shades green serve as indicators of the magnitude of accuracy augmentation, with the darker green hues representing more substantial improvements in accuracy.
8

---

Published as a conference paper at ICLR 2024
FodAgg
Fadityper
Fellag
Slukesseare Incrose Fedifyper
6.11
AM
19
63
20
To00a Louil Eaanting Tuab
0:33
Intcal Loobel Loansng Kiz:
Eniki Glabal Louning Rae
Leuming
buss Clichall camiing Rabi
(h)
Figure 4: FEDHYPER's robustness against diverse initial learning rates.
Feditypen- Server Momertum
FedAfg
Fedting Server Morrentum
Fedryypen- Fedkdam
FedAcam
CIRAR1O
CIBAR10
Shileespaare
30
40
30
45
40
180
100
200
203
300
000
FEDHYPER
(b) FEDHYPER FEDADAM
Figure 5: Integration of FEDHYPER with server momentum and FEDADAM.
The results in Figure 4 show that FEDHYPER improves accuracy by 1% 5% over FEDAVG for most initial learning rate configurations. Notably, FEDHYPER particularly effective at enhancing the accuracy of FEDAVG when it is faced with suboptimal initial learning rates. For instance, in the Shakespeare dataset (see Figure 4(c). FEDAVG excels with higher initial local learning rates (> 0.005), but underperforms suboptimal initial local learning rates like 0.001. In such scenarios, FEDHYPER 's ability to substantially boost accuracy, notably in cases with 0.001 initial local learning rate. When paired with 0.5 initial global rate, FEDAVG achieves 30% accuracy, whereas FEDHYPER raises it by almost 15% to 45.77%. The robustness of FEDHYPER against random initial learning rates can be visually demonstrated in Figure i.e., areas charac- terized by a lighter shade of red (i.e., indication of lower accuracy) in Figure 4(a) and Figure 4(c) transform into darker shades of green (representation of significant accuracy augmentation) in Fig- ure 4bb and Figure 4(d) The robustness manifested by FEDHYPER thus mitigates the necessity for meticulous explorations to initial learning rates within the FL paradigm.
(
Integrating FedHyper with Existing Optimizers. We evaluate the efficacy of FEDHYPER- G when is integrated with existing global optimization algorithms, i.e., server momentum (Liu al 2020) and FEDADAM (Reddi et al. 2020). The results in Figure S Jundersoree that FEDHYPER capable of improving the performance these widely-adopted FL optimizers. For instance, FEDHYPER- together with server momentum shows superior performance server momentum. Moreover, as Figure F debiictss when FEDHYPER- is paired with FEDADAM for the CIFARI0, there marked enhancement performance. These findings demonstrate that FEDHYPER possesses the versatility to serve not merely as a universal scheduler, but also as a universally applicable enhancement plugin that can elevate the capabilities of existing optimizers.
the combination of FEDAVG and
6 CONCLUSION
In this paper, we introduced FEDHYPER, universal and robust learning rate scheduling algorithm rooted in hypergradient descent. FEDHYPER can optimize both global and local learning rates throughout the training. Our experimental results have shown that FEDHYPER consistently out- performs baseline algorithms in terms of convergence rate and test accuracy. Our ablation stud- ies demonstrate the robustness of FEDHYpER under varied configurations of initial learning rates. Furthermore, our empirical evaluations reveal that FedHyper can seamlessly integrate with and augment the performance of existing optimization algorithms.

---

Published as a conference paper at ICLR 2024
REFERENCES
Shaashwat Agrawal, Sagnik Sarkar, Mamoun Alazab, Praveen Kumar Reddy Maddikunta, Thippa Reddy Gadekallu, Quoc-Viet Viet Pham, al. Genetic cft: Hyperparameter optimization in clustered federated learning. Computational Intelligence and Neuroscience, 2021, 2021.
Jonathan Barzilai and Jonathan M Borwein. Two-point step size gradient methods. IMA journal of numerical analysis, 8(1):141-148, 1988.
Atilim Gunes Baydin, Robert Comish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. arXiv preprint arXiv: 1703.04782, 2017.
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT 19th International Conference on Computational StatisticsParis France, Au- gust 22-27, 2010 Keynote, Invited and Contributed Papers, pp. 177-186. Springer, 2010.
Theodora S Brisimi, Ruidi Chen, Theofanie Mela, Alex Olshevsky, loannis Ch Paschalidis, and Wei Shi. Federated learning of predictive models from federated electronic health records. Interna- fional journal of medical informatics, 1 112:59-67, 2018.
Bozhou Chen, Hongzhi Wang, and Chenmin Ba. Differentiable self-adaptive learning rate. arXiv preprint 10. 10290, 2022.
Alex Graves, Santiago Fernánde7, and Jürgen Schmidhuber. Bidirectional Istm networks for im- proved phoneme classification and recognition. In International conference on artificial neural networks, pp. 799-804. Springer, 2005.
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv: 1.03604, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Fedexp: Speeding up federated averaging via extrapolation. arXiv preprint arXiv:2301 2301002300000004 2023.
Renlong Jic, Junbin Gao, Andrey Vasnev, and Minh-Ngoc Tran. Adaptive hierarchical hyper- gradient descent. International Journal of Machine Learning and Cybernetics, 37(1)):3785- 3805, 2022.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, al. Ad- vances and open problems in federated learning, Foundations and Trends in Machine Learning, 14(1-2):1-210, 2021.
Andrew K Kan. Federated hypergradient descent. arXiv preprint 1rXiv:2211.02/000 2022.
Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebas- tian Stich, and Ananda Theertha Suresh. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint arXiv:2008. 03606, 2020.
Mikhail Khodak, Tian Li, Liam Li, M Balcan, Virginia Smith, and Ameet Talwalkar. Weight sharing for hyperparameter optimization in federated learning. In Int. Workshop on Federated Learning for User Privacy and Data Confidentiality in Conjunction with ICML, volume 2020, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
10

---

Published as a conference paper at ICLR 2024
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv: wrXiv:190..0218, 2019.
Wei Liu, Li Chen, Yunfei Chen, and Wenyi Zhang. Accelerating federated learning via momentum gradient descent. IEEE Transactions on Parallel and Distributed Systems, 318):1754-1766 2020.
Agnes Lydia and Sagayaraj Francis. Adagrad an optimizer for stochastic gradient descent. Int. J. Inf. Comput. Sci, 6(5):566-568, 2019.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient- based hyperparameter optimiza- tion through reversible learning. International conference on machine learning, pp. 2113-2122. PMLR, 2015.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-eficieni learning of deep networks from decentralized data. In Artificial intelli- gence and statisties, pp. 1282-1282. PMLR, 2017.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.
Keith Rush, Zachary Charles, and Zachary Garrett. Federated automatic differentiation. arXiv preprint (rXiv:2301.07800 2023.
Dian Shi, Liang Li, Maoqiang Wu, Minglei Shu, Rong Yu, Miao Pan, and Zhu Han. To talk or to work: Dynamic batch sizes assisted time efficient federated learning over future mobile edge (011)):11088111
devices. IEEE Transactions on Wireless Communications, 21(12): 1038- 1050, 2022.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint arXiv:2202.0666 06440, 2020a.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611-7623 2020b.
Zhen Wang, Weirui Kuang, Ce Zhang, Bolin Ding, and Yaliang Li. Fedhpo-b: A benchmark suite federated hyperparameter optimization. arXiv preprint arXiv:2206.044 03966, 2022.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint 1XXiv1100.007444 2017.
Gang Yan, Hao Wang, and Jian Li. Seizing critical learning periods in federated learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 8788-8796 2022.
Kaichao You, Mingsheng Long, Jianmin Wang, and Michael I Jordan. How does learning rate decay help modern neural networks? arXiv preprint arXiv:1908 01878, 2019.
Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. A survey on federated learning. Knowledge- Based Systems, 216: 106775, 2021.
Yi Zhou, Parikshit Ram, Theodoros Salonidis, Nathalic Baracaldo. Horst Samulowitz, and Heiko Ludwig. Flora: Single-shot hyper-parameter optimization for federated learning. arXiv preprint arXiv:2112.08524, 2021.
Yi Zhou, Parikshit Ram, Theodoros Salonidis, Nathalie Baracaldo, Horst Samulowitz, and Heiko Ludwig. Single-shot hyper parameter optimization for federated learning: general algorithm & analysis. arXiv preprint
hyper-parameter
2022.

---

Published as a conference paper at ICLR 2024
A PROOF OF THEOREM 1: CONVERGENCE OF FEDHYPER
According to theoretical analysis of FEDNOVA (Wang et al. 2020b). an FL framework that follows the update rule in Eq. (13) will converge to a stationary point, and the optimization error will be bounded with the following assumptions:
Assumption (Smoothness). Each local objective function is Lipschitz smooth, that is. V (ar) VF v -||z vll. Vm E (., M
The stochastic gradient at each client is an unbiased estimator of the local gradient: Lg (<r {)] = V mm (2) and has bounded variance Fg (I {) F. (2) < o², Vm E {1,2, M 0² 2 O.
Assumption (Unbiased Gradient and Bounded Variance).
0 such that M 2m=1 M7 (x)]] < 6 | (z)(2) If local functions are identical to each other, 0.
Assumption (Bounded Dissimilarity). Existing constants 1/2 1. 2
then we have x²
where we adhere to our setting that each client contributes to the global model with the equal weight 1 Then we can rewrite the optimization error bound as follows:
(1))
Ao²
MBo²
MMCp²
min B V SVVPuu |1 0o
+0
+
+0
(24)
VMT
VM&T
KT
KT
Where A, B, and C are defined by:
M
1aml
M
A = (t)
Σ
B
Σ (I
113 mm,,m C = =max]] am 11- |I am -). (25)
M
M
m=m
where Om is a vector that can measure the local model update during local SGD, where the number of k-th value of am is Om [k]
g(0.3
Note that we have Bound 1 on global learning rate that ( min maaimmxxaa" J Ja ). we have
the upper and lower bound for (!) as follows:
S (t)) S Yas
(26)
Ya
73 For the local learning rate, have g(t,)) min (max [ (,(t.h) 1 78). Therefore, the maximum gis.k)
value of ratio fit.oy is 7/. when g(t.)) = 7a. and g(t.0) 1. Accordingly, the minimum of glc.) 75" We can derive the upper and lower bound also for ||am]] II and ||amll2 as follows: 73
2 < am,k S 7,00
7/
k 1
+ 1
< ||am|l |l2 S (k: - 1)7g
(27)
1
+
1
S
||am|| 1|2 S (k - 1 +
7/3
|mm 112
| 1
<
2
Then, we apply Eq. (26) and (27) to the first item of Eq. (25), and get:
o
(1))
50
%a
(28)
VMKT
VMKT
12

---

Published as a conference paper at ICLR 2024
Then, apply Eq. (26) and (27) to Eq. (25) and redefine A B and C:
am 11
A a
M
am
M
am
7/x
M amm
||
(29)
11
M
S 7/x
M
m==
M
B =
Σ m=1
(1 [I|m 112 -mm,-1)
M
(30)
[[(A: 1)7 + 112
M
7
m=1
C = max
11 - -1|am am,
k
(31)
< I 1)7 + 11²
1).
2
2
73
Then, we can combine the first and second items of Eq. (24) and get the new bound:
P
Q
min
so
+
(32)
(E(T]
MET
KT
where P is defined by:
No + Ao
M
002²
(33)
+ 1)Ya.
M
m=1
and Q is defined by:
Q = MBo² + MCp²
Σ
M
[[(ks
1)7 + 1
10² 102 + MM 1)78
-
1
112
1
k
+
I]].
(34)
7/
78
73
m=1
Note that we use the upper bound of A, B, C here. Now we have completed the proof of Theorem
B SUPPLEMENTARY EXPERIMENTAL RESULTS
B. LEARNING RATE CURVE OF FEDHYPER
As we analyzed in Section FEDHYPER adjusts the learning rates in way that the learning rates increase in the former training stages and decrease in the latter stages. It also aligns with our analysis of the relationship between the convergence and the value of learning rates in Figure [ To illustrate this point, we visualize the change in global learning rate through 0- 0-100 epochs in Figure 6 The global learning rate of FEDHYPER- increases from round 0 to 15, and starts to decrease. The value is greater than in the first 50 epochs and less than in the following 50 epochs. As for FEDEXP, the global learning rate value fluctuates between 1.0 and 1.5.
13

---

Published as a conference paper at ICLR 2024
FedHyper-G FedExp
Global Learning Rate
1.4
Men Whmn WM NN Mosaan
1.2
1.0
0.8
0
20
40
60
80
100
Communication Rounds
Figure 6: Comparison of global learning rate curve between FEDHYPER and FEDEXP.
B.2 THE TIME OVERHEAD OF FEDHyPER
FEDHYPER involves utilizing the inner product of gradients modify the learning rate. It may raise concern that FEDHYPER incurs additional computational expenses compared to the more basic FedAvg. Here claim that hypergradient computing in FEDHYPER only involves small amount of time overhead, which trivial compared to the original time consumption FEDAVG. To prove this, we have included an analysis of the time cost of our three schedulers and baselines. We ran the experiments on time cost of our three schedulers and baselines. The following results in Table the time cost (in seconds) of different optimization algorithms when training CIFARIO for 200 communication rounds:
FedHyper-G
FedHyper-CL.
FedAvg
FedExp
FedAdam
FedAdagrad
44060
45780
44031
45193
44045
43917
Table 1: Total time consumption in second of FEDHYPER and baselines
Different global schedulers have similar running times, however, the schedulers in FEDHYPER can reach convergence faster than baselines, so we can have less cost than our baselines ultimately. Specifically, FEDHYPER-CCI only increases less than 5% computation cost compared with FE- DAvG(SGD) but gets convergence up to 3 times faster than SGD. Therefore, FEDHYPER's faster convergence offsets this slight increase in computation time.
B.3 THE IMPACT OF NON-IID LEVEL
The data distribution on clients also affects FL performance. In Table we display the final accuracy of global models with FEDAVG and FEDHypeR in iid and non-iid data, and also different in non- iid Dirichlet distribution. We can conclude from the table that FEDHYPER contributes more to non- iid settings, especially with relatively small numbers. The accuracy increase of a 0.25 is 0.80% in FMNIST and 1.55% CIFAR1O, which the highest among the three a values. This might be attributed to the client-side local scheduler designed that adopts the global updates to restrict the increasing of local learning rates on some clients that might hinder the global convergence because of the data heterogi.
14

---

Published as a conference paper at ICLR 2024
FMNIST
CIFAR10
IID
0.75
0.5
0.25
IID
0.75
0.5
0.25
FedAvg FedHyper
98.62%
96.35%
96.36%
97.00%
67.14% 67.45%
56.71%
57.14%
53.55%
98.92%
97.03%
96.97%
97.80%
57.18%
58.42%
55.10%
Table 2: Accuracy FedHyper in different values
FedHyypr-GGtCL
FedHyper-G
FedHyper-CL
CIFAR10
Shakesseaae
60
53
51
49
47
45
Figure 7: Cooperation of FEDHYPER-O and FEDHYPER-
B.4 COMBINATION OF FEDHYPER-G AND FEDHYPER-CL
We show the results of FEDHYPER-G. FEDHYPER SL, and FEDHYPER- CL work alone in Figure and show that they both outperform baselines that optimize the global local training from the same dimension (e.g. hoth work on the server). However, FEDHYPER has another advantage over baselines, that is, the ability adjust both global and local learning rates in training process. To support this, we run experiments on CIFAR10 and Shakespeare by applying both FEDHYPER-C and FEDHYPER-CL. called FEDCYLLR-G++L We display the results in Figure and compare with FEDHYPER-GG and FEDHYPER-CLE The results show that FEDHYPER-GE G+CL L is still able to outperform both of the single adjusting algorithms, which indicates that the performance of FED- HYPER algorithms can superpose each other. This makes FEDHyPER more flexible and can be suitable for different user needs. Here we not show the results of combining FEDHYPER-C and FRDHYPER-SL. because they use the same hypergradient to adjust different learning rates. So the superposition effect not obvious.
B.5 How TO USE FEDHYPER IN REAL FL PROJECTS
We have three schedulers in FEDHYPER framework. However, not all of them are needed in projects. We highly recommend FL trainers select suitable algorithms according to their needs and budgets. Here we provide some suggestions on the algorithm selection in specific scenarios.
FedHyper-G only when the trainer has tight budget of computational resources on clients, e.g., when performing FL on edge devices, mobile terminal devices, or low-memory GPUs.
FedHyper-Sh only has similar scenario with FEDHYPER-G only. One thing difference is that it adds some extra communication costs in sending the local learning rates. Therefore. if the trainer does not have bottleneck in communication cost, she choose freely between FEDHYPER-C only and FEDHYPER- only while considering the features of the specific task (i.c., more sensitive global or local learning rates).
FedHyper-CL. only when the trainer has a tight budget of computational resources on the server but loose budget clients, e.g., FL service providers.
FedHyper-G and FedHyper-CL. when the trainer has loose budgets of computational resources on both server and clients, e.g. distributed large model training.
15

---

Published as a conference paper at ICLR 2024
Algorithm Workflow of FEDHYPER
Input: Initial Global Model W Number of Communication Rounds T, Number of Selected Clients each Round M, Initial Global Learning Rate a Initial Local Learning Rate each Round 8 3² 38 = = g Local Epoch number K, Local batches ;
Output: Trained Global Model W
1: for t in 0, ..... T 1 do
2: Server send W and 3 to all selected clients.
Clients: FedHyper-CI.
3: Compute global model update A W W
4: 1 = .
5: for k 0,.,...
do 6: Compute 9m (W on Wik and E.
7:
Update local learning rate: BLk
Br,k-1 + gm (Wrth)) Im (W (1 + E
8:
Clip: 3 minfmax (,8h, "g 1, 793
9:
Local model update: W2k+1 W 5t. Im (Wi,)))
10:
end for
11:
Send A W W to server.
Server:
FedHyper-G
12:
Compute global model update: A EP, At
13:
Update global learning rate: a² = a
A t-1
14:
Clip: a₁ min (max (at, 7a }.7a
15:
Update global model: W W a² A
end for
We do not encourage other combinations because we do not observe an obvious performance im- provement on them. We believe that FEDHYPER- + FEDHYPER- CL can achieve the best perfor- mance in our framework if the trainer has generous budget.
C ALGORITHM OF FEDHYPER
We obtain full FEDHyPER algorithm and show the cooperation of FEDHYPER-G and HEDHYPER-CL in Algorithm [ Note that FEDHYPER-ST uses the same hypergradient as FRDHyPER-C G so it not applied in order to simplify the algorithm.
16


Reviews:
[{"review_id": "nflBAOLWQN", "reviewer": "Reviewer_MWyj", "summary": "This work extends the traditional hypergradient learning rate scheduler to federated learning scenarios. Specifically, the authors propose a novel theoretical framework that jointly considers the global and local learning rates with respect to global and local updates. The authors have conducted sufficient experiments to validate the performance.", "questions": "FedHyper-SL and FedHyper-G appear to update different sets of hyperparameters using the same gradients. It would be beneficial if the authors could provide a more detailed comparison between these two variants, considering that FedHyper-G seems to be a special case of FedHyper-SL.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well written. The authors thoroughly reviewed the previously related work, hypergradient, and discussed the limitations of the work in detail. Then the authors naturally extended the method into federated learning scenarios.\n\n2. The proposed method is intuitive, and the theoretical guarantee is solid.\n\n3. The authors have conducted sufficient experiments to validate the proposed methods and discussed the suitability of several variants under different hardware constraints.", "weaknesses": "The proposed method is only evaluated on simple datasets and tasks, such as CIFAR-10 and FMNIST, and tested with small models. The hyperparameter choices for these simple scenarios would be relatively straightforward, particularly in the case of standard centralized training. It would be beneficial if the authors could test the performance improvements on more challenging cases, such as unbalanced or large datasets, or in fine-tuning settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3RtIVCHBxp", "reviewer": "Reviewer_aiMM", "summary": "This paper aims to introduce a structured approach to learning rate scheduling within the realm of Federated Learning, a critical step that enhances efficiency, particularly in the initialization phase. The authors present FedHyper, a framework that leverages hypergradient methods to adjust learning rates based on the inner product of gradients. FedHyper encompasses three distinct schedulers for comprehensive application: a global scheduler for the server-side learning rate, a server-side local scheduler, and a client-side local scheduler. Through rigorous testing on three datasets, FedHyper demonstrates superior performance compared to state-of-the-art baselines.", "questions": "See the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- By targeting a pivotal issue in Federated Learning, the study positions itself within a crucial niche. The focus on optimizing learning rate scheduling addresses a substantive bottleneck in the field, underlining the paper's relevance.\n- The hypergradient method is interesting and effective, and the proposed method could be applied to other hyperparameters as well. Simplicity and practicality are the hallmarks of the proposed scheduler, making it an attractive tool for real-world application. Its ease of use could significantly benefit practitioners in the field.\n- The convergence of FedHyper is theoretically proofed, making this work more sound.\n- The results from extensive experiments demonstrate the significant performance improvement in both convergence rate and final accuracy.\n- The paper excels in clarity and accessibility, effectively illustrating the mechanics of hypergradient descent in learning rate scheduling through well-conceptualized figures (e.g., Figures 1 and 2).", "weaknesses": "- The paper's primary methodology involves utilizing the inner product of gradients to modify the learning rate, a technique that potentially incurs additional computational expenses compared to the more basic FedAvg. The study falls short by not evaluating or discussing these potential overheads, leaving the reader uncertain about the practical trade-offs.\n- In FedHyper-CL, the author’s statement “directly applying Eq. (19) to local learning rates can lead to an imbalance in learning rates across clients” is not well supported by analysis or reference. So, the motivation of adding an item in FedHyper-CL is not clear.\n- The paper does not provide the direct comparison with FedAdam.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BUwp2CdtfZ", "reviewer": "Reviewer_EDET", "summary": "This paper introduces FedHyper, a novel learning rate scheduler adept at managing both global and local learning rates. FedHyper distinguishes itself by accelerating convergence and automating learning rate adjustments. To validate FedHyper's efficacy, the authors perform thorough experiments on benchmark datasets, offering a comprehensive evaluation of its performance.", "questions": "Please refer to Weakness.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1) The paper addresses a significant issue in Federated Learning (FL) by emphasizing the pivotal role of learning rate scheduling. It successfully argues the necessity for meticulous attention in this area and proposes pragmatic solutions, thereby contributing tangibly to advancements in FL.\n2) The adoption of the hypergradient method within this context is noteworthy. Its proven effectiveness in FL not only reinforces the method's utility but also broadens its appeal, suggesting it could be beneficial in a variety of other areas. This aspect of the paper stands out as particularly insightful.\n3) The experimental framework of the paper is commendable for its solidity. By utilizing benchmark datasets, the research provides ample evidence to support the stated contributions of FedHyper. This rigorous approach to experimentation substantiates the claims made, enhancing the paper's credibility.", "weaknesses": "1) One concern is the lack of empirical evidence supporting the client-side scheduler's strategy of employing a global model update to limit the growth of the local learning rate. The paper would greatly benefit from an ablation study to confirm the validity of this approach, ensuring that the strategy is both necessary and effective.\n2) The analysis seems incomplete when it comes to comparing FedHyper with FedAdam. The latter, considered a baseline, is conspicuously missing from the \"performance of FedHyper\" section after being included in the cooperation analysis. The reason for this omission is unclear, and it restricts a full understanding of how FedHyper stands against established methods.\n3) The paper falls short in explaining the underlying reasons behind FedHyper's superior ability to fine-tune learning rates compared to its contemporaries, such as FedExp. The missing intuitive rationale or clear justification leaves the reader questioning why FedHyper is ostensibly more efficient. Addressing this would make the advantages of FedHyper more transparent and convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M9PothyLjX", "reviewer": "Reviewer_xGw7", "summary": "This paper proposes FedHyper, which adopts Hypergradients to dynamically adjust learning rate for global learning rate (LR), server-side local LR, and client-side local LR. The authors demonstrate the effectiveness of FedHyper on various datasets and models, including image classification and language modeling tasks. FedHyper is shown to outperform other FL learning rate scheduling algorithms in terms of convergence rate and final accuracy on various datasets and models.", "questions": "- Why are the baseline methods different for the three settings in Figure 3?\n- Could you please provide a more detailed and intuitive explanation of hypergradients? I’m looking for a thorough understanding, and I’m willing to give a higher score for a comprehensive explanation.\n- Is the learning rate fixed for baseline methods like FedAvg?\n- Since FedHyper requires updating the learning rate using current and previous time gradients, does it incur significantly higher computational and resource costs compared to other baseline methods?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and easy to understand, with clear explanations and well-designed experiments. FEDHYPER is a versatile algorithm that can seamlessly integrate with and augment the performance of existing optimization algorithms. This versatility makes it a valuable tool for researchers and practitioners working in the field of FL.", "weaknesses": "The proposed FedHyper highly relies on the existing method, i.e., hypergradient. It's unclear why the learning rate can be updated following Eq. (4-5). It would be better to give the intuitive explanation for this. FedHyper does not discuss the issue of training cost in FL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)