Federated Hypergradient Descent
Authors: Andrew K Kan
Venue: NeurIPS 2022 Submitted (2022)
Keywords: federated learning
hyperparameter tuning
hypergradient descent

Abstract: In this work, we explore combining automatic hyperparameter tuning and optimization for federated learning (FL) in an online, one-shot procedure.  We apply a principled approach on a method for adaptive client learning rate, number of local steps, and batch size.  In our federated learning applications, our primary motivations are minimizing communication budget as well as local computational resources in the training pipeline.  Conventionally, hyperparameter tuning methods involve at least some degree of trial-and-error, which is known to be sample inefficient.  In order to address our motivations, we propose FATHOM (Federated AuTomatic Hyperparameter OptiMization) as a one-shot online procedure.  We investigate the challenges and solutions of deriving analytical gradients with respect to the hyperparameters of interest.  Our approach is inspired by the fact that all components involved in our training process are open-boxed, and this fact can be exploited in our algorithm impactfully.  We show that FATHOM is more communication efficient than Federated Averaging (FedAvg) with optimized, static valued hyperparameters, and is also more computationally efficient overall.  As a communication efficient, one-shot online procedure, FATHOM solves the bottleneck of costly communication and limited local computation, by eliminating a potentially wasteful tuning process, and by optimizing the hyperparamters adaptively throughout the training procedure without trial-and-error.  We show our numerical results through extensive empirical experiments with the Federated EMNIST-62 (FEMNIST) and Federated Stack Overflow (FSO) datasets, using FedJAX as our baseline framework.

24 Federated learning (FL) for on-device applications has its obvious social implications, due to its inherent privacy-protection feature. It opens up broad range of opportunities to allow massive number of devices collaborate in developing shared model by retaining private data on the devices. The ubiquity of machine learning (ML) on consumer data, coupled with the growth of privacy concerns, has pushed researchers and developers to look for new ways to protect and benefit end-users. In order for FL. to deliver its promise in deployed applications, there are still many open challenges remained to be solved. We are especially interested in the overall communication efficiency of the FL pipeline for it to be realistically deployed in unique communication environment over expensive Federated Hypergradient Descent
Anonymous Author(s)
Affiliation
Address
email
Abstract
In this work, we explore combining automatic hyperparameter tuning and opti- mization for federated learning (FL) in an online, one-shot procedure. We apply principled approach
2
method for adaptive client learning rate, number of local steps, and batch size. In our federated learning applications, our primary motivations are minimizing communication budget well local computational resources the training pipeline. Conventionally, hyperparameter tuning meth- ods involve at least some degree of trial-and-error, which is known be sample inefficient. In order to address our motivations, we propose FATHOM (Federated AuTomatic Hyperparameter OptiMization) as a one-shot online procedure. We investigate the challenges and solutions of deriving analytical gradients with respect the hyperparameters of interest. Our approach is inspired by the fact that we have full knowledge of all components involved in our training process, and this fact can be exploited in our algorithm impactfully. We show that FATHOM is more communication efficient than Federated Averaging (FedAvg) with optimized, static valued hyperparameters, and is also more computationally efficient overall. As communication efficient, one-shot online procedure, FATHOM solves the bottleneck of costly communication and limited local computation, by eliminat- ing potentially wasteful tuning process, and by optimizing the hyperparamters adaptively throughout the training procedure without trial-and-crroos We show our numerical results through extensive empirical experiments with the Federated EMNIST-62 (FEMNIST) and Federated Stack Overflow (FSO) datasets, using FedJAX our baseline framework.
4
6
10
11
12
13
14
15
17
19
21
Introduction
23
25
26
27
28
29
30
31
32 links. To begin, consider a typical step in a machine learning (ML) pipeline: hyperparameter tuning. Whether it is in centralized, distributed or federated setting, it is an essential step to achieve an optimal operation for the training process. At the heart of an ML training process is the optimization interested in using Federated Averaging (FedAvg) our baseline
34
algorithm. In particular,
Submitted 36th Conference Neural Information processing Systems (NeurIPS 2022). Do distribute.

---

36 federated optimization algorithm for our work. This is because, despite all the recent innovations in FL since its introduction 2016 by McMahan al. (0166. FedAvg remains the de facto standard in federated optimization for both research and practice, due to its simplicity and empirical effectiveness.
37
38
39 In order for FedAvg to operate effectively, requires properly tuned hyperparameter values.
40
Our work focuses specifically on hyperparameter optimization (HPO) of: 1) client learning rate, 2) number of local steps, well as 3) batch size, for FedAvg. We propose FATHOM (Federated AuTomatic Hyperparameter OptiMization), which is an online algorithm that operates as a one-shot procedure. In the rest of this paper, we will go through few notable recent tfat-oof-the-a art works on this topic, and make justifications for our new approach. Then we will derive a few key steps for our algorithm, followed by a theoretical convergence bound for adaptive learning rate and number of local steps in the non-convex regime. Lastly, we present numerical results on our empirical experiments with neural networks on the FEMNIST and FSO datasets.
41
42
43
44
45
48
47
48
Our contributions are as follows:
49
We derive gradients with respect to client learning rate and number of local steps for FedAvg. for an online optimization procedure. We propose FATHOM, a practical one-shot procedure for joint-optimization of hyperparameters and model parameters, for FedAAv.
50
51
We derive new convergence upper-bound with a relaxed condition (see Section A and remark D to highlight the benefits from the extra oogree--f-freedon that FATHOM delivers for performance gains.
59
54
55
We present empirical results that show tfth-offtheeart performance. To our knowledge, we are the first to show gain from an online HPO procedure over a well-tuned equivalent procedure with fixed hyperparameter values.
56
2
Related Work and Justifications for FATHOM
58
59
We explore the question whether the FATHOM approach is justified over the more recent, state-of- the-art methods that are designed for the same goal: a single-shot online hyperparamter optimization
60
61
procedure for FL. Zhou et al. [2022] proposed Federated Loss SuRface Aggregation (FLoRA), a
82
general single-shot HPO for FL, which works by treating HPO as black-box problem and by performing loss surface aggregation for training the global model. Khodak et al. [222]] draws inspiration from weight-sharing Neural Architectural Search (Pham et al. [0018 ,, Cai et al. (2019). and proposed FedEx, which is an online hyperparameter tuning algorithm that uses exponentiated gradients to update hyperparameters. On the other hand, Mostafaa [2019]s ' RMAH and Guo et al.
63
64
65
66
2022|² 's Auto-FedRL. both use REINFORCE (Williams 1992] in their agents to update hyperparam- 67
68
eters an online manner, by using relative loss as their trial rewards. One basic assumption among these methods, is that at least some of the gradients with respect to the hyperparameters are unavail- able directly. Generalized techniques are used to update these quantities, involving Monte-Carlo sampling and evaluation with held-out data. One key benefit with techniques such as these is their generalizability for wide range of different hyperparameters. On the other hand, we identify a few areas with these methods that we would like to improve on. One, information about the internals of
70
71
72
74 the procedure can and should be exploited. Two, communication overhead becomes a concern, since sufficient Monte-Cart sampling is required for some of these techniques to converge, an example
76 being the re-parametrization trick (Kingma and Welling (2013) which is used for FedEx, RMAH and Fedo.F From initial observations of their empirical results, while these methods
successful 78 in hyperparameter tuning and reaching target model accuracy as shown in these works, these goals are achieved in unspecified numbers of total communication rounds from works based on RL approaches
79
80
such as Mostafa| (219)] Guo et al. [2022].
81
The above observations justify exploring our problem differently from previous approaches. Our method exploits full knowledge of the training process, and does not require sufficient trials at potential expense of communication budget. Inspired by the hypergradient descent techniques
82
83
84 developed by [Baydin et al. (017]] and Amid et al. [2022] for centralized optimization learning rate, develop FATHOM by directing deriving analytical gradients with respect the hyperparameters 86 of interest. The result is a sample efficient method which offers both improvements in communication efficiency and reduced local computation in single-shot online optimization procedure. Meanwhile, FATHOM is not as flexibly applicable in optimizing a wide range of hyperparameters, since each
88

---

89
gradient needs to be derived separately to take advantage of our full knowledge of the training process. We believe this approach a performance advantage, at the expense of its flexibility.
90
81
There are other notable relevant works. Charles and Konečný 2020 and et et al. [2019] proved that
reducing the client learning during training necessary reach the true objective. Yet, line of
93
interesting works, such as Dai et al. [2020] and Holly et al. [202T] applies Bayesian Optimization
BO) on federated hyperparamter tuning. by treating it as closed-box optimization problem. Dai 94
cl al. 202T] further updates their use of BO in FL by incorporating differential privacy. However, 85
these BO-based works do not consider adaptive hyperparameters. Yet, another work <Wang and Joshi 96
97
2018 ) shares similarity to our approach of optimally adapting the number of local steps, with their adaptive communication strategy, AdaComm, in the distributed setting. However, their main interest is reducing wall-clock time. Lastly, around the same time of this writing. [Wan et al. [2022] publishes
96
99
100
their henchmark suite for FI. HPO, called FedHPO-B, which would he valuable to our future work.
Methodology
101
In this section we formalize the problem of hyperparameter optimization (HPO) for FL. We first review FedAvg, a de facto standard of federated optimization methods for research baseline and practice. Then, we present our method for maline-tuning of its hyperparameters, specifically client learning rate, number of local steps, and batch size. We call our method FATHOM (Federated AuTomatic Hyperparameter OptiMization).
103
105
3.1
Problem Definition
108
In this paper, we consider the empirical risk minimization (ERM) across all the client data, as an unconstrained optimization problem:
= min f(x) = (x) zERd
(1)
where ʃa R I is the loss function for data stored in local client index with d being the dimension of the parameters x, m is number of clients, and f = f(x,) where v, is a stationary solution to the ERM problem in eq
112
113
To facilitate some of the discussions that follow, it helps to define assumptions here as we do throughout the rest of this paper:
115
Assumption 1. (Unbiased Local Gradient Estimator) Let gx(x) be the unbiased, local gradient estimator of f(()), i.e., [xg] ff()), Vx, and E [m].
116
117
3.2
Federated Optimization and Tuning of Hyperparameters
118
Federated Averaging (FedAvg) We describe the operations of FedAvg from McMahan et al.
119
2016 as follows. At any round t, each of the m clients takes a total of K; local SGD steps, where K, Lv,/B, and where 14 is the number of data samples from client index i, B is batch size,
120
121
with epoch number E being a common baseline. In this version of FedAvg, heterogeneous data size is accommodated across clients, and the number of local steps can be manipulated via E and
122
123
B as hyperparameters. Each local SGD step updates the local model parameters of each client as follows: 21, . '/r.9i (xf ). where 1jr. is the local learning rate and A: E [K] is the local step index. To conclude each round, these clients return the local parameters æi,K, the server where it updates its global model, with Ii+1 = E, M, fyxl/x/ /vv where v = Σ, 14. To facilitate some of the discussions that follow, we define the following quantities:
125
126
K
A, L++1
Ft
Σ
A
where
A
Offline Hyperparameter Tuning Offline tuning is best to be summarized as follows. We first E V. We also define C. U x V. and (nL, K), where e C. Offline tuning would have the following objective:
define = (u u 2 0) with m) E U, and V {u E I 2 1} with
129
3

---

131
mincec (za c) s.t. x = argmingges :ER fusin(2, c). With abuse of notation, we use fralid for the objective function calculated from a validation dataset which is usually held-out before the procedure, and simin for the objective from training data which usually is just local client data. A few notable
132
133
134
offline tuning methods are as follows. Global grid-search from Holly et al. 202T] is an example
135
of offline tuning that iterates over the entire search grid defined as C, completing an optimization
136
process for each grid point and evaluating the result with a held-out validation set. Global Bayesian
Optimization from Holly et al. [202T] is another similar example of offline tuning that follows the
138
same template and objective. Instead of brute-force grid-search, c is sampled from a distribution De
over C, ie. Dc, that updates after every iteration.
140
Online Hyperparameter Optimization We are interested in an online procedure that combines hyperparameter optimization and model parameter optimization, with the following objective:
141
min fuain (22, c)
ZER
cEC
142
This formulation is the objective of our method, FATHOM, which we will discuss shortly in detail. It has the advantage of joint optimization in one-shot procedure. Furthermore, it does not assume the availability of a validation dataset.
144
145
3.3
Our Method: FATHOM
146
In this section we will introduce our method, FATHOM (Federated AuTomatic Hyperparameter OptiMization).. Recall from our joint objective, eq that both the model parameters, x, and hyperparameters of the optimization algorithm, are optimized jointly to minimize our objective function. An alternative view is treat as part of the parameters being optimized in a classic formulation, i.e. minyf(y) with y = (I, c). As previously mentioned, our method is inspired by hypergradient descent from Baydin et al. [2017] and by exponentiated gradient from Amid al. 2022). both proposed for centralized learning rate optimization. We will present how FATHOM exploits our knowledge of analytical gradients to update client learning rate, number of local steps, as well as batch size, for an online, one-shot optimization procedure.
147
148
149
150
151
152
153
154
Assumption 2. (Convexity w.r.t. πL and K) We assume (z() is convex w.r.t TL and K, even though we assume non-convexity w.r.t. Vz). Specifically, convexity w.r.t. K follows the definition in Murota 1998), accommodate the inleger space where K is defined.
156
157
158
Remark Assumption is necessary to guarantee the existence of subgradients derived in Theorems
159
7 a and it will be assumed for this work. In problems dealing with deep neural networks, it is reasonable to not assume convexity w.rl. hyperparameters. However, from our empirical results, we claim that the proposed algorithm is still able to operate as desired under this condition.
160
162
3.3.1
Hypergradient for Client Learning Rate
163
In this section, we derive the hypergradient for client learning rate in similar fashion as [Baydin et al. 2017]. with the difference being that they are mainly concerned with the centralized optimization problem, and that are concerned with the distributed setting where clients take local steps. We
184
165
186
derive the following hypergradient of the objective function as defined in eq taken with respect to the learning rate r such that it can be updated to obtain n..
187
(x)) (f.)) 0 + A.-1)
O
H TTL,L-1
Vz)))
(4)
dx,
168
where A, is the update step for the global parameters as defined in eq leading Tqn
03,
K
169
> 24 Σk= 9: (x We also make the approximation 12+1 32
A. - V f1x). We
170
can then write the normalized update, H., similar to Amid et al. 2022
as follows:
H. = fx(x))) (0ă.1//
(f()))
0A,-1
A.
A,-1
(5)
|
171
The resulting hypergradient is scalar, as expected, and can be used efficiently as part of the update 172 rule for 7/r., which we will see in Section B.3.4] The implementation is communication efficient, since
173 in each round, each client needs one extra scalar to send back to the server, and likewise the server
174 needs to broadcast one extra scalar back to the clients. It is also computationally efficient since
avoids calculating the full local gradient V (x:).

---

178
3.3.2
Hypergradient for Number of Local Steps
177
Since the number of local steps is an integer, i.e. K (k E k 2 1), this means f(z) does not exist for non-integer values of K. We formulate a subgradient as a surrogate of the hypergradient
178
as (x:)/OKK as follows. We will call this a hyper-subgraddent.
Theorem 1 When a piecewise function L defined for every value of Kg E [K] on such that
180
0.0 < < 1.0, we claim, under Assumption 2
that the following is a subgradient of f(r) at
182
K, Ko:
@h, (-)(())-(-)))) _
m
(x
(6)
v
where I represents the marginal fraction of local steps beyond Ku- We leave the proof (with illustration in Figure 2 in the Appendix section beginning in eq
184
185
The result from Theorem I is not sufficiently communication-cfficiee for implementing an update rule for K. This is because it would require the quantity s==""=1=1) to be communicated from each client to the server. To save communication, let us reuse what the server has in memory:
187
188
A.
7r. > m
1
Me E
K1-1 (x(x"")) If we let:
m
S. = Vs(()) (- VT
gi (
i=1
0
K,
0s:
N,
= vf(z)) ( 1T
(: 1) = ff(zz)) A.-1
(8)
0l
2m1
k=0
Vs(r))
Ar-
A
A
N. [V)())
A
A
AA
189
where eq is the normalized update as in Amid et al. (002). We claim that eq is positively- biased version of eq which has its practical importance due the fact that the last term in eq from Theorem resuulss in zero-mean, noisy gradients, when the local functions are nearing their local solutions, when fact, this the area where more local work is needed. Thus, a positive bias is
181
192
193
desirable to drive the number of local steps down. This result is also useful from a communication efficiency perspective its implementation, because the server has all the components to calculate this quantity, and would not require additional communication.
194
195
3.3.3
Regularization for Number of Local Steps
One of the goals for FATHOM is savings in local computation. To avoid excessive number of local steps, we further develop a regularization term for local computation against excessive K, which is a
198
proxy for the hypergradient of the local client functions at the end of each round : Off((z K )/OK.
199
200
Theorem 2. When a piecewise function Jr is defined for every value of Ko E [K] on such that
0.0 < < 1.0, we cluim, under Assumption that the following is a subgradient of Em (ek
201
at K, Ko:
0J: al WeAt
m
E[ga(s)
E[ga(s;
go (et
gi(2)
(i) (10)
203
where represents the fractiono of local steps beyond Ko- We leave the proof in the Appendix
section beginning in eq 24
In our algorithm,
the normalized update based on the following biased proxy, since eq 10
208 tends be noisy from g: Kk.
K
G. = 7
V/
min
gi
(11)
S.K
k=
Σ k=0 (x(x
G = 7L
Vi
min
(12)
KSK:
|| g:(1
5

---

207
where G3 is the normalized update. The proxy yields a bias towards smaller number of local steps, which is desirable for reducing local computation. We use this biased proxy against using a more typical regularization such as 1.2 for the number of local steps, based on initial empirical results for better performance..
208
210
3.3.4
Normalized Exponentiated Gradient Updates
212
For the update rules of the hyperparameters !/2. (client learning rate) and K (number of client local steps), we use the normalized exponentiated gradient descent method (EGN) with no momentum, rather than conventional linear update method such as the additive update of hypergradient descent proposed in Baydin et al. [0177]. It is reasonable to use exponentiated gradient (EG) methods for updates of hyperparameters that are strictly positive in value. EG methods also enjoy significantly faster convergence properties when only a small subset the dimensions are relevant, according to
213
215
217
Amid et al. [2022].
219
EG methods have been proposed in previous works for a variety of applications (Kooaak et al. [222]]
Amid et al. 2022]. Li et al. 2020 and analyzed in depth (Ghai et al. (019)). ). where its convergence
221
has been studied and validated (Li and Cevher (018)) Recently, Amid al. 2022] showed that EGN
222
is the same as the multiplicative update for hypergradient descent proposed in Baydin et al. [2017].
223
when the approximation exp(-) + is made. From our observations, we believe that momentum not needed for the effectiveness of EGN in our application,
validated in our numerical results.
225 We also opted-out of adding further complexity such as extra weights and activation functions to
model the relationships between M,, and Kg, because it would require more samples to optimize and because FATHOM is a one-shot procedure. Furthermore, due to the non non-stationary nature of these
227
values, opt for simpler scheme for faster performance.
229
Hence, for the update rule of client learning rate, ?],, we have:
1 = fJr,, exp ( (-7, H+)
(13)
where H, is as defined in eq For number of local steps, we observe that it is related to batch size in round t. B,, as follows. To accommodate heterogeneity of local dataset sizes among clients, have number of local data samples from client i to be 14. The number of local steps for client i 233 is K; = E B.J, where E, is number of epochs, with E, = 1 meaning the entire local dataset for each client be processed once per round. We derive update rules for E, and B, globally to optimize the number of local steps, without having to make any changes to our theoretical analysis to accommodate the heterogeneity of local dataset sizes:
231
235
E = Exexp 77 N. (.))
(14)
and
B+11 = exp ( 7B (-
(15)
238
where N₁ and G2 are defined in eq and eq respectively. These update rules accomplish the goal
of updating the number of local steps via E,/B, B, with F
B e; exp ( YeN, (TE - -7B) 37B)G, G.).
239
240
Typically, with YB 2 7E. (Yp - -r) Gr becomes a tunable regularization term as discussed at the
end of Section 3
242
3.3.5
Client Sampling
243
We present our method, FATHOM, as shown in Algorithm 1 considered in our discussions is partial client sampling. For our implementation to handle the
One practical factor we have not
stochastic nature of client sampling, the metric A for calculating H, in eq and N; in eq 9 is
245
246
modified by a smoothing function for noise filtering, i.e. A, = 0A, 1- s8m -I-a)) A, which is a
single-pole infinite impulse response filter (Oppenheim and Schater (0099 Oppenheimer et al. [2009])
with no bias compensation. We use the notation " 'sm"' for smoothed, and after many experiments, we
248
249
decide to use a 0.5 for all of our numerical results.

---

Algorithm 1: FATHOM gs(z)) is defined in Assumptions and m is the number of clients.
Input: Server initializes global model Tr=1, T as the end communication round, and:
tm = 0 a = 0.5
0.01 TE = 0.01 ;BB = 0.1
Output: 2r, as well as 1, Eg and B, for E [T]
for f= 1. T do
Sample client set S₁ out of m clients.
For each client i St, initialize: x; -.k==
24 and Ke,i = |p/Et/ Br)
Set A, = 0, and 6; +o0.
for k = ..... Ke
do
For each client i, compute in parallel an unbiased stochastic gradient gi(2) ). For each client i, calculate 6, min (0, 9: (2, Ax) where A 22 2t
For each client i, update in parallel its local solution: xt 1.1+ 3.8
(1 (2))))
end
Server calcualtes -ies, Us, where vs is the size of client dataset.
Server calculates A, = EiEs, Ai(v/v);; see eq
Ft A. Server updates global model x++11
Server calculates H. NN,- ||+l
T.
Am Am
modified from eq 5 and eq eq
Server calculates G eq 12
Server updates client learning rate M1, I. epochs, E and batch size Bi41 for the next
round; see eq 13. eq and eq eq
15
Server updates A 1.em
A a 1,sm for the next round
end
4 Theoretical Convergence
252
standard approach theoretical analysis of online optimization method such ours. is through
253
analyzing the regret bound Zinkevich 2003), Khodak et al. [2019], Kingma and Ba (2014), and Mokhtari al. (0166). Nonetheless, this approach does not tell us the impact on communication
254
255
efficiency by the online updates introduced from FATHOM. Therefore, we take an alternative
256
approach by extending the guarantees of FedAvg performance (Wang et al. (02T)) Reddi al.
2020], Gorbunov et al. [2020), Yang et al. 2021 Li et al. (2019), etc) to include both adaptive learning rate and adaptive number of local steps. We assume the special case in our analysis have full client participation. We prove that adaptive learning rate and adaptive number of local steps does not impact asymptotic convergence, despite the given relaxed conditions.
258
280
4.1 Assumptions
262
Assumption 3. (L-Lipschitz Continuous Gradient for Parameters X2) There exists a constant L > such that f f.(y)ll < L| yll. Vx, y E Rd,, and i E [m], where and y are the parameters in e
264
285
Assumption 4. (Bounded Local Variance) There exist a constant σı > 0, such that the variance of each local gradient estimator is bounded by E fı(z) (x))] < 1 Vz, and E [m].
266
287
Assumption (Bounded Second Moment) There exists a constant G > such that E, f. -
G. i [m]. Vz,.
289
4.2
Convergence Results
Theorem
3
Under Assumptions 7 and with full client participation, when FATHOM shown
271
in Algorithim ! is used to find a solution x. to the unconstrained problem defined in I. the sequence of outputs {2,} satisfies the following upper-bouue where, with slight abuse of notation,
273
=mingerr E. (
Efathom = =0(v
of G2
of
C
(16)
mKT
KT2
T2
7

---

274 with the following conditions: Tz =min min and ML, for all where
2Bom
&,D
D
2.50g L2O3T (IKLT(0)+G9)
2.53sK LAGHT
711 T
1
Σ
and
K
T Σ
1
K₁
(17)
and where
E, ML,K
B1 E. ML Krl / Ee Me.e
80 T[ E, 7 E, Ke]
(18)
E, Mi,
E. Kr Kr 7 E. m.))
T Ee Ki
B3 E. 1 Ma 2 1, K.
B2
(19)
Σ, LL
EL K3
We leave the proof in the Appendix beginning in eq 29
278
The values of 3o. 31, 32. 83, and 31 are dependent on the relative changes over the adaptive process of these components, according to Chebyshev's 's Sum Inequalities (Hardy et al. (1988] ). A special case is when these quantities equal to when both T,, and K₁ are constant, which recovers the standard upperbound for FedAvg from T6).
280
282
The definitions in 17 combined with the conditions for T, above is called the relaxed conditions this paper for the hyperparameters TI., and K,. The values of n1, and K₃ are adaptive during the optimization process between rounds t and = T, as long as the above conditions are satisfied for the guarantee e7(37) to hold. This relaxation presents opportunities for a scheme such as FATHOM to exploit for performance gain. For example, suppose T approaches for a prolonged training session. Then Ti would necessarily be sufficiently small for fathom to bounded by
Remark 2.
283
284
285
287
eq T6.. However, for early rounds i.e. small values, N1. S L be reasonably large and still can satisfy 17 for the benefit of accelerated learning and convergence progress early on. Similar strategy can be used for number of local steps minimize local computations towards later rounds.
289
291
In any case, these strategies are mere guidelines meant to remain within the worst case guarantee. However, Theorem Deseemmooffee Roffers the flexibility otherwise not available. We will now show the empirical performance gained by taking advantage of this flexibility.
293
IE PaRts
BALD SUTE
NoW Clees Nami
444
9.4
Kew
helr
0.2
323
3.0
0.20
3.0
ord
0.8
254
413
196
1M
028
156
PM0
1966
1/2
338
44
1906 1350
14 LoWe Patts
al Buth Size
Nuw Clexs faad
To0les
E20
1.
1M
DEVON makl Bal
033
20 54
75
306 35M
206
250 SK
114
10W
13K 115 201
310
250
96
75
1256
1536
11M 250
Figure I: Test Accuracy Performance with various values of initial client learning rate (LR_0), initial batch size (BatchSize 0), and number of clients per round (NumClients). Top row: FSO sims. Bottom row: FEMNIST sims. Baseline values for FEMNIST: LR_0=00 BatchSize_ (=20, NumClients=10. Baseline values for FSO: LR_ 0=0.32, BatchSize 0=16, NumClients=50.
8

---

Empirical Evaluation and Numerical Results
294
We present empirical evaluation of FATHOM proposed in Section and outlined in Algorithm
296
We conduct extensive simulations of federated learning in character recognition on the federated
EMNIST-62 dataset (FEMNIST) (Cohen al. (017) with CNN, and in natural language next-wer werd-word
298
prediction on the federated Stack Overtlow dataset (FSO)
with RNN. We defer most of the details of the experiment setup in Appendix Section (2019]
Our
300
choice of datasets, tasks and models, are exactly the same as the "EMNIST CR" task and the "SO
NWP" task from Reddi et al. [2020]. See Figure T and Table D and their captions for details of the
302
experiment results. Our evaluation lacks comparison with a few one-shot FL HPO methods discussed
303
earlier in the paper because of lack of standardized benchmark (until FedHPO- B Wang al. 2022
304
was published concurrently as this work) to be fair and comprehensive.
The underlying principle behind these experiments is evaluating the robustness of FATHOM versus FedAvg under various initial settings, to mirror realistic usage scenarios where the optimal hyperpa- rameter values unknown. For FATHOM,
306
start with the same initial hyperparameter values
308
as FedAvg. The test accuracy progress with respect to communication rounds is shown in Figure !] from these experiments. We also pick test accuracy targets for the two tasks. For FEMNIST CR we use 86% and for FSO NWP we use 23%. Table [ shows table of resource utilization metrics with respect to reaching these targets in our experiments, highlighting the communication efficiency as well as reduction in local computation from FATHOM in comparison to FedAvg. To our knowledge,
309
310
312
313
we are the first show gain from an online HPO procedure over well-tuned equivalent procedure with fixed hyperparameter values.
314
The federated learning simulation framework which we build our algorithms for our experiments FedJAX (Ro et al. (2021) which is under the Apache License. The server that runs the experiments equipped with Nvidia Tesla V100 SXM2 GPUs.
316
Table 1:
Resource utilization in communication and local computation to reach specified test accuracy target for each task. All evalutions are run for ten trials. Bold numbers highlight better performance. NA means target was not reached within 1500 rounds for FSO NWP and 2000 rounds for FEMNIST CR. in any of our trials. LR_0 is initial client learning rate, BS_0 is initial batch size, and NCPR is number of clients per round. All experiments use baseline initial values except where indicated. For clarification, M is used in place for "million,, and K for "thousand". Baseline fso (LR 0 = 0.32, BS 0 == 16, NCPR 50)
Dreline_femmiss (LR 0 0. 10, BS 0 20, NCPR 10)
Number of Rounds To
Local Gradients Calculated To
Tasks
Experiments
Reach Target Test Accuracy
Reach Target Test Accuracy
FATHOM
FedAvg
FATHOM
FedAvg
Baseline Iso
562 1 12
971 : II
85M + 1.2M
124M + 1.3M
LR 0 = 0.05 BS
871 +
NA
138M + 3.2M
NA
NWP
758 + 43
580 + 18
93M + 2.8M
74M + 2.5M
Target 23%
BS 0 256
801 + 28
NA
174M + 18M
NA
NCPR 25
970 + 49
1283 + 33
63M 2.7M
82M 3.8M
NCPR 200
396 17
684 26
280M 45M
350M 13M
Baseline_ feemistt
739 24
1098 15
1.5M 36K
2.2M 64K
LR 0 = 0.0S
905 + 21
1574 19
1.7M 28K
3.1M 28K
FEMNIST CR Target 86%
BS 0
708 + 17
1.2M 28K
1.7M 88K
BS 0 = 256
736 + 20
NA
2.0M 44K
NA
NCPR = 100
777 + 16
1436 + 18
22M + 0.27M
28M + 0.39K
NCPR = 200
790 + 16
1481 + 33
57M + 1.0M
59M + 1.3M
6
Conclusion and Future Work
319 In this work, we propose FATHOM for adaptive hyperparameters in federated optimization, specifi- cally for FedAvg. We analyze theoretically and evaluate empirically its potential benefits in conver- gence behavior as measured in test accuracy, and in reduction of local computations, by automatically
320
321
adapting the three main hyperparameters of FedAvg: client learning rate, and number of local steps via epochs and batch size. An example of future efforts to extend this work is using standardized benchmark such as Wang et al. 22022 for performance comparison against other FL HPO methods.
323

---

References
325
328
E. Amid. R. Anil, C. Fifty. and M. K. Warmuth. Step-size adaptation using exponentiated gradient updates. 2022.
328
A. G. Baydin, R. Cornish, D. M. Rubio, M. Schmidt, and F. Wood. Online learning rate adaptation with hypergradient descent, 2017.
330
H. Cai, L. Zhu, and S. Han. ProxylesNNAS: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations, 2019. URL atttps://axiiv wwwpexiv...ggggeoo org/pdf/ 1812
00332. pdf
333
Z. Charles and J. Konečný. On the outsized importance of learning rates in local update methods, 2020.
334
G. Cohen, S. Afshar, J. Tapson, and A. Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017
335
international joint conference on neural networks (ICCN), pages
IEEE, 2017.
336
Dai, K. H. Low, and P. Jaillet. Federated bayesian optimization via thompson sampling, 2020.
7. Dai, B. K. H. Low, and P. Jaillet. Differentially private federated bayesian optimization with distributed exploration, 2021.
338
U. Ghai, E. Hazan, and Y. Singer. Exponentiated gradient gradient descent, 2019.
340
E. Gorbunov, F. Hanzely, and P. Richtárík. Local sgd: Unified theory and new efficient methods, 2020.
341
P. Guo, D. Yang, A. Hatamizadeh, A. Xu, Z. Xu, W. Li, C. Zhao, D. Xu, S. Harmon, E. Turkbey, et al. Auto-fedrl: Federated hyperparameter optimization for multi nulti-institutional medical image segmentation. arXiv preprint arXiv:2203.066 06338, 2022.
342
344 G. Hardy, J. Littlewood, and G. Pólya. Inequalities. Cambridge Mathematical Library. Cambridge University Press, 1988. ISBN 9781107647399
bttps//ss////o google. http:bookss1d-BvAAApp.
346
S. Holly, T. Hiessl, S. R. Lakani, D. Schall, C. Heitzinger, and J. Kemnitz. Evaluation of hyperparameter- optimization approaches in an industrial federated learning system, 2021.
348
M. Khodak, M.-F. Balcan, and A. Talwalkar. Adaptive gradient-based meta-learning methods, 2019.
349
M. Khodak, R. Tu, Li, L. Li, N. Balcan, Smith, and A. Talwalkar. Federated hyperparameter tuning: Challenges, baselines, and connections to weight-sharing. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W.
350
Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https:: /attps://opone /openrevie
353
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, 2014.
354
D. P. Kingma and M. Welling, Auto-encoding variational bayes, 2013.
355
Li, M. Khodak, M.-F. Balcan, and A. Talwalkar. Geometry- gradient algorithms for neural architecture search, 2020.
356
X. Li, K. Huang, W. Yang, S. Wang, and 7. Zhang. On the convergence of fedavg on non-iid data, 2019.
358
Y.-H. Li and V. Cevher. Convergence of the exponentiated gradient method with armijo line search. Journal of Optimization Theory and Applications, 181(2):588 607, Dee 2018. ISSN 1573-2878. doi: 10.1007/ 110957-018 1428-9, URLhttp://aa doi org/10..1110 10010/0.0000///1500000
380
1428-9
H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-cffiiiiie learning of deep networks from decentralized data, 2016.
382
A. Mokhtari, S. Shahrampour, A. Jadhahaie, and A. Ribeiro. Online optimization in dynamic environments: Improved regret rates for strongly convex problems. 2016 IEEE 55th Conference on Decision and Control (CDC), Dee 2016.
365
doi: 10. 2..11///cc.2201677999777 URL http://dxx.... doi org/10, 1009/110./../// 2016
386
779937
387
H. Mostafa. Robust federated learning through representation matching and adaptive hyper paramearreetess, 2019.
K. Murota. Discrete convex analysis. Mathematical Progrumming. 83:313-371, 1998.
389
A. V. Oppenheim and R. W. Schafer. Discrete- Time Signal Processing. Prentice Hall Press, USA, 3rd edition, 2009. ISBN 0131988425.
10

---

371
H. Pham, M. Guan, B. Zoph. Q. Le. and J. Dean. Efficient neural architecture search via parameters sharing. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Leurning, volume of Proceedings of Machine Learning Research, pages 4005-4104, PMLR, 10-15 Jul 2018. URL
373
374
mlr. grees//v80/haane 18mm
375
Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečny, S. Kumar, and H. McMahan. Adaptive rederated optimization, 2020.
376
377
H. Ro, A. T. Suresh, and K. Wu. Fedjax: Federated learning simulation with jax. arXiv preprint arX1:2108.002117, 2021.
378
379
Tennsofloow-Federrtee-----
Tensorflow federated stack overflow dataset. 2019. URL https://www
380
tensorflos http://ww......./aaaaaaa///////a
381
Wang and G. Joshi. Adaptive communication strategies achieve the best error- rrootmmntimm trade-off in local- update sgd, 2018.
382
383
Wang, Z. Charles, Z. Xu, G. Joshi, H. McMahan, B. A. y Arcas, M. Al-Shedivat, G. Andrew, S. Avestimehr,
384
K. Daly, D. Data, S. Diggavi, H. Eichner, A. Gadhikar, Z. Garrett, A. M. Girgis, F. Hanzely, A. Hard, C. He,
385
S. Horvath, Z. Huo, A. Ingerman, M. Jaggi, T. Javidi, P. Kairouz, S. Kale, S. P. Karimireddy, J. Konecny,
386
S. Koyejo, T. Li, I.. Liu, M. Mohri, H. Qi, S. J. Reddi, P. Richtarik, K. Singhal, V. Smith, M. Soltanolkotabi,
W. Song, A. T. Suresh, S. U. Stich, A. Talwalkar, H. Wang, B. Woodworth, S. Wu, F. X. Yu, H. Yuan,
388
M. Zaheer, M. Zhang, Zhang. C. Zheng, C. Zhu, and W. Zhu, A field guide to federated optimization, 2021.
Z. Wang. W. Kuang, C. Zhang, B. Ding, and Y. Li. Fedhpo-b: benchmark suite for federated hyperparameter optimization, 2022.
390
R. J. Williams. Simple statistical fraliint- algorithms for connectionist reinforcement learning. Mach. Learn., 8(3 4):229-2566 may 1992. ISSN 0885-6125. doi: 10.1007/BB00099266. URL https //doi //doi.org//1 1007//00999/
392
393
394
H. Yang. M. Fang. and Liu, Achieving linear speedup with partial worker participation non-iid federated learning. 2021.
395
396
Zhou, P. Ram, T. Salonidis, N. Baracaldo, Samulowitz, and H. Ludwig. Single-shot hyper-parameter optimization for federated learning: A general algorithm and analysis, 2022.
397
398
M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. Proceedings of the Twentieth International Conference on International Conference on Machine Learning, ICML'03, page
399
400
928 935, AAAI Press, 2003. ISBN 1577351894.
Checklist
401
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODOO] to [Yes], [No], or [N/A]. You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
405
405
Did you include the license to the code and datasets? [Yes] The code is MIT licensed.
407
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
408
409
410
1. For all authors...
411
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
412
(b) Did you describe the limitations of your work? [Yes] Please refer to Sections 2 and C
414
(c) Did you discuss any potential negative societal impacts of your work? [No] Not
415
specifically, but it is alluded to how FL applications have social implications in the introductory section.
416
417
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
11

---

419
2. If you are including theoretical results...
420
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Please refer to Assumptions D
and S
422
(b) Did you include complete proofs of all theoretical results? [Yes] Yes, in the supple- mental material.
428
424
3.
If you ran experiments..
425
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as URL)? [Yes] Yes, in the supplemental material.
427
428
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Yes, in the supplemental material.
430
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] Yes, see Table !
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Yes, it is mentioned in Section S
438
434
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
435
(a) your work uses existing assets, did you eite the creators? [Yes] Yes, it is mentioned
438
in Section 5
(b) Did you mention the license of the assets? [Yes] Yes, it is mentioned in Section 5
(c) Did you include any assets either in the supplemental material URL? Yes]] Yes, in the supplemental material.
439
440
(d) Did you discuss whether and how consent was obtained from people whose data you' 're using/curating? [N/A]
441
442
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
444
5. If you used crowdsourcing or conducted research with human subjects...
445
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
447
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
448
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
450
12


Reviews:
[{"review_id": "lvRYXM_pF8i", "reviewer": "Reviewer_KttA", "summary": "This paper proposes a method for tuning the client step-size, number of epochs, and batch-size of FedAvg in a oneshot manner. They propose hypergradient-style updates to these quantities that can be done simultaneously with updating the parameters. The algorithm is evaluated on FEMNIST and StackOverflow.", "questions": "Notes:\n1. 12: what does “open-boxed” mean?\n2. 56: having difficulty following what this sentence is saying\n3. 68: what is a “relaxed condition”?\n4. 98: the two papers use different models and subsets of the data and so their settings are incomparable\n5. 129: I suggest using \\mathbb R instead of \\mathcal R to denote the set of real numbers\n6. 150: what do A, B, and I represent? also B was used earlier to denote the batch size\n7. 169: I do not believe either paper states this as their objective\n8. Theorem 1: this theorem also requires Assumption 1\n9. Theorem 2: this theorem requires Assumptions 1 and 2\n10. 225: the choice of using the bias is itself a hyperparameter\n11. 263: this is an online convex optimization method? My understanding was the function f itself is non-convex. ", "limitations": "The authors address the limitation of only being able to tune a few hyperparameters. Other limitations, especially the theoretical assumptions made and in the empirical evaluation, are less-discussed.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Post-rebuttal:** \nFollowing clarification from the authors, I have decided to raise my score slightly but still cannot fully endorse this submission. Overall I think it is somewhat unsatisfactory to have to make such a strong assumption (convexity of the loss w.r.t. the batch-size in the sense of Murota (1990)) just to obtain subgradients. In particular, what is a bit jarring is the fact that it is still unclear whether there is *any* scenario under which this assumption holds. Standard regularity assumptions, while restrictive, at least come with specific model classes and loss functions under which they hold. Here are some things that would strengthen the theoretical component of the work:\n1. Including a statement of the definition from Murota (1998) in the paper explicitly and showing where it is used in the proofs. \n2. Defining subgradients for non-convex functions in the discrete setting, as they can be for the continuous setting (Clarke, 1990).\n3. Providing an explicit setting where Assumption 2 holds.\n4. Providing experimental evidence that Assumption 2 holds in practice. In particular, this type of convexity should be easy to measure because the underlying set is not uncountable.\n\nSummary:\nThe paper is reasonably well-motivated, and the approach of targeting a few hyperparameters seems sensible. The experimental results indeed seem to demonstrate the utility of the proposed method. However, I have strong concerns regarding the assumptions the theoretical results and other concerns regarding the experimental comparison and discussion of past work that make me hesitant to accept the paper. \n\nStrengths:\n1. The paper tackles an important and timely problem in federated learning.\n2. The authors devote a good deal of time to the communication and local computation complexity of their approach, both in the method development and empirical comparison sections.\n3. The method is evaluated on relevant benchmarks in computer vision and NLP.\n4. Code is included.\n\nWeaknesses:\n1. The derivation of the hypergradients lack a lot of detail. Theorem 1 introduces a great deal of machinery to try and define a sub-gradient w.r.t. a continuous surrogate variable of a discrete function. How the piecewise function L corresponds to the objective, what l represents precisely (rather than in words), and how Theorem 1 is used in the later proofs should be explained more precisely. The derivation of the gradients for the batch size and number of epochs in (14) and (15) is also not thoroughly explained.\n2. Assumption 2, used by all three theorems in the paper, seems extremely strong. Is there at least an example of a function class in which this holds? Otherwise it seems the paper is avoiding the need to take derivatives through local iterations by optimizing an upper bound, and also hiding this in an assumption without discussing whether it is reasonable.\n3. The experimental evaluation does not compare to baselines other than FedAvg, such as general hyperparameter tuning methods or the FL-specific methods discussed in the related work.\n4. Some of the discussion of past federated hyperparameter optimization work is incorrect (see notes).\n5. The method only works for tuning three hyperparameters. The authors do acknowledge this limitation, and it is indeed reasonable to focus on specific hyperparameters, e.g. if they are especially important. However, it would be good to have a discussion of whether the ones chosen here are indeed the most impactful.\n\n&nbsp;  \n**References:**  \n1. Clarke. Optimization and nonsmooth analysis. SIAM, 1990.\n2. Murota. Discrete convex analysis. Mathematical Programming, 1998.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SABLVFhNjE", "reviewer": "Reviewer_bwW1", "summary": "I am not an expert in this area and do not know anything about federated learning. The last time I worked directly with hyperparameter optimization approaches was in 2017 but was focused on offline hyperparameter optimization and not in the context of federated learning. As such, my review is not very insightful.\n\n---\n\nThe paper proposes an approach for online hyperparameter optimization in a federated learning setting. The proposed approach (FATHOM) is more communication efficient than previous approaches and achieves better results than previous approaches with manually fine-tuned hyperparameters. However, since it assumes access to gradients it is restricted to a certain set of hyperparameters that can be optimized and the paper evaluates the approach by optimizing the client learning rate, number of local steps, and batch size. Compared to a baseline (FedAvg) the proposed approach is more efficient and obtains better final test accuracy.", "questions": "I can't comment on the evaluation protocol itself since I don't know the related work. The target accuracy numbers seem somewhat arbitrary (86% and 23%) but may be standard for this kind of evaluation.", "limitations": "The paper itself mentions several limitations, such as applicability only to specific hyperparameters for which gradients can be obtained.", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper motivates its approach well and clearly states its advantages and limitations compared to other approaches. The comparison to FedAvg is based on different hyperparameter initializations and shows improved results on most of them.\nI did not check the theoretical convergence part.\nThe evaluation of the approach seems clear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HRUZPBycpr1", "reviewer": "Reviewer_K7kJ", "summary": "This paper proposes to optimize the hyperparameters and parameters of a federated learning model jointly via gradient descent in an online fashion. To achieve this, the paper extends previous works on hypergradient to derives analytical gradients with respect to the local learning rate and the number of gradient steps. The paper analyzes the convergence of the proposed method and verifies the empirical improvement over FedAvg using real-world FL experiments.", "questions": "I've listed a few questions and comments as \"Weaknesses\" in the section above.", "limitations": "A limitation and potential negative societal impact are both discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed method is able to jointly optimize the hyperparameters and weight parameters of an FL model in a one-shot online manner, which is a natural and promising direction.\n- The approach taken by the paper is reasonable: derive the gradient w.r.t. the hyperparameters of interest, and then use gradient descent to optimize them together with model parameters.\n\nWeaknesses:\n- A few related works are missing which I have listed below. Reference [*a*] aims at single-shot hyperparameter optimization for FL, i.e., they aim to identify a single set of hyperparameters which work well for multiple applications, and hence should be cited in the first paragraph of Section 2; [*b*] also used Bayesian optimization for hyperparameter optimization in FL and hence should be added to lines 114-116; [*c*] provided a benchmark for federated hyperparameter optimization.     \n[*a*] Single-shot Hyper-parameter Optimization for Federated Learning: A General Algorithm & Analysis, 2022.   \n[*b*] Differentially Private Federated Bayesian Optimization with Distributed Exploration, NeurIPS 2021.   \n[*c*] FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization, 2022.   \n- In the experiments, the proposed method is only compared with vanilla FedAvg, which may not be enough especially since a number of related works on single-shot federated hyperparameter optimization have been listed in Section 2. Even if comparisons are not possible, some justifications should be given.\n- Theorem 3: instead of listing the detailed expressions of the entire theorem, I would prefer to see some interpretations/insights regarding the theoretical results. For example, how do the additional components introduced by the proposed algorithm (i.e., adaptive tuning of $\\eta$ and $K$) affect the convergence of FedAvg? Are there conditions under which these additional components can lead to better convergence then standard FedAvg according to Theorem 3?\n- Perhaps partially due to my unfamiliarity with gradient-based optimization, I feel that some of the technical details should be given more explanations. Firstly, Assumption 2 may need some further explanations and justifications. For example, $x_t$ and $\\mathbb{E}_t$ haven't been introduced till this point. A justification should be given as to how realistic this assumption is. Secondly, in Equation (5), some more explanations on how the last $\\approx$ is derived are needed. Next, in equations (14) and (15), why are these two equations designed in this way?\n- The paper has many typos, for example:   \n-- line 34: a -> an\n-- line 56: justify -> justifying; a -> an\n-- line 104: directing -> directly\n-- line 145: $K$ should be $K_i$?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject