Communication-Efficient Federated Bilevel Optimization with Global and Local Lower Level Problems
Authors: Junyi Li
Feihu Huang
Heng Huang
Venue: NeurIPS 2023 poster (2023)
Keywords: Federated Learning
Bilevel Optimization

Abstract: Bilevel Optimization has witnessed notable progress recently with new emerging efficient algorithms. However, its application in the Federated Learning setting remains relatively underexplored, and the impact of Federated Learning's inherent challenges on the convergence of bilevel algorithms remain obscure.
In this work, we investigate Federated Bilevel Optimization problems and propose a communication-efficient algorithm, named FedBiOAcc. The algorithm leverages an efficient estimation of the hyper-gradient in the distributed setting and utilizes the momentum-based variance-reduction acceleration. Remarkably, FedBiOAcc achieves a communication complexity $O(\epsilon^{-1})$, a sample complexity $O(\epsilon^{-1.5})$ and the linear speed up with respect to the number of clients. We also analyze a special case of the Federated Bilevel Optimization problems, where lower level problems are locally managed by clients. We prove that FedBiOAcc-Local, a modified version of FedBiOAcc, converges at the same rate for this type of problems. Finally, we validate the proposed algorithms through two real-world tasks: Federated Data-cleaning and Federated Hyper-representation Learning. Empirical results show superior performance of our algorithms.

Communication--fficcnii Federated Bilevel Optimization with Global and Local Lower Level Problems
Junyi Li Computer Science University of Maryland College Park, MD 20742 junyili. ai@gmail. com
Feihu Huang ECE University of Pittsburgh Pittsburgh, PA 15261
Heng Huang
Computer Science University of Maryland College Park, MD 20742
huaggeiihu20180gmen com
henghuanghh@@mail. com
Abstract
Bilevel Optimization has witnessed notable progress recently with new emerging efficient algorithms. However, its application in the Federated Learning setting remains relatively underexplored, and the impact Federated Learning's s inherent challenges the convergence of bilevel algorithms remain obscure. In this work, we investigate Federated Bilevel Optimization problems and propose a conmunniation--effcieet algorithm, named FedBiOAcc. The algorithm leverages an efficient estimation of the hyper-gradient in the distributed setting and utilizes acceleration. Remarkably, FedBiOAcc
the momentum- based achieves a communication complexity O),=)), sample complexity O(e-1.5) and the linear speed up with respect to the number clients. We also analyze special case of the Federated Bilevel Optimization problems, where lower level problems are locally managed by clients. We prove that FedBiOAcc- Local,, a modified version of FedBiOAcc, converges at the same rate for this type of problems. Finally, we validate the proposed algorithms through two real- real-world tasks: Federated Data- cleaning and Federated Hyppe--epressnnaaton Learning. Empirical results show superior performance of our algorithms.
Introduction
Bilevel optimization 51 48 has increasingly drawn attention due its wide-ranging applications in numerous machine learning tasks, including hyper- parameter optimization (42), meta- meta-learning [61] and neural architecture search (36). A bilevel optimization problem involves an upper problem and a lower problem, wherein the upper problem is a function of the minimizer of the lower prob- lem. Recently, great progress has been made to solve this type of problems, particularly through the development of efficient single-loop algorithms that rely on diverse gradient approximation techniques (23). However, the majority of existing bilevel optimization research concentrates on stan- dard, non-distributed settings, and how to solve the bilevel optimization problems under distributed settings have received much less attention. Federated learning (FL) [40] is a recently promising distributed learning paradigm. In FL, set of clients jointly solve machine learning task under the coordination of a central server. To protect user privacy and mitigate communication overhead, clients perform multiple steps of local update before communicating with the server. A variety of algorithms [50) 59 16, 26 [1] have been proposed to accelcrate this training process. However, most of these algorithms primarily address standard single-level optimization problems. In this work, we study the bilevel optimization problems in the Federated Learning setting and investigate the
"This work was partially supported by NSF IIS 1838627, 1837956, 1956002, 221 1492, CNS 2213701, CCF 2217003, 2225775.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table 1: Comparisons of the FederateddNonn-eddree
bilevel optimization algorithms for finding an c-stationary point of (T). Gelf. c) and Gc(g, c) denote the number of gradient evaluations w.z. f(m) (z. y) and g (m)) (z. y); (g. e) denotes the number of Jacobian -eecorr products; HV(g. e) is the number of Hessian-vecte products; & L/P is the condition number, p(k) is used when no dependence provided. Sample complexities are measured by client.
Setting
Algorithm
Communication
Ge(J.4)
Grlg.
(V
HV(a.))
Heterngemeity
Non-Fed
MRDO 5b
23
Olpls)e-
Olpinte
CommFedBio
33
Di
Diptste
Olpiale
Obilde
Ofpikle
FedNest
[
Ola
Olick
Ole
Olk
Federated
AgellD 53
H1
Opilee
Diplaje
Oiginje
OrA p(a)e
Olztate-
OLM OAA--ppoe)
Optieee
SinFBO
58
OLM 0
DLM p(
OLM (pte-
Lecal-BSCVR
Ougiaje
O piele
O pisle
OLM plate
OLM
FedBiOAcc (Ours)
O(M O(M
D(M-'126-11
OLM 1
OCM
following research question: Is it possible to develop communication-efiiiee federated algorithms tailored for bilevel optimization problems that also ensure rapid convergence rate?
More specifically, a general Federated Bilevel Optimization problem has the following form:
M
M
min Mxx) = WEB
plm) (x. yx), s.1. Uz = arg min yERd
g (m) (ax, v)
(1)
M
A federated bilevel optimization problem consists of upper and lower level problem, the upper problem f(x, v) 314 ΣM f(m (z, y) relies on the solution ye of the lower problem. and g(z, 3) == N7 Σm ( (x, Meanwhile, both the upper and the lower level problems are federated: In Eq. we have M clients, and each client has local upper problem ( y) and lower level problem y g/'m) (z, Compared single-level federated optimization problems, the estimation of the hyper-gradient in federated bilevel optimization problems is much more challenging. In Eq. Α.. the syper-gradienti is not linear w.r./ the local hyper-gradients of clients, whereas the gradient of a single-level Federated Optimization problem is the average of local gradients. Consequently, directly applying the vanilla local-sgd method 40) to federated bilevel problems results large bias. In the literature [49] 33 53). researchers evaluate the pper-gradient through multiple rounds of client-server communication, however, this approach leads high communication overhead. In contrast, we view the grperegradeent estimation as solving a quadratic federated problem and solving with the local-sgd method. More specifically, we formulate the solution of the federated bilevel optimization as three intertwined federated problems: the upper problem, the lower problem and the quadratic problem for the hyper-gradient estimation. Then we address the three problems using alternating gradient descent steps, furthermore, to manage the noise of the stochastic gradient and obtain the fast convergence rate, we employ momentum-bant based variance reduction technique.
Beyond the standard federated bilevel optimization problem as defined in Eq. another variant of Federated Bilevel Optimization problem, which entails locally managed lower-level problems, also frequently utilized in practical applications. For this type of problem, we can get an unbiased estimate of the global hyper-gradient using local Hyper-gradient thus we can solve it with a local-SGD like algorithm, named FedBiOAcc-Loca However, it is challenging to analyze the convergence of the algorithm. particular, we need to bound the intertwined client drift error, which intrinsic to FL and the bilevel-related errors e.g. the lower level solution bias. In fact, we prove that the FedBiOAce- Local algorithm attains the same fast rate as FedBiO algorithm.
Finally, we highlight the main contributions of our paper as follows:
1. We propose FedBiOAcc solve Federated Bilevel Optimization problems, the algorithm evaluates the hypergradient of federated bilevel optimization problems efficiently and achieves optimal convergence rate through momentum-basee variance reduction. Fed- BiOAcc has sample complexity of O(c-1.5) T,5), communication complexity of O(e-1) and achieves linear speed-up w.z. the number of clients.
2. We study Federated Bilevel Optimization problem with local lower level problem for the first time, where we show the convergence of a modified version of FedBiOAcc, named FedBiOAcc--ocal for this type of problems.
3. We validate the efficacy of the proposed FedBiOAcc algorithm through two real-world tasks: Federated Data Cleaning and Federated Hyper-represeenttiio Learning.

---

Notations V denotes full gradient, Vz denotes partial derivative for variable x, higher order deriva- tives follow similar rules. [K] represents the sequence of integers from K, z represents average of the sequence of variables (x((m) JM m== lg represents the global communication timestamp 8.
2 Related Works
Bilevel optimization dates back to at least the 1960s when [51] proposed a regularization method, and then followed by many research works [10 48 55 45), while in machine learning community, similar Optimization [30] 3
ideas in the name of implicit differentiation were also used in 2.8. Early algorithms for Bilevel Optimization solved the accurate solution of the lower problem for each upper variable. Recently, researchers developed algorithms that solve the lower problem with fixed number of steps, and use the pack-propagatton through time' technique compute the hyper-gradien 9 39 11 47
Very Recently, it witnessed a surge of interest in using implicit
differentiation derive single loop algorithms [14] 17 23 28 56 19 32 7 20 [8]. In particular,
proposes a way to iteratively evaluate the hyper-gradient to save computation. In this work, we view the hyper-gradient estimation of Federated Bilevel Optimization solving quadratic federated optimization problem and use a similar iterative evaluation rule as 32 in local update.
32 7
The bilevel optimization problem is also considered in the more general settings. For example, bilevel optimization with multiple lower tasks is considered in [55
furthermore, 57 38 [3] studies the
bilevel optimization problem in the decentralized setting, 25 studies the bilevel optimization problem in the asynchronous setting. In contrast, we study bilevel optimization problems under Federated Learning 40 setting. Federated learning is a promising privacy-preserving learning paradigm for distributed data. Compared to traditional data-center distributed learning, Federated Learning poses new challenges including data heterogeneity, privacy concerns, high communication cost, and unfairness. To deal with these challenges, various methods 226 35 46, 60) 41 34] are proposed. However, bilevel optimization problems are less investigated in the federated learning setting. [54] considered the distributed bilevel formulation, but it needs to communicate the Hessian matrix for every iteration, which is computationally infeasible. More recently, FedNest (49] has been proposed to tackle the general federated nest problems, including federated bilevel problems. However, this method evaluates the full hyper-gradient at every iteration; this leads to high communication overhead; furthermore, FedNest also uses SVRG to accelerate the training. Similar works that evaluate the
hyper-gradient with multiple rounds of client-server communication are 33 21 53 58 Finally,
there is a concurrent work [12] that investigates the possibility of local gradients on Federated Bilevel Optimization, however, it only considers the homogeneous case, this setting quite constrained and much simpler than the more general heterogeneous case we considered. Furthermore, 122 only considers the case where both the upper and the lower problem are federated, and omit the equally important case where the lower level problem is not federated.
Federated Bilevel Optimization
3.1 Some Mild Assumptions
Note that the formulation of Eq. I is very general, and we consider the stochastic heterogencous case in this work. More specifically, we assume:
flm) v) == E. E (m(m) (x. (()) (x, = E E (m)
(z. [)
where ((m) and DI are some probability distributions. Furthermore, we assume the local objectives could be potentially different: f(m) (v. y) * (()) (ar, y) (( (x, * g (x,, y) for + k m. [M]. Furthermore, we assume the following assumptions in our subsequent discussion:
Assumption 3.1. Function f(m) (xx, y) is possibly non-convex and a g( (x. v) is p-strongly convex w.tt for any given x.
Assumption 3.2. Function (m)) (x, u) is L-smooth and has Cy-bounded gradient;
Assumption 3.3. Function g (m) (xx,) v) is I--moothh and Vayg (x, y) and V. (yag(m) y) are Lipschitz continuous with constants Lay and Ly² respectively;
3

---

Algorithm Accelerated Federated Bilevel Optimization (FedBiOAce)
Input: Constants Cu. Cus Cu. 7/ 7/. T, T; learning rate schedule {ar}, t [T], initial state (z1, y1
u1);
2: Initialization: Set Vi (m) = y1. 2 (m) 21, u/mh) u1, w(()) Vyg(m) V1, By), v/(h)
/ (x), ( y1; B5z - Vayg(m) (xi, (01,31; y1; 1 and qi = Vgegin (mm (m))
;1 (m),
BBg2)u
ygf (m) 21 (m), B122) for m E [M
3: for t 1 to T do
((m) 1//
(m))
(m) [m() = Jt {m)
-aay (m) (m) 11+11 = P,(u (m) Tarelt )m))
4:
you!
5:
i mod
I 0 then
6:
(im) M E j= Di+1: T+t1 M "j=1
(() (m)
2
(1)
1+1
u1i M از= "it1
u
(m)
7:
else (m) (m) (m) (m) (m) (m)
8:
Mi+1 1Mt1 4+t 2t1
M441
d1t1
9:
end if
Get 1 (mm) and 1:+1 following Eq. 7
(m)
10:
11:
if mod then
(m) = M Σ}= We11 Mt1
()) (m)
M -ド= M (1) (m)
1411 9+4 M Σ .0
12:
wi+1
9:+1.
13:
else
(mm) = w Mit1 = 2 l.41 = amm)
(m)
14:
15:
end if
16:
end for
Assumption 3.4. We have unbiased stochastic first-order and second-order gradient oracle with bounded variance.
Assumption 3.5. For any m,j E [M] and 2 (x, v). we have:
f(m v (2)] Sr. ( (2) (2)-VQ (z)I Sg. IV "Vzgg(m (m)) (z)-Vavl Vaygli) (2)]l Sg.zy: (m V yigg
Sg.vo. where Cy. Sgz Sg.ay. Sg.wy. are constants.
study the non-coneex-trronly-oone bilevel optimization problems, this class of problems is widely studied in the non-distributed bilevel literature (2 14. Furthermore, Assumption 33.2 and Assumption 3.3 are also standard assumptions made in the non- distributed bilevel literature. Assumption 3.4 widely used in the study of stochastic optimization problems. For Assumption gradient difference is widely used in single level Federated Learning literature as measure of chinnt heterogeneity (28] 52 Please refer to the full version of Assumptions in Appendix.
As stated in The assumption 3.1
3.2 The FedBiOAce Algorithm
major difficulty in solving a Federated Bilevel Optimization problem Eq. ① evaluating the hyper-gradient (a:). For the function class coon-convex-stronnlyycoosse explicit form of hypergradient fı(z) exists as Vh(x)) = (<x, vo), where (2 y) is denoted as:
consider, the
( y) - v.f(zz y) V myg (x, y) x [V Va - y vf
(2)
Based on Assumption 3.1 3.3 we can verify (x. yz is the hyper-gradient clients only have access to their local data, for Vm [M]. the client evaluates:
14 But since the
(m) (x, v) -V (()) (x, v) - Vayg (z, v) x [V (m)g(m)
(m)) (I, w)] v (x, y),
(3)
It is straightforward to verify that ()(m) (x, y) is not an unbiased estimate of the full hyper-gradient i.e. (x yz) * M DM ((m) (x. yz). To address this difficulty, we can view the Hyper-gradient computation as the process of solving a federated optimization problem.
In fact, Evaluating Eq. 2 is equivalent the following two steps: first, solve the quadratic federated optimization problem I()):
1 M
min 1(2) nERd
Σ n² (V (m)) (x) (e. (V (m) v),
(4)
M

---

Suppose that we denote the solution of the above problem as u", then we have the following linear operation get the hypergradient:
M Σ
Vh(x)
M V (x, yu) V ayg (mm) (I, yo)
(5)
Compared to the formulation Eq. (2) Eq. and Eq. ( are more suitable for the distributed setting. In fact, both Eq. (4) and Eq. ( have a linear structure. Eq. 4 is (single-level) quadratic federated
optimization problem, and we could solve Eq. (4) through local-sgd [40), suppose that each client
maintains a variable fm) and performs the following update:
(m) Pr(u, (m)
Te (((mm) (u. (m); B)))
ue+1
Vi (m)) B) = V v (ym) :yt B).2))u (m)) y. (mm))) (ar (m)) (m)) ;B,22)
(m))
where V2(m) ( B) is client m's the stochastic gradient of Eq. 4 and (x(")) :ii (ml) denotes the upper and lower variable state at the timestamp I, the (-) denotes the projection to a bounded
ball of radius-r. Note that Clients perform multiple local updates of un before averaging. As for Eq. (5) each client evaluates (=(m)():) (x)) locally: (x)) V ( (1r Uz) Vayg (m)) (ar, Ya
and the server averages Vh(m) to get Vh(x). In summary, the linear structure of Eq. and Eq. (5) makes it suitable for local updates, therefore, reduce the communication cost.
More specifically, we perform alternative update of upper level variable (m)) the lower level variable Vi (m) and
computation variable u, ((m) For example, for each client m E [M/] we
perform the following local updates:
V 1 Vt (m)
(m)
To V (m) (2, (x())) Vv
(m)) (m)) .By), u441
(m) (r (m) - Te ( (u(m): B)) BB,1) - V Vayg(m) (mm) (m) (m)) _,t") (m²);
2+4 = (m) T/e V V.z(mm) f(m)) (xr) (m) , (m²)
)
(6)
Every 7 steps, the server averages clients' local states, this resembles the local-sgd method for single level federated optimization problems. Note that the update of the upper variable x; r,(")), we use as an estimation of u" in Eq. (5). An algorithm follows Eq. ( is shown in Algorithm 2 of Appendix and we refer to it as FedBiO.
ut (m)
Comparison with FedNest. The update rule of Eq. 6 is very different from that of FedNest [49 and its follow-ups 53]. In FedNest, a sub-routine named FedIHGP used to evaluate Eq. (2) at every global epoch. This involves multiple rounds of client-server communication and leads to higher communication overhead.
contrast, Eq. (6) formulates the hyper-gradient estimation as
an quadratic federated optimization problem, and then solves three intertwined federated problems through alternative updates of x, y and u.
Note that Eq. 6 updates the related variables through vanilla gradient descent steps. In the non- federated setting, gradient-based methods such as stocBiO 23 requires large-batch size (O(e to reach an e-stationary point, and we also analyze Algorithm in Appendix to show the same dependence. To control the noise and remove the dependence over large batch size, we apply the momentum-basn based variance-reductioon technique STORM [6]. In fact, Eq. (6) solves three intertwined optimization problems: the bilevel problem h(z), the lower level problem g(x, y) and the hyper- gradient computation problem Eq (4) So we control the noise in the process of solving each of the
(m) be the momentum estimator for three problems. More specifically, have !!!") (m)) and qi
(mm) (m) (m) and ut"" respectively, and we update them following the rule of STORM 6 it V/
a(m)
(m) = V (m) (2(m)
(m)) v(m)) (2441, 11.1. BBy) + (1 - (wa)) V (T)(")(( (m) :te
(m) ,B,))
(m)
= (V.. (m)) (x serti B5,1) - Vzyg((m) (m) (a (i++1: vithi Bo. (u(()) (m))
(m]
+ (1 - (v) (m) (V,) (m)) (1r (m)) Ut
(m)) BB,)) - Vayg (m)) (a :
(m))
BB, )u (mm)))
(m)
(V (m) (3 vot11
(m)
(, (m1) (m) {m) (m) B,,2))
4+1
+ +1-ccaa)) (1)
(m) (V,2g (1)( vi (m)) B 22 (m) V v (m) (m) we (m)) 13 BB,2)))
(m)
5

---

where Cu. Co and Cu are constants, Ot is the learning rate. Then we update the Itc and (m) as follows:
(m)) (m))
Wi+1 (m) yaya? (m) (m) = [(])
Toun (m) (m)
u Pr (mm) go (m))
(8)
where 7. 7) T are constants and Or the learning rate. The FedBiOAcc algorithm following Eq. (8)
is summarized in Algorithm [] As shown in line 6 and 12 of Algorithm | Every I iterations, we
average both variables and the momentum.
3.3 Convergence Analysis
In this section, we study the convergence property for the FedBiOAcc algorithm. For any E (7), we define the following virtual sequence:
M
M
M
Σ
(m)
(m)
Dt
M m=1
M
m=1
m==
we denote the average of the momentum similarly as We i/ and Tr. Lyapunov function Gr:
Then we consider the following
G. = h(t)) +
18 (11 - y2,1l + l| - uz 1
96MM
1
M
V
(m))
(x)
(m) (1m)
)
)1?
NY
64ar
M
m=1
+
9b
Σ
M
V
(2) (mm) (mm)) )u V (m)(m) (I((m)
(mm)
(mm)
(m1) ))1?
64cz
M
m=1
9b
Σ
M
V f(m (m) m/ V ayy (m (z)
(m))
, yul ju
(m)) (m))
mm))
(9)
64a
M
where Ua, denotes the solution of the lower level problem (T -) Ma. V Vy TT. Us - V, Va) denotes the solution of Eq 4 state Tt. Besides, 7, 1), 7 are learning rates and L, L are constants. Note that the first three terms of G;: h(TT), 11 Va. 112. [|ū, u2 || measures the errors of three federated problems: the upper level problem. the lower level problem and the syper-gradient estimation. Then the last three terms measure the estimation error of the momentum variables: Wa, Di and qe. The convergence proof primarily concentrates on bounding these errors. please see Lemma C.2 C.6 in the Appendix for more details. Meanwhile, as in the single level federated optimization problems, local updates lead to client-drift error. More specifically, need bound ||=r(m)) 3x
llv, (m) Tell and ||u; (mm) ūz 122, please see Lemma C.7 C.11 for more details. Finally, we have the following convergence theorem:
8 (u+t)i/a: t E (T), for some constant ō and and let cv, Cas Cn choose some value, 1), Y and t, be some small values decided by the Lipschitz constants of h(r), we choose the minibatch size to be b2 = by = b and the first batch be bj O16),, then we have:
Theorem 3.6. Suppose in Algorithm 1 we choose learning rate
[IVIEEEE =0
=
K x229/31 I
K 16/3
T
(bMT)2/3
To reach an e-stationary point, we need T 0 0( -15). 1= 0 ((b -1.
As stated in the Theorem, to reach an e-stationary point, we need T 0 (bM) -1-15 then the sample complexity for each client is Gc(f. c) = O(M - -1.5), Gc(g. c) 1 x5 1..6) So FedBiOAce achieves the linear speed up w.r.t the number of clients M. Next, suppose we choose I (1///3((MM)))) -0055), then the number of communication round E 29/29/3 This matches the optimal communication complexity of the single level optimization problems as in the STEM (27). Furthermore, compared to FedNest and its variants, FedBiOAce has improved both the communication complexity and the iteration complexity. As for LocalBSCVR [I2), FedBiOAcc obtains same rate, but incorporates the heterogeneous case. Note that it is much more challenging to analyze the heterogeneous case. In fact, if we assume homogeneous clients, we have local hyper-gradient (Eq. (3) equals the global hyper-gradient (Eq. (2)) then we do not need to use the quadratic federated optimization problem view in Section 3.2, while the theoretical analysis is also simplified significantly.
Ju(g.C c) ( (M--11&5111)) 1.5), Hu(g, c) R0

---

4 Federated Bilevel Optimization with Local Lower Level Problems
In this section, consider an alternative formulation of the Federated Bilevel Optimization problems as follows:
min h(x)= mERP
Σ glm) U/e )/( s.t. g/(h) arg min gf (m)) (z.y) M
1
(10)
wERd
Same as Eq. T Eq. 10p has federated upper level problem, however, Eq. (00) has unique lower
level problem for each client, which is different from Eq. ( In fact, federated bilevel optimization problem Eq (T) can be viewed as a special type of standard federated learning problems. we
denote (m (x) = f(m) (z, (m)), then Eq. (10) can be written as min hxz) = 37 Σm (x))
.EEP
But due to the bilevel structure of h(m)) (x). Eq. 100 more challenging than the standard Federated Learning problems.
Hyper-gradient Estimation. Assume Assumption 3.1 Assumption 3.3 hold, then the hyper-gradient Φ(x, yx) = Σm ((m) (ar, yt), where ((m) (x., v) is defined in Eq. (3). in other words, the local hyper-gradient ((m) (z. y) is an unbiased estimate of the full hyper-gradient. This fact makes it possible to solve Eq. (TO) with local-sgd like methods. More specifically, we solve the local bilevel problem (m))()) (x) multiple steps on each client and then the server averages the local states from clients. Please refer to Algorithm 3 and the variance-reduction acceleration Algorithm 4 in the respectively.
Appendix. For ease of reference, name them FedBiO-Local and FedBiOAcc-
Several challenges exist in analyzing FedBiO-Local and FedBiOAco- First, Eq. involves Hessian inverse, only evaluate it approximately through the Neumann series [37 as:
((m)) (x, y3 Ex) = vn (m)) (I, yi E.) - TV 2n9 (m) (x,, yi En)
Q-1
Σ
II (I- Q-9
ye (m) (z. V (z, (2),) (r)
(11)
and we assume its elements are mutually independent. ((()) (x. yi Ex) is a biased estimate of (1m)((z, y). but with bounded bias and variance (Please see Proposition D.2 for more details.) Furthermore, reduce the computation cost, each client solves the local lower level problem approximately and we update the upper and lower level variable alternatively. The idea of alternative update is widely used in the non non-distributed bilevel optimization [23] (56). However, in the federated setting, client variables drift away when performing multiple local steps. As result, the variable drift error and the bias caused by inexact solution of the lower level problem intertwined with each other. For example, in the local update, clients optimize the lower level variable
where &ur = (,.0===.....).-....
=
(( towards the minimizer y g but after the communication step, (1² is smoothed among clients, as result, the target of (!!")) changes which causes a huge bias.
In the appendix, we show the FedBiOAce- Local algorithm achieves the same optimal convergence rate as FedBiOAce, which has iteration complexity 005-1.5) and communication complexity Ole '). However, since the lower level problem in Eq. (TO) is unique for each client, FedBiOAce- does not have the property of linear speed-up the number of clients FedBiOAcc does.
Numerical Experiments
In this section, we assess the performance of the proposed FedBiOAcc algorithm through two federated bilevel tasks: Federated Data Cleaning and Federated ryper-representtatin Learning. The Federated Data Cleaning task involves global lower level problems, while the rlpeerrepressnttation Learning task involves local lower level problems. The implementation is carried out using PyTorch, and the Federated I Learning environment is simulated using the PyTorch. Distribubd package. Our experiments were conducted on servers equipped with an AMD EPYC 7763 64-core CPU and NVIDIA V100 GPUs.
7

---

5
Figure 1: Validation Error vs Communication Rounds. From Left Right: / 0.8, The local step 7 is set as 5 for FedBiO, FedBiOAcc and FedAvg.
106 1000
1001 AU0IE
Figure 2: Validation Error vs Communication Rounds with different number of clients per epoch. From Left to Right: p = 0.1, 0.4, 0.8, 0.95. The local step is set as 5.
5.1 Federated Data Cleaning
In this section, we consider the Federated Data Cleaning task. In this task, we are given a noisy training dataset whose labels are corrupted by noise and clean validation set. Then we aim to find weights for training samples such that model that is learned over the weighted training set performs well on the validation set. This is federated bilevel problem when the noisy training set distributed over multiple clients. The formulation of the task is included in Appendix B.I. This task a specialization of Eq. (
Dataset and Baselines. We create 10 clients and construct datasets based on MNIST 31]. For the training set, each client randomly samples 4500 images (no overlap among clients) from 10 classes and then randomly uniformly perturb the labels of p (0 < < 1) percent samples. For the validation set, each client randomly selects 50 clean images from a different class. In other words, the man. client only has validation samples from the mus class. This single-class validation setting introduces high level of heterogeneity, such that individual clients are unable to conduct local cleaning due to they only have clean samples from one class. In our experiments, we test our FedBiOAcc algorithm, including the FedBiO algorithm (Algorithm 2 in Appendix) which does not use variance reduction; additionally, we also consider some baseline methods: a baseline that directly performs FedAvg 40 on the noisy dataset, this helps to verify the usefulness of data cleaning; Local-BSGVE [12]. FedNest [49]. CommFedBiO [33], AggITD 53 and FedMBO [21]. Note that BocaR-BSGVR is designed for the homogeneous setting, and the last four baselines all need multiple rounds of client-server communication to evaluate the hyper-gradient at each global epoch. We perform grid search to find the best hyper-parameters for each method and report the best results. Specific choices are included in Appendix B.1.
In figure D we compare the performance of different methods at various noise levels p. Note that the larger the p value, the more noisy the training data are. The noise level can be illustrated by the performance of the FedAvg algorithm, which learns over the noisy data directly. As shown in the figure, FedAvg learns almost nothing when p = 0.95. Next, our algorithms are robust under various heterogeneity levels. When the noise level in the training set increases as the value of p increases, learning relies more on the signal from the heterogeneous validation set, and our algorithms consistently outperform other baselines. Finally, in figure we vary the number of clients sampled per epoch, and the experimental results show that our FedBiOAcc converges faster with more clients in the training per epoch; in figure we vary the number of local steps under different noisy levels. Interestingly, the algorithm benefit more from the local training under larger noise.
8

---

Cunmuniition Toutco
Figure 3: Validation Error Communication Rounds with different number of local steps I. From Left to Right: = 0.1,0.4, 0.8, 0.95.
No ool
20
A0
Figure 4: Validation Error Communication Rounds. The top row shows the result for the Omniglot Dataset and the bottom row shows MinilmageNet. From Left to Right: 5-way-1-shot, 5-way-5 10-way-1-shot, 20-way-5-shot. The local step I is set to 5.
5.2 Federated
Learning
In the Hyper--epresenntaiinn learning task, we learn hyper-representation of the data such that linear classifier can be learned quickly with a small number of data samples. A mathematical formulation of the task is included in Appendix B.2. Note that this task is instantiation of Eq. (TOp, due to the fact that each client has its own tasks, and thus only the upper level problem is federated. We consider the Omniglot (29) and MinilmageNet 44 data sets. As in the non-distributed setting, we perform N W-way-K -hot classification.
In this experiment, compare FedBiOAcc-Local (Algorithm 4 in the Appendix) with three baselines FedBiO-Local (AllllAlllaan (Algorithm 3 in the Appendix), DistBiO and DistBiOAce. Note that DistBiO and DistBiOAce are the distributed version of FedBiO-Local and FedBiOAce-Laae experiments, we implement DistBiO and DistBiOAcc by setting the local steps as for FedBiO-Local and FedBiOAcc-Local We perform grid search for the hypae-pprrmettr selection for both methods and choose the best ones, the specific choices of hyper-parameters are deferred to Appendix B.2. The results are summarized Figure (full results are included Figure and Figure 6 of Appendix. As shown by the results, FedBiOAce converges faster than the baselines on both datasets and on all four types of classification tasks, which demonstrates the effectiveness of variance reduction and multiple steps of local training.
respectively. In the
6 Conclusion
In this paper, we study the Federated Bilevel Optimization problems and introduce FedBiOAce. In particular, FedBiOAce evaluates the hyper-gradiens by solving federated quadratic problem, and mitigates the noise through momentum- -baed variance reduction technique. We provide a rigorous convergence analysis for our proposed method and show that FedBiOAce has the optimal iteration complexity Oie -1.5) and communication complexity O(e-1), and it also achieves linear speed-up w.n. the number of clients. Besides, we study a type of novel Federated Bilevel Optimization problems with local lower level problems. We modify FedBiO for this type of problems and propose

---

FedBiOAce- Local. FedBiOAce- Local achieves the same optimal convergence rate as FedBiOAcc. Finally, we validate our algorithms with real-world tasks.
References
[1] A. K. R. Bayoumi, K. Mishchenko, and P. Richtarik. Tighter theory for local sgd on identical and heterogeneous data. In International Conference on Artificial Intelligence and Statistics, pages 4519-4529, 2020.
[2] Y. Bengio. Gradient-based optimization of hyperparameters. Neural computation, 12(8):1889) 1889 1900, 2000.
[3] D. Chen and M. T. Hagan. Optimal use regularization and cross-validation neural network modeling. In IJCNN'99. International Joint Conference Neural Networks. Proceedings (Cat. No. 99CH36339), volume 2, pages 1275-1280. IEEE, 1999.
[4] T. Chen, Y. Sun, and W. Yin. A single-timeseen stochastic bilevel optimization method. arXiv preprint wrXiv:2102.046711 2021.
[5] X. Chen, M. Huang. and S. Ma. Decentralized bilevel optimization. arXtv:2206.05670, 2022.
arXiv preprint
[6] A. Cutkosky and F. Orabona. Momentum- variance reduction in non-convex sgd. In Advances in Neural Information Processing Systems, pages 55236-15255, 2019.
[7] M. Dagréou, P. Ablin, S. Vaiter, and T. Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. arXiv preprint arXiv:2201. 2022.
[8] C. B. Do, C.-S. Foo, and A. Y. Ng. Efficient multiple hyperparameter learning for log-lincar models. In NIPS, volume 2007, pages 377-384. Citeseer, 2007.
[9] Domke. Generic methods for optimization-basee modeling. In Artificial Intelligncce and Statistics, pages 318-326. PMLR, 2012.
[10] M. C. Ferris and O. L. Mangasarian. Finite perturbation of convex programs. Applied Mathe- matics and Optimization, 23( 1):2263-227. 1991.
[11] L. Franceschi, M. Donini, P. Frasconi, and M. Pontil. Forward and reverse gradient- based hyperparameter optimization. Proceedings of the 34th International Conference on Machine Learning-Volumm 70, pages 1165-1173. JMLR. org, 2017.
[12] H. Gao. On the convergence of momentum-based algorithms for federated stochastic bilevel optimization problems. arXiv preprint arXiv:2204. 13299, 2022.
[13] H. Gao, B. Gu, and M. T. Thai. On the convergence of distributed stochastic bilevel optimization algorithms over a network. In International Conference on Artificial Intelligence and Statistics, pages 9238 9281. PMLR, 2023.
[14] S. Ghadimi and M. Wang. Approximation methods for bilevel programming. arXiv preprint 2018.
[15] Z. Guo, Q. Hu, L. Zhang, and T. Yang. Randomized stochastic variance-reduced methods for multi-task stochastic bilevel optimization. arXiv preprint arXiv:2105.00226, 2021.
[16] F. Haddadpour and M. Mahdavi. On the convergence of local descent methods in federated learning. arXiv preprint arXiv:1910.1442. 2019.
[17] M. Hong, H.-T. Wai, Z. Wang, and Z. Yang. A two-timescale framework for bilevel optimization: Complexity analysis and application actor-critic. arXiv preprint arXiv:2007.05//00 2020.
[18] Huang. On momentum-based gradient methods for bilevel optimization with nonconvex lower-level. arXiv preprint arXiv:2303.0394 2023.
10

---

[19] F. Huang and H. Huang. Biadam: Fast adaptive bilevel optimization methods. arXiv preprint arXiv:2106. 1139,, 2021.
[20] Huang, J. Li, S. Gao, and H. Huang. Enhanced bilevel optimization via bregman distance. Advances in Neural Information Processing Systems, 35:28928 209889988-899,, 2022.
[21] M. Huang, D. Zhang, and K. Ji. Achieving linear speedup in non-iid federated bilevel learning. arXiv preprint 200.::2300.05412,, 2023.
[22] K. and Y. Liang. Lower bounds and accelerated algorithms for bilevel optimization. arXiv preprint arXiv:2 2021.
[23] K. Ji, J. Yang, and Y. Liang. Provably faster algorithms for bilevel optimization and applications meta-learning. arXiv preprint arXiv:2010.07962 2020.
[24] K. Ji. J. Yang, and Y. Liang. Bilevel optimization: Convergence analysis and enhanced design. In International conference on machine learning, pages 4882-4892. PMLR, 2021.
[25] Y. Jiao, K. Yang. T. Wu, D. Song, and C. Jian. Asynchronous distributed bilevel optimization. arXiv preprint arXiv:2212.10048, 2022.
[26] S. Karimireddy, S. Kale, M. Mohri, S. J. Reddi, U. Stich, and A. T. Suresh. Scaf- fold: Stochastic controlled averaging for on-device federated learning. arXiv preprint arXiv:1910.06378, 2019.
[27] Khanduri, P. Sharma, H. Yang, M. Hong, J. Liu, K. Rajawat, and P. Varshney. Stem: A stochastic two-sided momentum algorithm achieving near-optimal sample and communication complexities for federated learning. Advances Neural Information Processing Systems, 146050-60611 2021.
[28] P. Khanduri, S. Zeng, M. Hong, H.-T. Wai, Z. Wang, and Z. Yang. A near-optimal algorithm for stochastic bilevel optimization via
arXiv preprint arXiv:2102.07:36, 2021.
[29] B. Lake, R. Salakhutdinov, J. Gross, and J. Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the annual meeting of the cognitive science society, volume 33, 2011.
[30] Larsen, L. K. Hansen, C. Svarer, and M. Ohlsson. Design and regularization of neural networks: the optimal use of a validation set. In Nearal Networks for Signal Processing VI. Proceedings of the 1996 IEEE Signal Processing Society Workshop, pages 62-71. IEEE, 1996.
[31] Y. LeCun, L. Bottou, Y. Bengio, and Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(11):227--2324 1998.
[32] J. Li, B. Gu, and H. Huang. A fully single loop algorithm for bilevel optimization without hessian inverse. In Proceedings the AAAI Conference on Artificial Intelligence, volume 36, pages 1426-7434, 2022.
[33] J. Li, J. Pei, and H. Huang. Communication-effcieen robust. federated learning with noisy labels. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 914-924, 2022.
[34] T. Li, S. Hu, A. Beirami, and V. Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pages 6357-6368. PMLR, 2021.
[35] X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang. On the convergence of fedavg non-iid data. arXiv preprint arXiv:1907. 907,02289, 2019.
[36] H. Liu, K. Simonyan, and Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.0000 09055. 2018.
[37] J. Lorraine, P. Vicol, and D. Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics, pages 1540-1552. PMLR, 2020.
11

---

[38] S. Lu, S. Zeng, X. Cui, M. Squillante, L. Horesh, B. Kingsbury. J. Liu, and M. Hong. A stochastic linearized augmented lagrangian method for decentralized bilevel optimization. Advances in Neural Information Processing Systems, 35:30638-3066 2022.
[39] D. Maclaurin, D. Duvenaud, and R. Adams. Gradient based hyperparameter optimization through reversible learning. In International Conference on Machine Learning, pages 2113- 2122, 2015.
[40] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-effiiiie learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273-1282. PMLR, 2017.
[41] M. Mohri, G. Sivek, and A. T. Suresh. Agnostic federated learning. arXiv preprint 1oX1v:190..00146, 2019.
[42] T. Okuno, A. Takeda, and A. Kawana. Hyperparameter learning via bilevel nonsmooth opti- mization. arXiv preprint arXiv:1806.01520, 2018.
[43] F. Pedregosa. Hyperparameter optimization with approximate gradient. arXiv preprint 02355, 2016.
[44] S. Ravi and H. Larochelle. Optimization as model for few-shot learning. In International conference on learning representations, 2017.
[45] S. Sabach and S. Shtern. A first order method for solving convex bilevel optimization problems. SIAM Journal on Optimization, 27(2):640-66 660, 2017.
[46] A. K. Sahu, T. Li, M. Sanjabi, M. Zaheer, A. Talwalkar, and V. Smith. On the convergence of federated optimization in heterogeneous networks. arXiv preprint arXiv:1812.06117,, 3, 2018.
[47] A. Shaban, C.-A. Cheng, N. Hatch, and B. Boots. Truncated back-propagation for bilevel optimization, arXiv preprint arXiv:1810.100600 2018.
[48] M. Solodov, An explicit descent method for bilevel convex optimization. Journal of Convex Analysis, 14(2):227, 2007.
[49] D. A. Tarzanagh, M. Li, C. Thrampoulidis, and S. Oymak. Fednest: Federated bilevel, minimax, and compositional optimization. arXiv preprint arXiv:2205.02215. 2022.
[50] S. Wang, T. Tuor, T. Salonidis, K. K. Leung. C. Makaya. T. He, and K. Chan. Adaptive federated learning in resource constrained edge computing systems. IEEE Journal on Selected Areas in Communications, 37(6):100--12211 2019.
[51] R. A. Willoughby. Solutions of ill poo-d problems (an tikhonov and vy arsenin). SIAM Review, 21(2):266, 1979.
[52] B. Woodworth. The minimax complexity of distributed optimization. arXiv preprint
arXiv:2109.00534 2021.
[53] P. Xiao and K. Ji. Communicatinn-effiienn federated hypergradient computation via aggregated iterative differentiation. arXiv preprint arXiv:2302.044966 2023.
[54] P. Xing, S. Lu, L. Wu, and H. Yu. Big-fed: Bilevel optimization enhanced graph-aided federated learning.
[55] Yamada, M. Yukawa, and M. Yamagishi. Minimizing the moreau envelope of nonsmooth convex functions over the fixed point set of certain quasi-nonexpansive mappings. In Fixed-Point Algorithms for Inverse Problems in Science and Engineering, pages 345-390. Springer, 2011.
[56] J. Yang, K. Ji, and Y. Liang. Provably faster algorithms for bilevel optimization. arXiv preprint arXiv:2106.0 04692, 2021.
[57] S. Yang, X. Zhang, and M. Wang. Decentralized gossip-based stochastic bilevel optimization over communication networks. arXiv preprint 2:Xiv:2206.10877 2022.
12

---

[58] Y. Yang, P. Xiao, and K. Ji. Simfbo: Towards simple, flexible and communication-efieee federated bilevel learning. arXiv preprint arXiv:2305.19442 2023.
[59] H. Yu, Yang. and Zhu. Parallel restarted sgd with faster convergence and less communica- tion: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5693-5700, 2019.
[60] Y. Zhao, M. Li, L.. Lai, N. Suda, D. Civin, and V. Chandra. Federated learning with non-iid data. arXiv preprint arXiv: 00582, 2018.
[61] L. Zintgraf, K. Shiarli, V. Kurin, Hofmann, and S. Whiteson. Fast context adaptation via meta-learning. In International Conference on Machine Learning, pages 7693-7702. PMLR, 2019.
13


Reviews:
[{"review_id": "GLKjHLTAUc", "reviewer": "Reviewer_rgL1", "summary": "This paper proposed a communication efficient federated bilevel algorithm in both global and local lower-level problem setting that can achieve the best theoretical communication complexity. Extensive numerical experiments are provided to test the effectiveness of the proposed algorithm. ", "questions": "The three-level optimization perspective appears to be applicable to the local lower-level bilevel problem as well. Is there a specific reason for selecting the Neumann series approximation for the local lower-level problem, apart from the warm start strategy? ", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "They provide convergence analysis for the proposed algorithm. The numerical experiments are abundant and solid to verify the effectiveness of the proposed algorithm. Also, the federated bilevel with local lower-level problem is intriguing. ", "weaknesses": "1.\tThe analysis technique for bilevel method that treats the Hessian-vector product as a solution of a quadratic problem was also utilized in [R1]-[R5]. I’m afraid this would alleviate some novelty of this paper. Also, some relevant literature on fully single loop bilevel algorithms like [R1]-[R2] and communication efficient federated bilevel methods [R4]-[R5] are missing. Especially, [R4] allows for any heterogeneity without Assumption 3.5 and [R5] achieves the same communication complexity as this work, could you elaborate the pros and cons of your work and [R4]-[R5]?  \n\n\n[R1] Junyi Li, Bin Gu, and Heng Huang. A fully single loop algorithm for bilevel optimization without hessian inverse. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.\n\n[R2] Michael Arbel and Julien Mairal. Amortized implicit differentiation for stochastic bilevel optimization. arXiv\npreprint arXiv:2111.14580, 2021.\n\n[R3] Mathieu Dagreou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. In Proceedings of the Advances in Neural Information Processing Systems, 2022.\n\n[R4] Quan Xiao, Han Shen, Wotao Yin, and Tianyi Chen. \"Alternating Implicit Projected SGD and Its Efficient Variants for Equality-constrained Bilevel Optimization.\" arXiv preprint arXiv:2211.07096, 2022.\n\n[R5] Yifan Yang, Peiyao Xiao, and Kaiyi Ji. \"SimFBO: Towards Simple, Flexible and Communication-efficient Federated Bilevel Learning.\" arXiv preprint arXiv:2305.19442, 2023.\n\n2.\tThe role of heterogeneity is not expressly delineated in Theorem 3.6. It would be beneficial to see a how heterogeneity influences the convergence rate.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments. Below are our responses to your questions and concerns:\n\nResponse to weakness:\n\n**W1** : Although the idea of viewing hyper-gradient estimation as solving a quadratic problem is investigated in several papers, these paper only consider the non-distributed setting, and we are the first one to apply this idea into the federated setting. In fact, it is even more favorable to apply this idea into the FL setting than the non-distributed setting. Since the global quadratic problem is a linear combination of the client-wise quadratic problems, we can apply the standard FedAvg algorithm to get an unbiased estimation of the true hyper-gradient in a communication-efficient way, furthermore, its optimization can also be efficiently integrated with the update of upper and lower level variables to have a three-level federated optimization problem. Furthermore,  we achieve the optimal convergence rate by combining our three-level federated optimization formulation with the momentum-based variance reduction technique.\n\n**Comparison with [4]**: This work adopts a similar strategy as FedNest, where they first update the lower level variable with multiple steps and then update the upper level variable multiple steps, in contrast, our FedBiOAcc algorithm performs a simple alternative updates between the lower level variable, upper level variable and the hyper-gradient estimation variable. Furthermore, [4] adopts the recently proposed ProxSkip for the update of the lower level variable, thus does not need the heterogeneity assumption, however, the best algorithm E2-AiPOD in [4] only sub-optimal communication rate of $O(\\epsilon^{-1.5})$, while our FedBiOAcc achieves the optimal $O(\\epsilon^{-1})$ rate. **Comparison with [5]**: Similar to our algorithm, SimFBO also consider a three-level optimization problem, however, it only gets sample complexity of $O(\\epsilon^{-2})$ which is sub-optimal, while our FedBiOAcc achieves the optimal $O(\\epsilon^{-1.5})$ sample complexity.  Finally, the convergence analysis in both [4] and [5] does not include the dependence over the condition number $\\kappa$, which is an important factor to affect the performance of the algorithm in practice, and they don't consider the special case of local lower level problem, which can incorporate various Personalized FL formulations.\n\n**W2**: The term related to the heterogeneity coefficients are part of the constant factors absorbed by the Big O notation. Please refer to Line 715 and 728 in the manuscript for the precise form of the dependence, approximately, we have $\\frac{1}{T}\\sum_{t = 1}^{T-1} \\mathbb{E} \\bigg[ \\|\\nabla h(\\bar{x}_t) \\|^2 \\bigg] = O(\\zeta^2)$, and $\\zeta$ denotes the maximum of the heterogeneity coefficients defined in Assumption 3.5.\n\n**Q1**: For the case of local lower-level bilevel problem, the local hyper-gradient is an unbiased estimation of the global hyper-gradient, therefore, we don't need to solve an extra quadratic federated optimization problem to estimate the global hyper-gradient as in the case of global lower level problem. To estimate the local hyper-gradient, we can either use Neumann series or solve a quadratic problem and they both don't need extra communication. We choose the Neumann series approach due to the following reason: Neumann series does not need to keep an extra variable $u$ as the quadratic approach, furthermore,  the Neumann series approach leads to a good approximation of the hyper-gradient with only a few number of hessian-vector products in practice, thus the quadratic approach does not outperform the Neumann series approach significantly in computational efficiency. \n\nReferences\n\n[1]. Quan Xiao, Han Shen, Wotao Yin, and Tianyi Chen. \"Alternating Implicit Projected SGD and Its Efficient Variants for Equality-constrained Bilevel Optimization.\" arXiv preprint arXiv:2211.07096, 2022.\n\n[2]. Yifan Yang, Peiyao Xiao, and Kaiyi Ji. \"SimFBO: Towards Simple, Flexible and Communication-efficient Federated Bilevel Learning.\" arXiv preprint arXiv:2305.19442, 2023.\n"}, {"review_id": "8jAOLW9DX5", "reviewer": "Reviewer_5Ddx", "summary": "this paper proposed new algorithms for federated bilevel optimization. The authors apply the idea of [1] to the federated setting, which views the hypergradient computation as solving a quadratic subproblem. Combining with the STORM algorithm, the author proposed fedbioacc with a convergence guarantee based on this idea and then extend the results to a simpler case with local lower-level problems. numerical experiments on data cleaning and hyper representation are reported.\n\n[1] Dagréou et al. \"A framework for bilevel optimization that enables stochastic and global variance reduction algorithms\"", "questions": "see weakness", "limitations": "see weakness", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. the proposed algorithms are pure single-loop algorithms compared with other global federated bilevel algorithms.\n2. the authors provide convergence guarantees for both proposed algorithms.", "weaknesses": "1. One of my primary concerns is that single-loop algorithms in bilevel optimization perform worse than double-loop algorithms because of the hyper gradient estimation error introduced by the inexactness of the lower-level solution. this is different from the minimax problem because the hypergradient in bilevel optimization involves Jacobian and hessian inverse computation, thus being more complex. In practice, people are using a double-loop algorithm to reduce this hyper gradient estimation error.\n\n2. the contribution of this paper looks trivial to me, the main ideas come from STORM and [Dagréou et al]. the proposed algorithm is very complex and thus difficult to implement and tune in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for spending time reviewing our paper and provide comments. Below are our responses to your concerns:\n\n**W1**: We agree with the reviewer that bilevel optimization is harder than the minimax optimization problems. However, our three-level perspective of the bilevel optimization can eliminate the difficulty of dealing with the Hessian Inverse: we estimate the hyper-gradient by solving a quadratic problem, and then we perform alternative update between the upper level variable, lower level variable and the hyper-gradient estimation variable. In fact, we theoretically prove the convergence of our single loop algorithm and empirically verify its efficacy through the Federated Data Cleaning task and the Federated Hyper-representation task.\n\n**W2** Our main contributions are three folds: Firstly, we view the estimation of hyper-gradient as solving a federated quadratic problem and therefore can get the unbiased estimation of the hyper-gradient in a communication-efficient way; Secondly, by combining with the momentum-based variance reduction, we obtain the optimal convergence rate (both the sample complexity and the communication complexity)  and achieves linear speed up w.r.t. the number of clients. It is non-trivial to perform analysis which requires a careful trade-off of various types of estimation errors. Finally, we also consider the special case of local lower level problem for the first time, this type of federated bilevel problems has wide application over the personalized FL setting. \n\nAs for the complexity, our algorithm is well-structured and for each of the three problems: upper level, lower level and quadratic problem, we perform the same type of acceleration operation: momentum-based variance reduction. As a result, our algorithm is straightforward to implement. Furthermore,  in terms of the hyper-parameter tuning, we can first set the base learning rate $\\frac{\\delta}{u^{1/3}}$ to be around 0.1, then choose the learning rate coefficient $\\gamma$, $\\eta$, $\\tau$ respectively according to the structure of each problem, finally, we choose the momentum coefficient $c_{\\omega}$, $c_{\\nu}$ and $c_u$ such that the initial momentum to be around 0.9/0.99."}, {"review_id": "DHplyDdS3j", "reviewer": "Reviewer_MLKp", "summary": "This paper focuses on the problem of bilevel optimization in a federated learning or FL environment. Bilevel optimization has various FL applications, and a few recent works proposed versions of bilevel optimization schemes for FL. A challenging step in bilevel optimization is the computation of the \"hypergradient\", and the existing schemes are able to obtain an estimate of the hypergradient, albeit a biased estimate, with a substantial communication overhead (multiple rounds of communication per server-side update). The paper reformulates the hypergradient computation as a quadratic federated optimization problem that can provide an unbiased estimate of the hypergradient, while requiring only a single round of communication for every server-side update. Based on this insight, the paper proposes a federated bilevel optimization algorithm that updates the two variables in the bilevel problem, and the hypergradient estimate alternately on each client, with an intermittent averaging step across all clients. To improve the convergence with a constant batch size, the paper utilizes momentum based variance reduction techniques for all the three variables. The theoretical analysis establishes convergence with an improved iteration complexity compared to existing federated bilevel schemes, with a significantly improved communication complexity. Next, the paper studies an alternate form of a federated bilevel problem where all clients have local lower level variables, with applications in personalized FL. This setup allows for the local hypergradient computation without any need for communication across clients. The paper presents a modification of the previous algorithm for this problem and establishes convergence. The empirical evaluation considers two applications and compares the proposed scheme against various baselines, highlighting the improved communication complexity of the proposed scheme across various problem settings.", "questions": "- Line 2 in algorithm 1: Should it be $q_1^{(m)}$ since it is computed using $g^{(m)}$ and $f^{(m)}$?\n- Line 176-177: The ordering of the momentum terms and the corresponding variables in the \"... respectively\" needs to be updated.\n- In theorem 1, the results provide precise dependence on $\\kappa$, but I am unable to find the definition of $\\kappa$ in the theorem statement or anywhere else in the main paper. Can you please clarify what this $\\kappa$ is supposed to signify?", "limitations": "I did not find any discussion on limitations in the main paper or the supplement. However, I do not anticipate any potential negative societal impact of this work.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Critical federated hypergradient estimation as a federated optimization.**\nA key strength of this paper is a simple (yet of significant practical impact) reformulation of the hypergradient estimation using a standard quadratic program. A key property that the authors leverage is the fact that the global quadratic objective can be decomposed into per-client quadratic objectives, which is not true of the global hypergradient (which cannot be decomposed into per-client hypergradients). This simple yet powerful insight is then utilized to obtain an estimate which, upon proper solution of the global least-squares problem, is unbiased, and can be efficiently updated along side the upper and lower level variables in the bilevel problem. While this global least-squares reformulation does facilitate an intuitive communication-efficient algorithm, the paper also proposes a momentum-based variance reduction scheme, and establishes a very favorable convergence rate both in terms of iterations (and thus sample complexity) and communication. The overall algorithm makes the solution of federated bilevel problems significantly more practical.\n\n**Favorable convergence rate.**\nAlong with the novel federated hypergradient estimation, this paper is also able to get a favorable convergence rate across various dimensions. First, the results are able to provide convergence with a constant batch size, and establish a sample complexity of $O(\\epsilon^{-1.5})$, which is very strong for bilevel optimization. Second, the analysis establishes a $(1/M)$-dependence on the sample complexity, indicating that having more parties in the FL setup allows for smaller per-client sample complexity. Finally, as I discussed earlier, the analysis establishes a $O(\\epsilon^{-1})$ communication complexity which matches the rate of the best single-level federated optimization algorithm.\n\n\n**Bilevel personalized FL.**\nThe paper explicitly studies a form of the federated bilevel optimization where each client has its own lower level problem, and I believe this general problem would encompass various personalized FL problems and applications. While the problem is easier in terms of the computation of the hypergradient, it is still an interesting problem, and the paper establishes convergence again with favorable iteration and communication complexities.\n\n\n**Positioning against existing literature.**\nIn addition to the section on related works, the authors do a great job at discussing their proposed scheme in comparison to existing literature during the algorithm development. For example, after presenting the main idea behind the communication efficient hypergradient computation, the paper positions this work in comparison to FedNest.\n\n**Intuitive presentation of the theoretical results.**\nIn addition to establishing favorable theoretical guarantees, the authors also present the theoretical results in an intuitive manner. First, they clearly present the Lyapunov function, highlighting the different terms in it corresponding to the different variable iterates. Then the relevant parts of the Appendix are referenced which makes it easy to verify the results while reading. After the main theorem is presented, the authors also do a good job of highlighting what it implies, and how it compares to other federated algorithms.\n\n", "weaknesses": "**Increased hyperparameter space.**\nThe proposed framework utilizes various hyperparameters such as the initial learning rates $\\gamma, \\eta, \\tau$, the learning rate schedule $\\lbrace \\alpha_t, t \\in [T] \\rbrace$, the momentum parameters $c_{\\omega / \\nu / u}$, the aggregation frequency $I$.\nAs per the theoretical analyses, it can be seen that the best convergence rate of any execution will critically depend on an appropriate setting of these problem-dependent hyperparameters. Since these hyperparameters often depend on quantities that cannot be efficiently estimated (such as Lipschitz constant), the practical bilevel implementations usually utilize some form of hyperparameter search. Hyperparameter optimization is known to be a hard unsolved problem in FL because of the overall communication overhead. This makes it hard to see how the proposed federated bilevel framework can live up to its practical potential -- one can view this proposed federated bilevel framework as having shifted the communication overhead from the model training stage to the hyperparameter optimization stage, without reducing the overall communication necessary for good training convergence (which involves trying various hyperparameters and training with them).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed and insightful comments.\n\nResponse to Weakness: \n\nWe agree with the reviewer over the increased hyper-parameter space. In fact, the optimal convergence rate of the momentum-based variance reduction technique is achieved by carefully balancing the learning rate, momentum coefficients. Next, since we view the Federated Bilevel Optimization problem as three intertwined federated optimization problems, naturally, the number of hyper-parameters is roughly three times compared to that of its non-distributed single level counterpart. In practice, we can first set the base learning rate $\\frac{\\delta}{u^{1/3}}$ to be around 0.1, then choose the learning rate coefficient $\\gamma$, $\\eta$, $\\tau$ respectively according to the structure of each problem, finally, we choose the momentum coefficient $c_{\\omega}$, $c_{\\nu}$ and $c_u$ such that the initial momentum to be around 0.9/0.99.\n\n\nResponse to Questions:\n\n**Q1**: Yes, there should be a super-script, i.e. $q_1^{(m)}$;\n\n**Q2**: Yes, the order of $\\omega^{(m)}_t$ and $\\nu^{(m)}_t$ should be adjusted;\n\n**Q3**: $\\kappa = \\frac{L}{\\mu}$ denotes the condition number, and $L$ is the smoothness coefficient, $\\mu$ is the strong-convexity coefficient for the lower problem."}, {"review_id": "f5KVA6G54D", "reviewer": "Reviewer_nAuw", "summary": "This paper proposes an algorithm named FedBiOAcc that solves the federated bilevel optimization problem. In particular, the hyper-gradient of the bilevel optimization problem is evaluated by solving a reformulated quadratic federated optimization problem. The upper-level problem, lower-level problem and the hyper-gradient estimation are optimized in an alternative manner. A rigorous convergence analysis of the algorithm is provided. Also, the paper novelly introduces a new problem of bilevel optimization with local lower-level problems and proves the same convergence rate for a FedBiOAcc-Local algorithm that adapts to this setting.", "questions": "1. While bilevel federated optimization is technically challenging, what is the motivation to solve bilevel problems in a federated manner? There might need more concrete motivations and a clear, practical application or usage. Is there a need for distributed computing resources in such problems? Is there a requirement to ensure data privacy for bilevel optimization practical problems?\n2. The convergence analysis of the paper claims linear convergence speed-ups with the number of clients $M$. Does the convergence always become faster with the number of clients? For example, it is possible that the clients are heterogeneous. Also, does the dataset size of each client affect the convergence? It is hard to believe that given a fixed total number of samples, distributing them among more clients makes learning easier/faster.\n3. [Minor] Some notations used in the paper were not explicitly explained before the first appearance. For example, the $\\tau$ and $\\mathcal{B}$ in Line 149. Also, what do the subscripts mean for $\\mathcal{B}$?", "limitations": "I do not find any particular limitations in the proposed method.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Reformulation of the general federated bilevel optimization into an easier-to-solve federated quadratic optimization problem is interesting and sound.\n2. The convergence analysis of both algorithms in the heterogeneous case is complex and sound.\n3. The paper introduces a new problem setting with local lower-level problems, which could be even more challenging to solve. The algorithm and analysis are adapted to this new setting.\n4. The formulations of the data cleaning task and the hyper-representation learning task into federated bilevel optimization problems are reasonable.", "weaknesses": "1. The motivation of the proposed problem can be better instantiated with actual needs or examples in real-life.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for spending time reviewing our manuscript and providing insightful comments. Below are our responses to your questions:\n\nResponse to Weakness:\n\nMany real-world applications in FL exhibit a nested structure, and therefore can be solved as a federated bilevel problem. Firstly, the data-cleaning task in the FL setting can be formulated as a bilevel problem, as we discussed in section 5.1 of the manuscript. Next, various formulations of the Personalized FL has a bilevel structure: in our manuscript, we consider the hyper-representation approach in Section 5.2, where all clients jointly learn a backbone network, and each client individually learns a linear classifier with its local data. Furthermore, another way of Personalized FL is by training a subnetwork on each client, meanwhile we need train a mask for each client to do the selection.\n\nResponse to Questions:\n\n**Q1**: Please refer to our response to the weakness above for some real world FL applications that are bilevel.\n\n**Q2**:  The linear speed-up w.r.t. the number of clients achieves under certain conditions. Firstly, by observing the second term of the convergence bound in Theorem 3.6, we can see the number of iterations actually scales with the **product of batch size and number of clients**, which means if the total number of samples are fixed, distributing them to more clients might not improve the convergence speed, as the batch-size might be decreased. Secondly, the linear speedup conclusion still holds when the heterogeneity exists, however, note that the heterogeneity coefficients exist in the constant factors of the convergence bound in Theorem 3.6. If we take this constant factor explicitly, we have $ \\frac{1}{T}\\sum_{t = 1}^{T-1} \\mathbb{E} \\bigg[ \\|\\nabla h(\\bar{x}_t) \\|^2 \\bigg] = O(\\frac{\\zeta^2\\kappa^{16/3}}{(bMT)^{2/3}})$, and $\\zeta$ denotes the heterogeneity coefficients. This equation shows that larger heterogeneity level leads to worse convergence bound, although increasing the number of clients $M$ still decrease the bound.\n\n**Q3**: $\\tau$ is the step size/learning rate to solve the quadratic problem (its value choice be found in Theorem 3.6), $\\mathcal{B}$ represents mini-batch of samples used at each iteration, furthermore, we use the subscripts to differentiate the samples that are used to estimate different properties. For example, In line 149, $\\mathcal{B}_{g,2}$ represents samples used for the estimation of hessian vector product. Note that, to guarantee the convergence, it is essential to use independent samples to estimate different properties (gradient, hessian-vector product etc.) at each iteration.\n"}, {"review_id": "WP18ygtZJy", "reviewer": "Reviewer_Cbyn", "summary": "The paper studied the federated bilevel optimization problems, in which the upper level losses are non convex and lover level losses are strongly convex. Two algorithms, named as FedBiOAcc and FedBiOAcc-Local,  were proposed for whether the lower level problems requires to be consensus or not. The authors provided both communication can computation complexity bounds for the proposed algorithms. Two real-world tasks are adopted to validate the theoretical results.", "questions": "* It is a good idea to estimate the hyper-gradient via solving the federated quadric problem. \n   * However, during the updating iteration, both $x_t$ and $y_t$ will be changing, and thus $l$. It leaves me concern that whether $u_t$ would converge. \n   * Also how would the initialization of $u_t$ affect the algorithm convergence? \n   * in line 161, would $u^*$ depend on $t$? \n   * BTW, would you remind me the meaning of subscript of $2$ in $\\nabla l^{(m)}$ in line 149? It looks like the definition is missing.\n\n* The paper leveraged the STORM type of variance reduction to accelerate the convergence. However, in the original STORM [1], as well as its variant SUSTAIN [2], the initial batch size is a fixed number. Why does it need $O(M^{-1}\\epsilon^{-0.5})$ initial batch size?\n\n* The numerical studies looks somehow unfair to me.\n  *  In terms of communication, most of the other algorithms only transmit two vectors $x_t$ and $y_t$ but FedBiOAcc needs three vectors $x_t$, $y_t$ and $u_t$. Instead of looking at communication rounds, would the communicated bytes be a more fair metric to measure the communication loads? \n  * The authors also didn't report the computation cost. Rather than the simple convergence rate (which omit lots of key factors), the computation time / number of gradient/Hessian computation would be a better metric?\n  * BTW, what is the FedBio? It doesn't have any description. What's the different between FedBio and FedBioAcc?\n\nSome minor questions\n* Why doesn't the paper also include local-upper-shared-lower level problems?\n* Would the results or algorithms be extended to the case with strongly convex / convex upper level problem?\n\n[1] Momentum-based variance reduction in non-convex sgd\n[2] A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentum", "limitations": "The authors didn't mention any limitation and potential negative societal impact.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The authors studied the bilevel optimization problems under federated learning framework and provided sufficient research motivation and literature review. They proposed recovering the hyper-gradient via solving a federated optimization problem, which is novel. ", "weaknesses": "There have been many existing works on federated/distributed bilevel optimization. The contribution of this paper might not be significant. Also, the numerical comparison needs to be more fair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for spending time reviewing our manuscript. Below are our responses to your concerns and questions:\n\nResponse to Weakness:\n\nOur main contributions are three folds: Firstly, we view the estimation of hyper-gradient as solving a federated quadratic problem and therefore can get the unbiased estimation of the hyper-gradient in a communication-efficient way; Secondly, by combining with the momentum-based variance reduction, we obtain the optimal convergence rate (both the sample complexity and the communication complexity) and achieves linear speed up w.r.t. the number of clients. It is non-trivial to perform analysis which requires a careful trade-off of various types of estimation errors. Finally, we also consider the special case of local lower level problem for the first time, this type of federated bilevel problems has wide application over the personalized FL setting.\n\nResponse to Questions:\n\n**Q1**: The estimation error to $x_t$, $y_t$ and $u_t$ are intertwined with each other (in particular, please refer to Lemma C.6 for a bound to the estimation error of $u_t$), and each of them is chasing a moving target. However, as shown by Theorem 3.6, $x_t$ converges to a stationary point of $h(x)$, then naturally, both $y_t$ and $u_t$  also converge as $x_t$ stops updating.\n\nThe initial value of $u_t$ affects the convergence, and it appears in the constant factor of the bound in Theorem 3.6. Please refer to Line 728 in the manuscript for its explicit dependence.\n\n$u^*$ depends on t, as it is a function of the current state of the upper level variable $x_t$ as defined in Eq. 4.\n\nIn line 149, we use the subscripts to differentiate the samples that are used to estimate different properties. For example, In line 149, \n$\\mathcal{B}_{g,2}$ represents samples used for the estimation of hessian vector product.\n\n**Q2**: The large initial batch size is needed to mitigate the effect of client-drift error, which is generated when clients perform local updates. Note that the STEM [1] method which applies STORM to the single level FL problem also requires a large initial batch size to reach the optimal convergence rate (See corollary 1 in [1]).\n\n**Q3**:  the metric of communication rounds is widely used in Federated Optimization [1] and Federated Bilevel Optimization [2,3] works for comparison of different methods. In particular, the state-of-the-art Federated Bilevel Optimization method FedNest [2] also uses communication round for comparison, and we follow the same experimental setting as their work. In fact, the communication rounds is consistent with the theoretical analysis where we hide the constant factors.\n\nWe use FedBiO to refer to the algorithm without using momentum-based variance reduction, and its update rule is shown in Eq.6 of the manuscript, where we update the lower variable, upper variable and hyper-gradient estimation variable alternatively.\n\n**Minor points**: The case of local-upper problem is also interesting, and our FedBiO/FedBiOAcc can be extended to this case by not averaging the upper level variable, we leave a more detailed discussion of this case as a future work. We can follow the same analysis framework to analyze the simpler strongly-convex/convex upper level problem case, and we also leave this as a future work.\n\nReferences\n\n[1] Khanduri, Prashant, et al. \"Stem: A stochastic two-sided momentum algorithm achieving near-optimal sample and communication complexities for federated learning.\" Advances in Neural Information Processing Systems 34 (2021): 6050-6061.\n\n[2] Tarzanagh, Davoud Ataee, et al. \"Fednest: Federated bilevel, minimax, and compositional optimization.\" International Conference on Machine Learning. PMLR, 2022.\n\n[3] Yang, Yifan, Peiyao Xiao, and Kaiyi Ji. \"SimFBO: Towards Simple, Flexible and Communication-efficient Federated Bilevel Learning.\" arXiv preprint arXiv:2305.19442 (2023)."}]

Author rebuttal:

Meta-review: The main strength and weakness of this paper are summarized as follows.

**Strength**:

1. The proposed method is single-loop and use a small batch size (except the initial one). 

2. The communication complexity and sample complexity are the best in literature, and the linear speed-up is achieved. They are even better than the most recent method, called SimFBO, in terms of sample complexity. However, **I think the authors must cite SimFBO and include it in the table 1.**

 Yifan Yang, Peiyao Xiao, and Kaiyi Ji. "SimFBO: Towards Simple, Flexible and Communication-efficient Federated Bilevel Learning." arXiv preprint arXiv:2305.19442, 2023


**Weakness**:

1. The high-level idea is not new. Solving the Hessian-vector product as a solution of a federated quadratic problem is not new. The algorithm proposed is essentially solving three intertwined federated optimization problems using STORM momentum-based variance reduction like in [Dagréou et al] for example. As pointed out by multiple reviewers, these techniques seem well-known. 

2. The single-loop method seem to not perform as well as a double-loop method in practice. 

3. There are too many tuning parameters and thus hard to implement. 


There are also some relatively minors issues not listed above. 

My overall evaluation is that the contribution of this work outweighs its limitation. After all, **this work achieves the best sample complexity and communication complexity simultaneously in literature.** This work seems to be the new SOTA given its theoretical guarantees.
Decision: Accept (poster)