SlowGuess/NeurIPS_2023_OCR huggingface dataset:
an Yang, Peiyao Xiao and Kaiyi Ji
{yyang99, peiyaoxi, kaiyiji}@buffalo.edu
Federated bilevel optimization (FBO) has shown great potential recently in machine learning and edge computing due to the emerging nested optimization structure in meta-learning, fine-tuning, hyperparameter tuning, etc. However, existing FBO algorithms often involve complicated computations and require multiple sub-loops per iteration, each of which contains a number of communication rounds. In this paper, we propose a simple and flexible FBO framework named SimFBO, which is easy to implement without sub-loops, and includes a generalized server-side aggregation and update for improving communication efficiency. We further propose System-level heterogeneity robust FBO (ShroFBO) as a variant of SimFBO with stronger resilience to heterogeneous local computation. We show that SimFBO and ShroFBO provably achieve a linear convergence speedup with partial client participation and client sampling without replacement, as well as improved sample and communication complexities. Experiments demonstrate the effectiveness of the proposed methods over existing FBO algorithms.
Recent years have witnessed significant progress in a variety of emerging areas including meta-learning and fine-tuning [11, 52], automated hyperparameter optimization [13, 10], reinforcement learning [31, 21], fair batch selection in machine learning [54], adversarial learning [76, 40], AI-aware communication networks [27], fairness-aware federated learning [75], etc. These problems share a common nested optimization structure, and have inspired intensive study on the theory and algorithmic development of bilevel optimization. Prior efforts have been taken mainly on the single-machine scenario. However, in modern machine learning applications, data privacy has emerged as a critical concern in centralized training, and the data often exhibit an inherently distributed nature [70]. This highlights the importance of recent research and attention on federated bilevel optimization, and has inspired many emerging applications including but not limited to federated meta-learning [9], hyperparameter tuning for federated learning [25], resource allocation over communication networks [27] and graph-aided federated learning [71], adversarial robustness on edge computing [46], etc. In general, the federated bilevel optimization problem takes the following mathematical formulation.
$$
\min  _ {x \in \mathbb {R} ^ {p}} \Phi (x) = F \left(x, y ^ {*} (x)\right) := \sum_ {i = 1} ^ {n} p _ {i} f _ {i} (x, y ^ {*} (x)) = \sum_ {i = 1} ^ {n} p _ {i} \mathbb {E} _ {\xi} \left[ f _ {i} \left(x, y ^ {*} (x); \xi_ {i}\right) \right]
$$
$$
\text {s . t .} y ^ {*} (x) = \underset {y \in \mathbb {R} ^ {q}} {\arg \min } G (x, y) := \sum_ {i = 1} ^ {n} p _ {i} g _ {i} (x, y) = \sum_ {i = 1} ^ {n} p _ {i} \mathbb {E} _ {\zeta} \left[ g _ {i} (x, y; \zeta_ {i}) \right] \tag {1}
$$
Figure 1: Comparison of AID-based federated hypergradient estimation (FHE) in FedNest [65] (left), ITD-based FHE in AggITD [69] (middle) and our proposed SimFBO (right) at each iteration.
where  $n$  is the total number of clients, the outer- and inner-functions  $f_{i}(x,y)$  and  $g_{i}(x,y)$  for each client  $i$  take the expectation forms w.r.t. the random variables  $\xi_{i}$  and  $\zeta_{i}$ , and are jointly continuously differentiable. However, efficiently solving the federated problem in eq. (1) suffers from several main challenges posed by the federated hypergradient (i.e.,  $\nabla \Phi(x)$ ) computation that contains the second-order global Hessian-inverse matrix, the lower- and upper-level data and system-level heterogeneity, and the nested optimization structure. To address these issues, [25, 65, 24, 22] proposed approximate implicit differentiation (AID)-based federated bilevel algorithms, which applied the idea of non-federated AID-based estimate in [15] to the federated setting, and involve two sub-loops for estimating the global lower-level solution  $y^{*}(x)$  and the Hessian-inverse-vector product, respectively. [69] then proposed AggITD by leveraging the idea of iterative differentiation, which improved the communication efficiency of AID-based approaches by synthesizing the lower-level optimization and the hypergradient computation into the same communication sub-loop. However, some limitations still remain in these approaches.
- First, the sub-loops, each with a large number of communication rounds, often compute products of a series of matrix-vector products, and hence can complicate the implementation and increase the communication cost.
- Second, the practical client sampling without replacement has not been studied in these methods due to challenges posed by the nested structure of AID- and ITD-based federated hypergradient estimators.
- Third, as observed in the single-level federated learning [67], in the presence of heterogeneous system capabilities such as diverse computing power and storage, clients can take a variable number of local updates or use different local optimizers, which may make these FBO algorithms converge to the stationary point of a different objective.
In this paper, we propose a communication-efficient federated bilevel method named SimFBO, which is simple to implement without sub-loops, flexible with a generalized server-side update, and resilient to system-level heterogeneity. Our specific contributions are summarized below.
A simple and flexible implementation. As illustrated in Figure 1, differently from AID- and ITD-based approaches that contain multiple sub-loops of communication rounds at each iteration, our proposed SimFBO is simpler with a single communication round per iteration, in which three variables  $y$ ,  $x$  and  $v$  are updated simultaneously for optimizing the lower- and upper-level problems, and approximating the Hessian-inverse-vector product. SimFBO also includes a generalized server-side update on  $x$ ,  $y$ ,  $v$ , which accommodates the client sampling without replacement, and allows for a flexible aggregation to improve the communication efficiency.
Resilient server-side updates to system-level heterogeneity. In the presence of heterogeneous local computation, we show that the naive server-side aggregation can lead to the convergence to a stationary point of a different objective. To this end, we propose System-level heterogeneity robust FBO (ShroFBO) building on a normalized version of the generalized server-side update with correction, which provably converges to a stationary point of the original objective.
Algorithm	System-level heterogeneity	Partial participation	Without replacement	Linear speedup	Samples complexity	Communication complexity
FedNest [65]	X	X	X	X	O(ε-2)	O(ε-2)
FBO-AggITD [69]	X	X	X	X	O(ε-2)	O(ε-2)
FedBiO [36]	X	X	X	✓	O(ε-2.5n-1)	O(ε-1.5)
FedMBO [24]	X	✓	X	✓	O(ε-2P-1)	O(ε-2)
SimFBO (this paper)	X	✓	✓	✓	O(ε-2P-1)	O(ε-1)
ShroFBO (this paper)	✓	✓	✓	✓	O(ε-2P-1)	O(ε-1)
Table 1: Comparison of different federated bilevel algorithms in the setting with heterogeneous data. We do not include the methods with momentum-based acceleration for a fair comparison.  $P \leq n$  is the number of sampled clients in each communication round.
Convergence analysis and improved complexity. As shown in Table 1, our SimFBO and ShroFBO both achieve a sample complexity (i.e., the number of samples needed to reach an  $\epsilon$ -accurate stationary point) of  $\mathcal{O}(\epsilon^{-2}P^{-1})$ , which matches the best result obtained by FedMBO [24] but under a more practical client sampling without replacement. Moreover, SimFBO and ShroFBO both achieve the best communication complexity (i.e., the number of communication rounds to reach an  $\epsilon$ -accurate stationary point) of  $\mathcal{O}(\epsilon^{-1})$ , which improves those of other methods by an order of  $\epsilon^{-1/2}$ . Technically, we develop novel analysis in characterizing the client drifts by the three variables, bounding the per-iteration progress in the global  $y$  and  $v$  updates, and proving the smoothness and bounded variance in local  $v$  updates via induction, which may be of independent interest.
Superior performance in practice. In the experiments, the proposed SimFBO method significantly improves over existing strong federated bilevel baselines such as AggITD, FedNest and LFedNest in both the i.i.d. and non-i.i.d. settings. We also validate the better performance of ShroFBO in the presence of heterogeneous local computation due to the resilient server-side updates.
2 SimFBO: A Simple and Flexible Framework
2.1 Preliminary: Federated Hypergradient Computation
The biggest challenge in FBO is to compute the federated hypergradient  $\nabla \Phi (x)$  due to the implicit and complex dependence of  $y^{*}(x)$  on  $x$ . Under suitable assumptions and using the implicit function theorem in [18], it has been shown that the  $\nabla \Phi (x)$  takes the form of
$$
\nabla \Phi (x) = \sum_ {i = 1} ^ {n} p _ {i} \nabla_ {x} f _ {i} (x, y ^ {*}) - \nabla_ {x y} ^ {2} G (x, y ^ {*}) \left[ \nabla_ {y y} ^ {2} G (x, y ^ {*}) \right] ^ {- 1} \sum_ {i = 1} ^ {n} p _ {i} \nabla_ {y} f _ {i} (x, y ^ {*})
$$
which poses several computational challenges in the federated setting. First, the second term at the right-hand side contains three global components in a nonlinear manner, and hence the direct aggregation of local hypergradients given by
$$
\sum_ {i = 1} ^ {n} p _ {i} \left(\nabla_ {x} f _ {i} (x, y ^ {*}) - \nabla_ {x y} ^ {2} g _ {i} (x, y ^ {*}) \left[ \nabla_ {y y} ^ {2} g _ {i} (x, y ^ {*}) \right] ^ {- 1} \nabla_ {y} f _ {i} (x, y ^ {*})\right)
$$
is a biased estimation of  $\nabla \Phi(x)$  due to the client drift. Second, it is infeasible to compute, store and communicate the second-order Hessian-inverse and Jacobian matrices due to the limited computing and communication resource. Although various AID- and ITD-based approaches have been proposed to address these challenges, they still suffer from several limitations (as we point out in the introduction) such as complicated implementation, high communication cost, lack of client sampling without replacement, and vulnerability to the system-level heterogeneity. To this end, we propose a simple, flexible and communication-efficient FBO framework named SimFBO in this section.
2.2 Federated Hypergradient Surrogate
To estimate the federated hypergradient efficiently, we use the surrogate  $\bar{\nabla} F(x,y,v) = \nabla_xF(x,y) - \nabla_{xy}^2 G(x,y)v$ , where  $v\in R^{d_y}$  is an auxiliary vector. Then, it suffices to find  $y$  and  $v$  as efficient
estimates of the solutions to the global lower-level problem and the global linear system (LS)  $\nabla_{yy}^2 G(x,y)v = \nabla_yF(x,y)$  that is equivalent to solving following quadratic programming.
$$
\begin{array}{l} \min  _ {v} R (x, y, v) = \frac {1}{2} v ^ {T} \nabla_ {y y} ^ {2} G (x, y) v - v ^ {T} \nabla_ {y} F (x, y) \\ = \sum_ {i = 1} ^ {n} p _ {i} \left(\underbrace {\frac {1}{2} v ^ {T} \nabla_ {y y} ^ {2} g _ {i} (x , y) v - v ^ {T} \nabla_ {y} f _ {i} (x , y)} _ {R _ {i} (x, y, v)}\right), \tag {2} \\ \end{array}
$$
where  $R_{i}(x,y,v)$  can be regarded as the loss function of client  $i$  for solving this global LS problem. Based on this surrogate, we next describe the proposed SimFBO framework.
2.3 Simple Local and Server-side Aggregations and Updates
Simple local update. Differently from FedNest [65] and AggITD [69] that perform the lower-level optimization, the federated hypergradient estimation and the upper-level update alternatively in different communication sub-loops, our SimFBO conducts the simple updates on all these three procedures simultaneously in each communication round. In specific, each communication round  $t$  first selects a subset  $C^{(t)}$  of participating clients without replacement. Then, each active client  $i \in C^{(t)}$  updates three variables  $y, v, x$  at  $k^{th}$  local iteration simultaneously as
$$
\left( \begin{array}{l} y _ {i} ^ {(t, k + 1)} \\ v _ {i} ^ {(t, k + 1)} \\ x _ {i} ^ {(t, k + 1)} \end{array} \right) \leftarrow \left( \begin{array}{l} y _ {i} ^ {(t, k)} \\ v _ {i} ^ {(t, k)} \\ x _ {i} ^ {(t, k)} \end{array} \right) - a _ {i} ^ {(t, k)} \left( \begin{array}{c} \eta_ {y} \nabla_ {y} g _ {i} \left(x _ {i} ^ {(t, k)}, y _ {i} ^ {(t, k)}; \zeta_ {i} ^ {(t, k)}\right) \\ \eta_ {v} \nabla_ {v} R _ {i} \left(x _ {i} ^ {(t, k)}, y _ {i} ^ {(t, k)}, v _ {i} ^ {(t, k)}; \psi_ {i} ^ {(t, k)}\right) \\ \eta_ {x} \bar {\nabla} f _ {i} \left(x _ {i} ^ {(t, k)}, y _ {i} ^ {(t, k)}, v _ {i} ^ {(t, k)}; \xi_ {i} ^ {(t, k)}\right) \end{array} \right) \tag {3}
$$
where  $\eta_y, \eta_v, \eta_x$  correspond to the local stepsizes,  $a_i^{(t,k)}$  is a client-specific coefficient to increase the flexibility of the framework,  $\zeta_i^{(t,k)}, \psi_i^{(t,k)}, \xi_i^{(t,k)}$  are independent samples, and the local hypergradient estimate takes the form of  $\nabla f_i(x,y,v;\xi) = \nabla_x f_i(x,y;\xi) - \nabla_{xy}^2 g_i(x,y;\xi)v_i$ . The variables  $y, v$  and  $x$  in eq. (3), which optimize the lower-level problem, the LS problem and the upper-level problem, are updated with totally  $\tau_i^{(t)}$  local steps. Note that the updates in eq. (3) also allow for parallel computation on  $x, v$  and  $y$  locally.
Local and server-side aggregation. After completing all local updates, the next step is to aggregate such local information on both the client and server sides. As shown in eq. (4), each participating client  $i \in C^{(t)}$  aggregates all the local gradients, and then communicate the aggregations  $q_{y,i}^{(t)}, q_{v,i}^{(t)}$  and  $q_{x,i}^{(t)}$  to the server. Then, on the server side, such local information is further aggregated to be  $q_{y}^{(t)}, q_{v}^{(t)}$  and  $q_{x}^{(t)}$ , which will be used for a subsequent generalized server-side update.
$$
q _ {y} ^ {(t)} = \sum_ {i \in C ^ {(t)}} \widetilde {p} _ {i} q _ {y, i} ^ {(t)} = \sum_ {i \in C ^ {(t)}} \widetilde {p} _ {i} \sum_ {k = 0} ^ {\tau_ {i} - 1} a _ {i} ^ {(t, k)} \nabla_ {y} g _ {i} \big (x _ {i} ^ {(t, k)}, y _ {i} ^ {(t, k)}; \zeta_ {i} ^ {(t, k)} \big),
$$
$$
q _ {v} ^ {(t)} = \sum_ {i \in C ^ {(t)}} \widetilde {p} _ {i} q _ {v, i} ^ {(t)} = \sum_ {i \in C ^ {(t)}} \widetilde {p} _ {i} \sum_ {k = 0} ^ {\tau_ {i} - 1} a _ {i} ^ {(t, k)} \nabla_ {v} R _ {i} \big (x _ {i} ^ {(t, k)}, y _ {i} ^ {(t, k)} v _ {i} ^ {(t, k)}; \psi_ {i} ^ {(t, k)} \big),
$$
$$
\underbrace {q _ {x} ^ {(t)} = \sum_ {i \in C ^ {(t)}} \widetilde {p} _ {i} q _ {x , i} ^ {(t)}} _ {\text {S e r v e r a g g r e g a t i o n}} = \sum_ {i \in C ^ {(t)}} \widetilde {p} _ {i} \underbrace {\sum_ {k = 0} ^ {\tau_ {i} - 1} a _ {i} ^ {(t , k)} \bar {\nabla} f _ {i} \left(x _ {i} ^ {(t , k)} , y _ {i} ^ {(t , k)} , v _ {i} ^ {(t , k)} ; \xi_ {i} ^ {(t , k)}\right)} _ {\text {L o c a l a g g r e g a t i o n}}, \tag {4}
$$
where  $\widetilde{p_i} := \frac{n}{|C^{(t)}|} p_i$  is the effective weight of client  $i \in C^{(t)}$  among all participating clients such that  $\mathbb{E}\left(\sum_{i \in C^{(t)}} \widetilde{p_i}\right) = 1$ . Note that in eq. (4), the local aggregation  $q_{y,i}^{(t)}$  (similarly for  $v$  and  $x$ ) can be regarded as a linear combination of all local stochastic gradients, and hence covers a variety of local optimizers such as stochastic gradient descent, momentum-based gradient, variance reduction by choosing different coefficients  $a_i^{(t,k)}$  for  $i \in C^{(t)}$ . This substantially enhances the flexibility of the proposed framework.
Algorithm 1 SimFBO and ShroFBO
1: Input: initialization  $\pmb{y}^{(0)},\pmb{v}^{(0)},\pmb{x}^{(0)}$  , number of communication rounds  $T$  , learning rates: client  $\{\eta_y,\eta_v,\eta_x\}$  server:  $\{\gamma_{y},\gamma_{v},\gamma_{x}\}$  , local update rounds:  $\{\tau_i^{(t)}\}$    
2: for  $t = 0,1,2,\dots,T$  do   
3: for  $i\in C^{(t)}$  in parallel do   
4:  $y_{i}^{(t,0)} = y^{(t)},v_{i}^{(t,0)} = v^{(t)},x_{i}^{(t,0)} = x^{(t)}$    
5: for  $k = 0,1,2,\ldots,\tau_i^{(t)} - 1$  do   
6: Locally update  $y_{i}^{(t,k)},v_{i}^{(t,k)}$  and  $x_{i}^{(t,k)}$  simultaneously via eq. (3)   
7: end for   
8: Client i locally aggregates gradients to compute  $q_{y,i}^{(t)},q_{v,i}^{(t)},q_{x,i}^{(t)}$  via eq. (4)   
9: Client i locally aggregates gradients to compute  $h_{y,i}^{(t)},h_{v,i}^{(t)},h_{x,i}^{(t)}$  defined in eq. (6)   
10: end for   
11: Client  $i\in C^{(t)}$  communicate  $\{q_{y,i}^{(t)},q_{v,i}^{(t)},q_{x,i}^{(t)}\}$  or  $\{h_y^{(t)},h_v^{(t)},h_x^{(t)}\}$  to the server   
12: Server aggregates local estimators to compute  $\{q_y^{(t)},q_v^{(t)},q_x^{(t)}\}$  using eq. (4)   
13: Server aggregates local estimators to compute  $\{h_y^{(t)},h_v^{(t)},h_x^{(t)}\}$  using eq. (7)   
14: Server updates using eq. (5)   
15: Server updates using eq. (8)   
16: end for
Server-side updates. Based on the aggregated gradients  $q_{y}^{(t)}$ ,  $q_{v}^{(t)}$  and  $q_{x}^{(t)}$ , we then perform server-level gradient-based updates on variables  $x, v$  and  $y$  simultaneously as
$$
y ^ {(t + 1)} = y ^ {(t)} - \gamma_ {y} q _ {y} ^ {(t)}, v ^ {(t + 1)} = \mathcal {P} _ {r} \left(v ^ {(t)} - \gamma_ {v} q _ {v} ^ {(t)}\right), x ^ {(t + 1)} = x ^ {(t)} - \gamma_ {x} q _ {x} ^ {(t)}, \tag {5}
$$
where  $\gamma_y, \gamma_v$  and  $\gamma_x$  are server-side updating step sizes for  $y, v, x$  and  $\mathcal{P}_r(v) \coloneqq \min \left\{1, \frac{r}{\|v\|} \right\} v$  is a simple projection on a bounded ball with a radius of  $r$ . There are a few remarks about the updates in eq. (5). First, in contrast to existing FBO algorithms such as [65, 69], our introduced server-side updates leverage not only the client-side step sizes  $\eta_y, \eta_v, \eta_x$ , but also the server-side step sizes  $\gamma_y, \gamma_v$  and  $\gamma_x$ . This generalized two-learning-rate paradigm can provide more algorithmic and theoretical flexibility, and provides improved communication efficiency in practice and in theory. Second, the projection  $\mathcal{P}_r(\cdot)$  serves as an important step to ensure the boundedness of variable  $v^{(t)}$ , and hence guarantee the smoothness of the global LS problem and the boundedness of the estimation variance in  $v$  and  $x$  updates, both of which are crucial and necessary in the final convergence analysis. Note that we do not impose such projection on the local  $v_i^{(t,k)}$  variables because we can prove via induction that they are bounded given the boundedness of  $v^{(t)}$  (see Proposition 1).
2.4 Resilient Server-side Updates against System-level Heterogeneity
Limitations under system-level heterogeneity. When clients have heterogeneous computing and storing capabilities (e.g., computer server v.s. phone in edge computing), an unequal number of local updates are often performed such that the global solution can be biased toward those of the clients with much more local steps or stronger optimizers. As observed in [67], this heterogeneity can deviate the iterates to minimize a different objective function. To explain this mismatch phenomenon, inspired by [57], we rewrite the server-side update on  $x$  (similarly for  $v$  and  $y$ ) in eq. (4) as
$$
q _ {x} ^ {(t)} = \sum_ {i = 1} ^ {n} p _ {i} q _ {i} ^ {(t, k)} = \underbrace {\left(\sum_ {j = 1} ^ {n} p _ {j} \| a _ {j} ^ {(t)} \| _ {1}\right)} _ {\rho^ {(t)}} \sum_ {i = 1} ^ {n} \underbrace {\frac {p _ {i} \| a _ {i} ^ {(t)} \| _ {1}}{\sum_ {j = 1} ^ {n} p _ {j} \| a _ {j} ^ {(t)} \| _ {1}}} _ {w _ {i}} \underbrace {\frac {q _ {x , i} ^ {(t)}}{\| a _ {i} ^ {(t)} \| _ {1}}} _ {h _ {x, i} ^ {(t)}}. \tag {6}
$$
where  $a_{i}^{(t)} = \left[a_{i}^{(t,0)},\dots,a_{i}^{(t,\tau_{i}^{(t)} - 1)}\right]^{T}$  collects all local coefficients of client  $i$ , and  $h_{x,i}^{(t)}$  normalizes the aggregated gradient  $q_{x,i}^{(t)}$  by  $1 / \| a_i^{(t)}\| _1$  such that  $\| h_{x,i}^{(t)}\|$  does not grow with the increasing of  $\tau_{i}^{(t)}$ .
Although such normalization can help to mitigate the system-level heterogeneity, the effective weight  $w_{i}$  can deviate from the true weight  $p_i$  of the original objective in eq. (1), and the iterates converge to the stationary point of a different objective that replaces all  $p_i$  by  $w_{i}$  in eq. (1) (see Theorem 1).
System-level heterogeneity robust FBO (ShroFBO). To address this convergence issue, we then propose a new method named ShroFBO with stronger resilience to such heterogeneity. Motivated by the normalized reformulation in eq. (6), ShroFBO adopts a different server-side aggregation as
$$
h _ {y} ^ {(t)} = \sum_ {i \in C ^ {(t)}} \widetilde {p} _ {i} h _ {y, i} ^ {(t)}, \quad h _ {v} ^ {(t)} = \sum_ {i \in C ^ {(t)}} \widetilde {p} _ {i} h _ {v, i} ^ {(t)}, \quad h _ {x} ^ {(t)} = \sum_ {i \in C ^ {(t)}} \widetilde {p} _ {i} h _ {x, i} ^ {(t)}, \tag {7}
$$
where  $\widetilde{p_i} \coloneqq \frac{n}{|C^{(t)}|} p_i$  and  $h_{y,i}^{(t)}, h_{v,i}^{(t)}, h_{x,i}^{(t)}$  are the normalized local aggregations defined in eq. (6). Accordingly, the server-side updates become
$$
y ^ {(t + 1)} = y ^ {(t)} - \rho^ {(t)} \gamma_ {y} h _ {y} ^ {(t)}, v ^ {(t + 1)} = \mathcal {P} _ {r} \big (v ^ {(t)} - \rho^ {(t)} \gamma_ {v} h _ {v} ^ {(t)} \big), x ^ {(t + 1)} = x ^ {(t)} - \rho^ {(t)} \gamma_ {x} h _ {x} ^ {(t)}. \tag {8}
$$
Differently from eq. (6), we select the client weights to be  $\widetilde{p}_i$  to enforce the correct convergence to the stationary point of the original objective in eq. (1), as shown in Theorem 2 later.
We make the following standard definitions and assumptions for the outer- and inner-level objective functions, as also adopted in stochastic bilevel optimization [26, 21, 30] and in federated bilevel optimization [65, 69, 24].
Definition 1. A mapping  $F$  is  $L$ -Lipschitz continuous if for  $\forall z, z'$ ,  $\| F(z) - F(z')\| \leq L\| z - z' \|$ .
Since the overall objective  $\Phi (x)$  is nonconvex, the goal is expected to find an  $\epsilon$ -accurate stationary point defined as follows.
Definition 2. We say  $z$  is an  $\epsilon$ -accurate stationary point of the objective function  $\Phi(x)$  if  $\mathbb{E}\|\nabla\Phi(z)\|^2 \leq \epsilon$ , where  $z$  is the output of an algorithm.
Assumption 1. For any  $x \in \mathbb{R}^{d_x}$ ,  $y \in \mathbb{R}^{d_y}$  and  $i \in \{1,2,\dots,n\}$ ,  $f_i(x,y)$  and  $g_i(x,y)$  are twice continuously differentiable, and  $g_i(x,y)$  is  $\mu_g$ -strongly convex w.r.t.  $y$ .
The following assumption imposes the Lipschitz continuity conditions on the upper- and lower-level objective functions and their derivatives.
Assumption 2. Function  $f_{i}(x,y)$  is  $L_{f}$ -Lipschitz continuous; the gradients  $\nabla f_{i}(x,y)$  and  $\nabla g_{i}(x,y)$  are  $L_{1}$ -Lipschitz continuous; the second-order derivatives  $\nabla^{2}f_{i}(x,y)$  and  $\nabla^2 g_i(x,y)$  are  $L_{2}$ -Lipschitz continuous; and the third-order derivatives  $\nabla^3 g_i(x,y)$  is  $L_{3}$ -Lipschitz continuous for some constants  $L_{f}, L_{1}, L_{2}, L_{3} > 0$ .
The Lipschitz continuity of the third-order derivative is necessary here to ensure the smoothness of  $v^{*}(x)$ , which guarantees the descent in the iterations of LS function (see Lemma 10), under our more challenging simultaneous and single-loop updating structure. Next, we assume the bounded variance conditions on the gradients and second-order derivatives.
Assumption 3. There exist constants  $\sigma_f^2$ ,  $\sigma_g^2$ ,  $\sigma_{gg}^2$  such that  $\mathbb{E}\big[\| \nabla f_i(x,y) - \nabla f_i(x,y;\xi)\| ^2\big]\leq \sigma_f^2$ ,  $\mathbb{E}\big[\| \nabla g_i(x,y) - \nabla g_i(x,y;\zeta)\| ^2\big]\leq \sigma_g^2$  and  $\mathbb{E}\big[\| \nabla^2 g_i(x,y) - \nabla^2 g_i(x,y;\zeta)\| ^2\big]\leq \sigma_{gg}^2$ .
Assumption 4. For any  $x \in \mathbb{R}^{d_x}$ ,  $y \in \mathbb{R}^{d_y}$ , there exist constants  $\beta_{gh} \geq 1$  and  $\sigma_{gh} \geq 0$  such that
$$
\sum_ {i = 1} ^ {n} w _ {i} \| \nabla_ {y} g _ {i} (x, y) \| ^ {2} \leq \beta_ {g h} ^ {2} \| \sum_ {i = 1} ^ {n} w _ {i} \nabla_ {y} g _ {i} (x, y) \| ^ {2} + \sigma_ {g h} ^ {2}.
$$
We have  $\beta_{gh} = 1$ , and  $\sigma_{gh} = 0$  when all  $g_i$ 's are identical.
This assumption of global heterogeneity uses  $\beta_{gh}$  and  $\sigma_{gh}$  to measure the dissimilarity of  $\nabla_y g_i(x,y)$  for all  $i$ .
3.2 Convergence and Complexity Analysis