SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Based on (8), we can define the  $\epsilon$ -stationary point of the original BLO problem (1) as follows.
Definition 3 (ε-stationary point). A point  $(\bar{x},\bar{y})$  is called ε-stationary point of (1) if  $\exists \bar{w}$  such that  $\mathcal{R}_x(\bar{x},\bar{y},\bar{w})\leq \epsilon ,\mathcal{R}_w(\bar{x},\bar{y},\bar{w})\leq \epsilon$  and  $\mathcal{R}_y(\bar{x},\bar{y})\leq \epsilon$
2.3 The importance of necessary conditions for BLO without additional CQs
Next, we emphasize the importance of deriving the necessary condition for the optimality of the nonconvex-PL BLO without additional CQs.
On the one hand, the necessary condition for the optimality of BLO is fundamental to the algorithm design and has been investigated for a long time in the optimization community [70, 71, 15, 17], but has not yet been fully understood [69]. One of the main challenges is that traditional CQs for constrained optimization are hard to check or do not hold in BLO [16, 70]. As a result, the development of mild CQs and the identification of BLO classes that inherently satisfy these CQs are considered significant contributions to the field [70, 71, 15, 17, 69]. Among those, the calmness condition is the weakest CQ [11].
On the other hand, recent BLO applications in machine learning often involve nonconvex LL problems,
the necessary condition for the solution of which is far less explored in either optimization or machine learning community. In the optimization community, most works focus on proving linear bilevel and sequential min-max optimization satisfy certain CQs [70, 16]. In the machine learning community, works on nonconvex-nonconvex BLO either relax the LL problem that introduces error [43, 57] or impose assumptions that are hard to verify in practice such as CRCQ [37] or Morse-Bott condition [2]. To the best of our knowledge, we are the first to propose the necessary condition for the optimality of a class of nonconvex-nonconvex BLO problems with checkable assumptions. Moreover, algorithms in constrained optimization are always CQ-dependent, e.g. the convergence of an algorithm depends on whether a particular CQ condition is satisfied. As we prove that standard CQs are invalid for nonconvex-PL BLO, several existing BLO may become theoretically less grounded [37, 28].
Figure 2: Illustration of our theory: relations of different CQs and BLO reformulation (2) and (3). Slater condition fails for both BLO reformulations so we do not include it.
3 An Alternating Method for Bilevel Problems under the PL Condition
In this section, we introduce our Generalized ALternating method for bilevel opTimization with convex LL problem, GALET for short, and then elucidate its relations with the existing algorithms.
3.1 Algorithm development
To attain the  $\epsilon$ -stationary point of (1) in the sense of Definition 3, we alternately update  $x$  and  $y$  to reduce computational costs. At iteration  $k$ , we update  $y^{k+1}$  by  $N$ -step GD on  $g(x^k, y)$  with  $y^{k,0} = y^k$  and  $y^{k+1} = y^{k,N}$  as
$$
y ^ {k, n + 1} = y ^ {k, n} - \beta \nabla g \left(x ^ {k}, y ^ {k, n}\right) \tag {10}
$$
Algorithm 1 GALET for nonconvex-PL BLO
1: Initialization {x0,y0}, stepsizes {α,β,ρ}
2: for k=0 to K-1 do
3: for n=0 to N-1 do ▷yk,0=yk
4: update yk,n+1 by (10)
5: end for ▷yk+1=yk,N
6: for t=0 to T-1 do ▷wk,0=0
7: update wk,t+1 by (12b)
8: end for ▷wk+1=wk,T
9: calculate dkx by (13)
10: update xk+1=xk-αdkx
11: end for
While setting  $N = 1$  is possible, we retain  $N$  for generality. We then update  $w$  via the fixed-point equation derived from (8b) and employ  $\nabla_{yy}g(x,y)$  ( $\nabla_yf(x,y) + \nabla_{yy}g(x,y)w$ ) as the increment for  $w$ . This increment can be viewed as the gradient of  $\mathcal{L}(x,y,w)$  defined as
$$
\mathcal {L} (x, y, w) := \frac {1}{2} \left\| \nabla_ {y} f (x, y) + \nabla_ {y y} ^ {2} g (x, y) w \right\| ^ {2} \tag {11}
$$
which is quadratic w.r.t.  $w$ , given  $x^k$  and  $y^{k + 1}$ .
However, unlike the LL objective  $g(x,y)$ , the objective  $\mathcal{L}(x,y,w)$  is Lipschitz smooth with respect to  $x$  and  $y$  only for bounded  $w$ , which makes it difficult to control the change of solution (11) under different  $x$  and  $y$ . Hence, we update  $w^{k + 1}$  via  $T$ -step GD on with  $w^{k,0} = 0$  and  $w^{k + 1} = w^{k,T}$  as
$$
w ^ {k, t + 1} = w ^ {k, t} - \rho d _ {w} ^ {k, t}, \tag {12a}
$$
$$
d _ {w} ^ {k, t} := \nabla_ {y y} ^ {2} g \left(x ^ {k}, y ^ {k + 1}\right) \left(\nabla_ {y} f \left(x ^ {k}, y ^ {k + 1}\right) + \nabla_ {y y} ^ {2} g \left(x ^ {k}, y ^ {k + 1}\right) w ^ {k, t}\right). \tag {12b}
$$
After obtaining the updated  $y^{k + 1}$  and  $w^{k + 1}$ , we update  $x^k$  by the fixed point equation of (8a) as
$$
x ^ {k + 1} = x ^ {k} - \alpha d _ {x} ^ {k}, \quad \text {w i t h} \quad d _ {x} ^ {k} := \nabla_ {x} f \left(x ^ {k}, y ^ {k + 1}\right) + \nabla_ {x y} ^ {2} g \left(x ^ {k}, y ^ {k + 1}\right) w ^ {k + 1}. \tag {13}
$$
We summarize our algorithm in Algorithm 1. We choose  $w^{k,0} = 0$  for simplicity, but  $w^{k,0} = w^0 \neq 0$  is also valid. Same convergence statement can hold since the boundedness and Lipschitz continuity of limit points  $\lim_{t\to \infty}w^{k,t}$  are still guaranteed. From the algorithm perspective, [2] shares similarities with us. However, without recognizing the property of GD converging to the minimal norm solution [52], Arbel and Mairal [2] introduces an additional Hessian into the objective (11), resulting in the calculation of fourth-order Hessian operation, which is more complex than GALET.
3.2 Relation with algorithms in nonconvex-strongly-convex BLO
We explain the relation between GALET and methods in the nonconvex-strongly-convex BLO.
First, if  $g(x,y)$  is strongly convex in  $y$ , minimizing  $\mathcal{L}(x,y,w)$  over  $w$  yields the unique solution  $w^{*}(x,y) = -\left(\nabla_{yy}^{2}g(x,y)\right)^{-1}\nabla_{y}f(x,y)$ . This means that optimizing  $\mathcal{L}(x,y,w)$  corresponds to implicit differentiation in the nonconvex-strongly-convex BLO [9, 27, 24, 32]. Therefore, we refer
the optimization of  $w$  as the shadow implicit gradient level.
On the other hand, if  $\nabla_{yy}^2 g(x,y)$  is positive definite, the problem (11) over  $w$  is equivalent to
$$
\min  _ {w} \left\{\frac {1}{2} w ^ {\top} \nabla_ {y y} ^ {2} g (x, y) w + w ^ {\top} \nabla_ {y} f (x, y) \right\} \tag {14}
$$
which can be verified by their first-order necessary conditions. Thus, one can update  $w$  by GD on (14) via
$$
d _ {w} ^ {k, t} := \nabla_ {y} f \left(x ^ {k}, y ^ {k + 1}\right) + \nabla_ {y y} ^ {2} g \left(x ^ {k}, y ^ {k + 1}\right) w ^ {k, t}. \tag {15}
$$
Note that the increment in (15) eliminates the extra Hessian in (12b) and recovers the updates for nonconvex-strongly-convex BLO in [35, 27, 32, 9, 66]. Actually, the a. an algorithm to find the stationary point of the nonconvexresults of our algorithm with the nonconvex-strongly-con (12b) on Example 1, where the global optimality is meas set (7) with the explicit form stated in (46). We observe t optimizer of the example in Lemma 1 so that the addition
Figure 3: The  $w$  update in (15) for nonconvex-strongly-convex does not work well for nonconvex-PL BLO.
Additional Hessian in (12b) is inevitable for a PL BLO. Figure 3 shows the comparative convex BLO algorithm using (15) instead of (14) based by the distance to the global optimal that the update (15) fails to find the global Hessian in (12b) is unavoidable.
4 Convergence Analysis
In this section, we provide the convergence rate analysis under convex PL LL settings. We first introduce the assumptions and the challenges. Next, we analyze the descent property and drifting error in LL, the bias of  $w^{k + 1}$  to the optimal solution of  $\min_w\mathcal{L}(x^k,y^{k + 1},w)$  and the descent property of UL. Finally, we define a new Lyapunov function and state the iteration complexity of GALET.
Assumption 1 (Lipschitz continuity). Assume that  $\nabla f, \nabla g$  and  $\nabla^2 g$  are Lipschitz continuous with  $\ell_{f,1}, \ell_{g,1}$  and  $\ell_{g,2}$ , respectively. Additionally, we assume  $f(x,y)$  is  $\ell_{f,0}$ -Lipschitz continuous over  $y$ .
Assumption 2 (Landscape of LL objective). Assume that  $g(x,y)$  is  $\mu_g$ -PL over  $y$ . Moreover, let  $\sigma_g > 0$  be the lower-bound of the positive singular values of Hessian, i.e.  $\sigma_g = \inf_{x,y} \{\sigma_{\min}^+(\nabla_{yy}^2 g(x,y))\}$ .
Remark 1. By definition, singular values are always nonnegative. We use  $\sigma_g$  to denote the lower bound of the non-zero singular values of  $\nabla_{yy}^2 g(x,y)$ . Assumption 2 means that the non-zero singular values are bounded away from 0 on the entire domain. Given Assumption 1 that the Hessian is globally Lipschitz, they together potentially rule out negative eigenvalues of the Hessian. However, different from strong convexity [9, 27, 24, 32, 28], this assumption still permits  $\nabla_{yy}^2 g(x,y)$  to possess zero eigenvalues and includes BLO problems of multiple LL solutions. As an example, this includes the loss of the (overparameterized) generalized linear model [52]. In addition, Assumption 2 is needed only in the convergence rate analysis along the optimization path to ensure the sequence  $\{w^k\}$  is well-behaved. Therefore, it is possible to narrow down either the Hessian Lipschitz condition or the lower bound of singular values to a bounded region, when the local optimal sets are bounded.
Challenges of analyzing GALET. The absence of strong convexity of the LL brings challenges to characterize the convergence of GALET. To see this, recall that in recent analysis of BLO such as [9], when  $g(x,\cdot)$  is strongly convex, the minimizer of LL is unique. Therefore, to quantify the convergence of Algorithm 1, one can use the Lyapunov function  $\mathbb{V}_{\mathrm{SC}}^k \coloneqq f(x^k,S(x^k)) + \| y^k -S(x^k)\| ^2$  where  $f(x^{k},S(x^{k}))$  and  $\| y^{k} - S(x^{k})\|^{2}$  are used to account for the UL descent and LL error caused by the alternating update and the inexactness, respectively. However,  $\mathbb{V}_{\mathrm{SC}}$  is not well-defined when  $S(x)$  is not unique. Therefore, it is necessary to seek for a new Lyapunov function.
4.1 Descent of each sequence in GALET
A nature alternative of  $\| y^k - S(x^k) \|^2$  under the PL condition is the LL optimality residual  $\mathcal{R}_y(x^k, y^k)$ , the evolution of which between two steps can be quantified by
$$
\mathcal {R} _ {y} \left(x ^ {k + 1}, y ^ {k + 1}\right) - \mathcal {R} _ {y} \left(x ^ {k}, y ^ {k}\right) = \underbrace {\mathcal {R} _ {y} \left(x ^ {k + 1} , y ^ {k + 1}\right) - \mathcal {R} _ {y} \left(x ^ {k} , y ^ {k + 1}\right)} _ {(1 7 \mathrm {b})} + \underbrace {\mathcal {R} _ {y} \left(x ^ {k} , y ^ {k + 1}\right) - \mathcal {R} _ {y} \left(x ^ {k} , y ^ {k}\right)} _ {(1 7 \mathrm {a})} \tag {16}
$$
where the first term characterizes the drifting LL optimality gap after updating  $x$ , and the second term shows the descent amount by one-step GD on  $y$ . Unlike the strongly convex case where  $S(x)$  is Lipschitz continuous, both  $g^{*}(x)$  and  $g(x,y)$  are not Lipschitz continuous, so we cannot directly bound the first difference term in (16) by the drifting of update  $\| x^{k + 1} - x^k\|$ . Owing to the opposite sign of  $g(x,y)$  and  $g^{*}(x)$ , we bound the first term in (16) by the smoothness of  $g(x,y)$  and  $g^{*}(x)$ . The second term in (16) can be easily bounded by the fact that running GD on PL function ensures the contraction of function value distance.
Lemma 4. Under Assumption 1-2, let  $y^{k + 1}$  be the point generated by (10) given  $x^k$  and the stepsize  $\beta \leq \frac{1}{\ell_{g,1}}$ . Then it holds that
$$
\mathcal {R} _ {y} \left(x ^ {k}, y ^ {k + 1}\right) \leq \left(1 - \beta \mu_ {g}\right) ^ {N} \mathcal {R} _ {y} \left(x ^ {k}, y ^ {k}\right) \tag {17a}
$$
$$
\mathcal {R} _ {y} \left(x ^ {k + 1}, y ^ {k + 1}\right) \leq \left(1 + \frac {\alpha \eta_ {1} \ell_ {g , 1}}{\mu_ {g}}\right) \mathcal {R} _ {y} \left(x ^ {k}, y ^ {k + 1}\right) + \left(\frac {\alpha \ell_ {g , 1}}{2 \eta_ {1}} + \frac {\ell_ {g , 1} + L _ {g}}{2} \alpha^ {2}\right) \| d _ {x} ^ {k} \| ^ {2} \tag {17b}
$$
where  $L_{g}\coloneqq \ell_{g,1}(1 + \ell_{g,1} / 2\mu_{g})$  and  $\eta_1 = \mathcal{O}(1)$  is a constant that will be chosen in the final theorem.
Rearranging the terms in (17a) and using the fact that  $g(x,y) \geq g^{*}(x)$ , the second term in (16) can be upper bounded by a negative term and  $\mathcal{O}(\beta^2)$ . Adding (17a) and (17b), letting  $\alpha, \beta$  is the same order and choosing  $\eta_{1} = \mathcal{O}(1)$  properly yield
$$
\mathcal {R} _ {y} \left(x ^ {k + 1}, y ^ {k + 1}\right) - \mathcal {R} _ {y} \left(x ^ {k}, y ^ {k}\right) \leq - \mathcal {O} (\alpha) \mathcal {R} _ {y} \left(x ^ {k}, y ^ {k}\right) + \mathcal {O} (\alpha) \| d _ {x} ^ {k} \| ^ {2}. \tag {18}
$$
When we choose  $\eta_{1}$  large enough, the second term will be dominated by the negative term  $-\mathcal{O}(\alpha)\| d_x^k\|^2$  given by the UL descent, so it will be canceled out.
(a) UL level:  $\alpha$
Figure 4: Trajectories of GALET under different choices of the parameters in UL, LM and LL level for Example 1. The  $x$ -axis denotes the value of  $x^k$ , the  $y$ -axis represents the first coordinates of  $y^k$ . The color of the point shows the optimality gap measured by (46) with the color bar shown right. The yellow square is the starting point and the red star  $\star$  represent the optimal  $(x^*, y^*)$ .
(b) Shadow implicit gradient level:  $T,\gamma$
(c) LL level:  $N,\beta$
With the iterates generated by GD initialized at 0 converging to the minimal norm solution of the least squares problem [52], we can characterize the drifting error of  $w$  using the Lipschitz continuity of its minimal norm solution. The following lemma characterizes the error of  $w^{k+1}$  to the minimal-norm solution of the shadow implicit gradient level  $w^{\dagger}(x,y) := -\left(\nabla_{yy}^{2}g(x,y)\right)^{\dagger}\nabla_{y}f(x,y)$ .
Lemma 5. Under Assumption 1-2, we let  $w^{k+1}$  denote the iterate generated by (12a) given  $(x^k, y^{k+1})$  and  $w^{k,0} = 0$ , and denote  $b_k := \| w^{k+1} - w^\dagger(x^k, y^{k+1}) \|$ . If we choose the stepsize  $\rho \leq \frac{1}{\ell_{g,1}^2}$ , then the shadow implicit gradient error can be bounded by  $b_k^2 \leq 2(1 - \rho \sigma_g^2)^T \ell_{f,0}^2 / \mu_g$ .
Instead of using  $f(x, S(x))$  which is ill-posed, we integrate the increment of  $x$  to get  $F(x, y; w) = f(x, y) + w^\top \nabla_y g(x, y)$ . Then by plugging the optimal minimal norm solution  $w^\dagger(x, y)$ , we choose  $F(x^k, y^k; w^\dagger(x^k, y^k))$  as the counterpart of  $f(x^k, S(x^k))$  in  $\mathbb{V}_{\mathrm{SC}}^k$ . We can quantify its difference between two adjacent steps by the smoothness of  $F(x, y; w)$ .
Lemma 6. Under Assumption 1-2, by choosing  $\alpha \leq \min \left\{\frac{1}{4L_F + 8L_w\ell_{g,1}},\frac{\beta\eta_2}{2L_w^2}\right\}$ , one has
$$
\begin{array}{l} F \left(x ^ {k + 1}, y ^ {k + 1}; w ^ {\dagger} \left(x ^ {k + 1}, y ^ {k + 1}\right)\right) - F \left(x ^ {k}, y ^ {k}; w ^ {\dagger} \left(x ^ {k}, y ^ {k}\right)\right) \\ \leq - \frac {\alpha}{8} \| d _ {x} ^ {k} \| ^ {2} + \frac {\alpha \ell_ {g , 1} ^ {2}}{2} b _ {k} ^ {2} + \frac {\beta \left(\eta_ {2} \ell_ {g , 1} ^ {2} + 2 N L _ {w} \ell_ {g , 1} ^ {2} + 2 \ell_ {f , 0} \ell_ {g , 2}\right) + L _ {F} \ell_ {g , 1} ^ {2} \beta^ {2}}{\mu_ {g}} \mathcal {R} _ {y} \left(x ^ {k}, y ^ {k}\right) \tag {19} \\ \end{array}
$$
where  $L_{w}$  is the constant defined in Appendix and constant  $\eta_{2}$  will be chosen in the final theorem.
4.2 A new Lyapunov function
Based on (18) and (19), we can define a new Lyapunov function for the nonconvex-PL BLO as
$$
\mathbb {V} ^ {k} := F \left(x ^ {k}, y ^ {k}; w ^ {\dagger} \left(x ^ {k}, y ^ {k}\right)\right) + c \mathcal {R} _ {y} \left(x ^ {k}, y ^ {k}\right) \tag {20}
$$
where  $c$  is chosen to balance the coefficient of the term  $\mathcal{R}_y(x^k,y^k)$  and  $\| d_x^k\| ^2$  such that
$$
\mathbb {V} ^ {k + 1} - \mathbb {V} ^ {k} \leq - \mathcal {O} (\alpha) \| d _ {x} ^ {k} \| ^ {2} - \mathcal {O} (\alpha) \mathcal {R} _ {y} \left(x ^ {k}, y ^ {k}\right) + \mathcal {O} (\alpha) b _ {k} ^ {2}. \tag {21}
$$
Telescoping to (21) results in the convergence of both  $\|d_x^k\|^2$  and  $\mathcal{R}_y(x^k, y^k)$ . By definition,  $\|d_x^k\|^2 = \mathcal{R}_x(x^k, y^{k+1}, w^{k+1})$ , so the convergence of  $\mathcal{R}_x(x^k, y^k, w^k)$  is implied by the Lipschitz continuity of increments and the fact that  $\|y^{k+1} - y^k\|$ ,  $\|w^{k+1} - w^k\| \to 0$ . Likewise, the convergence of  $\mathcal{R}_w(x^k, y^k, w^k)$  can be established by recognizing that  $b_k^2$  is sufficiently small when selecting a large  $T$ , ensuring that  $\mathcal{R}_w(x^k, y^{k+1}, w^{k+1})$  is also small, and consequently,  $\mathcal{R}_w(x^k, y^k, w^k)$ . The convergence results of GALET can be formally stated as follows.
Theorem 2. Under Assumption 1-2, choosing  $\alpha, \rho, \beta = \mathcal{O}(1)$  with some proper constants and  $N = \mathcal{O}(1), T = \mathcal{O}(\log(K))$ , then it holds that
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathcal {R} _ {x} (x ^ {k}, y ^ {k}, w ^ {k}) = \mathcal {O} \left(\frac {1}{K}\right), \quad \frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathcal {R} _ {w} (x ^ {k}, y ^ {k}, w ^ {k}) = \mathcal {O} \left(\frac {1}{K}\right), \quad \frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathcal {R} _ {y} (x ^ {k}, y ^ {k}) = \mathcal {O} \left(\frac {1}{K}\right).
$$
From Definition 3, one need  $\mathcal{O}(KT) = \mathcal{O}(\epsilon^{-1}\log (\epsilon^{-1}))\eqqcolon$ $\tilde{\mathcal{O}} (\epsilon^{-1})$  iterations in average to achieve  $\epsilon$  -stationary point of the BLO problem (1). The next remark shows that this complexity is optimal.
Remark 2 (Lower bound). The iteration complexity  $\tilde{\mathcal{O}} (\epsilon^{-1})$  in Theorem 2 is optimal in terms of  $\epsilon$ , up to logarithmic factors. This is because when  $f(x,y)$  is inde
Figure 5: Convergence results of GALET of hyper-cleaning task on MNIST dataset. Left: test loss v.s. iteration. Right: stationary measure v.s. iteration (the  $y$ -axis is in the logarithmic scale).
pendent of  $y$  and  $g(x,y) = 0$ , Assumptions 1 and 2 reduce to the smoothness assumption of  $f$  on  $x$ , and thus the  $\mathcal{O}(\epsilon^{-1})$  lower bound of nonconvex and smooth minimization in [6] also applies to BLO under Assumptions 1 and 2.
5 Preliminary Simulations and Conclusions
The goal of our simulations is to validate our theories and test the performance of GALET on actual learning applications. The experimental setting and parameter choices are included in the Appendix.
Our stationary measure is a necessary condition of the global optimality. As shown in Figure 1, GALET approaches the global optimal set of Example 1 and our stationary measure also converges to 0, while the value-function based KKT score does not.
UL stepsize is the most sensitive parameter in our algorithm. As shown in Figure 4, varying UL stepsize  $\alpha$  leads to different trajectories of GALET, while varying parameters in the LL and shadow implicit gradient level only cause small perturbations. This means that  $\alpha$  is the most sensitive parameter. The fact that GALET is robust to small values of  $T$  and  $N$  makes it computationally appealing for practical applications, eliminating the need to tune them extensively. This phenomenon also occurs in the hyper-cleaning task.
Our method converges fast on the actual machine learning application. We compare GALET with BOME [37], IAPTT-GM [43] and V-PBGD [57] in the hyper-cleaning task on the MNIST dataset. As shown in Figure 5, GALET converges faster than other methods and the convergence rate of GALET is  $\mathcal{O}(1 / K)$ , which matches Theorem 2. Table 2 shows that the test accuracy of GALET is comparable to other methods.
Conclusions and limitations. In this paper, we study BLO with lower-level objectives satisfying the PL condition. We first establish the stationary measure for nonconvex-PL BLO without additional CQs and then propose an alternating optimization algorithm that generalizes the existing alternating (implicit) gradient-based algorithms for bilevel problems with a strongly convex lower-level objective. Our algorithm termed GALET achieves the  $\tilde{\mathcal{O}} (\epsilon^{-1})$  iteration complexity for BLO with convex PL LL problems. Numerical experiment
Method	Accuracy
BOME	88.20
IAPTT-GM	90.13
V-PBGD	90.24
GALET	90.20
Table 2: Comparison of the test accuracy.
One potential limitation is that we only consider the deterministic BLO problem and require second-order information, and it is unclear whether our analysis can be generalized to handle the stochastic case and whether the same iteration complexity can be guaranteed with full first-order information. Another limitation is Assumption 2. This stipulates that the singular value of the LL Hessian must be bounded away from 0 to ensure the stability of the  $\{w^k\}$ .
The work of Q. Xiao and T. Chen was supported by National Science Foundation MoDL-SCALE project 2134168, the RPI-IBM Artificial Intelligence Research Collaboration (AIRC) and an Amazon Research Award. We also thank anonymous reviewers and area chair for their insightful feedback and constructive suggestions, which are instrumental in enhancing the clarity and quality of our paper.
Supplementary Material for "A Generalized Alternating Method for Bilevel Optimization under the Polyak-Lojasiewicz Condition"
A Auxiliary lemmas 15
B Proof of Stationary Metric 16
B.1 Theorem 3 and its proof 16
B.2 Proof of Lemma 1 16
B.3 Proof of Lemma 2 17
B.4 Proof of Lemma 3 17
B.5 Proof of Theorem 1 18
C Proof of Lower-level Error 18
D Proof of Shadow Implicit Gradient Error 19
D.1 Propositions of  $\mathcal{L}(x,y,w)$  and  $\mathcal{L}^* (x,y)$  19
D.2 Proof of Lemma 5 19
E Proof of Upper-level Error 19
E.1 Propositions of  $w^{\dagger}(x,y)$  19
E.2 Proof of Lemma 6 20
F Proof of Overall Descent of Lyapunov Function 23
G Additional Experiments and Details 25
A Auxiliary lemmas
We use  $\| \cdot \|$  to be the Euclidean norm and  $\mathcal{X}$  be the metric space with the Euclidean distance metric. Let  $x\in \mathcal{X}$  be a point and  $A\subset \mathcal{X}$  be a set, we define the distance of  $x$  and  $A$  as
$$
d (x, A) = \inf  \left\{\| x - a \| \mid a \in A \right\}.
$$
The following lemma relates the PL condition to the error bound (EB) and the quadratic growth (QG) condition.
Lemma 7 ([33, Theorem 2]). If  $g(x,y)$  is  $\ell_{g,1}$ -Lipschitz smooth and  $PL$  in  $y$  with  $\mu_g$ , then it satisfies the EB condition with  $\mu_g$ , i.e.
$$
\left\| \nabla_ {y} g (x, y) \right\| \geq \mu_ {g} d (y, S (x)). \tag {22}
$$
Moreover, it also satisfies the  $QG$  condition with  $\mu_g$ , i.e.
$$
g (x, y) - g ^ {*} (x) \geq \frac {\mu_ {g}}{2} d (y, S (x)) ^ {2}. \tag {23}
$$
Conversely, if  $g(x,y)$  is  $\ell_{g,1}$ -Lipschitz smooth and satisfies  $EB$  with  $\mu_g$ , then it is  $PL$  in  $y$  with  $\mu_g / \ell_{g,1}$ .
The following lemma shows that the calmness condition ensures the KKT conditions hold at the global minimizer.
Proposition 1 ([11, Proposition 6.4.4]). Let  $(x^{*},y^{*})$  be the global minimizer of a constrained optimization problem and assume that this problem is calm at  $(x^{*},y^{*})$ . Then the KKT conditions hold at  $(x^{*},y^{*})$ .
The following lemma gives the gradient of the value function.
Lemma 8 ([51, Lemma A.5]). Under Assumption 2 and assuming  $g(x,y)$  is  $\ell_{g,1}$ -Lipschitz smooth,  $g^{*}(x)$  is differentiable with the gradient
$$
\nabla g ^ {*} (x) = \nabla_ {x} g (x, y), \forall y \in S (x).
$$
Moreover,  $g^{*}(x)$  is  $L_{g}$ -smooth with  $L_{g} \coloneqq \ell_{g,1}(1 + \ell_{g,1} / 2\mu_{g})$ .
B Proof of Stationary Metric
B.1 Theorem 3 and its proof
Theorem 3. For nonconvex-PL BLO, A point  $(x^{*},y^{*})$  is a local (resp. global) optimal solution of (1) if and only if it is a local (resp. global) optimal solution of (3). The same argument holds for (2).
Proof. First, we prove the equivalence of (1) and reformulation (3). According to PL condition,
$$
\frac {1}{2} \| \nabla_ {y} g (x, y) \| ^ {2} \geq \mu_ {g} (g (x, y) - g ^ {*} (x)).
$$