SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Therefore, if  $\nabla_y g(x,y) = 0$ , it holds that  $0 \geq g(x,y) - g^*(x) \geq 0$  which yields  $g(x,y) = g^*(x)$ . On the other hand, if  $g(x,y) = g^*(x)$ , the first order necessary condition says  $\nabla_y g(x,y) = 0$ . As a consequence, we know that
$$
\nabla_ {y} g (x, y) = 0 \Leftrightarrow g (x, y) = g ^ {*} (x) \Leftrightarrow y \in S (x).
$$
In this way, for any local (resp. global) optimal solution  $(x^{*},y^{*})$  of (3), there exists a neighborhood  $U$  such that for any  $(x,y)\in U$  with  $\nabla_yg(x,y) = 0$ , it holds that  $f(x^{*},y^{*})\leq f(x,y)$ . Since  $\nabla_yg(x,y) = 0\Leftrightarrow y\in S(x)$ , we know  $(x^{*},y^{*})$  is also a local (resp. global) optimal solution of (2). The reverse holds true for the same reason.
The equivalence of Problem (1) and reformulation (2) can be proven in the same way.
First, the gradient of  $g(x,y)$  can be calculated by
$$
\nabla_ {y} g (x, y) = \left(x + y _ {1} - \sin (y _ {2})\right) \left[ 1, - \cos (y _ {2}) \right] ^ {\top}.
$$
Since  $\| [1, -\cos (y_2)]^\top \| \geq 1$  and  $g^{*}(x) = 0$ , we know
$$
\frac {1}{2} \| \nabla_ {y} g (x, y) \| ^ {2} = \frac {1}{2} (x + y _ {1} - \sin (y _ {2})) ^ {2} \| [ 1, - \cos (y _ {2}) ] ^ {\top} \| ^ {2} \geq (g (x, y) - g ^ {*} (x)).
$$
Thus  $g(x,y)$  satisfies the PL condition with  $\mu_g = 1$
Next, the LL problem in Example 1 is equal to  $x + y_{1} - \sin (y_{2}) = 0$  so that the BLO of
$$
\min  _ {x, y} f (x, y) = x ^ {2} + y _ {1} - \sin (y _ {2}), \quad \text {s . t .} y \in \arg \min  g (x, y) = \frac {1}{2} \left(x + y _ {1} - \sin (y _ {2})\right) ^ {2} \tag {24}
$$
is equivalent to
$$
\min  _ {x, y} f (x, y) = x ^ {2} + y _ {1} - \sin (y _ {2}), \quad \text {s . t .}   x + y _ {1} - \sin (y _ {2}) = 0.
$$
According to the constraint, we know  $y_{1} - \sin(y_{2}) = -x$  so that solving the BLO amounts to solving  $\min\{x^{2} - x\}$  with the constraint  $x + y_{1} - \sin(y_{2}) = 0$ . Therefore, the global minimizers of BLO are those points  $(x^{*}, y^{*})$  where  $x^{*} = 0.5$  and  $y^{*} = (y_{1}^{*}, y_{2}^{*})^{\top}$  with  $y_{1}^{*} - \sin(y_{2}^{*}) + 0.5 = 0$ .
On the other hand, reformulating (24) using (2) yields
$$
\min  _ {x, y} f (x, y) = x ^ {2} + y _ {1} - \sin \left(y _ {2}\right), \quad \text {s . t .} g (x, y) - g ^ {*} (x) = 0 \tag {25}
$$
If the calmness holds for (25), then according to Proposition 1, there exists  $w^{*}\neq 0$  such that
$$
\nabla_ {x} f \left(x ^ {*}, y ^ {*}\right) + \left(\nabla_ {x} g \left(x ^ {*}, y ^ {*}\right) - \nabla_ {x} g ^ {*} \left(x ^ {*}\right)\right) w ^ {*} = 0. \tag {26}
$$
However, for any  $w^{*}\neq 0,g^{*}(x) = 0$  and
$$
\nabla_ {x} f \left(x ^ {*}, y ^ {*}\right) + \left(\nabla_ {x} g \left(x ^ {*}, y ^ {*}\right) - \nabla_ {x} g ^ {*} \left(x ^ {*}\right)\right) w ^ {*} = 2 x ^ {*} + \left(x ^ {*} + y _ {1} ^ {*} - \sin \left(y _ {2} ^ {*}\right)\right) w ^ {*} = 2 x ^ {*} = 1 \neq 0
$$
which contradicts (26). Therefore, the calmness condition fails for (24) with reformulation (2).
Figure 6: Landscape of  $g(1, y)$  in Example 1 and the red line denotes  $S(1)$ . Different choices of  $x$  only translate the plot and share the same landscape.
B.3 Proof of Lemma 2
Figure 6 shows the landscape of  $g(x,y)$  when  $x = 1$  and the red line denotes its solution set. The figure reveals that  $g(x,y)$  is nonconvex in  $y$  with multiple LL solutions, and the solution set is closed but nonconvex.
Proof. It is easy to see that  $S(x) = \{y \mid x + y_1 = \sin(y_2)\}$  and  $g^*(x) = 0$ .
First, the Slater condition is invalid because  $\nabla_y g(x,y) = 0$  is an equality constraint.
Next, we can compute the Hessian and Jacobian of  $g(x,y)$  as
$$
\nabla_ {y y} ^ {2} g (x, y) = \left[ \begin{array}{c c} 1 & - \cos (y _ {2}) \\ - \cos (y _ {2}) & \cos^ {2} (y _ {2}) + \sin (y _ {2}) (x + y _ {1} - \sin (y _ {2})) \end{array} \right], \quad \nabla_ {y x} ^ {2} g (x, y) = \left[ \begin{array}{c} 1 \\ - \cos (y _ {2}) \end{array} \right].
$$
Since  $\nabla_{yx}^2 g(x,y)$  coincides with the first column of  $\nabla_{yy}^2 g(x,y)$ , we know that
$$
\operatorname {r a n k} \left(\left[ \nabla_ {y y} ^ {2} g (x, y), \quad \nabla_ {y x} ^ {2} g (x, y) \right]\right) = \operatorname {r a n k} \left(\nabla_ {y y} ^ {2} g (x, y)\right).
$$
For any global minimizer  $(x^{*},y^{*})$  of the bilevel problem, it should satisfy the LL optimality, i.e.  $y^{*}\in S(x^{*})$ , and thus  $x^{*} + y_{1}^{*} - \sin (y_{2}^{*}) = 0$ . In this way, we have
$$
\nabla_ {y y} ^ {2} g (x ^ {*}, y ^ {*}) = \left[ \begin{array}{c c} 1 & - \cos (y _ {2} ^ {*}) \\ - \cos (y _ {2} ^ {*}) & \cos^ {2} (y _ {2} ^ {*}) \end{array} \right].
$$
Thus,  $\mathrm{rank}(\nabla_{yy}^2 g(x^*,y^*)) = 1\neq 2$  which contradicts the LICQ and MFCQ conditions.
Moreover, it follows that  $\mathrm{rank}(\nabla_{yy}^2 g(x,y)) = 1$  if and only if
$$
\det (\nabla_ {y y} ^ {2} g (x, y)) = \sin (y _ {2}) (x + y _ {1} - \sin (y _ {2})) = 0.
$$
Thus, the CRCQ condition is equivalent to that there exists a neighborhood of  $(x^{*},y^{*})$  such that either  $\sin (y_2) = 0$  or  $x + y_{1} - \sin (y_{2}) = 0$  holds for any  $(x,y)$  in that neighborhood. We fix  $x^{*}$  and search such neighborhood for  $y$  first. It is clear that finding an area of  $y$  such that  $\sin (y_2)\equiv 0$  is impossible due to the nature of sin function. On the other hand,  $x^{*} + y_{1} - \sin (y_{2}) = 0$  is equivalent to  $y\in S(x^{*})$  so that making  $x^{*} + y_{1} - \sin (y_{2})\equiv 0$  in an neighborhood of  $y^{*}$  is also unrealistic since otherwise, the solution set  $S(x^{*})$  should be open, which contradicts that  $S(x)$  is closed. Thus, (6) violates CRCQ condition.
B.4 Proof of Lemma 3
Assume the PL and the Lipschitz smoothness constants of  $g(x, \cdot)$  are  $\mu_g$  and  $\ell_{g,1}$ , and the Lipschitz continuity constant of  $f(x, \cdot)$  is  $\ell_{f,0}$ . Consider  $\forall q$ , and  $\forall (x', y')$ , s.t.  $\nabla_y g(x', y') + q = 0$ , then letting  $y_q \in \mathrm{Proj}_{S(x')} (y')$  and according to Lemma 7, one has
$$
\| q \| = \| \nabla_ {y} g \left(x ^ {\prime}, y ^ {\prime}\right) \| \geq \mu_ {g} \| y ^ {\prime} - y _ {q} \|
$$
Since  $(x^{*},y^{*})$  solves (3) and  $(x^{\prime},y_{q})$  is also feasible to (3), one has  $f(x^{*},y^{*})\leq f(x^{\prime},y_{q})$  . Thus,
$$
f \left(x ^ {\prime}, y ^ {\prime}\right) - f \left(x ^ {*}, y ^ {*}\right) \geq f \left(x ^ {\prime}, y ^ {\prime}\right) - f \left(x ^ {\prime}, y _ {q}\right) \geq - \ell_ {f, 0} \| y ^ {\prime} - y _ {q} \| \geq - \frac {\ell_ {f , 0}}{\mu_ {g}} \| q \|.
$$
This justifies the calmness definition in (5) with  $M \coloneqq \frac{\ell_{f,0}}{\mu_g}$ .
B.5 Proof of Theorem 1
Proof. By applying Lemma 3 and Proposition 1, we know that KKT conditions are necessary, i.e.
$$
\nabla_ {x} f \left(x ^ {*}, y ^ {*}\right) + \nabla_ {x y} ^ {2} g \left(x ^ {*}, y ^ {*}\right) w ^ {*} = 0 \tag {27a}
$$
$$
\nabla_ {y} f \left(x ^ {*}, y ^ {*}\right) + \nabla_ {y y} ^ {2} g \left(x ^ {*}, y ^ {*}\right) w ^ {*} = 0 \tag {27b}
$$
$$
\nabla_ {y} g \left(x ^ {*}, y ^ {*}\right) = 0. \tag {27c}
$$
On the other hand, since
$$
\nabla_ {y} g \left(x ^ {*}, y ^ {*}\right) = 0 \Leftrightarrow g \left(x ^ {*}, y ^ {*}\right) - g ^ {*} \left(x ^ {*}\right) = 0
$$
$$
\nabla_ {y} f (x ^ {*}, y ^ {*}) + \nabla_ {y y} ^ {2} g (x ^ {*}, y ^ {*}) w ^ {*} = 0 \Rightarrow \nabla_ {y y} ^ {2} g (x, y) \left(\nabla_ {y} f (x ^ {*}, y ^ {*}) + \nabla_ {y y} ^ {2} g (x ^ {*}, y ^ {*}) w ^ {*}\right) = 0
$$
we arrive at the conclusion.
C Proof of Lower-level Error
In this section, we give the proof of Lemma 4. To prove (17a) in Lemma 4, by the smoothness of  $g(x,y)$  and the update rule, we have
$$
\begin{array}{l} g (x ^ {k}, y ^ {k, n + 1}) \leq g (x ^ {k}, y ^ {k, n}) + \langle \nabla_ {y} g (x ^ {k}, y ^ {k, n}), y ^ {k, n + 1} - y ^ {k, n} \rangle + \frac {\ell_ {g , 1}}{2} \| y ^ {k, n + 1} - y ^ {k, n} \| ^ {2} \\ \leq g (x ^ {k}, y ^ {k, n}) - \beta \| \nabla_ {y} g (x ^ {k}, y ^ {k, n}) \| ^ {2} + \frac {\beta^ {2} \ell_ {g , 1}}{2} \| \nabla_ {y} g (x ^ {k}, y ^ {k, n}) \| ^ {2} \\ = g \left(x ^ {k}, y ^ {k, n}\right) - \left(\beta - \frac {\beta^ {2} \ell_ {g , 1}}{2}\right) \| \nabla_ {y} g \left(x ^ {k}, y ^ {k, n}\right) \| ^ {2} \\ \stackrel {(a)} {\leq} g \left(x ^ {k}, y ^ {k, n}\right) - \left(2 \beta - \beta^ {2} \ell_ {g, 1}\right) \mu_ {g} \left(g \left(x ^ {k}, y ^ {k, n}\right) - g ^ {*} \left(x ^ {k}\right)\right) \tag {28} \\ \end{array}
$$
where (a) results from Assumption 2. Then subtracting  $g^{*}(x^{k})$  from both sides of (28), we get
$$
g \left(x ^ {k}, y ^ {k, n + 1}\right) - g ^ {*} \left(x ^ {k}\right) \leq \left[ 1 - \left(2 \beta - \beta^ {2} \ell_ {g, 1}\right) \mu_ {g} \right] \left(g \left(x ^ {k}, y ^ {k, n}\right) - g ^ {*} \left(x ^ {k}\right)\right).
$$
By plugging in  $\beta \leq \frac{1}{\ell_{g,1}}$  and applying  $N$  times, we get the conclusion in (17a).
To prove (17b), we decompose the error by
$$
\begin{array}{l} g \left(x ^ {k + 1}, y ^ {k + 1}\right) - g ^ {*} \left(x ^ {k + 1}\right) = \left(g \left(x ^ {k}, y ^ {k + 1}\right) - g ^ {*} \left(x ^ {k}\right)\right) \\ + \underbrace {\left[ \left(g \left(x ^ {k + 1} , y ^ {k + 1}\right) - g ^ {*} \left(x ^ {k + 1}\right)\right) - \left(g \left(x ^ {k} , y ^ {k + 1}\right) - g ^ {*} \left(x ^ {k}\right)\right) \right]} _ {J _ {1}}. \tag {29} \\ \end{array}
$$
To obtain the upper bound of  $J_{1}$ , we first notice that  $g(x, y^{k+1}) - g^{*}(x)$  is  $(\ell_{g,1} + L_g)$ -Lipschitz smooth over  $x$  according to the smoothness of  $g(x, y^{k+1})$  in Assumption 1 and  $g^{*}(x)$  from Lemma 8. Therefore, we have
$$
\begin{array}{l} J _ {1} \leq \langle \nabla_ {x} g (x ^ {k}, y ^ {k + 1}) - \nabla g ^ {*} (x ^ {k}), x ^ {k + 1} - x ^ {k} \rangle + \frac {\ell_ {g , 1} + L _ {g}}{2} \| x ^ {k + 1} - x ^ {k} \| ^ {2} \\ \leq - \alpha \langle \nabla_ {x} g (x ^ {k}, y ^ {k + 1}) - \nabla g ^ {*} (x ^ {k}), d _ {x} ^ {k} \rangle + \frac {\ell_ {g , 1} + L _ {g}}{2} \alpha^ {2} \| d _ {x} ^ {k} \| ^ {2} \\ \leq \alpha \| \nabla_ {x} g (x ^ {k}, y ^ {k + 1}) - \nabla g ^ {*} (x ^ {k}) \| \| d _ {x} ^ {k} \| + \frac {\ell_ {g , 1} + L _ {g}}{2} \alpha^ {2} \| d _ {x} ^ {k} \| ^ {2} \\ \stackrel {(a)} {\leq} \alpha \ell_ {g, 1} d \left(y ^ {k + 1}, S \left(x ^ {k}\right)\right) \| d _ {x} ^ {k} \| + \frac {\ell_ {g , 1} + L _ {g}}{2} \alpha^ {2} \| d _ {x} ^ {k} \| ^ {2} \\ \stackrel {(b)} {\leq} \frac {\alpha \eta_ {1} \ell_ {g , 1}}{2} d \left(y ^ {k + 1}, S \left(x ^ {k}\right)\right) ^ {2} + \frac {\alpha \ell_ {g , 1}}{2 \eta_ {1}} \| d _ {x} ^ {k} \| ^ {2} + \frac {\ell_ {g , 1} + L _ {g}}{2} \alpha^ {2} \| d _ {x} ^ {k} \| ^ {2} \\ \stackrel {(c)} {\leq} \frac {\alpha \eta_ {1} \ell_ {g , 1}}{\mu_ {g}} \left(g \left(x ^ {k}, y ^ {k + 1}\right) - g ^ {*} \left(x ^ {k}\right)\right) + \frac {\alpha \ell_ {g , 1}}{2 \eta_ {1}} \left\| d _ {x} ^ {k} \right\| ^ {2} + \frac {\ell_ {g , 1} + L _ {g}}{2} \alpha^ {2} \left\| d _ {x} ^ {k} \right\| ^ {2} \tag {30} \\ \end{array}
$$
where (a) is derived from Lipschitz continuity of  $\nabla g(x,y)$  and Lemma 8, (b) comes from Cauchy-Swartz inequality, (c) is due to Lemma 7. Plugging (30) into (29) yields the conclusion in (17b).
D Proof of Shadow Implicit Gradient Error
We first present some properties of  $\mathcal{L}(x,y,w)$  and then prove Lemma 5. We define
$$
\mathcal {L} ^ {*} (x, y) = \min _ {w} \mathcal {L} (x, y, w), \quad \mathcal {W} (x, y) = \underset {w} {\arg \min} \mathcal {L} (x, y, w)
$$
D.1 Propositions of  $\mathcal{L}(x,y,w)$  and  $\mathcal{L}^* (x,y)$
Proposition 2. Under Assumption 1-2,  $\mathcal{L}(x,y,w)$  is  $\sigma_g^2$ -PL and  $\ell_{g,1}^2$  Lipschitz smooth over  $w$ .
Proof. According to the definition of  $\mathcal{L}(x,y,w)$ , we can calculate the Hessian of it as
$$
\nabla_ {w w} ^ {2} \mathcal {L} (x, y, w) = \nabla_ {y y} ^ {2} g (x, y) \nabla_ {y y} ^ {2} g (x, y).
$$
Since  $\nabla_{ww}\mathcal{L}(x,y,w)\preceq \ell_{g,1}^2 I$ , we know that  $\mathcal{L}(x,y,w)$  is  $\ell_{g,1}^{2}$ -Lipschitz smooth with respect to  $w$ . On the other hand,  $\mathcal{L}(x,y,w)$  is  $\sigma_g^2$ -PL since  $\frac{1}{2}\| Aw + b\|^2$  is a strongly convex function composited with a linear function which is  $\sigma_{\mathrm{min}}^2 (A)$ -PL according to [33].
D.2 Proof of Lemma 5
According to Proposition 2 and following the same proof of Lemma (17a), we know that
$$
\begin{array}{l} \mathcal {L} \left(x ^ {k}, y ^ {k + 1}, w ^ {k + 1}\right) - \mathcal {L} ^ {*} \left(x ^ {k}, y ^ {k + 1}\right) \leq \left(1 - \rho \sigma_ {g} ^ {2}\right) ^ {T} \left(\mathcal {L} \left(x ^ {k}, y ^ {k + 1}, w ^ {k, 0}\right) - \mathcal {L} ^ {*} \left(x ^ {k}, y ^ {k + 1}\right)\right) \\ \leq \left(1 - \rho \sigma_ {g} ^ {2}\right) ^ {T} \ell_ {f, 0} ^ {2} \\ \end{array}
$$
where the last inequality results from  $\mathcal{L}(x^k, y^{k+1}, w^0) = \| \nabla_y f(x^k, y^{k+1})\|^2 / 2 \leq \ell_{f,0} / 2$  and  $\mathcal{L}^*(x^k, y^{k+1}) \geq 0$ . Then according to the EB condition, we know
$$
d (w ^ {k + 1}, \mathcal {W} (x ^ {k}, y ^ {k + 1})) ^ {2} \leq (1 - \rho \sigma_ {g} ^ {2}) ^ {T} \frac {2 \ell_ {f , 0} ^ {2}}{\mu_ {g}}
$$
On the other hand, since  $w^{k,t} \in \mathrm{Ran}(\nabla_{yy}^2 g(x^k, y^{k+1}))$  according to the update, then  $w^{k+1} \in \mathrm{Ran}(\nabla_{yy}^2 g(x^k, y^{k+1}))$  and thus
$$
\operatorname *{arg  min}_{w\in \mathcal{W}(x^{k},y^{k + 1})}\| w - w^{k + 1}\| = w^{\dagger}(x^{k},y^{k + 1}).
$$
As a result, the bias can be bounded by
$$
b _ {k} ^ {2} = \| w ^ {k + 1} - w ^ {\dagger} (x ^ {k}, y ^ {k + 1}) \| ^ {2} = d (w ^ {k + 1}, \mathcal {W} (x ^ {k}, y ^ {k + 1})) ^ {2} \leq (1 - \rho \sigma_ {g} ^ {2}) ^ {T} \frac {2 \ell_ {f , 0} ^ {2}}{\mu_ {g}}.
$$
E Proof of Upper-level Error
E.1 Propositions of  $w^\dagger(x, y)$
Before proving the UL descent, we first present some property of  $w^{\dagger}(x,y)$ .
Lemma 9 (Lipschitz continuity and boundedness of  $w^\dagger(x, y)$ ). Under Assumption 1-2, for any  $x, x_1, x_2$  and  $y, y_1, y_2$ , it holds that
$$
\| w ^ {\dagger} (x, y) \| \leq \ell_ {f, 0} / \sigma_ {g}, \| w ^ {\dagger} (x _ {1}, y _ {1}) - w ^ {\dagger} (x _ {2}, y _ {2}) \| \leq L _ {w} \| (x _ {1}, y _ {1}) - (x _ {2}, y _ {2}) \|
$$
$$
w i t h L _ {w} = \frac {\ell_ {f , 1}}{\sigma_ {g}} + \frac {\sqrt {2} \ell_ {g , 2} \ell_ {f , 0}}{\sigma_ {g} ^ {2}}.
$$
Proof. First,  $\left(\nabla_{yy}^2 g(x,y)\right)^\dagger$  is bounded since according to Assumption 2,
$$
\left\| \left(\nabla_ {y y} ^ {2} g (x, y)\right) ^ {\dagger} \right\| \leq \frac {1}{\sigma_ {\min} (\nabla_ {y y} ^ {2} g (x , y))} \leq \frac {1}{\sigma_ {g}}.
$$
On the other hand, according to [60, Theorem 3.3],
$$
\begin{array}{l} \left\| \left(\nabla_ {y y} ^ {2} g (x, y)\right) ^ {\dagger} - \left(\nabla_ {y y} ^ {2} g \left(x ^ {\prime}, y ^ {\prime}\right)\right) ^ {\dagger} \right\| \\ \leq \sqrt {2} \max  \left\{\| \left(\nabla_ {y y} ^ {2} g (x, y)\right) ^ {\dagger} \| ^ {2}, \| \left(\nabla_ {y y} ^ {2} g \left(x ^ {\prime}, y ^ {\prime}\right)\right) ^ {\dagger} \| ^ {2} \right\} \| \nabla_ {y y} ^ {2} g (x, y) - \nabla_ {y y} ^ {2} g \left(x ^ {\prime}, y ^ {\prime}\right) \| \\ \leq \frac {\sqrt {2} \ell_ {g , 2}}{\sigma_ {g} ^ {2}} \| (x, y) - \left(x ^ {\prime}, y ^ {\prime}\right) \| \\ \end{array}
$$
which says  $\left(\nabla_{yy}^{2}g(x,y)\right)^{\dagger}$  is  $(\sqrt{2}\ell_{g,2} / \sigma_g^2)$ -Lipschitz continuous.
By the definition (32),  $w^{\dagger}(x,y) = -\left(\nabla_{yy}^{2}g(x,y)\right)^{\dagger}\nabla_{y}f(x,y)$ . Therefore, the boundedness of  $w^{\dagger}(x,y)$  is given by
$$
\| w ^ {\dagger} (x, y) \| \leq \| \left(\nabla_ {y y} ^ {2} g (x, y)\right) ^ {\dagger} \| \| \nabla_ {y} f (x, y) \| \leq \frac {\ell_ {f , 0}}{\sigma_ {g}}.
$$
Besides, for any  $x_{1}, x_{2}$  and  $y_{1}, y_{2}$ , we have
$$
\begin{array}{l} \left\| w ^ {\dagger} \left(x _ {1}, y _ {1}\right) - w ^ {\dagger} \left(x _ {2}, y _ {2}\right) \right\| \\ = \| - \left(\nabla_ {y y} ^ {2} g (x _ {1}, y _ {2})\right) ^ {\dagger} \nabla_ {y} f (x _ {1}, y _ {1}) + \left(\nabla_ {y y} ^ {2} g (x _ {2}, y _ {2})\right) ^ {\dagger} \nabla_ {y} f (x _ {2}, y _ {2}) \| \\ \stackrel {(a)} {\leq} \left\| \left(\nabla_ {y y} ^ {2} g (x _ {1}, y _ {1})\right) ^ {\dagger} \right\| \left\| \nabla_ {y} f (x _ {1}, y _ {1}) - \nabla_ {y} f (x _ {2}, y _ {2}) \right\| \\ + \| \nabla_ {y} f (x _ {2}, y _ {2}) \| \| (\nabla_ {y y} ^ {2} g (x _ {1}, y _ {1})) ^ {\dagger} - (\nabla_ {y y} ^ {2} g (x _ {2}, y _ {2})) ^ {\dagger} \| \\ \stackrel {(b)} {\leq} \left(\frac {\ell_ {f , 1}}{\sigma_ {g}} + \frac {\sqrt {2} \ell_ {g , 2} \ell_ {f , 0}}{\sigma_ {g} ^ {2}}\right) \| (x _ {1}, y _ {1}) - (x _ {2}, y _ {2}) \| \\ \end{array}
$$
where (a) is due to  $C_1D_1 - C_2D_2 = C_1(D_1 - D_2) + D_2(C_1 - C_2)$  and Cauchy-Schwartz inequality and (b) comes from Lemma 9.
E.2 Proof of Lemma 6
We aim to prove the descent of upper-level by
$$
\begin{array}{l} F \left(x ^ {k + 1}, y ^ {k + 1}; w ^ {\dagger} \left(x ^ {k + 1}, y ^ {k + 1}\right)\right) - F \left(x ^ {k}, y ^ {k}; w ^ {\dagger} \left(x ^ {k}, y ^ {k}\right)\right) \\ = \underbrace {F \left(x ^ {k + 1} , y ^ {k + 1} ; w ^ {\dagger} \left(x ^ {k + 1} , y ^ {k + 1}\right)\right) - F \left(x ^ {k + 1} , y ^ {k + 1} ; w ^ {\dagger} \left(x ^ {k} , y ^ {k + 1}\right)\right)} _ {\text {L e m m a 1 1}} \\ + \underbrace {F \left(x ^ {k + 1} , y ^ {k + 1} ; w ^ {\dagger} \left(x ^ {k} , y ^ {k + 1}\right)\right) - F \left(x ^ {k} , y ^ {k + 1} ; w ^ {\dagger} \left(x ^ {k} , y ^ {k + 1}\right)\right)} _ {\text {L e m m a 1 2}} \\ + \underbrace {F \left(x ^ {k} , y ^ {k + 1} ; w ^ {\dagger} \left(x ^ {k} , y ^ {k + 1}\right)\right) - F \left(x ^ {k} , y ^ {k + 1} ; w ^ {\dagger} \left(x ^ {k} , y ^ {k}\right)\right)} _ {\text {L e m m a 1 3}} \\ + \underbrace {F \left(x ^ {k} , y ^ {k + 1} ; w ^ {\dagger} \left(x ^ {k} , y ^ {k}\right)\right) - F \left(x ^ {k} , y ^ {k} ; w ^ {\dagger} \left(x ^ {k} , y ^ {k}\right)\right)} _ {\text {L e m m a 1 4}}. \tag {31} \\ \end{array}
$$
For ease of use, we derive the gradient of  $F(x,y;w^{\dagger}(x,y))$  below.
$$
\nabla_ {x} F (x, y; w ^ {\dagger} (x, y)) = \nabla_ {x} f (x, y) + \nabla_ {x y} ^ {2} g (x, y) w ^ {\dagger} (x, y) \tag {32a}
$$
$$
\nabla_ {y} F (x, y; w ^ {\dagger} (x, y)) = \nabla_ {y} f (x, y) + \nabla_ {y y} ^ {2} g (x, y) w ^ {\dagger} (x, y) \tag {32b}
$$
Lemma 10. Under Assumption I-2,  $F(x,y;w)$  is  $(\ell_{f,1} + \ell_{g,2}\| w\|)$ -smooth over  $x$  and  $y$ .
Proof. This is implied by the Lipschitz smoothness of  $f(x,y)$  and  $\nabla_y g(x,y)$ .
Lemma 11. Under Assumption 1-2, we have
$$
\begin{array}{l} F \left(x ^ {k + 1}, y ^ {k + 1}; w ^ {\dagger} \left(x ^ {k + 1}, y ^ {k + 1}\right)\right) - F \left(x ^ {k + 1}, y ^ {k + 1}; w ^ {\dagger} \left(x ^ {k}, y ^ {k + 1}\right)\right) \\ \leq \frac {\beta \eta_ {2} \ell_ {g , 1} ^ {2}}{\mu_ {g}} \mathcal {R} _ {y} \left(x ^ {k}, y ^ {k}\right) + \left(\frac {\alpha^ {2} L _ {w} ^ {2}}{2 \beta \eta_ {2}} + \ell_ {g, 1} L _ {w} \alpha^ {2}\right) \| d _ {x} ^ {k} \| ^ {2} \tag {33} \\ \end{array}
$$
where  $\eta_{2}$  is a constant that will be chosen in Theorem 2.
Proof. According to the definition of  $F(x,y) = f(x,y) + w^{\top}\nabla_{y}g(x,y)$ , it holds that
$$
\begin{array}{l} F \left(x ^ {k + 1}, y ^ {k + 1}; w ^ {\dagger} \left(x ^ {k + 1}, y ^ {k + 1}\right)\right) - F \left(x ^ {k + 1}, y ^ {k + 1}; w ^ {\dagger} \left(x ^ {k}, y ^ {k + 1}\right)\right) \\ = f (x ^ {k + 1}, y ^ {k + 1}) + \langle \nabla_ {y} g (x ^ {k + 1}, y ^ {k + 1}), w ^ {\dagger} (x ^ {k + 1}, y ^ {k + 1}) \rangle - f (x ^ {k + 1}, y ^ {k + 1}) - \langle \nabla_ {y} g (x ^ {k + 1}, y ^ {k + 1}), w ^ {\dagger} (x ^ {k}, y ^ {k + 1}) \rangle \\ = \left\langle \nabla_ {y} g \left(x ^ {k + 1}, y ^ {k + 1}\right), w ^ {\dagger} \left(x ^ {k + 1}, y ^ {k + 1}\right) - w ^ {\dagger} \left(x ^ {k}, y ^ {k + 1}\right) \right\rangle \\ \leq \langle \nabla_ {y} g (x ^ {k}, y ^ {k + 1}), w ^ {\dagger} (x ^ {k + 1}, y ^ {k + 1}) - w ^ {\dagger} (x ^ {k}, y ^ {k + 1}) \rangle \\ + \left\langle \nabla_ {y} g \left(x ^ {k + 1}, y ^ {k + 1}\right) - \nabla_ {y} g \left(x ^ {k}, y ^ {k + 1}\right), w ^ {\dagger} \left(x ^ {k + 1}, y ^ {k + 1}\right) - w ^ {\dagger} \left(x ^ {k}, y ^ {k + 1}\right) \right\rangle \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(a)} {\leq} \frac {\beta \eta_ {2}}{2} \| \nabla_ {y} g (x ^ {k}, y ^ {k + 1}) \| ^ {2} + \frac {\alpha^ {2} L _ {w} ^ {2}}{2 \beta \eta_ {2}} \| d _ {x} ^ {k} \| ^ {2} + \ell_ {g, 1} L _ {w} \alpha^ {2} \| d _ {x} ^ {k} \| ^ {2} \\ \stackrel {(b)} {\leq} \frac {\beta \eta_ {2} \ell_ {g , 1} ^ {2}}{\mu_ {g}} \mathcal {R} _ {y} (x ^ {k}, y ^ {k}) + \left(\frac {\alpha^ {2} L _ {w} ^ {2}}{2 \beta \eta_ {2}} + \ell_ {g, 1} L _ {w} \alpha^ {2}\right) \| d _ {x} ^ {k} \| ^ {2} \\ \end{array}
$$
where (a) is earned by Young's inequality and the Lipschitz continuity of  $w^{\dagger}(x,y)$  and  $\nabla_y g$ , (b) is resulting from letting  $y^{*} = \arg \min_{y\in S(x^{k})}\| y - y^{k}\|$  and Lemma 7, that is
$$
\begin{array}{l} \| \nabla_ {y} g (x ^ {k}, y ^ {k + 1}) \| ^ {2} = \| \nabla_ {y} g (x ^ {k}, y ^ {k + 1}) - \nabla_ {y} g (x ^ {k}, y ^ {*}) \| ^ {2} \\ \leq \ell_ {g, 1} ^ {2} \| y ^ {k + 1} - y ^ {*} \| ^ {2} = \ell_ {g, 1} ^ {2} d (y ^ {k + 1}, S (x ^ {k})) ^ {2} \\ \leq \frac {2 \ell_ {g , 1} ^ {2}}{\mu_ {g}} \left(g \left(x ^ {k}, y ^ {k + 1}\right) - g ^ {*} \left(x ^ {k}\right)\right) \\ \leq \frac {2 \ell_ {g , 1} ^ {2}}{\mu_ {g}} \left(g \left(x ^ {k}, y ^ {k}\right) - g ^ {*} \left(x ^ {k}\right)\right) = \frac {2 \ell_ {g , 1} ^ {2}}{\mu_ {g}} \mathcal {R} _ {y} \left(x ^ {k}, y ^ {k}\right). \tag {34} \\ \end{array}
$$
Lemma 12. Under Assumption 1-2, we have
$$
\begin{array}{l} F \left(x ^ {k + 1}, y ^ {k + 1}; w ^ {\dagger} \left(x ^ {k}, y ^ {k + 1}\right)\right) - F \left(x ^ {k}, y ^ {k + 1}; w ^ {\dagger} \left(x ^ {k}, y ^ {k + 1}\right)\right) \\ \leq - \left(\frac {\alpha}{2} - \frac {L _ {F}}{2} \alpha^ {2}\right) \| d _ {x} ^ {k} \| ^ {2} + \frac {\alpha \ell_ {g , 1} ^ {2}}{2} b _ {k} ^ {2} \tag {35} \\ \end{array}
$$
where  $b_{k} \coloneqq \| w^{k + 1} - w^{\dagger}(x^{k},y^{k + 1})\|$ , and the constant  $L_{F}$  is defined as  $L_{F} = \ell_{f,0}(\ell_{f,1} + \ell_{g,2}) / \sigma_{g}$ .
Proof. Since  $F(x, y; w)$  is  $(\ell_{f,1} + \ell_{g,2} \| w \|)$  Lipschitz smooth with respect to  $x$ , then according to Lemma 9, we have
$$
\begin{array}{l} F \left(x ^ {k + 1}, y ^ {k + 1}; w ^ {\dagger} \left(x ^ {k}, y ^ {k + 1}\right)\right) \\ \leq F (x ^ {k}, y ^ {k + 1}; w ^ {\dagger} (x ^ {k}, y ^ {k + 1})) + \langle \nabla_ {x} f (x ^ {k}, y ^ {k + 1}) + \nabla_ {x y} ^ {2} g (x ^ {k}, y ^ {k + 1}) w ^ {\dagger} (x ^ {k}, y ^ {k + 1}), x ^ {k + 1} - x ^ {k} \rangle \\ + \frac {L _ {F}}{2} \| x ^ {k + 1} - x ^ {k} \| ^ {2} \\ = F (x ^ {k}, y ^ {k + 1}; w ^ {\dagger} (x ^ {k}, y ^ {k + 1})) - \alpha \langle \nabla_ {x} f (x ^ {k}, y ^ {k + 1}) + \nabla_ {x y} ^ {2} g (x ^ {k}, y ^ {k + 1}) w ^ {\dagger} (x ^ {k}, y ^ {k + 1}), d _ {x} ^ {k} \rangle + \frac {\alpha^ {2} L _ {F}}{2} \| d _ {x} ^ {k} \| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(a)} {=} F (x ^ {k}, y ^ {k + 1}; w ^ {\dagger} (x ^ {k}, y ^ {k + 1})) - \frac {\alpha}{2} \| d _ {x} ^ {k} \| ^ {2} - \frac {\alpha}{2} \| \nabla_ {x} f (x ^ {k}, y ^ {k + 1}) + \nabla_ {x y} ^ {2} g (x ^ {k}, y ^ {k + 1}) w ^ {\dagger} (x ^ {k}, y ^ {k + 1}) \| ^ {2} \\ + \frac {\alpha}{2} \| d _ {x} ^ {k} - \nabla_ {x} f (x ^ {k}, y ^ {k + 1}) - \nabla_ {x y} ^ {2} g (x ^ {k}, y ^ {k + 1}) w ^ {\dagger} (x ^ {k}, y ^ {k + 1}) \| ^ {2} + \frac {\alpha^ {2} L _ {F}}{2} \| d _ {x} ^ {k} \| ^ {2} \\ \end{array}
$$
$$
\stackrel {(b)} {\leq} F (x ^ {k}, y ^ {k + 1}; w ^ {\dagger} (x ^ {k}, y ^ {k + 1})) - \left(\frac {\alpha}{2} - \frac {L _ {F}}{2} \alpha^ {2}\right) \| d _ {x} ^ {k} \| ^ {2} + \frac {\alpha \ell_ {g , 1} ^ {2}}{2} b _ {k} ^ {2}
$$
where (a) is due to  $\langle A,B\rangle = \frac{\|A\|^2}{2} +\frac{\|B\|^2}{2} -\frac{\|A - B\|^2}{2}$ , (b) results from
$$
\left\| d _ {x} ^ {k} - \nabla_ {x} f \left(x ^ {k}, y ^ {k + 1}\right) - \nabla_ {x y} ^ {2} g \left(x ^ {k}, y ^ {k + 1}\right) w ^ {\dagger} \left(x ^ {k}, y ^ {k + 1}\right) \right\|
$$