SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {1}{T} \sum_ {r = 1} ^ {T} | (g \left(x ^ {r}, y ^ {r}\right) - g ^ {*} \left(x ^ {r}\right) - \delta) \lambda^ {r} | ^ {2} = \mathcal {O} \left(\frac {1}{T}\right) \tag {9c}
$$
where  $|\cdot|_{+}$  takes the positive part, and  $T$  denotes the total number of iterations.
Remark 4. From Theorem 1, we can conclude that the iteration complexity of SLM to reach an  $\epsilon$ -KKT stationary point is  $\mathcal{O}(1 / \epsilon)$ . It is also worth noting that this convergence rate is optimal in terms of  $\epsilon$ , as it matches the lower bound established by the first-order method for finding an  $\epsilon$ -stationary point of nonconvex smooth optimization problems [46].
Corollary 1. (Convergence Rate of SLM to  $\epsilon$ -KKT Solutions of the BO problem (2)) Suppose that A1-A5 hold and the local regularity condition holds. Assume that the iterates  $\{x^r, y^r, w^r, z^r, u^r, \lambda^r\}$  are generated by SLM. Given the condition that  $\delta = \epsilon$  and assuming  $\underline{\delta} = \epsilon$ , if the step-sizes  $\eta, \alpha = \mathcal{O}(1 / \lambda^r)$ ,  $\beta = \mathcal{O}(\epsilon^{1.5})$ ,  $\gamma = \mathcal{O}(1 / L_g)$ ,  $\tau = \mathcal{O}(1)$ ,  $T^r = \Omega (\log (1 / \epsilon))$  and  $p, \Lambda = \Theta (1 / \sqrt{\epsilon})$ , then  $\varrho = \mathcal{O}(\sqrt{\epsilon})$  and the iteration complexity of SLM for finding an  $\epsilon$ -KKT stationary solution is  $\mathcal{O}(1 / \epsilon^{3.5})$ .
Remark 5. Here, an  $\epsilon$ -KKT stationary solution of this FCO problem refers to a point  $\{x^{*}, y^{*}\}$  that satisfies  $\| \mathcal{G}(x^{*}, y^{*}) \|^{2} \leq \epsilon$  and  $|g(x^{*}, y^{*}) - g^{*}(x^{*}) - \delta|_{+}^{2} \leq \epsilon$ , and  $|(g(x^{*}, y^{*}) - g^{*}(x^{*}) - \delta)\lambda^{*}|^{2} \leq \epsilon$ .
4 Proof Sketch
In this section, we will present the main theorem proving techniques employed to establish the results in Theorem 1. Let
$$
D (w, z; \lambda) \triangleq \min  _ {x \in \mathcal {X}, y} K (x, y, w, z; \lambda), \tag {10a}
$$
$$
P (w, z) \triangleq \min  _ {x \in \mathcal {X}, y} \max  _ {0 \leq \lambda \leq \Lambda} K (x, y, w, z; \lambda), \tag {10b}
$$
$$
x ^ {*} (w, z; \lambda), y ^ {*} (w, z; \lambda) \triangleq \arg \min  _ {x \in \mathcal {X}, y} K (x, y, w, z; \lambda), \tag {10c}
$$
$$
\bar {x} ^ {*} (w, z), \bar {y} ^ {*} (w, z) \triangleq \arg \min  _ {x \in \mathcal {X}, y} \max  _ {0 \leq \lambda \leq \Lambda} K (x, y, w, z; \lambda). \tag {10d}
$$
These four quantities are associated with the dual error bounds. The first two quantities, given by (10a) and (10b), define the optimal values of  $x, y$  given  $w, z$ , and  $\lambda$ , while the last two quantities, defined by (10c) and (10d), represent the optimal solutions for these intermediate subproblems.
It is obvious that when  $p$  is sufficiently large,  $K(x,y,w,z;\lambda)$  is strongly convex jointly w.r.t.  $x$  and  $y$ . First, under A1 and A5, we can easily show that
$$
d _ {\mathcal {S} \left(x ^ {r}\right)} ^ {2} \left(u _ {T ^ {r}} ^ {r}\right) \leq \mu_ {g} \left(1 - \frac {\gamma}{2 \mu_ {g}}\right) ^ {T ^ {r}} \left(g \left(x ^ {r}, u _ {0} ^ {r}\right) - g ^ {*} \left(x ^ {r}\right)\right), \tag {11}
$$
so under A2 and the assumption that  $u^r$  is initialized within a bounded set, it follows that  $d_{\mathcal{S}(x^r)}^2 (u_{T^r}^r)$  decreases to 0 at a linear rate. This inequality (11) is instrumental in quantifying the bias term  $|g^{*}(x^{r}) - g(x^{r},u^{r + 1})|^{2}$  that arises due to the inaccurate estimation of  $\arg \min_y g(x,y)$ . Next, we will present the following three lemmas which play a crucial role in establishing the proof of Theorem 1.
4.1 Descent Lemmas
After one round update of SLM (i.e., from  $(x^{r},y^{r},w^{r},z^{r},\lambda^{r})$  to  $(x^{r + 1},y^{r + 1},w^{r + 1},z^{r + 1},\lambda^{r + 1})$ , we can obtain the following result.
Lemma 1. Under A1-A5, suppose that the sequence is generated by SLM. When  $\alpha, \eta \leq \mathcal{O}(1 / \lambda^{r + 1})$ ,  $0 < \beta < 1$ ,  $T^r = \Omega (\log (r\lambda^{r + 1}))$ , and  $p > L$ , there exists a constant  $\zeta$  such that
$$
\begin{array}{l} \mathcal {Q} \left(x ^ {r + 1}, y ^ {r + 1}, w ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right) - \mathcal {Q} \left(x ^ {r}, y ^ {r}, w ^ {r}, z ^ {r}; \lambda^ {r}\right) \\ \leq - \left(\frac {1}{4 \eta} - \frac {1 7 \tau \left(\ell_ {g} ^ {2} + \ell_ {g} ^ {\prime 2}\right) \sigma_ {3} ^ {2}}{\eta^ {2}}\right) \| x ^ {r + 1} - x ^ {r} \| ^ {2} - \left(\frac {1}{2 \alpha} - \frac {9 \tau \ell_ {g} ^ {2} \sigma_ {3} ^ {2}}{\alpha^ {2}}\right) \| y ^ {r + 1} - y ^ {r} \| ^ {2} \\ \left. - p \left(\frac {1}{2 \beta} - \left(\frac {1}{\zeta} + \frac {2 p}{p - L} + 3 6 \zeta \sigma_ {1} ^ {2}\right)\right) \| z ^ {r + 1} - z ^ {r} \| ^ {2} - p \left(\frac {1}{2 \beta} - \left(\frac {1}{\zeta} + \frac {2 p}{p - L}\right)\right) \| w ^ {r + 1} - w ^ {r} \| ^ {2} \right. \\ + 3 6 p \zeta \left(\| y ^ {*} (w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} (w ^ {r}, z ^ {r})) - \bar {y} ^ {*} (w ^ {r}, z ^ {r}) \| ^ {2} + \| x ^ {*} (w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} (w ^ {r}, z ^ {r})) - \bar {x} ^ {*} (w ^ {r}, z ^ {r}) \| ^ {2}\right) \\ - \tau \left(\frac {1}{5 6} - 4 8 p \zeta \sigma_ {2} ^ {2} \tau\right) | g (x ^ {*} (w ^ {r}, z ^ {r}; \lambda^ {r + 1}), y ^ {*} (w ^ {r}, z ^ {r}; \lambda^ {r + 1})) - g ^ {*} (x ^ {*} (w ^ {r}, z ^ {r}; \lambda^ {r + 1})) - \delta | ^ {2} \\ + \left(\mu_ {g} L _ {g} ^ {2} + 2 \ell_ {g} ^ {2}\right) \frac {D _ {\mathcal {S}}}{r ^ {2}} - \frac {1}{4 \tau} \| \lambda^ {r + 1} - \lambda^ {r} \| ^ {2} \tag {12} \\ \end{array}
$$
where  $\sigma_{1} \triangleq (p - L) / p$ ,  $\sigma_{2} = (p + L) / (p - L)$ ,  $\sigma_{3} = 1 / (p - L)$ ,  $L = L_{f} + \lambda (2L_{g} + L_{g}^{2} / (2\mu_{g}))$ ,  $\lambda_{+}(w,z) = \mathcal{P}_{+}[\lambda + \tau \nabla_{\lambda}K(x^{*}(w,z;\lambda),y^{*}(w,z;\lambda),w,z;\lambda)]$ ,  $\ell_{g}, \ell_{g}^{\prime}$  denote Lipschitz constant of  $g(x,y)$  and  $g^{*}(x)$  respectively, the potential function is
$$
\mathcal {Q} \left(x ^ {r}, y ^ {r}, w ^ {r}, z ^ {r}; \lambda^ {r}\right) \triangleq K \left(x ^ {r}, y ^ {r}, w ^ {r}, z ^ {r}; \lambda^ {r}\right) - 2 D \left(w ^ {r}, z ^ {r}; \lambda^ {r}\right) + 2 P \left(w ^ {r}, z ^ {r}\right), \tag {13}
$$
and  $D_{\mathcal{S}} \triangleq 2\mu_g^{-1}(\max g(x,u_0) - \min g^* (x))\forall x\in \mathcal{X},u_0\in \mathcal{B}(R)$ .
It can be observed that the coefficients in front of the terms  $\| x^{r + 1} - x^r\| ^2$ ,  $\| y^{r + 1} - y^r\| ^2$ ,  $\| z^{r + 1} - z^r\| ^2$ ,  $\| w^{r + 1} - w^r\| ^2$  and  $|g(x^{*}(w^{r},z^{r};\lambda^{r + 1}),y^{*}(w^{r},z^{r};\lambda^{r + 1})) - g^{*}(x^{*}(w^{r},z^{r};\lambda^{r + 1})) - \delta |^2$  can be negative when the step-sizes are properly chosen and the bias term can be summed up to a constant. Hence, to ensure a sufficient decrease of  $\mathcal{Q}(x^r,y^r,w^r,z^r;\lambda^r)$ , it is necessary to bound the positive terms w.r.t.  $\| y^{*}(w^{r},z^{r};\lambda_{+}^{r + 1}(w^{r},z^{r})) - \bar{y}^{*}(w^{r},z^{r})\|^{2}$  and  $\| x^{*}(w^{r},z^{r};\lambda_{+}^{r + 1}(w^{r},z^{r})) - \bar{x}^{*}(w^{r},z^{r})\|^{2}$ . To accomplish this, two novel dual error bounds are provided as follows.
4.2 Dual Error Bounds
Lemma 2. (Weak Dual Error Bound) Under A1-A3 and A5, suppose that the sequence  $\{x^r,y^r,w^r,z^r,u^r,\lambda^r\}$  is generated by SLM. Then, it holds that
$$
\begin{array}{l} \left\| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {y} ^ {*} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2} + \left\| x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {x} ^ {*} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2} \\ \leq \sigma_ {\text {w e a k}} \| \lambda^ {r + 1} - \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right) \| \| \lambda \left(w ^ {r}, z ^ {r}\right) - \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right) \| \tag {14} \\ \end{array}
$$
$$
\sigma_ {w e a k} \triangleq \frac {1 + \tau \left(2 \ell_ {g} + \ell_ {g} ^ {\prime}\right) \sigma_ {2}}{2 \tau (p - L)}, \quad \lambda (w, z) \in \arg \max  _ {0 \leq \lambda \leq \Lambda} K \left(\bar {x} ^ {*} (w, z), \bar {y} ^ {*} (w, z), w, z; \lambda\right). \tag {15}
$$
Although (14) has quantified the variations between  $y^{*}(w^{r},z^{r};\lambda_{+}^{r + 1}(w^{r},z^{r}))$  and  $\bar{y}^{*}(w^{r},z^{r})$  and between  $x^{*}(w^{r},z^{r};\lambda_{+}^{r + 1}(w^{r},z^{r}))$  and  $\bar{x}^{*}(w^{r},z^{r})$  when the dual variable is perturbed<sup>1</sup>, this bound is in a non-homogeneous form. Therefore, it is not sufficient to demonstrate the  $\mathcal{O}(1 / \epsilon)$  convergence rate of the sequence generated by SLM. Fortunately, under the local regularity condition, we can obtain the following stronger result, which provides further insight.
Lemma 3. (Strong Dual Error Bound) Under A1-A3, A5 and the local regularity condition, suppose that the sequence  $\{x^r, y^r, w^r, z^r, u^r, \lambda^r\}$  is generated by SLM. Then, there exists a constant  $\sigma_{\text{strong}}$  such that
$$
\begin{array}{l} \left\| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {y} ^ {*} \left(w ^ {r}, z ^ {r}\right) \right\| + \left\| x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {x} ^ {*} \left(w ^ {r}, z ^ {r}\right) \right\| \\ \leq 2 \sigma_ {\text {s t r o n g}} \tau \left| g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g ^ {*} \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - \delta \right| \tag {16} \\ \end{array}
$$
$$
\sigma_ {\text {s t r o n g}} \triangleq \sigma_ {\text {w e a k}} \frac {L _ {f} + p + \Lambda L _ {g}}{\sqrt {2 \mu_ {g} \underline {{\delta}}}}. \tag {17}
$$
Given this inequality, we can quantify the sufficient descent of the potential function, which further leads to the convergence rate of SLM.
In this section, we evaluate our proposed algorithm for solving the bilevel optimization related problems and compare the performance of our proposed SLM with the state-of-the-art methods.
5.1 Toy Example
First, we consider the following toy example
$$
\min  _ {x \in [ 1, 2 ]} f (x, y) \triangleq x ^ {2} + y ^ {2} + 3 x \sin^ {2} (y), \quad \text {s . t .} \quad g (x, y) - g ^ {*} (x) \leq \delta \tag {18}
$$
where the LL objective function is  $g(x,y) \triangleq xy^2 + 3x\sin^2(y)$  and  $\delta = 1 \times 10^{-3}$ . It can be easily checked that function  $g(x,y)$  satisfies the PL condition w.r.t. variable  $y$  and function  $f(x,y)$  is nonconvex w.r.t.  $x$  and  $y$  jointly.
In the numerical results, we initialize variables as  $x^{1} = y^{1} = u^{1} = 1$  and choose the step-sizes for updating these variables  $(u,y,x)$  as  $1\times 10^{-3}$  for all the compared methods. Additionally, we set  $p = 1$  and  $\beta = 0.5$  for SLM. It can be seen from Figure 1 that SLM provides the advantage
(a) Constraint violation
(b) LL objective value
(c) UL objective value
Figure 1: Convergence performance of SLM, BOME [15], and V-PBGD [16] on the toy example.
$^1\lambda_+^{r + 1}(w^r,z^r)$  is a perturbation of  $\lambda^{r + 1}$ .
of enforcing the constraint to be satisfied even the LL objective is noncovnex. Moreover, all the algorithms converge to the global optimal solution of the LL problem. It is worth noting that SLM achieves the lowest UL objective value compared to the other methods. This can be attributed to the relaxation introduced by  $\delta$  in achieving the optimal solution of the LL problem, which provides more flexibility in searching for lower UL objective values.
5.2 Data Hyper-Cleaning
Next, we further evaluate SLM for the data hypercleaning problem, which has gained widespread adoption in the machine learning community [3, 16]. The goal of this learning task is to identify and select the clean data samples from the polluted ones, enabling the pre-trained model to generalize effectively on the clean test dataset. The optimization problem is commonly formulated as follows:
$$
\min  _ {x, y} \ell^ {\text {v a l}} (y), \quad \text {s . t .} \quad y \in \arg \min  _ {y ^ {\prime}} \ell^ {\text {t r a i n}} \left(x, y ^ {\prime}\right) \tag {19}
$$
where  $\ell^{\mathrm{val}}$  and  $\ell^{\mathrm{train}}$  represent the validation (at the UL) and training (at the LL) losses, respectively. The LL loss function is defined as  $\ell^{\mathrm{train}}\triangleq \sum_{i = 1}^{m}\sigma (x_{i})\ell (y)$ , where  $m$  denotes the total number
of training data samples,  $\sigma (x)$  is the sigmoid function, and  $y$  denotes the weights of the neural network. Adhering to the experimental settings used to test the V-PBGD as shown in [16], we are given 5,000 training data samples from the MNIST dataset. Among these samples,  $50\%$  are randomly polluted with incorrect labels. Additionally, we have a validation dataset consisting of 5,000 samples and a separate test set comprising 10,000 samples. The model parameter  $y$  includes the neural network weights of a hidden layer with a size of  $10\times 784$  and bias terms with a size of 10.
It has been shown in [16] that V-PBGD outperforms significantly the state-of-the-art methods, including BOME [15], IAPTG [47], RHG [3], T-RHG [4], for this particular example. We adapt the code used to implement V-PBGD to evaluate the performance of SLM and set  $\tau = p = 20$ ,  $\beta = 0.5$ ,  $\delta = 1.2$  for SLM. From Figure 2, it can be observed that SLM initially converges at a similar rate to V-PBGD in terms of the test loss values. However, after a certain number of iterations, SLM achieves smaller loss values. The reason behind this observation is that once the constraint is satisfied, the optimization variables primarily focus on minimizing the UL objective values, leading to lower test losses in this example. Although SLM achieves a test accuracy similar to V-PBGD, with the highest test accuracy of  $90.44\%$ , when the number of iterations becomes large (e.g., exceeding 3,000 in the figure), the model begins to overfit the training data. However, SLM maintains a test accuracy of  $88.8\%$ , while V-PBGD can only attain  $88.1\%$ .
As for the constraint violation, we define  $c_{\mathrm{error}} \triangleq \ell^{\mathrm{train}}(x,y) - v(x)$ , where  $v(x)$  is computed through the updating process of variable  $u$ . This measure represents the distance between the LL objective value and the optimal value. V-PBGD results in a  $c_{\mathrm{error}}$  value of approximately 0.68, whereas SLM achieves a  $(c_{\mathrm{error}} - \delta)$  value of 0.001 when the total number of iterations exceeds 2,000 in the figure. This further reinforces the fact that due to the presence of the optimization variables appearing in both levels or the coupling between the LL and UL variables, it is highly challenging for  $y$  in the BO formulation to find the global optimal solution of the LL problem. In contrast, SLM can find KKT solutions for this class of structured FCO problems as long as the hyper-parameter  $\delta$  is appropriately defined.
5.3 Neural Network on MNIST Data Set
We also evaluate the performance of these algorithms on a 2-layer neural network. The size of the first layer is  $784 \times 300$ , and the size of the second layer is  $300 \times 10$ . The activation function used is the sigmoid function. For all the compared algorithms, we set the step-sizes of the block- $x$  and block- $y$  updates to 1. The step-size  $\gamma$  for the auxiliary block- $u$  update is set to 0.1. Additionally, we choose  $\tau = 0.01$ ,  $\beta = 0.5$ , and  $p = 20$  for SLM, while for BOME, we use  $\eta = 0.1$ . The remaining settings remain the same as mentioned in the main text for the linear case.
The results, averaged over 5 independent trials, are presented in Figure 3. It can be observed from Figure 3(a) that SLM with  $\delta = 1$  achieves the lowest test loss and converges faster compared to
Figure 2: Comparison between SLM and V-PBGD on the data hyper-cleaning task over the MNIST dataset.
(a) Test loss
(b) Test accuracy
Figure 3: Convergence performance of SLM, BOME [15], and V-PBGD [16] on the neural network.
BOME. When  $\delta = 0.5$ , the convergence rate of SLM becomes slower. Both of these observations align with the theoretical convergence analysis. The test loss obtained by V-PBGD initially decreases and then increases, which is reasonable as this method optimizes both the UL (validation) and LL (training) loss values, with the penalty parameter increasing as the number of iterations rises. In contrast, SLM primarily minimizes the objective loss values, specifically the validation loss in this case, leading to better generalization performance as long as the constraint is satisfied. The constraint tolerance parameter  $\delta$  and the dual variable implicitly play the tradeoff between the training and validation losses.
Regarding the test accuracy shown in Figure 3(b), SLM with  $\delta = 1$  achieves a minimum of  $94.74 \pm 0.04\%$ , which is the lowest among all the algorithms compared. V-PBGD can reach  $94.65 \pm 0.09\%$  but converges slower than SLM with  $\delta = 1$ . SLM with  $\delta = 0.5$  and BOME converge relatively quickly initially and reach peak accuracies of  $94.53 \pm 0.06\%$  and  $94.52 \pm 0.11\%$ , respectively. However, it is evident that the test accuracy achieved by BOME decreases rapidly as the algorithm proceeds, even though the test loss continues to decrease, forming a clear U-shape. This implies that BOME overfits the validation data when the number of iterations is large. In contrast, the test accuracy achieved by SLM remains relatively stable, or at least does not decline as rapidly as that attained by BOME. This further confirms that the use of structured constraints in this class aids in improving the generalization performance of bilevel models.
In this work, we focused on solving a class of structured nonconvex FCO problems, where the functional constraint satisfies the PL condition w.r.t. one block of variables. In practical applications, this class of FCO problems can effectively model a wide range of nested or hierarchical learning problems, including data hyper-cleaning, meta-learning, corset selection, and more. To address these challenges, we developed a smoothed primal-dual algorithm based on the Lagrangian method. The proposed algorithm achieves a convergence rate of  $\mathcal{O}(\epsilon^{-1})$  to the  $\epsilon$ -KKT solutions of this problem by utilizing only the first-order oracles for both the objective function and the constraint.
The major difficulty from a theoretical perspective lies in showing the dominance of the descent achieved by the algorithm over the ascent induced by enforcing the nonconvex functional constraint. To the best of our knowledge, this is the first result showcasing that the dual error bound also holds for solving this class of nonlinear constraints. We conducted extensive numerical experiments to evaluate the performance of the proposed SLM against benchmark nonconvex BO methods. The results demonstrate that SLM not only ensures the satisfaction of the functional constraint but also achieves small objective values.
It is important to note that this work focuses on the scenario with a single constraint. Future research endeavors will explore the extension of the algorithm design and theoretical results to more general settings. Addressing this limitation will contribute to a broader applicability of the proposed method.
[38] Y. Wang, W. Yin, and J. Zeng, “Global convergence of admm in nonconvex nonsmooth optimization,” Journal of Scientific Computing, vol. 78, pp. 29–63, 2019.
[39] M. Arbel and J. Mairal, “Non-convex bilevel games with critical point selection maps,” Proceedings of Advances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 8013–8026, 2022.
[40] L. Cannelli, F. Facchinei, V. Kungurtsev, and G. Scutari, "Asynchronous parallel algorithms for nonconvex optimization," Mathematical Programming, vol. 184, pp. 121-154, 2020.
[41] X. Xie, H. Kong, J. Wu, W. Zhang, G. Liu, and Z. Lin, "Maximum-and-concatenation networks," in Proceedings of International Conference on Machine Learning (ICML), pp. 10483-10494, 2020.
[42] S. Liang, R. Sun, and R. Srikant, "Revisiting landscape analysis in deep neural networks: Eliminating decreasing paths to infinity," SIAM Journal on Optimization, vol. 32, no. 4, pp. 2797-2827, 2022.
[43] C. Liu, L. Zhu, and M. Belkin, “Loss landscapes and optimization in over-parameterized nonlinear systems and neural networks,” Applied and Computational Harmonic Analysis, vol. 59, pp. 85–116, 2022.
[44] J. Mei, C. Xiao, C. Szepesvari, and D. Schuurmans, "On the global convergence rates of softmax policy gradient methods," in Proceedings of International Conference on Machine Learning (ICML), pp. 6820-6829, 2020.
[45] M. Fazel, R. Ge, S. Kakade, and M. Mesbahi, “Global convergence of policy gradient methods for the linear quadratic regulator,” in Proceedings of International Conference on Machine Learning (ICML), pp. 1467–1476, 2018.
[46] Y. Carmon, J. C. Duchi, O. Hinder, and A. Sidford, "Lower bounds for finding stationary points," Mathematical Programming, vol. 184, no. 1-2, pp. 71-120, 2020.
[47] R. Liu, Y. Liu, S. Zeng, and J. Zhang, "Towards gradient-based bilevel optimization with nonconvex followers and beyond," Proceedings of Advances in Neural Information Processing Systems (NeurIPS), vol. 34, pp. 8662-8675, 2021.
[48] R. R. Tyrrell, Convex Analysis. Princeton University Press Princeton, 1996.
[49] S. Kakutani, “A generalization of brouwer's fixed point theorem,” Duke Math. J., vol. 8, no. 3, pp. 457–459, 1941.
[50] D. Bertsekas, A. Nedic, and A. Ozdaglar, Convex Analysis and Optimization, vol. 1. Athena Scientific, 2003.
[51] M. Hardt and M. Simchowitz, "Course notes for EE227C (spring 2018): Convex optimization and approximation," 2018. https://ee227c.github.io/notes/ee227c-notes.pdf.
[52] R. Liu, X. Liu, S. Zeng, J. Zhang, and Y. Zhang, “Value-function-based sequential minimization for bi-level optimization,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
In this section, we provide some technical preliminaries for the proofs of the lemmas and theorems claimed in the main body of this paper and the supplementary material, including parameter definitions and supporting results.
A.1 Inequalities
Basic inequalities used in the proof include
1. Strong Convexity: if function  $f(x)$  is strongly convex with modulus  $\mu$ , then
$$
f (x) - f \left(x ^ {\prime}\right) \geq \frac {\mu}{2} \| x - x ^ {\prime} \| ^ {2}. \tag {20}
$$
2. Young's inequality with parameter  $\theta > 0$ :
$$
\langle x, y \rangle \leq \frac {1}{2 \theta} \| x \| ^ {2} + \frac {\theta}{2} \| y \| ^ {2}, \quad \forall x, y. \tag {21}
$$
3. Triangle inequality and reverse triangle inequality:
$$
\| x + y \| \leq \| x \| + \| y \|, \quad \| x - y \| \geq \| x \| - \| y \|. \tag {22}
$$
4. Cauchy-Schwarz inequality
$$
\langle x, y \rangle \leq \| x \| \| y \|. \tag {23}
$$
A.2 Notations
The definitions of the parameters and assumptions are further listed in Table 2.
Note that under A2 (i.e., compactness of the feasible set  $\mathcal{X}$ ) the Lipschitz continuity of  $f(x,y), g(x,y)$  w.r.t.  $x$  holds. In Section C.3, we will demonstrate that the iterates (such as  $y^r$ ,  $\bar{y}^*(w^r, z^r)$ ) for which we need to evaluate the gradients or function values of  $g(x,y)$ ,  $g^*(x)$ , and  $f(x,y)$  are bounded, which implies that the corresponding Lipschitz continuity holds. To be more precise,  $\ell_f, L_g, \ell_g, \ell_g'$  and others are listed in the section of notations in Table 2.
A.3 Primal Error Bounds
Recall that the definition of  $K(x,y,w,z;\lambda)$  is
$$
K (x, y, w, z; \lambda) \triangleq f (x, y) + \lambda \left(g (x, y) - g ^ {*} (x) - \delta\right) + \frac {p}{2} \| x - w \| ^ {2} + \frac {p}{2} \| y - z \| ^ {2}. \tag {24}
$$