Unlocking Global Optimality in Bilevel Optimization: A Pilot Study

## Abstract

Abstract Bilevel optimization has witnessed a resurgence of interest, driven by its critical role in trustworthy and efficient machine learning applications. Recent research has focused on proposing efficient methods with provable convergence guarantees. However, while many prior works have established convergence to stationary points or local minima, obtaining the global optimum of bilevel optimization remains an important yet open problem .
The difficulty lies in the fact that unlike many prior non-convex single-level problems, this bilevel problem does not admit a “benign" landscape, and may indeed have multiple spurious local solutions.
Nevertheless, attaining the global optimality is indispensable for ensuring reliability, safety, and cost-effectiveness, particularly in high-stakes engineering applications that rely on bilevel optimization. In this paper, we first explore the challenges of establishing a global convergence theory for bilevel optimization, and present two sufficient conditions for global convergence.
We provide algorithm-specific proofs to rigorously substantiate these sufficient conditions along the optimization trajectory, focusing on two specific bilevel learning scenarios: representation learning and data hypercleaning (a.k.a. reweighting). Experiments corroborate the theoretical findings, demonstrating convergence to global minimum in both cases.

### 1 Introduction

Bilevel optimization aims to handle two interconnected problems, where one problem is nested within another [].
Recently, bilevel optimization has gained significant attention due to their relevance in various machine learning, signal processing and wireless communication applications, including hyperparameter optimization [, , , ], meta-learning [], representation learning [], reinforcement learning with human feedback [, ], continual learning [, , ], adversarial learning [, ] and neural architecture search []; see recent survey [, , ].
In this paper, we focus on the optimistic bilevel optimization problem as

$$ $\min_{u,v\in\mathcal{S}(u)}~{}f(u,v),~{}~{}\text{ s.t. }~{}~{}\mathcal{S}(u)=\operatorname*{arg\,min}_{v}~{}g(u,v)$ (1) $$

where both the upper-level objective function $f:\mathbb{R}^{d_{1}}\times\mathbb{R}^{d_{2}}\rightarrow\mathbb{R}$ and the lower-level objective function $g:\mathbb{R}^{d_{1}}\times\mathbb{R}^{d_{2}}\rightarrow\mathbb{R}$ are continuously differentiable.

To tackle the above bilevel problems, various efficient algorithms have been proposed with rigorous guarantees on iteration and sample complexity, but most of them are only guaranteed to converge to the stationary points [, , , , , ] or locally optimal solution [, , ] instead of the global optima. However, identifying the global optimal solution for bilevel optimization is vital in various real-world applications where the quality of solutions can have significant impacts. For instance, in policy-making [], energy systems [, ], resource allocation [, , ] and network design [], globally optimal solutions can lead to more cost-efficient and sustainable outcomes than local solutions. Furthermore, in high-stakes fields like healthcare, law, and robotics, attaining global optima ensures that the AI models are aligned with human values with minimal risks of harmful generation [, ].
Thus, the goal of this paper is to study the global convergence (in contrast to convergence to local optima) of bilevel optimization for certain (not all) machine learning applications.

#### 1.1 Our main results

Recent advances in bilevel optimization algorithms can be generally classified into two categories.

Nested approaches solve the problem ([1](#S1.E1)) from its nested formulation $\mathsf{F}(u):=\min_{v\in\mathcal{S}(u)}f(u,v)$ by optimizing first over one variable $v$ and then over the order $u$; see e.g., [, , , ].

Constrained approaches incorporate the optimality condition of the lower-level problem in ([1](#S1.E1)) as a constraint in the upper-level problem and then optimize over $u$ and $v$ jointly; see e.g., [, , , ].

In this paper, we first investigate the loss landscape of the bilevel problem ([1](#S1.E1)) through the lens of nested and constrained bilevel reformulations, demonstrating that the constrained formulation is easier to yield a benign landscape. Then we analyze the landscape of the penalty reformulation of the constrained form of ([1](#S1.E1)), deriving two sufficient conditions ensuring a benign landscape of bilevel problems. Specifically, we define the penalized objective as $\mathsf{L}_{\gamma}(u,v):=f(u,v)+\gamma(g(u,v)-g^{*}(u))$, where $g^{*}(u)$ is the value function of $g^{*}(u):=\min_{v}g(u,v)$ and $\gamma>0$ is a penalty constant, and generalize the standard Polyak-Lojasiewicz (PL) condition to the two-variable case below that we prove to be the sufficient conditions of bilevel global convergence.

###### Definition 1 (Joint and blockwise PL condition) .

$\mathsf{L}_{\gamma}(u,v)$ satisfies joint PL with $\mu_{l}={\cal O}(\gamma)$ if

$$ $\small\|\nabla\mathsf{L}_{\gamma}(u,v)\|^{2}\geq 2\mu_{l}\left(\mathsf{L}_{\gamma}(u,v)-\min_{u,v}\mathsf{L}_{\gamma}(u,v)\right).$ (2) $$

In addition, $\mathsf{L}_{\gamma}(u,v)$ is said to be blockwise PL with $\mu_{u},\mu_{v}={\cal O}(\gamma)$ if both of the followings hold

$$ $\displaystyle\|\nabla_{u}\mathsf{L}_{\gamma}(u,v)\|^{2}\geq 2\mu_{u}\left(\mathsf{L}_{\gamma}(u,v)-\min_{u}\mathsf{L}_{\gamma}(u,v)\right)$ (3a) $\displaystyle\|\nabla_{v}\mathsf{L}_{\gamma}(u,v)\|^{2}\geq 2\mu_{v}\left(\mathsf{L}_{\gamma}(u,v)-\min_{v}\mathsf{L}_{\gamma}(u,v)\right).$ (3b) $$

Under either of the above conditions, we establish that the penalized bilevel gradient descent (PBGD) algorithm [, , , ], a fully first-order method, globally converges to the optimal solutions of ([1](#S1.E1)).
If it satisfies the joint PL condition, we use a Jacobi update (simultaneously update two variables)

$$ $\small v^{k+1}=v^{k}-\alpha\nabla_{v}\mathsf{L}_{\gamma}(u^{k},v^{k})~{}~{}~{}~{}\text{ and }~{}~{}~{}~{}u^{k+1}=u^{k}-\alpha\widehat{\nabla}_{u}\mathsf{L}_{\gamma}(u^{k},v^{k})$ (4a) or if it satisfies the blockwise PL, we use a Gauss-Seidel update (alternatively update two variables) $\small v^{k,t+1}=v^{k,t}-\beta\nabla_{v}\mathsf{L}_{\gamma}(u^{k},v^{k,t}),\,t=0,\cdots,T_{k}~{}~{}\text{ and }~{}~{}u^{k+1}=u^{k}-\alpha\widehat{\nabla}_{u}\mathsf{L}_{\gamma}(u^{k},v^{k,T_{k}})$ (4b) $$

where $\widehat{\nabla}_{u}\mathsf{L}_{\gamma}(u,v)$ is a (possibly biased) gradient estimator of $\nabla_{u}\mathsf{L}_{\gamma}(u,v)$.

We then validate the joint and blockwise PL conditions through two machine learning applications: representation learning and data hyper-cleaning (a.k.a. data reweighting) with a least squares loss.
We provide rigorous proof demonstrating that the loss surfaces reached by PBGD, when employing a Jacobi update and a Gauss-Seidel update, respectively, adhere to the joint PL condition and the blockwise PL condition throughout the optimization trajectory. As a result, we prove for the first time that PBGD is guaranteed to converge to the global optimum for both of these two problems.

###### Definition 1 (Joint and blockwise PL condition) .

#### 1.2 Related works

To put our work in context, we review prior contributions in two categories.

Stationary point and local convergence.
The recent interest in developing efficient gradient-based bilevel methods with nonasymptotic convergence guarantees has been stimulated by [, , , ].
Based on different Hessian inversion approximation techniques, these algorithms can be categorized into iterative differentiation [, , ] and implicit differentiation-based approaches [, , , , ].
Recent works have reformulated the bilevel optimization problem as a single-level constrained problem, and solved it via the penalty-based gradient method []; see also [, , , , ].
While the nonasymptotic analysis of stationary convergence has been extensively studied in bilevel optimization recently, finite-time convergence guarantee for local minimum remains under-explored. Recently, [] and [] have found the benefit of adding noise to gradient-based bilevel methods, which helps them to efficiently escape from saddle points and converge to local minima. However, none of them analyze the landscape and the convergence to global optimal solutions.

Global optimum convergence.
In general, finding the global optimal solution for bilevel optimization is NP-hard []. Historically, globally convergent algorithms for bilevel optimization [, ] were built upon the branch and bound method, a globally convergent single-level algorithm. Despite its theoretical soundness, the branch and bound method is generally inefficient due to its exhaustive search nature. Another line of research focused on the bilevel problems with specific structures. A semi-definite relaxation method has been introduced in [] for polynomial bilevel problems, and a dual reformulation has been developed in [] for quadratic bilevel problems with a linear lower-level. More recent efforts by [, ] solved Stackelberg prediction games with quadratic regularized least squares problems at the lower level by applying semi-definite relaxation and spherical constraints. However, these methods are problem-specific and are not suitable for more complex machine-learning scenarios, particularly where the lower-level problem presents multiple solutions.

Notation. Let $\mathbb{R},\mathbb{R}_{\geq 0},\mathbb{R}_{>0},\mathbb{Z}$ be the sets of real, nonnegative real, positive real and integer numbers. For a given matrix $A\in\mathbb{R}^{p\times q}$, let $\lambda_{i}(A)$ and $\sigma_{i}(A)$ be the $i$-th eigenvalue and singular value in descending order, respectively. Denote $\sigma_{\max}(A)=\sigma_{1}(A)$, $\sigma_{\min}(A)=\sigma_{\min\{p,q\}}(A)$ and $\sigma_{*}(A)=\sigma_{\max\{i|\sigma_{i}(A)>0\}}$ as the maximal, minimal, and minimal nonzero singular values. Let $A_{ij}$ represent the entry at the $i$-th row and $j$-th column of $A$. Let $\|A\|$ and $\|A\|_{2}$ be the Frobenius and spectral norms of $A$. Let $\theta_{>0}$ be $\theta$ if $\theta>0$ and infinity otherwise. Let $\mathds{1}(\cdot)$ denote the indicator function.

### 2 Challenges and Target of Convergence

In this section, we will reveal the complicated landscape of the nested objective $\mathsf{F}(u)=\min_{v\in\mathcal{S}(u)}f(u,v)$ and then state the target of global convergence.

#### 2.1 Challenges in the nested form of bilevel optimization

Establishing the global convergence for bilevel optimization algorithms is fundamentally challenging because the nested bilevel objective $\mathsf{F}(u)$ exhibits different properties compared to the upper- and lower-level objectives $f(u,v)$ and $g(u,v)$. This is primarily due to the distortion induced by $\mathcal{S}(u)$.
To gain some intuition on this distortion, consider the case where both upper and lower-level objectives satisfy the PL condition jointly over
$(u,v)$: there exists $\mu_{f},\mu_{g}>0$ such that

$$ $\displaystyle\|\nabla f(u,v)\|^{2}\geq 2\mu_{f}(f(u,v)-\min_{u,v}f(u,v))\text{ and }\|\nabla g(u,v)\|^{2}\geq 2\mu_{g}(g(u,v)-\min_{u,v}g(u,v)).$ (5) $$

The following example shows that the PL condition on both levels is not sufficient to guarantee the PL condition over the bilevel objective $\mathsf{F}(u)$, even if the lower-level solution mapping $\mathcal{S}(u)$ is linear.

Figure: Figure 1: Visualization of $g(u,v),f(u,v)$ and $\mathsf{F}(u)$ in Example [1](#Thmexample1). In (c), saddle points of $\mathsf{F}(u)$ are identified within the red region, suggesting that $\mathsf{F}(u)$ in Example [1](#Thmexample1) does not fulfill the PL condition.
Refer to caption: /html/2408.16087/assets/figure/g_ex1.png

###### Example 1 .

With $u\in\mathbb{R}$ and $v\in\mathbb{R}$, consider the following upper and lower-level objectives

$$ $\displaystyle f(u,v)=\frac{1}{2}\left(u-\sin(v)\right)^{2}~{}~{}\text{ and }~{}~{}g(u,v)=\frac{1}{2}(u-v)^{2}.$ $$

We can verify that both $f(u,v)$ and $g(u,v)$ satisfy the joint PL condition in ([5](#S2.E5)) and the lower-level problem parameterized by $u$ yields the unique solution $\mathcal{S}(u)=u$. However, the overall bilevel function $\mathsf{F}(u)=f(u,\mathcal{S}(u))=\frac{1}{2}(u-\sin(u))^{2}$ violates the PL condition over $u$. The graph of $\mathsf{F}(u)$ is shown in Figure [1](#S2.F1) and the formal proof is deferred to Appendix [C.1](#A3.SS1).

Similarly, joint convexity of both levels can not ensure the convexity of the bilevel objective $\mathsf{F}(u)$ even if the lower-level solution is unique; see the Example [3](#Thmexample3) in Appendix [C.3](#A3.SS3).

These suggest that the landscape of the nested bilevel objective $\mathsf{F}(u)$ can be easily distorted by the lower-level solution mapping $\mathcal{S}(u)$, which we do not have the direct access to control.

###### Example 1 .

#### 2.2 Seeking global optimum via penalty reformulation

To avoid directly dealing with $\mathcal{S}(u)$, some of recent works have focused on solving the bilevel optimization ([1](#S1.E1)) from the perspective of constrained optimization, where the lower-level problem is treated as a constraint of the upper-level problem; see e.g., [, , , , ] . Defining the value function as $g^{*}(u)=\min_{v}g(u,v)$, the bilevel problem in ([1](#S1.E1)) can be equivalently reformulated as

$$ $\displaystyle\min_{u,v}~{}f(u,v),~{}~{}\textrm{ s.t. }~{}~{}g(u,v)-g^{*}(u)\leq 0.$ (6) $$

An $(\epsilon_{1},\epsilon_{2})$ solution of the problem ([6](#S2.E6)) thus the bilevel problem ([1](#S1.E1)) can then be defined as follows.

###### Definition 2 (An $(\epsilon_{1},\epsilon_{2})$ solution of bilevel problem) .

Given $u^{*}\in\mathbb{R}^{d_{1}},v^{*}\in\mathbb{R}^{d_{2}}$,
we say point $(u^{*},v^{*})$ is an $(\epsilon_{1},\epsilon_{2})$ global solution to bilevel problem ([1](#S1.E1)) if $g(u^{*},v^{*})-g^{*}(u^{*})\leq\epsilon_{2}$ and for any $u\in\mathbb{R}^{d_{1}},v\in\mathbb{R}^{d_{2}}$ satisfying $g(u,v)-g^{*}(u)\leq\epsilon_{2}$, we have $f(u^{*},v^{*})\leq f(u,v)+\epsilon_{1}$.

To find an $(\epsilon_{1},\epsilon_{2})$ solution of ([6](#S2.E6)), one can resort to optimize its penalized reformulation []

$$ $\displaystyle\min_{u,v}~{}\mathsf{L}_{\gamma}(u,v):=f(u,v)+\gamma(g(u,v)-g^{*}(u)).$ (7) $$

It was shown in [, ] that, the $\epsilon$-solution for ([7](#S2.E7)) with a penalty parameter $\gamma={\cal O}(\epsilon^{-0.5})$ is the $(\epsilon,{\cal O}(\epsilon))$ solution for bilevel problem in Definition [2](#Thmdefinition2). Therefore, analyzing the landscape of the penalized objective $\mathsf{L}_{\gamma}(u,v)$ also leads to the global convergence to an $(\epsilon,\epsilon)$ solution of ([6](#S2.E6)).

Benefits of investigating $\mathsf{L}_{\gamma}(u,v)$ instead of $\mathsf{F}(u)$. The continuity and differentiablity of $g^{*}(u)$ is generally easier to be satisfied than that of the solution mapping $\mathcal{S}(u)$; see e.g., [, Example 3B.6]. This makes $\mathsf{L}_{\gamma}(u,v)$ more likely to be continuous and differentiable than $\mathsf{F}(u)$. On the other hand, as $\min_{v}\mathsf{L}_{\gamma}(u,v)$ is an equivalent-dimensional proxy of $\mathsf{F}(u)$ [], $\mathsf{L}_{\gamma}(u,v)$ offers a high dimensional approximation of $\mathsf{F}(u)$, smoothing out the ravine of $\mathsf{F}(u)$ in high-dimensional space; see Figure [2](#S5.F2).

###### Definition 2 (An ( ϵ 1 , ϵ 2 ) subscript italic-ϵ 1 subscript italic-ϵ 2 (\epsilon_{1},\epsilon_{2}) solution of bilevel problem) .

### 3 Global Convergence Condition in Bilevel Optimization

In this section, we will propose counterparts to the global convergence condition from single-level optimization for bilevel optimization based on the penalized constrained formulation ([7](#S2.E7)).

#### 3.1 Benign landscape conditions

To characterize the global convergence in bilevel optimization, we generalize the standard PL condition [], and define the joint PL and blockwise PL conditions of $\mathsf{L}_{\gamma}(u,v)$ in Definition [1](#Thmdefinition1).

Two "bilevel PL" conditions.
The joint PL condition in ([2](#S1.E2)) extends the standard PL condition to a two-variable setting by treating $(u,v)$ as a new single variable, while the blockwise PL condition in ([3](#S1.E3)) reflects the hierarchical structure of bilevel problems by treating the optimization over $u$ and $v$ as separate blocks.
For simplicity, we define condition ([3a](#S1.E3.1)) for the whole space, but it is also sufficient for global convergence if the condition ([3a](#S1.E3.1)) holds only for $v\in\operatorname*{arg\,min}_{v}\mathsf{L}_{\gamma}(u,v)$.
It is also important to note that the joint PL condition and the blockwise PL condition can not imply each other.

Rationale of two PL conditions.
The two PL conditions correspond to bilevel problems with isomorphic and heterogeneous levels, respectively.
Specifically, in representation learning (Section [5](#S5)), the upper-level and lower-level variables are model weights from different layers, maintaining a similar nature; and in data hyper-cleaning (Section [6](#S6)), the lower-level variables are still model weights, but the upper-level variables are the classification parameters for each sample, which is a distinct type of variable from the model weights.

Following [, ], if we choose $\gamma={\cal O}(\epsilon^{-0.5})$, the $\epsilon$-global convergence to the penalized problem ([7](#S2.E7)) implies the convergence to $(\epsilon,{\cal O}(\epsilon))$ global solution of the bilevel problem in Definition [2](#Thmdefinition2). Therefore, we can then focus on developing a globally convergent gradient based algorithm for the penalized problem $\mathsf{L}_{\gamma}(u,v)$ under joint or blockwise PL condition.

From benign landscape to global convergence.
Under the joint PL condition, if we have access to $\nabla\mathsf{L}_{\gamma}(u,v)$, simultaneously updating $(u,v)$ in a Jacobi manner ([4a](#S1.E4.1)) will ensure the global convergence of $\mathsf{L}_{\gamma}(u,v)$.
Under the blockwise PL condition, we can show that $\mathsf{L}_{\gamma}^{*}(u)=\min_{v}\mathsf{L}_{\gamma}(u,v)$ satisfies the PL condition. Then if we have access to $\nabla\mathsf{L}_{\gamma}^{*}(u)$, which can be obtained by the Danskin type theorem [] as
$\nabla\mathsf{L}_{\gamma}^{*}(u)=\nabla_{u}\mathsf{L}_{\gamma}(u,v),\forall v\in\operatorname*{arg\,min}_{v}\mathsf{L}_{\gamma}(u,v)$, then
alternating updating $u$ and $v$ in a Gauss-Seidel manner in ([4b](#S1.E4.2)) ensures the global convergence of $\mathsf{L}_{\gamma}^{*}(u)$ (and thus $\mathsf{L}_{\gamma}(u,v)$).

Figure: Algorithm 1 PBGD in Jacobi fashion

#### 3.2 A globally convergent algorithm: Penalty-based bilevel gradient descent

In this section, we revisit the penalty-based bilevel gradient descent (PBGD) algorithm [, , , ], an efficient fully first order method with provable stationary convergence. We will demonstrate that PBGD reaches the global optimum when the benign conditions are satisfied.

As the name implies, PBGD employs gradient descent on the penalized objective ([7](#S2.E7)). Thanks to the Danskin type theorem [], the gradient of penalized objective can be calculated by

$$ $\displaystyle\nabla_{u}\mathsf{L}_{\gamma}(u,v)=\nabla_{u}f(u,v)+\gamma(\nabla_{u}g(u,v)-\nabla_{u}g(u,w)),~{}\nabla_{v}\mathsf{L}_{\gamma}(u,v)=\nabla_{v}f(u,v)+\gamma\nabla_{v}g(u,v)$ $$

where $w\in\mathcal{S}(u)$. Therefore, at each iteration $k$, PBGD first updates $w^{k}$ to approximate $\mathcal{S}(u^{k})$ by $T_{k}$ step gradient descent on lower-level objective initialized by $w^{k,0}=0$

$$ $\displaystyle w^{k,t+1}=w^{k,t}-\beta\nabla_{v}g(u^{k},w^{k,t})$ (8a) and update $w^{k+1}=w^{k,{T_{k}}}$ . Then we update $u^{k}$ and $v^{k}$ via the gradient descent update on $\mathsf{L}_{\gamma}(u,v)$ simultaneously that we term the Jacobi version if $\mathsf{L}_{\gamma}(u,v)$ is jointly PL $\displaystyle u^{k+1}=u^{k}-\alpha\left(\nabla_{u}f(u^{k},v^{k})+\gamma\left(\nabla_{u}g(u^{k},v^{k})-\nabla_{u}g(u^{k},w^{k+1})\right)\right)$ (8b) $\displaystyle v^{k+1}=v^{k}-\alpha(\nabla_{v}f(u^{k},v^{k})+\gamma\nabla_{v}g(u^{k},v^{k}))$ (8c) or alternatingly that we term the Gauss-Seidel version if $\mathsf{L}_{\gamma}(u,v)$ is blockwise PL (with $v^{k,0}=v^{0}$ as the initialization and $v^{k+1}=v^{k,T_{k}}$ as the output) $\displaystyle v^{k,t+1}=v^{k,t}-\tilde{\beta}(\nabla_{v}f(u^{k},v^{k,t})+\gamma\nabla_{v}g(u^{k},v^{k,t})),\,t=1,2,\cdots,T_{k}$ (8d) $\displaystyle u^{k+1}=u^{k}-\alpha\left(\nabla_{u}f(u^{k},v^{k+1})+\gamma\left(\nabla_{u}g(u^{k},v^{k+1})-\nabla_{u}g(u^{k},w^{k+1})\right)\right).$ (8e) $$

The Jacobi and Gauss-Seidel versions of PBGD are summarized in Algorithm [1](#alg1) and [2](#alg2), respectively.

To prove the convergence, we make the following assumptions; see also in [, , , , , , ].

###### Assumption 1 .

Assume $f(u,v)$ and $g(u,v)$ are $\ell_{f}$- and $\ell_{g}$-smooth over $(u,v)$, $f(u,\cdot)$ is $\ell_{f,0}$-Lipschitz continuous over $v$, and $g(u,\cdot)$ is $\mu_{g}$-PL over $v$.

The following theorem shows that PBGD is globally convergent to the bilevel problem with almost linear convergence rate when the penalized problem has a benign landscape.

###### Theorem 1 (Global convergence of PBGD) .

Suppose Assumption [1](#Thmassumption1) holds, then $g^{*}(u)$ is smooth with $L_{g}:=\ell_{g}(1+\ell_{g}/2\mu_{g})$. Given a target accuracy $\epsilon$, we set $\gamma={\cal O}(\epsilon^{-0.5})$, stepsizes $\beta\leq\frac{1}{\ell_{g}}$, inner loop $T_{k}={\cal O}\left(\log\left(\gamma^{2}{\epsilon}^{-1}\right)\right)$. Then if $\mathsf{L}_{\gamma}(u,v)$ satisfies the joint PL condition ([2](#S1.E2)) with $\mu_{l}>0$ and we choose $\alpha\leq\frac{1}{\ell_{f}+\gamma(\ell_{g}+L_{g})}$, there exists $\epsilon_{\gamma}={\cal O}(\epsilon)$ s.t. the iterates of PBGD in Algorithm [1](#alg1) satisfies

$$ $\displaystyle f(u^{K},v^{K})-f(u,v)$ $\displaystyle\leq{\cal O}((1-\alpha\mu_{l})^{K})+{\cal O}(\epsilon)~{}~{}\text{ and }~{}~{}g(u^{K},v^{K})-\min_{v}g(u^{K},v)\leq\epsilon_{\gamma}={\cal O}(\epsilon)$ $$

for any $(u,v)$ with $g(u,v)-\min_{v}g(u,v)\leq\epsilon_{\gamma}={\cal O}(\epsilon)$.
Alternatively, if $\mathsf{L}_{\gamma}(u,v)$ satisfies the blockwise PL condition ([3](#S1.E3)) with $\mu_{u},\mu_{v}>0$ and we choose the stepsizes $\tilde{\beta}\leq\frac{1}{\ell_{f}+\gamma\ell_{g}},\alpha\leq\frac{1}{L_{\gamma}}$ where $L_{\gamma}:=(\ell_{f}+\gamma\ell_{g})(1+(\ell_{f}+\gamma\ell_{g})/2\mu_{v})+L_{g}$, then the iterates of PBGD in Algorithm [2](#alg2) satisfies

$$ $\displaystyle f(u^{K},v^{K+1})-f(u,v)$ $\displaystyle\leq{\cal O}((1-\alpha\mu_{u})^{K})+{\cal O}(\epsilon)~{}\text{ and }~{}g(u^{K},v^{K+1})-\min_{v}g(u^{K},v)\leq\epsilon_{\gamma}$ $$

for any $(u,v)$ with $g(u,v)-\min_{v}g(u,v)\leq\epsilon_{\gamma}={\cal O}(\epsilon)$.

Theorem [1](#Thmtheorem1) shows that, with properly selected stepsizes, PBGD employing either the Jacobi update under the joint PL condition or the Gauss-Seidel update under the blockwise PL condition, converges to an $({\cal O}(\epsilon),{\cal O}(\epsilon))$ solution for the bilevel problem within ${\cal O}(K\max T_{k})={\cal O}(\log(\epsilon^{-1})^{2})$ iterations. The proof of Theorem [1](#Thmtheorem1) is provided in Appendix [D](#A4).

###### Remark 1 .

It is worth mentioning that other first-order bilevel algorithms based on the penalty formulation ([7](#S2.E7)), such as F^2SA [, , ] and BOME [], could also have global convergence. We provided the global convergence based on the PBGD algorithm [] here as an example.

###### Assumption 1 .

###### Theorem 1 (Global convergence of PBGD) .

###### Remark 1 .

### 4 Ensuring Global Convergence Conditions

In this section, we will provide some key observations to help us establish the global convergence conditions in Section [3.1](#S3.SS1). Let us first revisit the definition of the penalized objective

$$ $\displaystyle\mathsf{L}_{\gamma}(u,v)~{}~{}~{}=~{}~{}~{}\overbrace{\underset{\underset{{\text{\scriptsize joint PL \& blockwise PL }}}{\uparrow}}{f(u,v)}+\underbrace{\gamma(g(u,v)-g^{*}(u)).}_{\text{Observation}~{}\ref{ob1}:\text{ joint PL \& blockwise PL over }v}}^{\text{Additivity of PL functions? Observation }\ref{ob2}}$ $$

We have the first observation on the landscape of $g(u,v)-g^{*}(u)$ over $(u,v)$ and that over $v$.

###### Observation 1 .

​Under Assumption [1](#Thmassumption1), $g(u,v)\!-\!g^{*}(u)$ is joint PL over $(u,v)$ and blockwise PL over $v$.

Therefore, if the upper-level objective additionally satisfies joint PL or blockwise PL condition, then whether $\mathsf{L}_{\gamma}(u,v)$ satisfies joint PL or blockwise PL condition over $v$ depends on the additivity of PL functions. In general, the sum of two joint/blockwise PL functions is not necessarily a joint/blockwise PL function. For example, in Example [1](#Thmexample1), joint PL in both levels leads to non-joint PL penalized objective. For the proof and the counterexamples in blockwise PL condition, see Appendix [C.2](#A3.SS2).

The lack of the additivity of PL functions in general impedes the development of a unified global convergence theory for bilevel problems. However, we highlight another key observation to establish the PL condition of $\mathsf{L}_{\gamma}(u,v)$ for a special class of problem. This observation implies that a strongly convex function composite with a linear mapping preserves the PL condition under additivity.

###### Observation 2 .

Let $h_{1}:\mathbb{R}^{m}\rightarrow\mathbb{R}$ and $h_{2}:\mathbb{R}^{n}\rightarrow\mathbb{R}$ be strongly convex functions with constant $\mu_{1}$ and $\mu_{2}$, respectively. Given any matrix $A\in\mathbb{R}^{m\times d},B\in\mathbb{R}^{n\times d}$, $h_{1}(Az)$ and $h_{2}(Bz)$ satisfy the PL condition over $z\in\mathbb{R}^{d}$ with constant $\mu_{1}\sigma_{*}(A)$ and $\mu_{2}\sigma_{*}(B)$, respectively. Moreover, $h_{1}(Az)+h_{2}(Bz)$ satisfies the PL condition with constant $\min\{\mu_{1}\sigma_{*}^{2}(A),\mu_{2}\sigma_{*}^{2}(B)\}$.

This observation is particularly valuable for analyzing global convergence in bilevel optimization problems involving linear neural networks with strongly convex loss functions, such as the least squares loss or cross-entropy loss within a bounded region. This may also shed light on a broader range of bilevel problems involving sophisticated neural network architectures in machine learning.

Dealing with problem-specific challenges.
In Sections [5](#S5) and [6](#S6), we will show how to use our bilevel global convergence framework in two problems: representation learning and data hyper-cleaning. Unfortunately, the sufficient benign landscape conditions in Section [3.1](#S3.SS1) are not directly satisfied in those specific problems. Nevertheless, we will present a fine-tuned analysis to judiciously verify the joint PL condition and blockwise PL condition along the optimization trajectory of PBGD.

###### Observation 1 .

###### Observation 2 .

### 5 Global Convergence in Representation Learning

In representation learning, we are given a training dataset $\{x_{i},y_{i}\}_{i=1}^{N}$ and a validation dataset $\{\tilde{x}_{i},\tilde{y}_{i}\}_{i=1}^{N^{\prime}}$ with data sample $x_{i},\tilde{x}_{i}\in\mathbb{R}^{m}$ and label $y_{i},\tilde{y}_{i}\in\mathbb{R}^{n}$. We aim to train a model capable of excelling with unseen data by adjusting only the top layer’s weights $W_{1}$ while keeping the backbone model $W_{2}$ fixed. For simplicity, we consider the two-layer linear neural network and the mean square loss in this paper, so the resultant bilevel problem can be formulated as

where $X_{{\rm trn}}=\left[x_{1},\cdots,x_{N}\right]^{\top}\in\mathbb{R}^{N\times m},X_{{\rm val}}=\left[\tilde{x}_{1},\cdots,\tilde{x}_{N^{\prime}}\right]^{\top}\in\mathbb{R}^{N^{\prime}\times m}$ are the training and validation data matrix, $Y_{{\rm trn}}=\left[y_{1},\cdots,y_{N}\right]^{\top}\in\mathbb{R}^{N\times n},Y_{{\rm val}}=\left[\tilde{y}_{1},\cdots,\tilde{y}_{N^{\prime}}\right]^{\top}\in\mathbb{R}^{N^{\prime}\times n}$ are the training and validation label matrix and $W_{1}\in\mathbb{R}^{m\times h},W_{2}\in\mathbb{R}^{h\times n}$ are the weight matrix. We consider the overparameterized case with $m\geq\max\{N,N^{\prime}\}$ so that $\mathcal{S}(W_{1})$ has multiple solutions. We also assume the neural network is wide $h\geq\max\{m,n\}$ and $X_{\rm trn},X_{\rm val}$ are of full row rank (^1^11Our theory also works for rank-deficient case by changing $\sigma_{\min}(X_{\rm trn}),\sigma_{\min}(X_{\rm trn})$ to $\sigma_{*}(X_{\rm trn}),\sigma_{*}(X_{\rm trn})$.).

Let us denote $\mathsf{L}_{\textrm{trn}}(W_{1},W_{2})=\frac{1}{2}\left\|Y_{{\rm trn}}-X_{{\rm trn}}W_{1}W_{2}\right\|^{2}$, $\mathsf{L}_{\textrm{val}}(W_{1},W_{2})=\frac{1}{2}\left\|Y_{{\rm val}}-X_{{\rm val}}W_{1}W_{2}\right\|^{2}$ as the loss of two-layer linear neural network. We are interested in the scenario where the backbone model, when learned from the validation dataset, also achieves a good fit on the training dataset. This ensures that there are hidden patterns applicable across training and validation datasets so that a generalizable backbone model exists.
Note that although we employ a two-layer neural network structure to model both the backbone and adaptation layer for representation learning, our analysis is adaptable to multi-layer neural networks as well.
To this end, we make the following assumption.

###### Assumption 2 .

For any $\epsilon_{1},\epsilon_{2}\in\mathbb{R}_{\geq 0}$, there exists $(W_{1}^{*},W_{2}^{*})$ being the $(\epsilon_{1},\epsilon_{2})$ solution to the bilevel representation learning problem ([9](#S5.E9)) such that $\mathsf{L}_{\rm trn}(W_{1}^{*},W_{2}^{*})-\min_{W_{1},W_{2}}\mathsf{L}_{\rm trn}(W_{1},W_{2})\leq\epsilon_{2}$.

Assumption [2](#Thmassumption2) ensures that the bilevel problem has at least one full-rank solution $W_{1}^{*}$ so that is not degenerate. We present some *sufficient conditions* for it in Appendix [F.1](#A6.SS1).

Figure: Figure 2: The landscape of $\mathsf{F}(W_{1})$ and $\mathsf{L}_{\gamma}(W_{1},W_{2})$ with different penalty constant $\gamma=0.1,1,50$ in representation learning. The orange terrain is $\mathsf{F}(W_{1})$, while the blue surface is $\mathsf{L}_{\gamma}(W_{1},W_{2})$. The black line is the trajectory of PBGD which converges to the global optimum of bilevel loss.
Refer to caption: /html/2408.16087/assets/figure/gamma0.1.png

To solve ([9](#S5.E9)), we resort to its penalized function as stated ([7](#S2.E7)) with the upper-level function $f(u,v)=\mathsf{L}_{\rm val}(W_{1},W_{2})$, and the lower-level objective $g(u,v)=\mathsf{L}_{\rm trn}(W_{1},W_{2})$, which is defined as

$$ $\displaystyle\mathsf{L}_{\gamma}(W_{1},W_{2}):=\mathsf{L}_{\rm val}(W_{1},W_{2})+\gamma(\mathsf{L}_{\rm trn}(W_{1},W_{2})-\mathsf{L}_{\rm trn}^{*}(W_{1}))$ (10) $$

where $\mathsf{L}_{{\rm trn}}^{*}(W_{1})=\min_{W_{2}}\mathsf{L}_{\textrm{trn}}(W_{1},W_{2})$ and we run PBGD [1](#alg1) with $(u,v,w)=(W_{1},W_{2},W_{3})$.

To verify the global convergence condition ([2](#S1.E2)), we first observe that both $\mathsf{L}_{\rm val}(W_{1},W_{2})$ and $\mathsf{L}_{\rm trn}(W_{1},W_{2})$ are in the form of a quadratic function composite with a linear mapping. Moreover, we find that $\mathsf{L}_{\rm trn}^{*}(W_{1})=0$ if $W_{1}$ is of full row rank and $W_{1}^{k}$ will maintain full row rank on the trajectory generated by PBGD, so that $\mathsf{L}_{\rm trn}(W_{1},W_{2})-\mathsf{L}_{\rm trn}^{*}(W_{1})$ will be in the form of a quadratic function composite with a linear mapping along the optimization trajectory. Therefore, Observation [2](#Thmobservation2) implies that $\mathsf{L}_{\gamma}(W_{1},W_{2})$ is a PL function. Figure [2](#S5.F2) shows the landscape comparison of $\mathsf{F}(W_{1}):=\min_{W_{2}\in\mathcal{S}(W_{1})}\mathsf{L}_{\rm val}(W_{1},W_{2})$ and $\mathsf{L}_{\gamma}(W_{1},W_{2})$ in representation learning. From Figure [2](#S5.F2), it can be seen that the nested bilevel function $\mathsf{F}(W_{1})$ is nonconvex and discontinuous, while the penalized loss $\mathsf{L}_{\gamma}(W_{1},W_{2})$ has a smoother landscape, effectively steering the weight matrix updated by PBGD towards the global optimal solution. Formally, we have the following lemma.

###### Lemma 1 (Joint PL condition and descent lemma over trajectory) .

Let $X_{\gamma}=[X_{\rm val};\sqrt{\gamma}X_{\rm trn}]$ be the concatenated data matrix. Then under Assumption [2](#Thmassumption2), if $\sigma_{\min}^{2}(W_{1}^{k})>0$, $\sigma_{\min}^{2}(W_{2}^{k})>0$, the joint PL inequality holds with $\mu_{k}=(\sigma_{\min}^{2}(W_{1}^{k})+\sigma_{\min}^{2}(W_{2}^{k}))\sigma_{*}^{2}(X_{\gamma})$, that is

$$ $\displaystyle\|\nabla\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\|^{2}$ $\displaystyle\geq 2\mu_{k}(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\min_{W_{1},W_{2}}\mathsf{L}_{\gamma}(W_{1},W_{2})).$ $$

The descent lemma holds with $L_{k}$ defined in ([52](#A6.Ex130)) and $\|\delta_{k}\|$ being the estimation error of $\nabla\mathsf{L}_{\gamma}^{*}(W_{1}^{k})$

$$ $\displaystyle\mathsf{L}_{\gamma}(W_{1}^{k+1},W_{2}^{k+1})$ $\displaystyle\leq{\mathsf{L}}_{\gamma}(W_{1}^{k},W_{2}^{k})-\left(\frac{\alpha}{2}-\alpha^{2}L_{k}\right)\|\nabla{\mathsf{L}}_{\gamma}(W_{1}^{k},W_{2}^{k})\|^{2}+\left(\frac{\alpha}{2}+\alpha^{2}L_{k}\right)\|\delta_{k}\|^{2}.$ $$

To this end, using the joint PL condition together with the standard descent lemma leads to one-step contraction in the optimality gap of the penalized objective. Through induction, we can then establish the lower bounds of $\sigma_{\min}^{2}(W_{1}^{k})$ and $\sigma_{\min}^{2}(W_{2}^{k})$, as well as the upper bounds of $\sigma_{\max}^{2}(W_{1}^{k})$ and $\sigma_{\max}^{2}(W_{2}^{k})$. Consequently, these yield $k$-independent lower and upper bounds for $\mu_{k}$ and $L_{k}$, which enable us to demonstrate the almost linear convergence of PBGD.

###### Theorem 2 (Global convergence of PBGD for representation learning) .

Under Assumption [2](#Thmassumption2) and choose $\gamma={\cal O}(\epsilon^{-0.5}),T_{k}={\cal O}(\log(\epsilon^{-1}))$, there exists $\mu={\cal O}(\gamma)$, $L={\cal O}(\gamma),\alpha={\cal O}(\gamma^{-1})$, such that $\mu_{k}\geq\mu,L_{k}\leq L$, and the iteration complexity of PBGD in Algorithm [1](#alg1) to achieve $(\epsilon,\epsilon)$ global optimal point of the bilevel representation learning problem ([9](#S5.E9)) is ${\cal O}(\log^{2}(\epsilon^{-1}))$.

###### Assumption 2 .

###### Lemma 1 (Joint PL condition and descent lemma over trajectory) .

###### Theorem 2 (Global convergence of PBGD for representation learning) .

### 6 Global Convergence in Data Hyper-cleaning

Data hyper-cleaning aims to leverage a small, clean validation dataset to enhance the quality of a larger training dataset, which may be hindered by some noisy or unreliable data points. This process is widely used when the access to clean data is costly but the noisy data is not like in recommendation systems [, ].
To do so, we are given a training dataset $\{x_{i},y_{i}\}_{i=1}^{N}$ with data sample $x_{i}\in\mathbb{R}^{m}$ and label $y_{i}\in\mathbb{R}^{n}$, where each label may be corrupted. We are also given the clean validation data $\{\tilde{x}_{i},\tilde{y}_{i}\}_{i=1}^{N^{\prime}}$ to guide the training. Let $u\in\mathbb{R}^{N}$ be the classification vector trained to label the noisy data and $W\in\mathbb{R}^{m\times n}$ be the model weight, the bilevel problem of data hyper-cleaning is given by
$\min_{u\in\mathcal{U},W\in S(u)}~{}~{}\frac{1}{2}\sum_{i=1}^{N^{\prime}}(\tilde{y}_{i}-\tilde{x}_{i}^{\top}W)^{2},~{}\textrm{ s.t. }~{}\mathcal{S}(u)=\operatorname*{arg\,min}_{W}~{}\frac{1}{2}\sum_{i=1}^{N}\psi(u_{i})(y_{i}^{\top}-x_{i}^{\top}W)^{2}$
where $\mathcal{U}=[-\bar{u},\bar{u}]^{N}$ is used to avoid trivial solution $u_{i}=-\infty$ and $\psi(\cdot)$ is the sigmoid function.

Let $X_{{\rm trn}}=\left[x_{1},\cdots,x_{N}\right]^{\top}\in\mathbb{R}^{N\times m},X_{{\rm val}}=\left[\tilde{x}_{1},\cdots,\tilde{x}_{N^{\prime}}\right]^{\top}\in\mathbb{R}^{N^{\prime}\times m}$, $Y_{{\rm trn}}=\left[y_{1},\cdots,y_{N}\right]^{\top}\in\mathbb{R}^{N\times n},Y_{{\rm val}}=\left[\tilde{y}_{1},\cdots,\tilde{y}_{N^{\prime}}\right]^{\top}\in\mathbb{R}^{N^{\prime}\times n}$ be the training and validation data and label matrix, $\psi_{N}(\cdot)$ denotes the diagnal matrix of element wise sigmoid function and $\sqrt{\cdot~{}}$ denote the elementwise square root operator. We can formulate the data hyper-cleaning problem as the following bilevel problem

Note that for data hyper-cleaning, where the primary concern involves the interaction between the classification parameter $u$ and the model weight $W$, we adopt a one-layer neural network structure for simplicity in notations. Nonetheless, our results can also be extended to multi-layer linear networks.

We consider the overparameterized case with $m\geq\max\{N,N^{\prime}\}$. To model a corrupted training setting, we assume the concatenate data matrix $[X_{\rm val};X_{\rm trn}]$ is rank-deficient so that the training and validation objectives do not share a common weight minimizer, but there exists a parameter $u^{*}\in\mathcal{U}$ such that the selected concatenated matrix $[X_{\rm val};\sqrt{\psi_{N}(u^{*})}X_{\rm trn}]$ is nearly full rank by neglecting the rows with small $\psi(u_{i})$, meaning that
$\operatorname{rank}([X_{\rm val};\sqrt{\psi_{N}(u^{*})}X_{\rm trn}])=N+N^{\prime}-\big{|}\{i:\psi(u_{i}^{*})<\tilde{\psi}\}\big{|}$,
where $\tilde{\psi}>0$ is a threshold close to 0 0 and $\big{|}\cdot\big{|}$ denotes the cardinality of the set $\{i:\psi(u_{i}^{*})<\tilde{\psi}\}$.

Let us denote $\ell_{\textrm{trn}}(u,W)=\frac{1}{2}\|\sqrt{\psi_{N}(u)}(Y_{{\rm trn}}-X_{{\rm trn}}W)\|^{2}$ and the value function in data hyper-cleaning as $\ell_{\textrm{trn}}^{*}(u)=\min_{W}\ell_{\textrm{trn}}(u,W)$. To solve ([11](#S6.E11)), we resort to its penalized reformulation

$$ $\displaystyle\ell_{\gamma}(u,W):=\ell_{\rm val}(W)+\gamma(\ell_{\rm trn}(u,W)-\ell_{\rm trn}^{*}(u))$ (12) $$

and run PBGD in Algorithm [2](#alg2) with $(u,v,w)=(u,W,Z)$. Since both $\ell_{\rm val}(W)$ and $\ell_{\rm trn}(u,W)$ are in the form of strongly convex functions composite with a linear mapping, $\ell_{\gamma}(u,W)$ is blockwise PL with respect to $W$ from Observation [2](#Thmobservation2). To prove the blockwise PL condition over $u$, we first show that $\mathcal{S}(u)$ is independent of $u$ and obtain the form of $\ell_{\rm trn}^{*}(u)$ by plugging in the closed form of $\mathcal{S}(u)$, for which we defer the details in Lemma [23](#Thmlemma23) in Appendix [G.1](#A7.SS1). As a result, the penalized objective has the closed-form expression as

$$ $\displaystyle\ell_{\gamma}(u,W)=\ell_{\rm val}(W)+\frac{\gamma}{2}\sum_{i=1}^{N}\psi(u_{i})\left[\|y_{i}^{\top}-x_{i}^{\top}W\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right].$ (13) $$

Thus the landscape of $\ell_{\gamma}(u,W)$ over $u$ is fully characterized by the sigmoid function, which is PL on $u\in\mathcal{U}$. Consequently, $\ell_{\gamma}(u,W)$ is blockwise PL. We formalize our findings in the next lemma.

###### Lemma 2 (Blockwise PL condition) .

If $X_{\rm trn}X_{\rm trn}^{\dagger}$ is a diagonal matrix, then for any $u\in\mathcal{U}$ and $W$, there exists $L_{w}^{\gamma},\mu_{w}^{\gamma}={\cal O}(\gamma)$ such that
$\ell_{\gamma}(u,W)$ is $L_{w}^{\gamma}$-smooth and $\mu_{w}^{\gamma}$-PL over $W$. Moreover, $\ell_{\gamma}(u,W)$ is $\gamma\ell_{\rm trn}(W)$ smooth and $\frac{\gamma c(W)\psi(\bar{u})(1-\psi(\bar{u}))^{2}}{4}$-PL over $u\in\mathcal{U}$, where $c(W)$ is defined as

$$ $\displaystyle c(W)=\min_{i}\left\{\|y_{i}^{\top}-x_{i}^{\top}W\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right\}_{>0}$ $$

which is the lower bound of the positive mismatch in the training loss.

Although the blockwise PL constant of $\ell_{\gamma}(u,W)$ depends on $W$, we can derive a uniform positive lower bound of $\mu_{u}:=\min_{u\in\mathcal{U}}c(W^{*}_{\gamma}(u))$ based on the acute matrix perturbation theory, and thus, we can obtain the global convergence results for PBGD (Algorithm [2](#alg2)) on data hyper-cleaning problem.

###### Theorem 3 (Global convergence of PBGD in data hyper-cleaning) .

Suppose $X_{\rm trn}X_{\rm trn}^{\dagger}$ is a diagonal matrix. If the stepsizes in Algorithm [2](#alg2) satisfy $\alpha\leq\frac{1}{L_{u}}={\cal O}(1/\gamma)$, $\beta\leq\frac{1}{L_{w}},\tilde{\beta}\leq\frac{1}{L_{w}^{\gamma}},T_{k}={\cal O}(\log(\epsilon^{-1}))$, and $\bar{u}\geq 1$, then the iteration complexity of PBGD in Algorithm [2](#alg2) to achieve $(\epsilon,\epsilon)$ global optimal point of the bilevel data hyper-cleaning problem ([11](#S6.E11)) is ${\cal O}(\log(\epsilon^{-1})^{2})$.

###### Lemma 2 (Blockwise PL condition) .

###### Theorem 3 (Global convergence of PBGD in data hyper-cleaning) .

### 7 Numerical Experiments

We test the convergence of PBGD in Algorithm [1](#alg1) for representation learning and in Algorithm [2](#alg2) for data hyper-cleaning; see results in Figure [3](#S7.F3) and the detailed setup in Appendix [H](#A8). It can be observed that PBGD converges almost at a linear rate to a certain accuracy, and the relative error decreases as $\gamma$ increases. The fluctuation in the upper-level error near convergence in representation learning is due to the global convergence result in Theorem [1](#Thmtheorem1) not being strictly decreasing because of the ${\cal O}(\epsilon)$ error at each step. When upper-level and lower-level errors are sufficiently small, the additional ${\cal O}(\epsilon)$ error has larger impact, leading to a slight increase in error. However, the final error remains small, around $10^{-5}$ when $K$ and $\gamma$ is large as $\gamma=10,500$. This validates Theorem [1](#Thmtheorem1) that by setting $\gamma$ large enough, PBGD globally converges to a target accuracy inversely determined by $\gamma$ at almost linear rate.

Figure: Figure 3: Relative errors at upper-level and lower-level in $\log$ scale of PBGD versus iteration $K$ under different $\gamma$, where $\mathsf{L}_{\rm val}^{*}=\min_{W_{1},W_{2}\in\mathcal{S}(W_{1})}\mathsf{L}_{\rm val}(W_{1},W_{2})$ and $\ell_{\rm val}^{*}=\min_{u,W\in\mathcal{S}(u)}\ell_{\rm val}(W)$. (a)–(b) are for PBGD [1](#alg1) in representation learning, and (c)-(d) are for PBGD [2](#alg2) in data hyper-cleaning.
Refer to caption: /html/2408.16087/assets/figure/upper-level.png

### 8 Conclusions, Broader impacts and Limitations

In this paper, we proposed two global convergence conditions for bilevel problems, tailored to isomorphic and heterogeneous bilevel learning problems. We proved that PBGD, in either Jacobi or Gauss-Seidel fashions, converges to the global optimal solution at an almost linear rate under these conditions respectively. These global conditions were rigorously verified through representation learning and data hyper-cleaning. Numerical results confirm that PBGD is globally convergent. One limitation of this work is that the problem-specific global convergence analysis focuses on a particular square loss and a linear neural network architecture. It is possible to extend our global convergence analysis to
nonlinear cases via means of random features and neural tangent kernels in future work.

This is the first work analyzing the global convergence of bilevel optimization in the context of modern machine learning applications.
As global convergence has been thoroughly investigated in single-level neural network training, min-max optimization and meta-learning, exploring global convergence in bilevel optimization presents a promising future direction. This could also shed light on neural network training in multi-modality multi-criteria learning. Unlike previous problems, the global convergence of bilevel optimization depends on the coupling structure between the upper-level and lower-level tasks. Different coupling modes lead to varying properties of the overall objective. Our work provides a global convergence analysis framework for bilevel optimization, distinguishing between isomorphic and heterogeneous upper and lower-level tasks. However, the current relations remains application-specific, and we have not developed an effective way to directly link different coupling modes in bilevel problems to distinct global conditions (joint or blockwise PL condition).

### Acknowledgement

The work of Q. Xiao and T. Chen was supported by National
Science Foundation MoDL-SCALE project 2134168, the RPI-IBM Artificial Intelligence Research Collaboration (AIRC) and an Amazon Research Award. We also thank Ziqing Xu and Liuyuan Jiang for their insightful feedback and constructive suggestions, which are instrumental in enhancing the clarity and quality of our paper.

## 

### Appendix A Additional Related Works

Stationary point convergence.
The recent interest in developing efficient gradient-based bilevel methods and analyzing their nonasymptotic convergence rate has been stimulated by [, , , ].
Based on different Hessian inversion approximation techniques, these algorithms can be categorized into iterative differentiation [, , ] and approximate implicit differentiation-based approaches [, , , , ]. Later on, the research has been extended to variance reduction and momentum based methods [, , ]; warm-started algorithms [, , ]; distributed approaches [, , ]; and methods that are able to tackle with constrained [, , ] or non-strongly-convex lower-level problem [, , , , ]. Recent works have reformulated the bilevel optimization problem as a single-level constrained problem, and solved it via the penalty-based gradient method []; see also [, , , , ]. This method not only further enhanced the computational efficiency by using only first-order information, but also broadened the applicability of the bilevel framework to a wider class of problems. Nevertheless, none of these attempts the issue of global optimum convergence.

### Appendix B Auxiliary Lemmas

##### Additional Notations.

Let matrix $A\in\mathbb{R}^{m\times n}$. The range space of $A$ is denoted as $\operatorname{Ran}(A)=\{Ay:y\in\mathbb{R}^{n}\}$, while the null space of $A$ is given by $\operatorname{Ker}(A)=\{y\in\mathbb{R}^{n}:Ay=0\}$. Let $A_{ij}$ represent the element located at the $i$-th row and $j$-th column of matrix $A$. Let $\|A\|$ and $\|A\|_{2}$ be the Frobenius norm and spectral norm of $A$, respectively. $\mathbb{B}(A,r):=\left\{A^{\prime}\in\mathbb{R}^{p\times q}\mid\|A^{\prime}-A\|\leq a\right\}$ denotes the closed ball with center $A$ and radius $a$. Let $\theta\in\mathbb{R}^{d}$ be a point and $S\subset\mathbb{R}^{d}$ be a set, we define the distance of $\theta$ to $S$ as $d(\theta,S)=\inf\{\|\theta-a\|\mid a\in S\}.$ Let us denote $\ell_{\textrm{trn}}(W)=\frac{1}{2}\left\|Y_{{\rm trn}}-X_{{\rm trn}}W\right\|^{2},\ell_{\textrm{val}}(W)=\frac{1}{2}\left\|Y_{{\rm val}}-X_{{\rm val}}W\right\|^{2}$ as the square loss.

###### Lemma 3 (Matrix Inequality) .

$$ $\displaystyle\langle A,B\rangle\leq\|A\|\cdot\|B\|,$ (14) $\displaystyle 2\|AB\|\leq\|A\|^{2}+\|B\|^{2},$ (15) $\displaystyle\|AB+CD\|^{2}\leq\left[\sigma_{\max}^{2}(A)+\sigma_{\max}^{2}(C)\right]^{2}\cdot\left[\|B\|^{2}+\|D\|^{2}\right],$ (16) $\displaystyle\|A\|^{2}+\|B\|^{2}\leq 2\|A+B\|^{2},$ (17) $\displaystyle\sigma_{\max}(AB)\leq\sigma_{\max}(A)\sigma_{\max}(B).$ (18) $$

Moreover, if $A$ has full column rank and $B\neq 0$ or if $A\neq 0$ and $B$ has full row rank, it holds that

$$ $\displaystyle\sigma_{\min}(AB)\geq\sigma_{\min}(A)\sigma_{\min}(B)$ (19) $$

###### Lemma 4 ( [ 80 , Lemma B.3] ) .

Let $A\in\mathbb{R}^{p\times s}$ be a rank-$s$ matrix. Then for any $B\in\mathbb{R}^{s\times q}$, we have

$$ $\sigma_{\min}(A)\|B\|\leqslant\|AB\|\leqslant\sigma_{\max}(A)\|B\|.$ $$

###### Lemma 5 .

In representation learning, we consider the overparameterized wide neural network case with $m\geq\max\{N,N^{\prime}\},h\geq\max\{m,n\}$ and $X_{\rm trn},X_{\rm val}$ are in full row rank. Therefore, we have,

$$ $\displaystyle\min_{W}\ell_{\rm trn}(W)=0,~{}~{}\min_{W_{1},W_{2}}{\mathsf{L}}_{\rm trn}(W_{1},W_{2})=0,~{}~{}\min_{W}\ell_{\rm val}(W)=0,~{}~{}\min_{W_{1},W_{2}}{\mathsf{L}}_{\rm val}(W_{1},W_{2})=0$ $\displaystyle\min_{W}(\ell_{\rm trn}+\gamma\ell_{\rm val})(W)=\min_{W_{1},W_{2}}(\mathsf{L}_{{\rm trn}}+\gamma\mathsf{L}_{{\rm trn}})(W_{1},W_{2})$ $$

holds for any $\gamma$.
Moreover, for $W_{1}$ with $\sigma_{\min}(W_{1})>0$, it holds that
$\mathsf{L}_{\rm trn}^{*}(W_{1})=\min_{W_{2}}{\mathsf{L}}_{\rm trn}(W_{1},W_{2})=0.$

###### Lemma 6 ( [ 33 , Theorem 2] ) .

We say a function $h(\theta)$ is $\mu_{h}$ PL if it satisfies

$$ $\displaystyle\|\nabla h(\theta)\|^{2}\geq 2\mu_{h}(h(\theta)-h^{*})$ $$

where $h^{*}=\min h(\theta)$.
If $h(\theta)$ is $L_{h}$-Lipschitz smooth and PL in $\theta$ with $\mu_{h}$, then it satisfies the error bound condition with $\mu_{h}$, i.e.

$$ $\displaystyle\|\nabla h(\theta)\|\geq\mu_{h}d(\theta,\mathcal{S}).$ (20) $$

where $\mathcal{S}=\operatorname*{arg\,min}h(\theta)$.
Moreover, it also satisfies the quadratic growth condition with $\mu_{h}$, i.e.

$$ $\displaystyle h(\theta)-h^{*}\geq\frac{\mu_{h}}{2}d(\theta,\mathcal{S})^{2}.$ (21) $$

where $h^{*}=\min h(\theta)$.
Conversely, if $h(\theta)$ is $L_{h}$-Lipschitz smooth and satisfies the error bound condition with $\mu_{h}$, then it is PL in $\theta$ with $\mu_{h}/L_{h}$.

###### Lemma 7 ( [ 51 , Theorem 5.2] ) .

Suppose that $h(\theta)$ is $L_{h}$ Lipschitz smooth and $\mu_{h}$ PL over $\theta$. If we execute $T$-step gradient descent following $\theta^{t+1}=\theta^{t}-\beta\nabla h(\theta^{t})$ with $\beta\leq\frac{1}{L_{h}}$, the output satisfies

$$ $\displaystyle h(\theta^{T})-\min h(\theta)\leq\left(1-\beta\mu_{h}\right)^{T}\left(h(\theta^{0})-\min h(\theta)\right)$ $$

Moreover, the iterates generated by gradient descent is bounded, i.e.

$$ $\displaystyle\|\theta_{T}-\theta_{0}\|\leq\sum_{t=0}^{\infty}\left\|\theta_{t+1}-\theta_{t}\right\|_{\ell_{2}}\leq\sqrt{\frac{8(h(\theta_{0})-\min h(\theta_{0}))}{\mu_{h}}}$ $$

The following lemma gives the gradient of the value function.

###### Lemma 8 ( [ 50 , Lemma A.5] ).

Suppose $g(u,v)$ is $\mu_{g}$ PL over $v$ and is $\ell_{g,1}$- Lipschitz smooth, then $g^{*}(u)$ is differentiable with the gradient

$$ $\displaystyle\nabla g^{*}(u)=\nabla_{u}g(u,v),~{}~{}\forall v\in\mathcal{S}(u).$ $$

Moreover, $g^{*}(u)$ is $L_{g}$-smooth with $L_{g}:=\ell_{g}(1+\ell_{g}/2\mu_{g})$.

###### Lemma 9 .

If $\|\delta_{k}\|\leq\delta$ and $\sigma_{\min}(W_{1}^{k})>0,\sigma_{\min}(W_{2}^{k})>0$, then the following inequalities hold

$$ $\displaystyle~{}~{}~{}~{}~{}2\left\|\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\|$ $\displaystyle\leq\|\nabla\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\|^{2}+\|\nabla\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\|^{2}$ (22) $\displaystyle~{}~{}~{}~{}~{}\left\|\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\|$ $\displaystyle\leq 2\sigma_{\max}^{2}(X_{\gamma})\sigma_{\max}(W^{k})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})+\delta\sigma_{\max}(W_{1}^{k})\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})}$ (23) $\displaystyle~{}~{}~{}~{}~{}\left\|\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)W_{2}^{k}+W_{1}^{k}\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\|$ $\displaystyle\leq\left(\sigma_{\max}^{2}(W_{1}^{k})+\sigma_{\max}^{2}(W_{2}^{k})\right)\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})}+\delta\sigma_{\max}(W_{2}^{k})$ (24) $$

Proof:
([22](#A2.Ex15)) holds from ([15](#A2.E15)) by letting $A=\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}$ and $B=\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})$.

Note that when $\sigma_{\min}(W_{1}^{k})>0,\sigma_{\min}(W_{2}^{k})>0$, we have $\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})=\widetilde{\mathsf{L}}_{\gamma}(W_{1}^{k},W_{2}^{k})=\ell_{\gamma}(W^{k})$ where $\ell_{\gamma}=\ell_{\rm val}+\gamma\ell_{\rm trn}$. Thus ([23](#A2.Ex17)) can be derived from

$$ $\displaystyle~{}~{}~{}~{}~{}\left\|\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\|$ $\displaystyle\leq\left\|\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\|+\delta\left\|\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\|$ $\displaystyle\leq\|\nabla\ell_{\gamma}(W^{k})(W_{2}^{k})^{\top}(W_{1}^{k})^{\top}\nabla\ell_{\gamma}(W^{k})\|+\delta\left\|\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\|$ $\displaystyle\leq\|\nabla\ell_{\gamma}(W^{k})W^{k,\top}\nabla\ell_{\gamma}(W^{k})\|+\delta\left\|(W_{1}^{k})^{\top}\nabla\ell_{\gamma}(W^{k})\right\|$ $\displaystyle\leq\sigma_{\max}(W^{k})\|\nabla\ell_{\gamma}(W^{k})\|^{2}+\delta\sigma_{\max}(W_{1}^{k})\|\nabla\ell_{\gamma}(W^{k})\|$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})+\delta\sigma_{\max}(W_{1}^{k})\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})}$ $$

where $W^{k}=W_{1}^{k}W_{2}^{k}$ and (a) holds because $\ell_{\gamma}$ is $\sigma_{\max}^{2}(X_{\gamma})$- smooth so that

$$ $\displaystyle\|\nabla\ell_{\gamma}(W^{k})\|^{2}$ $\displaystyle\leq 2\sigma_{\max}^{2}(X_{\gamma})(\ell_{\gamma}(W^{k})-\ell_{\gamma}^{*})\overset{\eqref{relation-lL},\eqref{M_2}}{\leq}2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*}).$ (25) $$

Likewise, we can derive ([24](#A2.Ex19)) similarly by

$$ $\displaystyle~{}~{}~{}~{}\left\|\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)W_{2}^{k}+W_{1}^{k}\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\|$ $\displaystyle\leq\left\|\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})W_{2}^{k}\right\|+\delta\sigma_{\max}(W_{2}^{k})+\|W_{1}^{k}\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\|$ $\displaystyle\leq\left\|\nabla\ell_{\gamma}(W^{k})(W_{2}^{k})^{\top}W_{2}^{k}\right\|+\delta\sigma_{\max}(W_{2}^{k})+\|W_{1}^{k}(W_{1}^{k})^{\top}\nabla\ell_{\gamma}(W^{k})\|$ $\displaystyle\leq\left(\sigma_{\max}^{2}(W_{1}^{k})+\sigma_{\max}^{2}(W_{2}^{k})\right)\|\nabla\ell_{\gamma}(W^{k})\|+\delta\sigma_{\max}(W_{2}^{k})$ $\displaystyle\overset{\eqref{inter2}}{\leq}\left(\sigma_{\max}^{2}(W_{1}^{k})+\sigma_{\max}^{2}(W_{2}^{k})\right)\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})}+\delta\sigma_{\max}(W_{2}^{k}).$ $$

###### Lemma 10 (Pseudoinverse of matrix product) .

For two real matrix $M_{1}\in\mathbb{R}^{m\times m}$ and $M_{2}\in\mathbb{R}^{m\times n}$, if $M_{1}$ is a diagonal matrix and is invertiable and $M_{2}M_{2}^{\dagger}$ is diagonal, then we have $(M_{1}M_{2})^{\dagger}=M_{2}^{\dagger}M_{1}^{-1}$.

Proof:
According to [, Theorem 2], $(M_{1}M_{2})^{\dagger}=M_{2}^{\dagger}M_{1}^{\dagger}=M_{2}^{\dagger}M_{1}^{-1}$ holds if and only if both $M_{1}^{\dagger}M_{1}M_{2}M_{2}^{\top}$ and $M_{1}^{\top}M_{1}M_{2}M_{2}^{\dagger}$ are Hermitian matrix. As $M_{1}$ and $M_{2}$ are real matrix, it remains to prove that both $M_{1}^{\dagger}M_{1}M_{2}M_{2}^{\top}$ and $M_{1}^{\top}M_{1}M_{2}M_{2}^{\dagger}$ are symmetric. By the invertiability of $M_{1}$,

$$ $\displaystyle M_{1}^{\dagger}M_{1}M_{2}M_{2}^{\top}$ $\displaystyle=M_{1}^{-1}M_{1}M_{2}M_{2}^{\top}=M_{2}M_{2}^{\top}$ $$

so $M_{1}^{\dagger}M_{1}M_{2}M_{2}^{\top}$ is symmetric.

For $M_{1}^{\top}M_{1}M_{2}M_{2}^{\dagger}$, the starting point is a fact that diagonal matrix is commutative with diagonal matrix. This is because for diagonal matrix $\Lambda_{1}$ and diagonal matrix $\Lambda_{2}$, it holds that

$$ $\displaystyle\Lambda_{1}\Lambda_{2}=(\Lambda_{2}^{\top}\Lambda_{1}^{\top})^{\top}=(\Lambda_{2}\Lambda_{1})^{\top}=\Lambda_{2}\Lambda_{1}$ $$

where the last equation is due to the symmetricity of $\Lambda_{2}\Lambda_{1}$.

Since both $M_{1}^{\top}M_{1}$ and $M_{2}M_{2}^{\dagger}$ are diagonal matrix, we then have

$$ $\displaystyle M_{1}^{\top}M_{1}M_{2}M_{2}^{\dagger}$ $\displaystyle=M_{2}M_{2}^{\dagger}M_{1}^{\top}M_{1}=(M_{1}^{\top}M_{1}M_{2}M_{2}^{\dagger})^{\top}$ $$

so that $M_{1}^{\top}M_{1}M_{2}M_{2}^{\dagger}$ is also symmetric, which completes the proof.

###### Definition 3 ( [ 63 , Acute matrix] ) .

Let $P_{A}=AA^{\dagger},P_{B}=BB^{\dagger}$ be the orthogonal projection matrix onto the range space $\operatorname{Ran}(A)$ and $\operatorname{Ran}(B)$. We say matrix $A\in\mathbb{R}^{m\times n}$ and $B\in\mathbb{R}^{m\times n}$ are acute if $\|P_{A}-P_{B}\|<1$. Moreover, a class of parameterized matrix family $\{A(u)\},u\in\mathcal{U}$, is said to be acute if for any $u^{1},u^{2}\in\mathcal{U}$, $A(u^{1})$ and $A(u^{2})$ are acute.

###### Proposition 1 ( [ 63 ] ) .

If $A\in\mathbb{R}^{m\times n}$ and $B\in\mathbb{R}^{m\times n}$ are acute, then $\operatorname{Ran}(A)=\operatorname{Ran}(B)$.

###### Lemma 11 ( [ 63 , Theorem 2.5] ) .

Matrix $A\in\mathbb{R}^{m\times n}$ and $B\in\mathbb{R}^{m\times n}$ are acute if and only if

$$ $\displaystyle\operatorname{rank}(A)=\operatorname{rank}(B)=\operatorname{rank}(P_{A}BR_{A})$ $$

where $P_{A}=AA^{\dagger},R_{A}=A^{\dagger}A$ are the orthogonal projection matrix onto the column space $\operatorname{Ran}(A)$ and the row space $\operatorname{Ran}(A^{\top})$.

###### Lemma 12 (Rank equations [ 22 ] ) .

For $A\in\mathbb{R}^{m\times n},B\in\mathbb{R}^{v\times p},C\in\mathbb{R}^{p\times q}$, if $A$ is full column rank, and $C$ is full row rank, then it holds that

$$ $\displaystyle\operatorname{rank}(AB)=\operatorname{rank}(B),~{}~{}\operatorname{rank}(BC)=\operatorname{rank}(B).$ $$

Moreover, for any $D\in\mathbb{R}^{m\times n}$, $\operatorname{rank}(D^{\top}D)=\operatorname{rank}(D)$.

###### Lemma 3 (Matrix Inequality) .

###### Lemma 4 ( [ 80 , Lemma B.3] ) .

###### Lemma 5 .

###### Lemma 6 ( [ 33 , Theorem 2] ) .

###### Lemma 7 ( [ 51 , Theorem 5.2] ) .

###### Lemma 8 ( [ 50 , Lemma A.5] ).

###### Lemma 9 .

###### Lemma 10 (Pseudoinverse of matrix product) .

###### Definition 3 ( [ 63 , Acute matrix] ) .

###### Proposition 1 ( [ 63 ] ) .

###### Lemma 11 ( [ 63 , Theorem 2.5] ) .

###### Lemma 12 (Rank equations [ 22 ] ) .

### Appendix C Proof for Examples 1 and 2

#### C.1 Proof for Example 1

The gradients of the two objectives in Example [1](#Thmexample1) can be computed as

$$ $\displaystyle\nabla f(u,v)=\left[\begin{array}[]{l}u-\sin(v)\\ -\left(u-\sin(v)\right)\cos(v)\end{array}\right]~{}~{}~{}\text{ and }~{}~{}~{}\nabla g(u,v)=\left[\begin{array}[]{l}u-v\\ -(u-v)\end{array}\right].$ $$

Besides, also using the fact that $\min_{u,v}f(u,v)=\min_{u,v}g(u,v)=0$, we have

$$ $\displaystyle\|\nabla f(u,v)\|^{2}$ $\displaystyle=\left(u-\sin(v)\right)^{2}\times\left(1+\cos(v)^{2}\right)$ $\displaystyle\geq 2\times\frac{1}{2}\left(u-\sin(v)\right)^{2}=2\times\left(f(u,v)-\min_{u,v}f(u,v)\right)$ $\displaystyle\|\nabla g(u,v)\|^{2}$ $\displaystyle=2(u-v)^{2}\geq 4\times\frac{1}{2}\left(u-v\right)^{2}=4\times\left(g(u,v)-\min_{u,v}g(u,v)\right)$ $$

which suggests $f(u,v)$ is $2$-PL and $g(u,v)$ is $1$-PL. The lower-level solution set is earned by setting $u=v$. In this way, the overall objective is $\mathsf{F}(u)=f(u,\mathcal{S}(u))=\frac{1}{2}(u-\sin(u))^{2}$ and it is minimized when $u=0$. By calculating its gradient $\nabla F(u)=(u-\sin(u))(1-\cos(u))$, we have

$$ $\displaystyle\|\nabla F(u)\|^{2}$ $\displaystyle=(u-\sin(u))^{2}(1-\cos(u))^{2}.$ $$

Since $(1-\cos(u))^{2}=0$ when $u=2k\pi,k\in\mathbb{Z}$. Therefore, there does not exist a positive $\mu$ such that

$$ $\displaystyle\|\nabla F(u)\|^{2}$ $\displaystyle=(u-\sin(u))^{2}(1-\cos(u))^{2}\geq 2\mu(F(u)-\min F(u))=\mu(u-\sin(u))^{2}$ $$

holds for any $u$. This means $\mathsf{F}(u)$ is not PL in $u$.

Figure: Figure 4: Visualization of $g(u,v),f(u,v)$ and $\mathsf{F}(u)$ in Example [3](#Thmexample3).
Refer to caption: /html/2408.16087/assets/figure/g_ex2.png

Figure: Figure 5: Visualization of $\mathsf{L}_{\gamma}(u,v)$ in Example [1](#Thmexample1) and [2](#Thmexample2). Saddle points exist for $\mathsf{L}_{\gamma}(u,v)$ in Example [1](#Thmexample1) and Example [2](#Thmexample2), suggesting that penalized objective does not satisfy the PL condition.
Refer to caption: /html/2408.16087/assets/figure/penalty_ex1_3.png

#### C.2 PL functions do not preserve PL condition under additivity in general

The following lemma shows that joint PL upper- and lower-level objectives can lead to non-joint PL penalized objectives.

###### Lemma 13 .

In Example [1](#Thmexample1), the joint PL condition on both levels does not guarantee the joint PL condition of $\mathsf{L}_{\gamma}(u,v)$; see also the landscape of the penalized function shown in Figure [5](#A3.F5).

Proof:
For Example [1](#Thmexample1), as $g^{*}(u)=0$ for any $u$, the gradient of $\mathsf{L}_{\gamma}(u,v)$ can be computed as

$$ $\displaystyle\nabla\mathsf{L}_{\gamma}(u,v)=\left[\begin{array}[]{l}u-\sin(v)+\gamma(u-v)\\ -\left(u-\sin(v)\right)\cos(v)-\gamma(u-v)\end{array}\right].$ $$

On the other hand, $\mathsf{L}_{\gamma}(u,v)$ is minimized when $u=v=0$ with the minimal value 0 0. For any $\gamma$, there exists $(\bar{u},\bar{v})$ with $\bar{u}=\frac{2\gamma\pi}{1+\gamma},\bar{v}=2\pi$ such that

$$ $\displaystyle\nabla\mathsf{L}_{\gamma}(\bar{u},\bar{v})=\left[\begin{array}[]{l}\bar{u}-\sin(\bar{v})+\gamma(\bar{u}-\bar{v})\\ -\left(\bar{u}-\sin(\bar{v})\right)\cos(\bar{v})-\gamma(\bar{u}-\bar{v})\end{array}\right]=\left[\begin{array}[]{l}\bar{u}+\gamma(\bar{u}-2\pi)\\ -\bar{u}-\gamma(\bar{u}-2\pi)\end{array}\right]=\left[\begin{array}[]{l}0\\ 0\end{array}\right].$ $$

However, $\mathsf{L}_{\gamma}(\bar{u},\bar{v})=\frac{1}{2}(\bar{u}-\sin(\bar{v}))^{2}+\frac{\gamma}{2}(\bar{u}-\bar{v})^{2}>0=\min_{u,v}\mathsf{L}(\bar{u},\bar{v})$. This means that saddle points $(\bar{u},\bar{v})$ exist for any $\gamma$ so that $\mathsf{L}_{\gamma}(\bar{u},\bar{v})$ does not satisfy the joint PL condition.

Similarly, the following example shows
that blockwise PL conditions on upper and lower-level objectives are also not sufficient to ensure the
blockwise PL condition of $\mathsf{L}_{\gamma}(u,v)$.

###### Example 2 .

With $u\in\mathbb{R}$ and $v=[v_{1},v_{2}]\in\mathbb{R}^{2}$, consider the bilevel problem with the following blockwise PL upper-level objective $f(u,v)$ and the blockwise PL lower-level objective $g(u,v)$, i.e.,

$$ $\displaystyle f(u,v)=\frac{1}{2}\left(\frac{u}{2}+v_{1}-\sin(v_{2})\right)^{2}~{}~{}~{}\text{ and }~{}~{}~{}g(u,v)=\frac{1}{2}\left(u+v_{1}+\sin(v_{2})\right)^{2}$ $$

where $g^{*}(u)=0$. However, the penalized loss function $\mathsf{L}_{\gamma}(u,v)$ is not blockwise PL. The landscape of $\mathsf{L}_{\gamma}(u,v)$ when $v_{1}=2$ is shown in Figure [5](#A3.F5).

Proof:
The gradients of two objectives in Example [2](#Thmexample2) can be computed as

$$ $\displaystyle\nabla f(u,v)=\left[\begin{array}[]{l}\frac{1}{2}\left(\frac{u}{2}+v_{1}-\sin(v_{2})\right)\\ \left(\frac{u}{2}+v_{1}-\sin(v_{2})\right)\\ -\left(\frac{u}{2}+v_{1}-\sin(v_{2})\right)\cos(v_{2})\end{array}\right]~{}~{}~{}\text{ and }~{}~{}~{}\nabla g(u,v)=\left[\begin{array}[]{l}\left(u+v_{1}+\sin(v_{2})\right)\\ \left(u+v_{1}+\sin(v_{2})\right)\\ \left(u+v_{1}+\sin(v_{2})\right)\cos(v_{2})\end{array}\right].$ $$

We first verify that both $f(u,v)$ and $g(u,v)$ are blockwise PL. As $u,v_{1}-\sin(v_{2}),v_{1}+\cos(v_{2})$ spread out in $\mathbb{R}$, then both $\frac{u}{2}+v_{1}-\sin(v_{2})=0$ and $u+v_{1}+\sin(v_{2})=0$ always have a solution. Therefore, we have $\min_{u}f(u,v)=\min_{v}f(u,v)=\min_{u}g(u,v)=\min_{v}g(u,v)=0$. On the other hand,

$$ $\displaystyle\|\nabla_{u}f(u,v)\|^{2}$ $\displaystyle=\frac{1}{4}\left(\frac{u}{2}+v_{1}-\sin(v_{2})\right)^{2}\geq\frac{1}{2}\times\left(f(u,v)-\min_{u}f(u,v)\right)$ $\displaystyle\|\nabla_{v}f(u,v)\|^{2}$ $\displaystyle=\left(\frac{u}{2}+v_{1}-\sin(v_{2})\right)^{2}\times(1+\cos^{2}(v_{2}))\geq 2\times\left(f(u,v)-\min_{v}f(u,v)\right)$ $\displaystyle\|\nabla_{u}g(u,v)\|^{2}$ $\displaystyle=\left(u+v_{1}+\sin(v_{2})\right)^{2}=2\times\left(g(u,v)-\min_{u}g(u,v)\right)$ $\displaystyle\|\nabla_{v}g(u,v)\|^{2}$ $\displaystyle=\left(u+v_{1}+\sin(v_{2})\right)^{2}\times(1+\cos^{2}(v_{2}))\geq 2\times\left(f(u,v)-\min_{v}f(u,v)\right)$ $$

which suggests both $f(u,v)$ and $g(u,v)$ are blockwise PL.

For any $\gamma$, since
$g^{*}(u)=0$, $\nabla_{u}\mathsf{L}_{\gamma}(u,v)=0$ gives the equation

$$ $\displaystyle\left(\gamma+\frac{1}{4}\right)u+\left(\gamma+\frac{1}{2}\right)v_{1}+\left(\gamma-\frac{1}{2}\right)\sin(v_{2})=0.$ (26) $$

It can be verified that both $u=v_{1}=v_{2}=0$ and $u=-\frac{4+8\gamma}{4\gamma+1},v_{1}=2,v_{2}=0$ are solutions to ([26](#A3.E26)), which yields two different objective values $\mathsf{L}_{\gamma}(u,v)=0$ and $\mathsf{L}_{\gamma}(u,v)=\frac{8\gamma^{2}+2}{(4\gamma+1)^{2}}$. This means there exists saddle points for $\mathsf{L}_{\gamma}(u,v)$ so that $\mathsf{L}_{\gamma}(u,v)$ is not blockwise PL over $u$. Same phenomenon happens for $v$, suggesting that $\mathsf{L}_{\gamma}(u,v)$ is also not blockwise PL over $u$.

###### Lemma 13 .

###### Example 2 .

#### C.3 Nonconvexity of 𝖥 ​ ( u ) 𝖥 𝑢 \mathsf{F}(u)

###### Example 3 .

With $u\in\mathbb{R}$ and $v\in\mathbb{R}$, consider the following upper and lower-level objectives

$$ $\displaystyle f(u,v)=v~{}~{}\text{ and }~{}~{}g(u,v)=\mathrm{e}^{v}+(u+v)^{4}.$ $$

It is obvious that both $f(u,v)$ and $g(u,v)$ are jointly convex with regard to $(u,v)$ and minimizing the lower-level problem parameterized by $u$ yields the unique solution $\mathcal{S}(u)=\{v\mid 4(u+v)^{3}+\mathrm{e}^{v}=0\}$. However, the overall bilevel loss function $\mathsf{F}(u)=f(u,\mathcal{S}(u))=\mathcal{S}(u)$ is not convex over $u$. A visualization of $\mathsf{F}(u)=\mathcal{S}(u)$ is shown in Figure [4](#A3.F4).

Proof:
Let us calculate the gradient and Hessian of $f(u,v)$ and $g(u,v)$ in Example [3](#Thmexample3) by

$$ $\displaystyle\nabla f(u,v)=\left[\begin{array}[]{l}0\\ 1\end{array}\right]~{}~{}~{}~{}~{}~{}~{}\text{ and }~{}~{}~{}~{}~{}~{}~{}\nabla g(u,v)=\left[\begin{array}[]{l}4(u+v)^{3}\\ 4(u+v)^{3}+\mathrm{e}^{v}\end{array}\right]$ $\displaystyle\nabla^{2}f(u,v)=\left[\begin{array}[]{l}0~{}~{}~{}~{}0\\ 0~{}~{}~{}~{}0\end{array}\right]~{}~{}\text{ and }~{}~{}\nabla^{2}g(u,v)=\left[\begin{array}[]{l}12(u+v)^{2}~{}~{}~{}~{}12(u+v)^{2}\\ 12(u+v)^{2}~{}~{}~{}~{}12(u+v)^{2}+\mathrm{e}^{v}\end{array}\right].$ $$

From the Hessian, both $f(u,v)$ and $g(u,v)$ are convex because the determinants of all $k\times k$ submatrix are nonnegative. Besides, the lower-level optimal solution set is obtained by letting $4(u+v)^{3}+\mathrm{e}^{v}=0$. Since when fixing $u$, $4(u+v)^{3}+\mathrm{e}^{v}$ is strictly increasing with respect to $v$ and spread out over $(-\infty,\infty)$, the solution set $\mathcal{S}(u)=\{v\mid 4(u+v)^{3}+\mathrm{e}^{v}=0\}$ is unique and nonempty.

We will then prove $\mathsf{F}(u)$ is not convex by contradiction.
We choose $u=\sqrt[3]{-\frac{1}{4}}$ and $\tilde{u}=\sqrt[3]{-\frac{\mathrm{e}}{4}}-1$, then $\mathcal{S}(u)=\{v\mid 4(u+v)^{3}+\mathrm{e}^{v}=0\}=0$ and $\mathcal{S}(\tilde{u})=\{v\mid 4(\tilde{u}+v)^{3}+\mathrm{e}^{v}=0\}=1$, so that

$$ $\mathsf{F}(u)=f(\sqrt[3]{-\frac{1}{4}},0)=0~{}~{}~{}{\rm and}~{}~{}~{}F(\tilde{u})=f(\sqrt[3]{-\frac{\mathrm{e}}{4}}-1,1)=1.$ $$

Assume that $\mathsf{F}(u)$ is convex, and thus the function value $\mathsf{F}(\bar{u})$ at the medium point $\bar{u}=\frac{u+\tilde{u}}{2}=\frac{\sqrt[3]{-\frac{1}{4}}+\sqrt[3]{-\frac{\mathrm{e}}{4}}-1}{2}$ satisfies $\mathsf{F}(\bar{u})\leq\frac{F(u)+F(\tilde{u})}{2}=\frac{1}{2}$, which means there exists $v\leq\frac{1}{2}$ such that $4(\bar{u}+v)^{3}+\mathrm{e}^{v}=0$. However, $4(\bar{u}+v)^{3}+\mathrm{e}^{v}$ is strictly increasing and

$$ $\displaystyle 4(\bar{u}+v)^{3}+\mathrm{e}^{v}\mid_{v=1/2}~{}=4\left(\frac{\sqrt[3]{-\frac{1}{4}}+\sqrt[3]{-\frac{\mathrm{e}}{4}}-1}{2}+\frac{1}{2}\right)^{3}+\mathrm{e}^{\frac{1}{2}}\approx-0.07<0.$ $$

So for any $v\leq\frac{1}{2}$, $4(\bar{u}+v)^{3}+\mathrm{e}^{v}<0$, which yields a contradiction, and thus, $\mathsf{F}(u)$ is not convex.

###### Example 3 .

### Appendix D Proof of Theorem 1

###### Lemma 14 (Gradient bias) .

Under Assumption [1](#Thmassumption1), denoting $d_{u}^{k}=(u^{k+1}-u^{k})/\alpha$, we have

$$ $\displaystyle\|d_{u}^{k}-\nabla_{u}\mathsf{L}_{\gamma}(u^{k},v^{k})\|^{2}\leq\frac{2\gamma^{2}\ell_{g}^{2}}{\mu_{g}}(1-\beta\mu_{g})^{T_{k}}(g(u^{k},w^{0})-g^{*}(u^{k})).$ (27) $$

Proof:
First, according to Lemma [8](#Thmlemma8), $\nabla g^{*}(u)=\nabla_{u}g(u,v)$, $\forall v\in\mathcal{S}(u)$. By the update rule, the bias comes from the inexactness of the value function’s gradient, i.e.

$$ $\displaystyle\|d_{u}^{k}-\nabla_{u}\mathsf{L}_{\gamma}(u^{k},v^{k})\|$ $\displaystyle=\gamma\|\nabla g^{*}(u^{k})-\nabla g(u^{k},w^{k+1})\|\leq\gamma\ell_{g}d(w^{k+1},\mathcal{S}(u^{k})).$ (28) $$

According to Lemma [7](#Thmlemma7) and Lemma [6](#Thmlemma6), we have

$$ $\displaystyle d(w^{k+1},\mathcal{S}(u^{k}))^{2}$ $\displaystyle\leq\frac{2}{\mu_{g}}(1-\beta\mu_{g})^{T_{k}}(g(u^{k},w^{0})-g^{*}(u^{k}))$ $$

from which plugging into ([28](#A4.E28)) yields the conclusion.

###### Lemma 14 (Gradient bias) .

#### D.1 Proof under the joint PL condition

From Lemma [8](#Thmlemma8), we have
the smoothness of $g^{*}(u)$ and thus $\mathsf{L}_{\gamma}(u,v)$. Therefore, by defining $d_{u}^{k}=(u^{k+1}-u^{k})/\alpha$ and by Taylor expansion, we have

$$ $\displaystyle\mathsf{L}_{\gamma}(u^{k+1},v^{k+1})$ $\displaystyle\leq\mathsf{L}_{\gamma}(u^{k},v^{k})+\langle\nabla_{u}\mathsf{L}_{\gamma}(u^{k},v^{k}),u^{k+1}-u^{k}\rangle+\langle\nabla_{v}\mathsf{L}_{\gamma}(u^{k},v^{k}),v^{k+1}-v^{k}\rangle$ $\displaystyle~{}~{}~{}~{}+\frac{\ell_{f}+\gamma(\ell_{g}+L_{g})}{2}\|u^{k+1}-u^{k}\|^{2}+\frac{\ell_{f}+\gamma(\ell_{g}+L_{g})}{2}\|v^{k+1}-v^{k}\|^{2}$ $\displaystyle\leq\mathsf{L}_{\gamma}(u^{k},v^{k})-\alpha\langle\nabla_{u}\mathsf{L}_{\gamma}(u^{k},v^{k}),d_{u}^{k}\rangle-\left(\alpha-\frac{(\ell_{f}+\gamma(\ell_{g}+L_{g}))\alpha^{2}}{2}\right)\|\nabla_{v}\mathsf{L}_{\gamma}(u^{k},v^{k})\|^{2}$ $\displaystyle~{}~{}~{}~{}+\frac{(\ell_{f}+\gamma(\ell_{g}+L_{g}))\alpha^{2}}{2}\|d_{u}^{k}\|^{2}$ $\displaystyle=\mathsf{L}_{\gamma}(u^{k},v^{k})-\frac{\alpha}{2}\|\nabla_{u}\mathsf{L}_{\gamma}(u^{k},v^{k})\|^{2}-\left(\frac{\alpha}{2}-\frac{(\ell_{f}+\gamma(\ell_{g}+L_{g}))\alpha^{2}}{2}\right)\|d_{u}^{k}\|^{2}$ $\displaystyle~{}~{}~{}~{}-\left(\alpha-\frac{(\ell_{f}+\gamma(\ell_{g}+L_{g}))\alpha^{2}}{2}\right)\|\nabla_{v}\mathsf{L}_{\gamma}(u^{k},v^{k})\|^{2}+\frac{\alpha}{2}\|d_{u}^{k}-\nabla_{u}\mathsf{L}_{\gamma}(u^{k},v^{k}\|^{2}$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}\mathsf{L}_{\gamma}(u^{k},v^{k})-\frac{\alpha}{2}\|\nabla_{u}\mathsf{L}_{\gamma}(u^{k},v^{k})\|^{2}-\frac{\alpha}{2}\|\nabla_{v}\mathsf{L}_{\gamma}(u^{k},v^{k})\|^{2}$ $\displaystyle~{}~{}~{}~{}~{}+\frac{\alpha\gamma^{2}\ell_{g}^{2}}{\mu_{g}}(1-\beta\mu_{g})^{T_{k}}(g(u^{k},w^{0})-g^{*}(u^{k}))$ $\displaystyle=\mathsf{L}_{\gamma}(u^{k},v^{k})-\frac{\alpha}{2}\|\nabla\mathsf{L}_{\gamma}(u^{k},v^{k})\|^{2}+\frac{\alpha\gamma^{2}\ell_{g}^{2}}{\mu_{g}}(1-\beta\mu_{g})^{T_{k}}(g(u^{k},w^{0})-g^{*}(u^{k}))$ $\displaystyle\stackrel{{\scriptstyle(b)}}{{\leq}}\mathsf{L}_{\gamma}(u^{k},v^{k})-\alpha\mu_{l}(\mathsf{L}_{\gamma}(u^{k},v^{k})-\min_{u,v}\mathsf{L}_{\gamma}(u,v))+\frac{\alpha\gamma^{2}\ell_{g}^{2}}{\mu_{g}}(1-\beta\mu_{g})^{T_{k}}(g(u^{k},w^{0})-g^{*}(u^{k}))$ $$

where (a) is because $\alpha\leq\frac{1}{\ell_{f}+\gamma(\ell_{g}+L_{g})}$ and Lemma [14](#Thmlemma14), and (b) is because of the joint PL condition. Subtracting both sides by $\min_{u,v}\mathsf{L}_{\gamma}(u,v)$ yields

$$ $\displaystyle\mathsf{L}_{\gamma}(u^{k+1},v^{k+1})-\min_{u,v}\mathsf{L}_{\gamma}(u,v)$ $\displaystyle\leq(1-\alpha\mu_{l})(\mathsf{L}_{\gamma}(u^{k},v^{k})-\min_{u,v}\mathsf{L}_{\gamma}(u,v))$ $\displaystyle~{}~{}~{}~{}+\frac{\alpha\gamma^{2}\ell_{g}^{2}}{\mu_{g}}(1-\beta\mu_{g})^{T_{k}}(g(u^{k},w^{0})-g^{*}(u^{k})).$ $$

Letting $T_{k}=\log\left(\frac{\gamma^{2}(g(u^{k},w^{0})-g^{*}(u^{k}))}{\epsilon}\right)$ and telescoping the above inequality yields

$$ $\displaystyle\mathsf{L}_{\gamma}(u^{K},v^{K})-\min_{u,v}\mathsf{L}_{\gamma}(u,v)$ $\displaystyle\leq(1-\alpha\mu_{l})^{K}(\mathsf{L}_{\gamma}(u^{0},v^{0})-\min_{u,v}\mathsf{L}_{\gamma}(u,v))+\sum_{k=1}^{K-1}(1-\alpha\mu_{l})^{\kappa}{\cal O}(\alpha\epsilon)$ $\displaystyle\leq(1-\alpha\mu_{l})^{K}(\mathsf{L}_{\gamma}(u^{0},v^{0})-\min_{u,v}\mathsf{L}_{\gamma}(u,v))+{\cal O}(\epsilon).$ $$

On the other hand, according to [, Theorem 1] and denoting $\epsilon_{\gamma}:=g(u^{K},v^{K})-g^{*}(u^{K})$, we have for any $\epsilon_{1}$ and $\gamma\geq 2\gamma^{*}=\frac{\ell_{f,0}^{2}\mu_{g}}{4}\epsilon_{1}^{-1}$, the following

$$ $\displaystyle\epsilon_{\gamma}\leq\frac{\epsilon+\epsilon_{1}}{\gamma-\gamma^{*}}\leq\frac{8\epsilon_{1}(\epsilon+\epsilon_{1})}{\ell_{f,0}^{2}\mu_{g}}~{}~{}\text{ and }~{}~{}f(u^{K},v^{K})-f(u,v)\leq(1-\alpha\mu_{l})^{K}(\mathsf{L}_{\gamma}(u^{0},v^{0})-\min_{u,v}\mathsf{L}_{\gamma}(u,v))+{\cal O}(\epsilon)$ $$

holds for any $(u,v)$ satisfying $g(u,v)-g^{*}(u)\leq\epsilon_{\gamma}$. The choice of $\epsilon_{1}={\cal O}(\sqrt{\epsilon})$ gives $\epsilon_{\gamma}={\cal O}(\epsilon)$ and $\gamma={\cal O}(\epsilon^{-0.5})$, which completes the proof.

#### D.2 Proof under the blockwise PL condition

Define $\mathsf{L}_{\gamma}^{*}(u)=\min_{v}\mathsf{L}_{\gamma}(u,v)$ and $\mathcal{S}_{\gamma}(u)=\operatorname*{arg\,min}_{v}\mathsf{L}_{\gamma}(u,v)$.
According to Lemma [8](#Thmlemma8), $\nabla g^{*}(u)=\nabla_{u}g(u,v),\forall v\in\mathcal{S}(u)$. Since $\mathsf{L}_{\gamma}(u,v)$ is $(\ell_{f}+\gamma\ell_{g})$ smooth over $v$, then
according to Lemma [8](#Thmlemma8), we also have $\nabla\mathsf{L}_{\gamma}^{*}(u)=\nabla_{u}\mathsf{L}_{\gamma}(u,v),\forall v\in\mathcal{S}_{\gamma}(u)$. Moreover, $\mathsf{L}_{\gamma}^{*}(u)$ is $L_{\gamma}$ smooth with $L_{\gamma}:=(\ell_{f}+\gamma\ell_{g})(1+(\ell_{f}+\gamma\ell_{g})/2\mu_{v})+L_{g}$. Also $\mathsf{L}_{\gamma}^{*}(u)$ is PL with $\mu_{u}$ because $\mathsf{L}_{\gamma}(u,v)$ is blockwise PL over $u$.

If we denote $d_{u}^{k}=(u^{k+1}-u^{k})/\alpha$, then by the update rule, the bias comes from the inexactness of the value function’s gradient, i.e.

$$ $\displaystyle\|d_{u}^{k}-\nabla\mathsf{L}_{\gamma}^{*}(u^{k})\|$ $\displaystyle=\|\nabla_{u}f(u^{k},v^{k+1})-\nabla_{u}f(u^{k},v_{\gamma})\|+\gamma\|\nabla g(u^{k},v^{k+1})-\nabla g(u^{k},v_{\gamma})\|$ $\displaystyle~{}~{}~{}~{}+\gamma\|\nabla g(u^{k},w^{k+1})-\nabla g(u^{k},v)\|$ $\displaystyle\leq(\ell_{f}+\gamma\ell_{g})d(v^{k+1},\mathcal{S}_{\gamma}(u^{k}))+\gamma\ell_{g}d(w^{k+1},\mathcal{S}(u^{k}))$ $$

where $v_{\gamma}\in\mathcal{S}_{\gamma}(u^{k}),v\in\mathcal{S}(u^{k})$.

According to Lemma [6](#Thmlemma6) and Lemma [7](#Thmlemma7), we have

$$ $\displaystyle d(w^{k+1},\mathcal{S}(u^{k}))^{2}$ $\displaystyle\leq\frac{2}{\mu_{g}}(1-\beta\mu_{g})^{T_{k}}(g(u^{k},w^{0})-g^{*}(u^{k}))$ $\displaystyle d(v^{k+1},\mathcal{S}_{\gamma}(u^{k}))^{2}$ $\displaystyle\leq\frac{2}{\mu_{v}}(1-\tilde{\beta}\mu_{v})^{T_{k}}(\mathsf{L}_{\gamma}(u^{k},w^{0})-\mathsf{L}_{\gamma}^{*}(u^{k})).$ $$

As a result, the bias of the gradient estimator can be bounded by

$$ $\displaystyle\|d_{u}^{k}-\nabla\mathsf{L}_{\gamma}^{*}(u^{k})\|^{2}$ $\displaystyle\leq\frac{4(\ell_{f}+\gamma\ell_{g})^{2}}{\mu_{v}}(1-\tilde{\beta}\mu_{v})^{T_{k}}(\mathsf{L}_{\gamma}(u^{k},w^{0})-\mathsf{L}_{\gamma}^{*}(u^{k}))$ $\displaystyle~{}~{}~{}~{}~{}+\frac{4\gamma^{2}\ell_{g}^{2}}{\mu_{g}}(1-\beta\mu_{g})^{T_{k}}(g(u^{k},w^{0})-g^{*}(u^{k})).$ (29) $$

Therefore, by Taylor expansion and the smoothness of $\mathsf{L}_{\gamma}^{*}(u)$, we have

$$ $\displaystyle\mathsf{L}_{\gamma}^{*}(u^{k+1})$ $\displaystyle\leq\mathsf{L}_{\gamma}^{*}(u^{k})+\langle\nabla\mathsf{L}_{\gamma}^{*}(u^{k}),u^{k+1}-u^{k}\rangle+\frac{L_{\gamma}}{2}\|u^{k+1}-u^{k}\|^{2}$ $\displaystyle\leq\mathsf{L}_{\gamma}^{*}(u^{k})-\alpha\langle\nabla\mathsf{L}_{\gamma}^{*}(u^{k}),d_{u}^{k}\rangle+\frac{L_{\gamma}\alpha^{2}}{2}\|d_{u}^{k}\|^{2}$ $\displaystyle=\mathsf{L}_{\gamma}^{*}(u^{k})-\frac{\alpha}{2}\|\nabla\mathsf{L}_{\gamma}^{*}(u^{k})\|^{2}-\frac{\alpha}{2}\|d_{u}^{k}\|^{2}+\frac{\alpha}{2}\|\nabla\mathsf{L}_{\gamma}^{*}(u^{k})-d_{u}^{k}\|^{2}+\frac{L_{\gamma}\alpha^{2}}{2}\|d_{u}^{k}\|^{2}$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}\mathsf{L}_{\gamma}^{*}(u^{k})-\frac{\alpha}{2}\|\nabla\mathsf{L}_{\gamma}^{*}(u^{k})\|^{2}+\frac{2\alpha(\ell_{f}+\gamma\ell_{g})^{2}}{\mu_{v}}(1-\tilde{\beta}\mu_{v})^{T_{k}}(\mathsf{L}_{\gamma}(u^{k},w^{0})-\mathsf{L}_{\gamma}^{*}(u^{k}))$ $\displaystyle~{}~{}~{}~{}~{}+\frac{2\alpha\gamma^{2}\ell_{g}^{2}}{\mu_{g}}(1-\beta\mu_{g})^{T_{k}}(g(u^{k},w^{0})-g^{*}(u^{k}))$ $\displaystyle\stackrel{{\scriptstyle(b)}}{{\leq}}\mathsf{L}_{\gamma}^{*}(u^{k})-\alpha\mu_{u}(\mathsf{L}_{\gamma}^{*}(u)-\min_{u,v}\mathsf{L}_{\gamma}(u,v))+\frac{2\alpha(\ell_{f}+\gamma\ell_{g})^{2}}{\mu_{v}}(1-\tilde{\beta}\mu_{v})^{T_{k}}(\mathsf{L}_{\gamma}(u^{k},w^{0})-\mathsf{L}_{\gamma}^{*}(u^{k}))$ $\displaystyle~{}~{}~{}~{}~{}+\frac{2\alpha\gamma^{2}\ell_{g}^{2}}{\mu_{g}}(1-\beta\mu_{g})^{T_{k}}(g(u^{k},w^{0})-g^{*}(u^{k}))$ $$

where (a) is because $\alpha\leq\frac{1}{L_{\gamma}}$ and Lemma [14](#Thmlemma14), and (b) is because of the PL condition of $\mathsf{L}_{\gamma}^{*}(u)$. Subtracting both sides by $\min_{u,v}\mathsf{L}_{\gamma}(u,v)$ yields

$$ $\displaystyle\mathsf{L}_{\gamma}^{*}(u^{k+1})-\min_{u,v}\mathsf{L}_{\gamma}(u,v)$ $\displaystyle\leq(1-\alpha\mu_{u})(\mathsf{L}_{\gamma}^{*}(u^{k})-\min_{u,v}\mathsf{L}_{\gamma}(u,v))$ $\displaystyle~{}~{}~{}~{}+\frac{2\alpha(\ell_{f}+\gamma\ell_{g})^{2}}{\mu_{v}}(1-\tilde{\beta}\mu_{v})^{T_{k}}(\mathsf{L}_{\gamma}(u^{k},w^{0})-\mathsf{L}_{\gamma}^{*}(u^{k}))$ $\displaystyle~{}~{}~{}~{}~{}+\frac{2\alpha\gamma^{2}\ell_{g}^{2}}{\mu_{g}}(1-\beta\mu_{g})^{T_{k}}(g(u^{k},w^{0})-g^{*}(u^{k})).$ $$

When $\mu_{v}={\cal O}(\gamma)$, the last term is dominating over the second term.
Letting $T_{k}=\log\left(\frac{\gamma^{2}(g(u^{k},w^{0})-g^{*}(u^{k}))}{\epsilon}\right)$ and telescoping the above inequality yields

$$ $\displaystyle\mathsf{L}_{\gamma}^{*}(u^{k+1})-\min_{u,v}\mathsf{L}_{\gamma}(u,v)$ $\displaystyle\leq(1-\alpha\mu_{u})^{k}(\mathsf{L}_{\gamma}^{*}(u^{k})-\min_{u,v}\mathsf{L}_{\gamma}(u,v))+{\cal O}(\epsilon)$ $$

Combining with the results of

$$ $\displaystyle\mathsf{L}_{\gamma}(u^{k},v^{k+1})-\mathsf{L}_{\gamma}^{*}(u^{k})\leq{\cal O}(\epsilon)\text{ and }\mathsf{L}_{\gamma}(u^{k+1},v^{k+2})-\mathsf{L}_{\gamma}^{*}(u^{k+1})\leq{\cal O}(\epsilon)$ $$

we know that

$$ $\displaystyle\mathsf{L}_{\gamma}(u^{K},v^{K+1})-\min_{u,v}\mathsf{L}_{\gamma}(u,v)$ $\displaystyle\leq(1-\alpha\mu_{u})^{K}(\mathsf{L}_{\gamma}(u^{0},v^{1})-\min_{u,v}\mathsf{L}_{\gamma}(u,v))+{\cal O}(\epsilon)$ $$

On the other hand, according to [, Theorem 1] and denoting $\epsilon_{\gamma}:=g(u^{K},v^{K})-g^{*}(u^{K})$, we have for any $\epsilon_{1}$ and $\gamma\geq 2\gamma^{*}=\frac{\ell_{f,0}^{2}\mu_{g}}{4}\epsilon_{1}^{-1}$, the following

$$ $\displaystyle\epsilon_{\gamma}\leq\frac{\epsilon+\epsilon_{1}}{\gamma-\gamma^{*}}\leq\frac{8\epsilon_{1}(\epsilon+\epsilon_{1})}{\ell_{f,0}^{2}\mu_{g}}~{}~{}\text{ and }~{}~{}f(u^{K},v^{K})-f(u,v)\leq(1-\alpha\mu_{l})^{K}(\mathsf{L}_{\gamma}(u^{0},v^{0})-\min_{u,v}\mathsf{L}_{\gamma}(u,v))+{\cal O}(\epsilon)$ $$

holds for any $(u,v)$ satisfying $g(u,v)-g^{*}(u)\leq\epsilon_{\gamma}$. The choice of $\epsilon_{1}={\cal O}(\sqrt{\epsilon})$ gives $\epsilon_{\gamma}={\cal O}(\epsilon)$ and $\gamma={\cal O}(\epsilon^{-0.5})$, which completes the proof.

### Appendix E Proof of Observations 1 - 2

#### E.1 Proof of Observation 1

Proof:
The blockwise PL condition of $\gamma(g(u,v)-g^{*}(u))$ over $v$ is obvious under Assumption [1](#Thmassumption1) because $g^{*}(u)$ is independent of $v$. On the other hand, with Assumption [1](#Thmassumption1), the gradient of $\gamma(g(u,v)-g^{*}(u))$ can be lower bounded by

$$ $\displaystyle\|\nabla\gamma(g(u,v)-g^{*}(u))\|^{2}$ $\displaystyle=\gamma^{2}\|\nabla_{u}g(u,v)-\nabla_{u}g^{*}(u)\|^{2}+\gamma^{2}\|\nabla_{v}g(u,v)\|^{2}$ $\displaystyle\geq\gamma^{2}\|\nabla_{v}g(u,v)\|^{2}$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{\geq}}\gamma\mu_{g}(\gamma(g(u,v)-g^{*}(u)))$ $\displaystyle\stackrel{{\scriptstyle(b)}}{{=}}\gamma\mu_{g}(\gamma(g(u,v)-g^{*}(u))-\min_{u,v}\gamma(g(u,v)-g^{*}(u)))$ $$

where (a) is because $g(u,v)$ is $\mu_{g}$ PL in $v$ and (b) is due to $\min_{u,v}\gamma(g(u,v)-g^{*}(u))=0$.

#### E.2 Proof of Observation 2

The result that $h_{1}(Az)$ and $h_{2}(Bz)$ satisfy the PL condition is well-known; see e.g., [, Appendix B].
We will show $h_{1}(Az)+h_{2}(Bz)$ satisfies the PL inequality. For arbitrary $z_{1},z_{2}$, we denote $u_{1}=Az_{1},u_{2}=Az_{2},v_{1}=Bz_{1},v_{2}=Bz_{2}$. By the strong convexity of $h_{1}$ and $h_{2}$, we have

$$ $\displaystyle h_{1}(Az_{2})$ $\displaystyle\geq h_{1}(Az_{1})+\nabla h_{1}(u_{1})^{\top}(Az_{2}-Az_{1})+\frac{\mu_{1}}{2}\|Az_{1}-Az_{2}\|^{2}$ $\displaystyle h_{2}(Bz_{2})$ $\displaystyle\geq h_{2}(Bz_{1})+\nabla h_{2}(v_{1})^{\top}(Bz_{2}-Bz_{1})+\frac{\mu_{2}}{2}\|Bz_{1}-Bz_{2}\|^{2}.$ $$

By the fact of $\nabla h_{1}(Az_{i})=A^{\top}\nabla h_{1}(u_{i})$ and $\nabla h_{2}(Bz_{i})=B^{\top}\nabla h_{2}(v_{i})$ for $i=1,2$, we get

$$ $\displaystyle h_{1}(Az_{2})$ $\displaystyle\geq h_{1}(Az_{1})+\nabla h_{1}(Az_{1})^{\top}(z_{2}-z_{1})+\frac{\mu_{1}}{2}\|Az_{1}-Az_{2}\|^{2}$ (30a) $\displaystyle h_{2}(Bz_{2})$ $\displaystyle\geq h_{2}(Bz_{1})+\nabla h_{2}(Bz_{1})^{\top}(z_{2}-z_{1})+\frac{\mu_{2}}{2}\|Bz_{1}-Bz_{2}\|^{2}$ (30b) $$

Letting $z_{2}=\operatorname{Proj}_{\mathcal{Z}^{*}}(z_{1})$ be the projection of $z_{1}$ onto the optimal solution set $\mathcal{Z}^{*}=\operatorname*{arg\,min}_{z}h_{1}(Az)+h_{2}(Bz)$. Clearly, $z_{2}$ is the solution of

$$ $\displaystyle\min_{z}\frac{1}{2}\|z-z_{1}\|^{2},\text{ s.t. }Az=Az^{*},Bz=Bz^{*}$ (31) $$

where $z^{*}\in\operatorname*{arg\,min}_{z}h_{1}(Az)+h_{2}(Bz)$ is an arbitrary optimal point. Strong duality holds for ([31](#A5.E31)) so that KKT conditions must hold at $z_{2}$, i.e. there exists $\lambda_{1}\in\mathbb{R}^{m},\lambda_{2}\in\mathbb{R}^{n}$ such that

$$ $\displaystyle z_{2}-z_{1}+A^{\top}\lambda_{1}+B^{\top}\lambda_{2}=0.$ (32) $$

Without loss of generality, we can consider only two cases: i) $\operatorname{Ran}(A^{\top})=\operatorname{Ran}(B^{\top})$; and, ii) $\operatorname{Ran}(A^{\top})\perp\operatorname{Ran}(B^{\top})$. Otherwise, we can decompose $\operatorname{Ran}(B^{\top})$ into the direct sum of two orthogonal subspaces: one parallel and the other orthogonal to $\operatorname{Ran}(A^{\top})$.

For Case i), it is clear that $z_{2}-z_{1}\in\operatorname{Ran}(A^{\top})$ and $z_{2}-z_{1}\in\operatorname{Ran}(B^{\top})$ by ([32](#A5.E32)). Therefore, letting $C^{\prime}=\mu_{1}\sigma_{*}^{2}(A)+\mu_{2}\sigma_{*}^{2}(B)$ and adding ([30a](#A5.E30.1)) and ([30b](#A5.E30.2)) up yields

$$ $\displaystyle h_{1}(Az_{2})+h_{2}(Bz_{2})$ $\displaystyle\geq h_{1}(Az_{1})+h_{2}(Bz_{1})+(\nabla h_{1}(Az_{1})+\nabla h_{2}(Bz_{1}))^{\top}(z_{2}-z_{1})+\frac{C^{\prime}}{2}\|z_{1}-z_{2}\|^{2}$ $\displaystyle\geq h_{1}(Az_{1})+h_{2}(Bz_{1})+\min_{z}\left[(\nabla h_{1}(Az_{1})+\nabla h_{2}(Bz_{1}))^{\top}(z-z_{1})+\frac{C^{\prime}}{2}\|z-z_{1}\|^{2}\right]$ $\displaystyle=h_{1}(Az_{1})+h_{2}(Bz_{1})-\frac{1}{C^{\prime}}\|\nabla h_{1}(Az_{1})+\nabla h_{2}(Bz_{1})\|^{2}.$ (33) $$

For Case ii), since $A^{\top}\lambda_{1},B^{\top}\lambda_{2}\in\operatorname{Ran}(A^{\top})\cup\operatorname{Ran}(B^{\top})$ by ([32](#A5.E32)), we have

$$ $z_{1}-z_{2}=A^{\top}\lambda_{1}+B^{\top}\lambda_{2}\in\operatorname{Span}(\operatorname{Ran}(A^{\top})\cup\operatorname{Ran}(B^{\top})).$ $$

By definition, there is a finite number of orthogonal $v_{i}\in\operatorname{Ran}(A^{\top})\cup\operatorname{Ran}(B^{\top})$ and of $a_{i}\in\mathbb{R}$ so that $z_{1}-z_{2}=a_{1}v_{1}+\cdots+a_{k}v_{k}$. As each $v_{i}$ is either in $\operatorname{Ran}(A^{\top})$ or $\operatorname{Ran}(B^{\top})$ and they are orthogonal, we have $\|z_{1}-z_{2}\|^{2}=\sum_{i=1}^{k}\|a_{i}v_{i}\|^{2}$ and we can focus on each of $v_{i}$ independently.

Let $C=\min\{\mu_{1}\sigma_{*}^{2}(A),\mu_{2}\sigma_{*}^{2}(B)\}$, since $v_{i}\in\operatorname{Ran}(A^{\top})\cup\operatorname{Ran}(B^{\top})$, we have that for each $v_{i}$, either $\|Aa_{i}v_{i}\|^{2}\geq\frac{C}{2}\|a_{i}v_{i}\|^{2}$ or $\|Ba_{i}v_{i}\|^{2}\geq\frac{C}{2}\|a_{i}v_{i}\|^{2}$.
Therefore, adding ([30a](#A5.E30.1)) and ([30b](#A5.E30.2)) yields

$$ $\displaystyle h_{1}(Az_{2})+h_{2}(Bz_{2})$ $\displaystyle\geq h_{1}(Az_{1})+h_{2}(Bz_{1})+(\nabla h_{1}(Az_{1})+\nabla h_{2}(Bz_{1}))^{\top}(z_{2}-z_{1})+\frac{C}{2}\|z_{1}-z_{2}\|^{2}$ $\displaystyle\geq h_{1}(Az_{1})+h_{2}(Bz_{1})+\min_{z}\left[(\nabla h_{1}(Az_{1})+\nabla h_{2}(Bz_{1}))^{\top}(z-z_{1})+\frac{C}{2}\|z-z_{1}\|^{2}\right]$ $\displaystyle=h_{1}(Az_{1})+h_{2}(Bz_{1})-\frac{1}{C}\|\nabla h_{1}(Az_{1})+\nabla h_{2}(Bz_{1})\|^{2}.$ (34) $$

According to ([33](#A5.Ex3)) and ([34](#A5.Ex7)), as $h_{1}(Az_{2})+h_{2}(Bz_{2})=\min_{z}h_{1}(Az)+h_{2}(Bz)$ and $C\leq C^{\prime}$, this completes the proof.

### Appendix F Proof for Representation Learning

In this section, we provide the proof of lemmas and theorems omitted for representation learning.

#### F.1 Sufficient conditions for Assumption 2

Assumption [2](#Thmassumption2) ensures that the bilevel problem has at least one full-rank solution $W_{1}^{*}$. Note that the full rank solution $W_{1}^{*}$ always exists for the single training or validation problem because the objective value is solely dependent on the product $W_{1}W_{2}$. For any solution pair $({W}_{1},{W}_{2})$, it is possible to transform ${W}_{1}$ into a full rank matrix ${W}_{1}^{*}$ by altering its kernel space components and simultaneously, one can identify a corresponding matrix $W_{2}^{*}$ that mirrors $W_{2}$ within the range space of $W_{1}$ and is nullified within the kernel space of $W_{1}$, such that ${W}_{1}{W}_{2}={W}_{1}^{*}{W}_{2}^{*}$. In this sense, Assumption [2](#Thmassumption2) quantifies the singularity of solving two non-singular problems together. To get more sense of Assumption [2](#Thmassumption2), we present some *sufficient conditions* for it.

###### Lemma 15 .

Assumption [2](#Thmassumption2) holds if one of the following conditions holds.

- (a)
$\exists(W_{1}^{*},W_{2}^{*})\in\operatorname*{arg\,min}_{W_{1},W_{2}}\mathsf{L}_{\rm val}(W_{1},W_{2})$ s.t. $(W_{1}^{*},W_{2}^{*})\in\operatorname*{arg\,min}_{W_{1},W_{2}}\mathsf{L}_{\rm trn}(W_{1},W_{2})$.
- (b)
The concatenated data matrix $[X_{\rm val};X_{\rm trn}]$ is of full row rank.

Proof: The condition in (a) indicates that $(W_{1}^{*},W_{2}^{*})$ is a minimizer of the validation loss. Therefore, for any $W_{1},W_{2}$, we have $\mathsf{L}_{\rm val}(W_{1}^{*},W_{2}^{*})\leq\mathsf{L}_{\rm val}(W_{1},W_{2})$. Furthermore, we are given any $\epsilon_{1},\epsilon_{2}$. Then for any $W_{2}$ in the $\epsilon_{2}$ lower-level optimal set, $\mathsf{L}_{\rm val}(W_{1}^{*},W_{2}^{*})\leq\mathsf{L}_{\rm val}(W_{1},W_{2})\leq\mathsf{L}_{\rm val}(W_{1},W_{2})+\epsilon_{1}$ holds. Together with the $\epsilon_{2}$ feasibility of $(W_{1}^{*},W_{2}^{*})$ to the bilevel problem, it means $(W_{1}^{*},W_{2}^{*})$ is an $(\epsilon_{1},\epsilon_{2})$ solution to the bilevel problem. Moreover, since $(W_{1}^{*},W_{2}^{*})$ is also a minimizer of the training loss, we have

$$ $\displaystyle\mathsf{L}_{\rm trn}^{*}(W_{1}^{*})=\mathsf{L}_{\rm trn}(W_{1}^{*},W_{2}^{*})=\min_{W_{1},W_{2}}\mathsf{L}_{\rm trn}(W_{1},W_{2})\overset{\rm Lemma~{}\ref{lm-rank}}{=\joinrel=}0\leq\epsilon_{2}$ $$

which verifies Assumption [2](#Thmassumption2).

For the condition in (b), it implies there exists

$$ $(W_{1}^{*},W_{2}^{*})\in\operatorname*{arg\,min}_{W_{1},W_{2}}\left(\mathsf{L}_{\rm val}(W_{1},W_{2})+\mathsf{L}_{\rm trn}(W_{1},W_{2})\right)$ $$

such that $\mathsf{L}_{\rm val}(W_{1}^{*},W_{2}^{*})+\mathsf{L}_{\rm trn}(W_{1}^{*},W_{2}^{*})=0$. Due to the nonnegativeness of both validation and training loss, it must hold that $\mathsf{L}_{\rm val}(W_{1}^{*},W_{2}^{*})=\mathsf{L}_{\rm trn}(W_{1}^{*},W_{2}^{*})=0$. This suggests $(W_{1}^{*},W_{2}^{*})$ is the joint minimizer of both training and validation loss, which satisfies condition (b), and thus, Assumption [2](#Thmassumption2) is verified.

###### Remark 2 .

Condition (a) says that the training and validation problem share at least a global solution, situating us within the interpolating regime between training and validation. This means that minimizing the combined objective leads to a solution that simultaneously minimizes both problems []. In this case, the model trained from representation learning matches the performance attained by optimizing both layers of model directly based on validation loss. Condition (b) is a sufficient condition of Condition (a) and Condition (b) itself means there are no redundant information contained in the training and validation set.

###### Lemma 15 .

###### Remark 2 .

#### F.2 Landscape of nested bilevel objective 𝖥 ​ ( W 1 ) 𝖥 subscript 𝑊 1 \mathsf{F}(W_{1})

According to the loss definition and the knowledge in linear algebra, we can derive the explicit form of the bilevel objective in representation learning as shown in the following lemma.

###### Lemma 16 (Analytical form of $\mathsf{F}(W_{1})$ and its nonconvexity) .

The bilevel objective for two layer linear neural network representation learning in ([9](#S5.E9)) can be explicitly written as

$$ $\displaystyle F(W_{1})=\frac{1}{2}\left\|A-(X_{{\rm val}}W_{1})\left(I-(X_{{\rm trn}}W_{1})^{\dagger}(X_{{\rm trn}}W_{1})\right)b\right\|^{2}$ (35) $$

with $A=Y_{{\rm val}}-X_{{\rm val}}W_{1}(X_{{\rm trn}}W_{1})^{\dagger}Y_{{\rm trn}}$ and $b=\left[(X_{{\rm val}}W_{1})\left(I-(X_{{\rm trn}}W_{1})^{\dagger}(X_{{\rm trn}}W_{1})\right)\right]^{\dagger}A$. Moreover, $\mathsf{F}(W_{1})$ is nonconvex even when $X_{\rm trn}=X_{\rm val}=Y_{\rm trn}=Y_{\rm val}=I$.

When $X_{\rm trn}=X_{\rm val}=Y_{\rm trn}=Y_{\rm val}=I$, it is obvious that any invertible matrix $W_{1}$ and its inverse $W_{2}=W_{1}^{-1}$ are the solutions to the bilevel problem ([1](#S1.E1)) as both upper and lower-level objective are the same. In this sense, the above lemma indicates that the landscape of the bilevel objective is not convex even in the trivial case. A possible explanation to this phenomenon is the discontinuity and nonconvexity of the inverse operator $W_{1}^{\dagger}$ at the rank-deficient point.

Proof:
First, according to [, Theorem 6.1], the lower-level solution set of ([9](#S5.E9)) can be represented by

$$ $\displaystyle\mathcal{S}(W_{1})=\left\{(X_{{\rm trn}}W_{1})^{\dagger}Y_{{\rm trn}}+\left(I-(X_{{\rm trn}}W_{1})^{\dagger}(X_{{\rm trn}}W_{1})\right)b,~{}\forall b\right\}=(X_{{\rm trn}}W_{1})^{\dagger}Y_{{\rm trn}}+\operatorname{Ker}(X_{{\rm trn}}W_{1}).$ (36) $$

Then $\forall W_{1}$, we aim to solve the following problem

$$ $\displaystyle\min_{b}\frac{1}{2}\left\|A-(X_{{\rm val}}W_{1})\left(I-(X_{{\rm trn}}W_{1})^{\dagger}(X_{{\rm trn}}W_{1})\right)b\right\|^{2}$ (37) $$

where $A=Y_{{\rm val}}-X_{{\rm val}}W_{1}(X_{{\rm trn}}W_{1})^{\dagger}Y_{{\rm trn}}$ is fixed when $W_{1}$ is given. Applying [, Theorem 6.1] to ([37](#A6.E37)) again, we know ([37](#A6.E37)) achieves minimal when

$$ $\displaystyle b$ $\displaystyle=\left[(X_{{\rm val}}W_{1})\left(I-(X_{{\rm trn}}W_{1})^{\dagger}(X_{{\rm trn}}W_{1})\right)\right]^{\dagger}A.$ $$

When $X_{\rm trn}=X_{\rm val}=Y_{\rm trn}=Y_{\rm val}=I$, we have

$$ $\displaystyle b=\left[W_{1}\left(I-W_{1}^{\dagger}W_{1}\right)\right]^{\dagger}A=0,~{}~{}\text{ and }~{}~{}A=Y_{{\rm val}}-X_{{\rm val}}W_{1}(X_{{\rm trn}}W_{1})^{\dagger}Y_{{\rm trn}}=I-W_{1}W_{1}^{\dagger}$ $$

Therefore, the bilevel objective has the form of $\mathsf{F}(W_{1})=\frac{1}{2}\|A\|^{2}=\frac{1}{2}\|I-W_{1}W_{1}^{\dagger}\|^{2}$. Without loose of generality, we consider $W_{1}\in\mathbb{R}^{2\times 2}$ and the objective value at two point

$$ $\displaystyle F\left(\left(\begin{array}[]{l}0~{}~{}0\\ 0~{}~{}1\end{array}\right)\right)=\frac{1}{2},~{}~{}~{}~{}F\left(\left(\begin{array}[]{l}2~{}~{}~{}~{}0\\ 0~{}-1\end{array}\right)\right)=0$ $$

However, the function value at the median point does not satisfy the convexity condition, i.e.

$$ $\displaystyle F\left(\frac{1}{2}\times\left(\begin{array}[]{l}0~{}~{}0\\ 0~{}~{}1\end{array}\right)+\frac{1}{2}\times\left(\begin{array}[]{l}2~{}~{}~{}~{}0\\ 0~{}-1\end{array}\right)\right)=F\left(\left(\begin{array}[]{l}1~{}~{}0\\ 0~{}~{}0\end{array}\right)\right)=\frac{1}{2}>\frac{1}{2}\times\frac{1}{2}+\frac{1}{2}\times 0=\frac{1}{4}.$ $$

As a result, $\mathsf{F}(W_{1})$ is not convex.

###### Lemma 16 (Analytical form of 𝖥 ​ ( W 1 ) 𝖥 subscript 𝑊 1 \mathsf{F}(W_{1}) and its nonconvexity) .

#### F.3 Preliminaries of penalty reformulation

Ideally, we want to solve the penalized problem ([10](#S5.E10)) instead of the original bilevel representation learning problem ([9](#S5.E9)). However, the following lemma shows that $\mathsf{L}_{{\rm trn}}(W_{1},\cdot)$ is not Lipschitz smooth and PL over $W_{2}$ uniformly for all $W_{1}\in\mathbb{R}^{m\times h}$. Therefore, the equivalence of the penalized problem to the bilevel problem and the differentiability of lower-level value function $\mathsf{L}_{{\rm trn}}^{*}(W_{1})$ cannot be established directly from []. In this section, we provide the complete proof of them.

###### Lemma 17 (Non-uniform smoothness and PL) .

The lower-level function $\mathsf{L}_{{\rm trn}}(W_{1},\cdot)$ is smooth with $\sigma_{\max}^{2}(X_{\rm trn})\sigma_{\max}^{2}(W_{1})$ and PL with $\sigma_{*}^{2}(X_{\rm trn}W_{1})$. Moreover, if $\sigma_{\min}(W_{1})>0$, then $\mathsf{L}_{{\rm trn}}(W_{1},\cdot)$ is PL with $\sigma_{\min}^{2}(X_{\rm trn})\sigma_{\min}^{2}(W_{1})$.

Proof:
First, we know that $h(\tilde{W})=\frac{1}{2}\|Y_{\rm trn}-\tilde{W}\|^{2}$ is $1$-strongly convex and $1$-Lipschitz smooth. Given matrix $W_{1}$, then according to [, Appendix B], $\mathsf{L}_{\textrm{trn}}(W_{1},W_{2})=h(X_{\rm trn}W_{1}W_{2})$ is in the form of strongly convex composite with a linear mapping so that it is $\sigma_{*}^{2}(X_{\rm trn}W_{1})$-PL over $W_{2}$. Also, $\mathsf{L}_{\textrm{trn}}(W_{1},W_{2})=h(X_{\rm trn}W_{1}W_{2})$ is $\sigma_{\max}^{2}(X_{\rm trn}W_{1})$-Lipschitz smooth over $W_{2}$. Therefore, the Lipschitz smoothness constant of $\mathsf{L}_{\rm trn}(W_{1},\cdot)$ can be upper bounded by

$$ $\displaystyle\sigma_{\max}^{2}(X_{\rm trn}W_{1})$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}\sigma_{\max}^{2}(X_{\rm trn})\sigma_{\max}^{2}(W_{1})$ $$

where (a) comes from ([18](#A2.E18)).
If $W_{1}$ and $X_{\rm trn}$ are full row rank, the PL constant is lower bounded by

$$ $\displaystyle\sigma_{*}^{2}(X_{\rm trn}W_{1})=\sigma_{\min}^{2}(X_{\rm trn}W_{1})$ $\displaystyle\stackrel{{\scriptstyle(b)}}{{\geq}}\sigma_{\min}^{2}(X_{\rm trn})\sigma_{\min}^{2}(W_{1})$ $$

where (b) is derived from ([19](#A2.E19)), $X_{\rm trn}\neq 0$ and $W_{1}$ is full row rank which is inferred from $W_{1}$ is a fat matrix with $\sigma_{\min}(W_{1})>0$.

###### Definition 4 ( $\epsilon$ - solution of penalized problem) .

We say $(W_{1}^{*},W_{2}^{*})$ is an $\epsilon$- global solution to penalized problem ([10](#S5.E10)) if for any $(W_{1},W_{2})$, it holds that

$$ $\mathsf{L}_{\gamma}(W_{1}^{*},W_{2}^{*})\leq\mathsf{L}_{\gamma}(W_{1},W_{2})+\epsilon.$ $$

###### Theorem 4 (Relations of global solutions) .

Suppose Assumption [2](#Thmassumption2) holds. Any $\epsilon_{2}$ solution to the penalized problem with $\gamma$ is an $(\epsilon_{2},\epsilon_{\gamma})$ solution to the bilevel problem ([9](#S5.E9)) for some $\epsilon_{\gamma}={\cal O}(\epsilon_{2}^{2})$. Conversely, for any $\epsilon_{1}$, there exists $\gamma^{*}={\cal O}(\epsilon_{1}^{-1})$ such that for any $\gamma>\gamma^{*}$, the global optimal solution of bilevel problem ([9](#S5.E9)) must be an $\epsilon_{1}$ optimal solution of penalized problem with $\gamma$.

The above theorem shows that to ensure the penalized problem ([10](#S5.E10)) is an $(\epsilon,\epsilon)$ approximate solution to the bilevel problem ([9](#S5.E9)), i.e. $\epsilon_{\gamma}={\cal O}(\epsilon)$, one need to choose $\epsilon_{2}={\cal O}(\sqrt{\epsilon})$ and $\gamma={\cal O}(\epsilon^{-0.5})$.

Proof: We will prove the relations through four steps.

###### Lemma 17 (Non-uniform smoothness and PL) .

###### Definition 4 ( ϵ italic-ϵ \epsilon - solution of penalized problem) .

###### Theorem 4 (Relations of global solutions) .

##### Step 1:

First, we prove that under Assumption [2](#Thmassumption2), there exists an $(\epsilon_{1},\epsilon_{2})$ solution to the bilevel representation learning problem ([9](#S5.E9)) with full rank $W_{1}$.

Choose any $(\epsilon_{1},\epsilon_{2})$ solution $(W_{1}^{*},W_{2}^{*})$ to the bilevel representation learning problem ([9](#S5.E9)) that satisfies Assumption [2](#Thmassumption2). If $W_{1}^{*}$ is not full rank, then we can find full rank $W_{1}$ and $W_{2}$ such that $W_{1}W_{2}=W_{1}^{*}W_{2}^{*}$ by the following process. By singular value decomposition, we can decompose $W_{1}^{*}=U\Sigma V^{\top}$ with $\Sigma=\left[\begin{array}[]{ll}\Sigma_{1}&0\\
0&0\end{array}\right]\in\mathbb{R}^{m\times h}$, and orthogonal matrix $U=[U_{1}~{}~{}U_{2}]\in\mathbb{R}^{m\times m}$ and $V=[V_{1}~{}V_{2}]\in\mathbb{R}^{h\times h}$. Also, by assuming $\operatorname{Rank}(A)=r$, we know that $U_{1}\in\mathbb{R}^{m\times r},V_{1}\in\mathbb{R}^{h\times r}$ and $\Sigma_{1}\in\mathbb{R}^{r\times r}$ are full rank submatrix. Therefore, $W_{1}^{*}$ can be decomposed by

$$ $\displaystyle W_{1}^{*}=[U_{1}~{}U_{2}]\left[\begin{array}[]{ll}\Sigma_{1}&0\\ 0&0\end{array}\right]\left[\begin{array}[]{l}V_{1}^{\top}\\ V_{2}^{\top}\end{array}\right]=[U_{1}\Sigma_{1}~{}~{}~{}0]\left[\begin{array}[]{l}V_{1}^{\top}\\ V_{2}^{\top}\end{array}\right]=U_{1}\Sigma_{1}V_{1}^{\top}$ $$

and $V_{2}$ is the orthogonal basis of $\operatorname{Ker}(W_{1}^{*})$. Furthermore, we can decompose $V_{2}=[V_{3}~{}V_{4}]\in\mathbb{R}^{h\times(h-r)}$ with $V_{3}\in\mathbb{R}^{h\times(m-r)}$. In this way, we can construct $W_{1}$ and $W_{2}$ as

$$ $W_{1}=W_{1}^{*}+U_{2}V_{2}^{\top}=U_{1}\Sigma_{1}V_{1}^{\top}+U_{2}V_{3}^{\top}\text{ and }W_{2}=V_{1}V_{1}^{\top}W_{2}^{*}$ $$

where $W_{1}$ is of full row rank $m$. By Lemma [5](#Thmlemma5), $\mathsf{L}_{\rm trn}^{*}(W_{1})=0$. On the other hand, as the training and validation losses only depend on the value of $W_{1}W_{2}$ and $W_{1}W_{2}=W_{1}^{*}W_{2}^{*}$, we have

$$ $\displaystyle\mathsf{L}_{\rm val}(W_{1},W_{2})=\mathsf{L}_{\rm val}(W_{1}^{*},W_{2}^{*}),~{}~{}\mathsf{L}_{\rm trn}(W_{1},W_{2})=\mathsf{L}_{\rm trn}(W_{1}^{*},W_{2}^{*}).$ $$

Since $(W_{1}^{*},W_{2}^{*})$ satisfies Assumption [2](#Thmassumption2) and $\mathsf{L}_{\rm trn}^{*}(W_{1})=0$, we have

$$ $\displaystyle\mathsf{L}_{\rm trn}(W_{1},W_{2})-\mathsf{L}_{\rm trn}^{*}(W_{1})=\mathsf{L}_{\rm trn}(W_{1},W_{2})=\mathsf{L}_{\rm trn}(W_{1}^{*},W_{2}^{*})\leq\epsilon_{2}.$ $$

With the fact that $\mathsf{L}_{\rm val}(W_{1},W_{2})=\mathsf{L}_{\rm val}(W_{1}^{*},W_{2}^{*})$, we know $(W_{1},W_{2})$ is also an $(\epsilon_{1},\epsilon_{2})$ solution of the bilevel representation learning problem and $W_{1}$ is full row rank by the construction.

##### Step 2:

Next, we will prove that any $\epsilon_{2}$ solution to the penalized problem with $\gamma$ is an $(\epsilon_{2},\epsilon_{\gamma})$ solution to the bilevel problem for some $\epsilon_{\gamma}$. For any $\epsilon_{2}$ solution of the penalized problem $(W_{1}^{\gamma},W_{2}^{\gamma})$, we can find $W_{2}\in\arg\min_{W_{2}^{\prime}\in\mathcal{S}(W_{1}^{\gamma})}\|W_{2}^{\prime}-W_{2}^{\gamma}\|$ and

$$ $\sigma_{\max}(W_{2})=\sigma_{\max}(\operatorname{Proj}_{\mathcal{S}(W_{1}^{\gamma})}(W_{2}^{\gamma}))\leq\sigma_{\max}(W_{2}^{\gamma}).$ $$

Then we can derive the Lipschitz continuity of $\mathsf{L}_{\rm val}(W_{1}^{\gamma},\cdot)$ by

$$ $\displaystyle\mathsf{L}_{\rm val}(W_{1}^{\gamma},W_{2}^{\gamma})-\mathsf{L}_{\rm val}(W_{1}^{\gamma},W_{2})$ $\displaystyle=\frac{1}{2}\|Y_{\rm val}-X_{\rm val}W_{1}^{\gamma}W_{2}^{\gamma}\|^{2}-\frac{1}{2}\|Y_{\rm val}-X_{\rm val}W_{1}^{\gamma}W_{2}\|^{2}$ $\displaystyle=\frac{1}{2}\langle 2Y_{\rm val}-X_{\rm val}W_{1}^{\gamma}(W_{2}^{\gamma}+W_{2}),-X_{\rm val}W_{1}^{\gamma}(W_{2}^{\gamma}-W_{2})\rangle$ $\displaystyle\leq L(\gamma)\|W_{2}^{\gamma}-W_{2}^{*}\|$ $$

where $L(\gamma)=(\sigma_{\max}(Y_{\rm val})+\sigma_{\max}(X_{\rm val})\sigma_{\max}(W_{1}^{\gamma})\sigma_{\max}(W_{2}^{\gamma}))\sigma_{\max}(X_{\rm val})\sigma_{\max}(W_{1}^{\gamma})$.
So it follows

$$ $\displaystyle~{}~{}~{}~{}\mathsf{L}_{\rm val}(W_{1}^{\gamma},W_{2}^{\gamma})+{\gamma}(\mathsf{L}_{\rm trn}(W_{1}^{\gamma},W_{2}^{\gamma})-\mathsf{L}_{\rm trn}(W_{1}^{\gamma}))$ $\displaystyle\leq\mathsf{L}_{\rm val}(W_{1}^{\gamma},W_{2})+\epsilon_{2}$ $\displaystyle\leq\mathsf{L}_{\rm val}(W_{1}^{\gamma},W_{2}^{\gamma})+L(\gamma)\|W_{2}^{\gamma}-W_{2}\|+\epsilon_{2}$ $\displaystyle\leq\mathsf{L}_{\rm val}(W_{1}^{\gamma},W_{2}^{\gamma})+\sqrt{\frac{2}{\sigma_{*}(X_{\rm trn}W_{1}^{\gamma})}}L(\gamma)\sqrt{\mathsf{L}_{\rm trn}(W_{1}^{\gamma},W_{2}^{\gamma})-\mathsf{L}_{\rm trn}(W_{1}^{\gamma})}+\epsilon_{2}$ (38) $$

where the first inequality comes from the definition of $\epsilon_{2}$ solution of penalized problem and $\mathsf{L}_{\rm trn}(W_{1}^{\gamma},W_{2})-\mathsf{L}_{\rm trn}(W_{1}^{\gamma})=0$, and the last inequality is derived from the PL condition of $\mathsf{L}_{\rm trn}(W_{1}^{\gamma})$ and Lemma [6](#Thmlemma6). According to ([F.3](#A6.Ex41)), we have either

$$ $\displaystyle\mathsf{L}_{\rm trn}(W_{1}^{\gamma},W_{2}^{\gamma})-\mathsf{L}_{\rm trn}(W_{1}^{\gamma})\leq\frac{2\epsilon_{2}}{\gamma}$ (39) $$

or $\epsilon_{2}<\frac{\gamma}{2}(\mathsf{L}_{\rm trn}(W_{1}^{\gamma},W_{2}^{\gamma})-\mathsf{L}_{\rm trn}(W_{1}^{\gamma}))$ and thus

$$ $\displaystyle\frac{\gamma}{2}(\mathsf{L}_{\rm trn}(W_{1}^{\gamma},W_{2}^{\gamma})-\mathsf{L}_{\rm trn}(W_{1}^{\gamma}))\leq\sqrt{\frac{2}{\sigma_{*}(X_{\rm trn}W_{1}^{\gamma})}}L(\gamma)\sqrt{\mathsf{L}_{\rm trn}(W_{1}^{\gamma},W_{2}^{\gamma})-\mathsf{L}_{\rm trn}(W_{1}^{\gamma})}$ $$

which yields

$$ $\displaystyle\mathsf{L}_{\rm trn}(W_{1}^{\gamma},W_{2}^{\gamma})-\mathsf{L}_{\rm trn}(W_{1}^{\gamma})\leq\frac{8}{\sigma_{*}(X_{\rm trn}W_{1}^{\gamma})}\left(\frac{L(\gamma)}{\gamma}\right)^{2}.$ (40) $$

By choosing $\gamma={\cal O}(\epsilon_{2}^{-1})$ and noting that $\{L(\gamma),\sigma_{*}(X_{\rm trn}W_{1}^{\gamma})\}={\cal O}(1)$, ([39](#A6.E39)) and ([40](#A6.E40)) indicate that

$$ $\displaystyle\epsilon_{\gamma}:=\mathsf{L}_{\rm trn}(W_{1}^{\gamma},W_{2}^{\gamma})-\mathsf{L}_{\rm trn}^{*}(W_{1}^{\gamma})\leq\max\left\{\frac{2\epsilon_{2}}{\gamma},\frac{8}{\sigma_{*}(X_{\rm trn}W_{1}^{\gamma})}\left(\frac{L(\gamma)}{\gamma}\right)^{2}\right\}={\cal O}\left(\frac{\epsilon_{2}}{\gamma}\right)={\cal O}\left(\epsilon_{2}^{2}\right).$ $$

Moreover, for any $(\tilde{W}_{1},\tilde{W}_{2})$ satisfying $\mathsf{L}_{\rm trn}(\tilde{W}_{1},\tilde{W}_{2})-\mathsf{L}_{\rm trn}^{*}(\tilde{W}_{1})\leq\epsilon_{\gamma}$, it holds that

$$ $\displaystyle\mathsf{L}_{\rm val}(W_{1}^{\gamma},W_{2}^{\gamma})+\gamma(\mathsf{L}_{\rm trn}(W_{1}^{\gamma},W_{2}^{\gamma})-\mathsf{L}_{\rm trn}^{*}(W_{1}^{\gamma}))\leq\mathsf{L}_{\rm val}(\tilde{W}_{1},\tilde{W}_{2})+\gamma(\mathsf{L}_{\rm trn}(\tilde{W}_{1},\tilde{W}_{2})-\mathsf{L}_{\rm trn}^{*}(\tilde{W}_{1}))+\epsilon_{2}$ $$

and thus

$$ $\displaystyle\mathsf{L}_{\rm val}(W_{1}^{\gamma},W_{2}^{\gamma})-\mathsf{L}_{\rm val}(\tilde{W}_{1},\tilde{W}_{2})\leq\gamma(\mathsf{L}_{\rm trn}(\tilde{W}_{1},\tilde{W}_{2})-\mathsf{L}_{\rm trn}^{*}(\tilde{W}_{1})-\epsilon_{\gamma})+\epsilon_{2}\leq\epsilon_{2}$ $$

which indicates $(W_{1}^{\gamma},W_{2}^{\gamma})$ is $(\epsilon_{2},\epsilon_{\gamma})$ optimal solution of the bilevel problem. This concludes that any $\epsilon_{2}$ solution to the penalized problem is an $(\epsilon_{2},\epsilon_{\gamma})$ solution to the bilevel problem.

##### Step 3:

To prove the converse part, we first claim that for any $\epsilon_{2}$, there exists an $\epsilon_{2}$ solution to the penalized problem with full rank $W_{1}$, which is built on the previous two steps. Take the $\epsilon_{2}$ solution $(W_{1}^{\gamma},W_{2}^{\gamma})$ of the penalized problem in the second step, it is an $(\epsilon_{2},\epsilon_{\gamma})$ optimal solution of the bilevel problem. From the first step, we know there exists an $(\epsilon_{2},\epsilon_{\gamma})$ optimal solution of the bilevel problem $(W_{1}^{\epsilon},W_{2}^{\epsilon})$ with full rank $W_{1}^{\epsilon}$. Therefore, we have $\mathsf{L}_{\rm val}(W_{1}^{\gamma},W_{2}^{\gamma})=\mathsf{L}_{\rm val}(W_{1}^{\epsilon},W_{2}^{\epsilon})$ and the feasibility

$$ $\displaystyle\mathsf{L}_{\rm trn}(W_{1}^{\epsilon},W_{2}^{\epsilon})-\mathsf{L}_{\rm trn}^{*}(W_{1}^{\epsilon})\leq\epsilon_{\gamma}=\mathsf{L}_{\rm trn}(W_{1}^{\gamma},W_{2}^{\gamma})-\mathsf{L}_{\rm trn}^{*}(W_{1}^{\gamma}).$ $$

Adding these two inequalities together, we have $\mathsf{L}_{\gamma}(W_{1}^{\epsilon},W_{2}^{\epsilon})\leq\mathsf{L}_{\gamma}(W_{1}^{\gamma},W_{2}^{\gamma})$, which suggests that $(W_{1}^{\epsilon},W_{2}^{\epsilon})$ is also an $\epsilon_{2}$ solution to the penalized problem. As $W_{1}^{\epsilon}$ is full rank, we prove the claim.

##### Step 4:

We will then prove the converse part, i.e. for any $\epsilon_{1}$, there exists $\gamma^{*}$ such that any global solution of bilevel problem is an $\epsilon_{1}$ solution to the penalized problem for any $\gamma\geq\gamma^{*}$. Take any global solution of bilevel problem and denote it as $(\tilde{W}_{1},\tilde{W}_{2})$. According to the first step, we know that there exists a global solution to the bilevel problem $(W_{1}^{*},W_{2}^{*})$ with full rank $W_{1}^{*}$. Also there exists an $\epsilon_{1}$ solution to the penalized problem $(W_{1}^{\gamma},W_{2}^{\gamma})$ with full rank $W_{1}^{\gamma}$. Therefore, we can restrict the bilevel and penalized problem on the following constrained sets

$$ $\displaystyle\mathcal{W}_{1}=\{W_{1}:r_{1}\leq\sigma_{\min}(W_{1})\leq\sigma_{\max}(W_{1})\leq R_{1}\},~{}~{}~{}\text{ and }~{}~{}~{}\mathcal{W}_{2}=\{W_{2}:\sigma_{\max}(W_{2})\leq R_{2}\}$ $$

where $r_{1}<\min\{\sigma_{\min}(W_{1}^{*}),\sigma_{\min}(W_{1}^{\gamma})\}$, $R_{i}>\max\{\sigma_{\max}(W_{i}^{*}),\sigma_{\max}(W_{i}^{\gamma})\}$, $i=1,2$. In this way, $\mathcal{W}_{2}$ is a convex closed set while $\mathcal{W}_{1}$ is a closed but nonconvex set, and

$$ $\min_{W_{1}\in\mathcal{W}_{1},W_{2}\in\mathcal{W}_{2}}\mathsf{L}_{\gamma}(W_{1},W_{2})=\min_{W_{1},W_{2}}\mathsf{L}_{\gamma}(W_{1},W_{2})$ $$

because the constrained sets include the minimizer of penalized problem. Also, $\mathsf{L}_{\rm val}(W_{1},\cdot)$ is uniformly Lipschitz continuous on $\mathcal{W}_{1}$ and $\mathcal{W}_{2}$, and $\mathsf{L}_{\rm trn}(W_{1},\cdot)$ is uniformly smooth and PL on $\mathcal{W}_{1}$ and $\mathcal{W}_{2}$. Adapted from [, Theorem 1], there exists $\gamma^{*}={\cal O}(\epsilon_{1}^{-1})$ s.t. for any $\gamma>\gamma^{*}$, we have (^2^22[, Theorem 1] only requires the convexity of $\mathcal{W}_{2}$)

$$ $\displaystyle\mathsf{L}_{\gamma}(W_{1}^{*},W_{2}^{*})$ $\displaystyle\leq\min_{W_{1}\in\mathcal{W}_{1},W_{2}\in\mathcal{W}_{2}}\mathsf{L}_{\gamma}(W_{1},W_{2})+\epsilon_{1}$ $\displaystyle=\min_{W_{1},W_{2}}\mathsf{L}_{\gamma}(W_{1},W_{2})+\epsilon_{1}$ $$

Since $\mathsf{L}_{\gamma}(\tilde{W}_{1},\tilde{W}_{2})=\mathsf{L}_{\rm val}(\tilde{W}_{1},\tilde{W}_{2})=\mathsf{L}_{\rm val}(W_{1}^{*},W_{2}^{*})=\mathsf{L}_{\gamma}(W_{1}^{*},W_{2}^{*})$, we know $(\tilde{W}_{1},\tilde{W}_{2})$ is an $\epsilon_{1}$ optimal point of the penalized problem, which concludes the proof.

The following lemma shows that $\mathsf{L}^{*}(W_{1})$ is not differentiable at the whole space, but it is differentiable for $W_{1}$ with lower and upper bounded singular value.

###### Lemma 18 (Danskin type theorem) .

Assume there exists $r,R\in\mathbb{R}_{>0}$ such that for all $W_{1}\in\mathcal{W}_{1}$, $0<r\leq\sigma_{\min}(W_{1})\leq\sigma_{\max}(W_{1})\leq R$. Then $\mathsf{L}_{{\rm trn}}^{*}(W_{1})$ is differentiable on $\mathcal{W}_{1}$ with the gradient

$$ $\displaystyle\nabla\mathsf{L}_{{\rm trn}}^{*}(W_{1})=\nabla_{W_{1}}\mathsf{L}_{{\rm trn}}(W_{1},W_{2}^{*}),\qquad\forall W_{2}^{*}\in\mathcal{S}(W_{1})\text{ with finite norm}.$ $$

Moreover, $\nabla\mathsf{L}_{{\rm trn}}^{*}(W_{1})=0$ on $\mathcal{W}_{1}$.

Proof:
First, $\forall W_{1}\in\mathcal{W}_{1}$, without loss of generality, we can set $W_{2}^{*}=(X_{{\rm trn}}W_{1})^{\dagger}Y_{{\rm trn}}\in\mathcal{S}(W_{1})$ with finite $\ell_{2}$ norm

$$ $\displaystyle\sigma_{\max}(W_{2}^{*})=\sigma_{\max}((X_{{\rm trn}}W_{1})^{\dagger}Y_{{\rm trn}})$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}\sigma_{\max}((X_{{\rm trn}}W_{1})^{\dagger})\sigma_{\max}(Y_{{\rm trn}})$ $\displaystyle=\frac{\sigma_{\max}(Y_{{\rm trn}})}{\sigma_{\min}(X_{{\rm trn}}W_{1})}\leq\frac{\sigma_{\max}(Y_{{\rm trn}})}{\sigma_{\min}(X_{{\rm trn}})\sigma_{\min}(W_{1})}\leq\frac{\sigma_{\max}(Y_{{\rm trn}})}{\sigma_{\min}(X_{{\rm trn}})r}$ $$

where (a) comes from $\sigma_{\max}(A^{\dagger})=1/\sigma_{\min}(A^{\dagger})$ if $\sigma_{\min}(A^{\dagger})>0$. The benefit of choosing $W_{2}^{*}$ in this way is to ensure $\nabla_{W_{2}}\mathsf{L}_{\textrm{trn}}(W_{1},W_{2}^{*})$ is Lipschitz continuous over $W_{1}$, which can be proved by

$$ $\displaystyle~{}~{}~{}~{}~{}\|\nabla_{W_{2}}\mathsf{L}_{\textrm{trn}}(W_{1},W_{2}^{*})-\nabla_{W_{2}}\mathsf{L}_{\textrm{trn}}(W_{1}^{\prime},W_{2}^{*})\|_{2}$ $\displaystyle=\|-(X_{{\rm trn}}W_{1})^{\top}(Y_{{\rm trn}}-X_{{\rm trn}}W_{1}W_{2}^{*})+(X_{{\rm trn}}W_{1}^{\prime})^{\top}(Y_{{\rm trn}}-X_{{\rm trn}}W_{1}^{\prime}W_{2}^{*})\|_{2}$ $\displaystyle\leq\|(X_{{\rm trn}}W_{1})^{\top}X_{\rm trn}(W_{1}-W_{1}^{\prime})W_{2}^{*}\|_{2}+\|(W_{1}-W_{1}^{\prime})^{\top}X_{\rm trn}^{\top}(Y_{{\rm trn}}-X_{{\rm trn}}W_{1}^{\prime}W_{2}^{*})\|_{2}$ $\displaystyle\leq\sigma_{\max}^{2}(X_{\rm trn})\sigma_{\max}(W_{1})\sigma_{\max}(W_{2}^{*})\|W_{1}-W_{1}^{\prime}\|$ $\displaystyle~{}~{}~{}~{}~{}+\sigma_{\max}(X_{\rm trn})[\sigma_{\max}(Y_{\rm trn})+\sigma_{\max}(X_{\rm trn})\sigma_{\max}(W^{\prime}_{1})\sigma_{\max}(W_{2}^{*})]\|W_{1}-W_{1}^{\prime}\|$ $\displaystyle\leq\left[\frac{\sigma_{\max}^{2}(X_{\rm trn})\sigma_{\max}(Y_{{\rm trn}})R}{\sigma_{\min}(X_{{\rm trn}})r}+\sigma_{\max}(X_{\rm trn})\sigma_{\max}(Y_{\rm trn})\left(1+\frac{\sigma_{\max}(X_{\rm trn})R}{\sigma_{\min}(X_{{\rm trn}})r}\right)\right]\|W_{1}-W_{1}^{\prime}\|$ $$

where $W_{1},W_{1}^{\prime}\in\mathcal{W}_{1}$.

On the other hand, for any $W_{1}\in\mathcal{W}_{1}$, as long as it does not belong to the boundary, we can find $\bar{a}>0$ such that for any $a\leq\bar{a}$, $W_{1}+ad\in\mathcal{W}_{1}$ holds for any unit direction $d\in\mathbb{R}^{m\times h}$. Even for $W_{1}$ at the boundary of $\mathcal{W}_{1}$, one can slightly enlarge $\mathcal{W}_{1}$ by adjusting $r$ and $R$ correspondingly to ensure the singular value of any $W_{1}\in\mathbb{B}(W_{1},\bar{a})$ is uniformly lower and upper bounded. Then based on the Lipschitz continuity of $\nabla_{W_{2}}\mathsf{L}_{\textrm{trn}}(W_{1},W_{2}^{*})$ over $W_{1}$ and $W_{2}$ and the PL property of $\mathsf{L}_{\textrm{trn}}(W_{1},\cdot)$, and according to [, Lemma A.3], there exist $L>0$, for any $0<a\leq\bar{a}$ and any unit direction $d\in\mathbb{R}^{m\times h}$, one can choose $W_{2}^{*}(a)\in S(W_{1}+ad)$ such that $\|W_{2}^{*}(a)-W_{2}^{*}\|\leq La$.
By the Taylor expansion,

$$ $\displaystyle\mathsf{L}_{{\rm trn}}^{*}(W_{1}+ad)-\mathsf{L}_{{\rm trn}}^{*}(W_{1})$ $\displaystyle=\mathsf{L}_{{\rm trn}}(W_{1}+ad,W_{2}^{*}(a))-\mathsf{L}_{{\rm trn}}(W_{1},W_{2}^{*})$ $\displaystyle=a\nabla_{W_{1}}^{\top}\mathsf{L}_{{\rm trn}}(W_{1},W_{2}^{*})d+\nabla_{W_{2}}^{\top}\mathsf{L}_{{\rm trn}}(W_{1},W_{2}^{*})(W_{2}^{*}(a)-W_{2}^{*})+{\cal O}(a^{2})$ $\displaystyle=a\nabla_{W_{1}}^{\top}\mathsf{L}_{{\rm trn}}(W_{1},W_{2}^{*})d+{\cal O}(a^{2})$ $$

By the definition of directional derivative, we know

$$ $\displaystyle\mathsf{L}_{{\rm trn}}^{*,\prime}(W_{1};d)=\lim_{r\rightarrow 0^{+}}\frac{\mathsf{L}_{{\rm trn}}^{*}(W_{1}+ad)-\mathsf{L}_{{\rm trn}}^{*}(W_{1})}{a}=\nabla_{W_{1}}^{\top}\mathsf{L}_{{\rm trn}}(W_{1},W_{2}^{*})d$ $$

Since the above relationship holds for any direction $d$, then we get $\nabla\mathsf{L}_{{\rm trn}}^{*}(W_{1})=\nabla_{W_{1}}\mathsf{L}_{{\rm trn}}(W_{1},W_{2}^{*})$. In addition, the above discussion holds for any bounded $W_{2}^{*}$, which yields the first part of the conclusion. Moreover, as $\nabla_{W_{1}}\mathsf{L}_{{\rm trn}}(W_{1},W_{2}^{*})=\nabla\ell_{\rm trn}(W_{1}W_{2}^{*})W_{2}^{*,\top}=X_{\rm trn}^{\top}\ell_{\rm trn}(W_{1}W_{2}^{*})W_{2}^{*,\top}$ and $\ell_{\rm trn}(W_{1}W_{2}^{*})=\min_{W_{2}}{\mathsf{L}}_{\rm trn}(W_{1},W_{2})=0$ according to Lemma [5](#Thmlemma5), we get $\nabla\mathsf{L}_{{\rm trn}}^{*}(W_{1})=\nabla_{W_{1}}\mathsf{L}_{{\rm trn}}(W_{1},W_{2}^{*})=\nabla_{W_{1}}\mathsf{L}_{{\rm trn}}(W_{1},W_{2}^{*})=0$.

###### Lemma 19 .

Under Assumption [2](#Thmassumption2), for any $\epsilon>0$, there exists an $\epsilon$-solution to the penalized problem $(W_{1}^{\epsilon},W_{2}^{\epsilon})$ with $\mathsf{L}_{\rm trn}^{*}(W_{1}^{\epsilon})=0$.

Lemma [19](#Thmlemma19) is pivotal in ensuring that the penalized problem is well-posed. Otherwise, we encounter the situation where the optimal weight $(W_{1}^{*},W_{2}^{*})$ learned from the penalized method

$$ $\mathsf{L}_{\rm trn}^{*}(W_{1}^{*})\neq\min_{W_{1},W_{2}}\mathsf{L}_{\rm trn}(W_{1},W_{2})=0.$ $$

This implies that the learned bottom layer weight fails to fit well on the training dataset – a scenario we want to avoid. Without making assumption on the penalized problem, the well-poseness of it is derived from the singularity assumption of the bilevel problem and their approximated equivalence.

Proof:
Lemma [19](#Thmlemma19) is a side product of Theorem [4](#Thmtheorem4). According to Step 2-1 in the proof of Theorem [4](#Thmtheorem4), we know that for any $\epsilon$, there exists an $\epsilon$-solution to the penalized problem with full rank $W_{1}$. Together with Lemma [5](#Thmlemma5), we arrive at the conclusion.

###### Lemma 18 (Danskin type theorem) .

###### Lemma 19 .

#### F.4 Proof of Theorem 1

We restate Theorem [1](#Thmlemma1) and the descent lemma in the following theorem.

###### Theorem 5 (Local joint PL and smoothness) .

Suppose Assumption [2](#Thmassumption2) holds, assume $\sigma_{\min}^{2}(W_{1}^{k})>0$, $\sigma_{\min}^{2}(W_{2}^{k})>0$ and denote $X_{\gamma}=[X_{\rm val};\sqrt{\gamma}X_{\rm trn}]$. Then the descent lemma holds with $L_{k}$ defined in ([52](#A6.Ex130)) and the local joint PL inequality holds with $\mu_{k}=(\sigma_{\min}^{2}(W_{1}^{k})+\sigma_{\min}^{2}(W_{2}^{k}))\sigma_{*}^{2}(X_{\gamma})$, i.e.

$$ $\displaystyle\|\nabla\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\|^{2}$ $\displaystyle\geq 2\mu_{k}(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})$ $\displaystyle\mathsf{L}_{\gamma}(Z^{k+1})$ $\displaystyle\leq{\mathsf{L}}_{\gamma}(Z^{k})-\left(\frac{\alpha}{2}-\alpha^{2}L_{k}\right)\|\nabla{\mathsf{L}}_{\gamma}(Z^{k})\|^{2}+\left(\frac{\alpha}{2}+\alpha^{2}L_{k}\right)\|\delta_{k}\|^{2}.$ $$

where $Z=(W_{1},W_{2}),Z^{k}=(W_{1}^{k},W_{2}^{k})$, $\alpha$ and $\mathsf{L}_{\gamma}^{*}=\min_{Z}\mathsf{L}_{\gamma}(Z)$. Moreover, one has

$$ $\displaystyle\mathsf{L}_{\gamma}(Z^{k+1})-\mathsf{L}_{\gamma}^{*}$ $\displaystyle\leq\left(1-\alpha\mu_{k}+2\alpha^{2}L_{k}\mu_{k}\right)\left({\mathsf{L}}_{\gamma}(Z^{k})-\mathsf{L}_{\gamma}^{*}\right)+\left(\frac{\alpha}{2}+\alpha^{2}L_{k}\right)\|\delta_{k}\|^{2}.$ $$

Proof: We first prove the local PL inequality by four steps.

###### Theorem 5 (Local joint PL and smoothness) .

##### Step 1-1: Local PL inequality for 𝖫 trn ​ ( W 1 , W 2 ) subscript 𝖫 trn subscript 𝑊 1 subscript 𝑊 2 \mathsf{L}_{{\rm trn}}(W_{1},W_{2}) and 𝖫 val ​ ( W 1 , W 2 ) subscript 𝖫 val subscript 𝑊 1 subscript 𝑊 2 \mathsf{L}_{{\rm val}}(W_{1},W_{2}) .

First, we know that $h(\tilde{W})=\frac{1}{2}\|Y_{\rm trn}-\tilde{W}\|^{2}$ is $1$-strongly convex and $1$-Lipschitz smooth. Given matrix $W$, then according to [, Appendix B], $\ell_{\textrm{trn}}(W)=h(X_{\rm trn}W)$ is in the form of strongly convex composite with a linear mapping so that it is $\sigma_{*}^{2}(X_{\rm trn})$-PL over $W$. As $X_{\rm trn}$ is full rank, we have $\sigma_{*}^{2}(X_{\rm trn})=\sigma_{\min}^{2}(X_{\rm trn})$ and $\ell_{\textrm{trn}}(W)=h(X_{\rm trn}W)$ is $\sigma_{\min}^{2}(X_{\rm trn})$-PL over $W$. On the other hand, we also have

$$ $\displaystyle\min_{W}\ell_{\rm trn}(W)=\min_{W_{1},W_{2}}\mathsf{L}_{{\rm trn}}(W_{1},W_{2})$ (41) $$

because

$$ $\displaystyle\min_{W}\ell_{\rm trn}(W)\leq\ell_{\rm trn}(W_{1}^{*}W_{2}^{*})=\min_{W_{1},W_{2}}\mathsf{L}_{{\rm trn}}(W_{1},W_{2})$ (42) $$

where $W_{1}^{*},W_{2}^{*}=\operatorname*{arg\,min}_{W_{1},W_{2}}\mathsf{L}_{{\rm trn}}(W_{1},W_{2})$. The reverse direction of ([42](#A6.E42)) holds true because for any $W^{*}\in\operatorname*{arg\,min}_{W}\ell_{\rm trn}(W)$, it can be decomposed to

$$ $\displaystyle W^{*}=\left[W^{*}~{}~{}0_{n\times(h-m)}\right]\left[\begin{array}[]{l}I_{m\times m}\\ 0_{(h-m)\times m}\end{array}\right].$ $$

Based on the above facts and denoting $W=W_{1}W_{2}$, we can prove the local PL property of $\mathsf{L}_{\rm trn}$ over $(W_{1},W_{2})$ by

$$ $\displaystyle\|\nabla\mathsf{L}_{{\rm trn}}(W_{1},W_{2})\|^{2}$ $\displaystyle=\|\nabla\ell_{\rm trn}(W)W_{2}^{\top}\|^{2}+\|W_{1}^{\top}\nabla\ell_{\rm trn}(W)\|^{2}$ $\displaystyle=\|W_{2}\nabla\ell_{\rm trn}(W)^{\top}\|^{2}+\|W_{1}^{\top}\nabla\ell_{\rm trn}(W)\|^{2}$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{\geq}}\sigma_{\min}^{2}(W_{2})\|\nabla\ell_{\rm trn}(W)\|^{2}+\sigma_{\min}^{2}(W_{1})\|\nabla\ell_{\rm trn}(W)\|^{2}$ $\displaystyle=(\sigma_{\min}^{2}(W_{1})+\sigma_{\min}^{2}(W_{2}))\|\nabla\ell_{\rm trn}(W)\|^{2}$ $\displaystyle\stackrel{{\scriptstyle(b)}}{{\geq}}2(\sigma_{\min}^{2}(W_{1})+\sigma_{\min}^{2}(W_{2}))\sigma_{\min}^{2}(X_{\rm trn})(\ell_{\rm trn}(W)-\min_{W}\ell_{\rm trn}(W))$ $\displaystyle\stackrel{{\scriptstyle(c)}}{{\geq}}2(\sigma_{\min}^{2}(W_{1})+\sigma_{\min}^{2}(W_{2}))\sigma_{\min}^{2}(X_{\rm trn})(\mathsf{L}_{{\rm trn}}(W_{1},W_{2})-\min_{W_{1},W_{2}}\mathsf{L}_{{\rm trn}}(W_{1},W_{2}))$ (43) $$

where (a) comes from Lemma [4](#Thmlemma4), (b) is derived from $\ell_{\rm trn}(W)$ is $\sigma_{\min}^{2}(X_{\rm trn})$-PL, and (c) is because $\min_{W}\ell_{\rm trn}(W)=\min_{W_{1},W_{2}}\mathsf{L}_{{\rm trn}}(W_{1},W_{2})$. Similarly, for the validation loss, it holds that

$$ $\displaystyle\|\nabla\mathsf{L}_{{\rm val}}(W_{1}^{k},W_{2}^{k})\|^{2}$ $\displaystyle\geq 2(\sigma_{\min}^{2}(W_{1})+\sigma_{\min}^{2}(W_{2}))\sigma_{\min}^{2}(X_{\rm val})(\mathsf{L}_{{\rm val}}(W_{1},W_{2})-\min_{W_{1},W_{2}}\mathsf{L}_{{\rm val}}(W_{1},W_{2})).$ $$

##### Step 1-2: Local PL inequality for 𝖫 ~ γ ( W 1 , W 2 ) = : 𝖫 val ( W 1 , W 2 ) + γ 𝖫 trn ( W 1 , W 2 ) \widetilde{\mathsf{L}}_{\gamma}(W_{1},W_{2})=:\mathsf{L}_{\rm val}(W_{1},W_{2})+\gamma\mathsf{L}_{\rm trn}(W_{1},W_{2}) .

By noticing the fact that $(\ell_{\rm val}+\gamma\ell_{\rm trn})(W)=\frac{1}{2}\|Y_{\gamma}-X_{\gamma}W\|^{2}$ with $Y_{\gamma}=[Y_{\rm val};\sqrt{\gamma}Y_{\rm trn}]$ and $X_{\gamma}=[X_{\rm val};\sqrt{\gamma}X_{\rm trn}]$, we have

$$ $\displaystyle\|\nabla\widetilde{\mathsf{L}}_{\gamma}(W_{1},W_{2})\|^{2}$ $\displaystyle=\|\nabla(\ell_{\rm val}+\gamma\ell_{\rm trn})(W)W_{2}^{\top}\|^{2}+\|W_{1}^{\top}\nabla(\ell_{\rm val}+\gamma\ell_{\rm trn})(W)\|^{2}$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{\geq}}2(\sigma_{\min}^{2}(W_{1})+\sigma_{\min}^{2}(W_{2}))\sigma_{*}^{2}(X_{\gamma})(\widetilde{\mathsf{L}}_{\gamma}(W_{1},W_{2})-\min_{W_{1},W_{2}}\widetilde{\mathsf{L}}_{\gamma}(W_{1},W_{2}))$ (44) $$

where (a) is derived similarly from the derivation of ([43](#A6.Ex79)).

##### Step 1-3: We then prove the relation of 𝖫 γ ​ ( W 1 , W 2 ) subscript 𝖫 𝛾 subscript 𝑊 1 subscript 𝑊 2 \mathsf{L}_{\gamma}(W_{1},W_{2}) and 𝖫 ~ γ ​ ( W 1 , W 2 ) subscript ~ 𝖫 𝛾 subscript 𝑊 1 subscript 𝑊 2 \widetilde{\mathsf{L}}_{\gamma}(W_{1},W_{2}) that min W 1 , W 2 ⁡ 𝖫 γ ​ ( W 1 , W 2 ) = min W 1 , W 2 ⁡ 𝖫 ~ γ ​ ( W 1 , W 2 ) . subscript subscript 𝑊 1 subscript 𝑊 2 subscript 𝖫 𝛾 subscript 𝑊 1 subscript 𝑊 2 subscript subscript 𝑊 1 subscript 𝑊 2 subscript ~ 𝖫 𝛾 subscript 𝑊 1 subscript 𝑊 2 \displaystyle\min_{W_{1},W_{2}}\mathsf{L}_{\gamma}(W_{1},W_{2})=\min_{W_{1},W_{2}}\widetilde{\mathsf{L}}_{\gamma}(W_{1},W_{2}). (45)

Since $\mathsf{L}_{\rm trn}(W_{1},W_{2})\geq 0$, we know $\mathsf{L}_{\rm trn}^{*}(W_{1})=\min_{W_{2}}\mathsf{L}_{\rm trn}(W_{1},W_{2})\geq 0$, and thus

$$ $\mathsf{L}_{\gamma}(W_{1},W_{2})=\mathsf{L}_{\rm val}(W_{1},W_{2})+\gamma\mathsf{L}_{\rm trn}(W_{1},W_{2})-\gamma\mathsf{L}_{\rm trn}^{*}(W_{1})\leq\mathsf{L}_{\rm val}(W_{1},W_{2})+\gamma\mathsf{L}_{\rm trn}(W_{1},W_{2}).$ $$

Taking the minimization over both sides yields

$$ $\displaystyle\min_{W_{1},W_{2}}\mathsf{L}_{\gamma}(W_{1},W_{2})\leq\min_{W_{1},W_{2}}(\mathsf{L}_{\rm val}(W_{1},W_{2})+\gamma\mathsf{L}_{\rm trn}(W_{1},W_{2}))=\min_{W_{1},W_{2}}\widetilde{\mathsf{L}}_{\gamma}(W_{1},W_{2}).$ (46) $$

By Lemma [19](#Thmlemma19), for any $\epsilon>0$, $\exists(W_{1}^{\epsilon},W_{2}^{\epsilon})$ is an $\epsilon$-solution of $\mathsf{L}_{\gamma}(W_{1},W_{2})$ with $\mathsf{L}_{\rm trn}^{*}(W_{1}^{*})=0$, so

$$ $\displaystyle\epsilon+\min_{W_{1},W_{2}}\mathsf{L}_{\gamma}(W_{1},W_{2})\geq\mathsf{L}_{\gamma}(W_{1}^{\epsilon},W_{2}^{\epsilon})$ $\displaystyle=\mathsf{L}_{\rm val}(W_{1}^{\epsilon},W_{2}^{\epsilon})+\gamma\mathsf{L}_{\rm trn}(W_{1}^{\epsilon},W_{2}^{\epsilon})$ $\displaystyle\geq\min_{W_{1},W_{2}}(\mathsf{L}_{\rm val}(W_{1},W_{2})+\gamma\mathsf{L}_{\rm trn}(W_{1},W_{2}))=\min_{W_{1},W_{2}}\widetilde{\mathsf{L}}_{\gamma}(W_{1},W_{2})$ $$

holds for any $\epsilon$. Letting $\epsilon\rightarrow 0$, we get

$$ $\displaystyle\min_{W_{1},W_{2}}\mathsf{L}_{\gamma}(W_{1},W_{2})\geq\min_{W_{1},W_{2}}\widetilde{\mathsf{L}}_{\gamma}(W_{1},W_{2})$ $$

Together with ([46](#A6.E46)), ([45](#A6.E45)) must hold true.

##### Step 1-4: Local PL property of 𝖫 γ ​ ( W 1 , W 2 ) subscript 𝖫 𝛾 subscript 𝑊 1 subscript 𝑊 2 \mathsf{L}_{\gamma}(W_{1},W_{2}) .

$$ $\displaystyle\|\nabla\mathsf{L}_{\gamma}(W_{1},W_{2})\|^{2}$ $\displaystyle=\|\nabla\ell_{\rm val}(W)W_{2}^{\top}+\gamma\nabla\ell_{\rm trn}(W)W_{2}^{\top}-\gamma\nabla\mathsf{L}_{{\rm trn}}^{*}(W_{1})\|^{2}$ $\displaystyle~{}~{}~{}~{}+\|W_{1}^{\top}\nabla\ell_{\rm val}(W)+\gamma W_{1}^{\top}\nabla\ell_{\rm trn}(W)\|^{2}$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}\|\nabla\ell_{\rm val}(W)W_{2}^{\top}+\gamma\nabla\ell_{\rm trn}(W)W_{2}^{\top}\|^{2}+\|W_{1}^{\top}\nabla\ell_{\rm val}(W)+\gamma W_{1}^{\top}\nabla\ell_{\rm trn}(W)\|^{2}$ $\displaystyle=\|\nabla(\ell_{\rm val}+\gamma\ell_{\rm trn})(W)W_{2}^{\top}\|^{2}+\|W_{1}^{\top}\nabla(\ell_{\rm val}+\gamma\ell_{\rm trn})(W)\|^{2}$ $\displaystyle\stackrel{{\scriptstyle(b)}}{{\geq}}2(\sigma_{\min}^{2}(W_{1})+\sigma_{\min}^{2}(W_{2}))\sigma_{*}^{2}(X_{\gamma})\left(\widetilde{\mathsf{L}}_{\gamma}(W_{1},W_{2})-\min_{W_{1},W_{2}}\widetilde{\mathsf{L}}_{\gamma}(W_{1},W_{2})\right)$ $\displaystyle\stackrel{{\scriptstyle(c)}}{{=}}2(\sigma_{\min}^{2}(W_{1})+\sigma_{\min}^{2}(W_{2}))\sigma_{*}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1},W_{2})-\min_{W_{1},W_{2}}\mathsf{L}_{\gamma}(W_{1},W_{2}))$ $\displaystyle=2\mu_{k}(\mathsf{L}_{\gamma}(W_{1},W_{2})-\min_{W_{1},W_{2}}\mathsf{L}_{\gamma}(W_{1},W_{2}))$ $$

where (a) comes from Lemma [18](#Thmlemma18) and (b) is derived from ([44](#A6.Ex82)), and (c) holds because of ([45](#A6.E45)) and $\mathsf{L}_{\rm trn}^{*}(W_{1})=0$ when $\sigma_{\min}(W_{1})>0$.

Next, we will prove the descent lemma. Denoting $Z^{k}=(W_{1}^{k},W_{2}^{k})$, the update of PBGD can be formulated as

$$ $\displaystyle Z^{k+1}=Z^{k}-\alpha(\nabla\mathsf{L}_{\gamma}(Z^{k})+\delta_{k})$ (47) $$

where $\delta_{k}=\gamma(\nabla\mathsf{L}^{*}_{\rm trn}(W_{1}^{k})-\nabla_{W_{1}}\mathsf{L}_{\rm trn}(W_{1}^{k},W_{3}^{k+1}))$. Let us denote

$$ $\displaystyle H(\kappa)$ $\displaystyle=\nabla^{2}\widetilde{\mathsf{L}}_{\gamma}\left((1-\kappa)W_{1}^{k}+\kappa W_{1}^{k+1},(1-\kappa)W_{2}^{k}+\kappa W_{2}^{k+1}\right)$ $\displaystyle=\nabla^{2}\widetilde{\mathsf{L}}_{\gamma}\left(W_{1}^{k}-\alpha\kappa(\nabla_{W_{1}}\mathsf{L}_{\gamma}(Z^{k})+\delta_{k}),W_{2}^{k}-\alpha\kappa\nabla_{W_{2}}\mathsf{L}_{\gamma}(Z^{k})\right)$ $\displaystyle=\nabla^{2}\widetilde{\mathsf{L}}_{\gamma}\left(W_{1}^{k}-\alpha\kappa(\nabla_{W_{1}}\widetilde{\mathsf{L}}_{\gamma}(Z^{k})+\delta_{k}),W_{2}^{k}-\alpha\kappa\nabla_{W_{2}}\widetilde{\mathsf{L}}_{\gamma}(Z^{k})\right)$ $\displaystyle=\nabla^{2}\widetilde{\mathsf{L}}_{\gamma}\left(Z^{k}-\alpha\kappa(\nabla\widetilde{\mathsf{L}}_{\gamma}(Z^{k})+\delta_{k})\right)$ (48) $$

then by second order Taylor expansion, we have

$$ $\displaystyle~{}~{}~{}~{}~{}\mathsf{L}_{\gamma}(Z^{k+1})\leq\widetilde{\mathsf{L}}_{\gamma}(Z^{k+1})$ $\displaystyle=\widetilde{\mathsf{L}}_{\gamma}(Z^{k})+\langle\nabla\widetilde{\mathsf{L}}_{\gamma}(Z^{k}),Z^{k+1}-Z^{k}\rangle+\int_{0}^{1}(1-\kappa)\left\langle Z_{t+1}-Z_{t},H(\kappa)(Z^{k+1}-Z^{k})\right\rangle d\kappa$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}{\mathsf{L}}_{\gamma}(Z^{k})-\alpha\langle\nabla{\mathsf{L}}_{\gamma}(Z^{k}),\nabla\mathsf{L}_{\gamma}(Z^{k})+\delta_{k}\rangle+\int_{0}^{1}(1-\kappa)\left\langle Z_{t+1}-Z_{t},H(\kappa)(Z^{k+1}-Z^{k})\right\rangle d\kappa$ $\displaystyle\stackrel{{\scriptstyle(b)}}{{\leq}}{\mathsf{L}}_{\gamma}(Z^{k})-\frac{\alpha\|\nabla{\mathsf{L}}_{\gamma}(Z^{k})\|^{2}}{2}+\frac{\alpha\|\delta_{k}\|^{2}}{2}+\alpha^{2}\int_{0}^{1}(1-\kappa)\left\langle\nabla\mathsf{L}_{\gamma}(Z^{k})+\delta_{k},H(\kappa)(\nabla\mathsf{L}_{\gamma}(Z^{k})+\delta_{k})\right\rangle d\kappa$ $\displaystyle\stackrel{{\scriptstyle(c)}}{{=}}{\mathsf{L}}_{\gamma}(Z^{k})-\frac{\alpha\|\nabla{\mathsf{L}}_{\gamma}(Z^{k})\|^{2}}{2}+\frac{\alpha\|\delta_{k}\|^{2}}{2}+\alpha^{2}\|\nabla\mathsf{L}_{\gamma}(Z^{k})+\delta_{k}\|^{2}\int_{0}^{1}(1-\kappa)\left\langle g_{k},H(\kappa)g_{k}\right\rangle d\kappa$ $\displaystyle\stackrel{{\scriptstyle(d)}}{{\leq}}{\mathsf{L}}_{\gamma}(Z^{k})-\frac{\alpha\|\nabla{\mathsf{L}}_{\gamma}(Z^{k})\|^{2}}{2}+\frac{\alpha\|\delta_{k}\|^{2}}{2}+2\alpha^{2}\left(\|\nabla\mathsf{L}_{\gamma}(Z^{k})\|^{2}+\|\delta_{k}\|^{2}\right)\int_{0}^{1}(1-\kappa)L_{k}d\kappa$ $\displaystyle={\mathsf{L}}_{\gamma}(Z^{k})-\left(\frac{\alpha}{2}-\alpha^{2}L_{k}\right)\|\nabla{\mathsf{L}}_{\gamma}(Z^{k})\|^{2}+\left(\frac{\alpha}{2}+\alpha^{2}L_{k}\right)\|\delta_{k}\|^{2}$ (49) $$

where (a) is derived from ([47](#A6.E47)), $\sigma_{\min}(W_{1}^{k})>0$, Lemma [5](#Thmlemma5) and Lemma [18](#Thmlemma18), (b) holds because

$$ $\langle a,b\rangle=\frac{\|a\|^{2}}{2}+\frac{\|b\|^{2}}{2}-\frac{\|a-b\|^{2}}{2}\geq\frac{\|a\|^{2}}{2}-\frac{\|a-b\|^{2}}{2}$ $$

$g_{k}$ in (c) is defined as $\frac{\nabla\mathsf{L}_{\gamma}(Z^{k})+\delta_{k}}{\|\nabla\mathsf{L}_{\gamma}(Z^{k})+\delta_{k}\|}$, and (d) is earned by Lemma [22](#Thmlemma22).

Finally, according to ([49](#A6.Ex104)) and the local PL property of $\mathsf{L}_{\gamma}(W_{1},W_{2})$, we have

$$ $\displaystyle\mathsf{L}_{\gamma}(Z^{k+1})$ $\displaystyle\leq{\mathsf{L}}_{\gamma}(Z^{k})-\left(\frac{\alpha}{2}-\alpha^{2}L_{k}\right)\|\nabla{\mathsf{L}}_{\gamma}(Z^{k})\|^{2}+\left(\frac{\alpha}{2}+\alpha^{2}L_{k}\right)\|\delta_{k}\|^{2}$ $\displaystyle\leq{\mathsf{L}}_{\gamma}(Z^{k})-\left(\alpha-2\alpha^{2}L_{k}\right)\mu_{k}(\mathsf{L}_{\gamma}(Z^{k})-\mathsf{L}_{\gamma}^{*})+\left(\frac{\alpha}{2}+\alpha^{2}L_{k}\right)\|\delta_{k}\|^{2}.$ $$

Subtracting both sides by $\mathsf{L}_{\gamma}^{*}$ yields

$$ $\displaystyle\mathsf{L}_{\gamma}(Z^{k+1})-\mathsf{L}_{\gamma}^{*}$ $\displaystyle\leq\left(1-\alpha\mu_{k}+2\alpha^{2}L_{k}\mu_{k}\right)\left({\mathsf{L}}_{\gamma}(Z^{k})-\mathsf{L}_{\gamma}^{*}\right)+\left(\frac{\alpha}{2}+\alpha^{2}L_{k}\right)\|\delta_{k}\|^{2}.$ $$

We give the characteristic of $L_{k}$, the bound of $\left\langle g_{k},H(\kappa)g_{k}\right\rangle$, by showing $\left\langle g_{k},H(0)g_{k}\right\rangle$ and $|\langle g_{k},(H(\kappa)-H(0))g_{k}\rangle|$ are bounded subsequently. The upper bound of $\left\langle g_{k},H(0)g_{k}\right\rangle$ is adapted from [, Lemma D.1] as it is independent on the update direction, as long as $g_{k}$ is normalized.

###### Lemma 20 ( [ 75 , Lemma D.1] ) .

Let $g_{k}=\frac{\nabla\mathsf{L}_{\gamma}(Z^{k})+\delta_{k}}{\|\nabla\mathsf{L}_{\gamma}(Z^{k})+\delta_{k}\|}$ and $H(\kappa)$ defined in ([48](#A6.Ex97)), then it holds that

$$ $\displaystyle~{}~{}~{}~{}\left\langle g_{k},H(0)g_{k}\right\rangle$ $\displaystyle=\frac{1}{\alpha^{2}\|\nabla\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\|^{2}}\cdot\frac{d^{2}}{ds^{2}}M(s)\Big{|}_{s=0}$ $\displaystyle\leq\frac{1}{\alpha^{2}\|\nabla\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\|^{2}}\left(\langle\nabla\ell_{\gamma}(A(s)),\frac{d^{2}}{ds^{2}}A(s)\rangle\Big{|}_{s=0}+\sigma_{\max}^{2}(X_{\gamma})\|\frac{d}{ds}A(s+\kappa)\|^{2}\Big{|}_{s=0}\right)$ $\displaystyle\leq\sigma_{\max}^{2}(X_{\gamma})(\sigma_{\max}^{2}(W_{1}^{k})+\sigma_{\max}^{2}(W_{2}^{k}))+\sqrt{2\sigma_{\max}^{2}(X_{\gamma})({\mathsf{L}}_{\gamma}(W_{1}^{k},W_{2}^{k})-{\mathsf{L}}_{\gamma}^{*})}.$ (50) $$

To show the boundedness of $|\langle g_{k},(H(\kappa)-H(0))g_{k}\rangle|$, we define the loss at the intermediate point and the product of $W_{1}^{k}-s\alpha(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k})$ and $W_{2}^{k}-s\alpha\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})$ as follows.

$$ $\displaystyle M(s)$ $\displaystyle=\mathsf{L}_{\gamma}\left(W_{1}^{k}-s\alpha\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right),W_{2}^{k}-s\alpha\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right)$ $\displaystyle A(s)$ $\displaystyle=W^{k}-s\alpha\left(\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)W_{2}^{k}+W_{1}^{k}\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right)$ $\displaystyle~{}~{}~{}~{}+s^{2}\alpha^{2}\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})$ $$

where $W^{k}=W_{1}^{k}W_{2}^{k}$ and $\ell_{\gamma}=\ell_{\rm val}+\gamma\ell_{\rm trn}$. In this way, we have

$$ $\displaystyle M(0)=\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k}),~{}~{}~{}~{}~{}M(1)=\mathsf{L}_{\gamma}(W_{1}^{k+1},W_{2}^{k+1}),~{}~{}~{}\text{ and }~{}~{}~{}M(s)=\ell_{\gamma}(A(s)).$ $$

Then establishing the bound of $\|H(\kappa)-H(0)\|$ depends on the amount of $\|A(\kappa)-A(0)\|$ because $H(\kappa)=\nabla^{2}\widetilde{\mathsf{L}}_{\gamma}(M(\kappa))=\nabla^{2}\widetilde{\mathsf{L}}_{\gamma}(\ell_{\gamma}(A(\kappa)))$.

###### Lemma 21 .

For any $\kappa\in[0,1)$, if $\|\delta_{k}\|\leq\delta$ and $\sigma_{\min}(W_{1}^{k})>0,\sigma_{\min}(W_{2}^{k})>0$, then it holds that

$$ $\displaystyle\|A(\kappa)-A(0)\|$ $\displaystyle\leq\alpha\left(\sigma_{\max}^{2}(W_{1}^{k})+\sigma_{\max}^{2}(W_{2}^{k})\right)\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})}$ $\displaystyle~{}~{}~{}~{}+\alpha\delta\sigma_{\max}(W_{2}^{k})+2\alpha^{2}\sigma_{\max}^{2}(X_{\gamma})\sigma_{\max}(W^{k})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})$ $\displaystyle~{}~{}~{}~{}+\alpha^{2}\delta\sigma_{\max}(W_{1}^{k})\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})}.$ (51) $$

Proof:
According to the definition, we have

$$ $\displaystyle\|A(\kappa)-A(0$ $\displaystyle)\|=\left\|-\kappa\alpha\left(\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)W_{2}^{k}+W_{1}^{k}\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right)\right.$ $\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}\left.+\kappa^{2}\alpha^{2}\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\|$ $\displaystyle~{}~{}~{}~{}\leq\kappa\alpha\left\|\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)W_{2}^{k}+W_{1}^{k}\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\|$ $\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}+\kappa^{2}\alpha^{2}\left\|\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\|$ $\displaystyle\overset{\eqref{aux2}\&\eqref{aux3}}{\leq}\kappa\alpha\left(\sigma_{\max}^{2}(W_{1}^{k})+\sigma_{\max}^{2}(W_{2}^{k})\right)\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})}$ $\displaystyle~{}~{}~{}~{}~{}~{}~{}+\kappa\alpha\delta\sigma_{\max}(W_{2}^{k})+2\kappa^{2}\alpha^{2}\sigma_{\max}^{2}(X_{\gamma})\sigma_{\max}(W^{k})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})$ $\displaystyle~{}~{}~{}~{}~{}~{}~{}+\kappa^{2}\alpha^{2}\delta\sigma_{\max}(W_{1}^{k})\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})}.$ $$

Letting $\kappa<1$, we get the conclusion.

###### Lemma 22 .

For any $\kappa\in[0,1)$, let $\delta:=\|\delta_{k}\|$, then it holds that

$$ $\displaystyle\left\langle g_{k},H(\kappa)g_{k}\right\rangle$ $\displaystyle\leq\sigma_{\max}^{2}(X_{\gamma})(\sigma_{\max}^{2}(W_{1}^{k})+\sigma_{\max}^{2}(W_{2}^{k}))+3\alpha\delta\sigma_{\max}^{2}(X_{\gamma})\sigma_{\max}(W_{2}^{k})$ $\displaystyle~{}~{}~{}~{}+\left(1+3\alpha\sigma_{\max}^{2}(X_{\gamma})(\sigma_{\max}^{2}(W_{1}^{k})+\sigma_{\max}^{2}(W_{2}^{k}))\right)\sqrt{2\sigma_{\max}^{2}(X_{\gamma})({\mathsf{L}}_{\gamma}(W_{1}^{k},W_{2}^{k})-{\mathsf{L}}_{\gamma}^{*})}$ $\displaystyle~{}~{}~{}~{}+3\alpha^{2}\sigma_{\max}^{2}(X_{\gamma})\delta\sigma_{\max}(W_{1}^{k})\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})}$ $\displaystyle~{}~{}~{}~{}+6\alpha^{2}\sigma_{\max}^{4}(X_{\gamma})\sigma_{\max}(W^{k})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})=:L_{k}$ (52) $$

Proof: First, we observe that $\left\langle g_{k},H(\kappa)g_{k}\right\rangle$ is the second-order directional derivative of $\mathsf{L}_{\gamma}$ with respect to the update direction, i.e.

$$ $\displaystyle\left\langle g_{k},H(\kappa)g_{k}\right\rangle=\frac{1}{\alpha^{2}\|\nabla\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\|^{2}}\cdot\frac{d^{2}}{ds^{2}}M(s+\kappa)\Big{|}_{s=0}$ (53) $$

For the directional derivative, we have

$$ $\displaystyle~{}~{}~{}~{}~{}\frac{d^{2}}{ds^{2}}M(s+\kappa)\Big{|}_{s=0}$ $\displaystyle=\frac{d^{2}}{ds^{2}}\ell_{\gamma}(A(s+\kappa))\Big{|}_{s=0}$ $\displaystyle=\frac{d}{ds}\langle\nabla\ell_{\gamma}(A(s+\kappa)),\frac{d}{ds}A(s+\kappa)\rangle\Big{|}_{s=0}$ $\displaystyle=\langle\nabla\ell_{\gamma}(A(s+\kappa)),\frac{d^{2}}{ds^{2}}A(s+\kappa)\rangle\Big{|}_{s=0}+\langle\frac{d}{ds}A(s+\kappa),\nabla^{2}\ell_{\gamma}(A(s+\kappa))\frac{d}{ds}A(s+\kappa)\rangle\Big{|}_{s=0}$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}\langle\nabla\ell_{\gamma}(A(s+\kappa)),\frac{d^{2}}{ds^{2}}A(s+\kappa)\rangle\Big{|}_{s=0}+\sigma_{\max}^{2}(X_{\gamma})\|\frac{d}{ds}A(s+\kappa)\|^{2}\Big{|}_{s=0}$ (54) $$

where (a) uses the fact that $\ell_{\gamma}$ is $\sigma_{\max}^{2}(X_{\gamma})$ smooth. Then we bound $\langle\nabla\ell_{\gamma}(A(s+\kappa)),\frac{d^{2}}{ds^{2}}A(s+\kappa)\rangle\Big{|}_{s=0}$ and $\|\frac{d}{ds}A(s+\kappa)\|^{2}\Big{|}_{s=0}$ as follows.

$$ $\displaystyle~{}~{}~{}~{}\langle\nabla\ell_{\gamma}(A(s+\kappa)),\frac{d^{2}}{ds^{2}}A(s+\kappa)\rangle\Big{|}_{s=0}$ $\displaystyle=2\langle\nabla\ell_{\gamma}(A(\kappa)),\alpha^{2}\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\rangle$ $\displaystyle=2\langle\nabla\ell_{\gamma}(A(\kappa))-\nabla\ell_{\gamma}(A(0)),\alpha^{2}\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\rangle$ $\displaystyle~{}~{}~{}~{}~{}+2\langle\nabla\ell_{\gamma}(A(0)),\alpha^{2}\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\rangle$ $\displaystyle\leq 2\alpha^{2}\sigma_{\max}^{2}(X_{\gamma})\|A(\kappa)-A(0)\|\|(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k})\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\|$ $\displaystyle~{}~{}~{}~{}+\langle\nabla\ell_{\gamma}(A(s)),\frac{d^{2}}{ds^{2}}A(s)\rangle\Big{|}_{s=0}$ (55) $$

On the other hand,

$$ $\displaystyle~{}~{}~{}~{}\|\frac{d}{ds}A(s+\kappa)\|^{2}\Big{|}_{s=0}$ $\displaystyle=\left\|\alpha\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)W_{2}^{k}+\alpha W_{1}^{k}\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right.$ $\displaystyle~{}~{}~{}~{}~{}~{}\left.-2\kappa\alpha^{2}\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\|^{2}$ $\displaystyle=\alpha^{2}\left\|\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)W_{2}^{k}+W_{1}^{k}\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\|^{2}$ $\displaystyle~{}~{}~{}~{}+4\kappa^{2}\alpha^{4}\left\|\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\|^{2}$ $\displaystyle~{}~{}~{}~{}-4\kappa\alpha^{3}\left\langle\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k}),\right.$ $\displaystyle~{}~{}~{}~{}~{}~{}\left.\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)W_{2}^{k}+W_{1}^{k}\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\rangle$ $\displaystyle\leq\|\frac{d}{ds}A(s)\|^{2}\Big{|}_{s=0}+4\kappa^{2}\alpha^{4}\left\|\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\|^{2}$ $\displaystyle~{}~{}~{}~{}-4\kappa\alpha^{3}\left\langle\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k}),\right.$ $\displaystyle~{}~{}~{}~{}~{}~{}\left.\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)W_{2}^{k}+W_{1}^{k}\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\rangle$ (56) $$

Plugging ([55](#A6.Ex141)) and ([56](#A6.Ex151)) into ([54](#A6.Ex135)), we get

$$ $\displaystyle~{}~{}~{}~{}~{}\frac{d^{2}}{ds^{2}}M(s+\kappa)\Big{|}_{s=0}$ $\displaystyle\leq 2\alpha^{2}\sigma_{\max}^{2}(X_{\gamma})\|A(\kappa)-A(0)\|\|(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k})\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\|$ $\displaystyle~{}~{}~{}~{}+\langle\nabla\ell_{\gamma}(A(s)),\frac{d^{2}}{ds^{2}}A(s)\rangle\Big{|}_{s=0}+\sigma_{\max}^{2}(X_{\gamma})\|\frac{d}{ds}A(s)\|^{2}\Big{|}_{s=0}$ $\displaystyle~{}~{}~{}~{}+4\kappa^{2}\alpha^{4}\sigma_{\max}^{2}(X_{\gamma})\left\|\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\|^{2}$ $\displaystyle~{}~{}~{}~{}~{}-4\kappa\alpha^{3}\sigma_{\max}^{2}(X_{\gamma})\left\langle\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k}),\right.$ $\displaystyle~{}~{}~{}~{}~{}~{}\left.\left(\nabla_{W_{1}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\right)W_{2}^{k}+W_{1}^{k}\nabla_{W_{2}}\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})\right\rangle$ $\displaystyle\leq\alpha^{2}\sigma_{\max}^{2}(X_{\gamma})\|A(\kappa)-A(0)\|\|\nabla\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\|^{2}$ $\displaystyle~{}~{}~{}~{}+\left[\sigma_{\max}^{2}(X_{\gamma})(\sigma_{\max}^{2}(W_{1}^{k})+\sigma_{\max}^{2}(W_{2}^{k}))+\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\widetilde{\mathsf{L}}_{\gamma}(W_{1}^{k},W_{2}^{k})-{\mathsf{L}}_{\gamma}^{*})}\right]\alpha^{2}\|\nabla\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\|^{2}$ $\displaystyle~{}~{}~{}~{}+4\kappa^{2}\alpha^{4}\sigma_{\max}^{4}(X_{\gamma})\sigma_{\max}(W^{k})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})\|\nabla\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\|^{2}$ $\displaystyle~{}~{}~{}~{}+2\kappa^{2}\alpha^{4}\sigma_{\max}^{2}(X_{\gamma})\delta\sigma_{\max}(W_{1}^{k})\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})}\|\nabla\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\|^{2}$ $\displaystyle~{}~{}~{}~{}+2\kappa\alpha^{3}\sigma_{\max}^{2}(X_{\gamma})\left(\sigma_{\max}^{2}(W_{1}^{k})+\sigma_{\max}^{2}(W_{2}^{k})\right)\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})}\|\nabla\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\|^{2}$ $\displaystyle~{}~{}~{}~{}+2\kappa\alpha^{3}\sigma_{\max}^{2}(X_{\gamma})\delta\sigma_{\max}(W_{2}^{k})\|\nabla\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\|^{2}$ $$

where the last inequality follows from Lemma [9](#Thmlemma9) and ([50](#A6.Ex112)).
Plugging the above bound and the bound of $\|A(\kappa)-A(0)\|$ in ([51](#A6.Ex119)) into ([53](#A6.E53)) and note that $\kappa\leq 1$, we obtain that

$$ $\displaystyle\left\langle g_{k},H(\kappa)g_{k}\right\rangle$ $\displaystyle=\frac{1}{\alpha^{2}\|\nabla\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})+\delta_{k}\|^{2}}\cdot\frac{d^{2}}{ds^{2}}M(s+\kappa)\Big{|}_{s=0}$ $\displaystyle\leq\alpha\sigma_{\max}^{2}(X_{\gamma})\left(\sigma_{\max}^{2}(W_{1}^{k})+\sigma_{\max}^{2}(W_{2}^{k})\right)\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})}$ $\displaystyle~{}~{}~{}~{}+\alpha\delta\sigma_{\max}^{2}(X_{\gamma})\sigma_{\max}(W_{2}^{k})+2\alpha^{2}\sigma_{\max}^{4}(X_{\gamma})\sigma_{\max}(W^{k})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})$ $\displaystyle~{}~{}~{}~{}+\alpha^{2}\delta\sigma_{\max}^{2}(X_{\gamma})\sigma_{\max}(W_{1}^{k})\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})}$ $\displaystyle~{}~{}~{}~{}+\sigma_{\max}^{2}(X_{\gamma})(\sigma_{\max}^{2}(W_{1}^{k})+\sigma_{\max}^{2}(W_{2}^{k}))+\sqrt{2\sigma_{\max}^{2}(X_{\gamma})({\mathsf{L}}_{\gamma}(W_{1}^{k},W_{2}^{k})-{\mathsf{L}}_{\gamma}^{*})}$ $\displaystyle~{}~{}~{}~{}+4\alpha^{2}\sigma_{\max}^{4}(X_{\gamma})\sigma_{\max}(W^{k})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})$ $\displaystyle~{}~{}~{}~{}+2\alpha^{2}\sigma_{\max}^{2}(X_{\gamma})\delta\sigma_{\max}(W_{1}^{k})\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})}$ $\displaystyle~{}~{}~{}~{}+2\alpha\sigma_{\max}^{2}(X_{\gamma})\left(\sigma_{\max}^{2}(W_{1}^{k})+\sigma_{\max}^{2}(W_{2}^{k})\right)\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})}$ $\displaystyle~{}~{}~{}~{}+2\alpha\sigma_{\max}^{2}(X_{\gamma})\delta\sigma_{\max}(W_{2}^{k})$ $\displaystyle=\sigma_{\max}^{2}(X_{\gamma})(\sigma_{\max}^{2}(W_{1}^{k})+\sigma_{\max}^{2}(W_{2}^{k}))+3\alpha\delta\sigma_{\max}^{2}(X_{\gamma})\sigma_{\max}(W_{2}^{k})$ $\displaystyle~{}~{}~{}~{}+\left(1+3\alpha\sigma_{\max}^{2}(X_{\gamma})(\sigma_{\max}^{2}(W_{1}^{k})+\sigma_{\max}^{2}(W_{2}^{k}))\right)\sqrt{2\sigma_{\max}^{2}(X_{\gamma})({\mathsf{L}}_{\gamma}(W_{1}^{k},W_{2}^{k})-{\mathsf{L}}_{\gamma}^{*})}$ $\displaystyle~{}~{}~{}~{}+3\alpha^{2}\sigma_{\max}^{2}(X_{\gamma})\delta\sigma_{\max}(W_{1}^{k})\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})}$ $\displaystyle~{}~{}~{}~{}+6\alpha^{2}\sigma_{\max}^{4}(X_{\gamma})\sigma_{\max}(W^{k})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})$ $$

which completes the proof.

###### Lemma 20 ( [ 75 , Lemma D.1] ) .

###### Lemma 21 .

###### Lemma 22 .

#### F.5 Proof of Theorem 2

We restate Theorem [2](#Thmtheorem2) in a formal way as follows.

###### Theorem 6 (Almost linear convergence rate) .

Suppose that Assumption [2](#Thmassumption2) holds, letting $\alpha_{1}$ be the smallest positive solution of the following equation of $\alpha$

$$ $\displaystyle 5\alpha\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{0},W_{2}^{0})-\mathsf{L}_{\gamma}^{*})=(1-\exp(-\sqrt{\alpha}))(c_{1}+c_{2}(2-\exp(\sqrt{\alpha})))\sigma_{*}^{2}(X_{\gamma}).$ (57) $$

If the above equation does not have a positive solution, set $\alpha_{1}=\infty$. Similarly, let
$\alpha_{2}$ be the positive solution of $\alpha L=1$ with $L$ defined in ([65](#A6.Ex198)). Moreover, let $\alpha_{3}<(\log(2+\frac{c_{1}}{c_{2}}))^{2}$. Then for any $0<\alpha<\min\{\alpha_{1},\alpha_{2},\alpha_{3}\}$ and $0<\beta\leq\alpha_{2}$, there exists $T={\cal O}(\log\gamma+\log\epsilon^{-1}+K)$ such that $\delta\leq\min\left\{\sqrt{\frac{\alpha\mu^{2}({\mathsf{L}}_{\gamma}(W_{1}^{0},W_{2}^{0})-{\mathsf{L}}_{\gamma}^{*})}{3}},{\cal O}(\epsilon^{1/4})\right\}$ and

$$ $\displaystyle~{}~{}~{}~{}\left(2\alpha\delta\sqrt{c_{2}^{1}\exp(\sqrt{\alpha})}+2\alpha^{2}\delta^{2}+\frac{3\alpha^{2}\delta^{2}c_{2}\exp(\sqrt{\alpha})\sigma_{\max}^{2}(X_{\gamma})}{\mu}\right)K$ $\displaystyle\leq\frac{\alpha c_{2}\exp(\sqrt{\alpha})\sigma_{\max}^{2}(X_{\gamma})}{\mu}\left({\mathsf{L}}_{\gamma}(W_{1}^{0},W_{2}^{0})-{\mathsf{L}}_{\gamma}^{*}\right).$ (58) $$

Therefore, when $\gamma={\cal O}(\epsilon^{-0.5})$, $\mu={\cal O}(\gamma),L={\cal O}(\gamma)$, $\alpha={\cal O}(\gamma^{-1})$, we have $\alpha\mu={\cal O}(1)$ and to achieve the $(\epsilon,\epsilon)$ stationary point, i.e. ${\mathsf{L}}_{\rm val}(W_{1}^{k},W_{2}^{k})-\min_{W_{1},W_{2}}{\mathsf{L}}_{\rm val}(W_{1},W_{2})\leq\epsilon$ and ${\mathsf{L}}_{\rm trn}(W_{1}^{k},W_{2}^{k})-\min_{W_{2}}{\mathsf{L}}_{\rm trn}(W_{1},W_{2})\leq\epsilon$, one need $KT={\cal O}(\log(\epsilon^{-1})^{2})$ iterations.

Proof: Denote $W^{k}=W_{1}^{k}W_{2}^{k}$ and $D_{k}=(W_{1}^{k})^{\top}W_{1}^{k}-(W_{2}^{k})^{\top}W_{2}^{k}$ as the imbalanced matrix. We will prove this theorem by induction. Define the following properties as

- •
$\mathrm{P}_{1}(k)$: $\mathsf{L}_{\gamma}(W_{1}^{k+1},W_{2}^{k+1})-\mathsf{L}_{\gamma}^{*}\leq\left(1-\frac{\alpha\mu}{2}\right)\left({\mathsf{L}}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*}\right)+\frac{3\alpha\delta^{2}}{4}$
- •
$\mathrm{P}_{2}(k)$: $w_{1}\leq\sigma_{\min}(W^{k+1})\leq\sigma_{\max}(W^{k+1})\leq w_{2}$
- •
$\mathrm{P}_{3}(k)$: $\|D_{k+1}-D_{0}\|\leq\frac{5\alpha c_{2}\exp(\sqrt{\alpha})\sigma_{\max}^{2}(X_{\gamma})}{\left(c_{1}+2c_{2}(1-\exp(\sqrt{\alpha}))\right)\sigma_{*}^{2}(X_{\gamma})}\left({\mathsf{L}}_{\gamma}(W_{1}^{0},W_{2}^{0})-{\mathsf{L}}_{\gamma}^{*}\right)$
- •
$\mathrm{P}_{4}(k)$: $c_{1}^{i}+2c_{2}^{i}(1-\exp(\sqrt{\alpha}))\leq\sigma_{\min}^{2}(W_{i}^{k+1})\leq\sigma_{\max}^{2}(W_{i}^{k+1})\leq c_{2}^{i}\exp(\sqrt{\alpha}),~{}~{}i=1,2$

where $c_{1}=c_{1}^{1}+c_{1}^{2},c_{2}=c_{2}^{1}+c_{2}^{2}$. Suppose that the above properties hold at iteration $0,\cdots,k-1$, we aim to prove that $\mathrm{P}_{1}(k),\mathrm{P}_{2}(k),\mathrm{P}_{3}(k),\mathrm{P}_{4}(k)$ hold recursively.

###### Theorem 6 (Almost linear convergence rate) .

##### Step 1: P 1 ​ ( k ) subscript P 1 𝑘 \mathrm{P}_{1}(k) holds.

If we can prove the following uniform bounds of the local PL, smoothness constants and the lower-level error

$$ $\displaystyle\mu_{k}\geq\mu,~{}~{}L_{k}\leq L,~{}~{}\|\delta_{k}\|\leq\delta$ (59) $$

then it holds that

$$ $\displaystyle 1-\alpha\mu_{k}+2\alpha^{2}L_{k}\mu_{k}$ $\displaystyle=1-\mu_{k}(\alpha-2\alpha^{2}L_{k})\qquad\text{ and }\qquad\left(\frac{\alpha}{2}+\alpha^{2}L_{k}\right)\|\delta_{k}\|^{2}\leq\left(\frac{\alpha}{2}+\alpha^{2}L\right)\delta^{2}$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}1-\mu(\alpha-2\alpha^{2}L_{k})\qquad\qquad\qquad\qquad\qquad\qquad\qquad\stackrel{{\scriptstyle(b)}}{{\leq}}\frac{3\alpha\delta^{2}}{4}$ $\displaystyle\stackrel{{\scriptstyle(c)}}{{\leq}}1-\frac{\alpha\mu}{2}$ (60) $$

where (a) comes from ${\alpha\leq\frac{1}{L}\leq\frac{1}{L_{k}}}$, (b) and (c) are due to $\alpha\leq\frac{1}{4L}$ and $L_{k}\leq L$. Then according to Theorem [1](#Thmlemma1), we get the conclusion.

Next, we start to prove ([59](#A6.E59)). According to $\mathrm{P}_{4}(k-1)$, the lower bound of $\mu_{k}$ is earned by

$$ $\displaystyle\mu_{k}$ $\displaystyle=(\sigma_{\min}^{2}(W_{1}^{k})+\sigma_{\min}^{2}(W_{2}^{k}))\sigma_{*}^{2}(X_{\gamma})\geq\left(c_{1}+c_{2}(2-\exp(\sqrt{\alpha}))\right)\sigma_{*}^{2}(X_{\gamma})=:\mu$ (61) $$

where $c_{1}=c_{1}^{1}+c_{1}^{2}$ and $c_{2}=c_{2}^{1}+c_{2}^{2}$. We then prove the upper bound of $\|\delta_{k}\|$. By definition,

$$ $\displaystyle\|\delta_{k}\|$ $\displaystyle=\gamma^{2}\|\nabla_{W_{1}}\mathsf{L}_{\rm trn}(W_{1}^{k},W_{3}^{k+1})-\nabla_{W_{1}}\mathsf{L}_{\rm trn}(W_{1}^{k})\|$ $\displaystyle\leq\gamma^{2}\sigma_{\max}^{2}(X_{\rm trn})\sigma_{\max}^{2}(W_{1}^{k})\operatorname{dist}(W_{3}^{k+1},\mathcal{S}(W_{1}^{k}))$ $\displaystyle\leq 2\gamma^{2}\sigma_{\max}^{4}(X_{\rm trn})\sigma_{\max}^{4}(W_{1}^{k})\left(1-\frac{\sigma_{\min}^{2}(X_{\rm trn})\sigma_{\min}^{2}(W_{1}^{k})}{\sigma_{\max}^{2}(X_{\rm trn})\sigma_{\max}^{2}(W_{1}^{k})}\right)^{T}\left(\mathsf{L}_{\rm trn}(W_{1}^{k},W_{3}^{0})-\mathsf{L}_{\rm trn}^{*}(W_{1}^{k})\right)$ $\displaystyle\leq 2\gamma^{2}\sigma_{\max}^{4}(X_{\rm trn})\sigma_{\max}^{4}(W_{1}^{k})\left(1-\frac{\sigma_{\min}^{2}(X_{\rm trn})\sigma_{\min}^{2}(W_{1}^{k})}{\sigma_{\max}^{2}(X_{\rm trn})\sigma_{\max}^{2}(W_{1}^{k})}\right)^{T}\mathsf{L}_{\rm trn}(W_{1}^{k},W_{3}^{0})$ $$

where the first inequality is due to the Lipschitz continuity of $\mathsf{L}_{\rm trn}(W_{1}^{k},\cdot)$, the second inequality comes from Lemma [7](#Thmlemma7) and Lemma [17](#Thmlemma17). According to $\mathrm{P}_{4}(k-1)$, $\sigma_{\max}(W_{1}^{k}),\sigma_{\min}(W_{1}^{k})$ is upper and lower bounded, so the upper bound of $\|\delta_{k}\|\leq\delta$ exists and it exponentially decreases with $T$.

Finally, we prove the upper bound of $L_{k}$. As ${\mathsf{L}}_{\gamma}(W_{1}^{k},W_{2}^{k})=\widetilde{\mathsf{L}}_{\gamma}(W_{1}^{k},W_{2}^{k})$ when $\sigma_{\min}(W_{i}^{k})>0$, according to the definition, it holds that

$$ $\displaystyle L_{k}$ $\displaystyle\overset{\eqref{Lk_def}}{=\joinrel=}\sigma_{\max}^{2}(X_{\gamma})(\sigma_{\max}^{2}(W_{1}^{k})+\sigma_{\max}^{2}(W_{2}^{k}))+3\alpha\delta\sigma_{\max}^{2}(X_{\gamma})\sigma_{\max}(W_{2}^{k})$ $\displaystyle~{}~{}~{}~{}+\left(1+3\alpha\sigma_{\max}^{2}(X_{\gamma})(\sigma_{\max}^{2}(W_{1}^{k})+\sigma_{\max}^{2}(W_{2}^{k}))\right)\sqrt{2\sigma_{\max}^{2}(X_{\gamma})({\mathsf{L}}_{\gamma}(W_{1}^{k},W_{2}^{k})-{\mathsf{L}}_{\gamma}^{*})}$ $\displaystyle~{}~{}~{}~{}+3\alpha^{2}\sigma_{\max}^{2}(X_{\gamma})\delta\sigma_{\max}(W_{1}^{k})\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})}$ $\displaystyle~{}~{}~{}~{}+6\alpha^{2}\sigma_{\max}^{4}(X_{\gamma})\sigma_{\max}(W^{k})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*}).$ (62) $$

Therefore, to prove the upper bound of $L_{k}$, we need the following upper bounds

$$ $\displaystyle\allowdisplaybreaks\sigma_{\max}^{2}(W_{1}^{k})+\sigma_{\max}^{2}(W_{2}^{k})$ $\displaystyle\overset{\mathrm{P}_{4}(k-1)}{\leq}c_{2}\exp(\sqrt{\alpha})$ $\displaystyle{\mathsf{L}}_{\gamma}(W_{1}^{k},W_{2}^{k})-{\mathsf{L}}_{\gamma}^{*}$ $\displaystyle\overset{\mathrm{P}_{1}(k-1)}{\leq}(1-\frac{\alpha\mu}{2})^{k}({\mathsf{L}}_{\gamma}(W_{1}^{0},W_{2}^{0})-{\mathsf{L}}_{\gamma}^{*})+\frac{3\alpha\delta^{2}}{4}\sum_{\kappa=0}^{k-1}(1-\frac{\alpha\mu}{2})^{\kappa}$ $\displaystyle~{}~{}~{}~{}~{}\leq\left(1-\frac{\alpha\mu}{2}\right)^{k}({\mathsf{L}}_{\gamma}(W_{1}^{0},W_{2}^{0})-{\mathsf{L}}_{\gamma}^{*})+\frac{3\delta^{2}}{2\mu}\mathds{1}_{k\geq 1}$ (63) $\displaystyle~{}~{}~{}~{}~{}\stackrel{{\scriptstyle(a)}}{{\leq}}{\mathsf{L}}_{\gamma}(W_{1}^{0},W_{2}^{0})-{\mathsf{L}}_{\gamma}^{*}$ (64) $\displaystyle\sigma_{\max}^{2}(W_{i}^{k})\overset{\mathrm{P}_{4}(k-1)}{\leq}c_{2}^{i}$ $\displaystyle\exp(\sqrt{\alpha}),~{}~{}~{}~{}~{}\sigma_{\max}(W^{k})\overset{\mathrm{P}_{2}(k-1)}{\leq}w_{2},~{}~{}~{}~{}~{}\|\delta_{k}\|\leq\delta.$ $$

where (a) comes from $\delta\leq\sqrt{\frac{\alpha\mu^{2}({\mathsf{L}}_{\gamma}(W_{1}^{0},W_{2}^{0})-{\mathsf{L}}_{\gamma}^{*})}{3}}$.
Plugging the above upper bounds to ([62](#A6.Ex189)), we get

$$ $\displaystyle L_{k}$ $\displaystyle\leq\sigma_{\max}^{2}(X_{\gamma})c_{2}\exp(\sqrt{\alpha})+3\alpha\delta\sigma_{\max}^{2}(X_{\gamma})\sqrt{c_{2}\exp(\sqrt{\alpha})}$ $\displaystyle~{}~{}~{}~{}+\left(1+3\alpha\sigma_{\max}^{2}(X_{\gamma})c_{2}\exp(\sqrt{\alpha})\right)\sqrt{2\sigma_{\max}^{2}(X_{\gamma})({\mathsf{L}}_{\gamma}(W_{1}^{k},W_{2}^{k})-{\mathsf{L}}_{\gamma}^{*})}$ $\displaystyle~{}~{}~{}~{}+3\alpha^{2}\sigma_{\max}^{2}(X_{\gamma})\delta\sqrt{c_{2}\exp(\sqrt{\alpha})}\sqrt{2\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})}$ $\displaystyle~{}~{}~{}~{}+6\alpha^{2}\sigma_{\max}^{4}(X_{\gamma})w_{2}(\mathsf{L}_{\gamma}(W_{1}^{k},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})=:L.$ (65) $$

After obtaining the bounds for $\mu_{k},\|\delta_{k}\|,L_{k}$, $\mathrm{P}_{1}(k)$ holds because of Theorem [1](#Thmlemma1) and ([60](#A6.Ex181)).

##### Step 2: P 2 ​ ( k ) subscript P 2 𝑘 \mathrm{P}_{2}(k) holds.

Since $(\ell_{\rm val}+\gamma\ell_{\rm trn})(W)$ is $\sigma_{\max}^{2}(X_{\gamma})$-Lipschitz smooth, we have

$$ $\displaystyle(\ell_{\rm val}+\gamma\ell_{\rm trn})(W^{k+1})\leq(\ell_{\rm val}+\gamma\ell_{\rm trn})(W)+\langle\nabla(\ell_{\rm val}+\gamma\ell_{\rm trn})(W),W^{k+1}-W\rangle+\frac{\sigma_{\max}^{2}(X_{\gamma})}{2}\|W^{k+1}-W\|^{2}.$ $$

Setting $W=W^{*}\in\operatorname*{arg\,min}_{W}(\ell_{\rm val}+\gamma\ell_{\rm trn})(W)$ yields

$$ $\displaystyle(\ell_{\rm val}+\gamma\ell_{\rm trn})(W^{k+1})\leq\min_{W}(\ell_{\rm val}+\gamma\ell_{\rm trn})(W)+\frac{\sigma_{\max}^{2}(X_{\gamma})}{2}\|W^{k+1}-W^{*}\|^{2}.$ $$

and thus

$$ $\displaystyle\frac{\sigma_{*}^{2}}{2}(X_{\gamma})\|W^{k+1}-W^{*}\|^{2}$ $\displaystyle\leq(\ell_{\rm val}+\gamma\ell_{\rm trn})(W^{k+1})-\min_{W}(\ell_{\rm val}+\gamma\ell_{\rm trn})(W)$ $\displaystyle=\widetilde{\mathsf{L}}_{\gamma}(W_{1}^{k+1},W_{2}^{k+1})-\min_{W_{1},W_{2}}\widetilde{\mathsf{L}}_{\gamma}(W_{1},W_{2})$ $\displaystyle={\mathsf{L}}_{\gamma}(W_{1}^{k+1},W_{2}^{k+1})-{\mathsf{L}}_{\gamma}^{*}.$ $$

Also, $(\ell_{\rm val}+\gamma\ell_{\rm trn})(W)$ is $\sigma_{*}^{2}(X_{\gamma})$-PL, so the quadratic growth condition with holds by Lemma [6](#Thmlemma6)

$$ $\displaystyle 2\sigma_{*}^{2}(X_{\gamma})\|W^{k+1}-W^{*}\|^{2}$ $\displaystyle\leq(\ell_{\rm val}+\gamma\ell_{\rm trn})(W^{k+1})-\min_{W}(\ell_{\rm val}+\gamma\ell_{\rm trn})(W)={\mathsf{L}}_{\gamma}(W_{1}^{k+1},W_{2}^{k+1})-{\mathsf{L}}_{\gamma}^{*}$ $$

where $W^{*}=\operatorname*{arg\,min}_{W\in\operatorname*{arg\,min}_{W}(\ell_{\rm val}+\gamma\ell_{\rm trn})(W)}\|W^{k+1}-W\|^{2}$.
As a result, we have

$$ $\displaystyle\sigma_{\max}(W^{k+1})$ $\displaystyle=\sigma_{\max}\left(W^{k+1}-W^{*}+W^{*}\right)$ $\displaystyle\leq\sigma_{\max}\left(W^{*}\right)+\|W^{k+1}-W^{*}\|_{2}$ $\displaystyle\leq\sigma_{\max}\left(W^{*}\right)+\|W^{k+1}-W^{*}\|$ $\displaystyle\leq\sigma_{\max}\left(W^{*}\right)+\sqrt{\frac{1}{2\sigma_{*}^{2}(X_{\gamma})}({\mathsf{L}}_{\gamma}(W_{1}^{k+1},W_{2}^{k+1})-{\mathsf{L}}_{\gamma}^{*})}$ $\displaystyle\overset{\eqref{bound_general}}{\leq}\sigma_{\max}\left(W^{*}\right)+\sqrt{\frac{1}{2\sigma_{*}^{2}(X_{\gamma})}({\mathsf{L}}_{\gamma}(W_{1}^{0},W_{2}^{0})-{\mathsf{L}}_{\gamma}^{*})}=:w_{2}$ $$

where the first inequality is derived from the Weyl’s inequality.
Similarly, the lower bound of singular value is achieved by the Weyl’s inequality

$$ $\displaystyle\sigma_{\min}(W^{k+1})$ $\displaystyle=\sigma_{\min}\left(W^{k+1}-W^{*}+W^{*}\right)$ $\displaystyle\geq\sigma_{\min}\left(W^{*}\right)-\|W^{k+1}-W^{*}\|_{2}$ $\displaystyle\geq\sigma_{\min}\left(W^{*}\right)-\|W^{k+1}-W^{*}\|$ $\displaystyle\geq\sigma_{\min}\left(W^{*}\right)-\sqrt{\frac{1}{2\sigma_{*}^{2}(X_{\gamma})}({\mathsf{L}}_{\gamma}(W_{1}^{0},W_{2}^{0})-{\mathsf{L}}_{\gamma}^{*})}.$ $$

As the singular value is always nonnegative, we can define a lower bound

$$ $\displaystyle w_{1}:=\left[\sigma_{\min}\left(W^{*}\right)-\sqrt{\frac{1}{2\sigma_{*}^{2}(X_{\gamma})}({\mathsf{L}}_{\gamma}(W_{1}^{0},W_{2}^{0})-{\mathsf{L}}_{\gamma}^{*})}\right]_{+}$ (66) $$

which is strict positive when initializing $W^{0}_{1},W^{0}_{2}$ close to the optimal.

##### Step 3: P 3 ​ ( k ) subscript P 3 𝑘 \mathrm{P}_{3}(k) holds.

Denoting $W^{k}=W_{1}^{k}W_{2}^{k},\ell_{\gamma}=\ell_{\rm val}+\gamma\ell_{\rm trn}$ and utilizing the PBGD update

$$ $\displaystyle W_{1}^{k+1}=W_{1}^{k}-\alpha(\nabla\ell_{\gamma}(W^{k})(W_{2}^{k})^{\top}+\delta_{k})\quad\text{ and }\quad W_{2}^{k+1}=W_{2}^{k}-\alpha(W_{1}^{k})^{\top}\nabla\ell_{\gamma}(W^{k})$ (67) $$

we can expand the difference of imbalance matrix as follows

$$ $\displaystyle D_{k+1}-D_{k}$ $\displaystyle=W_{1}^{k+1,\top}W_{1}^{k+1}-W_{2}^{k+1,\top}W_{2}^{k+1}-(W_{1}^{k})^{\top}W_{1}^{k}+(W_{2}^{k})^{\top}W_{2}^{k}$ $\displaystyle=\left(W_{1}^{k}-\alpha(\nabla\ell_{\gamma}(W^{k})(W_{2}^{k})^{\top}+\delta_{k})\right)^{\top}\left(W_{1}^{k}-\alpha(\nabla\ell_{\gamma}(W^{k})(W_{2}^{k})^{\top}+\delta_{k})\right)$ $\displaystyle~{}~{}~{}~{}-\left(W_{2}^{k}-\alpha(W_{1}^{k})^{\top}\nabla\ell_{\gamma}(W^{k})\right)^{\top}\left(W_{2}^{k}-\alpha(W_{1}^{k})^{\top}\nabla\ell_{\gamma}(W^{k})\right)-(W_{1}^{k})^{\top}W_{1}^{k}+(W_{2}^{k})^{\top}W_{2}^{k}$ $\displaystyle=-2\alpha(W_{1}^{k})^{\top}(\nabla\ell_{\gamma}(W^{k})(W_{2}^{k})^{\top}+\delta_{k})+\alpha^{2}(\nabla\ell_{\gamma}(W^{k})(W_{2}^{k})^{\top}+\delta_{k})^{\top}(\nabla\ell_{\gamma}(W^{k})(W_{2}^{k})^{\top}+\delta_{k})$ $\displaystyle~{}~{}~{}~{}+2\alpha\left((W_{1}^{k})^{\top}\nabla\ell_{\gamma}(W^{k})\right)^{\top}W_{2}^{k}-\alpha^{2}\left((W_{1}^{k})^{\top}\nabla\ell_{\gamma}(W^{k})\right)^{\top}\left((W_{1}^{k})^{\top}\nabla\ell_{\gamma}(W^{k})\right)$ $\displaystyle=-2\alpha(W_{1}^{k})^{\top}\delta_{k}+\alpha^{2}(\nabla\ell_{\gamma}(W^{k})(W_{2}^{k})^{\top}+\delta_{k})^{\top}(\nabla\ell_{\gamma}(W^{k})(W_{2}^{k})^{\top}+\delta_{k})$ $\displaystyle~{}~{}~{}~{}-\alpha^{2}\left((W_{1}^{k})^{\top}\nabla\ell_{\gamma}(W^{k})\right)^{\top}\left((W_{1}^{k})^{\top}\nabla\ell_{\gamma}(W^{k})\right)$ $$

Taking the norm of both sides yields

$$ $\displaystyle\|D_{k+1}-D_{k}\|$ $\displaystyle\leq 2\alpha\delta\sigma_{\max}(W_{1}^{k})+\alpha^{2}\|\nabla\ell_{\gamma}(W^{k})(W_{2}^{k})^{\top}+\delta_{k}\|^{2}+\alpha^{2}\|(W_{1}^{k})^{\top}\nabla\ell_{\gamma}(W^{k})\|^{2}$ $\displaystyle\leq 2\alpha\delta\sigma_{\max}(W_{1}^{k})+2\alpha^{2}(\sigma_{\max}^{2}(W_{2}^{k})\|\nabla\ell_{\gamma}(W^{k})\|^{2}+\delta^{2})+\alpha^{2}\sigma_{\max}^{2}(W_{1}^{k})\|\nabla\ell_{\gamma}(W^{k})\|^{2}$ $\displaystyle=2\alpha\delta\sigma_{\max}(W_{1}^{k})+2\alpha^{2}\delta^{2}+\alpha^{2}(2\sigma_{\max}^{2}(W_{2}^{k})+\sigma_{\max}^{2}(W_{1}^{k}))\|\nabla\ell_{\gamma}(W^{k})\|^{2}$ $\displaystyle\leq 2\alpha\delta\sqrt{c_{2}^{1}\exp(\sqrt{\alpha})}+2\alpha^{2}\delta^{2}+2\alpha^{2}c_{2}\exp(\sqrt{\alpha})\sigma_{\max}^{2}(X_{\gamma})(\mathsf{L}_{\gamma}(W^{k}_{1},W_{2}^{k})-\mathsf{L}_{\gamma}^{*})$ $\displaystyle\stackrel{{\scriptstyle\eqref{bound_induction}}}{{\leq}}2\alpha\delta\sqrt{c_{2}^{1}\exp(\sqrt{\alpha})}+2\alpha^{2}\delta^{2}$ $\displaystyle~{}~{}~{}~{}+2\alpha^{2}c_{2}\exp(\sqrt{\alpha})\sigma_{\max}^{2}(X_{\gamma})\left(\left(1-\frac{\alpha\mu}{2}\right)^{k}({\mathsf{L}}_{\gamma}(W_{1}^{0},W_{2}^{0})-{\mathsf{L}}_{\gamma}^{*})+\frac{3\delta^{2}}{2\mu}\right)$ (68) $$

where the third inequality is due to $\mathrm{P}_{4}(k-1)$ and $\ell_{\gamma}$ is $\sigma_{\max}^{2}(X_{\gamma})$ Lipschitz smooth.

As a result, it follows

$$ $\displaystyle\|D_{k+1}-D_{0}\|$ $\displaystyle\leq\sum_{\kappa=0}^{k}\|D_{\kappa+1}-D_{\kappa}\|$ $\displaystyle\overset{\eqref{D_k+1-k}}{\leq}\sum_{\kappa=0}^{k}2\alpha\delta\sqrt{c_{2}^{1}\exp(\sqrt{\alpha})}+2\alpha^{2}\delta^{2}$ $\displaystyle~{}~{}~{}~{}+\sum_{\kappa=0}^{k}2\alpha^{2}c_{2}\exp(\sqrt{\alpha})\sigma_{\max}^{2}(X_{\gamma})\left(\left(1-\frac{\alpha\mu}{2}\right)^{k}({\mathsf{L}}_{\gamma}(W_{1}^{0},W_{2}^{0})-{\mathsf{L}}_{\gamma}^{*})+\frac{3\delta^{2}}{2\mu}\right)$ $\displaystyle\leq\left(2\alpha\delta\sqrt{c_{2}^{1}\exp(\sqrt{\alpha})}+2\alpha^{2}\delta^{2}+\frac{3\alpha^{2}\delta^{2}c_{2}\exp(\sqrt{\alpha})\sigma_{\max}^{2}(X_{\gamma})}{\mu}\right)K$ $\displaystyle~{}~{}~{}~{}+\frac{4\alpha c_{2}\exp(\sqrt{\alpha})\sigma_{\max}^{2}(X_{\gamma})}{\mu}\left({\mathsf{L}}_{\gamma}(W_{1}^{0},W_{2}^{0})-{\mathsf{L}}_{\gamma}^{*}\right)$ $\displaystyle\leq\frac{5\alpha c_{2}\exp(\sqrt{\alpha})\sigma_{\max}^{2}(X_{\gamma})}{\left(c_{1}+c_{2}(2-\exp(\sqrt{\alpha}))\right)\sigma_{*}^{2}(X_{\gamma})}\left({\mathsf{L}}_{\gamma}(W_{1}^{0},W_{2}^{0})-{\mathsf{L}}_{\gamma}^{*}\right)$ $$

where the last inequality holds by ([58](#A6.Ex178)) and ([61](#A6.E61)).

##### Step 4: P 4 ​ ( k ) subscript P 4 𝑘 \mathrm{P}_{4}(k) holds.

According to [, Appendix C], the initial weights can be bounded by

$$ $\displaystyle c_{1}^{1}\leq\sigma_{\min}(W_{1}^{0})\leq\sigma_{\max}(W_{1}^{0})\leq c_{2}^{1}$ $\displaystyle c_{1}^{2}\leq\sigma_{\min}(W_{2}^{0})\leq\sigma_{\max}(W_{2}^{0})\leq c_{2}^{2}$ $$

and the singular values of weight matrix can be bounded by the imbalance matrix

$$ $\displaystyle b_{l}^{1}=c_{1}^{1}-2\|D_{k+1}-D_{0}\|\leq\sigma_{\min}(W_{1}^{k+1})\leq\sigma_{\max}(W_{1}^{k+1})\leq c_{2}^{1}+\|D_{k+1}-D_{0}\|=b_{u}^{1}$ $\displaystyle b_{l}^{2}=c_{1}^{2}-2\|D_{k+1}-D_{0}\|\leq\sigma_{\min}(W_{2}^{k+1})\leq\sigma_{\max}(W_{2}^{k+1})\leq c_{2}^{2}+\|D_{k+1}-D_{0}\|=b_{u}^{2}.$ $$

Moreover, we have

$$ $\displaystyle b_{u}^{i}$ $\displaystyle=c_{2}^{i}+\|D_{k+1}-D_{0}\|$ $\displaystyle\leq c_{2}^{i}+\frac{5\alpha c_{2}\exp(\sqrt{\alpha})\sigma_{\max}^{2}(X_{\gamma})}{\left(c_{1}+2c_{2}(1-\exp(\sqrt{\alpha}))\right)\sigma_{*}^{2}(X_{\gamma})}\left({\mathsf{L}}_{\gamma}(W_{1}^{0},W_{2}^{0})-{\mathsf{L}}_{\gamma}^{*}\right)$ $\displaystyle\leq c_{2}^{i}+(1-\exp(-\sqrt{\alpha}))\left(c_{1}+c_{2}(2-\exp(\sqrt{\alpha}))\right)\sigma_{*}^{2}(X_{\gamma})\times\frac{c_{2}\exp(\sqrt{\alpha})}{\left(c_{1}+c_{2}(2-\exp(\sqrt{\alpha}))\right)\sigma_{*}^{2}(X_{\gamma})}$ $\displaystyle\leq\exp(\sqrt{\alpha})c_{2}^{i}$ $$

where the first inequality holds due to $\mathbb{P}_{3}(k)$, and the second inequality holds because of the condition ([57](#A6.E57)).
Since $b_{l}^{i}+2b_{u}^{i}=c_{1}^{i}+2c_{2}^{1}$ and $b_{u}^{i}\leq\exp(\sqrt{\alpha})c_{2}^{i}$, $\mathrm{P}_{4}(k)$ holds because

$$ $\displaystyle c_{1}^{i}+2c_{2}^{i}(1-\exp(\sqrt{\alpha}))\leq b_{l}^{1}\leq\sigma_{\min}(W_{i}^{k+1})\leq\sigma_{\max}(W_{i}^{k+1})\leq b_{u}^{i}\leq\exp(\sqrt{\alpha})c_{2}^{i}.$ $$

Therefore, the iterates of PBGD satisfy

$$ $\displaystyle{\mathsf{L}}_{\gamma}(W_{1}^{k},W_{2}^{k})-{\mathsf{L}}_{\gamma}^{*}$ $\displaystyle\leq\left(1-\frac{\alpha\mu}{2}\right)^{k}({\mathsf{L}}_{\gamma}(W_{1}^{0},W_{2}^{0})-{\mathsf{L}}_{\gamma}^{*})+{\cal O}\left(\frac{\epsilon^{0.5}}{\mu}\right).$ $$

Together with Theorem [1](#Thmtheorem1) and Theorem [4](#Thmtheorem4), we arrive at the conclusion.

### Appendix G Proof for Data Hyper-cleaning

We provide the omitted proof of lemmas and theorems for data hyper-cleaning.

#### G.1 Independence of the lower-level solution set 𝒮 ​ ( u ) 𝒮 𝑢 \mathcal{S}(u)

###### Lemma 23 .

If $X_{\rm trn}X_{\rm trn}^{\dagger}$ is a diagonal matrix, then for any $u$, $\mathcal{S}(u)$ that is independent of $u$ and thus

$$ $\!\!\ell_{\rm trn}^{*}(u)=\frac{1}{2}\left\|\sqrt{\psi_{N}(u)}\left(I-X_{\rm trn}X_{\rm trn}^{\dagger}\right)Y_{\rm trn}\right\|^{2}=\frac{1}{2}\sum_{i=1}^{N}\psi(u_{i})\|y_{i}\|^{2}\mathds{1}\left([X_{\rm trn}X_{\rm trn}^{\dagger}]_{ii}\neq 1\right).$ (69) $$

Proof:
According to [, Theorem 6.1], one of the lower-level solutions $W^{*}$ of ([11](#S6.E11)) should satisfy

$$ $\displaystyle W^{*}$ $\displaystyle=(\sqrt{\psi_{N}(u)}X_{{\rm trn}})^{\dagger}\sqrt{\psi_{N}(u)}Y_{{\rm trn}}$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}X_{{\rm trn}}^{\dagger}\sqrt{\psi_{N}(u)}^{-1}\sqrt{\psi_{N}(u)}Y_{{\rm trn}}$ $\displaystyle=(X_{{\rm trn}})^{\dagger}Y_{{\rm trn}}$ (70) $$

where (a) holds because $\sqrt{\psi_{N}(u)}$ is invertiable for any $u\in\mathcal{U}$ and Lemma [10](#Thmlemma10). Therefore,

$$ $\displaystyle\mathsf{L}_{\rm trn}^{*}(u)$ $\displaystyle=\frac{1}{2}\left\|\sqrt{\psi_{N}(u)}\left(I-X_{\rm trn}X_{\rm trn}^{\dagger}\right)Y_{\rm trn}\right\|^{2}=\frac{1}{2}\sum_{i=1}^{N}\psi(u_{i})\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{ii}\neq 1)$ $$

where $[X_{\rm trn}X_{\rm trn}^{\dagger}]_{ii}$ denotes the element of matrix $[X_{\rm trn}X_{\rm trn}^{\dagger}]$ at the position $(i,i)$.

###### Remark 3 .

Lemma [23](#Thmlemma23) suggests that the $(\epsilon,0)$ solution to the bilevel problem is meaningless since the static lower-level solution set makes the bilevel objective in ([11](#S6.E11)) independent of $u$. This is also aligned with the observation that the model achieved by data reweighting is independent of data weight when the training dataset is linear independent [, Theorem 1]. However, the $\epsilon$-lower-level solution relies on $u$ (the rationale is detailed below). This could also explain the reason why solving bilevel problem by nested approach to the $(\epsilon,0)$ stationary point has accuracy drop over the $(\epsilon,\epsilon)$ stationary point attained by penalized method in the data hyper-cleaning task [].

Explanation of Remark [3](#Thmremark3).
The gradient of lower-level objective can be computed as

$$ $\displaystyle\nabla_{W}\ell_{\rm trn}(u,W)$ $\displaystyle=-\frac{1}{2}(\sqrt{\psi_{N}(u)}X_{\rm trn})^{\top}\sqrt{\psi_{N}(u)}(Y_{\rm trn}-X_{\rm trn}W)$ $\displaystyle=-\frac{1}{2}X_{\rm trn}^{\top}\psi_{N}(u)(Y_{\rm trn}-X_{\rm trn}W)$ $\displaystyle=\frac{1}{2}\sum_{i=1}^{N}\psi(u_{i})x_{i}(x_{i}^{\top}W-y_{i}^{\top}).$ (71) $$

When the weight of $i$-th sample $\psi(u_{i})$ is close to 0 0, $i$-th sample minimally influences the lower-level optimization. Ideally, the optimal response weight which fits the dataset excluding the
$i$-th sample, will also perform approximately well on the weighted objective.

To see it,
let $u^{i}=(\bar{u},\cdots,\bar{u},-\bar{u},\bar{u},\cdots,\bar{u})$ with $-\bar{u}$ at the $i$-th position and $\bar{u}$ elsewhere. We can then choose the minimal norm solution $W(u^{i})\in\operatorname*{arg\,min}_{W}\frac{1}{2}\sum_{j\neq i}\|x_{j}^{\top}W-y_{j}^{\top}\|^{2}$ such that $\sum_{j\neq i}x_{j}(x_{j}^{\top}W-y_{j}^{\top})=0$. According to Lemma [23](#Thmlemma23), $W(u^{i})\in\operatorname*{arg\,min}_{W}\frac{1}{2}\sum_{j\neq i}\|x_{j}^{\top}W-y_{j}^{\top}\|$ is also solution to $\frac{1}{2}\sum_{j\neq i}\psi(u_{j})\|x_{j}^{\top}W-y_{j}^{\top}\|^{2}$ so that $\sum_{j\neq i}\psi(u_{j})x_{j}(x_{j}^{\top}W-y_{j}^{\top})=0$. By ([71](#A7.Ex248)),

$$ $\displaystyle\nabla_{W}\ell_{\rm trn}(u^{i},W(u^{i}))=\frac{\psi(-\bar{u})}{2}x_{i}(x_{i}^{\top}W(u^{i})-y_{i}^{\top})$ $$

holds for any $i$. Taking the norm yields

$$ $\displaystyle\|\nabla_{W}\ell_{\rm trn}(u^{i},W(u^{i}))\|$ $\displaystyle=\frac{\psi(-\bar{u})}{2}\|x_{i}(x_{i}^{\top}W(u^{i})-y_{i}^{\top})\|$ $$

which is very small because $\psi(-\bar{u})$ is close to 0 0 and $\|x_{i}(x_{i}^{\top}W(u^{i})-y_{i}^{\top})\|$ is bounded. According to the PL property of the training loss, we know the function value gap $\ell_{\rm trn}(u^{i},W(u^{i}))-\ell_{\rm trn}^{*}(u^{i})$ is also small enough, suggesting that $W(u^{i})$ is an $\epsilon$-solution of lower-level for some $\epsilon$.

###### Lemma 23 .

###### Remark 3 .

#### G.2 Proof of Lemma 2 : Local blockwise PL and smoothness

We first restate Lemma [2](#Thmlemma2) in a formal way as follows.

###### Lemma 24 (Local blockwise PL and smoothness of $\ell_{\gamma}(u,W)$ ) .

If $X_{\rm trn}X_{\rm trn}^{\dagger}$ is a diagonal matrix, then for any $u\in\mathcal{U}$ and any $W$, the penalized objective $\ell_{\gamma}(u,W)$ is $L_{w}^{\gamma}$-smooth and $\mu_{w}^{\gamma}$-PL over $W$, where the constants are defined as

$$ $\displaystyle\mu_{w}^{\gamma}:=\sigma_{*}^{2}\left(X_{\rm val}^{\top}X_{\rm val}+\gamma(1-\psi(\bar{u}))X_{\rm trn}^{\top}X_{\rm trn}\right),~{}~{}~{}~{}L_{w}^{\gamma}:=\sigma_{\max}^{2}\left(X_{\rm val}^{\top}X_{\rm val}+\gamma\psi(\bar{u})X_{\rm trn}^{\top}X_{\rm trn}\right).$ $$

Similarly, the lower-level objective $\ell_{\rm trn}(u,W)$ is $L_{w}$-smooth and $\mu_{w}$-PL over $W$, with the constants

$$ $\displaystyle\mu_{w}:=\sigma_{*}^{2}\left((1-\psi(\bar{u}))X_{\rm trn}^{\top}X_{\rm trn}\right),~{}~{}~{}L_{w}:=\sigma_{\max}^{2}\left((1-\psi(\bar{u}))X_{\rm trn}^{\top}X_{\rm trn}\right).$ $$

Moreover, $\ell_{\gamma}(u,W)$ is $\gamma\ell_{\rm trn}(W)$ smooth and $\frac{\gamma c(W)\psi(\bar{u})(1-\psi(\bar{u}))^{2}}{4}$-PL over $u\in\mathcal{U}$, where we define

$$ $\displaystyle c(W)=\min_{i}\left\{\|y_{i}^{\top}-x_{i}^{\top}W\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right\}_{>0}$ $$

as the lower bound of the positive mismatch in the training loss. If there is no positive mismatch, we can set $c(W)$ to any positive number.

Proof: We first show that Sigmoid function $\psi(u_{i})$ is $\psi(\bar{u})(1-\psi(\bar{u}))^{2}$ PL and $1$ smooth over $u_{i}$, where $u_{i}$ is the $i$-th element in $u$. According to the definition, the gradient of the Sigmoid function can be computed as $\nabla\psi(u_{i})=\psi(u_{i})(1-\psi(u_{i}))$, and its Hessian has the form of

$$ $\displaystyle\nabla^{2}\psi(u_{i})$ $\displaystyle=\psi(u_{i})(1-\psi(u_{i}))^{2}+\psi(u_{i})(-\psi(u_{i})(1-\psi(u_{i})))$ $\displaystyle=\psi(u_{i})(1-\psi(u_{i}))^{2}-\psi(u_{i})^{2}(1-\psi(u_{i}))$ $\displaystyle=\psi(u_{i})(1-\psi(u_{i}))(1-2\psi(u_{i}))\leq 1$ (72) $$

which validates the $1$ smoothness. On the other hand,

$$ $\displaystyle\|\nabla\psi(u_{i})\|^{2}$ $\displaystyle=\psi(u_{i})^{2}(1-\psi(u_{i}))^{2}$ $\displaystyle\geq\{\min_{-\bar{u}\leq u_{i}\leq\bar{u}}\psi(u_{i})(1-\psi(u_{i}))^{2}\}\psi(u_{i})$ $\displaystyle\geq\min\{\psi(-\bar{u})(1-\psi(-\bar{u}))^{2},\psi(\bar{u})(1-\psi(\bar{u}))^{2}\}\psi(u_{i})$ $\displaystyle\geq\min\{\psi(\bar{u})^{2}(1-\psi(\bar{u})),\psi(\bar{u})(1-\psi(\bar{u}))^{2}\}\psi(u_{i})$ $\displaystyle\geq\psi(\bar{u})(1-\psi(\bar{u}))^{2}\psi(u_{i})$ (73) $\displaystyle\geq\psi(\bar{u})(1-\psi(\bar{u}))^{2}(\psi(u_{i})-\min_{u_{i}}\psi(u_{i}))$ $$

where the third inequality holds since $\psi(-\bar{u})=1-\psi(\bar{u})$, the fourth inequality comes from $\bar{u}>0$ so that $\psi(\bar{u})>1-\psi(\bar{u})$, and the last inequality follows from $\min_{u_{i}}\psi(u_{i})=0$. This proves that $\psi(u_{i})$ is $\psi(\bar{u})(1-\psi(\bar{u}))^{2}$ PL when $u_{i}\in[-\bar{u},\bar{u}]$.

Recall that the penalized objective can be written as

$$ $\displaystyle\ell_{\gamma}(u,W)=\ell_{\rm val}(W)+\frac{\gamma}{2}\sum_{i=1}^{N}\psi(u_{i})\left[\|y_{i}^{\top}-x_{i}^{\top}W\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right].$ $$

Therefore, the gradient of $\mathsf{L}_{\gamma}(u,W)$ satisfies

$$ $\displaystyle\|\nabla_{u}\ell_{\gamma}(u,W)\|^{2}$ $\displaystyle=\sum_{i=1}^{N}\|\nabla_{u_{i}}\ell_{\gamma}(u,W)\|^{2}$ $\displaystyle=\sum_{i=1}^{N}\left\|\frac{\gamma}{2}\nabla\psi(u_{i})\left[\|y_{i}^{\top}-x_{i}^{\top}W\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right]\right\|^{2}$ $\displaystyle=\frac{\gamma^{2}}{4}\sum_{i=1}^{N}\nabla\psi(u_{i})^{2}\left[\|y_{i}^{\top}-x_{i}^{\top}W\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right]^{2}$ $\displaystyle\stackrel{{\scriptstyle\eqref{1234}}}{{\geq}}\frac{\gamma^{2}}{4}\sum_{i=1}^{N}\psi(\bar{u})(1-\psi(\bar{u}))^{2}\psi(u_{i})\left[\|y_{i}^{\top}-x_{i}^{\top}W\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right]^{2}$ $\displaystyle\geq\frac{\gamma^{2}c(W)\psi(\bar{u})(1-\psi(\bar{u}))^{2}}{4}\sum_{i=1}^{N}\psi(u_{i})\left[\|y_{i}^{\top}-x_{i}^{\top}W\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right]$ $\displaystyle=\frac{\gamma c(W)\psi(\bar{u})(1-\psi(\bar{u}))^{2}}{2}(\ell_{\gamma}(u,W)-\mathsf{L}_{\rm val}(W))$ $\displaystyle\geq\frac{\gamma c(W)\psi(\bar{u})(1-\psi(\bar{u}))^{2}}{2}(\ell_{\gamma}(u,W)-\min_{u}\ell_{\gamma}(u,W))$ (74) $$

where the last inequality holds from $\min_{u}\mathsf{L}_{\gamma}(u,W)=\mathsf{L}_{\rm val}(W)$. This shows $\ell_{\gamma}(u,W)$ is $\frac{\gamma c(W)\psi(\bar{u})(1-\psi(\bar{u}))^{2}}{4}$ PL over $u$. The Hessian of $\ell_{\gamma}(u,W)$ can be calculated by

$$ $\displaystyle\|\nabla^{2}_{u}\ell_{\gamma}(u,W)\|$ $\displaystyle=\sum_{i=1}^{N}\left\|\frac{\gamma}{2}\nabla^{2}\psi(u_{i})\left[\|y_{i}^{\top}-x_{i}^{\top}W\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right]\right\|$ $\displaystyle\leq\frac{\gamma}{2}\sum_{i=1}^{N}\|\nabla^{2}\psi(u_{i})\|\|y_{i}^{\top}-x_{i}^{\top}W\|^{2}$ $\displaystyle\leq\frac{\gamma}{2}\sum_{i=1}^{N}\|y_{i}^{\top}-x_{i}^{\top}W\|^{2}=\gamma\ell_{\rm trn}(W)$ (75) $$

which indicates $\ell_{\gamma}(u,W)$ is $\gamma\ell_{\rm trn}(W)$ smooth over $u$.

For the property over $W$, we can use the matrix form of the objective and define

$$ $\displaystyle\widetilde{\ell}_{u}(W):=\frac{1}{2}\left\|Y_{\rm val}-X_{\rm val}W\right\|^{2}+\frac{\gamma}{2}\left\|\sqrt{\psi_{N}(u)}\left(Y_{\rm trn}-X_{\rm trn}W\right)\right\|^{2}=\frac{1}{2}\left\|Y_{\gamma}(u)-X_{\gamma}(u)W\right\|^{2}$ $$

where $X_{\gamma}(u)=[X_{\rm val};\sqrt{\gamma\psi_{N}(u)}X_{\rm trn}]$ and $Y_{\gamma}(u)=[Y_{\rm val};\sqrt{\gamma\psi_{N}(u)}Y_{\rm trn}]$.

Then according to [, Appendix B], $\widetilde{\ell}_{u}(W)$ is $\sigma_{\max}^{2}(X_{\gamma}(u))$ smooth and $\sigma_{*}^{2}(X_{\gamma}(u))$ PL. Since

$$ $\ell_{\gamma}(u,W)=\widetilde{\ell}_{u}(W)-h(u)~{}~{}\text{with}~{}~{}h(u)=-\frac{\gamma}{2}\sum_{i=1}^{N}\psi(u_{i})\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)$ $$

where $h(u)$ is independent of $W$, we have $\ell_{\gamma}(u,W)$ is $\sigma_{\max}^{2}(X_{\gamma}(u))$ smooth over $W$ and

$$ $\displaystyle\|\nabla_{W}\ell_{\gamma}(u,W)\|^{2}=\|\nabla\widetilde{\ell}_{u}(W)\|^{2}$ $\displaystyle\geq 2\sigma_{*}^{2}(X_{\gamma}(u))\left(\widetilde{\ell}_{u}(W)-\min_{W}\widetilde{\ell}_{u}(W)\right)$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}2\sigma_{*}^{2}(X_{\gamma}(u))\left({\ell}_{\gamma}(u,W)-\min_{W}{\ell}_{\gamma}(u,W)\right)$ (76) $$

indicating that $\ell_{\gamma}(u,W)$ is $\sigma_{*}^{2}(X_{\gamma}(u))$ PL over $W$. On the other hand, we have

$$ $\displaystyle X_{\gamma}(u)^{\top}X_{\gamma}(u)$ $\displaystyle=[X_{\rm val}^{\top}~{}~{}X_{\rm trn}^{\top}\sqrt{\gamma\psi_{N}(u)}]\left[\begin{array}[]{l}X_{\rm val}\\ \sqrt{\gamma\psi_{N}(u)}X_{\rm trn}\end{array}\right]$ $\displaystyle=X_{\rm val}^{\top}X_{\rm val}+\gamma X_{\rm trn}^{\top}\psi_{N}(u)X_{\rm trn}.$ $$

As $\sigma(\cdot)$ is strictly increasing and $\psi(-u)=1-\psi(u)$, we have $(1-\psi(\bar{u}))I\preccurlyeq\psi_{N}(u)\preccurlyeq\psi(\bar{u})I$ and

$$ $\displaystyle X_{\rm val}^{\top}X_{\rm val}+\gamma(1-\psi(\bar{u}))X_{\rm trn}^{\top}X_{\rm trn}\preccurlyeq X_{\gamma}(u)^{\top}X_{\gamma}(u)\preccurlyeq X_{\rm val}^{\top}X_{\rm val}+\gamma\psi(\bar{u})X_{\rm trn}^{\top}X_{\rm trn}.$ $$

Moreover, the definition of $X_{\gamma}(u)$ suggests that it is of constant rank when $u\in\mathcal{U}$.
In this way, the singular value of the data matrix will be bounded by

$$ $\displaystyle\sigma_{*}^{2}(X_{\gamma}(u))\geq\sigma_{*}\left(X_{\rm val}^{\top}X_{\rm val}+\gamma(1-\psi(\bar{u}))X_{\rm trn}^{\top}X_{\rm trn}\right)=:\mu_{w}^{\gamma}$ (77) $\displaystyle\sigma_{\max}^{2}(X_{\gamma}(u))\leq\sigma_{\max}\left(X_{\rm val}^{\top}X_{\rm val}+\gamma\psi(\bar{u})X_{\rm trn}^{\top}X_{\rm trn}\right)=:L_{w}^{\gamma}$ (78) $$

which means $\ell_{\gamma}(u,W)$ is uniformly smooth over $W$ with $L_{w}$ and is uniformly PL over $W$ with $\mu_{w}$.
Similarly, the lower-level objective $\ell_{\rm trn}(u,W)$ is uniformly smooth over $W$ with $L_{w}$ and PL over $W$ with $\mu_{w}$ which are defined as

$$ $\displaystyle\mu_{w}:=(1-\psi(\bar{u}))\sigma_{*}\left(X_{\rm trn}^{\top}X_{\rm trn}\right),~{}~{}~{}L_{w}:=\sigma_{\max}\left(X_{\rm trn}^{\top}X_{\rm trn}\right).$ $$

###### Lemma 24 (Local blockwise PL and smoothness of ℓ γ ​ ( u , W ) subscript ℓ 𝛾 𝑢 𝑊 \ell_{\gamma}(u,W) ) .

#### G.3 Lipschitz continuity of solution to the penalized problem

###### Lemma 25 .

There exists $W^{*}_{\gamma}(u)\in\operatorname*{arg\,min}\ell_{\gamma}(u,W)$ such that $W^{*}_{\gamma}(u)$ is $L_{wu}^{*}$ Lipschitz continuous over $u$ where $L_{wu}^{*}={\cal O}(1)$. Moreover, $\|W_{\gamma}^{*}(u)\|\leq L_{w}^{*}={\cal O}(1)$.

Proof: This result can be deduced from a general result under the PL condition and the smoothness of
$\ell_{\gamma}(u,\cdot)$, as demonstrated by [, Lemma A.3]. Given that $\ell_{\gamma}(u,\cdot)$ is a squared loss composite with a linear mapping —- a specific case of a PL function – we aim to separately derive the Lipschitz continuity of the solution set for clarity and simplicity.

Let $W^{*}_{\gamma}(u)=X_{\gamma}(u)^{\dagger}Y_{\gamma}(u)$ be the minimal norm solution of $\min_{W}\ell_{\gamma}(u,W)$, where $X_{\gamma}(u)=[X_{\rm val};\sqrt{\gamma\psi_{N}(u)}X_{\rm trn}]$ and $Y_{\gamma}(u)=[Y_{\rm val};\sqrt{\gamma\psi_{N}(u)}Y_{\rm trn}]$. According to ([77](#A7.Ex283)) and ([78](#A7.Ex284)), we have

$$ $\displaystyle\|X_{\gamma}(u)^{\dagger}\|_{2}$ $\displaystyle\leq\frac{1}{\sigma_{*}(X_{\gamma}(u))}\leq\frac{1}{\sqrt{\mu_{w}^{\gamma}}}={\cal O}(\gamma^{-\frac{1}{2}})$ $\displaystyle\|X_{\gamma}(u^{1})^{\dagger}-X_{\gamma}(u^{2})^{\dagger}\|$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}\sqrt{2}\max\left\{\|X_{\gamma}(u^{1})\|_{2}^{2},\|X_{\gamma}(u^{2})\|_{2}^{2}\right\}\|X_{\gamma}(u^{1})-X_{\gamma}(u^{2})\|$ $\displaystyle\leq\frac{\sqrt{2}}{\sigma_{*}(X_{\gamma}(\bar{u}))^{2}}\|X_{\gamma}(u^{1})-X_{\gamma}(u^{2})\|$ $\displaystyle\stackrel{{\scriptstyle(b)}}{{\leq}}\frac{\sqrt{2}}{\mu_{w}^{\gamma}}\sqrt{\gamma}\|(\sqrt{\psi_{N}(u^{1})}-\sqrt{\psi_{N}(u^{2})})X_{\rm trn}\|$ $\displaystyle\leq\frac{\sqrt{2}}{\mu_{w}^{\gamma}}\sqrt{\gamma}\|\sqrt{\psi_{N}(u^{1})}-\sqrt{\psi_{N}(u^{2})}\|_{2}\|X_{\rm trn}\|$ $\displaystyle\stackrel{{\scriptstyle(c)}}{{\leq}}\frac{\sqrt{2}}{2\mu_{w}^{\gamma}\sqrt{1-\psi(\bar{u})}}\sqrt{\gamma}\|X_{\rm trn}\|\|\psi_{N}(u^{1})-\psi_{N}(u^{2})\|_{2}$ $\displaystyle\stackrel{{\scriptstyle(d)}}{{\leq}}\frac{\sqrt{2}}{2\mu_{w}^{\gamma}\sqrt{1-\psi(\bar{u})}}\sqrt{\gamma}\|X_{\rm trn}\|\|u^{1}-u^{2}\|={\cal O}(\gamma^{-\frac{1}{2}})\|u^{1}-u^{2}\|$ $$

where (a) results from [, Theorem 3.3], (b) is derived from the definition of $X_{\gamma}(u)$, and (c) is because $\sqrt{a}-\sqrt{b}=\frac{a-b}{\sqrt{a}+\sqrt{b}}$ and the bound of $\sqrt{\psi(u_{i}^{1})}+\sqrt{\psi(u_{i}^{1})}\geq 2\sqrt{1-\psi(\bar{u})},\forall i$ and (d) comes from the $1$-Lischitz continuity of the Sigmoid function $\psi(\cdot)$.

Similarly, we can derive the bound for $Y_{\gamma}(u)$ as

$$ $\displaystyle\|Y_{\gamma}(u)\|_{2}=\sigma_{\max}(Y_{\gamma}(u))$ $\displaystyle\leq\sqrt{\sigma_{\max}(Y_{\rm val}^{\top}Y_{\rm val}+\gamma\psi(\bar{u})Y_{\rm trn}^{\top}Y_{\rm trn})}={\cal O}(\gamma^{\frac{1}{2}})$ $\displaystyle\|Y_{\gamma}(u^{1})-Y_{\gamma}(u^{2})\|$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}\sqrt{\gamma}\|(\sqrt{\psi_{N}(u^{1})}-\sqrt{\psi_{N}(u^{2})})Y_{\rm trn}\|$ $\displaystyle\stackrel{{\scriptstyle(b)}}{{\leq}}\frac{\sqrt{\gamma}\|Y_{\rm trn}\|}{2\sqrt{1-\psi(\bar{u})}}\|u^{1}-u^{2}\|={\cal O}(\gamma^{\frac{1}{2}})\|u^{1}-u^{2}\|.$ $$

where (a) comes from the definition and (b) is derived from the Lipschitz continuity of Sigmoid function, $\|AB\|\leq\|A\|_{2}\|B\|$ and $\sqrt{a}-\sqrt{b}=\frac{a-b}{\sqrt{a}+\sqrt{b}}$.
As a result, for any $u^{1}$ and $u^{2}$, it holds that

$$ $\displaystyle\|W^{*}_{\gamma}(u^{1})$ $\displaystyle-W^{*}_{\gamma}(u^{2})\|\leq\|X_{\gamma}(u^{1})^{\dagger}\|_{2}\|Y_{\gamma}(u^{1})-Y_{\gamma}(u^{2})\|+\|Y_{\gamma}(u^{2})\|_{2}\|X_{\gamma}(u^{1})^{\dagger}-X_{\gamma}(u^{2})^{\dagger}\|$ $\displaystyle\leq\left(\frac{\sqrt{\gamma}\|Y_{\rm trn}\|}{2\sqrt{\mu_{w}^{\gamma}(1-\psi(\bar{u}))}}+\frac{\sqrt{2\gamma\sigma_{\max}(Y_{\rm val}^{\top}Y_{\rm val}+\gamma\psi(\bar{u})Y_{\rm trn}^{\top}Y_{\rm trn})}}{2\mu_{w}^{\gamma}\sqrt{1-\psi(\bar{u})}}\|X_{\rm trn}\|\right)\|u^{1}-u^{2}\|.$ $$

Let us denote

$$ $\displaystyle L_{wu}^{*}:=\frac{\sqrt{\gamma}\|Y_{\rm trn}\|}{2\sqrt{\mu_{w}^{\gamma}(1-\psi(\bar{u}))}}+\frac{\sqrt{2\gamma\sigma_{\max}(Y_{\rm val}^{\top}Y_{\rm val}+\gamma\psi(\bar{u})Y_{\rm trn}^{\top}Y_{\rm trn})}}{2\mu_{w}^{\gamma}\sqrt{1-\psi(\bar{u})}}\|X_{\rm trn}\|={\cal O}(1)$ (79) $$

then $W_{\gamma}^{*}(u)$ is $L_{wu}^{*}$ Lipschitz continuous on $u$.

Besides, the boundedness of $W_{\gamma}^{*}(u)$ can be earned by

$$ $\displaystyle\|W_{\gamma}^{*}(u)\|$ $\displaystyle\leq\|X_{\gamma}(u)^{\dagger}\|_{2}\|Y_{\gamma}(u)\|\leq\max\{n,N+N^{\prime}\}\|X_{\gamma}(u)^{\dagger}\|_{2}\|Y_{\gamma}(u)\|_{2}$ $\displaystyle\leq\frac{\max\{n,N+N^{\prime}\}\sqrt{\sigma_{\max}(Y_{\rm val}^{\top}Y_{\rm val}+\gamma\psi(\bar{u})Y_{\rm trn}^{\top}Y_{\rm trn})}}{\sqrt{\mu_{w}^{\gamma}}}=:L_{w}^{*}={\cal O}(1).$ $$

###### Lemma 25 .

#### G.4 Global solution relations in data hyper-cleaning

Based on the smoothness and PL of $\ell_{\rm trn}(u,W)$ over $W$, we are expected to analyze the approximate behavior of the penalized problem ([12](#S6.E12)) to the bilevel hyper-cleaning problem ([11](#S6.E11)). Since upper-level problem is not Lipschitz continuous globally, the result in [] can not be applied directly. In light of Remark [3](#Thmremark3), we only focus on whether $\epsilon_{2}$ solution of the penalized problem can recover some approximate solution of the bilevel problem in the following lemma.

###### Theorem 7 .

For any $\gamma$ and any $\epsilon_{2}$, suppose that there exists an $\epsilon_{2}$ solution $(u,W)$ to the $\gamma$-penalized problem ([12](#S6.E12)) which has bounded norm $\|W\|\leq R$ and $R$ is independent of $u$ and $\gamma$. For such $\epsilon_{2}$ solution $(u,W)$, there exists $\gamma^{*}={\cal O}(\epsilon_{1}^{-1})$ such that for any $\gamma>\gamma^{*}$, $(u,W)$ is also an $(\epsilon_{2},\epsilon_{\gamma})$ solution to the bilevel problem ([11](#S6.E11)) for some $\epsilon_{\gamma}\leq\frac{\epsilon_{1}+\epsilon_{2}}{\gamma-\gamma^{*}}$.

Similar to the representation learning, to ensure the penalized problem ([12](#S6.E12)) is an $\epsilon$-approximate solution to the bilevel data hyper-cleaning problem ([11](#S6.E11)), i.e. $\epsilon_{\gamma}={\cal O}(\epsilon)$, one need to choose $\epsilon_{1}={\cal O}(\sqrt{\epsilon})$ and set the penalty parameter $\gamma={\cal O}(\epsilon^{-0.5})$. Therefore, if we can verify the iterates generates by PBGD achieves $\epsilon$-solution of the penalized problem and are bounded with radius independent of $\gamma$ and $u$, then PBGD can converge to some approximate solution of bilevel problem.

Proof: Given $\gamma$, we first prove the Lipschitz continuity of the upper-level objective (validation loss) at the lower-level solution set, i.e. for any $\epsilon_{2}$ solution $(u,W)$ of $\gamma$-penalized problem with $\|W\|\leq R$, let $W_{u}=\operatorname*{arg\,min}_{W}\ell_{\rm trn}(u,W)$, then we have $\|W_{u}\|=\|\operatorname{Proj}_{\mathcal{S}(u)}(W)\|\leq\|W\|\leq R$ and

$$ $\displaystyle\ell_{\rm val}(W)-\ell_{\rm val}(W_{u})$ $\displaystyle=\frac{1}{2}\|Y_{\rm val}-X_{\rm val}W\|^{2}-\frac{1}{2}\|Y_{\rm val}-X_{\rm val}W_{u}\|^{2}$ $\displaystyle=\frac{1}{2}\langle 2Y_{\rm val}-X_{\rm val}(W+W_{u}),X_{\rm val}(W-W_{u})\rangle$ $\displaystyle\geq-\left(\|Y_{\rm val}\|+\|X_{\rm val}\|\|W\|\right)\|X_{\rm val}\|\|W-W_{u}\|$ $\displaystyle\geq-\left(\|Y_{\rm val}\|+\|X_{\rm val}\|R\right)\|X_{\rm val}\|\|W-W_{u}\|.$ $$

Then by choosing $\gamma^{*}=\frac{\left(\|Y_{\rm val}\|+\|X_{\rm val}\|R\right)^{2}\|X_{\rm val}\|^{2}}{2\mu_{w}\epsilon_{1}}$, it follows that

$$ $\displaystyle~{}~{}~{}~{}\ell_{\rm val}(W)+\gamma^{*}(\ell_{\rm trn}(u,W)-\ell_{\rm trn}^{*}(u))-\ell_{\rm val}(W_{u})$ $\displaystyle\geq-\left(\|Y_{\rm val}\|+\|X_{\rm val}\|R\right)\|X_{\rm val}\|\|W-W_{u}\|+\gamma^{*}(\ell_{\rm trn}(u,W)-\ell_{\rm trn}^{*}(u))$ $\displaystyle\geq-\left(\|Y_{\rm val}\|+\|X_{\rm val}\|R\right)\|X_{\rm val}\|\|W-W_{u}\|+\frac{\mu_{w}\gamma^{*}}{2}\|W-W_{u}\|^{2}$ $\displaystyle\geq\min_{z\in\mathbb{R}_{+}}-\left(\|Y_{\rm val}\|+\|X_{\rm val}\|R\right)\|X_{\rm val}\|z+\frac{\mu_{w}\gamma^{*}}{2}z^{2}$ $\displaystyle=-\frac{\left(\|Y_{\rm val}\|+\|X_{\rm val}\|R\right)^{2}\|X_{\rm val}\|^{2}}{2\mu_{w}\gamma^{*}}=-\epsilon_{1}.$ (80) $$

According to the definition of $\epsilon_{2}$ stationary point of $\gamma$ penalized problem, we have

$$ $\displaystyle\ell_{\rm val}(W)+\gamma(\ell_{\rm trn}(u,W)-\ell_{\rm trn}^{*}(u))$ $\displaystyle\leq\ell_{\rm val}(W_{u})+\gamma(\ell_{\rm trn}(u,W_{u})-\ell_{\rm trn}^{*}(u))+\epsilon_{2}$ $\displaystyle\leq\ell_{\rm val}(W_{u})+\epsilon_{2}$ $\displaystyle\stackrel{{\scriptstyle\eqref{numberthis}}}{{\leq}}\ell_{\rm val}(W)+\gamma^{*}(\ell_{\rm trn}(u,W)-\ell_{\rm trn}^{*}(u))+\epsilon_{1}+\epsilon_{2}.$ $$

Substracting $\ell_{\rm val}(W)$ from both sides and rearraging yields

$$ $\displaystyle\epsilon_{\gamma}:=\ell_{\rm trn}(u,W)-\ell_{\rm trn}^{*}(u)$ $\displaystyle\leq\frac{\epsilon_{1}+\epsilon_{2}}{\gamma-\gamma^{*}}$ $$

when $\gamma>\gamma^{*}$. Then for any $(u^{\prime},W^{\prime})$ satisfying $\ell_{\rm trn}(u^{\prime},W^{\prime})-\ell_{\rm trn}^{*}(u^{\prime})\leq\epsilon_{\gamma}$, it holds that

$$ $\displaystyle\ell_{\rm val}(W)+\gamma(\ell_{\rm trn}(u,W)-\ell_{\rm trn}^{*}(u))$ $\displaystyle\leq\ell_{\rm val}(W^{\prime})+\gamma(\ell_{\rm trn}(u^{\prime},W^{\prime})-\ell_{\rm trn}^{*}(u^{\prime}))+\epsilon_{2}.$ $$

Rearraging terms yields

$$ $\displaystyle\ell_{\rm val}(W)$ $\displaystyle\leq\ell_{\rm val}(W^{\prime})+\gamma(\ell_{\rm trn}(u^{\prime},W^{\prime})-\ell_{\rm trn}^{*}(u^{\prime})-\epsilon_{\gamma})+\epsilon_{2}\leq\ell_{\rm val}(W^{\prime})+\epsilon_{2}$ $$

which means $(u,W)$ is an $(\epsilon_{2},\epsilon_{\gamma})$ solution to bilevel problem.

###### Theorem 7 .

#### G.5 Parameterized data and label matrix family are acute

###### Lemma 26 .

For $u\in\mathcal{U}$, data matrix family $\{X_{\gamma}(u),u\in\mathcal{U}\}$ is acute, and $\{Y_{\gamma}(u),u\in\mathcal{U}\}$ is acute. Therefore, $\operatorname{Ran}(X_{\gamma}(u))$ remains the same for $u\in\mathcal{U}$ and so does $\operatorname{Ran}(Y_{\gamma}(u))$.

Proof:
For any $u$, we can write $X_{\gamma}(u)$ and $Y_{\gamma}(u)$ as

$$ $\displaystyle X_{\gamma}(u)=\left[\begin{array}[]{l}I\\ ~{}~{}~{}~{}~{}~{}\sqrt{\gamma\sigma_{N}(u)}\end{array}\right]\left[\begin{array}[]{l}X_{\rm val}\\ X_{\rm trn}\end{array}\right],~{}~{}~{}Y_{\gamma}(u)=\left[\begin{array}[]{l}I\\ ~{}~{}~{}~{}~{}~{}\sqrt{\gamma\sigma_{N}(u)}\end{array}\right]\left[\begin{array}[]{l}Y_{\rm val}\\ Y_{\rm trn}\end{array}\right].$ $$

According to Lemma [12](#Thmlemma12), it is obvious that $\operatorname{rank}(X_{\gamma}(u))=\operatorname{rank}([X_{\rm val};X_{\rm trn}])$ and $\operatorname{rank}(Y_{\gamma}(u))=\operatorname{rank}([Y_{\rm val};Y_{\rm trn}])$, which verifies the constant rank condition in Lemma [11](#Thmlemma11). To show the acute property, we need to further prove that for any $u^{1}$ and $u^{2}$, if we denote $A=X_{\gamma}(u^{1})$ and $B=X_{\gamma}(u^{2})$, then $\operatorname{rank}(A)=\operatorname{rank}(P_{A}BR_{A})$ with $P_{A}=AA^{\dagger}$ and $R_{A}=A^{\dagger}A$.

To do so, we first notice that, there exists a diagonal matrix $\Lambda$ such that $B=\Lambda A$ where

$$ $\displaystyle\Lambda=\left[\begin{array}[]{l}I\\ ~{}~{}~{}~{}~{}~{}\sqrt{\sigma_{N}(u^{2})/\sigma_{N}(u^{1})}\end{array}\right].$ $$

Then we can write

$$ $\displaystyle\operatorname{rank}(P_{A}BR_{A})$ $\displaystyle=\operatorname{rank}(AA^{\dagger}\Lambda AA^{\dagger}A)\stackrel{{\scriptstyle(a)}}{{=}}\operatorname{rank}(AA^{\dagger}\Lambda A)$ (81) $$

where (a) is because $AA^{\dagger}A=A$. Furthermore, by singular value decomposition, we can decompose $A=U\Sigma V^{\top}$ with $\Sigma=\left[\begin{array}[]{ll}\Sigma_{1}&0\\
0&0\end{array}\right]\in\mathbb{R}^{(N+N^{\prime})\times m}$, and orthogonal matrix $U=[U_{1}~{}U_{2}]\in\mathbb{R}^{(N+N^{\prime})\times(N+N^{\prime})}$ and $V=[V_{1}~{}V_{2}]\in\mathbb{R}^{m\times m}$. Also, by denoting $\operatorname{rank}(A)=r$, we know that $U_{1}\in\mathbb{R}^{(N+N^{\prime})\times r},V_{1}\in\mathbb{R}^{m\times r}$ and $\Sigma_{1}\in\mathbb{R}^{r\times r}$ are full rank. Thus, $A$ can be decomposed by

$$ $\displaystyle A=[U_{1}~{}U_{2}]\left[\begin{array}[]{ll}\Sigma_{1}&0\\ 0&0\end{array}\right]\left[\begin{array}[]{l}V_{1}^{\top}\\ V_{2}^{\top}\end{array}\right]=[U_{1}\Sigma_{1}~{}~{}~{}0]\left[\begin{array}[]{l}V_{1}^{\top}\\ V_{2}^{\top}\end{array}\right]=U_{1}\Sigma_{1}V_{1}^{\top}.$ $$

Besides, based on linear algebra knowledge, $A^{\dagger}=V_{1}\Sigma_{1}^{-1}U_{1}^{\top}$.
In this way, we can further write

$$ $\displaystyle\operatorname{rank}(P_{A}BR_{A})$ $\displaystyle\stackrel{{\scriptstyle\eqref{rank1}}}{{=}}\operatorname{rank}(AA^{\dagger}\Lambda A)=\operatorname{rank}(U_{1}U_{1}^{\top}\Lambda U_{1}\Sigma_{1}V_{1}^{\top})$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}\operatorname{rank}(U_{1}^{\top}\Lambda U_{1})\stackrel{{\scriptstyle(b)}}{{=}}\operatorname{rank}((\sqrt{\Lambda}U_{1})^{\top}\sqrt{\Lambda}U_{1})$ $\displaystyle\stackrel{{\scriptstyle(c)}}{{=}}\operatorname{rank}(\sqrt{\Lambda}U_{1})\stackrel{{\scriptstyle(d)}}{{=}}\operatorname{rank}(U_{1})=r=\operatorname{rank}(A)$ (82) $$

where (a) is derived from Lemma [12](#Thmlemma12) with full column rank $U_{1}$ and full row rank $V_{1}^{\top},\Sigma_{1}$, (b) is because $\Lambda$ is a nonsingular diagonal matrix, (c) and (d) come from Lemma [12](#Thmlemma12). Combining ([81](#A7.E81)) and ([82](#A7.Ex336)) with Lemma [11](#Thmlemma11), we know $A=X_{\gamma}(u^{1})$ and $B=X_{\gamma}(u^{2})$ are acute. Due to the arbitrary choices of $u^{1},u^{2}\in\mathcal{U}$, the family $\{X_{\gamma}(u),u\in\mathcal{U}\}$ is acute, which also holds for $\{Y_{\gamma}(u),u\in\mathcal{U}\}$. By the acute matrices in Proposition [1](#Thmproposition1), the range space remains the same, which completes the proof.

###### Lemma 26 .

#### G.6 Monotonicity of loss functions

###### Lemma 27 .

Given $u^{1},u^{2}\in\mathcal{U}$ and assuming $u^{1}\leq u^{2}$ in the sense that $u^{1}_{i}\leq u^{2}_{i}$ for any $i$, then

$$ $\displaystyle\psi(u_{i}^{1})\|y_{i}^{\top}-x_{i}^{\top}W_{\gamma}^{*}(u^{1})\|^{2}\leq\psi(u_{i}^{2})\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(u^{2})\|^{2}$ $\displaystyle\|y_{i}^{\prime\top}-x_{i}^{\prime\top}W_{\gamma}^{*}(u^{1})\|^{2}\leq\|y_{i}^{\prime\top}-x_{i}^{\prime\top}W_{\gamma}^{*}(u^{2})\|^{2}.$ $$

where $\{x_{i},y_{i}\}$ are training data samples and $\{x_{i}^{\prime},y_{i}^{\prime}\}$ are validation data samples. Moreover,

$$ $\displaystyle\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\leq\|y_{i}^{\top}-x_{i}^{\top}W_{\gamma}^{*}(u)\|^{2}.$ $$

Proof:
For a matrix $A$, we denote $A_{i}$ as the $i$-th row of $A$.
For any $u\in\mathcal{U}$, we have

$$ $\displaystyle\|y_{i}^{\top}-x_{i}^{\top}W_{\gamma}^{*}(u)\|^{2}$ $\displaystyle=\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(u)\|^{2}$ $\displaystyle=\|y_{i}^{\top}-x_{i}^{\top}X_{\gamma}(u)^{\dagger}Y_{\gamma}(u)\|^{2}$ $\displaystyle=\|(Y_{\rm trn}-X_{\rm trn}X_{\gamma}^{\dagger}(u)Y_{\gamma}(u))_{i}\|^{2}$ $\displaystyle=\gamma^{-1}\psi(u_{i})^{-1}\|(Y_{\gamma}(u)-X_{\gamma}(u)X_{\gamma}^{\dagger}(u)Y_{\gamma}(u))_{(N^{\prime}+i)}\|^{2}$ $\displaystyle=\gamma^{-1}\psi(u_{i})^{-1}\|((I-X_{\gamma}(u)X_{\gamma}^{\dagger}(u))Y_{\gamma}(u))_{(N^{\prime}+i)}\|^{2}$ $\displaystyle=\gamma^{-1}\psi(u_{i})^{-1}\|\operatorname{Proj}_{\operatorname{Ran}(X_{\gamma}(u))^{\perp}}(Y_{\gamma}(u))_{(N^{\prime}+i)}\|^{2}$ (83) $$

Then for $u^{1}\leq u^{2}$, it holds that

$$ $\displaystyle\|y_{i}^{\top}-x_{i}^{\top}W_{\gamma}^{*}(u^{1})\|^{2}$ $\displaystyle=\gamma^{-1}\psi(u_{i}^{1})^{-1}\|\operatorname{Proj}_{\operatorname{Ran}(X_{\gamma}(u^{1}))^{\perp}}(Y_{\gamma}(u^{1}))_{(N^{\prime}+i)}\|^{2}$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}\gamma^{-1}\psi(u_{i}^{1})^{-1}\|\operatorname{Proj}_{\operatorname{Ran}(X_{\gamma}(u^{2}))^{\perp}}(Y_{\gamma}(u^{1}))_{(N^{\prime}+i)}\|^{2}$ $\displaystyle=\gamma^{-1}\psi(u_{i}^{1})^{-1}\|((I-X_{\gamma}(u^{2})X_{\gamma}^{\dagger}(u^{2}))Y_{\gamma}(u^{1}))_{(N^{\prime}+i)}\|^{2}$ $\displaystyle\stackrel{{\scriptstyle(b)}}{{\leq}}\gamma^{-1}\psi(u_{i}^{1})^{-1}\|((I-X_{\gamma}(u^{2})X_{\gamma}^{\dagger}(u^{2}))Y_{\gamma}(u^{2}))_{(N^{\prime}+i)}\|^{2}$ $\displaystyle=\gamma^{-1}\psi(u_{i}^{2})^{-1}\|((I-X_{\gamma}(u^{2})X_{\gamma}^{\dagger}(u^{2}))Y_{\gamma}(u^{2}))_{(N^{\prime}+i)}\|^{2}\times\frac{\psi(u_{i}^{2})}{\psi(u_{i}^{1})}$ $\displaystyle=\frac{\psi(u_{i}^{2})}{\psi(u_{i}^{1})}\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(u^{2})\|^{2}$ $$

where (a) is because $\operatorname{Ran}(X_{\gamma}(u^{1}))=\operatorname{Ran}(X_{\gamma}(u^{2}))$ from Lemma [26](#Thmlemma26), (b) is because each element in $Y_{\gamma}(u^{1})$ is no greater than $Y_{\gamma}(u^{2})$. For the validation loss, it can be proved similarly, while the only difference is the fact that

$$ $\displaystyle\|y_{i}^{\prime\top}-x_{i}^{\prime\top}W_{\gamma}^{*}(u)\|^{2}$ $\displaystyle=\|(Y_{\gamma}(u)-X_{\gamma}(u)X_{\gamma}^{\dagger}(u)Y_{\gamma}(u))_{i}\|^{2}$ $$

without magnitude of $\gamma^{-1}\psi(u_{i})^{-1}$ because $Y_{\gamma}(u)$ and $X_{\gamma}(u)$ do not have such magnitude on validation data points.

Finally, as $\ell_{\rm trn}(u,W)-\ell_{\rm trn}^{*}(u)\geq 0$ holds for any $u\in\mathcal{U}$ and $W$, we have

$$ $\displaystyle\frac{\gamma}{2}\sum_{i=1}^{N}\psi(u_{i})\left[\|y_{i}^{\top}-x_{i}^{\top}W\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right]\geq 0$ $$

holds for any $u\in\mathcal{U}$. Then letting $u$ be the vector which equals to $\bar{u}$ at $i$-th position and equals to $-\bar{u}$ elsewhere and taking $\bar{u}\rightarrow\infty$, we know

$$ $\displaystyle\|y_{i}^{\top}-x_{i}^{\top}W\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\geq 0$ $$

holds for any $i$ and $W$. Taking $W=W_{\gamma}^{*}(u)$ yields the conclusion.

###### Lemma 28 .

If $X_{\rm trn}X_{\rm trn}^{\dagger}$ is diagonal matrix, $\ell_{\gamma}^{*}(u)$ is uniformly PL over $u\in\mathcal{U}$ with $\mu_{u}$ and is uniformly smooth over $u\in\mathcal{U}$ with $L_{u}$, and the constants are defined as

$$ $\displaystyle\mu_{u}:=\frac{\gamma\psi(\bar{u})(1-\psi(\bar{u}))^{2}\min_{i}\|y_{i}\|^{2}}{4}={\cal O}(\gamma),~{}~{}~{}~{}L_{u}:=\frac{\gamma}{2}\sum_{i=1}^{N}\|y_{i}\|^{2}={\cal O}(\gamma).$ (84) $$

Moreover, the gradient of $\ell_{\gamma}^{*}(u)$ is also bounded by $L_{u}$, i.e. $\|\nabla_{u}\ell_{\gamma}^{*}(u)\|\leq L_{u}$.

Proof:
Since $\ell_{\gamma}(u,\cdot)$ is smooth and PL, according to [, Lemma A.5], we know $\ell_{\gamma}^{*}(u)=\min_{W}\ell_{\gamma}(u,W)$ is smooth with the gradient

$$ $\nabla\ell_{\gamma}^{*}(u)=\nabla_{u}\ell_{\gamma}(u,W),~{}~{}~{}~{}~{}\forall W\in\operatorname*{arg\,min}_{W}\ell_{\gamma}(u,W)=\operatorname*{arg\,min}_{W}\tilde{\ell}_{\gamma}(u,W).$ $$

We write the minimal-norm optimal solution of penalized problem as $W^{*}_{\gamma}(u)\in\operatorname*{arg\,min}_{W}\ell_{\gamma}(u,W)$. Plugging $W^{*}_{\gamma}(u)$ into ([74](#A7.Ex270)), we know

$$ $\displaystyle\|\nabla\ell_{\gamma}^{*}(u)\|^{2}=\|\nabla_{u}\ell_{\gamma}(u,W^{*}_{\gamma}(u))\|^{2}$ $\displaystyle\geq\frac{\gamma c(W^{*}_{\gamma}(u))\psi(\bar{u})(1-\psi(\bar{u}))^{2}}{2}(\ell_{\gamma}(u,W^{*}_{\gamma}(u))-\min_{u}\ell_{\gamma}(u,W^{*}_{\gamma}(u)))$ $\displaystyle=\frac{\gamma c(W^{*}_{\gamma}(u))\psi(\bar{u})(1-\psi(\bar{u}))^{2}}{2}(\ell_{\gamma}^{*}(u)-\min_{u}\ell_{\gamma}^{*}(u))$ $$

which suggests that $\nabla\ell_{\gamma}^{*}(u)$ is $\frac{\gamma c(W^{*}_{\gamma}(u))\psi(\bar{u})(1-\psi(\bar{u}))^{2}}{4}$ PL over $u\in\mathcal{U}$. Besides, letting $u_{0}=[\bar{u},\cdots,\bar{u}]$, we have

$$ $\displaystyle c(W^{*}_{\gamma}(u))$ $\displaystyle=\min_{i}\left\{\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(u)\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right\}_{>0}$ $\displaystyle=\min_{i}\left\{\gamma^{-1}\psi(u_{i})^{-1}\|((I-X_{\gamma}(u)X_{\gamma}^{\dagger}(u))Y_{\gamma}(u))_{(N^{\prime}+i)}\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right\}_{>0}$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}\min_{i}\left\{\gamma^{-1}\psi(u_{i})^{-1}\|((I-X_{\gamma}(u)X_{\gamma}^{\dagger}(u))Y_{\gamma}(u))_{(N^{\prime}+i)}\|^{2}\right\}_{>0}$ $\displaystyle=\min_{i}\left\{\gamma^{-1}\psi(u_{i})^{-1}\|\operatorname{Proj}_{\operatorname{Ran}(X_{\gamma}(u))^{\perp}}(Y_{\gamma}(u))_{(N^{\prime}+i)}\|^{2}\right\}_{>0}$ $\displaystyle\stackrel{{\scriptstyle(b)}}{{\geq}}\frac{\psi(-\bar{u})}{\psi(\bar{u})}\min_{i}\left\{\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(-u_{0})\|^{2}\right\}_{>0}={\cal O}(1)$ (85) $$

where (a) comes from

$$ $\gamma^{-1}\psi(u_{i})^{-1}\|((I-X_{\gamma}(u)X_{\gamma}^{\dagger}(u))Y_{\gamma}(u))_{(N^{\prime}+i)}\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)>0$ $$

if and only if $\gamma^{-1}\psi(u_{i})^{-1}\|((I-X_{\gamma}(u)X_{\gamma}^{\dagger}(u))Y_{\gamma}(u))_{(N^{\prime}+i)}\|^{2}>0$ and $[X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}=1$, (b) is due to Lemma [27](#Thmlemma27).
This means $\ell_{\gamma}^{*}(u)$ is uniformly PL over $u\in\mathcal{U}$ with constant

$$ $\displaystyle\mu_{u}:=\frac{\gamma\psi(\bar{u})(1-\psi(\bar{u}))^{2}\min_{i}\left\{\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(-u_{0})\|^{2}\right\}_{>0}}{4}={\cal O}(\gamma).$ (86) $$

Moreover, $\nabla^{2}_{u}\ell_{\gamma}(u^{\prime},W)$ at $W=W_{\gamma}^{*}(u)$ can be upper bounded by

$$ $\displaystyle\|\nabla^{2}_{u}\ell_{\gamma}(u^{\prime},W^{*}_{\gamma}(u))\|$ $\displaystyle=\sum_{i=1}^{N}\left\|\frac{\gamma}{2}\nabla^{2}\psi(u_{i}^{\prime})\left[\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(u)\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right]\right\|$ $\displaystyle\leq\frac{\gamma}{2}\sum_{i=1}^{N}\|\nabla^{2}\psi(u_{i}^{\prime})\|\left[\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(u)\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right]$ $\displaystyle\leq\frac{\gamma}{2}\sum_{i=1}^{N}\left[\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(u)\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right]$ $\displaystyle\leq\frac{\gamma}{2}\sum_{i=1}^{N}\left[\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(u)\|^{2}\right]$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}\frac{\gamma}{2}\sum_{i=1}^{N}\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(u_{0})\|^{2}$ (87) $$

where (a) comes from similar derivation as of ([85](#A7.Ex362)). Similarly, for any $W=aW_{\gamma}(u^{1})+(1-a)W_{\gamma}(u^{2})$ where $a\in[0,1]$, $\nabla^{2}_{uW}\ell_{\gamma}(u^{\prime},W)$ can be bounded by

$$ $\displaystyle\|\nabla^{2}_{uW}\ell_{\gamma}(u^{\prime},W)\|$ $\displaystyle=\sum_{i=1}^{N}\left\|\gamma\nabla\psi(u_{i}^{\prime})x_{i}(y_{i}^{\top}-x_{i}^{\top}W)\right\|$ $\displaystyle\leq\gamma\sum_{i=1}^{N}\|x_{i}(y_{i}^{\top}-x_{i}^{\top}W)\|$ $\displaystyle=\gamma\sum_{i=1}^{N}\|x_{i}(y_{i}^{\top}-x_{i}^{\top}(aW_{\gamma}(u^{1})+(1-a)W_{\gamma}(u^{2}))\|$ $\displaystyle\leq a\gamma\sum_{i=1}^{N}\|x_{i}(y_{i}^{\top}-x_{i}^{\top}W_{\gamma}(u^{1}))\|+(1-a)\gamma\sum_{i=1}^{N}\|x_{i}(y_{i}^{\top}-x_{i}^{\top}W_{\gamma}(u^{2}))\|$ $\displaystyle\leq\gamma\sum_{i=1}^{N}\|x_{i}\|\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(u_{0})\|={\cal O}(\gamma).$ (88) $$

Together ([87](#A7.Ex368)) and ([88](#A7.Ex373)) indicate that $\ell^{*}_{\gamma}(u)$ is smooth because for any $u^{1},u^{2}\in\mathbb{R}^{N}$, it holds that

$$ $\displaystyle~{}~{}~{}~{}~{}\|\nabla_{u}\ell_{\gamma}^{*}(u^{1})-\nabla_{u}\ell_{\gamma}^{*}(u^{2})\|$ $\displaystyle=\|\nabla_{u}\ell_{\gamma}(u^{1},W^{*}_{\gamma}(u^{1}))-\nabla_{u}\ell_{\gamma}(u^{2},W^{*}_{\gamma}(u^{2}))\|$ $\displaystyle\leq\|\nabla_{u}\ell_{\gamma}(u^{1},W^{*}_{\gamma}(u^{1}))-\nabla_{u}\ell_{\gamma}(u^{2},W^{*}_{\gamma}(u^{1}))\|+\|\nabla_{u}\ell_{\gamma}(u^{2},W^{*}_{\gamma}(u^{1}))-\nabla_{u}\ell_{\gamma}(u^{2},W^{*}_{\gamma}(u^{2}))\|$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}\|\nabla_{u}^{2}\ell_{\gamma}(u^{\prime},W^{*}_{\gamma}(u^{1}))\|\|u^{2}-u^{1}\|+\|\nabla_{uw}^{2}\ell_{\gamma}(u^{2},W^{\prime})\|\|W^{*}_{\gamma}(u^{1})-W^{*}_{\gamma}(u^{2})\|$ $\displaystyle\stackrel{{\scriptstyle(b)}}{{\leq}}\frac{\gamma}{2}\sum_{i=1}^{N}\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(u_{0})\|^{2}\|u^{1}-u^{2}\|+\gamma\sum_{i=1}^{N}\|x_{i}\|\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(u_{0})\|L_{wu}^{*}\|u^{1}-u^{2}\|$ $$

where (a) comes from the mean value theorem with $u^{\prime}=au^{1}+(1-a)u^{2}$ and $W^{\prime}=bW_{\gamma}^{*}(u^{1})+(1-b)W_{\gamma}^{*}(u^{2})$ for some $a,b\in[0,1]$, and (b) holds from ([87](#A7.Ex368)) and ([88](#A7.Ex373)).
Therefore, we can define the smoothness constant of $\ell^{*}_{\gamma}$ as

$$ $\displaystyle L_{u}:=\frac{\gamma}{2}\sum_{i=1}^{N}\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(u_{0})\|^{2}+\gamma\sum_{i=1}^{N}\|x_{i}\|\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(u_{0})\|L_{wu}^{*}={\cal O}(\gamma).$ (89) $$

Besides the bounded Hessian, the gradient of $\ell_{\gamma}(u,W^{*}_{\gamma}(u))$ is also bounded by $L_{u}$ because

$$ $\displaystyle~{}~{}~{}~{}~{}\|\nabla_{u}\ell_{\gamma}^{*}(u)\|=\|\nabla_{u}\ell_{\gamma}(u,W^{*}_{\gamma}(u))\|$ $\displaystyle=\sum_{i=1}^{N}\left\|\frac{\gamma}{2}\nabla\psi(u_{i})\left[\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(u)\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right]\right\|$ $\displaystyle\leq\frac{\gamma}{2}\sum_{i=1}^{N}\|\nabla\psi(u_{i})\|\left[\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(u)\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right]$ $\displaystyle\leq\frac{\gamma}{2}\sum_{i=1}^{N}\left[\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(u)\|^{2}\right]\stackrel{{\scriptstyle(a)}}{{\leq}}\frac{\gamma}{2}\sum_{i=1}^{N}\|y_{i}^{\top}-x_{i}^{\top}W^{*}_{\gamma}(u_{0})\|^{2}\leq L_{u}$ (90) $$

where (a) can be obtained by ([87](#A7.Ex368)).

###### Lemma 27 .

###### Lemma 28 .

#### G.7 Gradient estimation error

###### Lemma 29 .

The gradient estimator has the bounded error

$$ $\displaystyle~{}~{}~{}~{}\left\|\nabla\ell^{*}_{\gamma}(u)-\gamma\left(\nabla_{u}\ell_{\rm trn}(u^{k},{W}^{k+1})-\nabla_{u}\ell_{\rm trn}(u^{k},Z^{k+1})\right)\right\|$ $\displaystyle\leq\gamma\left(C(Z^{0})\sqrt{\frac{2\ell_{\rm trn}(Z^{0})}{\mu_{w}}}+C(W^{0})\sqrt{\frac{2\ell_{\gamma}(W^{0})}{\mu_{w}^{\gamma}}}\right)(1-\beta\mu_{w})^{T/2}.$ (91) $$

where $\beta\mu_{w}=\min\{\beta_{1}\mu_{w},\beta_{2}\mu_{w}^{\gamma}\}$.

Proof:
Let $Z^{\prime}=\operatorname{Proj}_{\mathcal{S}(u^{k})}(Z^{k+1})$, we know $\|Z^{\prime}\|\leq\|Z^{k+1}\|$ and thus,

$$ $\displaystyle\|\nabla\ell_{\rm trn}^{*}(u^{k})-\nabla_{u}\ell_{\rm trn}(u^{k},Z^{k+1})\|$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}\|\nabla_{u}\ell_{\rm trn}(u^{k},Z^{\prime})-\nabla_{u}\ell_{\rm trn}(u^{k},Z^{k+1})\|$ $\displaystyle=\left\|\frac{1}{2}\sum_{i=1}^{N}\nabla\psi(u_{i})(\|y_{i}^{\top}-x_{i}^{\top}Z^{\prime}\|^{2}-\|y_{i}^{\top}-x_{i}^{\top}Z^{k+1}\|^{2})\right\|$ $\displaystyle\leq\frac{1}{2}\sum_{i=1}^{N}\|2y_{i}-x_{i}^{\top}(Z^{\prime}+Z^{k+1})\|\|x_{i}^{\top}(Z^{\prime}-Z^{k+1})\|$ $\displaystyle\leq\sum_{i=1}^{N}(\|y_{i}\|+\|x_{i}\|\|Z^{k+1}\|)\|x_{i}\|\|Z^{\prime}-Z^{k+1}\|$ $\displaystyle=\sum_{i=1}^{N}(\|y_{i}\|+\|x_{i}\|\|Z^{k+1}\|)\|x_{i}\|d(Z^{k+1},\mathcal{S}(u^{k}))$ (92) $$

where (a) results from the Danskin type theorem [, Lemma A.5] that $\nabla\ell_{\rm trn}^{*}(u^{k})=\nabla_{u}\ell_{\rm trn}(u^{k},Z),~{}\forall Z\in\mathcal{S}(u^{k})$.
Then according to Lemmas [6](#Thmlemma6) and [7](#Thmlemma7), at each iteration $k$, we have

$$ $\displaystyle\|Z^{k+1}\|$ $\displaystyle\leq\|Z^{k+1}-Z^{0}\|+\|Z^{0}\|\leq\|Z^{0}\|+\sqrt{\frac{8\ell_{\rm trn}(u^{k},Z^{0})}{\mu_{w}}}\stackrel{{\scriptstyle(a)}}{{\leq}}\|Z^{0}\|+\sqrt{\frac{8\ell_{\rm trn}(Z^{0})}{\mu_{w}}}$ (93) $$

and

$$ $\displaystyle d(Z^{k+1},\mathcal{S}(u^{k}))^{2}=d(Z^{k,T},\mathcal{S}(u^{k}))^{2}$ $\displaystyle\leq\frac{2}{\mu_{w}}\left(\ell_{\rm trn}(u^{k},Z^{k,T})-\min_{Z}\ell_{\rm trn}(u^{k},Z)\right)$ $\displaystyle\leq\frac{2(1-\beta\mu_{w})^{T}}{\mu_{w}}\left(\ell_{\rm trn}(u^{k},Z^{k,0})-\min_{Z}\ell_{\rm trn}(u^{k},Z)\right)$ $\displaystyle=\frac{2(1-\beta\mu_{w})^{T}}{\mu_{w}}\left(\ell_{\rm trn}(u^{k},Z^{0})-\min_{Z}\ell_{\rm trn}(u^{k},Z)\right)$ $\displaystyle\stackrel{{\scriptstyle(b)}}{{\leq}}\frac{2(1-\beta\mu_{w})^{T}\ell_{\rm trn}(Z^{0})}{\mu_{w}}$ (94) $$

where (a) and (b) hold because $\min_{Z}\ell_{\rm trn}(u^{k},Z)\geq 0$ and $\psi(u)\leq 1$. Plugging ([93](#A7.E93)), ([94](#A7.Ex392)) into ([G.7](#A7.Ex385)) and defining $C(Z^{0})=\sum_{i=1}^{N}(\|y_{i}\|+\|x_{i}\|(\|Z^{0}\|+\sqrt{\frac{8\ell_{\rm trn}(Z^{0})}{\mu_{w}}}))\|x_{i}\|$, we obtain that

$$ $\displaystyle\|\nabla\ell_{\rm trn}^{*}(u^{k})-\nabla\ell_{\rm trn}(u^{k},Z^{k+1})\|$ $\displaystyle\leq C(Z^{0})\sqrt{\frac{2\ell_{\rm trn}(Z^{0})}{\mu_{w}}}(1-\beta_{1}\mu_{w})^{T/2}.$ (95) $$

Similarly, if we define $\tilde{\ell}_{\gamma}(u,W)=\ell_{\rm val}(W)+\gamma\ell_{\rm trn}(u,W)$ and $\tilde{\ell}^{*}_{\gamma}(u):=\min_{W}\tilde{\ell}_{\gamma}(u,W)$, as $\tilde{\ell}_{\gamma}(u,\cdot)$ is also smooth and PL, the gradient estimator of $\tilde{\ell}_{\gamma}^{*}$ can be also bounded by

$$ $\displaystyle\|\nabla\tilde{\ell}_{\gamma}^{*}(u^{k})-\nabla_{u}\tilde{\ell}_{\gamma}(u^{k},W^{k+1})\|$ $\displaystyle\leq\gamma C(W^{0})\sqrt{\frac{2\ell_{\gamma}(W^{0})}{\mu_{w}^{\gamma}}}(1-\beta_{2}\mu_{w}^{\gamma})^{T/2}.$ (96) $$

We then define $\ell^{*}_{\gamma}(u):=\min_{W}\ell_{\gamma}(u,W)$ and since $\ell_{\gamma}(u,W)=\tilde{\ell}_{\gamma}(u,W)-\gamma\ell_{\rm trn}^{*}(u)$, we have

$$ $\displaystyle\ell^{*}_{\gamma}(u)=\min_{W}\ell_{\gamma}(u,W)=\min_{W}\tilde{\ell}_{\gamma}(u,W)-\gamma\ell_{\rm trn}^{*}(u)=\tilde{\ell}_{\gamma}^{*}(u)-\gamma\ell_{\rm trn}^{*}(u)$ $$

and thus $\ell^{*}_{\gamma}(u)$ is differentiable and $\nabla\ell^{*}_{\gamma}(u)=\nabla\tilde{\ell}_{\gamma}^{*}(u)-\gamma\nabla\ell_{\rm trn}^{*}(u)$.

Therefore, the gradient estimator of the penalized objective $\ell^{*}_{\gamma}$ can be bounded by

$$ $\displaystyle~{}~{}~{}~{}\left\|\nabla\ell^{*}_{\gamma}(u)-\gamma\left(\nabla_{u}\ell_{\rm trn}(u^{k},{W}^{k+1})-\nabla_{u}\ell_{\rm trn}(u^{k},Z^{k+1})\right)\right\|$ $\displaystyle\leq\|\nabla\tilde{\ell}_{\gamma}^{*}(u^{k})-\nabla_{u}\tilde{\ell}_{\gamma}(u^{k},W^{k+1})\|+\gamma\|\nabla\ell_{\rm trn}^{*}(u^{k})-\nabla\ell_{\rm trn}(u^{k},Z^{k+1})\|$ $\displaystyle\leq\gamma C(Z^{0})\sqrt{\frac{2\ell_{\rm trn}(Z^{0})}{\mu_{w}}}(1-\beta_{1}\mu_{w})^{T/2}+\gamma C(W^{0})\sqrt{\frac{2\ell_{\gamma}(W^{0})}{\mu_{w}^{\gamma}}}(1-\beta_{2}\mu_{w}^{\gamma})^{T/2}$ $\displaystyle\leq\gamma\left(C(Z^{0})\sqrt{\frac{2\ell_{\rm trn}(Z^{0})}{\mu_{w}}}+C(W^{0})\sqrt{\frac{2\ell_{\gamma}(W^{0})}{\mu_{w}^{\gamma}}}\right)(1-\beta\mu_{w})^{T/2}$ (97) $$

where $\beta\mu_{w}=\min\{\beta_{1}\mu_{w},\beta_{2}\mu_{w}^{\gamma}\}$.

###### Lemma 29 .

#### G.8 Proof of Theorem 3

Proof:
We first prove the error bound condition of $\ell^{*}_{\gamma}(u)$ over the constraint $u\in\mathcal{U}$. We denote

$$ $\displaystyle c_{i}(u):=\|y_{i}^{\top}-x_{i}^{\top}W_{\gamma}^{*}(u)\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)$ (98) $$

so that $\ell_{\gamma}^{*}(u):=\ell_{\rm val}(W_{\gamma}^{*}(u))+\frac{\gamma}{2}\sum_{i=1}^{N}\psi(u_{i})c_{i}(u)$.
Since $\ell_{\gamma}(u,\cdot)$ is uniformly PL, so the gradient of $\ell_{\gamma}^{*}(u)$ can be calculated by the Danskin type theorem as

$$ $\displaystyle\nabla_{u_{i}}\ell_{\gamma}^{*}(u)$ $\displaystyle=\frac{\gamma}{2}\nabla\psi(u_{i})\left[\|y_{i}^{\top}-x_{i}^{\top}W_{\gamma}^{*}(u)\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right]$ $\displaystyle=\frac{\gamma}{2}\psi(u_{i})(1-\psi(u_{i}))\left[\|y_{i}^{\top}-x_{i}^{\top}W_{\gamma}^{*}(u)\|^{2}-\|y_{i}\|^{2}\mathds{1}([X_{\rm trn}X_{\rm trn}^{\dagger}]_{i}\neq 1)\right]$ $\displaystyle=\frac{\gamma}{2}\psi(u_{i})(1-\psi(u_{i}))c_{i}(u)\geq 0$ $$

which means $\ell_{\gamma}^{*}(u)$ is non-decreasing and (projected) gradient flow will never fluctuate, i.e. $u^{k+1}\leq u^{k}$ holds for any $k$. In this way, the projection operator is effective at most at the end point. If $\ell_{\gamma}^{*}(u)$ attains minimum for some $u\in\operatorname{int}\mathcal{U}$, then $c_{i}(u)=0$. In this case, iterates generated by projected gradient descent converge to $\min_{u\in\mathcal{U}}\ell_{\gamma}^{*}(u)=\min_{u}\ell_{\gamma}^{*}(u)$ because the projection operator is ineffective along the trajectory.

If $\ell_{\gamma}^{*}(u)$ attains minimum on the boundary $u\in\mathcal{U}$, we will then prove the sequence generated by projected gradient descent will still converge to $\min_{u\in\mathcal{U}}\ell_{\gamma}^{*}(u)$ by contradiction.
It is clear that $\lim_{k\rightarrow\infty}\ell_{\gamma}^{*}(u^{k})<\min_{u\in\mathcal{U}}\ell_{\gamma}^{*}(u)$ can not hold because $u^{k}\in\mathcal{U}$. Therefore, without loose of generality,
we assume that $\lim_{k\rightarrow\infty}\ell_{\gamma}^{*}(u^{k})>\min_{u\in\mathcal{U}}\ell_{\gamma}^{*}(u)$. Then according to the non-decreasing property of $\ell_{\gamma}^{*}(u)$, we know $\lim_{k\rightarrow\infty}u^{k}>-\bar{u}$ so that the projection operator is ineffective at any iteration $K$. By smoothness and denoting $\widetilde{\nabla}\ell_{\gamma}^{*}(u^{k}):=\gamma\left(\nabla_{u}\ell_{\rm trn}(u^{k},{W}^{k+1})-\nabla_{u}\ell_{\rm trn}(u^{k},Z^{k+1})\right),\ell_{\gamma}^{*}=\min\ell_{\gamma}^{*}(u)$, for any $k\leq K$,

$$ $\displaystyle\ell_{\gamma}^{*}(u^{k+1})$ $\displaystyle\leq\ell_{\gamma}^{*}(u^{k})+\langle\nabla\ell_{\gamma}^{*}(u^{k}),-\alpha\widetilde{\nabla}\ell_{\gamma}^{*}(u^{k})\rangle+\frac{\alpha^{2}L_{u}}{2}\|\widetilde{\nabla}\ell_{\gamma}^{*}(u^{k})\|^{2}$ $\displaystyle\leq\ell_{\gamma}^{*}(u^{k})-\alpha\|\nabla\ell_{\gamma}^{*}(u^{k})\|^{2}+\frac{\alpha^{2}L_{u}}{2}\|\nabla\ell_{\gamma}^{*}(u^{k})\|^{2}+\alpha\langle\nabla\ell_{\gamma}^{*}(u^{k}),-\widetilde{\nabla}\ell_{\gamma}^{*}(u^{k})+\nabla\ell_{\gamma}^{*}(u^{k})\rangle$ $\displaystyle~{}~{}~{}~{}+\frac{\alpha^{2}L_{u}}{2}\langle\widetilde{\nabla}\ell_{\gamma}^{*}(u^{k})+\nabla\ell_{\gamma}^{*}(u^{k}),\widetilde{\nabla}\ell_{\gamma}^{*}(u^{k})-\nabla\ell_{\gamma}^{*}(u^{k})\rangle$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}\ell_{\gamma}^{*}(u^{k})-\alpha\|\nabla\ell_{\gamma}^{*}(u^{k})\|^{2}+\frac{\alpha^{2}L_{u}}{2}\|\nabla\ell_{\gamma}^{*}(u^{k})\|^{2}+\alpha L_{u}(\bar{u}+2)\|\nabla\ell_{\gamma}^{*}(u^{k})-\widetilde{\nabla}\ell_{\gamma}^{*}(u^{k})\|$ $\displaystyle\stackrel{{\scriptstyle(b)}}{{\leq}}\ell_{\gamma}^{*}(u^{k})-\alpha\mu_{u}(\ell_{\gamma}^{*}(u^{k})-\ell_{\gamma}^{*})+\alpha L_{u}(\bar{u}+2)\|\nabla\ell_{\gamma}^{*}(u^{k})-\widetilde{\nabla}\ell_{\gamma}^{*}(u^{k})\|$ $\displaystyle\stackrel{{\scriptstyle(c)}}{{\leq}}\ell_{\gamma}^{*}(u^{k})-\alpha\mu_{u}(\ell_{\gamma}^{*}(u^{k})-\ell_{\gamma}^{*})$ $\displaystyle~{}~{}~{}~{}~{}+\alpha L_{u}(\bar{u}+2)\gamma\left(C(W^{0})\sqrt{\frac{2\ell_{\gamma}(W^{0})}{\mu_{w}^{\gamma}}}+C(Z^{0})\sqrt{\frac{2\ell_{\rm trn}(Z^{0})}{\mu_{w}}}\right)(1-\beta\mu_{w})^{T/2}$ (99) $$

where (a) comes from the Cauchy-Swartz inequality, $\|\nabla\ell_{\gamma}^{*}(u)\|\leq L_{u}$ in Lemma [28](#Thmlemma28), $\alpha L_{u}\leq 1$ and

$$ $\displaystyle\alpha\|\widetilde{\nabla}\ell_{\gamma}^{*}(u^{k})\|=\|u^{k}-\operatorname{Proj}_{\mathcal{U}}\left(u^{k}-\alpha\widetilde{\nabla}\ell_{\gamma}^{*}(u^{k})\right)\|\leq 2\bar{u}$ $$

(b) results from , and (c) is derived from ([97](#A7.Ex397)).
Subtracting $\ell_{\gamma}^{*}$ from the both sides of ([99](#A7.Ex407)), we get

$$ $\displaystyle\ell_{\gamma}^{*}(u^{k+1})-\ell_{\gamma}^{*}$ $\displaystyle\leq(1-\alpha\mu_{u})(\ell_{\gamma}^{*}(u^{k})-\ell_{\gamma}^{*})$ $\displaystyle+\alpha L_{u}(\bar{u}+2)\gamma\left(C(W^{0})\sqrt{\frac{2\ell_{\gamma}(W^{0})}{\mu_{w}^{\gamma}}}+C(Z^{0})\sqrt{\frac{2\ell_{\rm trn}(Z^{0})}{\mu_{w}}}\right)(1-\beta\mu_{w})^{T/2}.$ (100) $$

Telescoping ([100](#A7.Ex410)) from $k=1$ to $K$
yields

$$ $\displaystyle\!\!\ell_{\gamma}^{*}(u^{K})-\ell_{\gamma}^{*}$ $\displaystyle\leq(1-\alpha\mu_{u})^{K}(\ell_{\gamma}^{*}(u^{k})-\ell_{\gamma}^{*})$ $\displaystyle~{}~{}~{}~{}+\frac{L_{u}(1+2\bar{u})\gamma\left(C(W^{0})\sqrt{\frac{2\ell_{\gamma}(W^{0})}{\mu_{w}^{\gamma}}}+C(Z^{0})\sqrt{\frac{2\ell_{\rm trn}(Z^{0})}{\mu_{w}}}\right)}{\mu_{l}}(1-\beta\mu_{w})^{T/2}.$ (101) $$

Taking $K\rightarrow\infty$, we know $\lim_{k\rightarrow\infty}\ell_{\gamma}^{*}(u^{k})=\min\ell_{\gamma}^{*}(u)\leq\min_{u\in\mathcal{U}}\ell_{\gamma}^{*}(u)$ which yields a contradiction to $\lim_{k\rightarrow\infty}\ell_{\gamma}^{*}(u^{k})>\min_{u\in\mathcal{U}}\ell_{\gamma}^{*}(u)$.
In conclusion, choosing $\gamma={\cal O}(\epsilon^{0.5})$, and to achieve the $\epsilon$-stationary point of the penalized objective, we can set $T={\cal O}(\log\epsilon^{-1})$ and $K={\cal O}(\log\epsilon^{-1})$.

Besides, according to Lemma [25](#Thmlemma25), the minimum norm solution $W_{\gamma}^{*}(u)=\operatorname*{arg\,min}_{W}\ell_{\gamma}(u,W)$ is bounded by $L_{w}^{*}={\cal O}(1)$. Moreover, according to [], GD on linear regression converges to the closest minimizer to the initialization. Therefore, the iterates of PBGD satisfies

$$ $\displaystyle\|W^{k}\|\leq\|W^{k}-\operatorname{Proj}_{\mathcal{W}_{\gamma}^{*}(u^{k})}(W^{0})\|+\|\operatorname{Proj}_{\mathcal{W}_{\gamma}^{*}(u^{k})}(W^{0})\|\leq{\cal O}(1)(1-\beta\mu_{w})^{T/2}+L_{w}^{*}$ $$

where the bound is independent of $\gamma$. Then according to Theorem [7](#Thmtheorem7), the $\epsilon$-stationary point of the penalized objective with $\gamma={\cal O}(\epsilon^{0.5})$ recovers an $(\epsilon,\epsilon)$ optimal point of the bilevel problem. Therefore, the iteration complexity of PBGD to achieve an $(\epsilon,\epsilon)$ optimal point is $TK={\cal O}((\log\epsilon^{-1})^{2})$.

### Appendix H Numerical Experiments

In this section, we provide numerical results for global convergence of PBGD in representation learning and data hyper-cleaning.

#### H.1 Representation learning

Considering the overparameterized and wide neural network case, we choose $N=30,N^{\prime}=20,m=40,n=10,h=300$. First, we respectively generate data matrix $X_{\rm trn}\in\mathbb{R}^{N\times m},X_{\rm val}\in\mathbb{R}^{N^{\prime}\times m}$ from Gaussian distribution $\mathcal{N}(5,0.01)$ and $\mathcal{N}(-3,0.01)$ to model different cluster of data. Then we generate optimal $W_{1}^{*}\in\mathbb{R}^{m\times h},W_{2}^{*}\in\mathbb{R}^{h\times n}$ from Gaussian distribution $\mathcal{N}(0,0.01)$ and $\mathcal{N}(2,0.01)$, respectively. Moroever, we generate the optimal weight under validation dataset $\widetilde{W}_{2}^{*}\in\mathbb{R}^{h\times n}$ by $\widetilde{W}_{2}^{*}\sim\mathcal{N}(W_{2}^{*},0.001)$. This ensures that $\|\widetilde{W}_{2}^{*}-W_{2}^{*}\|$ is not too large with high probability, satisfying Assumption [2](#Thmassumption2). Finally, we use $W_{1}^{*},W_{2}^{*},\widetilde{W}_{2}^{*}$ to generate the label matrix. For the bottom layer, we want both training label and validation clean label finds the shared optimal weight $W_{1}^{*}$, However, for the adaptation layer, they should exhibit distinct behaviors due to the different clusters. Specifically, the training labels should find the optimal weight $W_{2}^{*}$, while the validation label should find the optimal validation adaptation weight $\widetilde{W}_{2}^{*}$. Inspired by this, we generate the label matrix by

$$ $\displaystyle Y_{\rm trn}\sim\mathcal{N}(X_{\rm trn}W_{1}^{*}W_{2}^{*},0.01),~{}~{}~{}Y_{\rm val}\sim\mathcal{N}(X_{\rm trn}W_{1}^{*}\widetilde{W}_{2}^{*},0.01).$ $$

We test the PBGD in Algorithm [1](#alg1) on this synthetic representation learning problem and plot the upper-level and lower-level relative error versus iteration in Figure [3](#S7.F3). We measure upper-level relative error by $\mathsf{L}_{\rm val}(W_{1},W_{2})-\mathsf{L}_{\rm val}^{*}$ where $\mathsf{L}_{\rm val}^{*}=\min_{W_{1},W_{2}\in\mathcal{S}(W_{1})}\mathsf{L}_{\rm val}(W_{1},W_{2})$, and lower-level relative error is measured by $\mathsf{L}_{\rm trn}(W_{1},W_{2})-\mathsf{L}_{\rm trn}^{*}(W_{1})$. By the closed form of $\mathsf{L}_{\rm trn}^{*}(W_{1})$ in Lemma [16](#Thmlemma16), lower-level relative error is accessible. On the other hand, $\mathsf{L}_{\rm val}^{*}\approx\mathsf{L}_{\rm val}(W_{1}^{*},W_{2}^{*})$ because $W_{2}^{*}\in\mathcal{S}(W_{1}^{*})$ is feasible and $\|W_{2}^{*}-\widetilde{W}_{2}^{*}\|\leq\epsilon$ so that $\mathsf{L}_{\rm val}(W_{1}^{*},W_{2}^{*})$ is closed to the unconstrained minimal value $\mathsf{L}_{\rm val}(W_{1}^{*},\widetilde{W}_{2}^{*})$ which is also small. As $\mathsf{L}_{\rm val}(W_{1}^{*},W_{2}^{*})$ is only an estimate of $\mathsf{L}_{\rm val}^{*}$, there exists cases where $\mathsf{L}_{\rm val}(W_{1},W_{2})-\mathsf{L}_{\rm val}(W_{1}^{*},W_{2}^{*})<0$, so in practice, we use $|\mathsf{L}_{\rm val}(W_{1},W_{2})-\mathsf{L}_{\rm val}(W_{1}^{*},W_{2}^{*})|$ to estimate the upper-level relative error. Since the convergence of lower-level relative error suggests $W_{2}\rightarrow W_{2}^{*}$, the convergence of upper-level relative error will indicate that $W_{1}\rightarrow W_{1}^{*}$.

We select the best stepsizes $\alpha,\beta$ and the number of inner loop $T_{k}=T$ by grid search. It can be observed from Figure [3](#S7.F3) that PBGD converges almost at a linear rate to a certain accuracy, and the relative error decreases as $\gamma$ increases. The fluctuation in the upper-level error near convergence is due to the global convergence result in Theorem [1](#Thmtheorem1) not being strictly decreasing because of the ${\cal O}(\epsilon)$ error at each step. When upper-level and lower-level relative errors are sufficiently small, the additional ${\cal O}(\epsilon)$ error has larger impact, leading to a slight increase in error. However, the final error remains small, around $10^{-5}$ when $K$ and $\gamma$ is large enough. This validates Theorem [1](#Thmtheorem1) that by setting $\gamma$ large enough, PBGD globally converges to a target accuracy determined by $\gamma$ at almost linear rate.

#### H.2 Data hyper-cleaning

Considering the overparameterized linear regression with a small clean validation dataset and a large dirty training dataset, we choose $N=100,N^{\prime}=10,m=200,n=10$. First, we respectively generate data matrix $X_{\rm trn}\in\mathbb{R}^{N\times m},X_{\rm val}\in\mathbb{R}^{N^{\prime}\times m}$ from Gaussian distribution $\mathcal{N}(5,0.01)$ and $\mathcal{N}(-3,0.01)$ to model different cluster of data. Then we generate optimal clean weight $W^{*}\in\mathbb{R}^{m\times n}$ from Gaussian distribution $\mathcal{N}(1,0.01)$ and generate the clean label matrix for validation dataset as $Y_{\rm val}\sim\mathcal{N}(X_{\rm val}W^{*},0.001)$. For the training label matrix, we first generate optimal classification parameters $\psi(u_{i})\sim\operatorname{Bernoulli}(0.2)$ and then generate the label matrix as

$$ $Y_{\rm trn}\sim\mathcal{N}(X_{\rm trn}W^{*},0.01)+\psi(u)\odot\mathcal{N}(10,10)$ $$

where $\psi(u)=[\psi(u_{1});\psi(u_{2});\cdots;\psi(u_{N})]$ and $\odot$ denotes the Hadamard product. This ensures the training dataset is polluted with probability $0.2$.

We run PBGD in Algorithm [2](#alg2) on this synthetic data hyper-cleaning problem and plot the upper-level and lower-level relative errors versus iteration in Figure [3](#S7.F3). We measure the lower-level relative error by $\ell_{\rm trn}(u,W)-\ell_{\rm trn}^{*}(u)$ with closed form in Lemma [23](#Thmlemma23) and the upper-level relative error by $\ell_{\rm val}(W)-\ell_{\rm val}^{*}$ where $\ell_{\rm val}^{*}=\min_{u,W\in\mathcal{S}(u)}\ell_{\rm val}(W)$. We estimate $\ell_{\rm val}^{*}\approx\min_{W}\ell_{\rm val}(W)=\ell_{\rm val}(W^{*})$ because there exists $u$ such that the selected data matrix $\sqrt{\psi_{N}(u)}X_{\rm trn}$ is almost full rank so that selected training dataset and validation dataset share a joint minimizer $W^{*}$.

We select the best stepsizes $\alpha,\beta,\tilde{\beta}$ and the number of inner loop $T_{k}=T$ by grid search. It can be observed from Figure [3](#S7.F3) that PBGD converges almost at a linear rate to a certain accuracy, and the relative error, especially at the lower level, decreases as $\gamma$ increases. The final error remains small, around $10^{-5}$ when $K$ and $\gamma$ is large enough. This coincides with our Theorem [1](#Thmtheorem1) that PBGD globally converges to a target accuracy inversely determined by $\gamma$ at almost linear rate. Furthermore, the lower-level relative error is more sensitive to the choice of $\gamma$, as noted in Theorem [1](#Thmtheorem1), where the lower-level relative error $\epsilon_{\gamma}$ is inversely related to $\gamma$.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: In Line 1226, why is the blockwise PL condition of $L_\gamma$ over $u$ sufficient to ensure the PL condition for $L^*_\gamma$?
soundness: 3
strengths: 1.	The study of convergence of bilevel algorithms to global solutions is an interesting topic, and this paper offers an approach.
2.	The paper includes concrete application examples that validate the assumptions necessary for establishing global convergence results.
summary: This paper studies the convergence properties of a penalized bilevel gradient descent (PBGD) algorithm, aiming to obtain global optimal solutions of bilevel optimization problems under the joint and blockwise Polyak-Łojasiewicz (PL) conditions. The joint and blockwise PL conditions are validated in the context of two specific applications: representation learning and data hyper-cleaning. Numerical experiments are provided to substantiate the theoretical results.
weaknesses: 1.	While the topic of global optimal convergence in bilevel optimization is engaging, the approach presented in this work does not appear as innovative as suggested by the title. The main idea relies on the joint/blockwise PL condition of the penalized objective $L_\gamma$. However, it is well known that when the PL condition holds, any stationary point is globally optimal, and the proximal-gradient method can achieve linear convergence to this global optimum (see, e.g., Hamed Karimi, Julie Nutini, and Mark Schmidt, Linear Convergence of Gradient and Proximal-Gradient Methods under the Polyak-Łojasiewicz Condition, ECML PKDD 2016). Furthermore, the convergence of PBGD to a stationary point of $L_\gamma$ under the PL condition has been well studied in existing literature (e.g., Bo Liu, Mao Ye, Stephen Wright, Peter Stone, BOME! Bilevel Optimization Made Easy: A Simple First-Order Approach, NeurIPS 2022, and Shen, Han, and Tianyi Chen, On Penalty-Based Bilevel Gradient Descent Method, ICML 2023). Thus, the approach in this work may lack novelty, and the contribution seems somewhat incremental.
2.	Although the authors have put considerable effort into verifying that the joint/blockwise PL condition can be satisfied in specific applications, such as representation learning and data hyper-cleaning, only very restricted cases are analyzed, with strong assumptions imposed. For instance, Assumption 2 in the representation learning setting and the assumption $X_{trn}X_{trn}^{\dagger}$ is a diagonal matrix in data hyper-cleaning narrow the applicability of the results and limit their general applicability. The theoretical analysis appears heavily dependent on these assumptions, raising doubts about whether the joint/blockwise PL condition would hold in broader or more practical cases, or even in other bilevel optimization applications.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How should one choose between joint and blockwise PL conditions for a given application?
1. Could you please clarify which aspects of the convergence results would generalize to more complex settings like non-linear models?
1. What practical takeaways does this work provide for achieving global convergence in more complex bilevel applications?
1. How robust are the convergence results if the PL conditions are only approximately met?
soundness: 3
strengths: The main strength is that it is a pioneering work that studies the challenging and important problem of global convergence in bilevel optimization, a topic with substantial real-world relevance. The proposed analysis extends PL to both joint and blockwise PL conditions and verifies them on two application cases. Overall, the paper is well-organized and easy to follow.
summary: This paper presents a theoretical framework for achieving global convergence in bilevel optimization. The authors propose that a constrained reformulation generally yields a benign landscape, and they analyze the global convergence of penalized bilevel gradient descent (PBGD) algorithm for bilevel objectives under the proposed joint and blockwise PL conditions. The paper illustrates that the specific applications of representation learning and data hyper-cleaning can satisfy these PL conditions. Theoretical results are then supported by experiments conducted on these applications.
weaknesses: I have several concerns and comments on the submission (please correct me if I am wrong):

1. The applicability of the developed theorem seems unclear. The proof closely dependent on and follow existing convergence theorems for PBGD, and it’s unclear whether the analysis could extend to other bilevel algorithms. The non-additivity of PL conditions poses a great challenge for applying the developed theorem and no practical solutions are provided. The two applications studied rely on linear models and strong convexity of loss, which is overly idealized and simplified.

1. Moreover,  in line 228 (Section 3), the authors mention that convergence analysis may need “fine-tuning per application,” but it remains unclear which parts of the analysis are generally hold, such as whether the iteration complexity $𝑂(log⁡(𝜖^{−1}))$ generally holds to other settings that satisfy PL conditions. It also mention that "This may also shed light on a broader range of bilevel problems involving sophisticated neural network architectures in machine learning", but the paper lacks clearly summaries practical takeaways got from the developed theorem for achieving global convergence in such complex applications with modern non-linear deep models.

1. The numerical analysis lacks depth and discussion on robustness. I am suggesting throughly evaluating how values of parameters $\alpha$, $\beta$, $\gamma$ are set theoretically as well as practically, and whether the observed results match theoretical expectations on the convergence rate. Also, exploring how slight violations of PL conditions affect convergence would help clarify the robustness.

1. Section 2 provides an example to illustrate the complexity of the nested objective $𝐹(𝑢)$ caused by the lower-level mapping $𝑆(𝑢)$, but it lacks rigorous analysis of how the constrained formulation reliably produces a more benign landscape and to what extend. A precise definition of benign landscape in the context of bilevel optimization is also helpful. The conclusion that constrained reformulation yields a benign landscape relies heavily on prior literature (lines 211-215) rather than in-depth analysis in this paper.

1. In line 373 (page 7), matrix $𝑊_3$ is introduced without a clear explanation.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. Could the authors expand Section 1.1 with detailed theorems? The sentence following C3, “The joint and blockwise PL condition… are not assumptions, but the properties of the penalty reformulation,” is confusing. The authors should clarify the assumptions needed to establish global convergence rigorously.

2. In what specific way is “global optimality” used in the paper?
soundness: 3
strengths: The paper offers conditions that ensure global convergence in bilevel optimization by generalizing the Polyak-Lojasiewicz (PL) condition.
summary: The paper explores global convergence in bilevel optimization, a crucial yet challenging objective due to the non-convexity and potential for multiple local solutions in bilevel problems. To address this, the authors propose sufficient conditions for global convergence and illustrate these in bilevel learning applications such as representation learning and data hyper-cleaning.
weaknesses: While global optimality is underscored as essential, the precise definition or context of “global optimality” within this framework is unclear. A clear explanation of how this term is specifically applied in their method would strengthen the paper.

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 3
questions: Can it be applied to a more general bi-level optimization with constraints in (1)?
soundness: 4
strengths: Achieving Global optimality is an important property.
summary: The paper proposed two PL conditions, by satisfying which the global optimality of bi-level optimization can be achieved using simple algorithms like Gauss-Seidel.
weaknesses: The paper's assumptions are very restrictive. For most bilevel optimization problems, the Joint and blockwise PL conditions cannot be guaranteed, and even checking these conditions can be challenging.  The representative problems illustrated in the paper are very specific simple cases. For example, only linear models can satisfy the assumption for representation learning.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. What does the a pilot mean in the title?

2. Line 057, a benign landscape, is there a direct meaning for that?

3. Line 53, the  goal of this paper, this sentence is not important. Do not need to emp{}. 

4. The numerical results seem too little? Does the proposed method outperform SOTA bi-level methods?

5. What are the best convergence results for bi-level optimization method before this paper?

6. line 414, what does \gamma to be O(\epsilon^{-0.5}) mean? If gamma is very very large (with a very large constant), can the algorithm still converge? What is the meaning of O(xx) here?

7. Will PL condition a bit too strong?
soundness: 3
strengths: good. This paper is overall well written and provide plenty of theoretical results.

The proposed method also solves the neural network cases. That's especially good.
summary: This paper propose a new bilevel optimization algorithm. This paper is generally very well written and provide plenty of theoretical results. Overall this paper is clear a good paper. If all these results are correct, this paper should be clearly accepted (However, I am inadequate to go through all proofs).
weaknesses: 1. Experiments are not adequate. 

2. Some fonts seem strange.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1.	Major Concerns:

(a) In line 261, Danskin theorem is mentioned, then the gradient is calculated. Also, the variable $\omega$ is introduced later. I think it would be better to explain the connection and point out that the using Danskin theorem, the auxiliary variable $\omega$ will help us to find a good estimation of the gradient with respect to $u$.

(b) It may be better to put Algorithm 1 and 2 on Page 6 after the authors have summary these algorithms. It will give the readers a smooth reading experience.

(c) In section 6, you may want to specific the choice of $\alpha$ and $\beta$ and make sure that they satisfied the conditions stated in Theorem 2 and 3.

(d) If possible, adding more baseline methods would help readers better understand the convergence rate of the PBGD method. This is not necessary given the limited time.

2.	Minor Concerns:

(a) The sentence in line 199 is not very clear, please double check.

(b) There’s a “?” in line 309, please make sure it is correct.

(c) Misspelling in Line 973 and Line 2189. “invertiable” to “invertible”.
soundness: 4
strengths: Clear Problem Statement: The authors articulate the limitations of existing methods, particularly those that only guarantee convergence to local minima or stationary points, which motivate them for pursuing global convergence.

Timeliness and Relevance: The paper proof the global convergent rate for a certain type of bilevel optimization problems. Given the increasing application of bilevel optimization in machine learning and high-stakes fields, this work has substantial relevance.

Theoretical Contribution: The authors provide sufficient conditions for achieving global optimality. By leveraging the penalty reformulation approach, the paper establishes an almost linear global convergent rate for some linear bilevel optimization problems.

Experimental Validation: The empirical results test on bilevel learning problems like representation learning and data hyper-cleaning. The preliminary computational results support the almost linear convergence theorem.
summary: This paper studies the global convergence rate of bilevel optimization. The main result is that if the penalized objective satisfies the PL condition, then the bilevel problem have almost linear global convergence rate if PBGD method is used to solve the problem. Then the authors give two applications: representation learning and data hyper-cleaning. These problems can be formulated as bilevel optimization problems, and their penalized objectives satisfy the PL condition. Thus, when applying PBGD algorithm, they should converge almost linearly. The preliminary computational results also support the theorem.
weaknesses: Assumptions and Limitations: While the paper claims global convergence for bilevel problems, it focuses primarily on linear models. Expanding the theoretical foundation to nonlinear models or other loss functions would improve the paper’s generalizability.

Comparative Analysis: While the paper mentions other approaches, a direct empirical comparison with state-of-the-art methods for bilevel optimization would strengthen its validation.

Connection between Theory and Experiment: the author should clearly specified the connections between the theory and experiment so that the experimental results can support the theory. For example: in section 6, the author should specific the choice of the step length and make sure that they satisfied the conditions stated in Theorem 2 and 3.

Meta-review: While the reviewers have expressed concerns about some of the applicability of the assumptions and the value of the numerical studies, the authors have substantively resolved most of these concerns during the rebuttal period, and the least positive reviewers have failed to comment on whether their concerns have been resolved. After reviewing the paper personally, I believe there are fundamental contributions to the field of bilevel optimization that are worthy of publication at this time.

Decision: Accept (Poster)