SlowGuess/NeurIPS_2023_OCR huggingface dataset:
IBM Research, Thomas J. Watson Research Center
Yorktown Heights, NY 10598
Functional constrained optimization (FCO) has emerged as a powerful tool for solving various machine learning problems. However, with the rapid increase in applications of neural networks in recent years, it has become apparent that both the objective and constraints often involve nonconvex functions, which poses significant challenges in obtaining high-quality solutions. In this work, we focus on a class of nonconvex FCO problems with nonconvex constraints, where the two optimization variables are nonlinearly coupled in the inequality constraint. Leveraging the primal-dual optimization framework, we propose a smoothed first-order Lagrangian method (SLM) for solving this class of problems. We establish the theoretical convergence guarantees of SLM to the Karush-Kuhn-Tucker (KKT) solutions through quantifying dual error bounds. By establishing connections between this structured FCO and equilibrium-constrained nonconvex problems (also known as bilevel optimization), we apply the proposed SLM to tackle bilevel optimization oriented problems where the lower-level problem is nonconvex. Numerical results obtained from both toy examples and hyper-data cleaning problems demonstrate the superiority of SLM compared to benchmark methods.
In this paper, we consider a structured constrained optimization framework that has broad applications in diverse machine learning problems. The goal of this problem is to minimize the objective function using two blocks of optimization variables which are coupled in the functional constraint. More formally, we express this class of nonlinear constrained programming problems as follows:
$$
\min  _ {x \in \mathcal {X}, y} f (x, y), \quad \text {s . t .} \quad g (x, y) - g ^ {*} (x) \leq \delta \tag {1}
$$
where feasible set  $\mathcal{X}$  is convex and compact, the objective function  $f(x,y)$  is smooth and nonconvex, constraint  $g(x,y)$  is smooth and nonconvex and it also satisfies the Polyak-Lojasiewicz (PL) condition with respect to (w.r.t.)  $y$ ,  $g^{*}(x) \triangleq \min_{y} g(x,y)$  (which is also called value function [1]) is obtained by minimizing  $g(x,y)$  over  $y$ , and constant  $\delta > 0$ .
When  $\delta = 0$ , problem (1) reduces to the classic bilevel optimization (BO) problem (aka mathematical programs with equilibrium constraints [2]). More specifically, it takes the form of
$$
\min  _ {x \in \mathcal {X}, y} f (x, y), \quad \text {s . t .} \quad y \in \mathcal {S} (x) \triangleq \underset {y ^ {\prime}} {\arg \min } g (x, y ^ {\prime}), \tag {2}
$$
where  $f(x,y)$  and  $g(x,y')$  denote the upper-level (UL) and lower-level (LL) objective functions,  $S(x)$  represents the set that contains the global optimal solutions of the LL problem w.r.t. the block-y'. This BO formulation has been shown to be a useful framework for modeling various multi-task
Table 1: Comparison of existing related representative works on nonconvex functional constrained and/or bilevel optimization, where non-singleton: the existence of multiple global optimal solutions at the LL problem, oracle: requirements of accessing first-order and/or second-order derivative of either UL or LL loss functions, LL/C: the property that the LL objective functor/constraint satisfies in problem (2) or (1),  $\widetilde{\mathcal{O}}$ : denotes that the iteration complexity hides the dependence on the hyperparameters, e.g.,  $\delta$ , cvx: convex, ncvx: nonconvex, scvx: strongly convex.
Algorithms	Problem	Opt. Framework	Oracle	Non-Singleton	LL/C	Rate
ConEx [12]	primal-dual	first-order	✓	ncvx	O(1/ε3/2)
ITD/AID [13]	(2)	ITD/AID	second-order	scvx	O(1/ε)
F2SA [14]	(2)	penalty	first-order	scvx	O(1/ε3/2)
BOME [15]	(2)	penalty	first-order	✓	PL	O(1/ε3)
V-PBGD [16]	(2)	penalty	first-order	✓	PL	O(1/ε4)
SLM	(1)	primal-dual	first-order	✓	PL	O(1/ε)
machine learning problems, e.g., hyper-parameter optimization [3, 4], actor-critic schemes in reinforcement learning [5], meta-learning [6, 7], AUC maximization [8], distributed bilevel optimization [9, 10, 11], etc. In these scenarios, the decision/optimization variables are coupled in both levels of the minimization problems and the evaluation of the UL gradient depends on the optimal solution at the LL. Consequently, it is necessary to optimize the UL and LL loss functions jointly w.r.t. the block variables  $x$  and  $y$ .
Bilevel Optimization. Motivated by the numerous applications of BO, developing the efficient solvers for handling these problems has attracted great interest in the optimization community. There are two major directions: 1) focus on solving the BO problem in a general manner; 2) improve the computation or iteration complexity of the BO algorithms. It has been shown in [17] that a first-order method can find the global optimal solution when the UL objective function is strongly convex and the LL problem is convex. However, the strong convexity assumption restricts the applicability of the algorithm to many real-world machine learning problems. In contrast, some recent works have considered the case where the UL objective function is nonconvex and have shown that, given an oracle that computes the Jacobian and the inverse of Hessian matrices of the LL objective function, the complexity of finding an  $\epsilon$ -stationary point is  $\mathcal{O}(\epsilon^{-1})$  [18, 13], which is the same as that of gradient descent. But this setting is still restrictive as they still assume that the LL problem is strongly convex.
Non-Singleton at the LL. When the LL problem is not strongly convex, the computation of the UL gradient  $\nabla f(x,S(x))$  can become ill-posed due to the presence of multiple optimal solutions. To address this issue, a new metric called  $(\delta ,\epsilon)$ -Goldstein stationary point was introduced in [19], which aims to capture the discontinuity of  $\nabla f(x,S(x))$ . Under the assumption that  $f(x,\cdot)$  and  $g(x,\cdot)$  are convex, it was shown that an inexact gradient-free method can find the Goldstein stationary point at a rate of  $\mathcal{O}(\delta^{-1}\epsilon^{-4})$ . In addition to the new metric used for theoretical convergence analysis, the concept of Hölderian error bound was employed in [20], which imposes further conditions on the shape of the LL objective function. For a class of simple BO problems where only one set of block variables needs to be optimized, the conditional gradient-based bilevel optimization (CG-BiO) algorithm proposed in [20] was shown to converge to stationary points at a rate of  $\mathcal{O}(\epsilon^{-1})$  or  $\mathcal{O}(\epsilon^{-2})$ , depending on whether the Hölderian error bound holds.
In [21], the author considered the case of a nonconvex LL objective function but with an assumption that eliminates the presence of zero eigenvalues of the LL Hessian matrix. Another property used to characterize the landscape of the LL function is the PL condition. Based on the formulation in (1), researchers [16, 15] developed penalty methods for solving BO problems and discussed the relationship between local/global optimal solutions of the penalized problem formulation and the constrained problem (2) (i.e., when  $\delta = \epsilon$  in (1)). It was shown that the value-gap-based penalty-based bilevel gradient descent (V-PBGD) algorithm [16] can converge to stationary points of the reformulated single-level problem. Additionally, under the assumption of the constant rank con
strain quantification condition, the bilevel optimization made easy (BOME) algorithm developed in [15] can achieve Karush-Kuhn-Tucker (KKT) points of the BO problem in (2).
Computational Issue. Another line of research focuses on reducing the computational complexity of implementing BO optimization algorithms. As mentioned earlier, computing the UL gradient can be computationally expensive. Existing works have employed strategies such as the Neumann series for calculating the inverse of Hessian matrices [18], or utilized reverse-mode iterative differentiation (ITD) and approximate implicit differentiation (AID) techniques [22, 13] for hypergradient computation. However, these strategies introduce a double-loop structure in the algorithm design, requiring users to specify hyperparameters for the inner loop subroutine. Subsequently, single-loop BO algorithms have been developed [23, 24], which alleviate the burden of hyperparameter tuning in the inner loop. Nevertheless, they still require the computation of the Jacobian matrix and Hessian-vector product.
To circumvent these computations, some existing works reformulate the BO problem as a single-level functional constrained optimization (FCO) problem, as shown in (1), and apply well-established constrained optimization techniques, such as primal-dual methods, for solving the BO problem using only gradients. In [25], it was shown that when the UL problem is strongly convex and the LL problem is convex, the primal-dual algorithm can find the global optimal solution at a rate of  $\mathcal{O}(\epsilon^{-1/2})$ . For the case where the UL problem is nonconvex, the fully first-order stochastic approximation  $(\mathrm{F}^2\mathrm{SA})$  method proposed in [14] can find an  $\epsilon$ -stationary solution at a rate of  $\mathcal{O}(\epsilon^{-3/2})$  when the LL problem is strongly convex. Similarly, penalty methods [26, 16] also just require the computation of UL and LL gradients while achieving the desired optimization results.
Functional Constrained Optimization. The formulation based on function values establishes a connection between FCO and BO [1, 27, 28]. By leveraging existing well-developed constrained optimization methods [12, 29, 30], efficient solutions to the BO problem can still be obtained. When the block variables  $x$  and  $y$  are linearly coupled, the alternating direction method of multipliers (ADMM) [31] has emerged as a successful approach by decoupling the optimization process into subproblems. Recent advancements in smoothed augmented Lagrangian methods [30, 32, 33] have improved the convergence rates for solving nonconvex linearly constrained problems. For nonlinear functional constraints with one block of variables, the constraint extrapolation (ConEx) method [12] has shown promising results in finding KKT points with a convergence rate of  $\mathcal{O}(\epsilon^{-3/2})$  for nonconvex smooth functions. Besides, the properties of PL functions studied in min-max optimization problems [34] are expected to be useful in demonstrating the continuity of  $\nabla f(x, S(x))$  for BO problems within the framework of primal-dual methods. A comparison between our work and closely related previous works on FCO and BO is shown in Table 1.
1.2 Main Contributions of This Work
In this work, we propose a novel approach called the smoothed first-order Lagrangian method (SLM) to tackle the FCO problem presented in (1). The key advantage of SLM is that it only requires the computation of gradients, making it computationally efficient. Remarkably, under mild assumptions and a local regularity condition, we establish a strong error bound for SLM to solve this class of problems, and prove that SLM can converge to the KKT points even when the nonconvex objective in the constraint has multiple optimal solutions. To summarize, the main contributions of this work are highlighted as follows:
This class of structured FCO problems is generic and can be used to formulate multiple machine learning problems. The proposed SLM is computationally efficient as it relies only on the gradient of the objective functions. Moreover, the obtained convergence rate of the proposed SLM, which is  $\mathcal{O}(\epsilon^{-1})$ , is optimal for finding the  $\epsilon$ -KKT points of this class of problems.
SLM can be applied to solve nonconvex bilevel related optimization problems in the presence of nonconvexity and non-singleton at the lower level, when the LL objective function satisfies the PL condition.
The numerical results showcase the competitiveness of the proposed SLM compared to state-of-the-art BO algorithms in both a toy example and a data hyper-cleaning problem w.r.t. convergence speed and achievable test accuracy.
Due to space constraints, all technical proofs are provided in the supplementary material.
Notations: The distance between  $y$  and the set  $\mathcal{S}(x)$  is defined as  $d_{\mathcal{S}(x)}(y) \triangleq \min_{y' \in \mathcal{S}(x)} \| y - y' \|$ . A high-dimensional ball with a radius of  $R$  is denoted as  $\mathcal{B}(R)$ .
2 Smoothed First-Order Lagrangian Method
In this section, we will introduce the three-layer structure of our first-order method designed to solve problem (1). Towards this end, we can write the Lagrangian of this constrained problem as follows:
$$
\mathcal {L} (x, y; \lambda) \triangleq f (x, y) + \lambda \left(g (x, y) - g ^ {*} (x) - \delta\right) \tag {3}
$$
where the nonnegative  $\lambda$  denotes the Lagrange multiplier or dual variable.
Next, it is natural to have
$$
\nabla_ {x} \mathcal {L} (x, y; \lambda) = \nabla_ {x} f (x, y) + \lambda (\nabla_ {x} g (x, y) - \nabla_ {x} g ^ {*} (x)), \tag {4a}
$$
$$
\nabla_ {y} \mathcal {L} (x, y; \lambda) = \nabla_ {y} f (x, y) + \lambda \nabla_ {y} g (x, y), \tag {4b}
$$
$$
\nabla_ {\lambda} \mathcal {L} (x, y; \lambda) = g (x, y) - g ^ {*} (x) - \delta . \tag {4c}
$$
In contrast to the classical primal-dual algorithm design, here,  $g^{*}(x)$  is typically unknown. Based on the definition of  $g^{*}(x)$ , it is not hard to know that obtaining a closed-form expression for  $\nabla g^{*}(x)$  requires the computation of the Jacobian matrix and the inverse of Hessian matrices even for the case when  $g(x,y)$  is strongly convex w.r.t.  $y$ . Although many existing works have provided the iterative methods of computing these quantities for each iteration [13, 23, 24], the calculational cost and/or memory budget are still high for large-scale problems. Hence, it is motivated to have an efficient way of approximating the value and gradient of  $g^{*}(x)$  in the algorithm design.
2.1 Bottom-Layer
One of the most straightforward approaches is to generate a sequence  $u_{t}$  indexed by  $t$  as an inner loop to obtain  $u^{*}(x) \triangleq \arg \min_{u} g(x, u)$ . When the objective function  $g(x, y)$  satisfies certain properties, such as the PL condition or the strongly convex condition, applying gradient descent can lead to finding the global optimal solution of this subproblem with a linear convergence rate [35].
Let  $r$  denote the index of the outer iteration steps. Then, the approximate of  $u^{*}(x)$  at each iteration, i.e.,  $u^r$ , can be estimated through the following process:
$$
u _ {t + 1} ^ {r} = u _ {t} ^ {r} - \gamma \nabla_ {u} g \left(x ^ {r}, u _ {t} ^ {r}\right), \tag {5}
$$
where  $u_{t}^{r}$  represents the  $t$ th iterate of the inner loop,  $\gamma$  denotes the step-size, and  $u_{0}^{r}$  is initialized within a bounded ball (which will ensure the boundedness of the iterates). After running this algorithm for  $T^{r}$  iterations at each step, we set  $u^{r + 1} = u_{T^r}^r$  for updating the subsequent optimization variables.
2.2 Medium-Layer
Given that the quality of  $g^{*}(x)$  can be guaranteed, the most interesting part lies in solving the constrained problem w.r.t. block  $y$ . Even if we assume that  $g(x,y)$  follows the PL condition, the nonconvexity of  $f(x,y)$  still poses challenges for the first-order methods in finding the optimal solution. To address this, we propose adding the quadratic proximal terms that renders the Lagrangian w.r.t.  $x$  and  $y$  strongly convex, as follows:
$$
K (x, y, w, z; \lambda) \triangleq \mathcal {L} (x, y; \lambda) + \frac {p}{2} \| x - w \| ^ {2} + \frac {p}{2} \| y - z \| ^ {2}. \tag {6}
$$
Here,  $K(x,y,w,z;\lambda)$  can be regarded as a modified Lagrangian for this problem. Let  $L_{f}$  and  $L_{g}$  denote as the smoothness constants of  $f$  and  $g$ , respectively. Choosing a sufficiently large value for  $p$  (specifically,  $p = \Omega(L_{f} + \lambda L_{g})$ ) ensures that  $K(x,y,w,z;\lambda)$  becomes strongly convex.
Subsequently, we can develop the following gradient-based primal-dual algorithm for solving this constrained subproblem:
$$
\lambda^ {r + 1} = \mathcal {P} _ {+} \left[ \lambda^ {r} + \tau \nabla_ {\lambda} \widehat {K} \left(x ^ {r}, y ^ {r}, w ^ {r}, z ^ {r}, u ^ {r + 1}; \lambda^ {r}\right) \right], \tag {7a}
$$
$$
y ^ {r + 1} = y ^ {r} - \alpha \nabla_ {y} K \left(x ^ {r}, y ^ {r}, w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), \tag {7b}
$$
$$
z ^ {r + 1} = z ^ {r} + \beta \left(y ^ {r + 1} - z ^ {r}\right), \tag {7c}
$$
where  $\widehat{K}(x,y,w,z,u;\lambda) \triangleq \widehat{\mathcal{L}}(x,y,u;\lambda) + \frac{p}{2}\|x - w\|^2 + \frac{p}{2}\|y - z\|^2$  and  $\widehat{\mathcal{L}}(x,y,u;\lambda) \triangleq f(x,y) + \lambda(g(x,y) - g(x,u) - \delta)$ ,  $\tau, \alpha, \beta$  are the step-sizes, and  $\mathcal{P}_{+}$  denotes the projection of the dual variable onto a box constraint. In particular,  $\mathcal{P}_{+}[\lambda]$  is given by  $\mathcal{P}_{+}[\lambda] = 0$  if  $\lambda \leq 0$ ;  $\mathcal{P}_{+}[\lambda] = \lambda$  if  $0 < \lambda \leq \Lambda$ ;  $\mathcal{P}_{+}[\lambda] = \Lambda$  if  $\lambda > \Lambda$ .
Remark 1. The auxiliary sequence  $z^r$  is defined as an exponentially weighted average sequence of  $y^{r+1}$ , which has been previously used in smoothed gradient descent ascent for nonconvex min-max optimization [36, 37] problems and in the smoothed proximal augmented Lagrangian method for linearly constrained nonconvex objective problems [32]. However, to the best of our knowledge, this is the first time that the smoothed primal-dual method is applied to solve nonconvex objective problems with nonlinear functional constraints.
2.3 Top-Layer
After updating the variables  $y, z, u$ , and  $\lambda$ , the variables  $x$  and  $w$  are updated as follows:
$$
x ^ {r + 1} = \mathcal {P} _ {\mathcal {X}} \left[ x ^ {r} - \eta \nabla_ {x} \widehat {K} \left(x ^ {r}, y ^ {r + 1}, w ^ {r}, z ^ {r + 1}, u ^ {r + 1}; \lambda^ {r + 1}\right) \right], \tag {8a}
$$
$$
w ^ {r + 1} = w ^ {r} + \beta \left(x ^ {r + 1} - w ^ {r}\right), \tag {8b}
$$
where  $\mathcal{P}_{\chi}$  denotes the projection operator, and  $\eta$  is the step-size.
The detailed algorithm description is provided in Algorithm 1.
Algorithm 1 Smoothed first-order Lagrangian Method (SLM)
Initialization: step-sizes:  $\tau, \eta, \alpha, \beta, \gamma$ , variables:  $x^1, w^1, y^1, z^1, \lambda^1$   
1: for  $r = 1, 2, \dots, T$  do  
2: for  $t = 0, 1, 2, \dots, T^r - 1$  do  
3: Initialize  $u_0^r \in \mathcal{B}(R)$   
4:  $u_{t+1}^r = u_t^r - \gamma \nabla_u g(x^r, u_t^r)$   
5: end for  
6:  $u^{r+1} = u_{Tr}^r$   
7:  $\lambda^{r+1} = P_+ \left[ \lambda^r + \tau \left( g(x^r, y^r) - g(x^r, u^{r+1}) - \delta \right) \right]$   
8:  $y^{r+1} = y^r - \alpha \left( \nabla_y f(x^r, y^r) + \lambda^{r+1} \nabla_y g(x^r, y^r) + p(y^r - z^r) \right)$   
9:  $z^{r+1} = z^r + \beta (y^{r+1} - z^r)$   
10:  $x^{r+1} = \mathcal{P}_X[x^r - \eta (\nabla_x f(x^r, y^{r+1}) + \lambda^{r+1} (\nabla_x g(x^r, y^{r+1}) - \nabla_x g(x^r, u^{r+1})) + p(x^r - w^r)]$   
11:  $w^{r+1} = w^r + \beta (x^{r+1} - w^r)$   
12: end for
3 Theoretical Convergence Results
Before presenting the theoretical convergence guarantees of SLM, it is necessary to introduce the following main classes of assumptions. These assumptions are essential in establishing the descent properties of some quantifiable (potential) function, enabling SLM to converge to KKT points of this class of nonconvex FCO problems. More detailed definitions and properties regarding these assumptions are deferred to the supplement.
The assumptions are mainly related to the continuity and boundedness of the objective function, as well as the mathematical property of the constraint.
A1. (Smoothness) Assume that functions  $f(x,y), g(x,y)$  are differentiable and jointly smooth with constants  $L_{f}, L_{g}$  w.r.t. both  $x, y$ .
A2. (Compactness) The feasible set  $\mathcal{X}$  (projection friendly) is convex and compact.
A3. (Boundedness) Assume that the objective function  $f(x,y)$  is lower bounded and denoted as  $f$ .
A4. (Coercivity) The set  $\{y|f(x,y)\leq R,g(x,y) - g^{*}(x)\leq \delta \}$  is bounded for any  $R > 0$
A5. (PL condition) Function  $g(x,y)$  satisfies the PL condition, i.e., there exists a constant  $\mu_g$  such that  $\| \nabla_y g(x,y) \|^2 \geq 2\mu_g(g(x,y) - g^*(x)), \forall x,y$ .
Remark 2. Assumptions A1-A3 are standard in the optimization literature, while A4, also referred to as coerciveness, is a commonly used assumption in theoretical convergence analyses to guarantee boundedness of the iterates, as observed in methods like smoothed-GDA [36], ADMM [38], BO algorithms [39, Assumption 3], asynchronous algorithms [40, Assumption 5], etc. In practice, regularization terms are often included in the objective function, which can ensure that the level set is
bounded [41]. For instance, as suggested in [39], introducing a small  $\ell_2$ -penalty to a non-negative loss (e.g. cross-entropy or mean-squared loss) can establish boundedness for both the level set and the iterates. An example involving neural networks can be found in [42, Theorem 2]. A5 is a reasonable and practical assumption for neural network applications. Previous studies, such as [43], have demonstrated theoretically that when neural networks are overparametrized, the loss function satisfies the PL condition. Other examples include the nonconvex discounted return objective in reinforcement learning [44], and the loss function of a specific class of linear quadratic regulator models [45, Lemma 3].
Local Regularity Condition. Let  $\bar{x}^{*}(w,z),\bar{y}^{*}(w,z)$  be the KKT solution of problem  $\min_{x,y}f(x,y)+$ $\frac{p}{2}\| x - w\| ^2 +\frac{p}{2}\| y - z\| ^2$  , s.t.  $g(x,y)\leq g^{*}(x) + \delta$  . Then, there exist positive constants  $\underline{\delta}$  and  $\varrho$  such that when  $\| w - \bar{x}^{*}(w,z)\|^{2} + \| z - \bar{y}^{*}(w,z)\|^{2}\leq \varrho$  , the following inequality holds  $g(\bar{x}^{*}(w,z),\bar{y}^{*}(w,z))\geq g^{*}(\bar{x}^{*}(w,z)) + \underline{\delta} (w,z)$  , where  $\underline{\delta} (w,z)\geq \underline{\delta}$
Remark 3. Given the feasibility of achieving the global optimal solution of  $g^{*}(\bar{x}^{*}(w,z))$  and the requirement for variable  $y$  to minimize both objective functions, it is reasonable to expect that  $\bar{y}^{*}(w,z)$  can be close to  $\arg \min g(\bar{x}^{*}(w,z),y)$  up to some very small constant. It is worth noting that this assumption is fairly mild, as we only require this condition to hold in the neighborhoods of the KKT points.
Given these assumptions, we are now in a position to provide the following theoretical convergence guarantees for SLM.
3.2 Convergence Rates of SLM
Let  $\mathcal{G}(x^r, y^r)$  be defined as  $\mathcal{G}(x^r, y^r) \triangleq [\eta^{-1}(x^r - \mathcal{P}_{\mathcal{X}}(x^r - \eta \nabla_x \mathcal{L}(x^r, y^r; \lambda^r)); \nabla_y \mathcal{L}(x^r, y^r; \lambda^r)]$ . We denote  $\| \mathcal{G}(x^r, y^r) \|$  as the stationary gap.
Theorem 1. (Convergence Rate of SLM to the KKT Points of problem (1)) Suppose that A1-A5 are satisfied, and the local regularity condition holds. Assume that the iterates  $\{x^r,y^r,w^r,z^r,u^r,\lambda^r\}$  are generated by SLM. If the step-sizes are chosen as  $\eta ,\alpha \sim \mathcal{O}(1 / \lambda^r)$ $\gamma = \mathcal{O}(1 / L_g)$ $p = \Theta (\lambda^r)$ $\tau = \mathcal{O}(1)$ $\beta = \mathcal{O}(\delta^{1.5})$  when  $0 <   \delta <  1$  , and  $\beta = \mathcal{O}(1)$  when  $\delta >1$  , and  $T^r = \Omega (\log (r\lambda^r))$ $\Lambda \sim \Theta (1 / \sqrt{\delta})$  , then,  $\varrho = \mathcal{O}(\sqrt{\delta})$  and the following results hold
A1. Every limit point of  $\{x^r, y^r, w^r, z^r, u^r, \lambda^r\}$  generated by SLM is a KKT point of problem (1).  
A2.
$$
\frac {1}{T} \sum_ {r = 1} ^ {T} \| \mathcal {G} \left(x ^ {r}, y ^ {r}\right) \| ^ {2} = \mathcal {O} \left(\frac {1}{T}\right), \tag {9a}
$$
$$
\frac {1}{T} \sum_ {r = 1} ^ {T} \left| g \left(x ^ {r}, y ^ {r}\right) - g ^ {*} \left(x ^ {r}\right) - \delta \right| _ {+} ^ {2} = \mathcal {O} \left(\frac {1}{T}\right) \tag {9b}
$$