SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yankun Huang
Department of Business Analytics
Iowa City, IA 52242
yankun-huang@uiowa.edu
Qihang Lin
qihang-lin@uiowa.edu
We consider a non-convex constrained optimization problem, where the objective function is weakly convex and the constraint function is either convex or weakly convex. To solve this problem, we consider the classical switching subgradient method, which is an intuitive and easily implementable first-order method whose oracle complexity was only known for convex problems. This paper provides the first analysis on the oracle complexity of the switching subgradient method for finding a nearly stationary point of non-convex problems. Our results are derived separately for convex and weakly convex constraints. Compared to existing approaches, especially the double-loop methods, the switching gradient method can be applied to non-smooth problems and achieves the same complexity using only a single loop, which saves the effort on tuning the number of inner iterations.
Continuous optimization with nonlinear constraints arises from many applications of machine learning and statistics. Examples include Neyman-Pearson classification [62] and learning with fairness constraints [77]. In this paper, we consider the following general nonlinear constrained optimization
$$
f ^ {*} \equiv \min  _ {\mathbf {x} \in \mathcal {X}} f (\mathbf {x}) \quad \text {s . t .} \quad g (\mathbf {x}) \leq 0, \tag {1}
$$
where  $\mathcal{X} \subset \mathbb{R}^d$  is a closed convex set that allows a computationally easy projection operator,  $f$  is weakly-convex,  $g$  is either convex or weakly convex, and functions  $f$  and  $g$  are not necessarily smooth. When  $g(\mathbf{x}) \equiv \max_{i=1,\dots,m} g_i(\mathbf{x})$ , (1) is equivalent to an optimization problem with multiple nonlinear constraints  $g_i(\mathbf{x}) \leq 0$  for  $i = 1,\dots,m$ .
A weakly convex function can be non-convex, so computing an optimal solution of (1) is challenging in general even without constraints. For this reason, theoretical analysis for gradient-based algorithms for non-convex problems mostly focuses on an algorithm's (oracle) complexity for finding an  $\epsilon$ -stationary solution for (1). When a problem is non-smooth, finding an  $\epsilon$ -stationary solution is generally difficult even if the problem is convex [47]. Hence, in this paper, we consider finding a nearly  $\epsilon$ -stationary solution for (1), whose definition will be stated later in Definition 3.2.
In the past decade, there have been many studies on non-convex constrained optimization. However, most of the existing algorithms and their theoretical complexity analysis are developed by assuming  $f$  and  $g_{i}$ 's are all smooth or can be written as the sum of a smooth and a simple non-smooth functions. A non-exhaustive list of the works under such an assumption includes [34, 80, 81, 55, 44, 45, 43, 52, 63, 53, 50, 12, 22, 9, 21, 40, 51]. Their results cannot be applied to (1) due to non-smoothness in the problem.
Non-smoothness is common in optimization in machine learning, e.g., when a non-smooth loss function is applied, but there are much fewer studies on non-smooth non-convex constrained optimization. Under the weak-convexity assumption, an effective approach for solving a non-smooth non-convex problem with theoretical guarantees is the (inexact) proximal point method, where a quadratic proximal term is added to objective and constraint functions to construct a strongly convex constrained subproblem and then a sequence of solutions can be generated by solving this subproblem in exactly and updating the center of the proximal term. Oracle complexity for this method to find a nearly  $\epsilon$ -stationary has been established by [13, 54, 39] under different constraint qualifications.
The inexact proximal point method is a double-loop algorithm where the inner loop is another optimization algorithm for solving the aforementioned strongly convex subproblems. The complexity results in [13, 54, 39] require the inner loop solves each subproblem to a targeted optimality gap. However, the optimality gap is hard to evaluate and thus cannot be used to terminate the inner loop. Although the number of inner iterations needed to achieve the targeted gap can be bounded theoretically, the bound usually involves some constants that are unknown or can only be estimated conservatively. Hence, using the theoretical iteration bound to stop the inner loop usually leads to significantly more inner iterations than needed, making the whole algorithm inefficient. In practices, users often need to tune the number of inner iterations to improve algorithm's efficiency, which is an inconvenience common to most double-loop methods.
On the contrary, a single-loop algorithm is usually easier to implement as it does not require tuning the number of inner iterations. Therefore, the main contribution of this paper is showing that a single-loop first-order algorithm can find a nearly  $\epsilon$ -stationary point of (1) with oracle complexity  $O(1/\epsilon^4)$ , which matches state-of-the-art results obtained only by the double-loop methods [13, 54, 39]. The algorithm we study is the classical switching subgradient (SSG) method proposed by Polyak [59] in 1967, which is intuitive and easy to implement but has only been analyzed before in the convex case. We first show that the SSG method has complexity complexity  $O(1/\epsilon^4)$  when  $g$  is convex. Then, we show that the same complexity result holds when  $g$  is weakly convex and appropriate constraint qualification holds. We also present some practical examples where all of our assumptions hold, including a fair classification problem subject to the demographic parity constraint [1]. Our technical novelty is inventing a switching stepzize rule to accompany the switching subgradient. In particular, we use a fixed stepsize when the solution is updated by the objective's subgradient while use an adaptive Polyak's stepsize [33, 60] when the solution is updated by the constraint's subgradient. This allows us to leverage the local error bound of the constraint function to keep the solution nearly feasible during the algorithm, which prevents the solution from being trapped at an infeasible stationry point. To the best of our knowledge, this paper is the first to establish the complexity of a single-loop first-order method for weakly convex non-smooth nonlinear constrained optimization. In the appendix, we also extend our algorithms and complexity analysis to the stochastic setting.
Non-convex constrained optimization has a long history [36, 16, 29, 17, 30, 6, 18] and the interest on this subject has been still growing in the machine learning community because of its new applications such as learning with fairness constraints (see e.g., [77]).
In recent literature, the prevalent classes of algorithms for non-convex constrained optimization include the augmented Lagrangian method (ALM), the penalty method [37, 79, 80, 81, 38, 42, 55, 44, 45, 69, 43, 52, 63, 53, 50, 40, 51], and the sequential quadratic programming method [11, 34, 12, 22, 10, 9, 8, 21]. Besides, an inexact projected gradient method is developed by [15] and a level conditional gradient method is developed by [20]. However, these works all focus on the case where  $g$  is smooth and  $f$  is either smooth or equals  $f_{1} + f_{2}$ , where  $f_{1}$  is smooth and non-convex while  $f_{2}$  is non-smooth and convex and has a simple structure that allows a closed-form solution for the proximal mapping  $\arg \min_{\mathbf{y}} \{f_{2}(\mathbf{y}) + \frac{\rho}{2}\| \mathbf{y} - \mathbf{x}\|^{2}\}$  for any  $\mathbf{x}$ . There are relatively fewer works on non-convex non-smooth constrained problems. An alternating direction method of multipliers (ADMM) and an ALM are studied by [75] and [78], respectively, for non-convex non-smooth problems with linear constraints while our study considers nonlinear non-smooth constraints. The methods by [20] and [12] can be extended to a structured non-smooth case where  $f = f_{1} + f_{2}$  with  $f_{1}$  being smooth non-convex and  $f_{2} = \max_{\mathbf{y}}\{\mathbf{y}^{\top}A\mathbf{x} - \phi (\mathbf{y})\}$  with a convex  $\phi$ , and  $g$  has a similar structure. The method by [14] can handle a specific non-smooth non-convex constraint, i.e.,  $g(\mathbf{x}) = \lambda \| \mathbf{x}\| _1 - h(\mathbf{x})$
where  $h$  is a convex and smooth. Compared to these works, our results apply to a more general non-smooth problem without those structure assumptions.
When  $f$  and  $g$  in (1) are weakly convex and non-smooth, the inexact proximal point method has been studied by [13, 54, 39] under different constraint qualifications and different notions of stationarity. Their complexity analysis utilizes the relationship between the gradient of the Moreau envelope of (1) and the near stationarity of a solution, which is originally used to analyze complexity of subgradient methods for weakly convex non-smooth unconstrained problems [24, 25, 27, 2, 28, 61, 83]. Our analysis utilizes a similar framework. The methods [13, 54, 39] are double-loop while our algorithm only uses a single loop and achieves the same complexity of  $O(1 / \epsilon^4)$  as them under similar assumptions.
The SSG algorithm is first proposed by Polyak [59]. It has been well-studied for convex problems [58, 7, 49, 67, 72, 73, 65, 68, 66, 3] and quasi-convex problems [67]. This paper provides the first complexity analysis for the SSG method under weak convexity assumption. Non-smooth non-convex optimization has also been studied without weak convexity assumption by [82, 46, 64, 47, 19, 70, 71]. These works analyze the complexity of first-order methods for computing an  $(\epsilon, \delta)$ -Goldstein approximate stationary point, which is a more general stationarity notation than what we consider here. However, these works only focus on unconstrained problems.
3 Algorithm and near stationarity
Let  $\| \cdot \|$  be the  $\ell_2$ -norm. For  $h: \mathbb{R}^d \to \mathbb{R} \cup \{+\infty\}$ , the subdifferential of  $h$  at  $\mathbf{x}$  is
$$
\partial h (\mathbf {x}) = \left\{\boldsymbol {\zeta} \in \mathbb {R} ^ {d} \mid h \left(\mathbf {x} ^ {\prime}\right) \geq h (\mathbf {x}) + \langle \boldsymbol {\zeta}, \mathbf {x} ^ {\prime} - \mathbf {x} \rangle + o \left(\| \mathbf {x} ^ {\prime} - \mathbf {x} \|\right), \mathbf {x} ^ {\prime} \rightarrow \mathbf {x} \right\},
$$
and  $\zeta \in \partial h(\mathbf{x})$  is a subgradient of  $h$  at  $\mathbf{x}$ . We say  $h$  is  $\mu$ -strongly convex ( $\mu \geq 0$ ) on  $\mathcal{X}$  if
$$
h (\mathbf {x}) \geq h (\mathbf {x} ^ {\prime}) + \langle \boldsymbol {\zeta}, \mathbf {x} - \mathbf {x} ^ {\prime} \rangle + \frac {\mu}{2} \| \mathbf {x} - \mathbf {x} ^ {\prime} \| ^ {2}
$$
for any  $(\mathbf{x},\mathbf{x}^{\prime})\in \mathcal{X}\times \mathcal{X}$  and any  $\zeta \in \partial h(\mathbf{x}^{\prime})$  . We say  $h$  is  $\rho$  -weakly convex  $(\rho \geq 0)$  on  $\mathcal{X}$  if
$$
h (\mathbf {x}) \geq h (\mathbf {x} ^ {\prime}) + \langle \boldsymbol {\zeta}, \mathbf {x} - \mathbf {x} ^ {\prime} \rangle - \frac {\rho}{2} \| \mathbf {x} - \mathbf {x} ^ {\prime} \| ^ {2}
$$
for any  $(\mathbf{x},\mathbf{x}^{\prime})\in \mathcal{X}\times \mathcal{X}$  and any  $\zeta \in \partial h(\mathbf{x}^{\prime})$  . We denote the normal cone of  $\mathcal{X}$  at  $\mathbf{x}$  by  $\mathcal{N}_{\mathcal{X}}(\mathbf{x})$  and the relative interior of  $\mathcal{X}$  by relint(x). We say a point  $\mathbf{x}$  is  $\epsilon$  -feasible if  $\mathbf{x}\in \mathcal{X}$  and  $g(\mathbf{x})\leq \epsilon$  . Let  $\delta_{\mathcal{X}}(\mathbf{x})$  be the zero-infinity characteristic function of set  $\mathcal{X}$  ,  $\operatorname {proj}_{\mathcal{X}}(\cdot)$  be the projection mapping to  $\mathcal{X}$  and  $\mathrm{dist}(\mathbf{x},\mathcal{A}):= \min_{\mathbf{y}\in \mathcal{A}}\| \mathbf{x} - \mathbf{y}\|$  for set  $\mathcal{A}$
We make the following assumptions on (1) throughout the paper.
Assumption 3.1. Functions  $f$  and  $g$  are continuous with  $\partial f(\mathbf{x}) \neq \emptyset$  and  $\partial g(\mathbf{x}) \neq \emptyset$  on  $\mathcal{X}$ , and there exists  $M$  such that  $\| \zeta_f \| \leq M$  and  $\| \zeta_g \| \leq M$  for any  $\mathbf{x} \in \mathcal{X}$ ,  $\zeta_f \in \partial f(\mathbf{x})$  and  $\zeta_g \in \partial g(\mathbf{x})$ .
Since (1) is non-convex, finding an  $\epsilon$ -optimal solution is intractable in general. For a non-convex problem, the target is typically to find a stationary point of (1), which is a point  $\mathbf{x}^* \in \mathcal{X}$  that satisfies the following Karush-Kuhn-Tucker (KKT) conditions:
$$
\boldsymbol {\zeta} _ {f} ^ {*} + \lambda^ {*} \boldsymbol {\zeta} _ {g} ^ {*} \in - \mathcal {N} _ {\mathcal {X}} (\mathbf {x} ^ {*}), \quad \lambda^ {*} g (\mathbf {x} ^ {*}) = 0, \quad g (\mathbf {x} ^ {*}) \leq 0, \quad \lambda^ {*} \geq 0,
$$
where  $\lambda^{*} \in \mathbb{R}$  is a Lagrangian multiplier,  $\zeta_{f}^{*} \in \partial f(\mathbf{x}^{*})$  and  $\zeta_{g}^{*} \in \partial g(\mathbf{x}^{*})$ . Typically, an exact stationary point can only be approached by an algorithm at full convergence, which may require infinitely many iterations. Within a finite number of iterations, an algorithm can only generate an  $\epsilon$ -stationary point [13], which is a point  $\widehat{\mathbf{x}} \in \mathcal{X}$  satisfying
$$
\operatorname {d i s t} \left(\widehat {\boldsymbol {\zeta}} _ {f} + \widehat {\lambda} \widehat {\boldsymbol {\zeta}} _ {g}, - \mathcal {N} _ {\mathcal {X}} (\widehat {\mathbf {x}})\right) \leq \epsilon , \quad | \widehat {\lambda} g (\widehat {\mathbf {x}}) | \leq \epsilon^ {2}, \quad g (\widehat {\mathbf {x}}) \leq \epsilon^ {2}, \quad \widehat {\lambda} \geq 0, \tag {2}
$$
where  $\widehat{\lambda} \in \mathbb{R}$  is a Lagrangian multiplier,  $\widehat{\zeta}_f \in \partial f(\widehat{\mathbf{x}})$  and  $\widehat{\zeta}_g \in \partial g(\widehat{\mathbf{x}})$ . However, because  $f$  and  $g$  are non-smooth, computing an  $\epsilon$ -stationary point is still challenging even for an unconstrained problem. Nevertheless, under the weak convexity assumption, it is possible to compute a nearly  $\epsilon$ -stationary point, which we will introduce next.
Given  $\hat{\rho} \geq 0$ ,  $\tilde{\rho} \geq 0$  and  $\mathbf{x} \in \mathcal{X}$ , we define a quadratically regularized problem of (1) as
Algorithm 1 Switching Subgradient (SSG) Method by Polyak [59]
1: Input:  $\mathbf{x}^{(0)}\in \mathcal{X}$ , total number of iterations  $T$ , tolerance of infeasibility  $\epsilon_t\geq 0$ , stepsize  $\eta_t$ , and a starting index  $S$  for generating outputs.
2: Initialization:  $I = \emptyset$  and  $J = \emptyset$ .
3: for iteration  $t = 0,1,\dots ,T - 1$  do
4: if  $g(\mathbf{x}^{(t)}) \leq \epsilon_t$  then
5: Set  $\mathbf{x}^{(t + 1)} = \mathrm{proj}_{\mathcal{X}}\bigl (\mathbf{x}^{(t)} - \eta_t\boldsymbol{\zeta}_f^{(t)}\bigr)$  for any  $\zeta_f^{(t)}\in \partial f\bigl (\mathbf{x}^{(t)}\bigr)$  and, if  $t\geq S,I = I\cup \{t\}$
6: else
7: Set  $\mathbf{x}^{(t + 1)} = \mathrm{proj}_{\mathcal{X}}(\mathbf{x}^{(t)} - \eta_t\boldsymbol{\zeta}_q^{(t)})$  for any  $\zeta_q^{(t)}\in \partial g(\mathbf{x}^{(t)})$  and, if  $t\geq S$ ,  $J = J\cup \{t\}$ .
8: end if
9: end for
10: Output I:  $\mathbf{x}^{(\tau)}$  with  $\tau$  randomly sampled from  $I$  using  $\mathrm{Prob}(\tau = t) = \eta_t / \sum_{s\in I}\eta_s$ .
11: Output II:  $\mathbf{x}^{(\tau)}$  with  $\tau$  randomly sampled from  $I\cup J$  using  $\mathrm{Prob}(\tau = t) = \eta_t / \sum_{s\in I\cup J}\eta_s$
$$
\varphi (\mathbf {x}) \equiv \min  _ {\mathbf {y} \in \mathcal {X}} \left\{f (\mathbf {y}) + \frac {\hat {\rho}}{2} \| \mathbf {y} - \mathbf {x} \| ^ {2}, s. t. g (\mathbf {y}) + \frac {\tilde {\rho}}{2} \| \mathbf {y} - \mathbf {x} \| ^ {2} \leq 0 \right\}, \tag {3}
$$
$$
\widehat {\mathbf {x}} (\mathbf {x}) \equiv \underset {\mathbf {y} \in \mathcal {X}} {\arg \min } \left\{f (\mathbf {y}) + \frac {\hat {\rho}}{2} \| \mathbf {y} - \mathbf {x} \| ^ {2}, s. t. g (\mathbf {y}) + \frac {\tilde {\rho}}{2} \| \mathbf {y} - \mathbf {x} \| ^ {2} \leq 0 \right\}. \tag {4}
$$
Following the literature on weakly convex optimization [24, 27, 23, 13, 54, 39], we use the value of  $\|\widehat{\mathbf{x}}(\mathbf{x}) - \mathbf{x}\|$  as a measure of the quality of a solution  $\mathbf{x}$  because it can be interpreted as a stationarity measure. For the purpose of illustration, we assume for now that  $\widehat{\mathbf{x}}(\mathbf{x})$  is uniquely defined and there exists a Lagrangian multiplier  $\widehat{\lambda} \in \mathbb{R}$  such that the following KKT conditions of (4) hold.
$$
\begin{array}{l} \widehat {\boldsymbol {\zeta}} _ {f} + \hat {\rho} (\widehat {\mathbf {x}} (\mathbf {x}) - \mathbf {x}) + \widehat {\lambda} \left(\widehat {\boldsymbol {\zeta}} _ {g} + \tilde {\rho} (\widehat {\mathbf {x}} (\mathbf {x}) - \mathbf {x})\right) \in - \mathcal {N} _ {\mathcal {X}} (\widehat {\mathbf {x}} (\mathbf {x})), \\ \widehat {\lambda} \left(g (\widehat {\mathbf {x}} (\mathbf {x})) + \frac {\tilde {\rho}}{2} \| \widehat {\mathbf {x}} (\mathbf {x}) - \mathbf {x} \| ^ {2}\right) = 0, \quad g (\widehat {\mathbf {x}} (\mathbf {x})) + \frac {\tilde {\rho}}{2} \| \widehat {\mathbf {x}} (\mathbf {x}) - \mathbf {x} \| ^ {2} \leq 0, \quad \widehat {\lambda} \geq 0, \tag {5} \\ \end{array}
$$
where  $\widehat{\zeta}_f\in \partial f(\widehat{\mathbf{x}} (\mathbf{x}))$  and  $\widehat{\zeta}_g\in \partial g(\widehat{\mathbf{x}} (\mathbf{x}))$ . Therefore, as long as  $\| \widehat{\mathbf{x}} (\mathbf{x}) - \mathbf{x}\| \leq \epsilon$ , we have
$$
\left. \operatorname {d i s t} \left(\widehat {\boldsymbol {\zeta}} _ {f} + \widehat {\lambda} \widehat {\boldsymbol {\zeta}} _ {g}, - \mathcal {N} _ {\mathcal {X}} (\widehat {\mathbf {x}} (\mathbf {x}))\right) \leq (\hat {\rho} + \widehat {\lambda} \tilde {\rho}) \epsilon , \quad | \widehat {\lambda} g (\widehat {\mathbf {x}} (\mathbf {x})) | = \widehat {\lambda} \tilde {\rho} \epsilon^ {2} / 2, \quad g (\widehat {\mathbf {x}} (\mathbf {x})) \leq 0. \right. \tag {6}
$$
This means  $\widehat{\mathbf{x}} (\mathbf{x})$  is an  $O(\epsilon)$ -stationary point of the original problem (1) in the sense of (2). Since  $\mathbf{x}$  is within an  $\epsilon$ -distance from  $\widehat{\mathbf{x}} (\mathbf{x})$ , we call such an  $\mathbf{x}$  a nearly  $\epsilon$ -stationary point of (1). We formalize this definition as follows.
Definition 3.2. Suppose  $\widehat{\mathbf{x}}(\mathbf{x})$  is defined in (4) with  $\epsilon \geq 0$ . A (stochastic) point  $\mathbf{x} \in \mathcal{X}$  is a (stochastic) nearly  $\epsilon$ -stationary point of (1) if  $\mathbb{E}[\|\widehat{\mathbf{x}}(\mathbf{x}) - \mathbf{x}\|] \leq \epsilon$ .
Of course, we can claim  $\widehat{\mathbf{x}} (\mathbf{x})$  is an  $O(\epsilon)$ -stationary point of (1) based on (6) only when  $\widehat{\lambda}$  in (5) exists and does not go to infinity as  $\epsilon$  approaches zero. Fortunately, we can show in Lemmas 4.2 and 5.2 that this is true under some constraint qualifications, which justifies Definition 3.2.
We present the SSG method in Algorithm 1 for finding a nearly  $\epsilon$ -stationary point of (1). At iteration  $t$ , we check if the current solution  $\mathbf{x}^{(t)}$  is nearly feasible in the sense that  $g(\mathbf{x}^{(t)}) \leq \epsilon_t$  for a predetermined tolerance of infeasibility  $\epsilon_t$ . If yes, the algorithm performs a subgradient step along the subgradient of  $f$ . Otherwise, the algorithm switches the updating direction to the subgradient of  $g$ . The algorithm records the iteration indexes of the nearly feasible solutions in set  $I$  and other indexes in set  $J$ . The final output is randomly sampled from the iterates in  $I$  or  $I \cup J$  with a distribution weighted by the stepsizes  $\eta_t$ 's. An index  $S$  is set so the algorithm only starts to record  $I$  and  $J$  when  $t \geq S$ . We study the theoretical oracle complexity of Algorithm 1 for finding a nearly  $\epsilon$ -stationary point, which is defined as the total number of times for which the algorithm queries the subgradient or function value of  $f$  or  $g$ . Our results are presented separately when  $g$  is convex and when  $g$  is weakly convex.
4 Convex constraints
In this section, we first consider a relatively easy case where  $f$  is weakly convex but  $g$  is convex. In particular, we make the following assumptions in addition to Assumption 3.1 in this section.
Assumption 4.1. The following statements hold:
A.  $f(\mathbf{x})$  is  $\rho$ -weakly convex on  $\mathcal{X}$  and  $g(\mathbf{x})$  is convex on  $\mathcal{X}$ .
B. (Slater's condition) There exists  $\mathbf{x}_{\mathrm{feas}} \in \mathrm{relint}(\mathcal{X})$  such that  $g(\mathbf{x}_{\mathrm{feas}}) < 0$ .
C. There exists  $D \in \mathbb{R}$  such that  $\| \mathbf{x} - \mathbf{x}'\| \leq D$  for any  $\mathbf{x}$  and  $\mathbf{x}'$  in  $\mathcal{X}$ .
In this section, we choose parameters in (3) such that
$$
\hat {\rho} > \rho \text {a n d} \tilde {\rho} = 0. \tag {7}
$$
Under Assumption 4.1, (7) guarantees that (3) is strictly feasible, its objective function is  $(\hat{\rho} - \rho)$ -strongly convex and its constraint function is convex, so  $\widehat{\mathbf{x}}(\mathbf{x})$  in (4) is unique and  $\widehat{\lambda}$  in (5) exists. We first present an upper bound of  $\widehat{\lambda}$  that is independent of  $\mathbf{x}$ . The proof is in Section A.1.
Lemma 4.2. Suppose Assumptions 3.1 and 4.1 hold. Given any  $\mathbf{x} \in \mathcal{X}$ , let  $\widehat{\mathbf{x}}(\mathbf{x})$  be defined as in (4) with  $(\hat{\rho}, \tilde{\rho})$  satisfying (7) and  $\hat{\lambda}$  be the associated Lagrangian multiplier satisfying (5). We have
$$
\widehat {\lambda} \leq \Lambda := (M D + \hat {\rho} D ^ {2}) / (- g (\mathbf {x} _ {\text {f e a s}})). \tag {8}
$$
For simplicity of notation, we denote  $\widehat{\mathbf{x}} (\mathbf{x}^{(t)})$  defined in (4) by  $\widehat{\mathbf{x}}^{(t)}$ . Let  $\mathbb{E}_{\tau}[\cdot ]$  be the expectation taken only over the random index  $\tau$  when the algorithms stop. We present the convergence properties of Algorithm 1 when  $\epsilon_{t}$  and  $\eta_t$  are static and diminishing. The proof is provided in Section A.4.
Theorem 4.3. Suppose Assumptions 3.1 and 4.1 hold and  $\Lambda$  is as in (8). Let  $\widehat{\mathbf{x}} (\mathbf{x}^{(t)})$  be defined as in (4) with  $(\hat{\rho},\hat{\rho})$  satisfying (7) and  $\mathbf{x}^{(\tau)}$  is generated by Output I. Algorithm 1 guarantees  $\mathbb{E}_{\tau}[\| \widehat{\mathbf{x}}^{(\tau)} - \mathbf{x}^{(\tau)}\| ]\leq \epsilon$  and  $\mathbb{E}_{\tau}[g(\mathbf{x}^{(\tau)})]\leq \frac{\epsilon^2(\hat{\rho} - \rho)}{1 + \Lambda}$  in either of the following cases.
Case I:  $S = 0$ ,  $\epsilon_{t} = \frac{\epsilon^{2}(\hat{\rho} - \rho)}{1 + \Lambda}$ ,  $\eta_{t} = \frac{2\epsilon^{2}(\hat{\rho} - \rho)}{5(1 + \Lambda)M^{2}}$  and  $T \geq \frac{25M^2D^2(1 + \Lambda)^2}{4\epsilon^4(\hat{\rho} - \rho)^2} = O(1 / \epsilon^4)$ .
Case II:  $S = T / 2$ ,  $\epsilon_{t} = \frac{5MD}{\sqrt{t + 1}}$ ,  $\eta_{t} = \frac{D}{M\sqrt{t + 1}}$  and  $T \geq \frac{50M^2D^2(1 + \Lambda)^2}{\epsilon^4(\hat{\rho} - \rho)^2} = O(1 / \epsilon^4)$ .
Algorithm 1 is single-loop with  $O(1)$  oracle complexity per iteration, so its total complexity is just  $T = O(1 / \epsilon^4)$ , which matches the start-of-the-art complexity by [13, 54, 39]. This result can be generalized to the case where there exist additional linear equality constraints  $\mathbf{A}\mathbf{x} = \mathbf{b}$ . See the extension of Lemma 4.2 in Section A.1 and Remark A.2.
Assumption 4.1C can be relaxed to only require that the feasible set, i.e.,  $S = \{\mathbf{x} \in \mathcal{X} \mid g(\mathbf{x}) \leq 0\}$ , is bounded instead of  $\mathcal{X}$ . The same complexity can be achieved by Algorithm 1 by using a switching step size rule similar to  $\eta_t$  in Proposition 5.5. This result is provided in Section A.8 but we recommend interested readers to read Section 5 first to get introduced to this special step size.