SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark 4.4. Property  $\mathbb{E}_{\tau}[g(\mathbf{x}^{(\tau)})] \leq \frac{\epsilon^2(\hat{\rho} - \rho)}{1 + \Lambda}$  in the theorems above is not required by Definition 3.2. By Assumption 3.1A,  $\mathbb{E}_{\tau}[g(\mathbf{x}^{(\tau)})] \leq \mathbb{E}_{\tau}[g(\widehat{\mathbf{x}}^{(\tau)})] + M\mathbb{E}_{\tau}[\| \widehat{\mathbf{x}}^{(\tau)} - \mathbf{x}^{(\tau)}\|] \leq M\epsilon$ , which means a nearly  $\epsilon$ -stationary point must be  $O(\epsilon)$ -feasible by definition. Property  $\mathbb{E}_{\tau}[g(\mathbf{x}^{(\tau)})] \leq \frac{\epsilon^2(\hat{\rho} - \rho)}{1 + \Lambda}$  implies  $O(\epsilon^2)$ -feasibility for the output, which is even better.
When  $g$  is  $\mu$ -strongly convex with  $\mu > 0$ , we can show that the complexity of Algorithm 1 is still  $O(1 / \epsilon^4)$  but one can simply set  $\epsilon_t = 0$ , which makes  $\eta_t$  the only tuning parameter. This makes this single-loop method even more attractive. Due to space limit, we include this result in Section A.5. We also extend our result to the stochastic case in Section A.6 and A.7.
5 Weakly convex constraints
Next we consider the case where both  $f$  and  $g$  are weakly convex but not necessarily convex. Let
$$
g _ {+} (\mathbf {x}) = \max  \{g (\mathbf {x}), 0 \}, \quad \mathcal {L} = \{\mathbf {x} \in \mathcal {X} \mid g (\mathbf {x}) = 0 \} \text {a n d} \mathcal {S} = \{\mathbf {x} \in \mathcal {X} \mid g (\mathbf {x}) \leq 0 \}.
$$
We make the following assumptions in addition to Assumption 3.1 in this section.
Assumption 5.1. The following statements hold:
A.  $f(\mathbf{x})$  and  $g(\mathbf{x})$  are  $\rho$ -weakly convex on  $\mathcal{X}$ .
B. There exist  $\bar{\epsilon} > 0$ ,  $\theta > 0$  and  $\bar{\rho} > \rho$  such that, for any  $\bar{\epsilon}^2$ -feasible solution  $\mathbf{x}$ , there exists  $\mathbf{y} \in \mathrm{relint}(\mathcal{X})$  such that  $g(\mathbf{y}) + \frac{\bar{\rho}}{2} \| \mathbf{y} - \mathbf{x} \|^2 \leq -\theta$ .
C.  $f\coloneqq \inf_{\mathbf{x}\in \mathcal{X}}f(\mathbf{x}) > - \infty .$
Assumption 5.1B is called the uniform Slater's condition by [54]. We will present two real-world examples in Section B.1 that satisfy this assumption, including a fair classification problem under demographic parity constraint, which is one of the applications in our numerical experiments in Section 6.2. In this section, we choose parameters in (3) such that
$$
\bar {\rho} \geq \hat {\rho} = \tilde {\rho} > \rho . \tag {9}
$$
Under Assumption 5.1, (9) guarantees that (3) is uniformly strictly feasible for any  $\epsilon^2$ -feasible  $\mathbf{x}$ , and the objective and constraint functions of (3) are both  $(\hat{\rho} - \rho)$ -strongly convex, so  $\widehat{\mathbf{x}}(\mathbf{x})$  is uniquely defined and  $\widehat{\lambda}$  in (5) exists. In addition, Assumption 5.1 has the following three implications that play important roles in our complexity analysis.
First,  $\widehat{\lambda}$  in (5) can be bounded by a constant independent of  $\mathbf{x}$  and  $\epsilon$  as long as  $\mathbf{x}$  is  $\epsilon^2$ -feasible with  $\epsilon \leq \bar{\epsilon}$ . This result is similar to Lemma 1 by Ma et al. [54] except that they require  $\mathcal{X}$  to be bounded but we do not. The proof is given in Section B.2.
Lemma 5.2. Suppose Assumptions 3.1 and 5.1 hold. Given any  $\epsilon^2$ -feasible  $\mathbf{x}$  with any  $\epsilon \leq \bar{\epsilon}$ , let  $\widehat{\mathbf{x}}(\mathbf{x})$  defined as in (4) satisfying (9) and  $\widehat{\lambda}$  is the associated Lagrangian multiplier satisfying (5). We have
$$
\left\| \widehat {\mathbf {x}} (\mathbf {x}) - \mathbf {x} \right\| \leq M / \hat {\rho} \quad a n d \quad \widehat {\lambda} \leq \Lambda^ {\prime} := 2 M / \sqrt {2 \theta (\hat {\rho} - \rho)}. \tag {10}
$$
Second, the subgradient of the constraint function  $g(\mathbf{x}) + \delta_{\mathcal{X}}(\mathbf{x})$  on  $\mathcal{L}$  is uniformly away from the origin. The proof is provided in Section B.2.
Lemma 5.3. Suppose Assumptions 3.1 and 5.1 hold. It holds for any  $\mathbf{x} \in \mathcal{L}$  that
$$
\min  _ {\boldsymbol {\zeta} _ {g} \in \partial g (\mathbf {x}), \mathbf {u} \in \mathcal {N} _ {\chi} (\mathbf {x})} \| \boldsymbol {\zeta} _ {g} + \mathbf {u} \| \geq \nu := \sqrt {2 \theta (\hat {\rho} - \rho)}. \tag {11}
$$
Lastly, note that  $S$  is the optimal set of  $\min_{\mathbf{x} \in \mathcal{X}} g_+(x)$ , which is a  $\rho$ -weakly convex non-smooth optimization problem with an optimal value of zero. Lemma 5.3 implies that  $g_+(x)$  is sharp near the boundary of  $S$ , meaning that  $g_+(x)$  satisfies a linear error bound in an  $O(1)$ -neighborhood of  $S$ . A similar result for a convex  $g_+$  is given in Lemma 1 in [76]. In the lemma below, we extend their result for a weakly convex  $g_+$ . The proof is in Section B.2 and the second conclusion is directly from [26].
Lemma 5.4. Suppose Assumptions 3.1 and 5.1 hold. It holds for any  $\mathbf{x}$  satisfying  $\mathrm{dist}(\mathbf{x},\mathcal{S})\leq \frac{\nu}{\rho}$  that
$$
(\nu / 2) \operatorname {d i s t} (\mathbf {x}, S) \leq g _ {+} (\mathbf {x}). \tag {12}
$$
Moreover,  $\nu \leq 2M$  and  $\min_{\mathbf{x} \in \mathcal{X}} g_+(\mathbf{x})$  has no stationary point satisfying  $0 < \mathrm{dist}(\mathbf{x}, \mathcal{S}) < \frac{\nu}{\rho}$ .
Since  $g$  is non-convex, Algorithm 1 may not even find a nearly feasible solution if  $\mathbf{x}^{(t)}$  is trapped at a stationary point of  $g$  with  $g(\mathbf{x}) > 0$ , that is, a sub-optimal stationary point of  $\min_{\mathbf{x} \in \mathcal{X}} g_+(g(\mathbf{x}))$ . Fortunately, the second conclusion of Lemma 5.4 indicates that this situation can be avoided by keeping  $\mathrm{dist}(\mathbf{x}^{(t)}, \mathcal{S}) = O(\epsilon^2)$  during the algorithm. To do so, we start with  $\mathbf{x}^{(0)} \in \mathcal{S}$  and use  $\epsilon_t = O(\epsilon^2)$  in Algorithm 1. Moreover, we apply a switching stepsize rule that sets  $\eta_t = O(\epsilon^2)$  when  $t \in I$  and  $\eta_t = g(\mathbf{x}^{(t)}) / \| \boldsymbol{\zeta}_g^{(t)}\|^2$  when  $t \in J$ , the latter of which is known by the Polyak's stepsize [33, 60]. This way, when  $g(\mathbf{x}^{(t)}) \leq \epsilon_t$ , (12) ensures  $\mathrm{dist}(\mathbf{x}^{(t)}, \mathcal{S}) = O(\epsilon^2)$ . When  $g(\mathbf{x}^{(t)}) > \epsilon_t$ , (12) ensures  $\mathrm{dist}(\mathbf{x}^{(t)}, \mathcal{S})$  Q-linearly converges to zero [26], which also guarantees  $\mathrm{dist}(\mathbf{x}^{(t)}, \mathcal{S}) = O(\epsilon^2)$ . As a result, we have  $g(\mathbf{x}^{(t)}) \leq \epsilon^2$  for any  $t$ , so problem (3) with  $\mathbf{x} = \mathbf{x}^{(t)}$  and  $(\hat{\rho}, \tilde{\rho})$  satisfying (9) will be strictly feasible according to Assumption 5.1B. This finding is given in the proposition below with its proof in Section B.3.
Proposition 5.5. Suppose Assumptions 3.1 and 5.1 hold and  $\epsilon \leq \bar{\epsilon}$ . Also, suppose  $\mathbf{x}^{(t)}$  is generated by Algorithm 1 using  $\mathbf{x}^{(0)} \in S$ ,  $\epsilon_t = \frac{\nu}{4} \min \left\{ \frac{\epsilon^2}{M}, \frac{\nu}{(4\rho)} \right\}$  and
$$
\eta_ {t} = \left\{ \begin{array}{l l} \frac {\nu}{4 M ^ {2}} \min \left\{\epsilon^ {2} / M, \nu / (4 \rho) \right\} & i f t \in I \\ g (\mathbf {x} ^ {(t)}) / \| \boldsymbol {\zeta} _ {g} ^ {(t)} \| ^ {2} & i f t \in J. \end{array} \right.
$$
Then  $\mathrm{dist}(\mathbf{x}^{(t)},\mathcal{S})\leq \min \left\{\epsilon^2 /M,\nu /(4\rho)\right\}$  and  $g(\mathbf{x}^{(t)})\leq \epsilon^2$  for any  $t\geq 0$ . As a consequence,  $\mathbf{x}^{(t)}$  is  $\epsilon^2$ -feasible to (3) where  $\mathbf{x} = \mathbf{x}^{(t)}$  and  $(\hat{\rho},\tilde{\rho})$  satisfies (9).
Let  $\widehat{\mathbf{x}}^{(t)}$  be  $\widehat{\mathbf{x}} (\mathbf{x}^{(t)})$  defined in (4) with  $(\hat{\rho},\tilde{\rho})$  satisfying (9). The complexity result for the case of weakly convex constraints is as follows. The proof can be found in Section B.3.
Theorem 5.6. Under the same assumptions as Proposition 5.5, Algorithm 1 guarantees  $\mathbb{E}_{\tau}[\| \widehat{\mathbf{x}}^{(\tau)} - \mathbf{x}^{(\tau)}\| ]\leq C\epsilon$  and  $\mathbb{E}_{\tau}[g(\mathbf{x}^{(\tau)})]\leq \epsilon^2$ , where  $C\coloneqq 2\sqrt{\frac{1 + \Lambda'}{\hat{\rho} - \rho}}$ , if we use Output II and set
$$
S = 0 a n d T \geq \frac {8 M ^ {2} \left(f (\mathbf {x} ^ {(0)}) - \underline {{f}} + 3 M ^ {2} / (2 \hat {\rho})\right)}{\hat {\rho} (1 + \Lambda^ {\prime}) \nu \epsilon^ {2} \min \left\{\epsilon^ {2} / M , \nu / (4 \rho) \right\}} = O (1 / \epsilon^ {4}).
$$
This theorem indicates that Algorithm 1 finds a nearly  $(C\epsilon)$ -stationary point with complexity  $O(1 / \epsilon^4)$ . To obtain a nearly  $\epsilon$ -stationary point with  $\epsilon \leq \bar{\epsilon}$ , one only needs to replace  $\epsilon$  in  $\eta_t$ ,  $\epsilon_t$  and  $T$  in this theorem above by  $\epsilon / \max\{C, 1\}$ . This will only change the constant factor in the  $O(1 / \epsilon^4)$  complexity. This complexity matches the start-of-the-art complexity by [13, 54, 39].
We demonstrate the performance of the SSG method on two fairness-aware classification problems, which are instances of (1) with convex and weakly convex  $g$ 's, respectively. We compare with two different double-loop inexact proximal point (IPP) methods [13, 54, 39]. The IPP method approximately solves a strongly convex constrained subproblem in each outer iteration, and we use the SSG method in this paper and the ConEx method in [13] as the solvers (inner loop) because they both have the best theoretical complexity for that subproblem. We use IPP-SSG and IPP-ConEx to denote these two implementations of the IPP method. All experiments are conducted using MATLAB 2022b on a computer with the CPU 3.20GHz x Intel Core i7-8700 and 16GB memory.
6.1 Classification problem with ROC-based fairness
Given a feature vector  $\mathbf{a} \in \mathbb{R}^d$  and a class label  $b \in \{1, -1\}$ , the goal in a binary linear classification problem is to learn a model  $\mathbf{x} \in \mathbb{R}^d$  to predict  $b$  based on the score  $\mathbf{x}^\top \mathbf{a}$ . Let  $\mathcal{D} = \{(\mathbf{a}_i, b_i)\}_{i=1}^n$  be a training set and  $\ell(\cdot)$  be a convex non-increasing loss function. Model  $\mathbf{x}$  can be learned by solving
$$
L ^ {*} = \min  _ {\mathbf {x} \in \mathcal {W}} \left\{L (\mathbf {x}) := \frac {1}{n} \sum_ {i = 1} ^ {n} \ell \left(b _ {i} \mathbf {x} ^ {\top} \mathbf {a} _ {i}\right) \right\}, \tag {13}
$$
where  $\mathcal{W} = \{\mathbf{x}\in \mathbb{R}^d\mid \| \mathbf{x}\| \leq r\}$ . Solving (13) may ensure good classification performance of  $\mathbf{x}$  but not its fairness. Suppose there exist two additional datasets. One contains the feature vectors of a protected group, denoted by  $\mathcal{D}_p = \{\mathbf{a}_i^p\}_{i = 1}^{n_p}$ , and the other one contains the feature vectors of an unprotected group, denoted by  $\mathcal{D}_u = \{\mathbf{a}_i^u\}_{i = 1}^{n_u}$ . We want to enhance the fairness of  $\mathbf{x}$  between these two groups using the ROC-based fairness metric proposed by [74]. Suppose we set a threshold  $\theta$  and classify data  $\mathbf{a}$  as positive if  $\mathbf{x}^\top \mathbf{a}\geq \theta$  and as negative otherwise. The ROC-based fairness measure and its continuous approximation are defined as
$$
\begin{array}{l} \max _ {\theta \in \Theta} \left| \frac {1}{n _ {p}} \sum_ {i = 1} ^ {n _ {p}} \mathbb {I} (\mathbf {x} ^ {\top} \mathbf {a} _ {i} ^ {p} \geq \theta) - \frac {1}{n _ {u}} \sum_ {i = 1} ^ {n _ {u}} \mathbb {I} (\mathbf {x} ^ {\top} \mathbf {a} _ {i} ^ {u} \geq \theta) \right| \\ \approx R (\mathbf {x}) := \max  _ {\theta \in \Theta} \left| \frac {1}{n _ {p}} \sum_ {i = 1} ^ {n _ {p}} \sigma \left(\mathbf {x} ^ {\top} \mathbf {a} _ {i} ^ {p} - \theta\right) - \frac {1}{n _ {u}} \sum_ {i = 1} ^ {n _ {u}} \sigma \left(\mathbf {x} ^ {\top} \mathbf {a} _ {i} ^ {u} - \theta\right) \right|, \tag {14} \\ \end{array}
$$
where  $\sigma(z) = \exp(z) / (1 + \exp(z))$  is the sigmoid function and  $\Theta$  is a finite set of thresholds. If the value of this measure is small, model  $\mathbf{x}$  produces similar predicted positive rates for the protected and unprotected groups on various  $\theta$ 's, indicating the fairness of the model. To obtain a fair  $\mathbf{x}$ , we balance (13) and (14) by solving
$$
\min  _ {\mathbf {x} \in \mathcal {W}} R (\mathbf {x}) \text {s . t .} L (\mathbf {x}) \leq L ^ {*} + \kappa , \tag {15}
$$
Table 1: Information of the datasets. Groups are males VS females in a9a, users with age within [25, 60] VS outside [25, 60] in bank, and caucasian VS non-caucasian in COMPAS.
DATASETS	n	d	LABEL	GROUPS
A9A	48,842	123	INCOME	GENDER
BANK	41,188	54	SUBSCRIPTION	AGE
COMPAS	6,172	16	RECIDIVISM	RACE
Figure 1: Performances vs number of iterations on classification problems with ROC-based fairness.
where  $\kappa$  is a slackness parameter indicating how much we are willing to increase the classification loss in order to reduce  $R(\mathbf{x})$  to obtain a more fair model. Problem (15) is an instance of (1) satisfying Assumptions 3.1 and 4.1 with  $\rho = \beta$  where  $\beta$  is defined in (77) in Section B.1.
We solve (15) on three datasets:  $a9a$  [41], bank [56] and COMPAS [4]. The information of these datasets is given in Table 1. We split each dataset into two subsets with a ratio of  $2:1$ . The larger set is used as  $\mathcal{D}$  in the constraint and the smaller set is further split into  $\mathcal{D}_p$  and  $\mathcal{D}_u$  based on a binary group variable listed in Table 1.
In our experiments, we choose  $\ell(z) = (1 - z)_+$  and first solve (13) using the subgradient method with a large enough number of iterations to obtain  $L^*$  and a solution  $\mathbf{x}_{\mathrm{ERM}}$ . Then we set  $\kappa = 0.001L^*$  and  $r = 5\|\mathbf{x}_{\mathrm{ERM}}\|$ , and let  $\Theta$  consist of 400 points equally spaced between  $\min_i \mathbf{x}_{\mathrm{ERM}}^\top \mathbf{a}_i - 0.5(\max_i \mathbf{x}_{\mathrm{ERM}}^\top \mathbf{a}_i - \min_i \mathbf{x}_{\mathrm{ERM}}^\top \mathbf{a}_i)$  and  $\max_i \mathbf{x}_{\mathrm{ERM}}^\top \mathbf{a}_i + 0.5(\max_i \mathbf{x}_{\mathrm{ERM}}^\top \mathbf{a}_i - \min_i \mathbf{x}_{\mathrm{ERM}}^\top \mathbf{a}_i)$ .
All methods are initialized at  $\mathbf{x}_{\mathrm{ERM}}$ . We implemented the SSG method with both static and diminishing step sizes. For the static step size, we select  $\epsilon_{t}$  from  $\{10^{-6}, 2 \times 10^{-6}, 5 \times 10^{-6}, 10^{-5}\}$  and  $\eta_{t}$  from  $\{2 \times 10^{-4}, 5 \times 10^{-4}, 10^{-3}, 2 \times 10^{-3}\}$ . For the diminishing step size, we set  $\epsilon_{t} = \frac{E_{1}}{\sqrt{t + 1}}$  and  $\eta_{t} = \frac{E_{2}}{\sqrt{t + 1}}$  and select  $E_{1}$  from  $\{5 \times 10^{-5}, 10^{-4}, 2 \times 10^{-4}, 5 \times 10^{-4}\}$  and  $E_{2}$  from  $\{0.02, 0.05, 0.1, 0.2\}$ . We
select the best set of parameters that produces the smallest objective value after 5000 iterations. For IPP, we select  $\hat{\rho}$  from  $\max\{\rho, 1\} \times \{1, 1.5, 2\}$  for all three datasets, and the proximal point subproblem is approximately solved by SSG and ConEx both with 100 iterations. For IPP-SSG, we apply a static stepsize with  $\epsilon_t$  and  $\eta_t$  tuned in the same way as SSG. For IPP-ConEx, following the notation in [13], we set  $\theta_t = \frac{t}{t + 1}$ ,  $\eta_t = c_1(t + 1)$  and  $\tau_t = \frac{c_2}{t + 1}$ , and select  $c_1$  from  $\{20, 50, 100, 200\}$  and  $c_2$  from  $\{0.002, 0.005, 0.01, 0.02\}$  by the same procedure adopted by SSG.
We report the performances versus number of iterations of all methods on each dataset in Figure 1. The performances versus the used CPU time will be given in Section C in the Appendix. For SSG, the  $x$ -axis represents the total number of iterations while, for IPP, it represents the total number of inner iterations across all outer iterations. The  $y$ -axis in each row represents the objective value, infeasibility and near stationarity achieved at each iteration, respectively. To measure near stationarity, we solve (3) with  $\mathbf{x} = \mathbf{x}^{(t)}$  and parameters (7) using the SSG method with 2500 iterations and use the last iterate as an approximation of  $\widehat{\mathbf{x}}(\mathbf{x}^{(t)})$ . We make sure that the change of  $\| \widehat{\mathbf{x}}(\mathbf{x}^{(t)}) - \mathbf{x}^{(t)} \|$  is less than  $1\%$  if the number of iterations is increased to 5000. Then we plot  $\| \widehat{\mathbf{x}}(\mathbf{x}^{(t)}) - \mathbf{x}^{(t)} \|$  as near stationarity in Figure 1. Since computing  $\widehat{\mathbf{x}}(\mathbf{x}^{(t)})$  with a high precision for each  $t$  is time-consuming, we only report near stationarity at 100 equally spaced iterations.
According to Figure 1, the SSG method with a diminishing stepsize has the best performance on all three datasets in the sense that it reduces the objective value and the (approximate) near stationarity measure faster than others while keeping the solutions nearly feasible. However, the SSG method with a static stepsize is not always better than the IPP methods. This is consistent with our theoretical finding that the SSG and IPP methods have the same oracle complexity.
6.2 Classification problem with demographic parity
Following the notation in the previous subsection, we consider a binary classification problem with a constraint enforcing demographic parity [1]. The measure of demographic parity and its continuous approximation are
$$
\left| \frac {1}{n _ {p}} \sum_ {i = 1} ^ {n _ {p}} \mathbb {I} \left(\mathbf {x} ^ {\top} \mathbf {a} _ {i} ^ {p} \geq 0\right) - \frac {1}{n _ {u}} \sum_ {i = 1} ^ {n _ {u}} \mathbb {I} \left(\mathbf {x} ^ {\top} \mathbf {a} _ {i} ^ {u} \geq 0\right) \right| \approx R _ {0} (\mathbf {x}) := \left| \frac {1}{n _ {p}} \sum_ {i = 1} ^ {n _ {p}} \sigma \left(\mathbf {x} ^ {\top} \mathbf {a} _ {i} ^ {p}\right) - \frac {1}{n _ {u}} \sum_ {i = 1} ^ {n _ {u}} \sigma \left(\mathbf {x} ^ {\top} \mathbf {a} _ {i} ^ {u}\right) \right|. \tag {16}
$$
Fairness measure  $R_0(\mathbf{x})$  is a special case of  $R(\mathbf{x})$  with  $\Theta = \{0\}$ . If  $R_0(\mathbf{x})$  is small, model  $\mathbf{x}$  produces similar predicted positive rates for the protected and unprotected groups. To obtain a fair  $\mathbf{x}$ , we balance (13) and (16) by solving
$$
\min  L (\mathbf {x}) + \lambda \operatorname {S C A D} (\mathbf {x}) \text {s . t .} R _ {0} (\mathbf {x}) \leq \kappa . \tag {17}
$$
Different from (15), the fairness measure is used as the constraint in (17) while the objective function becomes the empirical hinge loss plus the smoothly clipped absolute deviation (SCAD) regularizer [35] for promoting a sparse solution. Here,  $\lambda$  is a regularization parameter and
$$
\operatorname {S C A D} (\mathbf {x}) := \sum_ {i = 1} ^ {d} s \left(x _ {i}\right), \quad s \left(x _ {i}\right) = \left\{ \begin{array}{l l} 2 \left| x _ {i} \right| & 0 \leq \left| x _ {i} \right| \leq 1 \\ - x _ {i} ^ {2} + 4 \left| x _ {i} \right| + 1 & 1 <   \left| x _ {i} \right| \leq 2 \\ 3 & 2 <   \left| x _ {i} \right|. \end{array} \right. \tag {18}
$$
We prove in Section B.1 that (17) is an instance of (1) satisfying Assumptions 3.1 and 5.1 with  $\rho = \max \{2\lambda, \beta\}$  where  $\beta$  is defined in (77) in Section B.1.
We set  $\lambda = 0.2$  for all datasets and set  $\kappa = 0.005$ , 0.02 and 0.02 for  $a9a$ , bank and COMPAS, respectively. For SSG, we select  $\epsilon_{t}$  from  $\{10^{-6}, 2 \times 10^{-6}, 5 \times 10^{-6}, 10^{-5}\}$  and select  $\eta_{t}$  from  $\{10^{-4}, 2 \times 10^{-4}, 5 \times 10^{-4}, 7.5 \times 10^{-4}\}$  for  $t \in I$  while set  $\eta_{t} = g(\mathbf{x}^{(t)}) / \| \boldsymbol{\zeta}_{g}^{(t)}\|^{2}$  for  $t \in J$ . We select the best set of parameters that produces the smallest objective value after 50000 iterations. For IPP, we select  $\hat{\rho}$  from  $\max\{\rho, 1\} \times \{1, 1.5, 2\}$  and the proximal point subproblem is approximately solved by SSG and ConEx both with 600 iterations. For IPP-SSG, we apply a static stepsize with  $\epsilon_{t}$  and  $\eta_{t}$  tuned in the same way as SSG. For IPP-ConEx, following the notation in [13], we set  $\theta_{t} = \frac{t}{t + 1}$ ,  $\eta_{t} = c_{1}(t + 1)$  and  $\tau_{t} = \frac{c_{2}}{t + 1}$  and select  $c_{1}$  from  $\{20, 50, 100, 200\}$  and  $c_{2}$  from  $\{0.002, 0.005, 0.01, 0.02\}$  by the same procedure adopted by SSG. Due to the limit of space, we present the performances vs number of iterations and CPU runtime of all methods in Figure 3 and Figure 4 respectively in Section C in the Appendix.
We study the oracle complexity of the switching subgradient (SSG) method for finding a nearly  $\epsilon$ -stationary point of a non-smooth weakly convex constrained optimization problem. We show that the complexity of the SSG method matches the best result in literature that is achieved only by double-loop methods. On the contrary, the SSG method is single-loop and easier to implement with reduced tuning effort. This is the first complexity result for a single-loop first-order method for a weakly-convex non-smooth constrained problem.
This work was supported by NSF award 2147253.
[1] Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reductions approach to fair classification. In International Conference on Machine Learning, pages 60–69. PMLR, 2018.
[2] Ahmet Alacaoglu, Yura Malitsky, and Volkan Cevher. Convergence of adaptive algorithms for constrained weakly convex optimization. Advances in Neural Information Processing Systems, 34:14214-14225, 2021.
[3] Mohammad S. Alkousa. On modification of an adaptive stochastic mirror descent algorithm for convex optimization problems with functional constraints. In Computational Mathematics and Applications, pages 47-63. Springer, 2020.
[4] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. ProPublica, May, 23, 2016.
[5] Yossi Arjevani, Yair Carmon, John C. Duchi, Dylan J. Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming, 199(1-2):165-214, 2023.
[6] Alfred Auslender. An extended sequential quadratically constrained quadratic programming algorithm for nonlinear, semidefinite, and second-order cone programming. Journal of Optimization Theory and Applications, 156(2):183-212, 2013.
[7] Anastasia Bayandina, Pavel Dvurechensky, Alexander Gasnikov, Fedor Stonyakin, and Alexander Titov. Mirror descent and convex optimization problems with non-smooth inequality constraints. In Large-Scale and Distributed Optimization, pages 181-213. Springer, 2018.
[8] Albert S. Berahas, Frank E. Curtis, Michael J. O'Neill, and Daniel P. Robinson. A stochastic sequential quadratic optimization algorithm for nonlinear equality constrained optimization with rank-deficient jacobian. arXiv preprint arXiv:2106.13015, 2021.
[9] Albert S. Berahas, Frank E. Curtis, Daniel Robinson, and Baoyu Zhou. Sequential quadratic optimization for nonlinear equality constrained stochastic optimization. SIAM Journal on Optimization, 31(2):1352-1379, 2021.
[10] Albert S. Berahas, Miaolan Xie, and Baoyu Zhou. A sequential quadratic programming method with high probability complexity bounds for nonlinear equality constrained stochastic optimization. arXiv preprint arXiv:2301.00477, 2023.
[11] Jérôme Bolte and Edouard Pauwels. Majorization-minimization procedures and convergence of sqp methods for semi-algebraic and tame programs. Mathematics of Operations Research, 41(2):442-465, 2016.
[12] Digvijay Boob, Qi Deng, and Guanghui Lan. Level constrained first order methods for function constrained optimization. arXiv preprint arXiv:2205.08011, 2022.