Title: Oracle Complexity of Single-Loop Switching Subgradient Methods for Non-Smooth Weakly Convex Functional Constrained Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Constrained optimization

first-order method

non-smooth optimization

non-convex optimization
Abstract: We consider a non-convex constrained optimization problem, where the objective function is weakly convex and the constraint function is either convex or weakly convex. To solve this problem, we consider the classical switching subgradient method, which is an intuitive and easily implementable first-order method whose oracle complexity was only known for convex problems. This paper provides the first analysis on the oracle complexity of the switching subgradient method for finding a nearly stationary point of non-convex problems. Our results are derived separately for convex and weakly convex constraints. Compared to existing approaches, especially the double-loop methods, the switching gradient method can be applied to non-smooth problems and achieves the same complexity using only a single loop, which saves the effort on tuning the number of inner iterations.

Oracle Complexity of Single-Loop Switching Subgradient Methods for Non-Smooth Weakly Convex Functional Constrained Optimization

Abstract

We consider a non-convex constrained optimization problem, where the objective function is weakly convex and the constraint function is either convex or weakly convex. To solve this problem, we consider the classical switching subgradient method, which is an intuitive and easily implementable first-order method whose oracle complexity was only known for convex problems. This paper provides the first analysis on the oracle complexity of the switching subgradient method for finding a nearly stationary point of non-convex problems. Our results are derived separately for convex and weakly convex constraints. Compared to existing approaches, especially the double-loop methods, the switching gradient method can be applied to non-smooth problems and achieves the same complexity using only a single loop, which saves the effort on tuning the number of inner iterations.

1 Introduction

Continuous optimization with nonlinear constraints arises from many applications of machine learning and statistics. Examples include Neyman-Pearson classification [62] and learning with fairness constraints [77]. In this paper, we consider the following general nonlinear constrained optimization

where X ⊂ R d is a closed convex set that allows a computationally easy projection operator, f is weakly-convex, g is either convex or weakly convex, and functions f and g are not necessarily smooth. When g ( x ) ≡ max i =1 ,...,m g i ( x ) , (1) is equivalent to an optimization problem with multiple nonlinear constraints g i ( x ) ≤ 0 for i = 1 , . . . , m .

A weakly convex function can be non-convex, so computing an optimal solution of (1) is challenging in general even without constraints. For this reason, theoretical analysis for gradient-based algorithms for non-convex problems mostly focuses on an algorithm's (oracle) complexity for finding an ϵ -stationary solution for (1). When a problem is non-smooth, finding an ϵ -stationary solution is generally difficult even if the problem is convex [47]. Hence, in this paper, we consider finding a nearly ϵ -stationary solution for (1), whose definition will be stated later in Definition 3.2.

In the past decade, there have been many studies on non-convex constrained optimization. However, most of the existing algorithms and their theoretical complexity analysis are developed by assuming f and g i 's are all smooth or can be written as the sum of a smooth and a simple non-smooth functions. A non-exhaustive list of the works under such an assumption includes [34, 80, 81, 55, 44, 45, 43, 52, 63, 53, 50, 12, 22, 9, 21, 40, 51]. Their results cannot be applied to (1) due to non-smoothness in the problem.

Non-smoothness is common in optimization in machine learning, e.g., when a non-smooth loss function is applied, but there are much fewer studies on non-smooth non-convex constrained optimization. Under the weak-convexity assumption, an effective approach for solving a non-smooth non-convex problem with theoretical guarantees is the (inexact) proximal point method, where a quadratic proximal term is added to objective and constraint functions to construct a strongly convex constrained subproblem and then a sequence of solutions can be generated by solving this subproblem inexactly and updating the center of the proximal term. Oracle complexity for this method to find a nearly ϵ -stationary has been established by [13, 54, 39] under different constraint qualifications.

The inexact proximal point method is a double-loop algorithm where the inner loop is another optimization algorithm for solving the aforementioned strongly convex subproblems. The complexity results in [13, 54, 39] require the inner loop solves each subproblem to a targeted optimality gap. However, the optimality gap is hard to evaluate and thus cannot be used to terminate the inner loop. Although the number of inner iterations needed to achieve the targeted gap can be bounded theoretically, the bound usually involves some constants that are unknown or can only be estimated conservatively. Hence, using the theoretical iteration bound to stop the inner loop usually leads to significantly more inner iterations than needed, making the whole algorithm inefficient. In practices, users often need to tune the number of inner iterations to improve algorithm's efficiency, which is an inconvenience common to most double-loop methods.

On the contrary, a single-loop algorithm is usually easier to implement as it does not require tuning the number of inner iterations. Therefore, the main contribution of this paper is showing that a singleloop first-order algorithm can find a nearly ϵ -stationary point of (1) with oracle complexity O (1 /ϵ 4 ) , which matches state-of-the-art results obtained only by the double-loop methods [13, 54, 39]. The algorithm we study is the classical switching subgradient (SSG) method proposed by Polyak [59] in 1967, which is intuitive and easy to implement but has only been analyzed before in the convex case. We first show that the SSG method has complexity complexity O (1 /ϵ 4 ) when g is convex. Then, we show that the same complexity result holds when g is weakly convex and appropriate constraint qualification holds. We also present some practical examples where all of our assumptions hold, including a fair classification problem subject to the demographic parity constraint [1]. Our technical novelty is inventing a switching stepzize rule to accompany the switching subgradient. In particular, we use a fixed stepsize when the solution is updated by the objective's subgradient while use an adaptive Polyak's stepsize [33, 60] when the solution is updated by the constraint's subgradient. This allows us to leverage the local error bound of the constraint function to keep the solution nearly feasible during the algorithm, which prevents the solution from being trapped at an infeasible stationry point. To the best of our knowledge, this paper is the first to establish the complexity of a single-loop first-order method for weakly convex non-smooth nonlinear constrained optimization. In the appendix, we also extend our algorithms and complexity analysis to the stochastic setting.

2 Related work

Non-convex constrained optimization has a long history [36, 16, 29, 17, 30, 6, 18] and the interest on this subject has been still growing in the machine learning community because of its new applications such as learning with fairness constraints (see e.g., [77]).

In recent literature, the prevalent classes of algorithms for non-convex constrained optimization include the augmented Lagrangian method (ALM), the penalty method [37, 79, 80, 81, 38, 42, 55, 44, 45, 69, 43, 52, 63, 53, 50, 40, 51], and the sequential quadratic programming method [11, 34, 12, 22, 10, 9, 8, 21]. Besides, an inexact projected gradient method is developed by [15] and a level conditional gradient method is developed by [20]. However, these works all focus on the case where g is smooth and f is either smooth or equals f 1 + f 2 , where f 1 is smooth and non-convex while f 2 is non-smooth and convex and has a simple structure that allows a closed-form solution for the proximal mapping arg min y { f 2 ( y ) + ρ 2 ∥ y -x ∥ 2 } for any x . There are relatively fewer works on non-convex non-smooth constrained problems. An alternating direction method of multipliers (ADMM) and an ALM are studied by [75] and [78], respectively, for non-convex non-smooth problems with linear constraints while our study considers nonlinear non-smooth constraints. The methods by [20] and [12] can be extended to a structured non-smooth case where f = f 1 + f 2 with f 1 being smooth non-convex and f 2 = max y { y ⊤ A x -ϕ ( y ) } with a convex ϕ , and g has a similar structure. The method by [14] can handle a specific non-smooth non-convex constraint, i.e., g ( x ) = λ ∥ x ∥ 1 -h ( x )

where h is a convex and smooth. Compared to these works, our results apply to a more general non-smooth problem without those structure assumptions.

When f and g in (1) are weakly convex and non-smooth, the inexact proximal point method has been studied by [13, 54, 39] under different constraint qualifications and different notions of stationarity. Their complexity analysis utilizes the relationship between the gradient of the Moreau envelope of (1) and the near stationarity of a solution, which is originally used to analyze complexity of subgradient methods for weakly convex non-smooth unconstrained problems [24, 25, 27, 2, 28, 61, 83]. Our analysis utilizes a similar framework. The methods [13, 54, 39] are double-loop while our algorithm only uses a single loop and achieves the same complexity of O (1 /ϵ 4 ) as them under similar assumptions.

The SSG algorithm is first proposed by Polyak [59]. It has been well-studied for convex problems [58, 7, 49, 67, 72, 73, 65, 68, 66, 3] and quasi-convex problems [67]. This paper provides the first complexity analysis for the SSG method under weak convexity assumption. Non-smooth non-convex optimization has also been studied without weak convexity assumption by [82, 46, 64, 47, 19, 70, 71]. These works analyze the complexity of first-order methods for computing an ( ϵ, δ ) -Goldstein approximate stationary point, which is a more general stationarity notation than what we consider here. However, these works only focus on unconstrained problems.

3 Algorithm and near stationarity

Let ∥ · ∥ be the ℓ 2 -norm. For h : R d → R ∪ { + ∞} , the subdifferential of h at x is

and ζ ∈ ∂h ( x ) is a subgradient of h at x . We say h is µ -strongly convex ( µ ≥ 0 ) on X if

for any ( x , x ′ ) ∈ X × X and any ζ ∈ ∂h ( x ′ ) . We say h is ρ -weakly convex ( ρ ≥ 0 ) on X if

for any ( x , x ′ ) ∈ X × X and any ζ ∈ ∂h ( x ′ ) . We denote the normal cone of X at x by N X ( x ) and the relative interior of X by relint ( X ) . We say a point x is ϵ -feasible if x ∈ X and g ( x ) ≤ ϵ . Let δ X ( x ) be the zero-infinity characteristic function of set X , proj X ( · ) be the projection mapping to X , and dist ( x , A ) := min y ∈A ∥ x -y ∥ for set A .

We make the following assumptions on (1) throughout the paper.

̸

Assumption 3.1. Functions f and g are continuous with ∂f ( x ) = ∅ and ∂g ( x ) = ∅ on X , and there exists M such that ∥ ζ f ∥ ≤ M and ∥ ζ g ∥ ≤ M for any x ∈ X , ζ f ∈ ∂f ( x ) and ζ g ∈ ∂g ( x ) .

̸

Since (1) is non-convex, finding an ϵ -optimal solution is intractable in general. For a non-convex problem, the target is typically to find a stationary point of (1), which is a point x ∗ ∈ X that satisfies the following Karush-Kuhn-Tucker (KKT) conditions:

where λ ∗ ∈ R is a Lagrangian multiplier, ζ ∗ f ∈ ∂f ( x ∗ ) and ζ ∗ g ∈ ∂g ( x ∗ ) . Typically, an exact stationary point can only be approached by an algorithm at full convergence, which may require infinitely many iterations. Within a finite number of iterations, an algorithm can only generate an ϵ -stationary point [13], which is a point ̂ x ∈ X satisfying

where ̂ λ ∈ R is a Lagrangian multiplier, ̂ ζ f ∈ ∂f ( ̂ x ) and ̂ ζ g ∈ ∂g ( ̂ x ) . However, because f and g are non-smooth, computing an ϵ -stationary point is still challenging even for an unconstrained problem. Nevertheless, under the weak convexity assumption, it is possible to compute a nearly ϵ -stationary point , which we will introduce next.

Given ˆ ρ ≥ 0 , ˜ ρ ≥ 0 and x ∈ X , we define a quadratically regularized problem of (1) as

Algorithm 1 Switching Subgradient (SSG) Method by Polyak [59]

1: Input: x (0) ∈ X , total number of iterations T , tolerance of infeasibility ϵ t ≥ 0 , stepsize η t , and a starting index S for generating outputs.

2: Initialization: I = ∅ and J = ∅ .

3: for iteration t = 0 , 1 , · · · , T -1 do

4: if g ( x ( t ) ) ≤ ϵ t then

6: else

X g g 8: end if

7: Set x ( t +1) = proj ( x ( t ) -η t ζ ( t ) ) for any ζ ( t ) ∈ ∂g ( x ( t ) ) and, if t ≥ S , J = J ∪ { t } .

9: end for

10: Output I: x ( τ ) with τ randomly sampled from I using Prob ( τ = t ) = η t / ∑ s ∈ I η s .

11: Output II: x ( τ ) with τ randomly sampled from I ∪ J using Prob ( τ = t ) = η t / ∑ s ∈ I ∪ J η s .

Following the literature on weakly convex optimization [24, 27, 23, 13, 54, 39], we use the value of ∥ ̂ x ( x ) -x ∥ as a measure of the quality of a solution x because it can be interpreted as a stationarity measure. For the purpose of illustration, we assume for now that ̂ x ( x ) is uniquely defined and there exists a Lagrangian multiplier ̂ λ ∈ R such that the following KKT conditions of (4) holds.

where ̂ ζ f ∈ ∂f ( ̂ x ( x )) and ̂ ζ g ∈ ∂g ( ̂ x ( x )) . Therefore, as long as ∥ ̂ x ( x ) -x ∥ ≤ ϵ , we have

This means ̂ x ( x ) is an O ( ϵ ) -stationary point of the original problem (1) in the sense of (2). Since x is within an ϵ -distance from ̂ x ( x ) , we call such an x a nearly ϵ -stationary point of (1). We formalize this definition as follows.

Definition 3.2. Suppose ̂ x ( x ) is defined in (4) with ϵ ≥ 0 . A(stochastic) point x ∈ X is a (stochastic) nearly ϵ -stationary point of (1) if E [ ∥ ̂ x ( x ) -x ∥ ] ≤ ϵ .

Of course, we can claim ̂ x ( x ) is an O ( ϵ ) -stationary point of (1) based on (6) only when ̂ λ in (5) exists and does not go to infinity as ϵ approaches zero. Fortunately, we can show in Lemmas 4.2 and 5.2 that this is true under some constraint qualifications, which justifies Definition 3.2.

We present the SSG method in Algorithm 1 for finding a nearly ϵ -stationary point of (1). At iteration t , we check if the current solution x ( t ) is nearly feasible in the sense that g ( x ( t ) ) ≤ ϵ t for a predetermined tolerance of infeasibility ϵ t . If yes, the algorithm performs a subgradient step along the subgradient of f . Otherwise, the algorithm switches the updating direction to the subgradient of g . The algorithm records the iteration indexes of the nearly feasible solutions in set I and other indexes in set J . The final output is randomly sampled from the iterates in I or I ∪ J with a distribution weighted by the stepsizes η t 's. An index S is set so the algorithm only starts to record I and J when t ≥ S . We study the theoretical oracle complexity of Algorithm 1 for finding a nearly ϵ -stationary point, which is defined as the total number of times for which the algorithm queries the subgradient or function value of f or g . Our results are presented separately when g is convex and when g is weakly convex.

4 Convex constraints

In this section, we first consider a relatively easy case where f is weakly convex but g is convex. In particular, we make the following assumptions in addition to Assumption 3.1 in this section.

Assumption 4.1. The following statements hold:

f ( x ) is ρ -weakly convex on X and g ( x ) is convex on X .

(Slater's condition) There exists x feas ∈ relint ( X ) such that g ( x feas ) < 0 .

There exists D ∈ R such that ∥ x -x ′ ∥ ≤ D for any x and x ′ in X .

In this section, we choose parameters in (3) such that

Under Assumption 4.1, (7) guarantees that (3) is strictly feasible, its objective function is (ˆ ρ -ρ ) -strongly convex and its constraint function is convex, so ̂ x ( x ) in (4) is unique and ̂ λ in (5) exists. We first present an upper bound of ̂ λ that is independent of x . The proof is in Section A.1.

Lemma 4.2. Suppose Assumptions 3.1 and 4.1 hold. Given any x ∈ X , let ̂ x ( x ) be defined as in (4) with (ˆ ρ, ˜ ρ ) satisfying (7) and ̂ λ be the associated Lagrangian multiplier satisfying (5) . We have

For simplicity of notation, we denote ̂ x ( x ( t ) ) defined in (4) by ̂ x ( t ) . Let E τ [ · ] be the expectation taken only over the random index τ when the algorithms stop. We present the convergence properties of Algorithm 1 when ϵ t and η t are static and diminishing. The proof is provided in Section A.4.

Theorem 4.3. Suppose Assumptions 3.1 and 4.1 hold and Λ is as in (8) . Let ̂ x ( x ( t ) ) be defined as in (4) with (ˆ ρ, ˜ ρ ) satisfying (7) and x ( τ ) is generated by Output I. Algorithm 1 guarantees E τ [ ∥ ̂ x ( τ ) -x ( τ ) ∥ ] ≤ ϵ and E τ [ g ( x ( τ ) )] ≤ ϵ 2 (ˆ ρ -ρ ) 1+Λ in either of the following cases.

Algorithm 1 is single-loop with O (1) oracle complexity per iteration, so its total complexity is just T = O (1 /ϵ 4 ) , which matches the start-of-the-art complexity by [13, 54, 39]. This result can be generalized to the case where there exist additional linear equality constraints Ax = b . See the extension of Lemma 4.2 in Section A.1 and Remark A.2.

Assumption 4.1C can be relaxed to only require that the feasible set, i.e., S = { x ∈ X | g ( x ) ≤ 0 } , is bounded instead of X . The same complexity can be achieved by Algorithm 1 by using a switching step size rule similar to η t in Proposition 5.5. This result is provided in Section A.8 but we recommend interested readers to read Section 5 first to get introduced to this special step size.

Remark 4.4 . Property E τ [ g ( x ( τ ) )] ≤ ϵ 2 (ˆ ρ -ρ ) 1+Λ in the theorems above is not required by Definition 3.2. By Assumption 3.1A, E τ [ g ( x ( τ ) )] ≤ E τ [ g ( ̂ x ( τ ) )] + M E τ [ ∥ ̂ x ( τ ) -x ( τ ) ∥ ] ≤ Mϵ, which means a nearly ϵ -stationary point must be O ( ϵ ) -feasible by definition. Property E τ [ g ( x ( τ ) )] ≤ ϵ 2 (ˆ ρ -ρ ) 1+Λ implies O ( ϵ 2 ) -feasibility for the output, which is even better.

When g is µ -strongly convex with µ > 0 , we can show that the complexity of Algorithm 1 is still O (1 /ϵ 4 ) but one can simply set ϵ t = 0 , which makes η t the only tuning parameter. This makes this single-loop method even more attractive. Due to space limit, we include this result in Section A.5. We also extend our result to the stochastic case in Section A.6 and A.7.

5 Weakly convex constraints

Next we consider the case where both f and g are weakly convex but not necessarily convex. Let

We make the following assumptions in addition to Assumption 3.1 in this section.

Assumption 5.1. The following statements hold:

f ( x ) and g ( x ) are ρ -weakly convex on X .

There exist ¯ ϵ > 0 , θ > 0 and ¯ ρ > ρ such that, for any ¯ ϵ 2 -feasible solution x , there exists y ∈ relint ( X ) such that g ( y ) + ¯ ρ 2 ∥ y -x ∥ 2 ≤ -θ .

Assumption 5.1B is called the uniform Slater's condition by [54]. We will present two real-world examples in Section B.1 that satisfy this assumption, including a fair classification problem under demographic parity constraint, which is one of the applications in our numerical experiments in Section 6.2. In this section, we choose parameters in (3) such that

Under Assumption 5.1, (9) guarantees that (3) is uniformly strictly feasible for any ϵ 2 -feasible x , and the objective and constraint functions of (3) are both (ˆ ρ -ρ ) -strongly convex, so ̂ x ( x ) is uniquely defined and ̂ λ in (5) exists. In addition, Assumption 5.1 has the following three implications that play important roles in our complexity analysis.

First, ̂ λ in (5) can be bounded by a constant independent of x and ϵ as long as x is ϵ 2 -feasible with ϵ ≤ ¯ ϵ . This result is similar to Lemma 1 by Ma et al. [54] except that they require X to be bounded but we do not. The proof is given in Section B.2.

Lemma 5.2. Suppose Assumptions 3.1 and 5.1 hold. Given any ϵ 2 -feasible x with any ϵ ≤ ¯ ϵ , let ̂ x ( x ) defined as in (4) satisfying (9) and ̂ λ is the associated Lagrangian multiplier satisfying (5) . We have

Second, the subgradient of the constraint function g ( x ) + δ X ( x ) on L is uniformly away from the origin. The proof is provided in Section B.2.

Lemma 5.3. Suppose Assumptions 3.1 and 5.1 hold. It holds for any x ∈ L that

Lastly, note that S is the optimal set of min x ∈X g + ( x ) , which is a ρ -weakly convex non-smooth optimization problem with an optimal value of zero. Lemma 5.3 implies that g + ( x ) is sharp near the boundary of S , meaning that g + ( x ) satisfies a linear error bound in an O (1) -neighborhood of S . A similar result for a convex g + is given in Lemma 1 in [76]. In the lemma below, we extend their result for a weakly convex g + . The proof is in Section B.2 and the second conclusion is directly from [26].

Lemma 5.4. Suppose Assumptions 3.1 and 5.1 hold. It holds for any x satisfying dist ( x , S ) ≤ ν ρ that

Moreover, ν ≤ 2 M and min x ∈X g + ( x ) has no stationary point satisfying 0 < dist ( x , S ) < ν ρ .

Since g is non-convex, Algorithm 1 may not even find a nearly feasible solution if x ( t ) is trapped at a stationary point of g with g ( x ) > 0 , that is, a sub-optimal stationary point of min x ∈X g + ( x ) . Fortunately, the second conclusion of Lemma 5.4 indicates that this situation can be avoided by keeping dist ( x ( t ) , S ) = O ( ϵ 2 ) during the algorithm. To do so, we start with x (0) ∈ S and use ϵ t = O ( ϵ 2 ) in Algorithm 1. Moreover, we apply a switching stepsize rule that sets η t = O ( ϵ 2 ) when t ∈ I and η t = g ( x ( t ) ) / ∥ ζ ( t ) g ∥ 2 when t ∈ J , the latter of which is known by the Polyak's stepsize [33, 60]. This way, when g ( x ( t ) ) ≤ ϵ t , (12) ensures dist ( x ( t ) , S ) = O ( ϵ 2 ) . When g ( x ( t ) ) > ϵ t , (12) ensures dist ( x ( t ) , S ) Q-linearly converges to zero [26], which also guarantees dist ( x ( t ) , S ) = O ( ϵ 2 ) . As a result, we have g ( x ( t ) ) ≤ ϵ 2 for any t , so problem (3) with x = x ( t ) and (ˆ ρ, ˜ ρ ) satisfying (9) will be strictly feasible according to Assumption 5.1B. This finding is given in the proposition below with its proof in Section B.3.

Proposition 5.5. Suppose Assumptions 3.1 and 5.1 hold and ϵ ≤ ¯ ϵ . Also, suppose x ( t ) is generated by Algorithm 1 using x (0) ∈ S , ϵ t = ν 4 min { ϵ 2 /M,ν/ (4 ρ ) } and

Then dist ( x ( t ) , S ) ≤ min { ϵ 2 /M,ν/ (4 ρ ) } and g ( x ( t ) ) ≤ ϵ 2 for any t ≥ 0 . As a consequence, x ( t ) is ϵ 2 -feasible to (3) where x = x ( t ) and (ˆ ρ, ˜ ρ ) satisfies (9) .

Let ̂ x ( t ) be ̂ x ( x ( t ) ) defined in (4) with (ˆ ρ, ˜ ρ ) satisfying (9). The complexity result for the case of weakly convex constraints is as follows. The proof can be found in Section B.3.

Theorem 5.6. Under the same assumptions as Proposition 5.5, Algorithm 1 guarantees E τ [ ∥ ̂ x ( τ ) -x ( τ ) ∥ ] ≤ Cϵ and E τ [ g ( x ( τ ) )] ≤ ϵ 2 , where C := 2 √ 1+Λ ′ ˆ ρ -ρ , if we use Output II and set

This theorem indicates that Algorithm 1 finds a nearly ( Cϵ ) -stationary point with complexity O (1 /ϵ 4 ) . To obtain a nearly ϵ -stationary point with ϵ ≤ ¯ ϵ , one only needs to replace ϵ in η t , ϵ t and T in this theorem above by ϵ/ max { C, 1 } . This will only change the constant factor in the O (1 /ϵ 4 ) complexity. This complexity matches the start-of-the-art complexity by [13, 54, 39].

6 Numerical experiments

We demonstrate the performance of the SSG method on two fairness-aware classification problems, which are instances of (1) with convex and weakly convex g 's, respectively. We compare with two different double-loop inexact proximal point (IPP) methods [13, 54, 39]. The IPP method approximately solves a strongly convex constrained subproblem in each outer iteration, and we use the SSG method in this paper and the ConEx method in [13] as the solvers (inner loop) because they both have the best theoretical complexity for that subproblem. We use IPP-SSG and IPP-ConEx to denote these two implementations of the IPP method. All experiments are conducted using MATLAB 2022b on a computer with the CPU 3.20GHz x Intel Core i7-8700 and 16GB memory.

6.1 Classification problem with ROC-based fairness

Given a feature vector a ∈ R d and a class label b ∈ { 1 , -1 } , the goal in a binary linear classification problem is to learn a model x ∈ R d to predict b based on the score x ⊤ a . Let D = { ( a i , b i ) } n i =1 be a training set and ℓ ( · ) be a convex non-increasing loss function. Model x can be learned by solving

where W = { x ∈ R d | ∥ x ∥ ≤ r } . Solving (13) may ensure good classification performance of x but not its fairness. Suppose there exist two additional datasets. One contains the feature vectors of a protected group, denoted by D p = { a p i } n p i =1 , and the other one contains the feature vectors of an unprotected group, denoted by D u = { a u i } n u i =1 . We want to enhance the fairness of x between these two groups using the ROC-based fairness metric proposed by [74]. Suppose we set a threshold θ and classify data a as positive if x ⊤ a ≥ θ and as negative otherwise. The ROC-based fairness measure and its continuous approximation are defined as

where σ ( z ) = exp( z ) / (1 + exp( z )) is the sigmoid function and Θ is a finite set of thresholds. If the value of this measure is small, model x produces similar predicted positive rates for the protected and unprotected groups on various θ 's, indicating the fairness of the model. To obtain a fair x , we balance (13) and (14) by solving

a9a

bank

COMPAS

Objective

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Iteration t

10 5

0.08

0.1

0.12

0.14

0.16

0.18

0.2

0.22

SSG-static

SSG-diminishing

IPP-SSG

IPP-ConEx

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Iteration t

10 4

0.005

0.01

0.015

0.02

0.025

0.03

0.035

0.04

0.045

0.05

0.055

SSG-static

SSG-diminishing

IPP-SSG

IPP-ConEx

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Iteration t

10 4

0.008

0.01

0.012

0.014

0.016

0.018

0.02

0.022

0.024

SSG-static

SSG-diminishing

IPP-SSG

IPP-ConEx

Infeasibility

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Iteration t

10 5

-5

-4

-3

-2

-1

0

1

2

10 -4

SSG-static

SSG-diminishing

IPP-SSG

IPP-ConEx

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Iteration t

10 4

-4

-3

-2

-1

0

1

2

10 -4

SSG-static

SSG-diminishing

IPP-SSG

IPP-ConEx

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Iteration t

10 4

-1.5

-1

-0.5

0

0.5

1

10 -4

SSG-static

SSG-diminishing

IPP-SSG

IPP-ConEx

Near Stationarity

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Iteration t

10 5

0

0.005

0.01

0.015

0.02

0.025

0.03

0.035

0.04

0.045

SSG-static

SSG-diminishing

IPP-SSG

IPP-ConEx

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Iteration t

10 4

0

0.01

0.02

0.03

0.04

0.05

0.06

SSG-static

SSG-diminishing

IPP-SSG

IPP-ConEx

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Iteration t

10 4

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

SSG-static

SSG-diminishing

IPP-SSG

IPP-ConEx

where κ is a slackness parameter indicating how much we are willing to increase the classification loss in order to reduce R ( x ) to obtain a more fair model. Problem (15) is an instance of (1) satisfying Assumptions 3.1 and 4.1 with ρ = β where β is defined in (77) in Section B.1.

We solve (15) on three datasets: a9a [41], bank [56] and COMPAS [4]. The information of these datasets is given in Table 1. We split each dataset into two subsets with a ratio of 2 : 1 . The larger set is used as D in the constraint and the smaller set is further split into D p and D u based on a binary group variable listed in Table 1.

In our experiments, we choose ℓ ( z ) = (1 -z ) + and first solve (13) using the subgradient method with a large enough number of iterations to obtain L ∗ and a solution x ERM. Then we set κ = 0 . 001 L ∗ and r = 5 ∥ x ERM ∥ , and let Θ consist of 400 points equally spaced between min i x ⊤ ERM a i -0 . 5(max i x ⊤ ERM a i -min i x ⊤ ERM a i ) and max i x ⊤ ERM a i +0 . 5(max i x ⊤ ERM a i -min i x ⊤ ERM a i ) .

All methods are initialized at x ERM. We implemented the SSG method with both static and diminishing stepsizes. For the static stepsize, we select ϵ t from { 10 -6 , 2 × 10 -6 , 5 × 10 -6 , 10 -5 } and η t from { 2 × 10 -4 , 5 × 10 -4 , 10 -3 , 2 × 10 -3 } . For the diminishing stepsize, we set ϵ t = E 1 √ t +1 and η t = E 2 √ t +1 and select E 1 from { 5 × 10 -5 , 10 -4 , 2 × 10 -4 , 5 × 10 -4 } and E 2 from { 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 } . We

select the best set of parameters that produces the smallest objective value after 5000 iterations. For IPP, we select ˆ ρ from max { ρ, 1 } × { 1 , 1 . 5 , 2 } for all three datasets, and the proximal point subproblem is approximately solved by SSG and ConEx both with 100 iterations. For IPP-SSG, we apply a static stepsize with ϵ t and η t tuned in the same way as SSG. For IPP-ConEx, following the notation in [13], we set θ t = t t +1 , η t = c 1 ( t +1) and τ t = c 2 t +1 , and select c 1 from { 20 , 50 , 100 , 200 } and c 2 from { 0 . 002 , 0 . 005 , 0 . 01 , 0 . 02 } by the same procedure adopted by SSG.

We report the performances versus number of iterations of all methods on each dataset in Figure 1. The performances versus the used CPU time will be given in Section C in the Appendix. For SSG, the x -axis represents the total number of iterations while, for IPP, it represents the total number of inner iterations across all outer iterations. The y -axis in each row represents the objective value, infeasibility and near stationarity achieved at each iteration, respectively. To measure near stationarity, we solve (3) with x = x ( t ) and parameters (7) using the SSG method with 2500 iterations and use the last iterate as an approximation of ̂ x ( x ( t ) ) . We make sure that the change of ∥ ̂ x ( x ( t ) ) -x ( t ) ∥ is less than 1% if the number of iterations is increased to 5000 . Then we plot ∥ ̂ x ( x ( t ) ) -x ( t ) ∥ as near stationarity in Figure 1. Since computing ̂ x ( x ( t ) ) with a high precision for each t is time-consuming, we only report near stationarity at 100 equally spaced iterations.

According to Figure 1, the SSG method with a diminishing stepsize has the best performance on all three datasets in the sense that it reduces the objective value and the (approximate) near stationarity measure faster than others while keeping the solutions nearly feasible. However, the SSG method with a static stepsize is not always better than the IPP methods. This is consistent with our theoretical finding that the SSG and IPP methods have the same oracle complexity.

6.2 Classification problem with demographic parity

Following the notation in the previous subsection, we consider a binary classification problem with a constraint enforcing demographic parity [1]. The measure of demographic parity and its continuous approximation are

Fairness measure R 0 ( x ) is a special case of R ( x ) with Θ = { 0 } . If R 0 ( x ) is small, model x produces similar predicted positive rates for the protected and unprotected groups. To obtain a fair x , we balance (13) and (16) by solving

Different from (15), the fairness measure is used as the constraint in (17) while the objective function becomes the empirical hinge loss plus the smoothly clipped absolute deviation (SCAD) regularizer [35] for promoting a sparse solution. Here, λ is a regularization parameter and

We prove in Section B.1 that (17) is an instance of (1) satisfying Assumptions 3.1 and 5.1 with ρ = max { 2 λ, β } where β is defined in (77) in Section B.1.

We set λ = 0 . 2 for all datasets and set κ = 0 . 005 , 0 . 02 and 0 . 02 for a9a , bank and COMPAS , respectively. For SSG, we select ϵ t from { 10 -6 , 2 × 10 -6 , 5 × 10 -6 , 10 -5 } and select η t from { 10 -4 , 2 × 10 -4 , 5 × 10 -4 , 7 . 5 × 10 -4 } for t ∈ I while set η t = g ( x ( t ) ) / ∥ ζ ( t ) g ∥ 2 for t ∈ J . We select the best set of parameters that produces the smallest objective value after 50000 iterations. For IPP, we select ˆ ρ from max { ρ, 1 } × { 1 , 1 . 5 , 2 } and the proximal point subproblem is approximately solved by SSG and ConEx both with 600 iterations. For IPP-SSG, we apply a static stepsize with ϵ t and η t tuned in the same way as SSG. For IPP-ConEx, following the notation in [13], we set θ t = t t +1 , η t = c 1 ( t + 1) and τ t = c 2 t +1 and select c 1 from { 20 , 50 , 100 , 200 } and c 2 from { 0 . 002 , 0 . 005 , 0 . 01 , 0 . 02 } by the same procedure adopted by SSG. Due to the limit of space, we present the performances vs number of iterations and CPU runtime of all methods in Figure 3 and Figure 4 respectively in Section C in the Appendix.

7 Conclusion

We study the oracle complexity of the switching subgradient (SSG) method for finding a nearly ϵ -stationary point of a non-smooth weakly convex constrained optimization problem. We show that the complexity of the SSG method matches the best result in literature that is achieved only by double-loop methods. On the contrary, the SSG method is single-loop and easier to implement with reduced tuning effort. This is the first complexity result for a single-loop first-order method for a weakly-convex non-smooth constrained problem.

References

Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reductions approach to fair classification. In International Conference on Machine Learning , pages 60-69. PMLR, 2018.

Ahmet Alacaoglu, Yura Malitsky, and Volkan Cevher. Convergence of adaptive algorithms for constrained weakly convex optimization. Advances in Neural Information Processing Systems , 34:14214-14225, 2021.

Mohammad S. Alkousa. On modification of an adaptive stochastic mirror descent algorithm for convex optimization problems with functional constraints. In Computational Mathematics and Applications , pages 47-63. Springer, 2020.

Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. ProPublica, May, 23 , 2016.

Yossi Arjevani, Yair Carmon, John C. Duchi, Dylan J. Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming , 199(1-2):165-214, 2023.

Alfred Auslender. An extended sequential quadratically constrained quadratic programming algorithm for nonlinear, semidefinite, and second-order cone programming. Journal of Optimization Theory and Applications , 156(2):183-212, 2013.

Anastasia Bayandina, Pavel Dvurechensky, Alexander Gasnikov, Fedor Stonyakin, and Alexander Titov. Mirror descent and convex optimization problems with non-smooth inequality constraints. In Large-Scale and Distributed Optimization , pages 181-213. Springer, 2018.

Albert S. Berahas, Frank E. Curtis, Michael J. O'Neill, and Daniel P. Robinson. A stochastic sequential quadratic optimization algorithm for nonlinear equality constrained optimization with rank-deficient jacobians. arXiv preprint arXiv:2106.13015 , 2021.

Albert S. Berahas, Frank E. Curtis, Daniel Robinson, and Baoyu Zhou. Sequential quadratic optimization for nonlinear equality constrained stochastic optimization. SIAM Journal on Optimization , 31(2):1352-1379, 2021.

Albert S. Berahas, Miaolan Xie, and Baoyu Zhou. A sequential quadratic programming method with high probability complexity bounds for nonlinear equality constrained stochastic optimization. arXiv preprint arXiv:2301.00477 , 2023.

Jérôme Bolte and Edouard Pauwels. Majorization-minimization procedures and convergence of sqp methods for semi-algebraic and tame programs. Mathematics of Operations Research , 41(2):442-465, 2016.

Digvijay Boob, Qi Deng, and Guanghui Lan. Level constrained first order methods for function constrained optimization. arXiv preprint arXiv:2205.08011 , 2022.

Digvijay Boob, Qi Deng, and Guanghui Lan. Stochastic first-order methods for convex and nonconvex functional constrained optimization. Mathematical Programming , 197(1):215-279, 2023.

Digvijay Boob, Qi Deng, Guanghui Lan, and Yilin Wang. A feasible level proximal point method for nonconvex sparse constrained optimization. Advances in Neural Information Processing Systems , 33:16773-16784, 2020.

Morteza Boroun and Afrooz Jalilzadeh. Inexact-proximal accelerated gradient method for stochastic nonconvex constrained optimization problems. In 2021 Winter Simulation Conference (WSC) , pages 1-12. IEEE, 2021.

James V. Burke. A sequential quadratic programming method for potentially infeasible mathematical programs. Journal of Mathematical Analysis and Applications , 139(2):319-351, 1989.

Coralia Cartis, Nicholas I.M. Gould, and Philippe L. Toint. On the evaluation complexity of composite function minimization with applications to nonconvex nonlinear programming. SIAM Journal on Optimization , 21(4):1721-1739, 2011.

Coralia Cartis, Nicholas I.M. Gould, and Philippe L. Toint. Optimality of orders one to three and beyond: characterization and evaluation complexity in constrained nonconvex optimization. Journal of Complexity , 53:68-94, 2019.

Lesi Chen, Jing Xu, and Luo Luo. Faster gradient-free algorithms for nonsmooth nonconvex stochastic optimization. arXiv preprint arXiv:2301.06428 , 2023.

Yi Cheng, Guanghui Lan, and H. Edwin Romeijn. Functional constrained optimization for risk aversion and sparsity control. arXiv preprint arXiv:2210.05108 , 2022.

Frank E. Curtis, Michael J. O'Neill, and Daniel P. Robinson. Worst-case complexity of an sqp method for nonlinear equality constrained stochastic optimization. arXiv preprint arXiv:2112.14799 , 2021.

Frank E. Curtis, Daniel P. Robinson, and Baoyu Zhou. Inexact sequential quadratic optimization for minimizing a stochastic objective function subject to deterministic nonlinear equality constraints. arXiv preprint arXiv:2107.03512 , 2021.

Damek Davis and Dmitriy Drusvyatskiy. Complexity of finding near-stationary points of convex functions stochastically. arXiv preprint arXiv:1802.08556 , 2018.

Damek Davis and Dmitriy Drusvyatskiy. Stochastic subgradient method converges at the rate o ( k -1 / 4 ) on weakly convex functions. arXiv preprint arXiv:1802.02988 , 2018.

Damek Davis and Dmitriy Drusvyatskiy. Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization , 29(1):207-239, 2019.

Damek Davis, Dmitriy Drusvyatskiy, Kellie J. MacPhee, and Courtney Paquette. Subgradient methods for sharp weakly convex functions. Journal of Optimization Theory and Applications , 179(3):962-982, 2018.

Damek Davis and Benjamin Grimmer. Proximally guided stochastic subgradient method for nonsmooth, nonconvex problems. SIAM Journal on Optimization , 29(3):1908-1930, 2019.

Qi Deng and Wenzhi Gao. Minibatch and momentum model-based methods for stochastic weakly convex optimization. Advances in Neural Information Processing Systems , 34:2311523127, 2021.

Gianni Di Pillo and Luigi Grippo. On the exactness of a class of nondifferentiable penalty functions. Journal of Optimization Theory and Applications , 57(3):399-410, 1988.

Gianni Di Pillo and Luigi Grippo. Exact penalty functions in constrained optimization. SIAM Journal on Control and Optimization , 27(6):1333-1360, 1989.

Yoel Drori and Ohad Shamir. The complexity of finding stationary points with stochastic gradient descent. In International Conference on Machine Learning , pages 2658-2667. PMLR, 2020.

Dmitriy Drusvyatskiy and Courtney Paquette. Efficiency of minimizing compositions of convex functions and smooth maps. Mathematical Programming , 178(1-2):503-558, 2019.

Ivan Ivanovich Eremin. The relaxation method of solving systems of inequalities with convex functions on the left-hand side. In Doklady Akademii Nauk , volume 160, pages 994-996. Russian Academy of Sciences, 1965.

Francisco Facchinei, Vyacheslav Kungurtsev, Lorenzo Lampariello, and Gesualdo Scutari. Ghost penalties in nonconvex constrained optimization: Diminishing stepsizes and iteration complexity. Mathematics of Operations Research , 46(2):595-627, 2021.

Jianqing Fan and Runze Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American statistical Association , 96(456):1348-1360, 2001.

Roger Fletcher. An ℓ 1 penalty method for nonlinear constraints. In Numerical optimization 1984 , pages 26-40. SIAM Publications Philadelphia, 1985.

Wenbo Gao, Donald Goldfarb, and Frank E. Curtis. Admm for multiaffine constrained optimization. Optimization Methods and Software , 35(2):257-303, 2020.

Mingyi Hong, Zhi-Quan Luo, and Meisam Razaviyayn. Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems. SIAM Journal on Optimization , 26(1):337-364, 2016.

Zhichao Jia and Benjamin Grimmer. First-order methods for nonsmooth nonconvex functional constrained optimization with or without slater points. arXiv preprint arXiv:2212.00927 , 2022.

Lingzi Jin and Xiao Wang. A stochastic primal-dual method for a class of nonconvex constrained optimization. Computational Optimization and Applications , 83(1):143-180, 2022.

Ron Kohavi et al. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In KDD , volume 96, pages 202-207, 1996.

Weiwei Kong, Jefferson G. Melo, and Renato D.C. Monteiro. Complexity of a quadratic penalty accelerated inexact proximal point method for solving linearly constrained nonconvex composite programs. SIAM Journal on Optimization , 29(4):2566-2593, 2019.

Weiwei Kong, Jefferson G. Melo, and Renato D.C. Monteiro. Iteration complexity of a proximal augmented lagrangian method for solving nonconvex composite optimization problems with nonlinear convex constraints. Mathematics of Operations Research , 48(2):1066-1094, 2023.

Weiwei Kong, Jefferson G. Melo, and Renato D.C. Monteiro. Iteration complexity of an inner accelerated inexact proximal augmented lagrangian method based on the classical lagrangian function. SIAM Journal on Optimization , 33(1):181-210, 2023.

Weiwei Kong and Renato D.C. Monteiro. An accelerated inexact dampened augmented lagrangian method for linearly-constrained nonconvex composite optimization problems. Computational Optimization and Applications , 85(2):509-545, 2023.

Guy Kornowski and Ohad Shamir. Oracle complexity in nonsmooth nonconvex optimization. Advances in Neural Information Processing Systems , 34:324-334, 2021.

Guy Kornowski and Ohad Shamir. On the complexity of finding small subgradients in nonsmooth optimization. arXiv preprint arXiv:2209.10346 , 2022.

Guanghui Lan, Arkadi Nemirovski, and Alexander Shapiro. Validation analysis of mirror descent stochastic approximation method. Mathematical Programming , 134(2):425-458, 2012.

Guanghui Lan and Zhiqiang Zhou. Algorithms for stochastic optimization with function or expectation constraints. Computational Optimization and Applications , 76(2):461-498, 2020.

Zichong Li, Pin-Yu Chen, Sijia Liu, Songtao Lu, and Yangyang Xu. Rate-improved inexact augmented lagrangian method for constrained nonconvex optimization. In International Conference on Artificial Intelligence and Statistics , pages 2170-2178. PMLR, 2021.

Zichong Li, Pin-Yu Chen, Sijia Liu, Songtao Lu, and Yangyang Xu. Zeroth-order optimization for composite problems with functional constraints. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7453-7461, 2022.

Zichong Li and Yangyang Xu. Augmented lagrangian-based first-order methods for convexconstrained programs with weakly convex objective. INFORMS Journal on Optimization , 3(4):373-397, 2021.

Qihang Lin, Runchao Ma, and Yangyang Xu. Complexity of an inexact proximal-point penalty method for constrained smooth non-convex optimization. Computational Optimization and Applications , 82(1):175-224, 2022.

Runchao Ma, Qihang Lin, and Tianbao Yang. Quadratically regularized subgradient methods for weakly convex optimization with weakly convex constraints. In International Conference on Machine Learning , pages 6554-6564. PMLR, 2020.

Jefferson G. Melo, Renato D.C. Monteiro, and Hairong Wang. Iteration-complexity of an inexact proximal accelerated augmented lagrangian method for solving linearly constrained smooth nonconvex composite optimization problems. arXiv preprint arXiv:2006.08048 , 2020.

Sérgio Moro, Paulo Cortez, and Paulo Rita. A data-driven approach to predict the success of bank telemarketing. Decision Support Systems , 62:22-31, 2014.

Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization , 19(4):15741609, 2009.

Yurii Nesterov. Lectures on convex optimization , volume 137. Springer, 2018.

Boris T. Polyak. A general method of solving extremum problems. Soviet Mathematics Doklady , 8(3):593-597, 1967.

Boris T. Polyak. Minimization of unsmooth functionals. USSR Computational Mathematics and Mathematical Physics , 9(3):14-29, 1969.

Hassan Rafique, Mingrui Liu, Qihang Lin, and Tianbao Yang. Weakly-convex-concave minmax optimization: provable algorithms and applications in machine learning. Optimization Methods and Software , 37(3):1087-1121, 2022.

Philippe Rigollet and Xin Tong. Neyman-pearson classification, convexity and stochastic constraints. Journal of Machine Learning Research , 12:2831-2855, 2011.

Mehmet Fatih Sahin, Ahmet Alacaoglu, Fabian Latorre, Volkan Cevher, et al. An inexact augmented lagrangian framework for nonconvex optimization with nonlinear constraints. Advances in Neural Information Processing Systems , 32:13966-13978, 2019.

Ohad Shamir. Can we find near-approximately-stationary points of nonsmooth nonconvex functions? arXiv preprint arXiv:2002.11962 , 2020.

Fedor S. Stonyakin, Mohammad S. Alkousa, Alexey N. Stepanov, and Aleksandr A. Titov. Adaptive mirror descent algorithms for convex and strongly convex optimization problems with functional constraints. Journal of Applied and Industrial Mathematics , 13(3):557-574, 2019.

Fedor S. Stonyakin, Mohammad S. Alkousa, Alexander A. Titov, and Victoria V. Piskunova. On some methods for strongly convex optimization problems with one functional constraint. In International Conference on Mathematical Optimization Theory and Operations Research , pages 82-96. Springer, 2019.

Fedor S. Stonyakin, Alexey N. Stepanov, Alexander A. Titov, and Alexander V. Gasnikov. Mirror descent for constrained optimization problems with large subgradient values. arXiv preprint arXiv:1908.00218 , 2019.

Fedor S. Stonyakin and Alexander A. Titov. One mirror descent algorithm for convex constrained optimization problems with non-standard growth properties. arXiv preprint arXiv:1803.01329 , 2018.

Arnesh Sujanani and Renato D.C. Monteiro. An adaptive superfast inexact proximal augmented lagrangian method for smooth nonconvex composite optimization problems. Journal of Scientific Computing , 97(2):34, 2023.

Lai Tian and Anthony Man-Cho So. Computing goldstein ( ϵ, δ ) -stationary points of lipschitz functions in ˜ O ( ϵ -3 δ -1 ) iterations via random conic perturbation. arXiv preprint arXiv:2112.09002 , 2021.

Lai Tian, Kaiwen Zhou, and Anthony Man-Cho So. On the finite-time complexity and practical computation of approximate stationarity concepts of lipschitz functions. In International Conference on Machine Learning , pages 21360-21379. PMLR, 2022.

Alexander A. Titov, Fedor S. Stonyakin, Mohammad S. Alkousa, Seydamet S. Ablaev, and Alexander V. Gasnikov. Analogues of switching subgradient schemes for relatively lipschitzcontinuous convex programming problems. In International Conference on Mathematical Optimization Theory and Operations Research , pages 133-149. Springer, 2020.

Alexander A. Titov, Fedor S. Stonyakin, Alexander V. Gasnikov, and Mohammad S. Alkousa. Mirror descent and constrained online optimization problems. In International Conference on Optimization and Applications , pages 64-78. Springer, 2019.

Robin Vogel, Aurélien Bellet, and Stéphan Clémençon. Learning fair scoring functions: Bipartite ranking under roc-based fairness constraints. In International Conference on Artificial Intelligence and Statistics , pages 784-792. PMLR, 2021.

Yu Wang, Wotao Yin, and Jinshan Zeng. Global convergence of admm in nonconvex nonsmooth optimization. Journal of Scientific Computing , 78(1):29-63, 2019.

Tianbao Yang and Qihang Lin. RSG: Beating subgradient method without smoothness and strong convexity. Journal of Machine Learning Research , 19(1):236-268, 2018.

Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodriguez, and Krishna P. Gummadi. Fairness constraints: A flexible approach for fair classification. Journal of Machine Learning Research , 20(1):2737-2778, 2019.

Jinshan Zeng, Wotao Yin, and Ding-Xuan Zhou. Moreau envelope augmented lagrangian method for nonconvex optimization with linear constraints. Journal of Scientific Computing , 91(2):61, 2022.

Jiawei Zhang and Zhi-Quan Luo. A proximal alternating direction method of multiplier for linearly constrained nonconvex minimization. SIAM Journal on Optimization , 30(3):2272-2302, 2020.

Jiawei Zhang and Zhi-Quan Luo. A global dual error bound and its application to the analysis of linearly constrained nonconvex optimization. SIAM Journal on Optimization , 32(3):2319-2346, 2022.

Jiawei Zhang, Wenqiang Pu, and Zhi-Quan Luo. On the iteration complexity of smoothed proximal alm for nonconvex optimization problem with convex constraints. arXiv preprint arXiv:2207.06304 , 2022.

Jingzhao Zhang, Hongzhou Lin, Stefanie Jegelka, Suvrit Sra, and Ali Jadbabaie. Complexity of finding stationary points of nonconvex nonsmooth functions. In International Conference on Machine Learning , pages 11173-11182. PMLR, 2020.

Siqi Zhang and Niao He. On the convergence rate of stochastic mirror descent for nonsmooth nonconvex optimization. arXiv preprint arXiv:1806.04781 , 2018.

Peer reviews:
comments: SUMMARY:
This paper studies switching subgradient method for solving weakly-convex objective (weakly-)convex constrained problems. The oracle complexity for finding a nearly stationary point is derived for both convex constraint case and weakly-convex constrained (with good initialization). Numerical experiments validate the efficiency of the proposed algorithms on classical machine learing tasks.

STRENGTHS:
The paper is overall well written. It is the first single-loop first-order method handling a weakly-convex objective (weakly-)convex constrained problems, which is much convenient to implement compared to double-loop algorithms. The theoretical complexity result is clearly stated and competitive with sound proofs.

WEAKNESSES:
No obvious weaknesses and please see the questions part for some detailed comments.

QUESTIONS:
1. For the weakly-convex constrained case, if we cannot have feasible initial point, does the algorithm SSG converge? If so, what kind of point will it arrive? As we know, it is usually a NP-hard problem to obtain a feasible point for nonconvex constraint. In this case, does the SSG converge to the stationary point satisfying $0\in \partial (f+1_{\mathcal{X}\cap [g\leq0]})(x)$?
2. In Assumption 4.1, it is assumed that the Slater's condition holds. Does it mean that we cannot include the equality constraint $g(x)=0$ for the following analysis?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
The paper considers a weakly-convex constrained optimization problem and establishes the oracle complexity of the switching subgradient method for finding a nearly stationary point.

STRENGTHS:
1. Considering a single-loop algorithm, the paper can establish the complexity $O(1/\epsilon^4)$ for a case of weakly-convex constraint. This is a nice result. 

2. The paper has a rigorous convergence analysis.

WEAKNESSES:
In the numerical experiments, the paper compares a single-loop method with double-loop methods but reports the evolution of objective functions over iterations. This is not fair. The evolution over time should be reported.

QUESTIONS:
Can the analysis be extended to the case when Assumption 4.1 C is removed (for example, when $\mathcal X$ is unbounded)?

LIMITATIONS:
n.a.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This paper examines the oracle complexity of the switching subgradient method used for solving non-convex constrained optimization problems. The target functions are weakly convex, while the constraint functions are either convex or weakly convex. Remarkably, this method matches the complexity of double-loop methods while only necessitating a single loop implementation.

STRENGTHS:
The authors have produced a commendable piece of work in the domain of weakly convex optimization, a specific category of nonconvex problems noted for its favorable theoretical convergence attributes. Although there is an abundance of single-loop algorithms such as subgradient or proximal point for weakly convex optimization, much of the existing literature has focused on unconstrained or simply-constrained issues. It is therefore somewhat unexpected that there are no analogous results in functionally constrained settings. Seen in this light, the technical novelty and contribution of this paper are remarkable.

WEAKNESSES:
While the authors have provided a comprehensive review of the literature, there seems to be an overemphasis on marginally relevant references. The broad coverage of optimization beyond weakly convex settings, although interesting, does not directly contribute to the key message of the paper.

QUESTIONS:
Assumption 4.1. assumes the boundedness of the solution, is this easily satisfied in practical scenarios?  Moreover,  should you include such a constraint in your second application?



Can you extend your algorithm for weakly convex problems with linear equality constraints?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-13