Title: Faster Accelerated First-order Methods for Convex Optimization with Strongly Convex Function Constraints
Venue: NeurIPS 2024
Accepted: yes
Keywords: Convex optimization; Accelerated primal dual algorithm; Sparse Optimization
Abstract: In this paper, we introduce faster accelerated primal-dual algorithms for minimizing a convex function subject to strongly convex function constraints. 
Prior to our work, the best complexity bound was $\mathcal{O}(1/{\varepsilon})$, regardless of the strong convexity of the constraint function.
It is unclear whether the strong convexity assumption can enable even better convergence results. 
To address this issue, we have developed novel techniques to progressively estimate the strong convexity of the Lagrangian function.
Our approach, for the first time, effectively leverages the constraint strong convexity, obtaining an improved complexity of $\mathcal{O}(1/\sqrt{\varepsilon})$. This rate matches the complexity lower bound for strongly-convex-concave saddle point optimization and is therefore order-optimal.
We show the superior performance of our methods in sparsity-inducing constrained optimization, notably Google's personalized PageRank problem. Furthermore, we show that a restarted version of the proposed methods can effectively identify the optimal solution's sparsity pattern within a finite number of steps, a result that appears to have independent significance.

Faster Accelerated First-order Methods for Convex Optimization with Strongly Convex Function Constraints

Abstract

In this paper, we introduce faster accelerated primal-dual algorithms for minimizing a convex function subject to strongly convex function constraints. Prior to our work, the best complexity bound was O (1 /ε ) , regardless of the strong convexity of the constraint function. It is unclear whether the strong convexity assumption can enable even better convergence results. To address this issue, we have developed novel techniques to progressively estimate the strong convexity of the Lagrangian function. Our approach, for the first time, effectively leverages the constraint strong convexity, obtaining an improved complexity of O (1 / √ ε ) . This rate matches the complexity lower bound for strongly-convex-concave saddle point optimization and is therefore order-optimal. We show the superior performance of our methods in sparsity-inducing constrained optimization, notably Google's personalized PageRank problem. Furthermore, we show that a restarted version of the proposed methods can effectively identify the optimal solution's sparsity pattern within a finite number of steps, a result that appears to have independent significance.

1 Introduction

In this paper, we are interested in the following convex function-constrained problem:

where f : R n → R is a convex continuous function and bounded from below and g i : R n → R , i = 1 , 2 , . . . , m , are strongly convex continuous functions. An important application of this problem, commonly encountered in statistics and engineering, involves the objective f ( x ) as a proximalfriendly regularizer and g i ( x ) as a data-driven loss function used to gauge model fidelity.

To apply first-order methods for the above function-constrained problems, a common strategy involves a double-loop procedure that repeatedly employs fast first-order methods, such as Nesterov's accelerated method, to solve specific strongly convex proximal subproblems. Popular methods among this category include Augmented Lagrangian methods [18, 33], level-set methods [21], penalty methods [17]. When both f ( x ) and g i ( x ) are convex and smooth (or composite), it has been found that these double-loop algorithms can attain an iteration complexity of O (1 /ε ) to achieve an ε -error

in both the optimality gap and constraint violation. When the objective is strongly convex, the complexity can be further improved to O (1 / √ ε ) ([33, 21]).

In contrast to these double-loop algorithms, single-loop algorithms remain popular due to their simplicity in implementation. Along this research line, [32] developed a first-order algorithm based on linearizing the augmented Lagrangian function, which obtains an iteration complexity of O (1 /ε ) . [34] extended the augmented Lagrangian method to stochastic function-constrained problems where both objective and constraint exhibit an expectation form. Viewing (1) as a special case of the min-max problem:

[11] proposed to solve (1) and (2) by an accelerated primal-dual method (APD), which generalizes the primal-dual hybrid gradient method [6] initially developed for saddle point optimization with bilinear coupling term. Under mild conditions, APD achieves the best iteration complexity of O (1 /ε ) for general convex constrained problem and a further improved complexity of O (1 / √ ε ) when f ( x ) is strongly convex. [4] proposed a unified constrained extrapolation method that can be applied to both deterministic and stochastic constrained optimization problems.

Despite these recent progresses, to the best of our knowledge, all available algorithms are suboptimal in the presence of strongly convex function constraints (1). Specifically, direct applications of previously discussed algorithms yield an O (1 /ε ) complexity, which is inferior to the O (1 / √ ε ) optimal bound for the strongly-convex-concave saddle point problem [22]. It is somewhat unsatisfactory that the strong convexity of g ( x ) has not been found helpful in further algorithmic acceleration. The core underlying issue arises from the dynamics of saddle point optimization: it is the strong convexity of L ( · , y ) that offers more potential acceleration advantages, yet the strong convexity of L ( · , y ) is substantially harder to estimate than that of g ( x ) . This difficulty is compounded by the interplay between g ( x ) and the varying dual sequence { y k } . The challenge naturally leads us to question: Is it possible to further improve the convergence rate of first-order methods for solving the strongly convex constrained problem (1) ?

Key intuitions We make an assumption that the minimizer of f ( x ) is infeasible for the function constraint g i ( x ) ≤ 0 , 1 ≤ i ≤ m . If this assumption were not made, we would be dealing with an unconstrained optimization problem that would not depend on g ( x ) . This assumption also implies that the optimal dual variables are non-zero, and as a result, the Lagrangian function is strongly convex with respect to x . By leveraging the strong convexity, we can use more aggressive step sizes and achieve faster convergence rates compared to other state-of-the-art algorithms.

Applications in sparsity-constrained optimization We consider the constrained Lasso-type problem, which minimizes a sparsity-inducing regularizer while explicitly ensuring data-driven error remains controlled:

where g ( · ) is a convex smooth loss term. A motivating application is the approximate personalized PageRank problem [8], where g ( x ) = 1 2 ⟨ x , Q x ⟩ - ⟨ b , x ⟩ is strongly convex quadratic and Q integrates the graph Laplacian with an identity matrix. Compared to the standard Lasso problem [30], min x ∈ R n g ( x )+ λ ∥ x ∥ 1 , the constrained problem (3) offers enhanced control over the data fitting error. This advantage, however, is counterbalanced by the challenge of dealing with a nonlinear constraint. Besides concerns about the efficiency in solving (3), it is often desired to show the active set (or sparsity) identification, namely, the nonzero patterns of the optimal solution x ∗ can be identified by the solution sequence { x k } in a finite number of iterations. Identifying the embedded solution structure within a broader context is referred to as the manifold identification problem [31, 12]. Exploiting the sparsity pattern is particularly desirable in large-scale PageRank problems, as it could result in significant runtime savings. For the regularized Lasso-type problem, it has been known that proximal gradient methods (e.g. [14, 19, 24]) possess the finite active-set identification property. Specifically, [24] introduced 'active set complexity', which is defined as the number of iterations required before an algorithm is guaranteed to have reached the optimal manifold, and they proved the proximal gradient method with constant stepsize can identify the optimal manifold in a finite number of iterations. However, for the problem (3), it remains unclear whether first-order methods can identify the sparsity pattern in finite time.

Contributions We address the theoretical questions about strongly convex constrained optimization and the application of sparse optimization. Our contributions are summarized as follows.

First, we present a new accelerated primal-dual algorithm with progressive strong convexity estimation (APDPro) for solving problem (1). APDPro employs a novel strategy to estimate the lower bound of the dual variables, which leads to a gradually refined estimated strong convexity modulus of L ( · , y ) . With additional cut constraints on the dual update, APDPro is able to separate the dual search space from the origin point, which is critical for maintaining the desired strong convexity over the entire solution path. With these two important ingredients, APDPro exhibits an O ( ( ∥ x 0 -x ∗ ∥ + D Y ) / √ ε ) complexity bound to obtain an ε -error on the function value gap and constraint violation, where D Y is a known upper-bound of ∥ y 0 -y ∗ ∥ . Moreover, we show that for the last iterate to have an ε error (i.e., ∥ x K -x ∗ ∥ 2 ≤ ε ), APDPro requires a total iteration of O ( ( ∥ x 0 -x ∗ ∥ + ∥ y 0 -y ∗ ∥ ) / √ ε ) . Both complexity results appear new in the literature for strongly convex-constrained optimization.

Second, we present a new restart algorithm (rAPDPro) which calls APDPro repeatedly with the input parameters properly changing over time. Different from APDPro, rAPDPro dynamically adjusts the iteration number of APDPro in each epoch based on the progressive strong convexity estimation. We show that rAPDPro exhibits a complexity of O ( log( D X / √ ε ) + D Y / √ ε ) to ensure ε -error in the last iterate convergence where D X is the estimated diameter of the primal feasible domain. While it is difficult to improve the overall O (1 / √ ε ) bound, rAPDPro appears to be more advantageous when D X and D Y are the same order of ∥ x 0 -x ∗ ∥ and ∥ y 0 -y ∗ ∥ , respectively, and D X ≫ D Y . In addition, we show that a similar restart strategy can further accelerate the standard APD. The multistage-accelerated primal dual method (msAPD) obtains a comparable O (1 / √ ε ) complexity of APDPro without introducing additional cut constraint.

Third, we apply our proposed methods to the sparse learning problem (3). In view of the theoretical analysis, all our methods converge at an O (1 / √ ε ) rate, which is substantially better than the rates of state-of-the-art first-order algorithms. Moreover, we conduct a new analysis to show that the restart algorithm rAPDPro has the favorable feature of identifying the optimal sparsity pattern. Note that such active-set/manifold identification is substantially more challenging to prove due to the coupling of dual variables and constraint functions. To establish the desired property, we develop asymptotic convergence of the dual sequence to the optimal solution, which can be of independent interest.

Outline Section 2 sets notations and assumptions for the later analysis. Section 3 presents the APDPro algorithm and develops its stepsize rule and complexity rate. Section 4 presents the restart APDPro (rAPDPro) algorithm. Section 5 applies our proposed methods for sparsity-inducing optimization and shows the sparsity identification result for rAPDPro. Section 6 empirically examine the convergence performance and sparsity identification of our proposed algorithms. Finally, we draw the conclusion in Section 7. All the missing proofs are provided in the appendix sections.

2 Preliminaries

We use bold letters like x to represent vectors. Suppose x ∈ R n , q ≥ 1 , we use ∥ x ∥ q = ( ∑ n i =1 | x ( i ) | q ) 1 /q to represent the l q -norm, where x ( i ) is the i -th element of x . For brevity, ∥ x ∥ stands for l 2 -norm. For a matrix A , we denote the matrix norm induced by 2-norm as ∥ A ∥ = sup ∥ x ∥≤ 1 ∥ A x ∥ . The normal cone of U at u is denoted as N U ( u ) := { v | ⟨ v , x -u ⟩ ≤ 0 , ∀ x ∈ U} . Let B ( x , r ) be the closed ball centered at x with radius r > 0 , i.e., B ( x , r ) = { y | ∥ y -x ∥ ≤ r } . We denote the set of feasible solutions by X G := { x | g i ( x ) ≤ 0 , ∀ i ∈ [ m ] } and write the constraint function as G ( x ) := [ g 1 ( x ) , . . . , g m ( x )] ⊤ . We assume each g i ( x ) is a µ i strongly convex function, and denote µ := [ µ 1 , . . . , µ m ] ⊤ . Let [ m ] := { 1 , . . . , m } for integer m . We denote minimum and maximum strongly convexity µ := min j ∈ [ m ] { µ j } , and ¯ µ := max j ∈ [ m ] { µ j } and the vector of elements 0 by 0 . The Lagrangian function of problem (1) is given by L ( x , y ) := f ( x ) + ⟨ y , G ( x ) ⟩ where y ∈ R m + .

Definition 1 (KKT condition) . We say that x ∗ satisfies the KKT condition of (1) if there exists a Lagrangian multiplier vector y ∗ ∈ R m + such that 0 ∈ ∂ x L ( x ∗ , y ∗ ) and ⟨ y ∗ , G ( x ∗ ) ⟩ = 0 .

The KKT condition is necessary for optimality when a constraint qualification (CQ) holds at x ∗ . We assume Slater's CQ (Assumption 1) holds, which guarantees that an optimal solution is also a KKT point [3].

Assumption 1. There exists a strictly feasible point ˜ x ∈ R n such that G ( ˜ x ) < 0 .

We use ˜ x to denote a strictly feasible point throughout the paper. Moreover, we require Assumption 2 to circumvent any trivial solution.

Assumption 2. For any x ∗ 0 ∈ argmin x ∈ R n f ( x ) , there exists an i ∈ [ m ] such that g i ( x ∗ 0 ) > 0 .

Remark 1. Assumption 2 is essential for our analysis. While verifying Assumption 2 can be indeed challenging, it is achievable for the sparsity-inducing problem considered in our paper. In this example, the solution x ∗ 0 = 0 is the single minimizer of the sparsity penalty.

Next, we give several useful properties about the optimal solutions of problem (1). Please refer to Appendix D.1 for the proof of Proposition 1 and Appendix D.2 for the proof of Proposition 2.

Proposition 1. Suppose Assumption 1 holds. Then, for any optimal solution x ∗ of problem (1) , there exists y ∗ ∈ R m such that KKT condition holds. Moreover, y ∗ falls into set Y := { y | ∥ y ∥ 1 ≤ ¯ c } , where ¯ c := f ( ˜ x ) -min x ∈ R n f ( x ) min i ∈ [ m ] {-g i ( ˜ x ) } .

Proposition 2. Under Assumption 2, x ∗ is the unique solution of (1) . Furthermore, set Y ∗ = argmax y ∈ R m + L ( x ∗ , y ) is convex and bounded.

In view of Assumption 2, Proposition 2, and closedness of the subdifferential set of proper convex functions [2, Theorem 3.9], [27, Chapter 23], we know that dist ( ∂f ( x ∗ ) , 0 ) > 0 , where dist ( ∂f ( x ∗ ) , 0 ) := min ξ ∈ ∂f ( x ∗ ) ∥ ξ ∥ . Furthermore, we make the following assumption:

Assumption 3. Throughout the paper, suppose that a constant r satisfying

̸

is known.

We give some important examples for which the lower bound r can be estimated. Suppose f ( x ) is a Lasso regularizer, i.e., f ( x ) = ∥ x ∥ 1 , then r = 1 satisfies (4). More general, consider the group Lasso regularizer, i.e., f ( x ) = ∑ B i =1 p i ∥ x ( i ) ∥ , where x ( i ) ∈ R b i and ∑ B i =1 b i = n , B is the number of blocks, then r = min i ∈ [ B ] { p i } when x ∗ = 0 . Another example is f ( x ) = c ⊤ x , then we have r = ∥ c ∥ .

Remark 2. Condition (4) is similar to the bounded gradient assumption that has been used for accelerating the convergence of the Frank-Wolfe algorithm. See Appendix B for more discussions.

When considering the Lipschitz continuity of function in R n , even quadratic functions are not Lipschitz continuous. However, the Lipschitz continuity of g i ( x ) is crucial for algorithm convergence. Therefore, we define the bounded feasible region in the following proposition, with its proof provided in Appendix D.3.

Proposition 3. Let X := B ( ˜ x , min i ∈ [ m ] 2 √ -2 g i ( x ∗ i ) µ i ) , where x ∗ i = argmin x ∈ R n g i ( x ) . Then under Assumptions 1 and 2, we have x ∗ ∈ int X .

Assumption 4. There exist L X , L G > 0 such that

where ∇ G ( x ) := [ ∇ g 1 ( x ) , · · · , ∇ g m ( x )] ∈ R n × m and X is defined in Proposition 3.

The Lipschitz smoothness of the Lagrangian function with respect to the primal variable x is crucial for the convergence of algorithms. Given that the dual variable y is bounded from above, and considering the smoothness of the constraint functions, we can derive the smoothness of the Lagrangian function. Combining (5) and the fact ∥ y ∥ ≤ ∥ y ∥ 1 ≤ ¯ c, ∀ y ∈ Y , we obtain that

where L XY = ¯ cL X . For set X , Y , we use D X and D Y to denote their diameters, respectively, i.e., D X := max x 1 , x 2 ∈X ∥ x 1 -x 2 ∥ and D Y := max y 1 , y 2 ∈Y ∥ y 1 -y 2 ∥ .

Algorithm 1 Accelerated Primal-Dual Algorithm with Progressive Strong Convexity Estimation (APDPro)

Require: τ 0 > 0 , σ 0 > 0 , x 0 ∈ X , y 0 ∈ Y , ρ 0 ≥ 0 , N > 0

1: Initialize: ( x -1 , y -1 ) ← ( x 0 , y 0 ) , ¯ x 0 ← x 0 , σ -1 ← σ 0 , T 0 = 0

2: Set ∆ XY = 1 2 τ 0 D 2 X + 1 2 σ 0 D 2 Y

3: for k = 0 , 1 , . . . , N do

4: Y k ← { y ∈ R m + | ∥ y ∥ 1 · µ ≥ ρ k }⋂ Y ,

5: z k ← (1 + σ k -1 /σ k ) G ( x k ) -( σ k -1 /σ k ) G ( x k -1 )

6: y k +1 ← argmin y ∈Y k ∥ y -( y k + σ k z k ) ∥ 2

7: x k +1 ← prox f, X ( x k -τ k ∇ G ( x k ) y k +1 , τ k )

8: Compute t k , ¯ x k +1 ← ( T k ¯ x k + t k x k +1 ) / ( T k + t k ) , T k +1 ← T k + t k

9: Update ρ k +1 ← IMPROVE( x k , ¯ x k , σ 0 τ k -1 ∆ XY σ k -1 , ∆ XY T k , ρ k )

10: Update τ k +1 and σ k +1 depending on ρ k +1

11: end for

12: Output: x N +1 , y N +1

13: procedure IMPROVE( x , ¯ x , β , ¯ β , ρ old )

15: Set ρ new = max { ρ old , ρ }

16: return ρ new

17: end procedure

3 APD with progressive strong convexity estimation

We present the Accelerated Primal-Dual Algorithm with Progressive Strong Convexity Estimation (APDPro) to solve problem (1). For problem (1), APDPro achieves the improved convergence rate O (1 / √ ε ) without relying on the uniform strong convexity assumption [11, 22]. For the rest of this paper, we denote prox f, X ( x -η z , η ) := argmin ˆ x ∈X f (ˆ x ) + ⟨ z , ˆ x ⟩ + 1 2 η ∥ ˆ x -x ∥ 2 as the proximal mapping.

We describe APDPro in Algorithm 1. The main component of APDPro contains a dual ascent step to update y k based on the extrapolated gradient, followed by a primal proximal step to update x k . Compared with standard APD [11], APDPro has two more steps. First, line 4 of Algorithm 1 applies a novel cut constraint to separate the dual sequence { y k } from the origin, which allows us to leverage the strong convexity of the Lagrangian function and hence obtain a faster rate of convergence than APD. Second, to use the strong convexity more effectively, in line 9, we perform a progressive estimation of the strong convexity by using the latest iterates x k and ¯ x k . Throughout the algorithm process, we use a routine IMPROVE to construct a non-decreasing sequence { ρ k } , which provides increasingly refined lower bounds of the strong convexity of the Lagrangian function.

The IMPROVE step In order to estimate the strong convexity of the Lagrangian function, we rely on the subdifferential separation (eq. (4)) to bound the dual variables. From the first-order optimality condition in minimizing L ( x , y ∗ ) and the fact that x ∗ ∈ int X (Proposition 3), we have 0 ∈ ∂f ( x ∗ ) + ∇ G ( x ∗ ) y ∗ + N X ( x ∗ ) = ∂f ( x ∗ ) + ∇ G ( x ∗ ) y ∗ . It follows from (4) that

where the last inequality use the fact that ∥ · ∥ ≤ ∥ · ∥ 1 . Note that the bound ∥ y ∗ ∥ 1 ≥ r/ ∥∇ G ( x ∗ ) ∥ can not be readily used in the algorithm implementation because x ∗ is generally unknown. To resolve this issue, we develop more concrete dual lower bounds by using the generated solution ˆ x in the proximity of x ∗ . As we will show in the analysis, APDPro keeps track of two primal sequences { x k } and { ¯ x k } , for which we can establish bounds on ∥ x k -x ∗ ∥ 2 and ( y ∗ ) ⊤ µ · ∥ ˆ x -x ∗ ∥ 2 / 2 , respectively. This drives us to develop the following lower bound property, with the proof provided in Appendix E.1.

Proposition 4. Suppose Assumption 4 holds. Let y ∗ ∈ Y ∗ be a dual optimal solution.

Suppose ( y ∗ ) ⊤ µ · ∥ ˆ x -x ∗ ∥ 2 ≤ 2 β , then we have

Our next goal is to conduct the convergence analysis for APDPro in Theorem 1 and Corollary 1. Complete proof details are provided in Appendix E.2 and E.3.

Theorem 1. Suppose for any y ∗ ∈ Y ∗ , ( y ∗ ) ⊤ µ ≥ ρ 0 holds, and let the sequence { τ k , σ k , t k , ρ k +1 } generated by Algorithm 1 satisfy:

Then, the set Y k is nonempty and Y ∗ ⊆ Y k . Let ∆( x , y ) := 1 2 τ 0 ∥ x -x 0 ∥ 2 + 1 2 σ 0 ∥ y -y 0 ∥ 2 , ¯ y K = T -1 K ∑ K -1 s =0 t s y s . The sequence { ¯ x k , x k , ¯ y k } generated by APDPro satisfies

Next, we develop more concrete complexity results in Corollary 1.

Corollary 1. Suppose that σ k , τ k , t k satisfy:

Then we have

where c ∗ := ( f ( x ∗ ) -min x f ( x ) ) / min i ∈ [ m ] {-g i ( ˜ x ) } > 0 , ˜ ρ k = 2 ∑ k s =0 ˆ ρ s s/ ( k ( k +1) ) and ˜ ρ k satisfy the following condition, ˆ ρ k +1 := √ ˆ ρ 2 k k 2 +(3 ρ k +1 ˆ ρ k ) k/ ( k +1) , ˆ ρ 1 = 3 √ ρ 1 /τ 0 .

Remark 3. In view of Corollary 1, APDPro obtains an iteration complexity of O (1 / √ ˜ ρ K ε ) , which is substantially better than the O (1 /ε ) bound of APD [11] and ConEx [4] when the strong convexity parameter ˜ ρ K is relatively large compared with ε .

Remark 4. Additionally, we argue that even when ˜ ρ K = O ( ε ) , APDPro can obtain the matching O (1 /ε ) bound of the state-of-the-art algorithms. Specifically, using the definition of σ k , τ k , we can easily derive the monotonicity of { σ k } . It follows from σ k +1 = τ k σ k /τ k +1 = τ k σ k / ( τ k / √ 1 + ρ k +1 τ k ) ≥ σ k , that T k = ∑ k -1 s =0 t k = σ -1 0 ∑ k -1 s =0 σ k ≥ k . Using a similar argument to that of Corollary 1, we obtain the bound f (¯ x K ) -f ( x ∗ ) ≤ O (1 /K ) and ∥ [ G (¯ x K )] + ∥ ≤ O (1 /K ) .

Remark 5. The implementation of APDPro requires knowing an upper bound on ∥ y ∗ ∥ . When the bound is unavailable, [11] developed an adaptive APD which still ensures the boundedness of dual sequence via line search. Since our main goal of this paper is to exploit the lower-bound rather than the upper bound of ∥ y ∗ ∥ , we leave the extension for the future work.

4 APDPro with a restart scheme

Note that in the worst case, APDPro exhibits an iteration complexity of O ( ( D X + D Y ) / √ ε ) , which has a linear dependence on the diameter. While the O (1 / √ ε ) is optimal [25], it is possible to improve the complexity with respect to the primal part from O ( D X / √ ε ) to O ( log ( D X / √ ε )) . To achieve this goal, we propose a restart scheme (rAPDPro) that calls APDPro repeatedly and present the details in Algorithm 2. Inspired by [16], we set the iteration number as a function of the estimated strong convexity, detailed in the TERMINATEITER procedure. For convenience in describing a double-loop algorithm, we use superscripts for the number of epochs and subscripts for the number of

Algorithm 2 Restarted APDPro (rAPDPro)

Require: ρ -1 ≥ 0 , ¯ σ > 0 , ν 0 ∈ (0 , 1) , δ ∈ (0 , 1) , x -1 , y -1 , S

N -1 N -1 N -1 1: Compute ¯ τ = (1 -ν 0 ) ( L XY + L 2 G ¯ σ/δ ) -1 2: for s = 0 , 1 , . . . , S do 3: τ s 0 = ¯ τ, σ s 0 = ¯ σ, ( x s -1 , y s -1 ) ← ( x s -1 N s -1 , y s -1 N s -1 ) , ( x s 0 , y s 0 ) ← ( x s -1 N s -1 , y s -1 N s -1 ) , ρ s 0 = ρ s -1 N s -1 4: Set ∆ XY = 1 τ s 0 D 2 X + 1 2 σ s 0 D 2 Y , σ s -1 ← σ s 0 , T s 0 = 0 , k = 0 , ˆ ρ s 0 = 1 , N s = ∞ 5: while k < N s do 6: Run line 4-10 of APDPro with index set ( s, k ) 7: Update N s , ˆ ρ s k +1 ← TERMINATEITER( ˆ ρ s k , ρ s k +1 , s, k ), k ← k +1 8: end while 9: end for 10: Output: x S N S , y S N S 11: procedure TERMINATEITER( ˆ ρ old , ρ, s, k ) 12: Compute ˆ ρ new = { 1 k +1 √ ˆ ρ 2 old k 2 +3 ρ ˆ ρ old k k > 1 3 √ ρ/τ 0 k = 1 13: Compute N = ⌈ max { 6(ˆ ρ new τ s 0 ) -1 , √ 2 s · 3 √ 2 D Y / ( ˆ ρ new D X √ τ s 0 σ s 0 ) }⌉ 14: return N, ˆ ρ new 15: end procedure

sub-iterations in parameters x , y , τ, σ , e.g., x S 1 meaning the x output of first iterations at S -th epoch. To avoid redundancy in the Algorithm 2, we call the APDPro iteration directly. Note that the notation system here is identical to that of APDPro, with the only difference being the use of superscripts to distinguish the number of epochs.

In Theorem 2, we show the overall convergence complexity of rAPDPro with the proof provided in Appendix F.1.

Theorem 2. Let { x s 0 } s ≥ 0 be the sequence generated by rAPDPro , then we have

As a consequence, rAPDPro will find a solution x S 0 such that ∥ x S 0 -x ∗ ∥ 2 ≤ ε for any ε ∈ (0 , D 2 X ) in at most S := ⌈ log 2 ( D 2 X /ε ) ⌉ epochs. Moreover, The iteration number of rAPDPro to find x S 0 such that ∥ x S 0 -x ∗ ∥ 2 ≤ ε is bounded by

where ϖ 1 and ϖ 2 satisfy ∑ S s =0 (ˆ ρ s N s ) -1 = ( ϖ 1 ) -1 ( S + 1) and ∑ S s =0 √ 2 s / ˆ ρ s N s = ( ϖ 2 ) -1 ∑ S s =0 √ 2 s , respectively.

Remark 6. The bound T ε depends on ε , ϖ 1 and ϖ 2 . If ϖ 1 = O ( ( -log 2 √ ε ) -1 ) or ϖ 2 = O ( √ ε ) , then we have T ε = ∞ , which implies that we can not guarantee ∥ x s 0 -x ∗ ∥ ≤ ε at finite iterations. T ε = ∞ implies that there exists an epoch with infinite sub-iterations. Hence, rAPDPro is reduced to APDPro if we only consider that epoch.

Remark 7. Comparison of rAPDPro and APDPro involves a number of factors. In particular, rAPDPro compares favorably against APDPro if ∥ x 0 -x ∗ ∥ = ˜ Ω( √ ε log D X ) . Moreover, the complexity (16) can be slightly improved if D X is replaced by any tighter upper bound of ∥ x s 0 -x ∗ ∥ . However, it is still unknown whether we can directly replace D X with ∥ x s 0 -x ∗ ∥ in (16) .

Dual Convergence For dual variables, we establish asymptotic convergence to the optimal solution, a key condition for developing the active-set identification in the later section. For ease in notation, it is more convenient to label the generated solution as a whole sequence using a single subscript index: x 1 , x 2 , . . . , x N ; y 1 , y 2 , . . . , y N . Hence, we use the index system j and ( s, k ) interchangeably. Note that { x s +1 0 , y s +1 0 } and { x s N s +1 , y s N s +1 } correspond to the same pair of points. We present the dual asymptotic result in the following theorem, with the proof provided in Appendix F.2.

Theorem 3. Assume ¯ τ -1 > ρ and choose ν 0 > 0 such that 1 > inf j ≥ 0 { σ j -1 /σ j } ≥ δ + ν 0 . We have ( x ∗ , y ∗ ) satisfy the KKT condition, where y ∗ is any limit point of { y j } generated by rAPDPro .

Remark 8. To establish the asymptotic convergence of the dual variable, we introduce an additional constant δ ∈ (0 , 1) , which implies that the initial step size must meet a stricter requirement than the convergence condition specified in Corollary 1. Since σ s k /σ s k -1 = √ 1 + ρ s k τ s k , { ρ s k } is bounded due to the boundedness of the dual variable, { τ s k } is monotonically decreasing, then inf 0 ≤ k ≤ N s { σ s k -1 /σ s k } ≥ (1 + ρ ¯ τ ) -1 / 2 . Hence, inequality, 1 > inf j ≥ 0 { σ j -1 /σ j } ≥ δ + ν 0 , is always satisfiable if we choose proper δ, ν 0 such that (1+ ρ ¯ τ ) -1 / 2 ≥ δ + ν 0 . Furthermore, Assumption (¯ τ ) -1 > ρ is mild. Since we always choose ¯ σ large enough in rAPDPro , ¯ τ can be sufficiently small.

Remark 9. Both algorithms proposed previously require solving quadratic optimization with linear constraints when updating dual variables, which may introduce implementation overheads when the constraint number is high. Inspired by the multi-stage algorithm, we additionally propose an algorithm (Multi-Stage APD, msAPD) that uses different step sizes in different stages and dynamically adjusts the number of iterations in each stage by leveraging strong convexity, as detailed in Appendix H.

5 Active-set identification in sparsity-inducing optimization

In this section, we apply our proposed algorithms to the aforementioned sparse learning problem:

where f ( x ) = ∑ B i =1 p i ∥ x ( i ) ∥ is the group Lasso regularizer and g ( x ) is a strongly convex function. Weuse x ( i ) to express the i -th block coordinates of x . The goal of this section is to show that rAPDPro can identify the sparsity pattern of the optimal solution of (17) in a finite number of iterations.

In general, suppose that f ( x ) has a separable structure f ( x ) = ∑ B i =1 f i ( x ( i ) ) , we define the active set A ( x ) for f ( x ) by A ( x ) := { i : ∂f i ( x ( i ) ) is not a singleton } . For f ( x ) = ∑ B i =1 p i ∥ x ( i ) ∥ , it is easy to see that A ( x ) is the index set of the zero blocks: A ( x ∗ ) = { i : x ∗ ( i ) = 0 } . Next, we describe one property for the optimal solution of (17) in Proposition 5 with the proof provided in Appendix G.1.

Proposition 5. Under Assumptions 1 and 2, the KKT point for (17) is unique.

To identify the sparsity pattern (active set) of the optimal solution, it is common to assume the existence of a non-degenerate optimal solution, which is stronger than the standard optimality condition [24, 29]. We say that x ∗ is non-degenerate if 0 ∈ ri ∂ L ( x ∗ , y ∗ ) = ri ( ∂f ( x ∗ )+ ∇ g ( x ∗ ) y ∗ ) for the Lagrangian multiplier y ∗ , where ri stands for the relative interior. More specifically, ( x ∗ , y ∗ ) satisfies the block-wise optimality condition

Inspired by [24], we use the radius η := min i ∈A ( x ∗ ) { p i -∥ [ ∇ g ( x ∗ ) y ∗ ] ( i ) ∥ } , which describes the certain distance between the gradient and "subdifferential boundary" of the active set. We demonstrate in the following theorem that the optimal sparsity pattern is identified when the iterates fall in a neighborhood dependent on η , with the proof provided in Appendix G.2.

Theorem 4. Set X := B ( ˜ x , min i ∈ [ m ] 2 √ -2 g i ( x ∗ i ) µ i + ζ ) with ζ > 0 and 3 L XY · (¯ τ +(2 L XY ) -1 ) · ζ > η ¯ τ in rAPDPro , then we have there exists a epoch ˆ S 0 such that x ∗ ( i ) = x s k ( i ) , s ≥ ˆ S 0 , ∀ k ∈ [ N s ] , ∀ i ∈ A ( x ∗ ) .

Remark 10. The active-set identification result is achieved using the optimality condition at the next iterate x k +1 i . To ensure x k +1 i ∈ int X , we define an expanded region, which prevents cases where the normal cone differs from { 0 } .

6 Numerical study

In this section, we examine the empirical performance of our proposed algorithms for solving the sparse Personalized PageRank [8, 9, 23]. The constrained form of Personalized PageRank can be

0

1

2

3

4

5

1e4

10

10

10

8

10

6

10

4

10

2

10 0

APD

Mirror-Prox

rAPDPro

msAPD

APD+restart

0

1

2

3

4

5

1e3

10

9

10

7

10

5

10

3

10

1

APD

Mirror-Prox

rAPDPro

msAPD

APD+restart

0.0

0.5

1.0

1.5

2.0

1e5

10

6

10

4

10

2

10 0

APD

Mirror-Prox

rAPDPro

msAPD

APD+restart

0

1

2

3

4

5

1e4

10

10

10

8

10

6

10

4

10

2

APD

Mirror-Prox

rAPDPro

msAPD

APD+restart

0

1

2

3

4

5

1e3

10

11

10

9

10

7

10

5

10

3

10

1

APD

Mirror-Prox

rAPDPro

msAPD

APD+restart

0.0

0.5

1.0

1.5

2.0

1e5

10

11

10

9

10

7

10

5

10

3

APD

Mirror-Prox

rAPDPro

msAPD

APD+restart

written as follows: min x ∈ R n ∥ D 1 / 2 x ∥ 1 s . t . 1 2 ⟨ x , Q x ⟩ -α ⟨ s , D -1 / 2 x ⟩ ≤ b, where Q,D and s are generated by graph. We implement both rAPDPro and msAPD. We skip APDPro as we observe that the restart strategy consistently improves the algorithm performance. For comparison, we consider the state-of-the-art accelerated primal-dual (APD) method [11], APD with restart mechanism at fixed iterations (APD+restart) and Mirror-Prox [13]. 6 small to medium-scale datasets from various domains in the Network Datasets [28] are selected in our experiments. All experiments are implemented on Mac mini M2 Pro, 32GB. Due to the page limit, we only report results on three datasets and leave more details in the last Appendix I.

We plot the relative function value gap | f ( x ) -f ( x ∗ ) | / | f ( x ∗ ) | and the feasibility violation max { G ( x ) , 0 } over the iteration number in Figure 1, respectively. Firstly, in terms of both optimality gap and constraint violation, the performance of rAPDPro and msAPD is significantly better than that of APD, APD+restart and Mirror-Prox. Additionally, rAPDPro and msAPD often converge to high-precision solutions. Secondly, based on the experimental results, it is indeed observed that msAPD exhibits a periodic variation in convergence performance, which aligns with our algorithm theory.

0

1

2

3

4

5

1e4

0.6

0.7

0.8

0.9

1.0

APD

Mirror-Prox

rAPDPro

msAPD

APD+restart

0

1

2

3

4

5

1e3

0.875

0.900

0.925

0.950

0.975

1.000

APD

Mirror-Prox

rAPDPro

msAPD

apd+restart

0.0

0.5

1.0

1.5

2.0

1e5

0.6

0.7

0.8

0.9

APD

Mirror-Prox

rAPDPro

msAPD

APD+restart

Next, we examine the algorithm's effectiveness in identifying sparsity patterns. We computed a nearly optimal solution x ∗ from MOSEK. Note that x ∗ is a dense vector. For numerical consideration, we truncate the coordinate values of x ∗ to zero if the absolute value is below 10 -8 and perform the same truncation to all the generated solutions of the compared algorithms. Then we use ( |A ( x ) ∩

A ( x ∗ ) | + |A c ( x ) ∩ A c ( x ∗ ) | ) /n to measure the accuracy of identifying the active set, where | · | denotes the set cardinality. For rAPDPro, we consider the last iterate x k while for APD, msAPD and Mirror-Prox, we plot the result on ¯ x k , as these are the solutions where the convergence rates are established. Figure 2 plots the experiment result, from which we observe that rAPDPro and msAPD are highly effective in identifying the active set. Often, they are able to recognize the structure of the active set within a small number of iterations. Overall, the experimental results show the great potential of our proposed algorithms in identifying the sparsity structure and are consistent with our theoretical analysis.

7 Conclusion

The key contribution of this paper is that we develop several new first-order primal-dual algorithms for convex optimization with strongly convex constraints. Using some novel strategies to exploit the strong convexity of the Lagrangian function, we substantially improve the best convergence rate from O (1 /ε ) to O (1 / √ ε ) . In the application of constrained sparse learning problems, the experimental study confirms the advantage of our proposed algorithms against state-of-the-art first-order methods for constrained optimization. Moreover, we show that one of our proposed algorithms rAPDPro has the favorable feature of identifying the sparsity pattern in the optimal solution. For future work, one direction is to apply the adaptive strategy, such as line search, to our framework to deal with cases when the dual bound is unavailable. Another interesting direction is to further exploit the active set identification property in a general setting. For example, it would be interesting to incorporate our algorithm with active constraint identification, which could be highly desirable when there are a large number of constraints. It would also be interesting to consider a more general convex objective when the proximal operator is not easy to compute.

References

Mosek ApS. Mosek optimization toolbox for matlab. User's Guide and Reference Manual, Version , 4(1), 2019.

Amir Beck. First-order methods in optimization . SIAM, 2017.

Dimitri P. Bertsekas. Nonlinear programming . Athena Scientific, 1999.

Digvijay Boob, Qi Deng, and Guanghui Lan. Stochastic first-order methods for convex and nonconvex functional constrained optimization. Mathematical Programming , pages 1-65, 2022.

Gábor Braun, Alejandro Carderera, Cyrille W Combettes, Hamed Hassani, Amin Karbasi, Aryan Mokhtari, and Sebastian Pokutta. Conditional gradient methods. arXiv preprint arXiv:2211.14103 , 2022.

Antonin Chambolle and Thomas Pock. On the ergodic convergence rates of a first-order primal-dual algorithm. Mathematical Programming , 159(1):253-287, 2016.

Joseph C Dunn. Rates of convergence for conditional gradient algorithms near singular and nonsingular extremals. SIAM Journal on Control and Optimization , 17(2):187-211, 1979.

Kimon Fountoulakis, Farbod Roosta-Khorasani, Julian Shun, Xiang Cheng, and Michael W Mahoney. Variational perspective on local graph clustering. Mathematical Programming , 174:553-573, 2019.

Kimon Fountoulakis and Shenghao Yang. Open problem: Running time complexity of accelerated ℓ 1 -regularized pagerank. In Conference on Learning Theory , pages 5630-5632. PMLR, 2022.

Dan Garber and Elad Hazan. Faster rates for the frank-wolfe method over strongly-convex sets. In International Conference on Machine Learning , pages 541-549. PMLR, 2015.

Erfan Yazdandoost Hamedani and Necdet Serhat Aybat. A primal-dual algorithm with line search for general convex-concave saddle point problems. SIAM Journal on Optimization , 31(2):1299-1329, 2021.

Warren L Hare and Adrian S Lewis. Identifying active constraints via partial smoothness and prox-regularity. Journal of Convex Analysis , 11(2):251-266, 2004.

Niao He, Anatoli Juditsky, and Arkadi Nemirovski. Mirror prox algorithm for multi-term composite minimization and semi-separable problems. Computational Optimization and Applications , 61(2):275-319, 2015.

Franck Iutzeler and Jérôme Malick. Nonsmoothness in machine learning: specific structure, proximal identification, and applications. Set-Valued and Variational Analysis , 28(4):661-678, 2020.

Michel Journée, Yurii Nesterov, Peter Richtárik, and Rodolphe Sepulchre. Generalized power method for sparse principal component analysis. Journal of Machine Learning Research , 11(2), 2010.

Guanghui Lan. First-order and stochastic optimization methods for machine learning . Springer, 2020.

Guanghui Lan and Renato DC Monteiro. Iteration-complexity of first-order penalty methods for convex programming. Mathematical Programming , 138(1):115-139, 2013.

Guanghui Lan and Renato DC Monteiro. Iteration-complexity of first-order augmented lagrangian methods for convex programming. Mathematical Programming , 155(1):511-547, 2016.

Sangkyun Lee, Stephen J Wright, and Léon Bottou. Manifold identification in dual averaging for regularized stochastic online learning. Journal of Machine Learning Research , 13(6), 2012.

Evgeny S Levitin and Boris T Polyak. Constrained minimization methods. USSR Computational mathematics and mathematical physics , 6(5):1-50, 1966.

Qihang Lin, Selvaprabu Nadarajah, and Negar Soheili. A level-set method for convex optimization with a feasible solution path. SIAM Journal on Optimization , 28(4):3290-3311, 2018.

Tianyi Lin, Chi Jin, and Michael I Jordan. Near-optimal algorithms for minimax optimization. In Conference on Learning Theory , pages 2738-2779. PMLR, 2020.

David Martínez-Rubio, Elias Wirth, and Sebastian Pokutta. Accelerated and sparse algorithms for approximate personalized pagerank and beyond. arXiv preprint arXiv:2303.12875 , 2023.

Julie Nutini, Mark Schmidt, and Warren Hare. 'active-set complexity' of proximal gradient: How long does it take to find the sparsity pattern? Optimization Letters , 13(4):645-655, 2019.

Yuyuan Ouyang and Yangyang Xu. Lower complexity bounds of first-order methods for convex-concave bilinear saddle-point problems. Mathematical Programming , 185(1):1-35, 2021.

H. Robbins and D. Siegmund. A convergence theorem for non negative almost supermartingales and some applications. In Jagdish S. Rustagi, editor, Optimizing Methods in Statistics , pages 233-257. Academic Press, 1971.

R Tyrrell Rockafellar. Convex analysis , volume 18. Princeton university press, 1970.

Ryan A. Rossi and Nesreen K. Ahmed. The network data repository with interactive graph analytics and visualization. In AAAI , 2015.

Yifan Sun, Halyun Jeong, Julie Nutini, and Mark Schmidt. Are we there yet? manifold identification of gradient-related proximal methods. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1110-1119. PMLR, 2019.

Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological) , 58(1):267-288, 1996.

Stephen J Wright. Identifiable surfaces in constrained optimization. SIAM Journal on Control and Optimization , 31(4):1063-1079, 1993.

Yangyang Xu. First-order methods for constrained convex programming based on linearized augmented lagrangian function. Informs Journal on Optimization , 3(1):89-117, 2021.

Yangyang Xu. Iteration complexity of inexact augmented lagrangian methods for constrained convex programming. Mathematical Programming , 185(1):199-244, 2021.

Liwei Zhang, Yule Zhang, Jia Wu, and Xiantao Xiao. Solving stochastic optimization with expectation constraints efficiently by a stochastic augmented lagrangian-type algorithm. INFORMS Journal on Computing , 34(6):2989-3006, 2022.

Appendix

Structure of the Appendix

The appendix is structured as follows: Appendix A introduces some limitations of our methods, primarily concerning the application scenarios of our algorithm. Appendix B includes comparisons between ours and some related Frank-Wolfe methods. We give some auxiliary lemmas in Appendix C, which are very important for the proofs presented later. Appendix D, E, F and G present the proof of conclusion in Section 2, 3, 4 and 5, respectively. Furthermore, Appendix H introduces a new algorithm to obtain a convergence rate without complicated dual updating. Finally, Appendix I offers more extensive details on our experiments.

A Limitations

In this paper, we focus on the theoretical analysis of convex optimization. Although our proposed algorithms for the convex optimization with strongly convex constraints can theoretically improve the existing results from O (1 /ε ) to O (1 / √ ε ) . However, we still need to point out that our optimization algorithm has the following limitations. One is the algorithm needs a lower bound on the norm of sub-gradients of the objective function in the optimal solution, which may not be satisfied for all functions. On the other hand, we require consistent smoothness of the constraints to ensure convergence, and how to use the line search method to ensure convergence is a future direction.

B Comparison with Frank-Wolfe

We note that the strongly convex function constraint in (1) is a special case of a strongly convex set constraint, as demonstrated in [15]. Over the strongly convex set, it has been shown that Frank-Wolfe Algorithm (FW) can obtain convergence rates substantially better than the worst-case O (1 /ε ) rate. Under the bounded gradient assumption, [7, 20] show that FW obtains linear convergence over a strongly convex set. Nevertheless, the uniform bounded gradient assumption appears to be stronger than ours, as we only impose the lower boundedness assumption on the optimal solution x ∗ and allow the objective to be non-differentiable. More recently, [10] shows that FW obtains an O (1 / √ ε ) rate when the gradient is the order of the square root of the function value gap. For more recent progress, please refer to [5]. Despite the attractive convergence property, FW exhibits certain limitations when applied to the general function constraints (1) addressed in this paper. Specifically, FW involves a sequence of linear optimization problems throughout the iterations. While linear optimization over certain strongly convex sets, such as ℓ p -ball, admits a closed-form solution, there exists no efficient routine to handle general function constraints explored in this paper.

C Auxiliary lemmas

The following three-point property is important in the convergence analysis.

Lemma 1. Let f : R n → R ∪{ + ∞} be a closed strongly convex function with modulus µ ≥ 0 . Give ¯ x ∈ X , where X is a compact convex set and t ≥ 0 , let x + = argmin x ∈X f ( x ) + t 2 ∥ x -¯ x ∥ 2 , then for all x ∈ X , we have

Proof. Since X is a convex compact set, ϕ ( x ) := I X ( x )+ f ( x )+ t 2 ∥ x -¯ x ∥ 2 is lower-semi-continuous and ( µ + t ) -strongly convex, where I X ( x ) = { 0 x ∈ X ∞ x / ∈ X . Using the optimality ( 0 ∈ ϕ ( x + ) ) and strong convexity, we have ϕ ( x ) ≥ ϕ ( x + ) + ⟨ 0 , x -x + ⟩ + t + µ 2 ∥ x + -x ∥ 2 , for any x ∈ X . This immediately gives the desired relation.

The following result is adjusted from the classic supermartingale convergence theorem [26, Theorem 1]. We give proof for completeness.

Lemma2. Let (Ω , F , P ) be a probability space and F 1 ⊂ F 2 ⊂ · · · be a sequence of subσ -algebras of F . For each j = 1 , 2 , · · · , let a j , b j and c j be non-negative F n -measure random variables such E [ a j +1 | F j ] ≤ a j -b j + c j , then we have lim j →∞ a j < ∞ exists and ∑ ∞ j =1 b j < ∞ a.s. when ∑ ∞ j =1 c j < ∞ .

Proof. Define d j = a j -∑ j -1 l =1 ( c l -b l ) and for any ¯ a > 0 , define t = inf { t : ∑ t l =1 c l > ¯ a } . If j < t , we have

where ( a ) holds by E [ a j +1 | F j ] ≤ a j + c j -b j

where ( a ) holds by (18). Therefore, we have { d min { t, ( j +1) } , F j , 1 ≤ j ≤ ∞} is a supermartingale. Since

holds for all j , where ( a ) holds by a min { t,j } , b l ≥ 0 . Then it follows from the martingale convergence theorem that lim j →∞ d min { t,j } exists and is finite a.s., i.e., lim j →∞ d j exists and is finite on { t = ∞} = { ∑ ∞ j =1 c j ≤ ¯ a } . Since ¯ a is arbitrary, we see that lim j →∞ d j exists and is finite a.s. on { ∑ ∞ j =1 c j < ∞} . By d j = a j -∑ j -1 l =1 ( c l -b l ) , we have lim j →∞ a j exists and is finite and ∑ ∞ j =1 b j < ∞ when { ∑ ∞ j =1 c j < ∞} .

D Proof details in Section 2

D.1 Proof of Proposition 1

Proof. Under Slater's CQ, it is standard to show that any optimal solution x ∗ will also satisfy the KKT condition. For example, one can refer to [3]. For any x ∈ X G , we have

where the equality is from the complementary slackness. In view of the above result and the Slater's condition (i.e., G (˜ x ) < 0 ), we have

Combining with fact ∥ y ∗ ∥ 1 min i ∈ [ m ] { -g i (˜ x ) } ≤ -⟨ y ∗ , G (˜ x ) ⟩ , then we have

where the last inequality is by f ( x ∗ ) ≥ min x ∈ R n f ( x ) .

D.2 Proof of Proposition 2

Proof. Weprove the uniqueness property by contradiction. Suppose that there exist ( x ∗ , y ∗ ) , (˜ x ∗ , ˜ y ∗ ) satisfying the KKT condition, then from the complementary slackness, optimality of x ∗ and ˜ x ∗ , we have

̸

Moreover, we have L (˜ x ∗ , ˜ y ∗ ) ≤ L ( x ∗ , ˜ y ∗ ) ≤ L ( x ∗ , y ∗ ) . Hence, we must have L (˜ x ∗ , ˜ y ∗ ) = L ( x ∗ , ˜ y ∗ ) . However, since Assumption 2 implies ˜ y ∗ = 0 , the strongly convex function L ( · , ˜ y ∗ ) has a unique optimizer. Therefore, we conclude that x ∗ = ˜ x ∗ .

Next, we show that the set of optimal dual variables for problem (1) is convex. Suppose that there exist two optimal dual variables y ∗ 1 and y ∗ 2 for the unique primal variable x ∗ , both satisfying the KKT condition, then we have ⟨ y ∗ 1 , G ( x ∗ ) ⟩ = ⟨ y ∗ 2 , G ( x ∗ ) ⟩ = 0 . This implies that any linear combination of y ∗ 1 and y ∗ 2 satisfy KKT condition, i.e., ⟨ a y ∗ 1 + b y ∗ 2 , G ( x ∗ ) ⟩ = 0 , ∀ a, b . From Proposition 1, we know any optimal dual variable falls into a bounded convex set Y . The intersection of two convex sets is also a convex set. Hence, we complete our proof.

D.3 Proof of Proposition 3

Proof. From the strong convexity of g i ( x ) , we have g i ( x ) ≥ g i ( x ∗ i ) + µ i 2 ∥ x -x ∗ i ∥ 2 , which implies

where ( a ) holds by g i (˜ x ) < 0 . In view of the triangle inequality and the above result, we have

Hence, x ∗ ∈ int B ( ˜ x , min i ∈ [ m ] 2 √ -2 g i ( x ∗ i ) µ i )

E Convergence analysis of APDPro

E.1 Proof of Proposition 4

Proof. Using the triangle inequality and (5), we have

Combining the above inequality and (8), we obtain

Next, we develop more specific lower bounds on ∥ y ∥ 1 . i). Inequality (9) can be easily verified since we have ∥ ˆ x -x ∗ ∥ ≤ √ 2 β . ii). Suppose ( y ∗ ) ⊤ µ · ∥ ˆ x -x ∗ ∥ 2 ≤ 2 β , then together with (22) we have

Note that the above inequality can be expressed as at 2 -bt -c ≤ 0 with t = ∥ y ∗ ∥ -1 / 2 1 , a = r, b = L X √ 2 β/µ and c = ∥∇ G (ˆ x ) ∥ . Standard analysis implies that t ≤ ( b + √ b 2 +4 ac ) / 2 a , which gives the desired bound (10).

E.2 Proof of Theorem 1

Proof. First, it is easy to verify by our construction that {Y k } is a monotone sequence: Y 1 ⊇ Y 2 ⊇ . . . ⊇ Y k . . . . Our goal is to show Y ∗ ⊆ Y k holds for any k ≥ 0 by induction. Note that Y ∗ ⊆ Y 0 immediately follows from our assumption that ( y ∗ ) ⊤ µ ≥ ρ 0 , for any y ∗ ∈ Y ∗ . Suppose that Y ∗ ⊆ Y k holds for k = 0 , . . . , K -1 , we claim:

For any x ∈ X and y ∈ Y ∗ , we have

Part 1. For k = 0 , 1 , 2 , . . . , K -1 , taking -⟨ z k , ·⟩ and f ( · ) + ⟨∇ G ( x k ) y k +1 , ·⟩ in Lemma 1, the following relations

hold for any x ∈ X and y ∈ ⋂ 0 ≤ s ≤ k Y s . The existence of such y follows from our induction hypothesis. Since y ⊤ k +1 G ( · ) is ρ k -strongly convex, we have

Combining this result and (25), we have

On the other hand, by the definition of z k , we have

where

Putting (28) and (30) together, we have

where the last inequality is by Lipschitz smoothness of ⟨ y k +1 , G ( · ) ⟩ .

Next, we bound the term ⟨ q k , y k +1 -y k ⟩ by Young's inequality, which gives

It follows from (31) and σ k 2 ∥ q k +1 ∥ 2 ≤ L 2 G σ k 2 ∥ x k +1 -x k ∥ 2 that

Multiply both sides of the above relation by t k and sum up the result for k = 0 , 1 , . . . , K -1 . In view of the parameter relation (11), we have

where ( a ) uses q 0 = 0 and x -1 = x 0 , and ( b ) holds by ρ 0 = 0 , t 0 = 1 and

Since L ( x , y ) is convex in x and linear in y , we have

Combining (33) and (34), we obtain

Dividing both sides by T K , we obtain the desired result (23).

Part 2. Next we show Y ∗ ⊆ Y K . Let y ∗ be any point in Y ∗ . Since (35) holds for any x ∈ X and y ∈ ∩ 0 ≤ k ≤ K -1 Y k ⊇ Y ∗ , we can place x = x ∗ , y = y ∗ ∈ Y ∗ in (23) to obtain

Moreover, the strong convexity of L ( · , y ∗ ) implies

Applying the above two inequalities yields

In view of (36) and Proposition 4, we have that

Moreover, since Y ∗ ⊆ Y K -1 , we have ( y ∗ ) T µ ≥ ρ K -1 . Hence we have ( y ∗ ) T µ ≥ ρ K where ρ K = max { ˆ ρ K , ρ K -1 } is the output of the IMPROVE procedure. Due to the construction of Y K , we immediately see that y ∗ ∈ Y K . This implies Y ∗ ⊆ Y K and completes our induction proof.

Next, we specify the stepsize selection in Lemma 3 and develop more concrete complexity results in Corollary 1.

Then we have

where ˜ ρ k = 2 ∑ k s =0 ˆ ρ s s k ( k +1) for k ≥ 1 . Moreover, suppose ¯ ρτ 0 ≤ 2 , where ¯ ρ = ¯ c · ¯ µ , then we have

Proof. We first use induction to show that 1 τ 2 k ≥ ˆ ρ 2 k 9 k 2 + 1 τ 2 0 . It is easy to see that 1 τ 2 k ≥ ˆ ρ 2 k 9 k 2 + 1 τ 2 0 holds for k = 1 by the definition ˆ ρ 1 = 3 √ ρ 1 /τ 0 and τ 1 = τ 0 (1+ ρ 1 τ 0 ) -1 2 . Assume 1 τ 2 k ≥ ˆ ρ 2 k 9 k 2 + 1 τ 2 0 holds for all k = 0 , . . . , K , then we have

which completes our induction. It follows from 1 τ 2 k ≥ ˆ ρ 2 k 9 k 2 + 1 τ 2 0 and the relation among T k , t k , σ k , τ k that, for any k ≥ 1

Similarly, we use induction to prove

It is easy to find that ˆ ρ 1 ≥ min { ρ 1 , ˆ ρ 1 } . We assume that ˆ ρ k ≥ min { ρ 1 , ˆ ρ 1 } , ∀ k ≥ 1 holds for any k = 1 , . . . , K . Considering ˆ ρ K +1 , we have

which completes the induction. Moreover, we use induction to show σ 2 k ≤ σ 2 0 ( k +1) 2 . It is obvious that the inequality holds for k = 0 . Assume the inequality holds for all k = 0 , . . . , K, then we have

where the last inequality use the relation ρ k ≤ ¯ ρ, ∀ k , and ¯ ρτ 0 ≤ 2 .

E.3 Proof of Corollary 1

Proof. First, we show that the sequences { τ k , σ k , t k , ρ k } generated by APDPro satisfy the relationship in (11) in Theorem 1. The first part of (11) can be derived using the monotonicity of { ρ k } as follows:

The second part of (11) can be easily verified using the parameters setting.

Next, we prove the last term in (11) by induction. Firstly, it easy to verify that for any σ 0 > 0 , there exists τ 0 ∈ (0 , ( L XY + L 2 G σ 0 ) -1 ] such that last term of (11) holds. Hence, when k = 0 , the last term of (11) is directly from the first term of (13). Suppose that the last term of (11) holds for k = 0 , . . . , K -1 . From σ K -1 /σ K = τ K /τ K -1 ≤ 1 , we have

Without loss of generality, place x = x ∗ , y = y + := ( ∥ y ∗ ∥ 1 + c ∗ ) [ G (¯ x K )] + ∥ [ G (¯ x K )] + ∥ in (23), and using ∥ y ∗ ∥ 1 ≤ ¯ c in Proposition 1. It is easy to see ∥ y + ∥ = ∥ y ∗ ∥ 1 + c ∗ ≤ ¯ c , and ∥ y + ∥ 1 ≥ ∥ y + ∥ = ∥ y ∗ ∥ 1 + c ∗ ≥ ∥ y ∗ ∥ 1 , Hence, we conclude that y + ∈ Y k , ∀ k ≥ 0 .

Now observe that L (¯ x K , y ∗ ) -L ( x ∗ , y ∗ ) ≥ 0 , which implies f (¯ x K ) + ⟨ y ∗ , G (¯ x K ) ⟩ -f ( x ∗ ) ≥ 0 . In view of ⟨ y ∗ , G (¯ x K ) ⟩ ≤ ⟨ y ∗ , [ G (¯ x K )] + ⟩ ≤ ∥ y ∗ ∥ · ∥ [ G (¯ x K )] + ∥ , then we have

Moreover, it follows from ∥ y ∗ ∥ 1 ≥ ∥ y ∗ ∥ that

Combining (45), (46) and (23), we obtain

In view of the bound in (38) and the relation between τ k , σ k , we can get

In view of (47) and (38), we have

F Convergence analysis of rAPDPro

F.1 Proof of Theorem 2

Proof. First, we show that the choice of τ s 0 = ¯ τ, σ s 0 = ¯ σ, ∀ s ≥ 0 satisfy the condition (13) in Corollary 1: ( τ s 0 ) -1 ≥ (1 -ν 0 )( τ s 0 ) -1 = L XY + cL 2 G σ s 0 /δ ≥ L XY + cL 2 G σ s 0 .

Next, we show (15) holds by induction. Clearly, (15) holds for s = 0 . Assume ∥ x s 0 -x ∗ ∥ 2 ≤ ∆ s holds for s = 0 , . . . , S -1 . Then by Theorem 1, we have

In view of the first bound in (38) and the relation between τ s N s , σ s N s , we can get

Combining (49) and (50) yields

Since the algorithm sets N s = ⌈ max { 6(ˆ ρ N s τ s 0 ) -1 , √ 2 s · 3 √ 2 D Y / ( ˆ ρ N s D X √ τ s 0 σ s 0 ) }⌉ , it follows that

which implies the desired result (15).

Let the algorithm run for S = ⌈ log 2 ( D 2 X /ε ) ⌉ epochs, then ∥ x S 0 -x ∗ ∥ 2 ≤ D 2 X · 2 -S ≤ ε . The total iteration number required by Algorithm 2 for attaining a solution x S 0 such that ∥ x S 0 -x ∗ ∥ 2 ≤ ε is

where ( a ) holds by ∑ S s =0 (ˆ ρ s N s ) -1 = ( ϖ 1 ) -1 ( S +1) and ∑ S s =0 √ 2 s / ˆ ρ s N s = ( ϖ 2 ) -1 ∑ S s =0 √ 2 s .

Now, we give some proof details in dual convergence results. Let

then we establish an important property about the solution sequence in the following lemma.

Lemma 4. Assume ¯ τ -1 > ρ and choose ν 0 > 0 such that

Then there exists an ν 1 > 0 such that for any j ≥ 0 and any KKT point ( x ∗ , ˜ y ∗ ) :

Proof. First, we give some results that will be used repeatedly in the following. For notation simplicity, we denote θ j = σ j -1 /σ j . In view of Lemma 3, and the parameter ergodic sequence generated by rAPDPro, we have { ( τ s k ) -1 , σ s k } is monotonically increasing sequence in k , ¯ τ = τ s 0 , ¯ σ = σ s 0 , t s 0 = 1 , ∀ s ≥ 0 , and there exist a ν 3 > 0 such that ¯ σ + ν 3 ≤ σ := min s { σ s N s } . Now, for rAPDPro, we claim that there exist ν 1 , ν 2 > 0 such that the following two conditions hold

1. For any j ≥ 0 , we have

and

2. For any j ≥ 0 , we have

Part 1. We first consider two subsequent points x j and x j +1 within the same epoch, and assume j ∼ ( s, k ) . Then, it follows from θ s k = σ s k -1 /σ s k that

Next, we use induction to show

When k = 0 , inequality (56) degenerates as the definition of τ s 0 , σ s 0 . Suppose (56) holds for k = 0 , 1 , . . . , K -1 . Then, from θ s K = σ s K -1 /σ s K = τ s K /τ s K -1 ≤ 1 , we have

which completes our induction proof. Hence, combining (55) and (56), we have

Furthermore, when switching to the next epoch ( s → s +1) , we have

where ( a ) holds by θ s 0 = 1 , δ < 1 , ( b ) follows from ( σ s N s ) -1 ≥ σ -1 . Hence, combining (55), (57) and (58), we completes our proof of (52) by setting ν 1 = min { 1 -¯ σσ -1 , ν 0 ¯ τ, ν 0 } .

Since rAPDPro reset the stepsize periodically and { t s k , ( τ s k ) -1 } k ∈ [ N s ] are two monotonically increasing sequences, hence

Consider inf k ∈ [ N s ] t s k σ s k (1 -δσ s k /σ s k -1 ) . Combining δ + ν 0 ≤ inf k ∈ [ N s ] { θ s k } , then

Furthermore, when switching to the next epoch ( s → s +1) , we have

where the last inequality holds by ¯ σ = σ s 0 ≤ σ s N s . Hence, it follows from (59), (60) and (61) that there exist ν 2 = min { ¯ τ -1 -ρ, ν 0 ¯ σ, ¯ σ (1 -δ ) } such (53) holds.

Part 2. for any j ≥ 0 , we have

Consider k ∈ { 0 , 1 , . . . , N s } . Inequality (51) implies (11) holds (see proof of Corollary 1 in Section E.3). Hence, for 0 ≤ k ≤ N s , we have

where j corresponds to ( s, k ) . Furthermore, consider switching to next epoch ( s → s +1) . Since t s k ( τ s k ) -1 is an increasing sequence in k , ρ s +1 0 > 0 , t s +1 0 = 1 , hence

Next, we have

where ( a ) holds by the definition of t s k = σ s k σ s 0 , ( b ) holds by { t s k , σ s k } is an increasing sequence in k , and ( c ) holds by θ s +1 0 = 1 . Hence, by (64) and (65), we have

where j corresponds to ( s, N s ) . By putting (63) and (66) together, we complete the proof of (62).

Placing ( x , y ) = ( x ∗ , ˜ y ∗ ) , ( x k +1 , y k +1 ) = ( x j +1 , y j +1 ) in (32) and multiplying t j on both sides, we have

where the last inequality holds by (62) and (52). It follows from (53), σ j -1 /σ j ≤ 1 and

that

Combining (67) and (68), we complete our proof of (54).

F.2 Proof of Theorem 3

Proof. Since { ( x j , y j ) } located in set X × Y is a bounded sequence, it must have a convergent subsequence lim n →∞ ( x j n , y j n ) = ( x ∗ , y ∗ ) , where y ∗ is the limit point. We claim that limit point ( x ∗ , y ∗ ) satisfies the KKT condition. Placing a j = t j Q j ( x ∗ , ˜ y ∗ ) , b j = ν 1 t j [(2 τ j ) -1 ∥ x j +1 -x j ∥ 2 + (2 σ j ) -1 ∥ y j +1 -y j ∥ 2 ] and c j = 0 in Lemma 2. It follows from (54) in Lemma 4 that a j ≥ 0 , b j > 0 . Hence, we have ∑ ∞ j =0 ∥ x j +1 -x j ∥ 2 < ∞ , and ∑ ∞ j =0 ∥ y j +1 -y j ∥ 2 < ∞ , which

implies lim n →∞ ∥ x j n -x j n +1 ∥ 2 = 0 and lim n →∞ ∥ y j n -y j n +1 ∥ 2 = 0 . There are two different cases for τ j n when j n →∞ , and we discuss the value of B j n +1 in (25) decided by τ j n in each of the two cases below.

Case 2: τ -1 j n = ∞ . It follows from (39) that τ -1 j n increases at order Θ( k ) , where j n ∼ ( s, k ) . By (23), we obtain ∥ x -x j n ∥ decreases at order O (1 /k ) ( j n ∼ ( s, k ) ). Hence, combining lim n →∞ ∥ x j n -x j n +1 ∥ 2 = 0 , we have B j n +1 ≤ 1 τ j n ( ∥ x -x j n +1 ∥∥ x j n +1 -x j n ∥ ) n →∞ -→ 0 . It follows from lim n →∞ x j n = x ∗ , lim n →∞ B j n +1 = 0 and (25) that

Hence, according to the first-order optimality condition, we have

Next, we show the complementary slackness holds for ( x ∗ , y ∗ ) . Since σ -1 j n has an upper bound ¯ σ -1 , ∥ y -y j n +1 ∥ ≤ D Y , lim n →∞ ∥ y j n -y j n +1 ∥ 2 = 0 and the definition of A j n +1 in (26), hence we obtain A j n +1 ≤ 1 σ j n ( ∥ y j n -y j n +1 ∥∥ y -y j n +1 ∥ ) n →∞ -→ 0 . Combining above, lim n →∞ y j n = y ∗ and (24), we have 0 ≤ -⟨ G ( x ∗ ) , y ∗ ⟩ ≤ -⟨ G ( x ∗ ) , y ⟩ , ∀ y ∈ Y . Moreover, due to the complementary slackness, there exists an ˆ y ∗ ∈ Y ∗ ⊆ Y such that -⟨ G ( x ∗ ) , ˆ y ∗ ⟩ = 0 . Hence, we must have ⟨ G ( x ∗ ) , y ∗ ⟩ = 0 , which, together with (69), implies that ( x ∗ , y ∗ ) is KKT point.

G Proof details for sparsity identification

Our proof strategy of active-set identification in rAPDPro is similar to those in unconstrained optimization [24]. Namely, we show that the optimal sparsity pattern is identified when the iterates fall in a properly defined neighborhood dependent on η . The next lemma shows that the primal and dual sequences indeed converge to the neighborhood of the optimal primal and dual solutions, respectively, in a finite number of iterations.

Lemma 5. There exists an ˆ S 1 such that

where ( x ∗ , y ∗ ) is the unique solution of problem (17) . Moreover, there exists an epoch ˆ S 0 ≥ ˆ S 1 such that ∀ s ≥ ˆ S 0 , we have

Proof. From Theorem 2 and 3, we have lim j →∞ ( x j , y j ) = ( x ∗ , y ∗ ) , where j corresponds to ( s, 0) . It implies that there exists an epoch ˆ S 1 such that (70) holds.

It follows from (35) that ∥ x s 1 -x ∗ ∥ ≤ √ σ s 0 τ s 0 /σ s 1 ( ∥ x s 0 -x ∗ ∥ 2 /τ s 0 + ∥ y s 0 -y ∗ ∥ 2 /σ s 0 ) . Hence, in order to prove ∥ x s 1 -x ∗ ∥ ≤ η 3 L XY · τ s k τ s k +(2 L XY ) -1 , we need to prove

From Corollary 1 and Theorem 2, 3, we know that the left hand side of (72) converges to 0 and right hand side of (72) is a positive constant. Hence, there exist a ˆ S 2 such that (72) holds, which implies (71) holds for k = 1 , s = ˆ S 2 . Now we use induction to prove, for ∀ k ∈ [ N ˆ S 2 ] , we have

When k = 1 , inequality (73) coincides with (72) with s = ˆ S 2 . Now, assume (73) holds for k , we aim to prove (73) holds for k +1 . It follows from (35) that

.

where ( a ) follows from induction, ( b ) holds by τ ˆ S 2 k σ ˆ S 2 k = τ ˆ S 2 k +1 σ ˆ S 2 k +1 and ( c ) holds by τ ˆ S 2 k +1 ≤ τ ˆ S 2 k Hence, we complete our proof of (73). From Theorem 2, we have ∥ x s 0 -x ∗ ∥ 2 ≤ D 2 X · 2 -s , which implies that there exists a ˆ S 3 = ⌈ 2 log 2 { D X ( η 3 L XY · ¯ τ ¯ τ +(2 L XY ) -1 ) -1 }⌉ such that ∥ x ˆ S 3 0 -x ∗ ∥ ≤ D X · √ 2 -ˆ S 3 ≤ η 3 L XY ¯ τ ¯ τ +(2 L XY ) -1 , which implies that ∥ x s 0 -x ∗ ∥ ≤ D 2 X · 2 -s ≤ η 3 L XY ¯ τ ¯ τ +(2 L XY ) -1 holds for any s ≥ ˆ S 3 .

It follows from the definition of ˆ S 1 in (70) and stepsize will be reset at different epoch, then we have (72) holds for s ≥ max { ˆ S 1 , ˆ S 2 } , which implies that (73) holds with substituting ˆ S 2 as any s ≥ max { ˆ S 1 , ˆ S 2 } . Furthermore, it follows from Theorem 3 that lim j →∞ y j = y ∗ , where j corresponds to ( s, k ) . Then there exists a ˆ S 4 such that the first term in (71) holds. Hence, we can obtain that there exist a ˆ S 0 = max { ˆ S 1 , ˆ S 2 , ˆ S 3 , ˆ S 4 } such that (71) holds.

It is worth noting that the primal neighborhood defined by the second term of (71) is a bit different from the fixed neighborhood in the standard analysis [24], which involves a constant stepsize. As APDPro sets τ s k = O (1 /k ) , both the point distance and neighborhood radius decay at the same O (1 /k ) rate. Hence, we use a substantially different analysis to show the sparsity identification in the constrained setting.

G.1 Proof of Proposition 5

Proof. The uniqueness of primal optimal solution x ∗ follows from Proposition 2. The KKT condition (ensured by Slater's CQ) implies

̸

̸

According to Assumption 2, we have x ∗ = 0 , hence A c ( x ∗ ) = { 1 , 2 , . . . , B } \ A ( x ∗ ) = ∅ . In view of (74), for any i ∈ A c ( x ) , we have p i x ∗ ( i ) / ∥ x ∗ ( i ) ∥ = -∇ ( i ) g ( x ∗ ) y ∗ , which gives a unique y ∗ .

G.2 Proof of Theorem 4

Proof. It follows from the Lipschitz smoothness of g ( · ) and property (71) that for any s ≥ ˆ S 0 , we have

Recall that the primal update has the following form

Since τ s k / ( τ s k +(2 L XY ) -1 ) is monotonically increasing with respect to τ s k , for the strictly feasible point ˜ x , we have

where ( a ) holds by (71), ¯ τ ≥ τ s k and ( b ) follows from the definition of x ∗ , ˜ x and ζ . Inequality (76) implies that x s k +1 ∈ int X , and hence N X ( x s k +1 ) = { 0 } . In view of the optimality condition, we have

Our next goal is to show [ x S k +1 ] ( i ) = x ∗ ( i ) satisfies condition (77) for i ∈ A ( x ∗ ) . Placing x ( i ) = x ∗ ( i ) in ∥ ∥ [ ∇ g ( x S k ) y S k +1 + 1 τ s k ( x -x s k ) ] ( i ) ∥ ∥ , we have

In above, ( a ) follows from (71) and (75), ( b ) follows from

and ( c ) holds by the definition of η . Combining (77) and (78), we have A ( x ∗ ) ⊆ A ( x s k +1 ) , s ≥ ˆ S 0 , ∀ k ∈ [ N s ] , which completes our proof.

H A multi-stage accelerated primal-dual algorithm

Both the previous algorithms need to solve a complicated dual problem that involves a linear cut constraint, posing a potential issue: the associated sub-problem might lack a closed-form solution. To resolve this issue, we present the Multi-Stage Accelerated Primal-Dual Algorithm (msAPD) in Algorithm 3, which obtains the same O (1 / √ ε ) complexity without introducing a new cut constraint. Our new method is a double-loop procedure for which an accelerated primal-dual algorithm with a pending sub-iteration number (APDPi) is running in each stage. While both APDPi and APDPro employ the IMPROVE step to estimate the dual lower bound, APDPi only relies on the lower bound estimation to change the inner-loop iteration number adaptively, but not the stepsize selection.

We develop the convergence property of APDPi, which paves the path to proving our main theorem. For the convergence analysis, it suffices to verify that the initial stepsize parameter τ s 0 , σ s 0 satisfy assumptions in Theorem 5.

Theorem 5. Let { ¯ x s k , ¯ y s k } be the sequence generated by APDPi , then we have

where ∆ s ( x ∗ , y ∗ ) ≜ 1 2 τ s 0 ∥ x s 0 -x ∗ ∥ 2 + 1 2 σ s 0 ∥ y s 0 -y ∗ ∥ 2 and ( x ∗ , y ∗ ) is a KKT point.

0.0

0.2

0.4

0.6

0.8

1.0

1e5

10

7

10

5

10

3

10

1

APD

Mirror-Prox

rAPDPro

msAPD

APD+restart

0.0

0.2

0.4

0.6

0.8

1.0

1e5

10

10

10

8

10

6

10

4

10

2

10 0

APD

Mirror-Prox

rAPDPro

msAPD

APD+restart

0.0

0.5

1.0

1.5

2.0

1e4

10

6

10

4

10

2

10 0

APD

Mirror-Prox

rAPDPro

msAPD

APD+restart

0.0

0.2

0.4

0.6

0.8

1.0

1e5

10

11

10

9

10

7

10

5

10

3

APD

Mirror-Prox

rAPDPro

msAPD

APD+restart

0.0

0.2

0.4

0.6

0.8

1.0

1e5

10

11

10

9

10

7

10

5

10

3

10

1

APD

Mirror-Prox

rAPDPro

msAPD

APD+restart

0.0

0.5

1.0

1.5

2.0

1e4

10

10

10

9

APD

Mirror-Prox

rAPDPro

msAPD

APD+restart

Algorithm 3 Multi-Stage APD (msAPD)

Proof. The stepsize τ s k = τ s 0 , σ s k = σ s 0 are unchanged at one epoch, which implies that ρ k +1 = 0 , i.e., (37) are satisfied. By the definition of τ s 0 and σ s 0 , we have ( τ s 0 ) -1 = L XY + L 2 G ˜ σ √ 2 s = L XY + L 2 G σ s 0 , which means equality holds at the first term in (37).

Since g i ( x ) is a strongly convex function with modulus µ i , then we have

Summing up the two inequalities above, we can get

We show msAPD obtains an O (1 / √ ε ) convergence rate, which matches the complexity of APDPro.

Theorem 6. Let { ¯ x s 0 } be the sequence computed by msAPD . Then, we have

For any ε ∈ (0 , D 2 X ) , msAPD will find a solution ¯ x s 0 ∈ X such that ∥ ¯ x s 0 -x ∗ ∥ 2 ≤ ε in at most ⌈ log 2 D 2 X /ε ⌉ epochs. Moreover, the overall iteration number performed by msAPD to find such a solution is bounded by

Proof. We first show that (81) holds by induction. It is easy to verify that (81) holds for s = 0 . Assume ∥ ¯ x s 0 -x ∗ ∥ 2 ≤ ∆ s = D 2 X · 2 -s holds for s = 0 , . . . , S -1 . By Theorem 5, we have

As the algorithm sets N S -1 = ⌈ max { 4 / ( ρ S -1 N S -1 τ S -1 0 ) , 2 D 2 Y / ( ρ S -1 N S -1 σ S -1 0 ∆ S ) }⌉ , the following inequalities hold:

Putting these pieces together, we have ∥ ¯ x S -x ∗ ∥ 2 ≤ 1 2 ∆ S + 1 2 ∆ S = ∆ S . Suppose the algorithm runs for S epochs to achieve the desired accuracy ε , i.e., ∥ x S 0 -x ∗ ∥ 2 ≤ D 2 X · 2 -S ≤ ε . Then the overall iteration number can be bounded by

where ( a ) holds by ρ N S ≥ ρ N 0 , ∀ s ≥ 0 , ( b ) follows from the definition of τ 0 and σ 0 .

Remark 11. Theorem 6 shows that msAPD obtains a worst-case complexity of O ( log( D X / √ ε ) + ( D X + D 2 Y /D X ) / √ ε ) , which is an upper bound of the complexity of rAPDPro (see Theorem 2). The complexities of msAPD and rAPDPro match when D X = Ω(1) D Y . Otherwise, rAPDPro appears to be much better in terms of dependence on D X / √ ε . On the other hand, msAPD has a simpler subproblem, which does not involve an additional cut constraint on the dual update.

I Experiment details

We examine the empirical performance for solving sparse Personalized PageRank. Let G = ( V, E ) be a connected undirected graph with n vertices. Denote the adjacency matrix of G by A , that is, A i,j = 1 if i ∼ j and 0 otherwise. Let D = diag( d 1 , . . . , d n ) be the matrix with the degrees { d i } n i =1 in its diagonal. Then the constrained form of Personalized PageRank can be written as follows:

where Q = D -1 / 2 ( D -1 -α 2 ( D + A ) ) D -1 / 2 , α ∈ (0 , 1) , s ∈ ∆ n is a teleportation distribution over the nodes of the graph G and b is a pre-specific target level.

0.0

0.2

0.4

0.6

0.8

1.0

1e5

0.6

0.7

0.8

0.9

1.0

APD

Mirror-Prox

rAPDPro

msAPD

APD+restart

0.0

0.2

0.4

0.6

0.8

1.0

1e5

0.6

0.7

0.8

0.9

1.0

APD

Mirror-Prox

rAPDPro

msAPD

APD+restart

0.0

0.5

1.0

1.5

2.0

1e4

0.6

0.7

0.8

0.9

1.0

APD

Mirror-Prox

rAPDPro

msAPD

APD+restart

Datasets We selected 6 small-to-median scale datasets from various domains in the Network Datasets [28]. We skip large-scale networks as MOSEK struggles to achieve the optimal solution, making it unsuitable for subsequent comparison of the optimality gap. We briefly describe these datasets in Table 1. For more details, please refer to the network repository.

Parameter tuning For all experiments, we set r = min i ∈ [ n ] | d i | , µ = λ min ( Q ) and L X = λ max ( Q ) , with λ min ( · ) , λ max ( · ) denoting the smallest and largest eigenvalue, respectively. For msAPD, we have made additional parameter adjustments. Based on our observations, due to a small estimated strongly convex coefficient, msAPD could not switch to the next cycle s early enough. To prevent msAPD from degrading to APD, we iterate according to the predefined number of sub-iterations and manually switch to the next set of parameters. We divide τ by √ 2 , multiply σ by √ 2 , and increase the number of sub-iterations in the next period by a factor of √ 2 . For all experiments, we tune the stepsize τ, σ, γ from { 0 . 0001 , 0 . 0005 , 0 . 001 , 0 . 005 , 0 . 01 } , where τ, σ are the initial stepsizes of rAPDPro, msAPD and APD, γ is the constant stepsize of Mirror-Prox. All algorithms start with the primal variables initialized as zero vectors and the dual variables initialized as ones.

Additional experiment results Figure 3 and Figure 4 describe the convergence performance and active set identification results on the last three datasets: DD68, DD242 and peking-1. Furthermore, we report the time consumption for the Personalized PageRank problem in Table 2. The table indicates that, although rAPDPro and msAPD require moderately complex computations to determine the lower bound of the strong convexity parameter, the two methods still accelerate the algorithm's convergence and can significantly reduce the overall convergence time.

Nonetheless, we observe that Mosek achieves significantly faster computational efficiency for small-scale problems than our algorithm. Therefore, we test the efficiency of rAPDPro on some large-scale instances. For large-scale instances, we consider the following problem min x ∈ R n ∥ x -1 ∥ 1 s . t . 1 2 x ⊤ Q i x + c ⊤ i x + d i ≤ 0 , i = 1 , . . . , m, where Q i are dense and positive definite matrix and generated randomly and c i are generated randomly. Furthermore, we set proper d i to make the feasible region is non-empty. When n = 5000 and m> 10 , MOSEK crashes on our computer, which means we can not get x ∗ for calculating the optimality gap. Therefore, we report the time required

for the algorithm to satisfy max {| f ( x ) -f ( x ∗ ) | / | f ( x ∗ ) | , max { G ( x ) , 0 }} ≤ 10 -3 and the time taken by the algorithm to complete 10,000 iterations. On this problem, results from small datasets indicate that the performance of the 10,000-step algorithm should be sufficient to meet our specified termination criteria.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We state the complete contributions in the Introduction section.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our work in Appendix A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We give all assumptions needed for the theorems we are proving, such as Assumption 1, 2, 3 and 4, to ensure the conclusion is correct.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our experimental reproduction scripts have been placed in the attachment.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our experiments use entirely publicly available datasets, and we are committed to making our code completely open source.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All details can be found in the paper and supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Since our algorithm is deterministic, our experimental results do not report standard deviation correlation results, but we have experimented on a wide range of datasets to demonstrate the robustness of our algorithm.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments are run on Mac mini M2 Pro, 32GB.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The code in submission is fully compliant with the NeurIPS code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators or original owners of assets are properly credited, and the license and terms of use are explicitly mentioned and respected in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide a complete document of our code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce faster accelerated primal-dual algorithms for minimizing a convex function subject to strongly convex function constraints.\n\nSTRENGTHS:\n1. The authors address the theoretical questions about strongly convex-constrained optimization and the application of sparse optimization.\n\n2. The authors present a new accelerated primal-dual algorithm with progressive strong convexity estimation (APDPro) for solving the problem (1).\n\n3. The authors present a new restart algorithm (rAPDPro) which calls APDPro repeatedly with the input parameters properly changing over time.\n\nWEAKNESSES:\n1. What is the 2-norm (see line 116)? what is the meaning of $\\bot$?\n\n2. The conditions of Theorem 1 are too strict and limited, which hinders the application of APDPro. There are also similar issues with the corollary 1.\n\n3. As illustrated by formulation (2), when $f$ is convex and $g_i$ is strongly convex, $L(\\cdot, y)$ is strongly convex and the convergence rate of gradient descent is linear, which is a well-known conclusion in optimization. However, the convergence rate of your method is sublinear [1, 2]. \n\n4. This paper lacks related work. There are several studies exploring optimization with the objective of minimizing a convex function subject to strongly convex function constraints.\n\n5. The improvement in the convergence rate comes at the cost of increased computational effort per step.\n\n6. Convex optimization with strongly convex function constraints has been explored by [1, 2], but the authors don’t cite these references. Moreover, Nesterov's Accelerated Gradient Method [3] can also achieve a complexity of $1/\\sqrt{\\epsilon}$. The authors did not improve the convergence rate compared to previous works.\n\n\n[1] Jorge Nocedal, Stephen J. Wright: Numerical Optimization. Springer 1999, ISBN 978-0-387-98793-4, pp. 1-634\n\n[2] Yurii E. Nesterov: Introductory Lectures on Convex Optimization - A Basic Course. Applied Optimization 87, Springer 2004, ISBN 978-1-4613-4691-3, pp. 1-236\n\n[3] A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights. J. Mach. Learn. Res. 17: 153:1-153:43 (2016)\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis method proposes new acceleration methods to solve the convex optimization problem with convex constraints. To do this, the authors propose to iteratively improve the lower bounds of the strong convexity parameter of the associated Lagrangian. In turn, this lower bound is used to create cutting planes for the domain of the dual variables. By using this new technique, the authors is able to improve the running time complexity of the base method APD from $O(1/k)$ to $O(1/k^2)$, where $k$ is the number of iterations. The authors verify the claim of running time improve empirically through several datasets.\n\nSTRENGTHS:\nThe core techniques (iteratively improve the lower bound of the strong convexity parameter and use it to add cutting planes to the domain of the dual variables) seem to be novel. The improvement on computational complexity is nontrivial and matches the lower bound for this class.\n\nThe writing is clear in terms of highlighting the high-level ideas and emphasizing which part is the authors' contribution.\n\nThe code is also available as part of the supplementary materials.\n\nWEAKNESSES:\n1. For Figure 2 on the active-set identification experiment, I am not seeing a big difference between APD (baseline) and rAPDPro (proposed method). So I am guessing the core empirical contribution is on improving the curve of optimality gap vs number of iterations. It would be nice to show the curve of optimality gap vs wall-clock running time.\n\nQUESTIONS:\n1. For the experimental section, you only show optimality gap vs number of iterations. Could you also report optimality gap vs wall-clock running time? You can report these results in table format during rebuttal.\n\n2. For the baseline method APD, is restart also applied? For fair comparison, I think you should compare rAPDPro with APD + restart.\n\n3. The authors obtain the optimal solutions via MOSEK. Could you report the optimality gap of the MOSEK solver together with what are shown in Figure 1?\n\n4. For Figure 3 and Figure 4 in the appendix, can you explain why the baselines (mirror-P and APD) fail and perform so poorly on the rightmost plot?\n\nLIMITATIONS:\nThe authors have already pointed out the limitations in this submission:\n\n1. The proposed method requires knowing an upper bound on $\\lVert \\mathbf{y}^* \\rVert$.\n\n2. If the evaluation of the proximal operator of $f$ is inexpensive, this method can be rightly applied; if not, this method will incur additional computational cost.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper introduces accelerated primal-dual algorithms for minimizing a convex function subject to strongly convex constraints. Currently, the best complextiy bound for these problems is $\mathcal{O}(1/\epsilon)$, even when the constraints are strongly convex. However, this work develops a technique to progressively estimate the strong convexity of the Lagrangian function, and thereby establishes an improved, and optimal, complexity bound of $\mathcal{O}(1/\sqrt{\epsilon})$. Further, a restarted version of the methods can identify the sparsity pattern of the optimal solution within a finite number of steps.

STRENGTHS:
The paper was well written and the problem is of interest to the wider research community.
The paper establishes convergence of their method, and exploits the strong convexity of the constraint functions to obtain an improved complexity of $\mathcal{O}(1/\sqrt{\epsilon})$.
That the restarted version of the algorithm can identify the sparsity pattern of the optimal solution in a finite number of steps, which is independently interesting.

WEAKNESSES:
The claim of "optimal rate" does assume that $\tilde \rho_K$ is small. If it is order epsilon, then the `usual' complexity bound holds (the authors are upfront about this as it is noted in Remark 2). This does weaken the "optimal" claim somewhat.

QUESTIONS:
1. In Definition 1, does the last condition (orthogonality of the vectors) imply complementarity? Does one not need $\mathbf{y^*}_i g_i(\mathbf{x^*}) = 0$ for all $i$?
2. On line 140, the word "closeness" is used. Do you mean "closedness"?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOverall, this paper introduces a new idea and new result which is that we can accelerate constraint optimization as soon as the constraint sets are strongly convex. This result is very interesting to the community, but the paper’s writing is really bad as it is and needs a huge improvement to be publishable in my opinion.\n\nSTRENGTHS:\nThis paper addresses an important problem, constrained optimization, and shows that we can achieve a Nesterov-like acceleration over such problems.\n\nWEAKNESSES:\nHere I list all the remarks that I would like to address, going from typo to concern. They are not all of equal importance, I list them conserving the order of the paper:\n\n- l.87: Can the authors be more precise on which upper bound it is? Is this some known upper bound that we need to use in the algorithm? Or $||y_0 - y_*||$ would work? This is not clear from here.\n\n- l.125: Authors should specify the derivative. Which variables are we deriving against?\n- l.125: $y\\geq 0$ is redundant with $y\\in\\mathbb{R}_+^m$.\n- l.125: I suggest to the authors to avoid such a sequence of different types of relations and instead write \"$G(x^*)\\leq 0$ and $<y^*, G(x^*)> = 0$\".\n\n- l.131: « easily verifiable ». Can the authors explicitly say how? I understand in the particular case they mention right after, but in general, it would require finding all minimizers of $f$ and check. But no practical algorithm is guaranteed to provide an exact solution to unconstraint minimization of $f$, and even less to provide all of them.\n- My personal point of view on Assumption 2 is that, while Problem 1 would be equivalent to an unconstrained minimization problem if this assumption is not verified, we would like to have an algorithm that covers both cases to avoid the need for determining in which case we are. Except if, as authors pretend, Assumption 2 is easily verifiable. It is fair to make an assumption if the most general case if hard to solve, but I think the authors should not underestimate the importance of this assumption in their text.\n\n- Section D1 eq 19: Authors should not use $\\Longrightarrow$ instead of « then » or « thus » or « it follows ». $A \\Longrightarrow B$ has a very precise signification. It means that if A, then B, but without knowing if A is verified. Which is different from stating A as a true statement and concluding with B.\n- Section D1 eq 19: The last $\\leq \\bar{c}$ should be replaced by an equality by definition of $\\bar{c}$.\n\n- l.141: Is the closeness of the subdifferential set of proper convex functions a result originally dated back to 2017? Otherwise, I think a more relevant reference should be preferred. This applied to other references in this paper. Authors should prefer original ones except a more recent one brings some very clear and new explanation to the claim.\n\n- Authors should add a sentence after the different statements they make with a clickable link to the place where we can find the proof in their paper/appendix.\n- l.141: « we derive a subdifferential separation result ». Where is it? From assumption 2, I understand that $d(\\partial f(x^*), 0) > 0$. So if $r$ depends on $f$, the statement is trivial, if it is uniform over all the problems, then it needs proof.\n\n- After stating Assumption 1, the authors should define $\\tilde{x}$ as a generic notation for the existing strictly feasible point so that it can be used later. Otherwise, the $\\tilde{x}$ is not defined in Propositions 1 and 3.\n\n- In general, I suggest the authors motivate their lemmas/propositions/theorems before stating them (e.g. Proposition 3). Otherwise, a reader does not know where all this is going.\n\n- Proposition 3: Why add a $\\zeta$? Since $\\tilde{x}$ is in the interior of $\\mathcal{X}_G$, the first inequality replacing $x_1$ and $x_2$ by respectively $\\tilde{x}$ and $x^*$ should be strict, and therefore $\\zeta$ can be set to 0.\n\n- eq.7: At this stage, the authors explain that we can straightforwardly prove some bound but we do not know why it is useful, nor what it means. More motivation and discussions are needed to improve the writing of this paper.\n\n- l.164: For better readability, I suggest the author use classical tools. What they define as $prox_{f, \\mathcal{X}}(x, z, \\eta)$ is simply the prox of $f$ at a different point: $prox_{f, \\mathcal{X}}(x - \\eta z, \\eta)$. Otherwise one can think that the authors assume access to many different oracles, the proximal operators of many different functions.\n\n- l.178: $N_\\mathcal{X} \\rightarrow \\mathcal{N}_\\mathcal{X}$\n\n\nAlgo 1:\n- l.4: « Compute $\\theta_k$ ». How? Eq.11?\n- $\\theta_k$ is only used in l.5 which defines $z_k$, only used in l.6. I suggest authors merge those lines into a single one.\n- l.8: 3 updates in one line. Authors should at least use « \\qquad » to separate them and increase readability.\n- l.10: Update how? In any way as soon as Thm1 assumptions are verified? $\\gamma$ is not defined in Thm1. Should we look at Cor 1? It is then defined as $\\sigma / \\tau$. Why do we update it after updating $\\sigma$ and $\\tau$? It is not even called in Algo 1.\n\nThm1:\n- Why assume that $t_{k+1} / \\sigma_{k+1} \\leq t_{k} / \\sigma_{k}$ (second inequality of eq.11) while in Algo 1, $t_k = \\sigma_k / \\sigma_0$? The assumed inequality is necessarily true and even an equality.\n- Concerning the third assumption in eq.11, we discussed it above: the expression of $\\theta$ should be replaced in the algorithm for better readability.\n- Replacing $\\theta_k$ by its expression, one sees that the second inequality of eq.12 simply reads $\\delta \\leq 1$.\nIn summary, the second and third assumptions of eq.11 and the second assumption of eq.12 should be removed and replaced by the sole assumption $\\delta\\leq 1$. Moreover, since $\\delta$ is used nowhere in the algorithm, nor in the consequence eq.13 of Theorem 1, but only mentioned in the first inequality of eq.12, we can replace it with its largest possible value, i.e. 1.\nFinally, in conclusion, authors should replace eq.11 and 12 by\n$\\tau_k(1/\\tau_{k+1} -  \\rho_{k+1}) \\leq \\sigma_k / \\sigma_{k+1}$ and $L_{XY} + L_G^2\\sigma_k \\leq 1/\\tau_k$, making the statement much clearer.\n\nCor1:\n- Again, $\\delta$ seems to be used only in the first assumption and neither the algorithm nor the guarantee depends on the choice of $\\delta$, so why not just fix it to the most permissive value, i.e. 1?\n- $\\theta$ is defined for a single used, so why not avoiding a new definition?\n- $t_k$ is finally fixed as in the algorithm. If the authors want to be more permissive in the theorem, $t_k$ must not be fixed in the algorithm.\n- A new notation arrived: $\\gamma$. Why? This is not used in the algorithm.\n- Moreover, all those assumptions can be simplified a lot. First, the last equation of the second line simply writes $\\tau_k^2\\gamma_k$ is constant. Removing $\\gamma_k$ and writing all this in terms of used quantities, we have that $\\sigma_k\\tau_k$ is. constant. In conclusion, the algorithm is parametrized by 3 sequences:  $t$, $\\tau$ and $\\sigma$, with $t\\propto\\sigma\\propto 1/\\tau $,  which can be greatly simplified using only 1 sequence.\n- The two only useful assumptions of theorem 1 now write (with this new proportionality assumption of Cor.1, introducing $c_0 = \\tau_k\\sigma_k$):\n    - $\\sigma_k / c_0 \\geq L_{XY} + L_G^2\\sigma_k$ and using that. $\\sigma_k$ is increasing as pointed out in Remark 3, it suffices to verify this inequality for $k=0$, explaining the first inequality following l.194 (to be taken for $\\delta=1$): $c_0 \\leq \\sigma_0 / (L_{XY} + L_G^2\\sigma_0)$.\n    - $c_0 \\rho_k \\geq (\\sigma_k^2 - \\sigma_{k-1}^2) / \\sigma_k$. Note that it shows $c_0$ should be taken as large as possible and then the previous bound on $c_0$ has to be taken as an equality. Finally, $\\rho_k$ is upper bounded, so is $(\\sigma_k^2 - \\sigma_{k-1}^2) / \\sigma_k$, showing that $\\sigma$ can only increase up to a certain speed, similar to the classical parameters of NAG algorithm. This clearly explains the fact that $t_k$ (as well as $\\sigma_k$ and $1/\\tau_k$) grows as $k$ and $T_k$ as $k^2$, giving eq.14. Remark 3 explains why we obtain at least a $1/k$ guarantee, but it not more complicated to see that we can actually accelerate. In my view, the algorithm, the thm, and the corollary have all been overcomplicated with many notations and assumptions and a lack of explanation. and intuition that would have made all this straightforward to understand.\n\n\nIn summary, I would say that the method is sound as soon as we have some $\\mu_{min}$ (main assumption of this paper) and some $r$ (eq.4) coming from the assumption that no minimizer of $f$ can be a solution to our problem. Note we also need to have access to such a $r$, which does not seem trivial in general.\n\n\nOverall, this paper introduces a new idea and new result which is that we can accelerate constraint optimization as soon as the constraint sets are strongly convex. This result is very interesting to the community, but the paper’s writing is really bad as it is and needs a huge improvement to be publishable in my opinion.\n\nQUESTIONS:\n- l.7-8: « Our approach, for the first time, effectively leverages the constraint strong convexity, obtaining an improved complexity of $O(1/\\sqrt{\\varepsilon})$ ».\n- l.27-28: « When the objective is strongly convex, the complexity can be further improved to $O(1/\\sqrt{\\varepsilon})$ [cite refs]».\n- l.42-43: « Specifically, direct applications of previously discussed algorithms yield an $O(1/\\varepsilon)$ complexity ».\n\nIs this acceleration result novel?\nWhile I understand that the authors mention a 2-loops vs single loop procedure, this must be clear from the beginning. I find the abstract a bit overselling if there actually is some method, even using 2 loops, that achieves this accelerated rate. I suggest the authors be more specific in their abstract if this is the case.\n\nLIMITATIONS:\nNo limitations other than the claimed assumptions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-19"}