Title: Inexact Augmented Lagrangian Methods for Conic Optimization: Quadratic Growth and Linear Convergence
Venue: NeurIPS 2024
Accepted: yes
Keywords: convex optimization

augmented Lagrangian method

quadratic growth

error bound

linear convergence
Abstract: Augmented Lagrangian Methods (ALMs) are widely employed in solving constrained optimizations, and some efficient solvers are developed based on this framework. Under the quadratic growth assumption, it is known that the dual iterates and the Karush–Kuhn–Tucker (KKT) residuals of ALMs applied to conic programs converge linearly. In contrast, the convergence rate of the primal iterates has remained elusive. In this paper, we resolve this challenge by establishing new $\textit{quadratic growth}$ and $\textit{error bound}$ properties for primal and dual conic programs under the standard strict complementarity condition. Our main results reveal that both primal and dual iterates of the ALMs converge linearly contingent solely upon the assumption of strict complementarity and a bounded solution set. This finding provides a positive answer to an open question regarding the asymptotically linear convergence of the primal iterates of ALMs applied to conic optimization.

Inexact Augmented Lagrangian Methods for Conic Programs: Quadratic Growth and Linear Convergence

Abstract

Augmented Lagrangian Methods (ALMs) are widely employed in solving constrained optimizations, and some efficient solvers are developed based on this framework. Under the quadratic growth assumption, it is known that the dual iterates and the Karush-Kuhn-Tucker (KKT) residuals of ALMs applied to semidefinite programs (SDPs) converge linearly. In contrast, the convergence rate of the primal iterates has remained elusive. In this paper, we resolve this challenge by establishing new quadratic growth and error bound properties for primal and dual SDPs under the strict complementarity condition. Our main results reveal that both primal and dual iterates of the ALMs converge linearly contingent solely upon the assumption of strict complementarity and a bounded solution set. This finding provides a positive answer to an open question regarding the asymptotically linear convergence of the primal iterates of ALMs applied to semidefinite optimization.

1 Introduction

We consider the standard primal and dual semidefinite programs (SDPs) of the form

where the problem data consists of a cost matrix C ∈ S n , a constant vector b ∈ R m , a linear map A : S n → R m A ( X ) = [ ⟨ A 1 , X ⟩ , · · · , ⟨ A m , X ⟩ ] T with A 1 , · · · , A m ∈ S n , and its adjoint operator A ∗ : R m → S n as A ∗ ( y ) = ∑ m i =1 A i y i that satisfy ⟨A ( X ) , y ⟩ = ⟨ X, A ∗ ( y ) ⟩ , ∀ X ∈ S n , y ∈ R m .

SDPs are a broad and powerful class of conic programs, which include linear programs and secondorder cone programs as special cases. The class of SDPs has found an extensive list of applications, including control theory [1], combinatorial optimization [2], polynomial optimization [3], machine learning [4-6], and beyond [7]. Meanwhile, many algorithms have been developed to solve (P) and (D), ranging from reliable interior point methods [8-10] to scalable first-order methods [11-14]. In particular, it is demonstrated that Augmented Lagrangian methods (ALMs) [15, 16] are suitable for solving large-scale optimization problems. Some efficient ALM-based algorithms have been developed to solve (P) and (D). For example, the celebrated low-rank matrix factorization [17] was integrated with ALM for efficient algorithm design. A semi-smooth Newtown-CG algorithm was proposed for ALM to solve SDPs with many affine constraints in [18]. An enhanced version was introduced in [19] to tackle degenerate SDPs by combining a warm starting strategy. The algorithms [18, 19] have been implemented into a solver, SDPNAL+ , which has shown very impressive numerical performance in benchmark problems. Other recent ALM-based algorithms include [20-22].

Despite the impressive practical performance, theoretical convergence guarantees of applying ALMs to (P) and (D) are less complete. Thanks to the seminal work [23], it is well-known that running ALMs for the primal (P) is equivalent to executing the proximal point method (PPM) for the dual (D). Thus, the classical convergences of ALM are typically inherited from PPM, which only guarantees that the dual iterates converge linearly when a Lipschitz continuity property is satisfied [24]. The Lipchitz condition requires (P) to have a unique solution. The uniqueness assumption was relaxed in [25], which allows multiple optimal solutions but requires an error bound property. Recently, this error bound property was established in [26] in a general conic optimization setup. The work [26] also proves linear convergence of KKT residuals when applying ALM to convex composite conic programming. However, due to the connections between ALM and PPM, all these results only guarantee linear convergence for the dual iterates, while the convergence rate of the primal iterates in ALM remains unclear. Indeed, it is noted in [21, 26] that the problem of whether the primal sequence generated by the ALM can also converge asymptotically linearly is open.

This asymmetry in convergence guarantees is notably dissatisfying, especially considering the elegant duality in (P) and (D). In this paper, our main goal is to establish linear convergence guarantees for both primal and dual iterates when applying an inexact ALM to SDPs (either (P) or (D)), under the usual assumption of strong duality and strict complementarity.

Our contributions. To resolve the challenge, we have made novel contributions from two aspects. On the problem structure side:

(a) Under the standard strict complementarity assumption, we establish the quadratic growth and error bound for both primal and dual SDPs (P) and (D) over any compact set containing an optimal solution (see Theorems 1 and 2). Our results not only extend the findings of [27] from a small (yet unknown) neighborhood to any compact set, benefiting the analysis of iterative algorithms, but also improves the recent results in [28-30]; see Remark 1 for a comparison.

(b) To establish the quadratic growth , we unveil a new characterization of the preimage of the subdifferential of the exact penalty functions for SDPs. This reveals a useful connection between exact penalty functions and indicator functions (see (11) in Lemma 3). We believe that this new characterization is of independent interest.

(c) Using the new characterization, we further provide a new and simple proof for the growth properties in the exact penalty functions of S n + (see Appendix B), and clarify some subtle differences in constructing exact penalty functions (see Section 3.1 and Remark B.1).

On the algorithm analysis side:

(a) By leveraging the established quadratic growth and error bound, we show that applying ALM to solve either the primal or dual SDPs (P) and (D) has linear convergence for both the primal and dual iterates (see Theorem 3).

(b) We establish symmetric versions of inexact ALMs for solving both the primal and dual conic programs (see Section 4 and Appendix E.3), which is less emphasized in the literature.

(c) We clarify the subtlety concerning the (in)feasibility of the dual iterates by inexact ALMs (see Section 4.2), which is an important issue concerning the relationship between PPM and ALM.

Related works : We here review some closely related results in the literature.

Quadratic growth and error bound. Many studies have been dedicated to establishing quadratic growth or error bound for conic optimizations under various assumptions [28, 31, 27, 32-35]. Starting from the famous Hoffman's lemma [36], a global error bound for linear systems has been established, which was later extended to a partially infinite-dimensional case [37]. Sturm investigated the error bound for SDPs, in which the exponent term is bounded by 2 d with d being the sigularity degree that is at most n -1 [32]. Zhang provided a simple proof of error bound for a feasibility system under Slater's condition [33]. Zhou and So provided a unified proof to establish error bound for a class of structured convex optimization [35]. The results in [35] are generalized to convex problems with spectral functions [31]. Drusvyatskiy and Lewis discussed quadratic growth for a convex problem of the form min x f ( Ax ) + g ( x ) under dual strict complementarity [34, Section 4]. Very recently, Ding and Udell established an error bound property for general conic programs via an elementary framework using strict complementarity [28]. All the aforementioned results have slightly different assumptions, resulting in different forms of quadratic growth properties. They are not directly applicable to our purpose of establishing linear convergences of primal and dual iterates in ALM; a detailed comparison will be provided in Remark 1 after introducing our main result.

Augmented Lagrangian method (ALM) and Proximal point method (PPM). The ALM is first introduced in [15, 16] to improve numerical performance of penalty methods. It is shown by Rockafellar that the augmented dual function is continuously differentiable [38], and the ALM can be viewed as an augmented dual ascent method [24], thus the convergence of ALM can be analyzed via dual ascent. The seminal work [23] established a strong relation between ALM and PPM: the dual iterate by ALM is the same as the proximal update on the dual function by PPM. From the convergence of PPM, we know that the dual iterates of ALM converge linearly when the preimage of the subdifferential map of the dual function is Lipschitz continuous at the origin [24]. However, the Lipchitz condition requires the solution to be unique, which is not suitable for many practical applications. Luque made an important extension in PPM to allow multiple solutions while maintaining linear convergence [25]. Recently, Cui et. al relaxed the Lipschitz assumption to upper -Lipschtiz continuous which allows multiple primal and dual solutions [26]. In addition, the KKT residuals of convex composite conic programs are shown to converge linearly under the assumption that the dual function satisfies quadratic growth [26]. The primal iterates of ALM are also known to converge linearly under a much stronger Lipschitz assumption on the augmented Lagrangian function [23, 26]. As pointed out in [26, Section 3], the Lipschitz property of the Lagrangian function is more conservative than the quadratic growth of the primal or dual function. To the best of our knowledge, when applying ALM to conic optimization (P) and (D), the problem of whether the primal iterates converge linearly under the normal quadratic growth condition remains open.

Paper outline. The rest of this paper is structured as follows. Section 2 reviews some preliminaries in SDPs and PPM. Section 3 establishes quadratic growth and error bound properties. Section 4 establishes the linear convergence of primal and dual iterates of ALMs under standard assumptions. Section 5 presents numerical experiments, and Section 6 concludes the paper. While all results are presented for SDPs, analogous versions exist for other conic programs, such as LPs and SOCPs.

2 Preliminaries

2.1 Strong duality and strict complementarity

Definition 1 (Strong duality) . We say (P) and (D) satisfies strong duality if we have p ⋆ = d ⋆ .

Strong duality only requires the optimal values to be the same. The existence of optimal primal and dual variables ( X ⋆ , y ⋆ , Z ⋆ ) is ensured by the primal and dual Slater's condition [8, Theorem 3.1], i.e., there exist primal and dual feasible points X and y satisfying X ∈ int( S n + ) and C -A ∗ ( y ) ∈ int( S n + ) . Let Ω P ⊆ S n and Ω D ⊆ R m × S n denote the optimal solutions of (P) and (D), respectively,

Under the primal and dual Slater's condition, the solution sets Ω P , Ω D are nonempty and compact (see e.g., [30, Section 2] or [29, Prop. 2.1]). We further have ⟨ Z ⋆ , X ⋆ ⟩ = 0 , ∀ X ⋆ ∈ Ω P , ( y ⋆ , Z ⋆ ) ∈ Ω D , which is called complementary slackness .

Next, we introduce another regularity condition: dual strict complementarity , which is the key to ensuring many nice properties in a range of conic problems [34, 28, 35].

Definition 2 (Dual strict complementarity for SDPs) . Given a pair of optimal solutions X ⋆ ∈ Ω P , ( y ⋆ , Z ⋆ ) ∈ Ω D , we say that ( X ⋆ , y ⋆ , Z ⋆ ) satisfies dual strict complementarity if X ⋆ ∈ relint( N ( S n + ) ◦ ( -Z ⋆ )) , where ( S n + ) ◦ is the polar cone of S n + , N ( S n + ) ◦ ( -Z ⋆ ) denotes the normal cone of ( S n + ) ◦ at -Z ⋆ , and relint denotes relative interior. If (P) and (D) have one such pair, we say that (P) and (D) satisfy dual strict complementarity.

We note that the usual notion of strict complementarity for SDPs is equivalent to Definition 2, as pointed out in [28, Appendix B]. For completeness, we also review the standard notion of strict complementarity in Appendix A.1. The notion of (dual) strict complementarity is not restrictive, and it holds generically for conic programs [39], [40, Theorem 15]. Recently, it has been revealed that many structured SDPs from practical applications also satisfy strict complementarity [41]. One can also define primal strict complementarity, and the primal and dual strict complementarity are not equivalent in general, but they are equivalent for the so-called self-dual cones, such as S n + .

2.2 Proximal point methods (PPMs)

It is now well-known that running ALMs for the primal (P) is equivalent to executing a PPM for the dual (D), and vice versa [23]. We here give a quick review of PPMs and their standard convergences.

Let f : R n → ¯ R be a proper closed convex function, and consider the problem f ⋆ = min x ∈ R n f ( x ) . Let S = argmin x f ( x ) , which we assume to be nonempty. We define the proximal operator as

where α > 0 . Starting with an initial point x 0 ∈ R n , the PPM generates a sequence of points as follows x k +1 = prox c k ,f ( x k ) , k = 0 , 1 , 2 , . . . , where { c k } k ≥ 0 is a sequence of positive numbers bounded away from zero. The proximal operator (2) is always well-defined thanks to strong convexity.

In practice, the proximal operator (2) may not be easily evaluated, and thus an inexact version is often used [42]. In particular, an inexact PPM generates a sequence of points as

Two classical types of inexactness, originating from the seminal work [42], are

where { ϵ k } k ≥ 0 and { δ k } k ≥ 0 are two sequences of inexactness for (3). The classical work [42] has established asymptotic convergence of the iterates x k from (3) when using criterion (a). Fast linear convergences have also been established under various regularity conditions, such as the inverse of the subdifferential ( ∂f ) -1 is locally Lipschitz at 0 [42] (or relaxed to be upper Lipschitz continuous at 0 [25]), and more generally ∂f is metrically subregular [27, 43] (or f satisfies quadratic growth [34, 44]). Recall that f satisfies quadratic growth at x ⋆ ∈ S if there exists a constant µ q > 0 such that

where U is a neighborhood containing the optimal solution x ⋆ . Convergence results for (3) are:

̸

Lemma 1. Let S = argmin x ∈ R n f ( x ) and suppose S = ∅ . Let { x k } k ≥ 0 be a sequence from (3) .

(a) If criterion (a) is used, then lim k →∞ x k = x ∞ ∈ S , i.e., x k converges to an optimal solution.

(b) In addition to (a) , if (b) is used and f satisfies (QG) at x ∞ ∈ S , then there exists ˆ k > 0 such that dist( x k +1 , S ) ≤ ˆ θ k dist( x k , S ) , ∀ k ≥ ˆ k, where ˆ θ k = θ k +2 δ k 1 -δ k with θ k = 1 / √ c k µ q +1 < 1 .

The first asymptotic convergence is taken from [42, Theorem 1], and different proofs for the second linear convergence under (QG) are available, and a simple one can be found in [44, Theorem 5.3]. Criterion (a) guarantees the iterate x k will arrive within a neighbor U where (QG) holds after some number ˆ k , then criterion (b) ensures the linear convergence; see [44, Theorem 5.3] for details.

Note that Lemma 1 only guarantees the (linear) convergence of iterates x k , which corresponds to the dual iterates in the ALM [23] (see Lemma 4 in Section 4). Thus, once a suitable condition like (QG) is established for (P) and (D), applying the ALM will enjoy linear convergence for dual iterates [21, 25, 26]. In this paper, we aim to establish linear convergences of both primal and dual iterates in the ALM, when applied to (P) or (D). For this, we first establish quadratic growth in Section 3.

3 Quadratic Growth in SDPs

In this section, we identify favorable quadratic growth and error bound properties for both primal and dual SDPs in (P) and (D), under the usual assumptions of strong duality and strict complementarity (cf. Definitions 1 and 2). These favorable structural properties will allow us to establish linear convergences of both primal and dual iterates of inexact ALMs applied to either (P) or (D).

3.1 Growth and error bound properties

To analyze PPM or ALM, we consider an equivalent reformulation of (P) or (D) in the following form

where g (and h , respectively) is chosen as either an indicator function δ S n + or an exact penalty function, defined as 2

where ( y ⋆ , Z ⋆ ) ∈ Ω D can be any optimal solution. In (5), the exact penalty function h is designed in the same way as (6) where the lower bound on ρ is replaced by ρ > tr ( X ⋆ ) and X ⋆ is any optimal solution in Ω P . The common feature in (4) and (5) is to move the difficult conic constraint X ∈ S n + and Z ∈ S n + as a nonsmooth term g ( X ) and h ( Z ) in the cost function. It is clear that for both indicator and exact penalty functions, we have g ( X ) = 0 , ∀ X ∈ S n + and h ( Z ) = 0 , ∀ Z ∈ S n + . For clarity, we state the following technical result.

Lemma 2. Suppose strong duality holds for (P) and (D) . Under the choice of an indicator function or an exact penalty function in (6) , the set of optimal solutions for (4) is the same as Ω P in (1a) , and the set of optimal solutions for (5) is the same as Ω D in (1b) .

The equivalence with an indicator function is obvious, and the equivalence with an exact penalty function has appeared in [30] and [29, Propositions 3.1&3.2]. Their proofs rely on a general characterization in exact penalty methods [45, Theorem 7.21]. In Appendix B, we present a simple, different, and self-contained proof without relying on prior results. Our proof uses a new characterization of the subdifferential of exact penalty functions (Proposition B.1), which might be of independent interest.

Our first main theoretical results are the following growth and error bound properties for primal and conic programs. The proofs will be outlined in Section 3.2.

Theorem 1 (Growth properties in the primal) . Suppose strong duality and dual strict complementarity hold for (P) and (D) . Consider the primal conic programs (P) and (4) . Let ( ¯ X ⋆ , ¯ y ⋆ , ¯ Z ⋆ ) satisfy strict complementarity, and the exact penalty parameter is chosen as ρ > tr ( ¯ Z ⋆ ) . For any compact set U ⊆ S n containing an optimal solution X ⋆ ∈ Ω P , there exist positive constants κ, γ, α > 0 such that

If Ω P is further compact, then set U in (7a) and (7b) can be chosen as any sublevel set of f P , i.e., { X ∈ S n | f P ( X ) ≤ β, A ( X ) = b } with a finite β .

In (7a), the impact of the conic constraint X ∈ S n + is reflected in the reformulated cost f P ( X ) . When using the indicator function as f P ( X ) = ⟨ C, X ⟩ + δ S n + ( X ) , the left-hand side of (7a) implicitly requires X ∈ S n + ; otherwise f P ( X ) = + ∞ and thus the inequality (7a) holds trivially. If using the exact penalty function (6), we have f P ( X ) = ⟨ C, X ⟩ + g ( X ) < + ∞ for X / ∈ S n + , which is closer to (7b). Indeed, our proof of (7b) utilizes a characterization of the exact penalty function (6).

Both (7a) and (7b) give useful bounds when X moves away from the optimal solution set Ω P in terms of suboptimality f P ( X ) -p ⋆ or ⟨ C, X ⟩ -p ⋆ , affine feasibility ∥A ( X ) -b ∥ , and conic feasibility dist( X, S n + ) . They are closely related to quadratic growth in (QG), and are also referred to error bound properties [46, 33, 28]. The error bound plays a vital role in proving fast linear convergence for many iterative algorithms [34, 35, 44, 47]. Indeed, some previous studies [27-30] have established similar results to Theorem 1 for conic optimization, especially SDPs. As we will detail in Remark 1 below, our results are more general and unified. Similarly, the growth and error bound properties also hold for the dual conic program.

Theorem 2 (Growth properties in the dual) . Suppose strong duality and dual strict complementarity hold for (P) and (D) . Consider the dual problems (D) and (5) . Let ( ¯ X ⋆ , ¯ y ⋆ , ¯ Z ⋆ ) satisfy strict complementarity, and the penalty parameter is chosen as ρ> tr ( ¯ X ⋆ ) . For any compact set V ⊆ R m × S n containing an optimal solution ( y ⋆ , Z ⋆ ) ∈ Ω D , there exist constants κ, γ, α > 0 such that

d ⋆ -⟨ b, y ⟩ + γ ∥ C -A ∗ ( y ) -Z ∥ + α · dist( Z, S n + ) ≥ κ · dist 2 (( y, Z ) , Ω D ) , ∀ ( y, Z ) ∈ V . (8b) If Ω D is further compact, then set V can be chosen as any sublevel set of f D , i.e., { ( y, Z ) ∈ R m × S n | f D ( y, Z ) ≤ β, C + Z = A ∗ ( y ) } with a finite β .

The compactness of Ω P and Ω D in Theorems 1 and 2 is not restrictive, as it can be ensured by the dual and primal Slater's conditions (and A ∗ is injective) respectively; see e.g., [29, Proposition 2.1]. Remark 1 . Our results in Theorems 1 and 2 are more general and unified than previous ones [27-30]. Theorems 1 and 2 hold for both the indicator function and the penalty function (6), while all previous results [27-30] have only discussed either one of them. We provide more comparisons below. The guarantees in (7a) and (8b) directly lead to a quadratic growth property of f P and f D :

When considering indicator functions, this property (9) is more general than [27, Corollary 3.1] since we allow for any compact set U (and also any sublevel set of f P or f D ). In contrast, the size of the neighborhood around an optimal solution X ⋆ in [27, Corollary 3.1] depends on the eigenvalues of an optimal dual solution Z ⋆ . Thus the neighborhood only exists but its size might be unknown. Furthermore, our results (7a) and (8a) allow for the residual of the affine constraints A ( X ) = b and C -A ∗ ( y ) = Z , while [27, Corollary 3.1] requires strict affine feasible points.

When considering exact penalty functions, similar quadratic growth properties appear in [29, 30]. However, the penalty ρ in [29, 30] needs to be as large as ρ ≥ 2 sup ( y ⋆ ,Z ⋆ ) ∈ Ω D tr ( Z ⋆ ) . Our results in Theorems 1 and 2 only require a smaller constant as ρ > tr ( ¯ Z ⋆ ) where ¯ Z ⋆ is an optimal dual solution satisfying strict complementarity, and this result is established via a new and simple argument (see Proposition B.2 in the appendix). Finally, an error bound similar to (7b) was also established in [28, Corollary 1] under similar assumptions of strong duality and strict complementarity. Our error bound (7b) does not require the absolute value of f P ( X ) -p ⋆ , and our proof strategy relies on a new characterization of exact penalty functions (see Lemma 3), which offers a new (possibly simpler) perspective than [28, Corollary 1]. The dual property (8b) is new and not discussed in [28]. □

Remark 2 (Conic programs) . Theorems 1 and 2 are stated specifically for SDPs (P) and (D). It is known that linear programs (LP) and second-order cone programs (SOCP) are special cases of SDPs. Therefore, results similar to Theorems 1 and 2 when replacing S n + by nonnegative orthant or second-order cone also exist. Theorems 1 and 2 hold for a general class of conic programs.

3.2 Proof sketches for Theorems 1 and 2

We here provide key proof sketches for Theorems 1 and 2. One key step is the following growth properties of an indicator function and a penalty function.

Lemma 3. Let ¯ X, ¯ Z ∈ S n + satisfying 〈 ¯ X, ¯ Z 〉 = 0 (i.e., both ¯ X and ¯ Z are on the boundary unless one of them is zero). If l : S n → ¯ R is either an indicator function l = δ S n + , or a penalty function

then we have that

Moreover, for any positive value µ ∈ (0 , ∞ ) , there exists a positive constant κ such that

Note that l in (10) is related to, but not identical to, the exact penalty function in (6). The proof of Lemma 3 heavily exploits the nice self-dual structure of the cone S n + . For better clarity, we provide the proofs of (11) and (12) in Proposition B.1 and Appendix C respectively.

Remark 3 . We believe that the results in Lemma 3 are of independent interest, especially in terms of the penalty function (10). Building upon (12), one can further show l is metrically subregular [48, Theorem 3.3]. The most closely related characterization is [27, Proposition 3.3], which focuses solely on the indicator function of the positive semidefinite cone S n + . However, even in this scenario, the result in [27, Proposition 3.3] only guarantees the existence of a small neighborhood around ¯ X . In contrast, our result (12) works for any closed ball B ( ¯ X,µ ) around ¯ X with a finite radius µ > 0 . □

Upon establishing Lemma 3, with bounded linear regularity (see Appendix A.4) that is guaranteed by dual strict complementarity, we can establish the following growth properties for (4) and (5).

Proposition 1. Consider primal and dual conic programs (P) and (D) and their equivalent forms in (4) and (5) respectively. Suppose strong duality and dual strict complementarity hold for (P) and (D) . Let ( X ⋆ , y ⋆ , Z ⋆ ) be a pair of primal and dual optimal solutions, i.e., X ⋆ ∈ Ω P and ( y ⋆ , Z ⋆ ) ∈ Ω D . For any µ > 0 , there exist positive constants κ 1 , γ 1 , κ 2 , γ 2 such that

The proof of Proposition 1 is given in Appendix D.1. We now see that Theorems 1 and 2 are direct consequences of Proposition 1 as max { 0 , λ max ( -X ) } ≤ dist( X, S n + ) for all X ∈ S n . The compactness statement in Theorems 1 and 2 comes from the fact that the solution set Ω P (resp. Ω D ) is compact if and only if any sublevel set of f P (resp. f D ) is compact (see, e.g., [29, Lemma D.1]).

4 Augmented Lagrangian Methods (ALMs) for SDPs

In this section, we prove that both primal and dual iterates of ALMs, when applied to (P) and (D), enjoy linear convergence under the usual assumption of strict complementarity.

4.1 Inexact Augmented Lagrangian Method

In principle, ALM can be applied to solve both (P) and (D). However, most existing results focus on solving (D) [18, 19]. For simplicity, we here focus on one formulation of the augmented Lagrangian function for (P). The dual formulation is presented in Appendix E.3 for completeness.

We start by introducing two dual variables y ∈ R m and Z ∈ S n + and defining the Lagrangian function for (P) as L 0 ( X,y,Z ) = ⟨ C, X ⟩ + ⟨ y, b -A ( X ) ⟩ + ⟨ Z, X ⟩ . The corresponding Lagrangian dual function and dual problem read as

It can be verified the dual in (13) is the same as the dual conic program (D). Given a penalty parameter r > 0 , the Augmented Lagrangian function of (P) corresponding to L 0 is defined as

where Π S n + ( · ) denotes the orthogonal projection onto the cone S n + . Note that L r is continuously differentiable as ∥ Π S n + ( · ) ∥ 2 is continuously differentiable [18, Section 1]. Given initial points ( y 0 , Z 0 ) ∈ R m × S n + and a sequence of positive scalars r k ↑ r ∞ < + ∞ , the inexact ALM generates a sequence of { X k } (primal variables) and { y k , Z k } (dual variables) as

For notational convenience, we write w = ( y, Z ) and w k = ( y k , Z k ) and consider the following two inexactness criteria for solving (15a) (since it can be a challenge to solve (15a) exactly):

The two stopping criteria above are suggested by the seminal work of Rockafellar [23].

4.2 Linear convergences of primal and dual iterates in ALM

After [23], one classical method for analyzing the convergence of the inexact ALM (15) is to utilize the connection between ALM and PPM. We review the following important lemma.

Lemma 4 ([23, Proposition 6]) . The dual iterates w k +1 = ( y k +1 , Z k +1 ) in (15b) and (15c) satisfy the following relationship

where g 0 is the Lagrangian dual function in (13) .

If (15a) is updated exactly, Lemma 4 confirms that w k +1 = prox r k , -g 0 ( w k ) : the dual iterate of the ALM agree with the PPM iterate for the dual problem (13). If (15a) is updated inexactly, the iterate w k +1 may not satisfy the affine constraint C = Z k +1 + A ∗ ( y k +1 ) . Moreover, viewing Lemma 4, the stopping criteria (A ′ ) and (B ′ ) naturally imply that the dual iterate w k +1 satisfies

Then, we can establish the convergence of the dual sequence { w k } in (15) from Lemma 1 via PPM. In particular, this observation was first discovered in [23] and later tailored in convex composite conic programmings in [26]. We summarize asymptotic convergences of (15) below.

̸

Proposition 2 (Asymptotic convergences) . Consider (P) and (D) . Assume strong duality holds and Ω D = ∅ . Let { X k , w k } be a sequence from the ALM (15) under (A ′ ) . The following hold.

(a) The dual sequence { w k } is bounded. Further, lim k →∞ w k = w ∞ ∈ Ω D (i.e., the whole sequence converges to a dual optimal solution).

(b) The primal feasibility and cost value gap satisfy

(c) If the primal solution set Ω P in (1a) is nonempty and bounded, then the primal sequence { X k } is also bounded, and all of its cluster points belongs to Ω P .

Proof. We give the sketch of proof for parts (a) and (b). A complete proof can be found in Appendix E.1. Part (a) comes directly from the PPM convergence Lemma 1 as Lemma 4 and the stopping criteria (A ′ ) naturally imply the dual iterate w k +1 satisfies ∥ w k +1 -prox r k , -g 0 ( w k ) ∥≤ ϵ k and ∑ ∞ k =1 ϵ k < ∞ .

In part (b), the first inequality uses the fact that Z k +1 + ( X k +1 -X k ) /r k ∈ S n + by performing Moreau decomposition [49, Exercise 12.22] on r k X k +1 -Z k and using the update rule (15b); The second inequality comes directly from (15c); The last inequality uses the definition of L r k ( X k +1 , w k ) in (14) and the fact min X ∈ S n L r k ( X,w k ) ≤ p ⋆ .

Part (c) is a consequence of part (b) and the fact that Ω P is bounded if and only if for any γ ∈ R 3 the set { X ∈ S n | dist( X, S n + ) ≤ γ 1 , ∥A ( X ) -b ∥ ≤ γ 2 , ⟨ C, X ⟩ ≤ γ 3 } is bounded [23, page 110].

Proposition 2 establishes the asymptotic convergences for both the primal and dual variables. The linear convergence of the dual iterates can also be deduced when the negative of the dual g 0 defined in (13) satisfies (QG). However, the rate of the primal iterates remains unclear. Here, leveraging the error bound (8b), we also derive a linear convergence of the primal iterates, which is one main technical contribution of this work. We summarize linear convergence results below.

̸

Theorem 3 (Linear convergences) . Consider (P) and (D) . Assume strong duality and dual strict complementarity holds (implying Ω P = ∅ and Ω D = ∅ ). Let { X k , w k } be a sequence from the ALM (15) under (A ′ ) and (B ′ ) . The following statements hold.

̸

(a) (Dual iterates and KKT residuals) There exist constants ˆ k > 0 such that for all k ≥ ˆ k , we have

where 0 < µ k < 1 , µ ′ k > 0 , and µ ′′ k > 0 are positive finite constants.

(b) (Primal iterates) If Ω P is bounded, then the primal sequence { X k } also converges linearly to the solution set Ω P , i.e., there a constant ˆ k > 0 such that for all k ≥ ˆ k ,

where τ k > 0 is a finite constant.

Proof. The proof of part (a) is largely motivated by the techniques in [26] that focus on ALMs on dual SDPs. Part (b) is a consequence of the error bound (7b). Here, we highlight the key steps of the proof of part (a) due to the page limit, and detailed arguments about part (a) can be found in Appendix E.2. Note that if dual strict complementarity holds, Theorem 2 (with h in (5) as an indicator function) guarantees the following two nice properties: 1) -g 0 in (13) satisfies quadratic growth in (9b); 2) the error bound (7b) holds.

By the discussion after Lemma 4, since (A) and (B ′ ) are in force, the dual sequence { w k } satisfies

Applying the convergence result of PPM in Lemma 1 yields the linear convergence of the dual distance, i.e., there exists a k 1 ≥ 0 such that

Using part (b) in Proposition 2 and ∥ w k +1 -w k ∥ ≤ 1 1 -δ k dist( w k , Ω D ) if δ k < 1 , we can show

Finally, as δ k → 0 , there exists k 2 ≥ 0 such that δ k < 1 and for all k ≥ k 2 . Thus, part (a) then follows from taking ˆ k = max { k 1 , k 2 } .

Note that the error bound (7b) and the assumption that Ω P is bounded guarantee that for any bounded set U containing Ω P , there exist constants α 1 , α 2 , α 3 > 0 such that

By Proposition 2 - part (c), the sequence { X k } is bounded. Let U ⊆ S n be a bounded set that contains Ω P and all the iterates { X k } . Combing (18) with (16), we have

Dividing both sides by α 3 leads to the desired result in part (b). This completes the proof.

To our best knowledge, achieving linear convergence for the primal iterates of inexact ALMs requires a significantly stronger condition on the Lagrangian function [23, Theorem 5] and implementing an additional stopping criterion [26, Proposition 3]. Unfortunately, as highlighted in [26, Section 3], such an assumption in [23, Theorem 5] can easily fail for general conic programs.

Our result Theorem 3, however, reveals that the linear convergence of the primal iterates also happens under the standard assumption of strict complementarity and bounded primal solution set. This suggests that primal linear convergence is often expected since strict complementary is a generic property of conic programs [40, Theorem 15]. Our result not only completes theoretical convergences of inexact ALMs but also offers insights for practical successes in [18, 19].

Remark 4 (Strict complementarity) . As we see in the proof of Theorem 3, strict complementarity is the key to ensure that 1) the negative of the function g 0 in (13) satisfies quadratic growth; 2) the error bound property (E.2) holds for the primal conic program. The quadratic growth is used to derive the dual linear convergence (16), and the error bound is to conclude the primal linear convergence. □

5 Numerical experiments

In this section, we present numerical experiments to examine the empirical performance of ALM introduced in Section 4. We consider two applications in combinational problems and machine learning, including the SDP relaxation of maximum cut (Max-Cut) problem [50] and linear Support Vector Machine (SVM). The Max-Cut problem and the linear SVM can be formulated as

where 1 is an all one vector, A ∈ R m × n and b ∈ R m are problem data in the linear SVM, diag( b )

denotes the diagonal matrix with b as the diagonal elements, and λ > 0 is a constant. We can see that Max-Cut is of the same form as (P). Despite linear SVM is not in the form of SDP (P) but a quadratic program, the corresponding ALM with similar convergence guarantees can also be derived since a quadratic program is a special case of SDP. For Max-Cut problem, we select the graph G 1 , G 2 , and G 3 from the website https://web.stanford.edu/~yyye/yyye/Gset/ and only take the first 20 × 20 submatrix as the considered problem data C . For linear SVM, we randomly generate the problem data A ∈ R 10 × 100 and b ∈ {-1 , 1 } 100 . We then apply the ALM (15) for those instances and compute the relative primal and dual cost value gap and the relative feasibility residuals as follows

The numerical results are presented in Figure 1. In all cases, the primal and dual cost value gap and the KKT residuals all converge linearly to at least the accuracy of 10 -5 . The oscillation or flattening behavior that appears in the tail (when the iterates are close to the solution set) could be due to the inaccuracy of the subproblem solver and computational impreciseness. A detailed theoretical analysis of such behavior is interesting, and we leave it to future work.

Further details on the algorithm parameters, problem instances, and more numerical experiments for machine learning applications can be found in Appendix F.

6 Conclusion

In this paper, we have established the quadratic growth and error bound of two different variants (4) and (5) of SDPs under the condition of strict complementarity. Central to our approach is the examination of the growth properties of the indicator and exact penalty functions. By leveraging these new quadratic growth and error bounds, we establish the linear convergence of both primal and dual iterates of inexact ALMs when applied to semidefinite programs, under the usual assumption of strict complementarity and a bounded solution set on the primal side. Our result not only fills a void in the convergence theory of inexact ALMs but also offers valuable insights into the exceptional numerical performance of solvers rooted in ALMs, such as SDPNAL+ [19]. We expect further interesting applications in machine learning and polynomial optimization [11] for inexact ALMs.

Acknowledgments and Disclosure of Funding

This work is supported by NSF ECCS 2154650, NSF CMMI 2320697, and NSF CAREER 2340713. We would like to thank Dr. Xin Jiang for the discussions on exact penalty functions, and Mr. Pranav Reddy for his assistance in proofreading the manuscript and contributions to the appendix.

References

Stephen Boyd, Laurent El Ghaoui, Eric Feron, and Venkataramanan Balakrishnan. Linear matrix inequalities in system and control theory . SIAM, 1994.

Renata Sotirov. SDP relaxations for some combinatorial optimization problems. In Handbook on Semidefinite, Conic and Polynomial Optimization , pages 795-819. Springer, 2012.

Grigoriy Blekherman, Pablo A Parrilo, and Rekha R Thomas. Semidefinite optimization and convex algebraic geometry . SIAM, 2012.

Gert RG Lanckriet, Nello Cristianini, Peter Bartlett, Laurent El Ghaoui, and Michael I Jordan. Learning the kernel matrix with semidefinite programming. Journal of Machine learning research , 5(Jan):27-72, 2004.

Aditi Raghunathan, Jacob Steinhardt, and Percy S Liang. Semidefinite relaxations for certifying robustness to adversarial examples. Advances in neural information processing systems , 31, 2018.

Ben Batten, Panagiotis Kouvaros, Alessio Lomuscio, and Yang Zheng. Efficient neural network verification via layer-based semidefinite relaxations and linear cuts. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , pages 2184-2190. International Joint Conferences on Artificial Intelligence Organization, 8 2021. Main Track.

Lieven Vandenberghe and Stephen Boyd. Applications of semidefinite programming. Applied Numerical Mathematics , 29(3):283-299, 1999.

Lieven Vandenberghe and Stephen Boyd. Semidefinite programming. SIAM review , 38(1):49-95, 1996.

Farid Alizadeh. Interior point methods in semidefinite programming with applications to combinatorial optimization. SIAM journal on Optimization , 5(1):13-51, 1995.

Henry Wolkowicz, Romesh Saigal, and Lieven Vandenberghe. Handbook of semidefinite programming: theory, algorithms, and applications , volume 27. Springer Science & Business Media, 2012.

Yang Zheng, Giovanni Fantuzzi, and Antonis Papachristodoulou. Chordal and factor-width decompositions for scalable semidefinite and polynomial optimization. Annual Reviews in Control , 52:243-279, 2021.

Zaiwen Wen, Donald Goldfarb, and Wotao Yin. Alternating direction augmented Lagrangian methods for semidefinite programming. Math. Program. Comput. , 2(3):203-230, 2010.

Brendan O'Donoghue, Eric Chu, Neal Parikh, and Stephen Boyd. Conic optimization via operator splitting and homogeneous self-dual embedding. Journal of Optimization Theory and Applications , 169(3):1042-1068, June 2016.

Yang Zheng, Giovanni Fantuzzi, Antonis Papachristodoulou, Paul Goulart, and Andrew Wynn. Chordal decomposition in operator-splitting methods for sparse semidefinite programs. Math. Program. , 180(1):489-532, 2020.

Magnus R Hestenes. Multiplier and gradient methods. Journal of optimization theory and applications , 4(5):303-320, 1969.

Michael JD Powell. A method for nonlinear constraints in minimization problems. Optimization , pages 283-298, 1969.

Samuel Burer and Renato DC Monteiro. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Math. Program. , 95(2):329-357, 2003.

Xin-Yuan Zhao, Defeng Sun, and Kim-Chuan Toh. A Newton-CG augmented Lagrangian method for semidefinite programming. SIAM J. Optim. , 20(4):1737-1765, 2010.

Liuqin Yang, Defeng Sun, and Kim-Chuan Toh. Sdpnal+: a majorized semismooth Newton-CG augmented Lagrangian method for semidefinite programming with nonnegative constraints. Math. Program. Computation , 7(3):331-366, 2015.

Alp Yurtsever, Joel A Tropp, Olivier Fercoq, Madeleine Udell, and Volkan Cevher. Scalable semidefinite programming. SIAM Journal on Mathematics of Data Science , 3(1):171-200, 2021.

Ying Cui, Chao Ding, Xu-Dong Li, and Xin-Yuan Zhao. Augmented Lagrangian methods for convex matrix optimization problems. Journal of the Operations Research Society of China , 10(2):305-342, 2022.

Ling Liang, Defeng Sun, and Kim-Chuan Toh. An inexact augmented Lagrangian method for second-order cone programming with applications. SIAM J. Optim. , 31(3):1748-1773, 2021.

R Tyrrell Rockafellar. Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Mathematics of operations research , 1(2):97-116, 1976.

R Tyrell Rockafellar. The multiplier method of hestenes and powell applied to convex programming. Journal of Optimization Theory and applications , 12(6):555-562, 1973.

Fernando Javier Luque. Asymptotic convergence analysis of the proximal point algorithm. SIAM Journal on Control and Optimization , 22(2):277-293, 1984.

Ying Cui, Defeng Sun, and Kim-Chuan Toh. On the R-superlinear convergence of the KKT residuals generated by the augmented Lagrangian method for convex composite conic programming. Math. Program. , 178:381-415, 2019.

Ying Cui, Defeng Sun, and Kim-Chuan Toh. On the asymptotic superlinear convergence of the augmented Lagrangian method for semidefinite programming with multiple solutions. arXiv preprint arXiv:1610.00875 , 2016.

Lijun Ding and Madeleine Udell. A strict complementarity approach to error bound and sensitivity of solution of conic programs. Optimization Letters , 17(7):1551-1574, 2023.

Feng-Yi Liao, Lijun Ding, and Yang Zheng. An overview and comparison of spectral bundle methods for primal and dual semidefinite programs. arXiv preprint arXiv:2307.07651 , 2023.

Lijun Ding and Benjamin Grimmer. Revisiting spectral bundle methods: Primal-dual (sub) linear convergence rates. SIAM J. Optim. , 33(2):1305-1332, 2023.

Ying Cui, Chao Ding, and Xinyuan Zhao. Quadratic growth conditions for convex matrix optimization problems associated with spectral functions. SIAM J. Optim. , 27(4):2332-2355, 2017.

Jos F Sturm. Error bounds for linear matrix inequalities. SIAM J. Optim. , 10(4):1228-1248, 2000.

Shuzhong Zhang. Global error bounds for convex conic problems. SIAM J. Optim. , 10(3):836851, 2000.

Dmitriy Drusvyatskiy and Adrian S Lewis. Error bounds, quadratic growth, and linear convergence of proximal methods. Mathematics of Operations Research , 43(3):919-948, 2018.

Zirui Zhou and Anthony Man-Cho So. A unified approach to error bounds for structured convex optimization problems. Math. Program. , 165:689-728, 2017.

Alan J Hoffman. On approximate solutions of systems of linear inequalities. Journal of Research of the National Bureau of Standards , 49(4), 1952.

James V Burke and Paul Tseng. A unified analysis of hoffman's bound via fenchel duality. SIAM J. Optim. , 6(2):265-282, 1996.

R Tyrrell Rockafellar. A dual approach to solving nonlinear programming problems by unconstrained optimization. Math. Program. , 5(1):354-373, 1973.

Mirjam Dür, Bolor Jargalsaikhan, and Georg Still. Genericity results in linear conic programming-a tour d'horizon. Mathematics of operations research , 42(1):77-94, 2017.

Farid Alizadeh, Jean-Pierre A Haeberly, and Michael L Overton. Complementarity and nondegeneracy in semidefinite programming. Math. Program. , 77(1):111-128, 1997.

Lijun Ding and Madeleine Udell. On the simplicity and conditioning of low rank semidefinite programs. SIAM J. Optim. , 31(4):2614-2637, 2021.

R Tyrrell Rockafellar. Monotone operators and the proximal point algorithm. SIAM journal on control and optimization , 14(5):877-898, 1976.

D Leventhal. Metric subregularity and the proximal point method. Journal of Mathematical Analysis and Applications , 360(2):681-688, 2009.

Feng-Yi Liao, Lijun Ding, and Yang Zheng. Error bounds, PL condition, and quadratic growth for weakly convex functions, and linear convergences of proximal point methods. arXiv preprint arXiv:2312.16775 , 2023.

Andrzej Ruszczynski. Nonlinear optimization . Princeton university press, 2011.

Jong-Shi Pang. Error bounds in math. program. Math. Program. , 79(1-3):299-332, 1997.

Ion Necoara, Yu Nesterov, and Francois Glineur. Linear convergence of first order methods for non-strongly convex optimization. Math. Program. , 175:69-107, 2019.

FJ Aragón Artacho and Michel H Geoffroy. Characterization of metric regularity of subdifferentials. Journal of Convex Analysis , 15(2):365, 2008.

R Tyrrell Rockafellar and Roger J-B Wets. Variational analysis , volume 317. Springer Science &Business Media, 2009.

Michel X Goemans and David P Williamson. Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM (JACM) , 42(6):1115-1145, 1995.

R Tyrrell Rockafellar. Convex analysis , volume 11. Princeton university press, 1997.

Eduardo H Zarantonello. Projections on convex sets in Hilbert space and spectral theory: Part i. Projections on convex sets: Part ii. Spectral theory. In Contributions to nonlinear functional analysis , pages 237-424. Elsevier, 1971.

Heinz H Bauschke and Jonathan M Borwein. On projection algorithms for solving convex feasibility problems. SIAM review , 38(3):367-426, 1996.

Heinz H Bauschke, Jonathan M Borwein, and Wu Li. Strong conical hull intersection property, bounded linear regularity, Jameson's property (G), and error bounds in convex optimization. Math. Program. , 86:135-160, 1999.

Michael L Overton. Large-scale optimization of eigenvalues. SIAM J. Optim. , 2(1):88-120, 1992.

MOSEK ApS. The MOSEK optimization toolbox for MATLAB manual. Version 9.0. , 2019.

Contents

Appendix

The appendix is divided into the following seven sections:

Appendix A presents some preliminaries in conic optimization and standard ALMs.

Appendix B presents a self-contained proof for the exact penalty functions and a new characterized of the preimage of the exact penalty functions.

Appendix C completes the proof of growth properties of the indicator function and exact penalty function.

Appendix D details the proof of Proposition 1 and presents a simple example to illustrate the growth property.

Appendix E.2 completes the missing proofs in Theorem 3.

Appendix E.3 discusses the convergence of ALMs applied to the dual problem (D).

Appendix F presents further details of numerical experiments in Section 5.

A Optimization Background

Notations For clarity, we summarize common notations here. We use R n to denote the n -dimensional real space and S n to denote the space of real n × n symmetric matrices. The cone of nonnegative orthant is denoted as R n + = { x ∈ R n | x i ≥ 0 , ∀ i = 1 , . . . , n } . The cone of positive and negative semidefinite matrices are defined as S n + = { X ∈ S n | v T Xv ≥ 0 , ∀ v ∈ R n } and S n -= -S n + respectively. For vectors in R n , the inner product ⟨· , ·⟩ is defined as ⟨ x, y ⟩ = x T y and the normal ∥ · ∥ stands for the l 2 norm. For matrices in S n , the inner product ⟨· , ·⟩ is defined as ⟨ X,Y ⟩ = tr ( X T Y ) and the norm ∥ · ∥ op , ∥ · ∥ ∗ and ∥ · ∥ denote the operator two norm, nuclear norm, and Frobenious norm respectively. Given a set S ⊆ R n , the relative interior of S is denoted as relint( S ) , and δ S ( x ) is defined as the indicator function of S , i.e., δ S ( x ) = 0 if x ∈ S and δ S ( x ) = ∞ otherwise. The closed ball with a radius µ around the center x ∈ R n is written as B ( x, µ ) = { y ∈ R n | ∥ y -x ∥ ≤ µ } .

A.1 Standard notion of strict complementarity in SDPs

In this subsection, we review the standard algebraic notion definition for SDPs which is equivalent to Definition 2. A proof of the equivalence can be found in [28, Appendix B].

Definition A.1 (Strict complementarity for SDPs) . Consider the pair of primal and dual SDPs (P) and (D). We say the strict complementarity holds if there exists a pair of primal and dual solutions ( X ⋆ , y ⋆ , Z ⋆ ) satisfying

With the complementarity slackness ⟨ X ⋆ , Z ⋆ ⟩ = 0 , we have X ⋆ Z ⋆ = Z ⋆ X ⋆ = 0 . This means there exists an orthonormal matrix Q ∈ R n × n with Q T Q = I , such that

and λ i w i = 0 , i = 1 , . . . , n . Then, (A.1) implies that exactly one of the two conditions λ i = 0 and w i = 0 is true in (A.2). In other words, the eigenvalues of X ⋆ and Z ⋆ have a complementary sparsity pattern.

Remark A.1 (Dual strict complementarity in the general case) . The dual strict complementarity in Definition 2 is consistent with [28, Defnition 2]. A variant of dual strict complementarity in the general case is given in [34, Section 4], which requires that

where g ( y ) = ⟨ b, y ⟩ -δ S n + ( C -A ∗ ( y )) . Definition 2 is also consistent with (A.3). Indeed, let a dual optimal solution ( y ⋆ , Z ⋆ ) satisfy (A.3). Considering ∂g ( y ) = b + A ( N S n + ( C -A ∗ ( y ))) and using [51, Corollary 6.6.2 and Theorem 6.6], we have

which means there exists a point s ∈ relint( N S n + ( Z ⋆ )) such that 0 = b + A ( s ) . Equivalently, there exists X ⋆ = -s ∈ relint( N ( S n + ) ◦ ( -Z ⋆ )) (recall that ( S n + ) ◦ = -S n + ) such that A ( X ⋆ ) = b . This is equivalent to Definition 2. □

A.2 Augmented Lagrangian Methods (ALMs) for inequality constraints

We here review some classic results of ALM. Consider a constrained convex optimization problem

where X 0 ⊆ R n is a closed convex set and f, f i : E → ¯ R are proper closed convex functions. We associate each inequality constraint in (A.4) with a dual variable z i ≥ 0 , and the Lagrangian function for (A.4) is L 0 ( x, z ) = f ( x ) + ∑ m i =1 z i f i ( x ) . The corresponding dual function is

One way to define the augmented Lagrangian for (A.4) is to transform the inequality constraints into equality constraints [45, Chapter 4.7]. After some manipulation (see [45, Equation 4.79] or Appendix A.3), the classical augmented Lagrangian with parameter r > 0 for (A.4) reads as

where Π R m + ( · ) denotes the Euclidean projection onto R m + . Note that L r is differentiable in z [52]. Given an initial point z 0 ∈ R m and a sequence of positive scalars r 0 ↑ r ∞ < + ∞ , the inexact ALM generates a sequence of { x k } (primal variables) and { z k } (dual variables) as

The convergence of x k and z k depends on the inexactness in (A.7a). For this, the seminal work by Rockafellar [23] points out an important connection between PPM and ALM, summarized below.

Lemma A.1 ([23, Proposition 6]) . The dual iterate z k +1 in (A.7b) satisfies the following relationship

where g 0 is the Lagrangian dual function in (A.5) , and ¯ L r denotes an extended form of the augmented Lagrangian L r as ¯ L r ( x, z ) = L r ( x, z ) , if x ∈ X 0 , otherwise ¯ L r ( x, z ) = ∞ .

Note the subtle difference between Lemmas 4 and A.1. If (A.7a) is updated exactly, Lemma A.1 confirms that z k +1 = prox c k , -g 0 ( z k ) . In other words, the dual iterates of the ALM are the same as the iterates from PPM on the dual problem (A.5). Moreover, by controlling the inexactness (A.7a), the convergence of the dual variables z k can be naturally guaranteed from Lemma 1. Motivated by (a) and (b), the following two criteria are commonly used in (A.7a)

where { ϵ k } k ≥ 0 and { δ k } k ≥ 0 are two sequences of inexactness. Note that (A) and (B) guarantee the dual variables z k from ALM (A.7) satisfy (a) and (b) with the inexact PPM (3), respectively. Let S = argmax z ≥ 0 g 0 ( z ) denote the set of optimal dual solutions for (A.4). Combining Lemma A.1 with Lemma 1, the iterates from the inexact ALM (A.7) has the following convergence guarantees.

̸

Lemma A.2. Assume S = ∅ . Let { x k , z k } be a sequence generated by the inexact ALM (A.7) .

(a) If criterion (A) is used, then lim k →∞ z k = z ∞ ∈ S (i.e., z k converges to a dual optimal solution), and the primal sequence { x k } satisfies f i ( x k +1 ) ≤ ∥ z k -z k +1 ∥ /c k , i = 1 , . . . , m (primal feasibility), and ⟨ c, x k +1 ⟩ -p ⋆ ≤ ( ϵ 2 k + ∥ z k -z k +1 ∥ 2 ) / (2 c k ) (cost value gap).

(b) If both criteria (A) and (B) are used, and the dual function -g 0 in (A.5) further satisfies (QG) with a coefficient µ q > 0 , then there exists a ˆ k > 0 such that dist( z k +1 , S ) ≤ µ k · dist( z k , S ) , ∀ k ≥ ˆ k, where µ k = θ k +2 δ k 1 -δ k with θ k = 1 / √ c k µ q +1 < 1 .

The first asymptotic convergence for lim k →∞ z k = z ∞ ∈ S directly comes from Lemma 1, and the asymptotic convergence in terms of primal feasibility and cost value gap is taken from [23, Theorem 4]. The second linear convergence in Lemma A.2 is only guaranteed for the dual iterates z k , which is also implied by Lemma 1. However, the rate of the primal iterates x k is unclear even with the quadratic growth condition. With a much stronger condition on the Lagrangian, the classical result in [23, Theorem 5] can ensure linear convergence of x k .

A.3 Augmented Lagrange functions for inequality constraints

One straightforward way to define the augmented Lagrangian for the inequality form (A.4) is to first transform (A.4) into the following equivalent problem

Then the augmented Lagrangian for the above equality-constrained problem with a penalty term r > 0 is defined as

Note that L r is a decoupled quadratic function in terms of v i . Minimizing L r ( x, v, z ) with respect to v ≥ 0 with a fixed x and y , we derive the optimal v as

Upon plugging the optimal v in the L r with some arrangements, we arrive at the classical augmented Lagrangian for the problem with inequality constraints (A.4)

A.4 Bounded Linear regularity and regularity conditions

We review an important concept: bounded linear regularity of a collection of closed convex sets.

Definition A.2 ( Bounded linear regularity [53, Definition 5.6] ) . Let D 1 , . . . , D m ⊆ R n be closed convex set. Suppose D := D 1 ∩ · · · ∩ D m is nonempty. The collection { D 1 , . . . , D m } is called boundedly linear regular if for every bounded set V ⊆ R n , there exists a constant κ > 0 such that

Bounded linear regularity allows us to bound (up to some constant) the distance to the intersection D (which can be complicated) by the maximum of the distance to D i , which is often easier to compute. The following result shows a sufficient condition for bounded linear regularity.

Lemma A.3 ([54, Collorary 3]) . Let D 1 , . . . , D m ⊆ R n be closed convex set. Suppose that D 1 , . . . , D r ( r ≤ m ) are polyhedra. Then { D 1 , . . . , D m } is boundedly linear regular if

̸

where relint denotes the relative interior.

This result indicates that a collection of closed convex sets is boundedly linear regular if the relative interiors of non-polyhedral sets have a non-empty intersection with the rest of the polyhedra. Lemma A.3 plays a key role in the proof of Theorems 1 and 2. In particular, as we will see in Appendix D.1, the optimal solution set of (P), i.e., Ω P , can be characterized as

where X 0 = { X ∈ S n + | A ( X ) = b } . Thus, assuming dual strict complementarity (there exists X ⋆ ∈ relint(( ∂g ) -1 ( -Z ⋆ )) ), given a compact set U ⊆ S n containing X ⋆ , Lemma A.3 ensures the following holds

where α 1 and α 2 are two positive constants.

B Exact penalty forms of conic optimization

In this section, we characterize the connection between exact penalty functions and indicator functions (6). Most importantly, we give a simple proof for the exact penalty functions. The main result is stated in Proposition B.2. We first state the preimage characterization of subdifferential mapping of the exact penalty function in Proposition B.1, which is a more general result of the first part of Lemma 3 (i.e., (11)). The second part of Lemma 3 (i.e., (12)) is postponed to Appendix C.

B.1 Subdifferential of the exact penalty function

Proposition B.1. Let Z ∈ S n + and define

Then for all S ∈ S n + satisfying ρ > tr ( S ) , it holds that

One can see that (11) is a special case of Proposition B.1 with S = ¯ Z . We postpone the proof of Proposition B.1 to Appendix B.2. Be aware that (B.2) does not mean the mappings ( ∂l ) -1 and ( δ S n + ) -1 are the same for each point in the domain, while the equality only holds for certain points satisfying the relationship with the penalty coefficient ρ in (B.1). We use a simple example to illustrate this subtlety.

Example B.1. Consider l : R → R as l ( x ) = 2max { 0 , -x } which is the scalar version of the function in (B.1) with ρ = 2 . The subdifferential ∂l : R ⇒ R and ∂δ R + = N R + can be computed as

Thus, the preimages can be verified as

This shows that ( ∂l ) -1 ( -z ) = ( δ R + ) -1 ( -z ) holds only when z < 2 .

With the preimage characterization in Proposition B.1, the following result on the exact penalty functions follows easily by choosing the penalty term ρ correctly.

Proposition B.2 (Exact penalty function) . Assuming strong duality holds for (P) and (D) , (4) is equivalent to the primal SDP (P) if the function g in (4) is chosen to be the corresponding indicator function, i.e., g = δ S n + , or the exact penalty function

where ( y ⋆ , Z ⋆ ) ∈ Ω D is an optimal dual solution of (D) .

Proof. The case for the indicator function δ S n + is clear. We consider the case for the exact penalty function. Note the dual of (4) is

Let Ω P 2 and Ω D 2 denote the optimal solution of (4) and (B.4) respectively. First, it is clear that Ω D 2 ⊆ Ω D as there is an extra constraint in (B.4), compared with (D). Second, as the problem is convex, a pair of primal and dual solutions ( ˆ X ⋆ , ˆ y ⋆ , ˆ Z ⋆ ) solves (4) and (B.4) if and only if the following KKT system holds

Then the primal solution set can be written as

where X 0 = { X ∈ S n | A ( X ) = b } . Note that ( y ⋆ , Z ⋆ ) is an optimal solution to (B.4) as ( y ⋆ , Z ⋆ ) is feasible in (B.4) and ⟨ b, y ⋆ ⟩ ≤ max y,Z (B.4) ≤ d ⋆ = ⟨ b, y ⋆ ⟩ by construction. We can write Ω P 2 as

where the last equality uses ( ∂g ) -1 ( -Z ⋆ ) = ( ∂δ S n + ) -1 ( -Z ⋆ ) in (B.2) as tr ( Z ⋆ ) < ρ and Z ⋆ ∈ S n + . On the other hand, we can write Ω P as

It is clear that Ω P 2 = Ω P

Remark B.1 (Dual problem) . Note that Proposition B.2 only guarantees the cost value and solution sets of (4) and (P) to be the same, while the solution sets of (D) and the dual of (4) may differ, as seen in the proof of Proposition B.2. To fully equate (D) and the dual of (4), one can choose the penalty coefficient ρ as ρ > sup ( y ⋆ ,Z ⋆ ) ∈ Ω D tr ( Z ⋆ ) . This choice of ρ can guarantee every optimal solution in (D) is also an optimal solution in (B.4) as the extra constraint in (B.4) (compared with (D)) does not affect the solution set. Combining this with the fact Ω D 2 ⊆ Ω D , we conclude that Ω D = Ω D 2 .

B.2 Proof for Proposition B.1

The following proposition showcases the subdifferential calculation of the function l ( X ) = ρ max { 0 , λ max ( -X ) } with ρ ≥ 0 . It is a standard result of the subdifferential of the maximal eigenvalue of symmetric matrices and standard subdifferential calculus.

Proposition B.3 ([55, Theorem 2] and [45, Theorem 2.87]) . Let ρ ≥ 0 and X ∈ S n with λ max ( -X ) having t multiplicity . The subdifferential of l ( X ) = ρ max { 0 , λ max ( -X ) } is characterized by

where columns of P ∈ R n × t are orthonormal eigenvectors corresponding to λ max ( -X ) .

Proof. The case ( ∂l ) -1 (0) = S n + can be easily observed from the first two cases in (B.5). We then consider the case Z ∈ S n + \ { 0 } . Given a X ∈ S n , we use the matrix P X ∈ R n × t to denote the orthonormal eigenvectors corresponding to λ max ( -X ) where t is the multiplicity of λ max ( -X ) . It follows that

where the domain S n + \ S n ++ and the strict inequlaity tr ( S ) < 1 in ( a ) is due to the choice ρ > tr ( Z ) , and ( c ) is a common form of N S n -( -Z ) [28, Complementary face in Section 3.3]. We argue ( b ) is correct by verifying the following two sets are equivalent.

( D 1 ) ⇒ ( D 2 ) : Let X ∈ D 1 . Write X = P α 0 P T α + P β Λ β P T β with P α ∈ R n × t and P β ∈ R n × ( n -t ) . As span( P 1 ) ⊆ span( P α ) , we have span( P β ) ⊆ span( P 2 ) . Thus, there exists U ∈ R n × ( n -r ) such that U T U = I n -r and P β = P 2 U . We then have

This means X ∈ D 2 .

( D 2 ) ⇒ ( D 1 ) : Let X ∈ D 2 . It is clear that span( P 1 ) is in the eigenspace of 0 of the matrix X . Choosing S = Λ 1 ρ ∈ S r ++ , we can recover Z as

Also, tr ( S ) = tr (Λ 1 ) /ρ < 1 , so we conclude X ∈ D 1 .

C Growth properties of indicator and exact penalty functions in Lemma 3

One key step in the proof of Theorems 1 and 2 is the growth properties of the indicator function or the penalty function (10), summarized in (12) in Lemma 3. In this section, we complete its proof details. We first establish a lower bound of 〈 ¯ Z,X 〉 in terms of the distance dist( X, N ( S n + ) ◦ ( -¯ Z )) . Our proof uses some techniques in [27].

Lemma C.1. Let ¯ X, ¯ Z ∈ S n + satisfying 〈 ¯ X, ¯ Z 〉 = 0 (i.e., both ¯ X and ¯ Z are on the boundary unless one of them is zero). For any positive µ ∈ (0 , ∞ ) , there exists a constant κ > 0 such that

where B ( ¯ X,µ ) = { X ∈ S n | ∥ X -¯ X ∥ ≤ µ } denotes a closed ball with radius µ and center ¯ X .

Note that (C.1) is the reformulation of

Indeed, (C.2) is the same as (12) in Lemma 3 when choosing l = δ S n + (recall that we have δ S n + ( ¯ X ) = 0 , and the relationship in (11)). Therefore, Lemma 3 for the indicator function l = δ S n + is a direct consequence of Lemma C.1. The proof of Lemma C.1 is given in Appendix C.1. In Appendix C.2, we will use Lemma C.1 to prove the growth property for the exact penalty function l in (10).

C.1 Proof of Lemma C.1

We only need to consider the case where Z = 0 , since the case Z = 0 is true by observing that

̸

We first introduce the following useful inequality for positive semidefinite matrices.

We now proceed with the proof. Fix a µ > 0 . Consider ¯ X, ¯ Z ∈ S n + and 〈 ¯ X, ¯ Z 〉 = 0 . Let X ∈ B ( ¯ X,µ ) ∩ S n + and write

where Λ 1 ∈ S r + is a diagonal matrix containing the positive eigenvalues of ¯ Z . Let P = [ P 1 P 2 ] be the orthonormal eigenvectors matrix. The projection of X onto N ( S n + ) ◦ ( -¯ Z ) can be verified as

Hence, the distance dist 2 ( X, N ( S n + ) ◦ ( -¯ Z )) can be computed as

where the second equality uses the unitary invariance. It remains to estimate the terms ∥ P T 1 XP 1 ∥ 2 and ∥ P T 1 XP 2 ∥ 2 .

By the assumption X ∈ B ( ¯ X,µ ) ∩ S n + and unitary invariance, we have

where the last inequality uses the fact that P T 1 ¯ XP 1 = 0 as 〈 ¯ X, ¯ Z 〉 = 0 and ¯ X, ¯ Z ∈ S n + , and the relationship ∥ · ∥ ≥ ∥ · ∥ op . It follows that

where the second inequality uses the generalized Cauchy-Schwarz inequality ( ∥· ∥ ∗ and ∥· ∥ op norms are dual to each other; we have | ⟨ X,Z ⟩ | ≤ ∥ X ∥ ∗ ∥ Z ∥ op , ∀ X,Z ∈ S n ), and the last inequality uses (C.4) and ∥ P T 1 XP 1 ∥ ∗ = tr ( P T 1 XP 1 ) as X ∈ S n + .

On the other hand, applying Lemma C.2 on P T XP yields

where the last inequality is due to (C.4). Moreover, we have

where λ min (Λ 1 ) > 0 . Putting (C.3), (C.5) and (C.6) together and choosing κ = λ min (Λ 1 ) 3 µ +2 ∥ ¯ X ∥ gives us

where the last inequality is due to (C.7). This completes the proof.

C.2 Growth property of the exact penalty function l in (10)

Throughout this section, we fix a pair of matrices ( ¯ X, ¯ Z ) ∈ S n + × S n + satisfying 〈 ¯ X, ¯ Z 〉 = 0 . We need to prove for any positive value µ ∈ (0 , ∞ ) , there exists a positive constant κ > 0 such that

Note that it is equivalent to showing that there exists a κ > 0 such that

as l ( ¯ X ) = 0 and 〈 ¯ Z, ¯ X 〉 = 0 by assumption. This result (C.9) has already applied a non-trivial fact ( ∂l ) -1 ( -¯ Z ) = ( N S n + ) -1 ( -¯ Z ) = N ( S n + ) ◦ ( -¯ Z ) for the exact penalty function (10), which has been established in Proposition B.1.

In the following, we will bound the terms R 1 and R 2 in (C.9). The term R 2 will rely on Lemma C.1. One key difference between (C.9) and (C.2) is that the penalty function l ( X ) < ∞ when X / ∈ S n + , while δ S n + ( X ) = ∞ , ∀ X / ∈ S n + .

Let us fix a µ > 0 , consider X ∈ B ( ¯ X,µ ) , and write ρ = δ + tr ( ¯ Z ) . We first consider the case when ¯ Z = 0 . Note that 〈 ¯ Z,X 〉 = 0 and N ( S n + ) ◦ (0) = S n + . The distance dist( X, N ( S n + ) ◦ (0)) can be upper bounded as

Choosing κ = ρ nµ yields

̸

We then consider the case ¯ Z = 0 .

We first bound R 1 = 〈 -¯ Z,X 〉 :

where the inequality applies the general Cauchy-Schwarz inequality ( ∥ · ∥ ∗ and ∥ · ∥ op norms are dual to each other; we have | ⟨ X,Z ⟩ | ≤ ∥ X ∥ ∗ ∥ Z ∥ op , ∀ X,Z ∈ S n ).

We then bound R 2 = dist 2 ( X, N ( S n + ) ◦ ( -¯ Z )) : Let Y be the projection of Π S n + ( X ) on to N ( S n + ) ◦ ( -¯ Z ) , i.e., Y = argmin V ∈N ( S n + ) ◦ ( -¯ Z ) ∥ Π S n + ( X ) -V ∥ . We then have

The first term ∥ X -Π S n + ( X ) ∥ 2 can be bounded by

where the last inequality comes from | λ min ( X ) -λ min ( ¯ X ) | = | λ max ( -X ) | ≤ µ as λ min ( ¯ X ) = 0 and ∥ X -¯ X ∥ ≤ µ.

The second term ∥ Π S n + ( X ) -Y ∥ 2 can be bounded by Lemma C.1, i.e., there is a κ ′ > 0 such that

Putting everything together and choosing κ = min { δ 2 nµ , κ ′ 2 } yields

D Growth growth under strict complementarity

As mentioned in the main text, the key step to prove Theorems 1 and 2 is to establish the growth property (12) in Lemma 3 which we have proved in Appendix C. Once Lemma 3 is established, with strict complementarity assumption, we can deduce Proposition 1. In this appendix, we first provide proof for Proposition 1 in Appendix D.1 and use a simple example to illustrate the quadratic growth property in Appendix D.2.

D.1 Proof of Proposition 1

The following proof is adapted from [27]. We first prove the primal case. Fix a µ > 0 and X ⋆ ∈ Ω P . As the problem is convex, a pair of primal and dual solutions ( ˆ X ⋆ , ˆ y ⋆ , ˆ Z ⋆ ) solves (4) and its dual problem if and only if the following KKT system holds

Combined with the fact that any primal and dual solution forms a pair, the optimal primal solution Ω P can be characterized as

where X 0 = { X ∈ S n | A ( X ) = b } . Let ( ¯ X ⋆ , ¯ y ⋆ , ¯ Z ⋆ ) be a pair of primal and dual solutions satisfying ¯ X ⋆ ∈ relint(( ∂g ) -1 ( -¯ Z ⋆ )) (strict complementarity assumption). There exist α 1 , κ 1 , κ 2 > 0 such that for all X ∈ B ( X ⋆ , µ ) the following holds

where the first inequality applies Lemma A.3, the second inequality applies ( A + B ) 2 ≤ 2 A 2 +2 B 2 for all A,B ≥ 0 , the third inequality uses the fact that X 0 is an affine space. On the other hand, we know 〈 ˆ X ⋆ , ¯ Z ⋆ 〉 = 0 for all ˆ X ⋆ ∈ Ω P naturally, so Lemma 3 can be applied here on Z ⋆ and ¯ Z ⋆ (the proof of Lemma 3 is provided in Appendix C.2). For all X ∈ B ( X ⋆ , µ ) it holds that

where the first inequality uses the growth property on g for X ⋆ and ¯ Z ⋆ (12), the second inequality uses (D.1), the third inequality uses the definition of the adjoint operator and Cauchy-Schwarz inequality, and the last inequality uses the fact that X is in a bounded set B ( X ⋆ , µ ) so there exists some γ ≥ 0 such that κ ∥A ( X ) -b ∥ ≤ γ . This completes the proof.

We move on to prove the dual case. Similarly, we let ( y ⋆ , Z ⋆ ) ∈ Ω D and µ > 0 and characterize the optimal dual solution as

where Z 0 = { ( y, Z ) ∈ R m × S n | C -A ∗ ( y ) = Z } . Let ( ¯ X ⋆ , ¯ y ⋆ , ¯ Z ⋆ ) be a pair of primal and dual solution satisfying ¯ Z ⋆ ∈ relint(( ∂h ) -1 ( -¯ X ⋆ )) . Note that

Therefore, the existence of such a pair of strict complementarity solution implies

̸

Hence, following the same argument to (D.1), we know there exists a constant κ 3 > 0 such that

for all ( y, Z ) ∈ B (( y ⋆ , Z ⋆ ) , µ ) . Also, as 〈 ¯ X ⋆ , Z ⋆ 〉 = 0 , we have

where the first inequality uses the growth property on h , the second equality rewrites ⟨ b, y ⟩ = 〈 ¯ X ⋆ , A ∗ ( y ) 〉 and Z ⋆ = C -A ∗ ( y ⋆ ) , and the last inequality uses the fact that ( y, Z ) is in a bounded set B (( y ⋆ , Z ⋆ ) , µ ) so there exists a γ ≥ 0 such that κ ∥ C -A ∗ ( y ) -Z ∥ ≤ γ .

D.2 Illustration of quadratic growth

In this subsection, we use a simple example to illustrate the quadratic growth property discussed in (9). In particular, we choose the exact penalty formulation for dual SDP (5) (the proof for the exact penalty function can be found in Appendix B). Moreover, the following example shows that even a simple 2 × 2 SDP can not have sharp growth in the objective function.

Example D.1. Consider a SDP (P) and (D) with the problem data

It can be verified that both primal and dual SDP have a unique solution

Therefore, the optimal cost value is 0 . Moreover, it also satisfies strict complementarity as rank( X ⋆ )+ rank( Z ⋆ ) = 1 + 1 = 2 (see Definition A.1). (8b) and (9b) then ensure the following exact penalized formulation having quadratic growth property

where ρ can be chosen as any number satisfying ρ > tr ( Z ⋆ ) = 2 . Let S = { (0 , 0) } be the optimal dual solution set and f ⋆ = 0 be the optimal cost value. Figure 2a shows the landscape of the function f with ρ = 4 and verifies that the function value f ( y ) is growing at least quadratically away from the distance to the solution set S = { (0 , 0) } with the quadratic constant κ = 0 . 3 . In Figure 2a, the yellow area indicates the part where only the linear component of f ( y ) , i.e. b T y , is active while the nonlinear penalty component ρ max { 0 , λ max ( A ∗ ( y ) -C ) } is 0 , and the blue region is the part where both the linear and nonlinear components of f ( y ) are active. The black line in Figure 2a is the boundary where λ max ( A ∗ ( y ) -C ) = 0 which can be characterized by the nonlinear equation y 1 y 2 -y 1 -y 2 = 0 with y 1 < 1 . This nonlinear direction also indicates that sharp growth of f is impossible. Indeed, let y 1 ∈ [0 , 1) and y 2 = y 1 y 1 -1 . Note that this choice of y satisfies the nonlinear equation y 1 y 2 -y 1 -y 2 = 0 and thus only the linear part -b T y in f ( y ) is in force. We then have

On the other hand, the distance to the solution set can be verified as

It becomes clear that it is impossible to have a constant κ > 0 such that

(a)

3-dimensional view

-1

-0.5

0

0.5

0

0.1

0.2

0.3

0.4

0.5

(b)

Sectional view

bTy+pmax{^max(A*y -C),0}

k·Dist²(y,S)

1.5

1

0.5

0

-1

1

-0.5

0.5

0

0

0.5

-0.5

y1

-1

y2

as the quadratic therm y 2 1 will dominant when y 1 is small, let alone the sharp growth κ · dist( y, S ) ≤ f ( y ) -f ⋆ . Moreover, the quadratic growth of indicator function formulation -b T y + δ S 2 + ( C -A ∗ ( y )) has also been verified as we only need to consider the linear component -b T y , which has already been confirmed in the previous case. Lastly, it is worth pointing out that the sublevel sets of the functions f and -b T y + δ S 2 + ( C - A ∗ ( y )) are different. Indeed, given a finite value β > 0 , the sublevel set { y ∈ R 2 | f ( y ) ≤ β } contains both parts of yellow and blue regions in Figure 2b. In contrast, the sublevel set { y ∈ R 2 | -b T y + δ S 2 + ( C -A ∗ ( y )) ≤ β } only contains the linear part in Figure 2b.

E Proofs and supplemental discussions in Section 4

In this section, we complete the missing proof in Section 4 and discuss the ALM applied to (D) with its corresponding convergence properties. This section is divided into three subsections. Appendix E.1 completes the proof of Proposition 2. Appendix E.2 finishes the proof of Theorem 3. Appendix E.3 discusses the ALM applied to (D).

E.1 Proof of Proposition 2

The proof of Proposition 2 requires a nice characterization of the ordinal dual function and the augmented dual function defined as

where L r is the augmented Lagrangian function (14), recall that we write w = ( y, Z ) .

Proposition E.1 ([38, Theorem 3.2]) . Let r > 0 . For all w ∈ R m × S n , it holds that

where g 0 is the dual function in (13) .

Proposition E.1 not only shows the augmented dual function g r is g 0 with a prior but also shows g r is continuously differentiable. In comparison, the ordinary dual function g 0 can be highly nonsmooth. We are ready to prove Proposition 2.

Part (a) of Proposition 2 comes directly from the PPM convergence Lemma 1 as Lemma 4 and the stopping criteria (A ′ ) naturally imply the dual iterate w k +1 satisfies ∥ w k +1 -prox r k , -g 0 ( w k ) ∥ ≤ ϵ k and ∑ ∞ k =1 ϵ k < ∞ .

The proof of (b) in Proposition 2 is in fact straightforward. Firstly, by Moreau decomposition [49, Exercise 12.22] and (15c), we have

Diving both sides by r k , we conclude X k +1 +( Z k +1 -Z k ) /r k ∈ S n + . Therefore, it follows that

Secondly, ∥A ( X k +1 ) -b ∥ = ∥ y k -y k +1 ∥ /r k comes directly from the update (15b). Lastly, note that

On the other hand, we have

where u ⋆ is the point that achieves the maximum in Proposition E.1 and the last inequality uses weakly duality. Therefore, we have

Since the convergence of { w k } is guaranteed in (a), the quantities dist( X k +1 , S n + ) , ∥A ( X k +1 ) -b ∥ , and ⟨ C, X k +1 ⟩ -p ⋆ also convergence to zero.

Part (c) is a direct consequence of part (b) and the fact that Ω P is bounded if and only if the set { X ∈ S n | dist( X, S n + ) ≤ γ 1 , ∥A ( X ) -b ∥ ≤ γ 2 , ⟨ C, X ⟩ -p ⋆ ≤ γ 3 } is bounded for any γ ∈ R 3 [23, page 110].

E.2 Proof of Theorem 3

In this subsection, we provide a complete proof of part (a) in Theorem 3. The proof is motivated by [26, Theorem 1] which focuses on dual SDPs only. To facilitate proof, we first introduce a standard result in proximal mapping that upper bounds the step length of the proximal step by the distance to the optimal solution set, which can be found in [42, Proposition 1] by setting z = x and z ′ = Π S ( x ) . We give a simple proof below.

̸

Proposition E.2. Let f : R n → ¯ R be a convex function. Denote S ⊆ R n as the set of minimizers of f , i.e., S = argmin x ∈ R n f ( x ) . Suppose S = 0 . Given a point x ∈ R n and a constant α > 0 , the proximal mapping holds that

Proof. From the optimality condition of proximal mapping, we have

which completes the proof.

Using Proposition E.2 and (B ′ ), we arrive at the following key inequality to show the linear convergence of KKT residuals. Proposition 2. It bounds the step length of the dual update by the current distance to the solution set.

Proposition E.3. [26, Lemma 3] Let { X k , w k } be the sequence generated by (15) under (B ′ ) . Then for all k ≥ 1 with δ k < 1 , it holds that

Proof. From the triangle inequality and the stopping criterion (B ′ ), we have

Rearranging terms and using Proposition E.2 yields

This completes the proof.

We are ready to start the proof of part (a) here. As dual strict complementary holds, the negative of the dual function (13), i.e., -g 0 , satisfies quadratic growth (9b). Using Lemma 4 and stopping criteria (B ′ ), we have ∥ w k +1 -prox r k , -g 0 ( w k ) ∥ ≤ δ k ∥ w k +1 -w k ∥ . Therefore, the linear reduction on dist( w k , Ω D ) comes directly from the PPM convergence Lemma 1, i.e., there exists a ˆ k 1 such that for all k ≥ ˆ k 1 , it holds that

We move to show the bound for residuals dist( X k +1 , S n + ) , ∥A ( X k +1 ) -b ∥ , and ⟨ C, X k +1 ⟩ -p ⋆ . Specifically, when δ k < 1 , it holds that

where we use part (b) in Proposition 2, ∥ y k -y k +1 ∥ ≤ ∥ w k -w k +1 ∥ , ∥ Z k -Z k +1 ∥ ≤ ∥ w k -w k +1 ∥ , and Proposition E.3. Again, using part (b) in Propositions 2 and E.3, when δ k < 1 , we have

Since δ k → 0 , there exists a ˆ k 2 such that δ k < 1 for all k ≥ ˆ k 2 . Taking ˆ k = max { ˆ k 1 , ˆ k 2 } completes the proof for part (a) with the constants

We move on to part (b). By Theorem 2, we know that the error bound (7b) holds as dual strict complementary holds, i.e., for any bounded set U ⊆ S n containing any X ⋆ ∈ Ω P , there exist constants α 1 , α 2 , α 3 > 0 such that

From the assumption that Ω P is bounded, Proposition 2 confirms that the sequence { X k } is bounded. Thus, we can choose a bounded set U ⊇ Ω P ⋃ ∪ k ≥ 1 { X k } . Combing (E.2) with (16), we have

Dividing both sides by α 3 leads to the desired result in part (b). This completes the proof.

E.3 ALMs for dual conic program (D)

In aiming to offer a comprehensive understanding of ALMs for solving (P) and (D), in this subsection, we provide the algorithm of ALM applied to solving the dual problem (D) along with the convergence analysis. This can be considered as an extension of the work in [26] and a counterpart to the ALM for solving (P) discussed in Section 4.

We first reformulate (D) as the equivalent minimization problem

Wethen introduce a dual variable X ∈ S n + and define the ordinary Lagrangian function as L 0 ( y, X ) = -⟨ b, y ⟩ + ⟨ X,C -A ∗ ( y ) ⟩ . The corresponding dual function and dual problem become

Then, the augmented Lagrangian with parameter r > 0 , analogous to (14), is

Given an initial point X 0 ∈ S n and a sequence of positive scalars r k ↑ r ∞ , the ALM generates a sequence of { y k } and { X k } following

Many of the same properties discussed in Section 4 hold. We will provide the precise statement in the remainder of this section. Similarly, we can consider the analogous stopping criteria of (A ′ ) and (B ′ ).

̸

Proposition E.4. Consider (P) and (D) . Assume strong duality holds and Ω P = ∅ . Let { y k , X k } be a sequence from the ALM (E.4) under (A ′ D ) and write Z k = C -A ∗ ( y k ) for all k ≥ 0 . The following statements hold.

(a) The dual sequence X k is bounded. Further, lim k →∞ X k = X ∞ ∈ Ω P (i.e., the whole sequence converges to a primal optimal solution).

(b) The dual feasibility and cost value gap satisfy

(c) If the solution set Ω D in (1b) is nonempty and bounded, then the primal sequence y k is also bounded, and all of its cluster points belongs to Ω D .

Proof. The proof of many of these statements is a change of the notations in the proof in Proposition 2.

Part (a) comes directly from the PPM convergence by noting Lemma 4 and the stopping criterion (A ′ D ) imply the dual iterate X k +1 satisfies ∥ X k +1 -prox r k , -g 0 ( X k ) ∥ ≤ ϵ k .

By the Moreau decomposition [49, Exercise 12.22], we have

Thus, Z k +1 + X k +1 -X k r k ∈ S n + . Therefore,

From the definition of the augmented Lagrangian function, we have

On the other hand, we know

where U ⋆ denoted as the point that achieves the maximum in the identity [38, Theorem 3.2]

and the last inequality comes from the definition of the dual problem in (E.3). Thus,

Part (c) is a consequence of part (b) and the fact that Ω D is bounded if and only if the set { ( y, Z ) ∈ R m × S n | d ⋆ -⟨ b, y ⟩ ≤ γ 1 , dist( Z, S n + ) ≤ γ 2 , ∥ C -A ∗ ( y ) -Z ∥ ≤ γ 3 } is bounded for any γ ∈ R 3 [23, page 110].

̸

Theorem E.1 (Linear convergences) . Consider primal and dual SDPs (P) and (D) . Assume strong duality and strict complementarity holds (implying Ω P = ∅ and Ω D = ∅ ). Let { y k , X k } be a sequence from the ALM (E.4) under (A ′ D ) and (B ′ D ) and write Z k = C -A ∗ ( y k ) for all k ≥ 0 . The following statements hold.

̸

(a) There exists constants ˆ k ≥ 0 , 0 < µ k < 1 and µ ′ k , µ ′′ k > 0 such that for all k ≥ ˆ k ,

(b) If Ω D is bounded, then the primal sequence { y k } also converges linearly to Ω D , i.e., there is a constant ˆ k ≥ 0 such that for all k ≥ ˆ k ,

Proof. Similar to Proposition E.3, if (B ′ D ) is used, we have

Therefore, there exists a ˆ k ≥ 0 such that ∀ k ≥ ˆ k, δ k < 1 and

where the first inequality uses part (b). Then, once again using (E.6) and part (b), we have

If strict complementarity holds, Theorem 1 (with g in (5) as an indicator function) guarantees that -g 0 in (E.3) satisfies quadratic growth in (9b). Thus, the linear convergence of dist( X k , Ω P ) comes from the PPM convergence Lemma 1, i.e., there exists a ˆ k 1 such that for all k ≥ ˆ k 1 , it holds that

Since δ k → 0 , there exists a ˆ k 2 such that δ k < 1 for all k ≥ ˆ k 2 . Taking ˆ k = max { ˆ k 1 , ˆ k 2 } completes the proof for part (a) with the constants

On the other hand, Theorem 1 also guarantees that the error bound (8b) holds, i.e., for any bounded set V ⊇ Ω D , there exist constants α 1 , α 2 , α 3 > 0 such that

By the design of (E.4), we always have ∥ C -A ∗ ( y k ) -Z k ∥ = 0 . By Proposition E.4 - part (c), the sequence { y k , Z k } is bounded. Thus, choosing a bounded set V ⊇ Ω D ⋃ ∪ k ≥ 1 { y k , Z k } and combing (E.7) with (E.5), we have

Dividing both sides by α 3 leads to the desired result. This completes the proof.

F Details of the numerical experiments

In Section 5, we consider the classical SDP relaxation of the Mac-Cut problem and the linear SVM. In this section, along with another popular signal processing problem - Lasso, we detail the problem formulation and report the numerical performance of ALM applied to those problems.

The SDP relaxation of Max-Cut:

where C ∈ S n is the problem data representing a weighted undirected graph, 1 ∈ R n is an all one vector, and Diag( X ) = [ X 11 , · · · , X nn ] T . This type of problem has been shown to satisfy strict complementarity and has unique primal and dual solutions for almost all data matrix C [41, Corollary 2.3].

Linear SVM:

where a i ∈ R n is the feature of the i -th data point, b i ∈ {-1 , 1 } is the corresponding label, and λ > 0 is a constant. Equivalently, the problem can reformulated in the following standard conic form

where A = [ a 1 , · · · , a m ] T ∈ R m × n and Diag( b ) ∈ S m denotes the diagonal matrix with the diagonal elements being b .

Lasso:

where A ∈ R m × n , b ∈ R m are problem data and λ > 0 is the weighted parameter balancing the sparsity of the solution and the accuracy of the linear solution. Equivalently, the problem can be rewritten in the conic form

We observe that, in all cases, ALM enjoys linear convergence in the primal cost value gap, dual cost value gap, and KKT residuals to the accuracy of at least 10 -5 , which is consistent with our theoretical findings in Theorem 3. We believe the oscillated or flattening behavior that appears in the tail (when the iterates are close to the solution set) is due to the inaccuracy of the subproblem solver and computational errors. The numerical result also indicates that when the augmented term r increases, ALM favors the decrease of the feasibility more, leading to faster convergence in the beginning phase. However, a large r may not lead to the best convergence in the long term.

Note that the projection term ∥ Π S n + ( Z -rX ) ∥ 2 in the subproblem (15a) can not be directly modeled by Yalmip. Therefore, we reformulate the problem as the following

where the first equality drops two independent terms and uses -S n + = S n -and ∥ Π S n + ( Y ) ∥ = ∥ -Π ( -S n + ) ( -Y ) ∥ = ∥ Π ( S n -) ( -Y ) ∥ for all Y ∈ S n , and the last equality uses min U ∈ S n + ∥ Y -U ∥ = ∥ Y -Π S n + ( Y ) ∥ = ∥ Π S n -( Y ) ∥ for all Y ∈ S n .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have clearly and accurately stated the contributions and scope of this paper in the abstract and introduction. We have also made proper assumptions and presented a clear comparison with results in the existing literature.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: We have clearly stated the assumptions on conic optimization in order to guarantee the quadratic growth properties and the linear convergence of inexact ALM.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the theorems, formulas, and proofs in the paper are numbered and crossreferenced. We have provided completed proofs for all theorems/lemmas. In the main text, a proof sketch is provided, and the details can be found in the supplementary material.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided a separated section in the supplementary material for experimental details.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We plan to make all code open access after the review period.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have a separate section in the supplementary materials for experimental setting.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided the type of computer and memory for reproducing our experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have followed closely the Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer:[Yes]

Justification: We have discussed the positive societal impacts from efficient algorithms to solve conic optimization. We do not see negative societal impacts of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents new theoretical results on the convergence of primal iterates in inexact augmented Lagrangian methods (ALMs) for conic optimization. The idea is to use strict complementarity (which is a standard assumption in conic optimization) to establish quadratic growth and error bound conditions for the primal and dual programs. These bounds are then used to show linear convergence for both the primal and dual iterates in inexact ALMs. The linear convergence of the primal iterates is shown in (brief) experiments.\n\nSTRENGTHS:\n* The work provides a thorough introduction to the problem background \n* The theory is presented rigorously!\n\nWEAKNESSES:\nI found the main weakness of the paper to be in its presentation of results.\n\n* The paper takes too long to reach the main results. Perhaps an informal version of Theorem 3 could be stated in the introduction?\n\n* There is a lot of notation/specific terminology in the paper, which makes it difficult to follow. One suggestion to reduce the complexity would be to focus on a particular problem class (LP, SOCP, or SDP) in the main paper. For example, if focusing on LPs, the authors could move the discussion of strict complementarity of LPs to earlier in the paper, which would give more intuition for the assumptions in the paper. Furthermore, the theorems in the main paper could be simplified to address LPs, which would make the results easier to understand for a wider audience. The general results for LPs, SOCPs, and SDPs could then be presented in the appendix.\n\n* I would have liked the paper to expound upon how their work relates to machine learning (outside of some brief citations in the introduction). Ideally, this could be done by showing linear convergence of primal iterates on a conic program from machine learning. One idea could be to solve an SVM problem (QP, which is a special case of SDP). \n\n* While Figure 1 suggests that primal iterates are converging linearly, I would like to see more iterations (say 20) in the plots to really verify this claim.\n\n* I found it hard to follow how the author’s results compare to previous convergence results in the literature. I’d suggest providing a more in-depth comparison in lines 341-349.\n\n* The paper takes too long to discuss inexact augmented Lagrangian methods (these are not discussed in-depth until page 7). This makes the flow of the paper confusing — I’d suggest covering inexact ALMs closer to the introduction.\n\nMinor comments: \n\n* Discussion of scalable first-order methods (line 28) should cite PDLP [1, 2]\n\n* “Lipschitz” is misspelled on lines 100 and 103\n\n[1] Applegate, David, Mateo Diaz, Oliver Hinder, Haihao Lu, Miles Lubin, Brendan ODonoghue, and Warren Schudy. 2021. “Practical Large-Scale Linear Programming Using Primal-Dual Hybrid Gradient.” In Advances in Neural Information Processing Systems, edited by M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan, 34:20243–57. Curran Associates, Inc.\n\n[2] Applegate, D., Hinder, O., Lu, H. et al. Faster first-order primal-dual methods for linear programming using restarts and sharpness. Math. Program. 201, 133–184 (2023).\n\nQUESTIONS:\n* Are there problems for which the dual iterates can converge linearly, but the primal iterates converge sublinearly? If so, could this be demonstrated in an experiment?\n\n* Line 265: What exactly is meant by “nice self-dual structure”?\n\nLIMITATIONS:\nPlease see “Weaknesses”", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents the convergence rate of the primal iterates of the augmented Lagrangian Methods (ALMs) which are widely employed in solving constrained optimizations. The authors develop new quadratic growth and error bound properties for primal and dual conic programs under the standard strict complementarity condition and then reveal that both primal and dual iterates of the ALMs converge linearly contingent upon the assumption of strict complementarity and a bounded solution set. The specific contributions include:\n1. Under the standard strict complementarity assumption, the quadratic growth and error bound for both primal and dual conic programs (P) and (D) over any compact set containing an optimal solution (see Theorems 1 and 2) are established.  \n2. A new characterization of the preimage of the subdifferential of the exact penalty functions is unveiled. \n3. They provide new and simple proof for the growth properties in the exact penalty functions and clarify some subtle differences in constructing exact penalty functions.\n4. They show the linear convergence for both the primal and dual iterates of  ALM to solve either the primal or dual conic programs.\n\nSTRENGTHS:\nOriginality: The paper introduces novel quadratic growth and error bound properties for primal and dual conic programs, addressing long-standing challenges in the field. The development of symmetric inexact ALMs for primal and dual problems is a creative extension of existing methods.\n\nQuality: The paper is of high quality, featuring a robust theoretical framework and rigorous mathematical proofs.\n\nClarity: The paper is clearly written and well-organized.\n\nSignificance: By addressing open questions and providing new theoretical insights, the work could influence future research.\n\nWEAKNESSES:\n1. The authors claim in Remark 1 that the results in Theorems 1 and 2 are more general and unified compared with known results since they allow for any compact set $\\mathcal{U}$. However, let $\\mathcal{U}=\\mathcal{U}_1\\cap \\mathcal{U}_2$ where $\\mathcal{U}_1$ and $\\mathcal{U}_2$  are compact, and there is not any optimal solution in $\\mathcal{U}_1$. If it is known that the error bound is constrained with a pair $\\gamma, \\kappa$ for $x\\in\\mathcal{U}_2$. For $x\\in\\mathcal{U}_1$, it is natural to get a $\\kappa$ through the minimizatin of the fraction $$\\frac{f(x)-p^*+\\gamma\\|\\mathcal{A}(x)-b\\|}{dist^p(x,\\Omega)}$$ since $\\mathcal{U}_1$ is compact and $dist^p(x,\\Omega)>0$. We can just take the minimizer of two $\\kappa$(s). The advantages of this error bound form in this paper is doubtable.\n2. Aftrer Definition 2, the authors claim that the notion of strict complementarity is not restrictive, and it holds generically for conic programs. After that,  the authors also presented: that it has been revealed that many structured SDPs from practical applications also satisfy strict complementarity. There are contradictions and misunderstandings in these two assertions.\n\nQUESTIONS:\n1. Lines 184-185: If the linear constraints $\\mathcal{A}(x)=b$ is empty, how to define the exact penalty function?\n2. Line 226: Why is there $\\mathbb{F}\\times \\mathbb{E}$ instead of $\\mathbb{F}$?\n3. Please consider citing the paper \"Rockafellar, R.T. Convergence of Augmented Lagrangian Methods in Extensions Beyond Nonlinear Programming. Math. Program. 199, 375–420 (2023). https://doi.org/10.1007/s10107-022-01832-5,\" and compare it with your work.\n\nTypos:\n1. Line 238: Should (8b) be replaced by (8a)?\n2. The notation $\\mathbb{R}^n$ should be changed to $\\mathbb{R}_{+}^n$.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops Inexact Augmented Lagrangian methods (ALM) for conic optimization problems with focus on Linear Programming (LP) or the non-negative orthant cone, Second-Order Cone (SOCP) problems and Semidefinite programming (SDP) given a linear objective. Under assumptions of strict complementarity and strong duality, the authors show local linear convergence for these problems by first showing quadratic growth property within a compact subset around the optimal solution for both the primal and dual programs and then leveraging the well-established concept that running ALM for the primal is equivalent to running Proximal Point Method (PPM) for the dual. The experiments show linear convergence for the SDP relaxation for the MaxCut problem, the SDP relaxation for matrix completion, and the moment relaxation of binary quadratic program (BQP) given the following metrics: primal distance, dual cost gap and the KKT-based residuals.\n\nSTRENGTHS:\nThe biggest strength of the paper is the theoretical contribution to understanding of local linear convergence of inexact ALM methods for a variety of conic problems. Previous works have only shown local linear convergence of dual iterates but this work shows local  linear convergence for both primal and dual iterates. The authors provide a thorough theoretical discussion and provide all the proofs in the appendix section.\n\nWEAKNESSES:\nOne weakness of the paper is the lack of characterization of how inexact evaluation of the proximal operator affects convergence. The authors use the property of linear convergence for Proximal Point Method (PPM) given inexact evaluation of proximal operator and quadratic growth assumption to show similar linear convergence for Inexact ALMs but there are no guarantees based on chosen error tolerance schedule (i.e. choice of $\\epsilon_k$ and $\\delta_k$ for all k). Another weakness of the paper is the lack of experimental evaluation of conic problems other than SDPs. The work shows theoretical guarantees of local linear convergence under strict complementarity and strong duality for three different types of conic problems: LPs, SOCPs and SDPs but experimental evaluation of LPs and SOCPs are lacking. It maybe because SDPs are a more general class than LPs or SOCPs but it would be good to have a variety of experiments that cover both convex/non-convex constraints as well as different LPs and SOCPs in addition to SDPs. There is also a lack of experimental evaluation to claim that there is local linear convergence given inexact evaluations of the subproblem in Equation (15a).\n\nQUESTIONS:\n1. Why were LPs and SOCPs not evaluated as part of the experimental evaluation but discussed in the theoretical sections and appendix?\n2. Have you considered showing the local linear convergence given different random starting points for Inexact ALM? \n3. Did you use inexact evaluation of the subproblem in Equation (15a)?\n4. Have you considered showing the local linear convergence given different error tolerance schedules for $\\epsilon_k$ and $\\delta_k$ for all k?\n5. Have you considered choosing the $\\mathcal{A}$ operator to hold non-convex constraints in the experimental evaluation? The MaxCut SDP and Matrix Completion SDP both have linear constraints and the BQM has a convex constraint of $x_i^2 = 1$ but your guarantees only only assume that $\\mathcal{A}$ is surjective.\n\nLIMITATIONS:\nThere were no discussions of limitations in the work or adequate discussion of future directions of work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the convergence of primal and dual iterates in Augmented Lagrangian Methods (ALMs) for conic optimization, particularly under quadratic growth assumptions. The authors establish that both primal and dual iterates of ALMs demonstrate linear convergence solely based on the strict complementarity assumption and a bounded solution set. This addresses a significant gap in the literature concerning the linear convergence of primal iterates, providing a theoretical foundation that has been previously unresolved.\n\nSTRENGTHS:\nOriginality: The paper attempts to establish the conditions under which both primal and dual iterates of Augmented Lagrangian Methods (ALMs) for conic optimization exhibit linear convergence.\n\nClarity: The paper is well-structured and written with a clear narrative that guides the reader through the complex theoretical developments. Definitions, theorems, and proofs are delineated, making the complex content accessible to readers unfamiliar with the subject area. \n\nSignificance: The findings have potential implications for both the theoretical and practical aspects of optimization.\n\nWEAKNESSES:\nPresentation of Key Contributions: One significant issue with the paper is the placement and treatment of one of its key contributions, Theorem 3, which appears towards the very end of the document. This theorem, which is central to the paper's claims and theoretical framework, is not proven within the main text, and the appendix provided does not sufficiently validate or elaborate on it. This placement and lack of rigorous proof within the main discourse may detract from the impact of the contribution, as it might be overlooked or undervalued by readers. To improve, the authors could consider integrating Theorem 3 more prominently within the main discussion and ensuring that a complete and accessible proof is included either in the main text or more comprehensively in the appendix.\n\nNumerical Validation: Another area where the paper falls short is in its numerical section. The section dedicated to numerical validation of the theoretical results is very brief and lacks depth/sufficient validation through several diverse problems. A more extensive set of numerical experiments is crucial to demonstrate the practical effectiveness and robustness of the proposed methods under varied conditions. The current numerical validation does not sufficiently cover the scope of the potential applications discussed in the paper. Readers are referred to Appendix H, where additional testing is still missing.\n\nQUESTIONS:\nTheorem 3 Proof: Can the authors provide a detailed proof of Theorem 3 within the main text or an enhanced appendix to help readers understand its foundational role in the paper's theoretical framework? \n\nNumerical Experiments: Could the authors expand on the range and depth of the numerical experiments presented? Specifically, how do the proposed methods perform under varying conditions and parameter settings?\n\nLIMITATIONS:\nThe paper does not explicitly discuss the limitations or potential negative societal impacts of the research, which is a critical component of scholarly communication.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}