Recently, many variance reduced stochastic alternating direction method of multipliers (ADMM) methods (e.g. SAGADMM, SDCA-ADMM and SVRG-ADMM) have made exciting progress such as linear convergence rates for strongly convex problems. However, the best known convergence rate for general convex problems is O(1/T ) as opposed to O(1/T ) of accelerated batch algorithms, where T is the number of iterations. Thus, there still remains a gap in convergence rates between existing stochastic ADMM and batch algorithms. To bridge this gap, we introduce the momentum acceleration trick for batch optimization into the stochastic variance reduced gradient based ADMM (SVRG-ADMM), which leads to an accelerated (ASVRG-ADMM) method. Then we design two different momentum term update rules for strongly convex and general convex cases. We prove that ASVRG-ADMM converges linearly for strongly convex problems. Besides having a low per-iteration complexity as existing stochastic ADMM methods, ASVRG-ADMM improves the convergence rate on general convex problems from O(1/T ) to O(1/T ). Our experimental results show the effectiveness of ASVRG-ADMM. Introduction In this paper, we consider a class of composite convex optimization problems min x∈Rd1 f(x) + h(Ax), (1) where A∈Rd2×d1 is a given matrix, f(x) := 1 n ∑n i=1fi(x), each fi(x) is a convex function, and h(Ax) is convex but possibly non-smooth. With regard to h(·), we are interested in a sparsity-inducing regularizer, e.g. `1-norm, group Lasso and nuclear norm. When A is an identity matrix, i.e. A = Id1 , the above formulation (1) arises in many places in machine learning, statistics, and operations research (Bubeck 2015), such as logistic regression, Lasso and support vector machine (SVM). We mainly focus on the large sample regime. In this regime, even first-order batch methods, e.g. FISTA (Beck and Teboulle 2009), become computationally burdensome due to their per-iteration complexity of O(nd1). As a result, stochastic gradient descent (SGD) with per-iteration complexity of O(d1) has Copyright c © 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. witnessed tremendous progress in the recent years. Especially, a number of stochastic variance reduced gradient methods such as SAG (Roux, Schmidt, and Bach 2012), SDCA (Shalev-Shwartz and Zhang 2013) and SVRG (Johnson and Zhang 2013) have been proposed to successfully address the problem of high variance of the gradient estimate in ordinary SGD, resulting in a linear convergence rate (for strongly convex problems) as opposed to sub-linear rates of SGD. More recently, the Nesterov’s acceleration technique (Nesterov 2004) was introduced in (Allen-Zhu 2016; Hien et al. 2016) to further speed up the stochastic variancereduced algorithms, which results in the best known convergence rates for both strongly convex and general convex problems. This motivates us to integrate the momentum acceleration trick into the stochastic alternating direction method of multipliers (ADMM) below. When A is a more general matrix, i.e. A 6=Id1 , the formulation (1) becomes many more complicated problems arising from machine learning, e.g. graph-guided fuzed Lasso (Kim, Sohn, and Xing 2009) and generalized Lasso (Tibshirani and Taylor 2011). To solve this class of composite optimization problems with an auxiliary variable y = Ax, which are the special case of the general ADMM form, min x∈Rd1,y∈Rd2 f(x) + h(y), s.t. Ax+By = c, (2) the ADMM is an effective optimization tool (Boyd et al. 2011), and has shown attractive performance in a wide range of real-world problems, such as big data classification (Nie et al. 2014). To tackle the issue of high periteration complexity of batch (deterministic) ADMM (as a popular first-order optimization method), Wang and Banerjee (2012), Suzuki (2013) and Ouyang et al. (2013) proposed some online or stochastic ADMM algorithms. However, all these variants only achieve the convergence rate of O(1/ √ T ) for general convex problems and O(log T/T ) for strongly convex problems, respectively, as compared with the O(1/T ) and linear convergence rates of accelerated batch algorithms (Nesterov 1983), e.g. FISTA, where T is the number of iterations. By now several accelerated and faster converging versions of stochastic ADMM, which are all based on variance reduction techniques, have been proposed, e.g. SAG-ADMM (Zhong and Kwok 2014b), SDCA-ADMM (Suzuki 2014) and SVRG-ADMM (Zheng and Kwok 2016). With regard to strongly convex problems, ar X iv :1 70 7. 03 19 0v 1 [ cs .L G ] 1 1 Ju l 2 01 7 Table 1: Comparison of convergence rates and memory requirements of some stochastic ADMM algorithms. General convex Strongly-convex Space requirement SAG-ADMM O(1/T ) unknown O(d1d2+nd1) SDCA-ADMM unknown linear rate O(d1d2+n) SCAS-ADMM O(1/T ) O(1/T ) O(d1d2) SVRG-ADMM O(1/T ) linear rate O(d1d2) ASVRG-ADMM O(1/T ) linear rate O(d1d2) Suzuki (2014) and Zheng and Kwok (2016) proved that linear convergence can be obtained for the special ADMM form (i.e. B = −Id2 and c = 0) and the general ADMM form, respectively. In SAG-ADMM and SVRG-ADMM, an O(1/T ) convergence rate can be guaranteed for general convex problems, which implies that there still remains a gap in convergence rates between the stochastic ADMM and accelerated batch algorithms. To bridge this gap, we integrate the momentum acceleration trick in (Tseng 2010) for deterministic optimization into the stochastic variance reduction gradient (SVRG) based stochastic ADMM (SVRG-ADMM). Naturally, the proposed method has low per-iteration time complexity as existing stochastic ADMM algorithms, and does not require the storage of all gradients (or dual variables) as in SCAS-ADMM (Zhao, Li, and Zhou 2015) and SVRGADMM (Zheng and Kwok 2016), as shown in Table 1. We summarize our main contributions below. • We propose an accelerated variance reduced stochastic ADMM (ASVRG-ADMM) method, which integrates both the momentum acceleration trick in (Tseng 2010) for batch optimization and the variance reduction technique of SVRG (Johnson and Zhang 2013). • We prove that ASVRG-ADMM achieves a linear convergence rate for strongly convex problems, which is consistent with the best known result in SDCA-ADMM (Suzuki 2014) and SVRG-ADMM (Zheng and Kwok 2016). • We also prove that ASVRG-ADMM has a convergence rate of O(1/T ) for non-strongly convex problems, which is a factor of T faster than SAG-ADMM and SVRG-ADMM, whose convergence rates are O(1/T ). • Our experimental results further verified that our ASVRG-ADMM method has much better performance than the state-of-the-art stochastic ADMM methods. Related Work Introducing y = Ax ∈R2 , problem (1) becomes min x∈Rd1,y∈Rd2 f(x) + h(y), s.t. Ax− y = 0. (3) Although (3) is only a special case of the general ADMM form (2), when B = −Id2 and c = 0, the stochastic (or online) ADMM algorithms and theoretical results in (Wang and Banerjee 2012; Ouyang et al. 2013; Zhong and Kwok 2014b; Zheng and Kwok 2016) and this paper are all for the more general problem (2). To minimize (2), together with the dual variable λ, the update steps of batch ADMM are yk = argminy h(y) + β 2 ‖Axk−1+By−c+λk−1‖ , (4) xk = argminx f(x) + β 2 ‖Ax+Byk−c+λk−1‖ , (5) λk = λk−1 +Axk +Byk − c, (6) where β>0 is a penalty parameter. To extend the batch ADMM to the online and stochastic settings, the update steps for yk and λk remain unchanged. In (Wang and Banerjee 2012; Ouyang et al. 2013), the update step of xk is approximated as follows: xk = argmin x x∇fik(xk−1) + 1 2ηk ‖x− xk−1‖G + β 2 ‖Ax+Byk−c+λk−1‖, (7) where we draw ik uniformly at random from [n] := {1, . . . , n}, ηk ∝ 1/ √ k is the step-size, and ‖z‖G = zGz with given positive semi-definite matrix G, e.g. G = Id1 in (Ouyang et al. 2013). Analogous to SGD, the stochastic ADMM variants use an unbiased estimate of the gradient at each iteration. However, all those algorithms have much slower convergence rates than their batch counterpart, as mentioned above. This barrier is mainly due to the variance introduced by the stochasticity of the gradients. Besides, to guarantee convergence, they employ a decaying sequence of step sizes ηk, which in turn impacts the rates. More recently, a number of variance reduced stochastic ADMM methods (e.g. SAG-ADMM, SDCA-ADMM and SVRG-ADMM) have been proposed and made exciting progress such as linear convergence rates. SVRG-ADMM in (Zheng and Kwok 2016) is particularly attractive here because of its low storage requirement compared with the algorithms in (Zhong and Kwok 2014b; Suzuki 2014). Within each epoch of SVRG-ADMM, the full gradient p̃ =∇f(x̃) is first computed, where x̃ is the average point of the previous epoch. Then ∇fik(xk−1) and ηk in (7) are replaced by ∇̃fIk(xk−1) = 1 |Ik| ∑ ik∈Ik (∇fik(xk−1)−∇fik(x̃)) + p̃ (8) and a constant step-size η, respectively, where Ik ⊂ [n] is a mini-batch of size b (which is a useful technique to reduce the variance). In fact, ∇̃fIk(xk−1) is an unbiased estimator of the gradient ∇f(xk−1), i.e. E[∇̃fIk(xk−1)]=∇f(xk−1). Accelerated Variance Reduced Stochastic ADMM In this section, we design an accelerated variance reduced stochastic ADMM method for both strongly convex and general convex problems. We first make the following assumptions: Each convex fi(·) is Li-smooth, i.e. there exists a constant Li>0 such that ‖∇fi(x)−∇fi(y)‖≤Li‖x−y‖, ∀x, y ∈ R, and L , maxi Li; f(·) is μ-strongly convex, i.e. there is μ > 0 such that f(x) ≥ f(y) +∇f(y)(x− y)+ μ2 ‖x−y‖ 2 for all x, y ∈R; The matrix A has full row rank. The first two assumptions are common in the analysis of first-order optimization methods, while the last one has Algorithm 1 ASVRG-ADMM for strongly-convex case Input: m, η, β > 0, 1 ≤ b ≤ n. Initialize: x̃= z̃, ỹ, θ, λ̃=− 1 β (A T )∇f(x̃0). 1: for s = 1, 2, . . . , T do 2: x0 = z s 0 = x̃ s−1, y 0 = ỹ s−1, λ0 = λ̃ s−1; 3: p̃ = ∇f(x̃s−1); 4: for k = 1, 2, . . . ,m do 5: Choose Ik⊆ [n]of size b, uniformly at random; 6: y k=argminy h(y)+ β 2 ‖Az s k−1+By− c+λk−1‖; 7: z k=z s k−1− η(∇̃fIk(x s k−1)+βA (Az k−1+By s k−c+λ s k−1)) γθ ; 8: xk=(1− θ)x̃s−1 + θz k; 9: λk=λ s k−1 +Az s k +By s k − c; 10: end for 11: x̃= 1 m ∑m k=1x s k, ỹ s=(1−θ)ỹs−1+ θ m ∑m k=1y s k, 12: λ̃=− 1 β (A T )†∇f(x̃s); 13: end for Output: x̃ , ỹ . been used in the convergence analysis of batch ADMM (?; Nishihara et al. 2015; Deng and Yin 2016) and stochastic ADMM (Zheng and Kwok 2016). The Strongly Convex Case In this part, we consider the case of (2) when each fi(·) is convex,L-smooth, and f(·) is μ-strongly convex. Recall that this class of problems include graph-guided Logistic Regression and SVM as notable examples. To efficiently solve this class of problems, we incorporate both the momentum acceleration and variance reduction techniques into stochastic ADMM. Our algorithm is divided into T epochs, and each epoch consists of m stochastic updates, where m is usually chosen to be O(n) as in (Johnson and Zhang 2013). Let z be an important auxiliary variable, its update rule is given as follows. Similar to (Zhong and Kwok 2014b; Zheng and Kwok 2016), we also use the inexact Uzawa method (Zhang, Burger, and Osher 2011) to approximate the sub-problem (7), which can avoid computing the inverse of the matrix ( 1 η Id1+βA A). Moreover, the momentum weight 0≤ θs ≤ 1 (the update rule for θs is provided below) is introduced into the proximal term 1 2η‖x−xk−1‖ 2 G similar to that of (7), and then the sub-problem with respect to z is formulated as follows: min z (z −z k−1) ∇̃fIk(xk−1)+ θs−1 2η ‖z −z k−1‖G + β 2 ‖Az +By k − c+ λk−1‖, (9) where ∇̃fIk(xk−1) is defined in (8), η < 1 2L , and G = γId1− ηβ θs−1 AA with γ ≥ γmin ≡ ηβ‖A A‖2 θs−1 +1 to ensure that G I similar to (Zheng and Kwok 2016), where ‖·‖2 is the spectral norm, i.e. the largest singular value of the matrix. Furthermore, the update rule for x is given by xk= x̃ +θs−1(z s k− x̃)=(1−θs−1)x̃+θs−1z k, (10) where θs−1(z k − x̃s−1) is the key momentum term (similar to those in accelerated batch methods (Nesterov 2004)), which helps accelerate our algorithm by using the iterate of the previous epoch, i.e. x̃s−1. Similar to xk, ỹ s = (1− θs−1)ỹ s−1+ θs−1 m ∑m k=1y s k. Moreover, θs can be set to a constant θ in all epochs of our algorithm, which must satisfy 0 ≤ θ ≤ 1− δ(b)/(α−1), where α = 1 Lη > 1+ δ(b), and δ(b) is defined below. The optimal value of θ is provided in Proposition 1 below. The detailed procedure is shown in Algorithm 1, where we adopt the same initialization technique for λ̃ as in (Zheng and Kwok 2016), and (·)† is the pseudo-inverse. Note that, when θ=1, ASVRG-ADMM degenerates to SVRG-ADMM in (Zheng and Kwok 2016). The Non-Strongly Convex Case In this part, we consider general convex problems of the form (2) when each fi(·) is convex, L-smooth, and h(·) is not necessarily strongly convex (but possibly non-smooth). Different from the strongly convex case, the momentum weight θs is required to satisfy the following inequalities: 1− θs θ2 s ≤ 1 θ2 s−1 and 0 ≤ θs ≤ 1− δ(b) α− 1 , (11) where δ(b) := n−b b(n−1) is a decreasing function with respect to the mini-batch size b. The condition (20) allows the momentum weight to decease, but not too fast, similar to the requirement on the step-size ηk in classical SGD and stochastic ADMM (?). Unlike batch acceleration methods, the weight must satisfy both inequalities in (20). Motivated by the momentum acceleration techniques in (Tseng 2010; Nesterov 2004) for batch optimization, we give the update rule of the weight θs for the mini-batch case: θs = √ θ4 s−1+ 4θ 2 s−1 − θ s−1 2 and θ0 = 1− δ(b) α− 1 . (12) For the special case of b = 1, we have δ(1) = 1 and θ0 = 1− 1 α−1 , while b=n (i.e. batch version), δ(n)=0 and θ0= 1. Since {θs} is decreasing, then θs ≤ 1− δ(b) α−1 is satisfied. The detailed procedure is shown in Algorithm 2, which has many slight differences in the initialization and output of each epoch from Algorithm 1. In addition, the key difference between them is the update rule for the momentum weight θs. That is, θs in Algorithm 1 can be set to a constant, while that in Algorithm 2 is adaptively adjusted as in (12). Convergence Analysis This section provides the convergence analysis of our ASVRG-ADMM algorithms (i.e. Algorithms 1 and 2) for strongly convex and general convex problems, respectively. Following (Zheng and Kwok 2016), we first introduce the following function P (x, y) := f(x)−f(x∗)−∇f(x∗)T(x− x∗)+h(y)−h(y∗)−h′(y∗)T(y− y∗) as a convergence criterion, where h′(y) denotes the (sub)gradient of h(·) at y. Indeed, P (x, y)≥ 0 for all x, y ∈ R. In the following, we give the intermediate key results for our analysis. Algorithm 2 ASVRG-ADMM for general convex case Input: m, η, β > 0, 1 ≤ b ≤ n. Initialize: x̃ = z̃, ỹ, λ̃, θ0 = 1− Lηδ(b) 1−Lη . 1: for s = 1, 2, . . . , T do 2: x0=(1−θs−1)x̃+θs−1z̃, y 0= ỹs−1, λ0= λ̃s−1; 3: p̃ = ∇f(x̃s−1), z 0= z̃s−1; 4: for k = 1, 2, . . . ,m do 5: Choose Ik⊆ [n]of size b, uniformly at random; 6: y k=argminy h(y)+ β 2 ‖Az s k−1+By− c+λk−1‖; 7: z k=z s k−1− η(∇̃fIk(x s k−1)+βA (Az k−1+By s k−c+λ s k−1)) γθs−1 ; 8: xk=(1− θs−1)x̃ + θs−1z k; 9: λk=λ s k−1 +Az s k +By s k − c; 10: end for 11: x̃= 1 m ∑m k=1x s k, ỹ =(1−θs−1)ỹ+ θs−1 m ∑m k=1y s k, 12: λ̃=λm, z̃ =z m, θs= √ θ4 s−1+4θ 2 s−1−θ 2 s−1 2 ; 13: end for Output: x̃ , ỹ . Lemma 1. E[‖∇̃fIk(xk−1)−∇f(xk−1)‖] ≤2Lδ(b) [ f(x̃)−f(xk−1)+(xk−1− x̃)∇f(xk−1) ] , where δ(b)= n−b b(n−1)≤1 and 1 ≤ b ≤ n. Lemma 2. Using the same notation as in Lemma 1, let (x∗, y∗, λ∗) denote an optimal solution of problem (2), and {(z k, xk, y k, λk, x̃, ỹ)} be the sequence generated by Algorithm 1 or 2 with θs ≤ 1− δ(b) α−1, where α= 1 Lη . Then the following holds for all k, E [ P (x̃, ỹ)− θs−1 m m ∑ k=1 ( (x∗−zs k)Aφk+(y−y k)Bφk )] ≤E [ P (x̃s−1, ỹs−1) 1/(1−θs−1) + θ s−1 ( ‖x∗− z 0‖G−‖x− z m‖G )
Output: Accelerated Variance Reduced Stochastic ADMM