Accelerated Variance Reduction Stochastic ADMM for Large-Scale Machine Learning

Recently, many stochastic variance reduced alternating direction methods of multipliers (ADMMs) (e.g., SAG-ADMM and SVRG-ADMM) have made exciting progress such as linear convergence rate for strongly convex (SC) problems. However, their best-known convergence rate for non-strongly convex (non-SC) problems is <inline-formula><tex-math notation="LaTeX">$\mathcal {O}(1/T)$</tex-math><alternatives><mml:math><mml:mrow><mml:mi mathvariant="script">O</mml:mi><mml:mo>(</mml:mo><mml:mn>1</mml:mn><mml:mo>/</mml:mo><mml:mi>T</mml:mi><mml:mo>)</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="shang-ieq1-3000512.gif"/></alternatives></inline-formula> as opposed to <inline-formula><tex-math notation="LaTeX">$\mathcal {O}(1/T^2)$</tex-math><alternatives><mml:math><mml:mrow><mml:mi mathvariant="script">O</mml:mi><mml:mo>(</mml:mo><mml:mn>1</mml:mn><mml:mo>/</mml:mo><mml:msup><mml:mi>T</mml:mi><mml:mn>2</mml:mn></mml:msup><mml:mo>)</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="shang-ieq2-3000512.gif"/></alternatives></inline-formula> of accelerated deterministic algorithms, where <inline-formula><tex-math notation="LaTeX">$T$</tex-math><alternatives><mml:math><mml:mi>T</mml:mi></mml:math><inline-graphic xlink:href="shang-ieq3-3000512.gif"/></alternatives></inline-formula> is the number of iterations. Thus, there remains a gap in the convergence rates of existing stochastic ADMM and deterministic algorithms. To bridge this gap, we introduce a new momentum acceleration trick into stochastic variance reduced ADMM, and propose a novel accelerated SVRG-ADMM method (called ASVRG-ADMM) for the machine learning problems with the constraint <inline-formula><tex-math notation="LaTeX">$Ax + By = c$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>A</mml:mi><mml:mi>x</mml:mi><mml:mo>+</mml:mo><mml:mi>B</mml:mi><mml:mi>y</mml:mi><mml:mo>=</mml:mo><mml:mi>c</mml:mi></mml:mrow></mml:math><inline-graphic xlink:href="shang-ieq4-3000512.gif"/></alternatives></inline-formula>. Then we design a linearized proximal update rule and a simple proximal one for the two classes of ADMM-style problems with <inline-formula><tex-math notation="LaTeX">$B = \tau I$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>B</mml:mi><mml:mo>=</mml:mo><mml:mi>τ</mml:mi><mml:mi>I</mml:mi></mml:mrow></mml:math><inline-graphic xlink:href="shang-ieq5-3000512.gif"/></alternatives></inline-formula> and <inline-formula><tex-math notation="LaTeX">$B\ne \tau I$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>B</mml:mi><mml:mo>≠</mml:mo><mml:mi>τ</mml:mi><mml:mi>I</mml:mi></mml:mrow></mml:math><inline-graphic xlink:href="shang-ieq6-3000512.gif"/></alternatives></inline-formula>, respectively, where <inline-formula><tex-math notation="LaTeX">$I$</tex-math><alternatives><mml:math><mml:mi>I</mml:mi></mml:math><inline-graphic xlink:href="shang-ieq7-3000512.gif"/></alternatives></inline-formula> is an identity matrix and <inline-formula><tex-math notation="LaTeX">$\tau$</tex-math><alternatives><mml:math><mml:mi>τ</mml:mi></mml:math><inline-graphic xlink:href="shang-ieq8-3000512.gif"/></alternatives></inline-formula> is an arbitrary bounded constant. Note that our linearized proximal update rule can avoid solving sub-problems iteratively. Moreover, we prove that ASVRG-ADMM converges linearly for SC problems. In particular, ASVRG-ADMM improves the convergence rate from <inline-formula><tex-math notation="LaTeX">$\mathcal {O}(1/T)$</tex-math><alternatives><mml:math><mml:mrow><mml:mi mathvariant="script">O</mml:mi><mml:mo>(</mml:mo><mml:mn>1</mml:mn><mml:mo>/</mml:mo><mml:mi>T</mml:mi><mml:mo>)</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="shang-ieq9-3000512.gif"/></alternatives></inline-formula> to <inline-formula><tex-math notation="LaTeX">$\mathcal {O}(1/T^2)$</tex-math><alternatives><mml:math><mml:mrow><mml:mi mathvariant="script">O</mml:mi><mml:mo>(</mml:mo><mml:mn>1</mml:mn><mml:mo>/</mml:mo><mml:msup><mml:mi>T</mml:mi><mml:mn>2</mml:mn></mml:msup><mml:mo>)</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="shang-ieq10-3000512.gif"/></alternatives></inline-formula> for non-SC problems. Finally, we apply ASVRG-ADMM to various machine learning problems, e.g., graph-guided fused Lasso, graph-guided logistic regression, graph-guided SVM, generalized graph-guided fused Lasso and multi-task learning, and show that ASVRG-ADMM consistently converges faster than the state-of-the-art methods.