Three Operator Splitting with Subgradients, Stochastic Gradients, and Adaptive Learning Rates

√ t ) convergence rate . The third extension ADAPTOS endows TOS with adaptive stepsizes . For the important setting of optimizing a convex loss over the intersection of convex sets ADAPTOS attains universal convergence rates , i.e. , the rate adapts to the unknown smoothness degree of the objective function . We compare our proposed methods with competing methods on various applications . 1 Introduction . We study convex optimization problems of the form min x∈Rn φ ( x ) : = f ( x ) + g ( x ) + h ( x ) , ( 1 ) where f : Rn → R and g , h : Rn → R ∪ { +∞ } are proper , lower semicontinuous and convex functions . Importantly , this template captures constrained problems via indicator functions . To avoid pathological examples , we assume that the relative interiors of dom ( f ) , dom ( g ) and dom ( h ) have a nonempty intersection . Problem ( 1 ) is motivated by a number of applications in machine learning , statistics , and signal processing , where the three functions comprising the objective φ model data fitting , structural priors , or decision constraints . Examples include overlapping group lasso ( Yuan et al. , 2011 ) , isotonic regression ( Tibshirani et al. , 2011 ) , dispersive sparsity ( El Halabi & Cevher , 2015 ) , graph transduction ( Shivanna et al. , 2015 ) , learning with correlation matrices ( Higham & Strabić , 2016 ) , and multidimensional total variation denoising ( Barbero & Sra , 2018 ) . An important technique for addressing composite problems is operator splitting ( Bauschke et al. , 2011 ) . However , the basic proximal- ( sub ) gradient method may be unsuitable for Problem ( 1 ) since it Alp Yurtsever and Alex Gu contributed equally to this paper . The paper is based primarily on the work done while Alp Yurtsever was at Massachusetts Institute of Technology . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . requires the prox-operator of g + h , computing which may be vastly more expensive than individual prox-operators of g and h. An elegant , recent method , Three Operator Splitting ( TOS , Davis & Yin ( 2017 ) , see Algorithm 1 ) offers a practical choice for solving Problem ( 1 ) when f is smooth . Importantly , at each iteration , TOS evaluates the gradient of f and the proximal operators of g and h only once . Moreover , composite problems with more than three functions can be reformulated as an instance of Problem ( 1 ) in a product-space and solved by using TOS . This is an effective method as long as each function has an efficient gradient oracle or proximal operator ( see Section 2 ) . Unfortunately , TOS is not readily applicable to many optimization problems that arise in machine learning . Most important among those are problems where only access to stochastic gradients is feasible , e.g. , when performing large-scale empirical risk minimization and online learning . Moreover , prox-operators for some complex penalty functions are computationally expensive and it may be more efficient to instead use subgradients . For example , proximal operator for the maximum eigenvalue function that appears in dual-form semidefinite programs ( e.g. , see Section 6.1 in ( Ding et al. , 2019 ) ) may require computing a full eigendecomposition . In contrast , we can form a subgradient by computing only the top eigenvector via power method or Lanczos algorithm . Contributions . With the above motivation , this paper contributes three key extensions of TOS . We tackle nonsmoothness in Section 3 and stochasticity in Section 4 . These two extensions enable us to use subgradients and stochastic gradients of f ( see Section 2 for a comparison with related work ) , and satisfy a O ( 1/ √ T ) error bound in function value after T iterations . The third main contribution is ADAPTOS in Section 5 . This extension provides an adaptive step-size rule in the spirit of AdaGrad ( Duchi et al. , 2011 ; Levy , 2017 ) for an important subclass of Problem ( 1 ) . Notably , for optimizing a convex loss over the intersection of two convex sets , ADAPTOS ensures universal convergence rates . That is , ADAPTOS implicitly adapts to the unknown smoothness degree of the problem , and ensures a Õ ( 1/ √ t ) convergence rate when the problem is nonsmooth but the rate improves to Õ ( 1/t ) if the problem is smooth and a solution lies in the relative interior of the feasible set . In Section 6 , we discuss empirical performance of our methods by comparing them against present established methods on various benchmark problems from COPT Library ( Pedregosa et al. , 2020 ) including the overlapping group lasso , total variation deblurring , and sparse and low-rank matrix recovery . We also test our methods on nonconvex optimization by training a neural network model . We present more experiments on isotonic regression and portfolio optimization in the supplements . Notation . We denote a solution of Problem ( 1 ) by x ? and φ ? : = φ ( x ? ) . The distance between a point x ∈ Rn and a closed and convex set G ⊆ Rn is dist ( x , G ) : = miny∈G ‖x − y‖ ; the projection of x onto G is given by projG ( x ) : = arg miny∈G ‖x− y‖ . The prox-operator of a function g : Rn → R ∪ { +∞ } is defined by proxg ( x ) : = arg miny∈Rn { g ( y ) + 12‖x− y‖ 2 } . The indicator function of G gives 0 for all x ∈ G and +∞ otherwise . Clearly , the prox-operator of an indicator function is the projection onto the corresponding set . 2 Background and related work . TOS , proposed recently by Davis & Yin ( 2017 ) , can be seen as a generic extension of various operator splitting schemes , including the forward-backward splitting , Douglas-Rachford splitting , forward-Douglas-Rachford splitting ( Briceño-Arias , 2015 ) , and the generalized forward-backward splitting ( Raguet et al. , 2013 ) . It covers these aforementioned approaches as special instances when the terms f , g and h in Problem ( 1 ) are chosen appropriately . Convergence of TOS is well studied when f has Lipschitz continuous gradients . It ensures O ( 1/t ) convergence rate in this setting , see ( Davis & Yin , 2017 ) and ( Pedregosa , 2016 ) for details . Other related methods that can be used for Problem ( 1 ) when f is smooth are the primal-dual hybrid gradient ( PDHG ) method ( Condat , 2013 ; Vũ , 2013 ) and the primal-dual three operator splitting methods in ( Yan , 2018 ) and ( Salim et al. , 2020 ) . These methods can handle a more general template where g or h is composed with a linear map , however , they require f to be smooth . The convergence rate of PDHG is studied in ( Chambolle & Pock , 2016 ) . Nonsmooth setting . We are unaware of any prior result that permits using subgradients in TOS ( or in other methods that can use the prox-operator of g and h separately for Problem ( 1 ) ) . The closest match is the proximal subgradient method which applies when h is removed from Problem ( 1 ) , and it is covered by our nonsmooth TOS as a special case . Stochastic setting . There are multiple attempts to devise a stochastic TOS in the literature . Yurtsever et al . ( 2016 ) studied Problem ( 1 ) under the assumption that f is smooth and strongly convex , and an unbiased gradient estimator with bounded variance is available . Their stochastic TOS has a guaranteed O ( 1/t ) convergence rate . In ( Cevher et al. , 2018 ) , they drop the strong convexity assumption , instead they assume that the variance is summable . They show asymptotic convergence with no guarantees on the rate . Later , Pedregosa et al . ( 2019 ) proposed a stochastic variance-reduced TOS and analyzed its non-asymptotic convergence guarantees . Their method gets O ( 1/t ) convergence rate when f is smooth . The rate becomes linear if f is smooth and strongly convex and g ( or h ) is also smooth . Recently , Yurtsever et al . ( 2021 ) studied TOS on problems where f can be nonconvex and showed that the method finds a first-order stationary point with O ( 1/ 3 √ t ) convergence rate under a diminishing variance assumption . They increase the batch size over the iterations to satisfy this assumption . None of these prior works cover the broad template we consider : f is smooth or Lipschitz continuous and the stochastic first-order oracle has bounded variance . To our knowledge , our paper gives the first analysis for stochastic TOS without strong convexity assumption or variance reduction . Other related methods are the stochastic PDHG in ( Zhao & Cevher , 2018 ) , the decoupling method in ( Mishchenko & Richtárik , 2019 ) , the stochastic primal-dual method in ( Zhao et al. , 2019 ) , and the stochastic primal-dual three operator splitting in ( Salim et al. , 2020 ) . The method in ( Zhao et al. , 2019 ) can be viewed as an extension of stochastic ADMM ( Ouyang et al. , 2013 ; Azadi & Sra , 2014 ) from the sum of two terms to three terms in the objective . Similar to the existing stochastic TOS variants , these methods either assume strong convexity or require variance-reduction . Adaptive step-sizes . The standard writings of TOS and PDHG require the knowledge of the smoothness constant of f for the step-size . Backtracking line-search strategies ( for finding a suitable step-size when the smoothness constant is unknown ) are proposed for PDHG in ( Malitsky & Pock , 2018 ) and for TOS in ( Pedregosa & Gidel , 2018 ) . These line-search strategies are significantly different than our adaptive learning rate . Importantly , these methods work only when f is smooth . They require extra function evaluations , and are thus not suitable for stochastic optimization . And their goal is to estimate the smoothness constant . In contrast , our goal is to design an algorithm that adapts to the unknown smoothness degree . Our method does not require function evaluations , and it can be used in smooth , nonsmooth , or stochastic settings . At the heart of our method lie adaptive online learning algorithms ( Duchi et al. , 2011 ; Rakhlin & Sridharan , 2013 ) together with online to offline conversion techniques ( Levy , 2017 ; Cutkosky , 2019 ) . Similar methods appear in the literature for other problem templates with no constraint or a single constraint in ( Levy , 2017 ; Levy et al. , 2018 ; Kavis et al. , 2019 ; Cutkosky , 2019 ; Bach & Levy , 2019 ) . Our method extends these results to optimization over the intersection of convex sets . When f is nonsmooth , ADAPTOS ensures a Õ ( 1/ √ t ) rate , whereas the rate improves to Õ ( 1/t ) if f is smooth and there is a solution in the relative interior of the feasible set . TOS for more than three functions . TOS can be used for solving problems with more than three convex functions by a product-space reformulation technique ( Briceño-Arias , 2015 ) . Consider min x∈Rd q∑ i=1 φi ( x ) , ( 2 ) where each component φi : Rd → R∪ { +∞ } is a proper , lower semicontinuous and convex function . Without loss of generality , suppose φ1 , . . . , φp are prox-friendly . Then , we can reformulate ( 2 ) in the product-space Rd× ( p+1 ) as min ( x0 , x1 , ... , xp ) ∈Rd× ( p+1 ) p∑ i=1 φi ( xi ) + q∑ i=p+1 φi ( x0 ) subject to x0 = x1 = . . . = xp . ( 3 ) This is an instance of Problem ( 1 ) with n = d× ( p+1 ) and x = ( x0 , x1 , . . . , xp ) . We can choose g ( x ) as the indicator of the equality constraint , f ( x ) = ∑q i=p+1 φi ( x0 ) , and h ( x ) = ∑p i=1 φi ( xi ) . Then , the ( sub ) gradient of f is the sum of ( sub ) gradients of φp+1 , . . . , φq ; proxg is a mapping that averages x0 , x1 , . . . , xp ; and proxh is the concatenation of the individual prox-operators of φ1 , . . . , φp . To our knowledge , TOS has been studied only for problems with smooth f , and this forces us to assign all nonsmooth components φi in ( 2 ) to the proximal term h in ( 3 ) . In this work , by enabling subgradient steps for nonsmooth f , we provide the flexibility to choose how to process each nonsmooth component φi in ( 3 ) , either by its proximal operator through h or by its subgradient via f . Algorithm 1 Three Operator Splitting ( TOS ) Input : Initial point y0 ∈ Rn , step-size sequence { γt } Tt=0 for t = 0 , 1 , 2 , . . . , T do zt = proxγtg ( yt ) Choose an update direction ut ∈ Rn { ut = ∇f ( zt ) captures the standard version of TOS } xt = proxγth ( 2zt − yt − γtut ) yt+1 = yt − zt + xt end for Return : Ergodic sequence x̄t and z̄t defined in ( 5 )

Summary: This paper provides an extension of the Three Operator Splitting methods for composite optimization, of the form $f + g + h$. Previous work assumed $f$ to be smooth. This work tackles two cases: 1) the case where $f$ is non-smooth, and the case where additionally, only a stochastic estimator of the (sub)gradient of $f$ is available. It can be seen as an extension of (Yurtsever et al 2018) in the non-smooth, non-strongly convex case. The paper provides a theoretical worst-case analysis of convergence, and gives non-asymptotic rates for these two cases. It also proposes an analyses an adaptive variant, where step size depends on the previous observed subgradient magnitudes, a la AdaGrad. It provides experimental validation of the method on a non-smooth problem, and on a non-convex non-smooth problem.