Convergent Adaptive Gradient Methods in Decentralized Optimization

1 INTRODUCTION . Distributed training of machine learning models is drawing growing attention in the past few years due to its practical benefits and necessities . Given the evolution of computing capabilities of CPUs and GPUs , computation time in distributed settings is gradually dominated by the communication time in many circumstances ( Chilimbi et al. , 2014 ; McMahan et al. , 2017 ) . As a result , a large amount of recent works has been focussing on reducing communication cost for distributed learning ( Alistarh et al. , 2017 ; Lin et al. , 2018 ; Wangni et al. , 2018 ; Stich et al. , 2018 ; Wang et al. , 2018 ; Tang et al. , 2019 ) . In the traditional parameter ( central ) server setting , where a parameter server is employed to manage communication in the whole network , many effective communication reductions have been proposed based on gradient compression ( Aji & Heafield , 2017 ) and quantization ( Chen et al. , 2010 ; Ge et al. , 2013 ; Jegou et al. , 2010 ) techniques . Despite these communication reduction techniques , its cost still , usually , scales linearly with the number of workers . Due to this limitation and with the sheer size of decentralized devices , the decentralized training paradigm ( Duchi et al. , 2011b ) , where the parameter server is removed and each node only communicates with its neighbors , is drawing attention . It has been shown in Lian et al . ( 2017 ) that decentralized training algorithms can outperform parameter server-based algorithms when the training bottleneck is the communication cost . The decentralized paradigm is also preferred when a central parameter server is not available . In light of recent advances in nonconvex optimization , an effective way to accelerate training is by using adaptive gradient methods like AdaGrad ( Duchi et al. , 2011a ) , Adam ( Kingma & Ba , 2015 ) or AMSGrad ( Reddi et al. , 2018 ) . Their popularity are due to their practical benefits in training neural networks , featured by faster convergence and ease of parameter tuning compared with Stochastic Gradient Descent ( SGD ) ( Robbins & Monro , 1951 ) . Despite a large amount of studies within the distributed optimization literature , few works have considered bringing adaptive gradient methods into distributed training , largely due to the lack of understanding of their convergence behaviors . Notably , Reddi et al . ( 2020 ) develop the first decentralized ADAM method for distributed optimization problems with a direct application to federated learning . An inner loop is employed to compute mini-batch gradients on each node and a global adaptive step is applied to update the global parameter at each outer iteration . Yet , in the settings of our paper , nodes can only communicate to their neighbors on a fixed communication graph while a server/worker communication is required in Reddi et al . ( 2020 ) . Designing adaptive methods in such settings is highly non-trivial due to the already complex update rules and to the interaction between the effect of using adaptive learning rates and the decentralized communication protocols . This paper is an attempt at bridging the gap between both realms in nonconvex optimization . Our contributions are summarized as follows : • In this paper , we investigate the possibility of using adaptive gradient methods in the decentralized training paradigm , where nodes have only a local view of the whole communication graph . We develop a general technique that converts an adaptive gradient method from a centralized method to its decentralized variant . • By using our proposed technique , we present a new decentralized optimization algorithm , called decentralized AMSGrad , as the decentralized counterpart of AMSGrad . • We provide a theoretical verification interface , in Theroem 2 , for analyzing the behavior of decentralized adaptive gradient methods obtained as a result of our technique . Thus , we characterize the convergence rate of decentralized AMSGrad , which is the first convergent decentralized adaptive gradient method , to the best of our knowledge . A novel technique in our framework is a mechanism to enforce a consensus on adaptive learning rates at different nodes . We show the importance of consensus on adaptive learning rates by proving a divergent problem instance for a recently proposed decentralized adaptive gradient method , namely DADAM ( Nazari et al. , 2019 ) , a decentralized version of AMSGrad . Though consensus is performed on the model parameter , DADAM lacks consensus principles on adaptive learning rates . After having presented existing related work and important concepts of decentralized adaptive methods in Section 2 , we develop our general framework for converting any adaptive gradient algorithm in its decentralized counterpart along with their rigorous finite-time convergence analysis in Section 3 concluded by some illustrative examples of our framework ’ s behavior in practice . Notations : xt , i denotes variable x at node i and iteration t. ‖·‖abs denotes the entry-wise L1 norm of a matrix , i.e . ‖A‖abs = ∑ i , j |Ai , j | . We introduce important notations used throughout the paper : for any t > 0 , Gt : = [ gt , N ] where [ gt , N ] denotes the matrix [ gt,1 , gt,2 , · · · , gt , N ] ( where gt , i is a column vector ) , Mt : = [ mt , N ] , Xt : = [ xt , N ] , ∇f ( Xt ) : = 1N ∑N i=1∇fi ( xt , i ) , Ut : = [ ut , N ] , Ũt : = [ ũt , N ] , Vt : = [ vt , N ] , V̂t : = [ v̂t , N ] , Xt : = 1N ∑N i=1 xt , i , U t : = 1 N ∑N i=1 ut , i and Ũt : = 1 N ∑N i=1 ũt , i . 2 DECENTRALIZED ADAPTIVE TRAINING AND DIVERGENCE OF DADAM . 2.1 RELATED WORK . Decentralized optimization : Traditional decentralized optimization methods include well-know algorithms such as ADMM ( Boyd et al. , 2011 ) , Dual Averaging ( Duchi et al. , 2011b ) , Distributed Subgradient Descent ( Nedic & Ozdaglar , 2009 ) . More recent algorithms include Extra ( Shi et al. , 2015 ) , Next ( Di Lorenzo & Scutari , 2016 ) , Prox-PDA ( Hong et al. , 2017 ) , GNSD ( Lu et al. , 2019 ) , and Choco-SGD ( Koloskova et al. , 2019 ) . While these algorithms are commonly used in applications other than deep learning , recent algorithmic advances in the machine learning community have shown that decentralized optimization can also be useful for training deep models such as neural networks . Lian et al . ( 2017 ) demonstrate that a stochastic version of Decentralized Subgradient Descent can outperform parameter server-based algorithms when the communication cost is high . Tang et al . ( 2018 ) propose the D2 algorithm improving the convergence rate over Stochastic Subgradient Descent . Assran et al . ( 2019 ) propose the Stochastic Gradient Push that is more robust to network failures for training neural networks . The study of decentralized training algorithms in the machine learning community is only at its initial stage . No existing work , to our knowledge , has seriously considered integrating adaptive gradient methods in the setting of decentralized learning . One noteworthy work ( Nazari et al. , 2019 ) propose a decentralized version of AMSGrad ( Reddi et al. , 2018 ) and it is proven to satisfy some non-standard regret . Adaptive gradient methods : Adaptive gradient methods have been popular in recent years due to their superior performance in training neural networks . Most commonly used adaptive methods include AdaGrad ( Duchi et al. , 2011a ) or Adam ( Kingma & Ba , 2015 ) and their variants . Key features of such methods lie in the use of momentum and adaptive learning rates ( which means that the learning rate is changing during the optimization and is anisotropic , i.e . depends on the dimension ) . The method of reference , called Adam , has been analyzed in Reddi et al . ( 2018 ) where the authors point out an error in previous convergence analyses . Since then , a variety of papers have been focussing on analyzing the convergence behavior of the numerous existing adaptive gradient methods . Ward et al . ( 2019 ) , Li & Orabona ( 2019 ) derive convergence guarantees for a variant of AdaGrad without coordinate-wise learning rates . Chen et al . ( 2019 ) analyze the convergence behavior of a broad class of algorithms including AMSGrad and AdaGrad . Zou & Shen ( 2018 ) provide a unified convergence analysis for AdaGrad with momentum . Noticeable recent works on adaptive gradient methods can be found in Agarwal et al . ( 2019 ) ; Luo et al . ( 2019 ) ; Zaheer et al . ( 2018 ) . 2.2 DECENTRALIZED OPTIMIZATION . In distributed optimization ( with N nodes ) , we aim at solving the following problem min x∈Rd 1 N N∑ i=1 fi ( x ) , ( 1 ) where x is the vector of parameters and fi is only accessible by the ith node . Through the prism of empirical risk minimization procedures , fi can be viewed as the average loss of the data samples located at node i , for all i ∈ [ N ] . Throughout the paper , we make the following mild assumptions required for analyzing the convergence behavior of the different decentralized optimization algorithms : A1 . For all i ∈ [ N ] , fi is differentiable and the gradients is L-Lipschitz , i.e. , for all ( x , y ) ∈ Rd , ‖∇fi ( x ) −∇fi ( y ) ‖ ≤ L‖x− y‖ . A2 . We assume that , at iteration t , node i accesses a stochastic gradient gt , i . The stochastic gradients and the gradients of fi have bounded L∞ norms , i.e . ‖gt , i‖ ≤ G∞ , ‖∇fi ( x ) ‖∞ ≤ G∞ . A3 . The gradient estimators are unbiased and each coordinate have bounded variance , i.e . E [ gt , i ] = ∇fi ( xt , i ) and E [ ( [ gt , i − fi ( xt , i ) ] j ) 2 ] ≤ σ2 , ∀t , i , j . Assumptions A1 and A3 are standard in distributed optimization literature . A2 is slightly stronger than the traditional assumption that the estimator has bounded variance , but is commonly used for the analysis of adaptive gradient methods ( Chen et al. , 2019 ; Ward et al. , 2019 ) . Note that the bounded gradient estimator assumption in A2 implies the bounded variance assumption in A3 . In decentralized optimization , the nodes are connected as a graph and each node only communicates to its neighbors . In such case , one usually constructs a N × N matrix W for information sharing when designing new algorithms . We denote λi to be its ith largest eigenvalue and define λ , max ( |λ2| , |λN | ) . The matrix W can not be arbitrary , its required key properties are listed in the following assumption : A4 . The matrix W satisfies : ( I ) ∑N j=1Wi , j = 1 , ∑N i=1Wi , j = 1 , Wi , j ≥ 0 , ( II ) λ1 = 1 , |λ2| < 1 , |λN | < 1 and ( III ) Wi , j = 0 if node i and node j are not neighbors . We now present the failure to converge of current decentralized adaptive method before introducing our proposed framework for general decentralized adaptive gradient methods . 2.3 DIVERGENCE OF DADAM Algorithm 1 DADAM ( with N nodes ) 1 : Input : α , current point Xt , u 1 2 , i = v̂0 , i = 1 , m0 = 0 and mixing matrix W 2 : for t = 1 , 2 , · · · , T do 3 : for all i ∈ [ N ] do in parallel 4 : gt , i ← ∇fi ( xt , i ) + ξt , i 5 : mt , i = β1mt−1 , i + ( 1− β1 ) gt , i 6 : vt , i = β2vt−1 , i + ( 1− β2 ) g2t , i 7 : v̂t , i = β3v̂t , i+ ( 1−β3 ) max ( v̂t−1 , i , vt , i ) 8 : xt+ 12 , i = ∑N j=1Wijxt , j 9 : xt+1 , i = xt+ 12 , i − α mt , i√ v̂t , i 10 : end for Recently , Nazari et al . ( 2019 ) initiated an attempt to bring adaptive gradient methods into decentralized optimization with Decentralized ADAM ( DADAM ) , shown in Algorithm 1 . DADAM is essentially a decentralized version of ADAM and the key modification is the use of a consensus step on the optimization variable x to transmit information across the network , encouraging its convergence . The matrix W is a doubly stochastic matrix ( which satisfies A4 ) for achieving average consensus of x . Introducing such mixing matrix is standard for decentralizing an algorithm , such as distributed gradient descent ( Nedic & Ozdaglar , 2009 ; Yuan et al. , 2016 ) . It is proven in Nazari et al . ( 2019 ) that DADAM admits a non-standard regret bound in the online setting . Nevertheless , whether the algorithm can converge to stationary points in standard offline settings such training neural networks is still unknown . The next theorem shows that DADAM may fail to converge in the offline settings . Theorem 1 . There exists a problem satisfying A1-A4 where DADAM fails to converge to a stationary points with∇f ( X̄t ) = 0 . Proof . Consider a two-node setting with objective function f ( x ) = 1/2 ∑2 i=1 fi ( x ) and f1 ( x ) = 1 [ |x| ≤ 1 ] 2x2 +1 [ |x| > 1 ] ( 4|x|−2 ) , f2 ( x ) = 1 [ |x−1| ≤ 1 ] ( x−1 ) 2 +1 [ |x−1| > 1 ] ( 2|x−1|−1 ) . We set the mixing matrix W = [ 0.5 , 0.5 ; 0.5 , 0.5 ] . The optimal solution is x∗ = 1/3 . Both f1 and f2 are smooth and convex with bounded gradient norm 4 and 2 , respectively . We also have L = 4 ( defined in A1 ) . If we initialize with x1,1 = x1,2 = −1 and run DADAM with β1 = β2 = β3 = 0 and ≤ 1 , we will get v̂1,1 = 16 and v̂1,2 = 4 . Since |gt,1| ≤ 4 , |gt,2| ≤ 2 due to bounded gradient , and ( v̂t,1 , v̂t,2 ) are non-decreasing , we have v̂t,1 = 16 , v̂t,2 = 4 , ∀t ≥ 1 . Thus , after t = 1 , DADAM is equivalent to running decentralized gradient descent ( DGD ) ( Yuan et al. , 2016 ) with a re-scaled f1 and f2 , i.e . running DGD on f ′ ( x ) = ∑2 i=1 f ′ i ( x ) with f ′ 1 ( x ) = 0.25f1 ( x ) and f ′ 2 ( x ) = 0.5f2 ( x ) , which unique optimal x′ = 0.5 . Define x̄t = ( xt,1 + xt,2 ) /2 , then by Th . 2 in Yuan et al . ( 2016 ) , we have when α < 1/4 , f ′ ( x̄t ) −f ( x′ ) = O ( 1/ ( αt ) ) . Since f ′ has a unique optima x′ , the above bound implies x̄t is converging to x′ = 0.5 which has non-zero gradient on function∇f ( 0.5 ) = 0.5 . Theorem 1 shows that , even though DADAM is proven to satisfy some regret bounds ( Nazari et al. , 2019 ) , it can fail to converge to stationary points in the nonconvex offline setting ( common for training neural networks ) . We conjecture that this inconsistency in the convergence behavior of DADAM is due to the definition of the regret in Nazari et al . ( 2019 ) . The next section presents decentralized adaptive gradient methods that are guaranteed to converge to stationary points under assumptions and provide a characterization of that convergence in finite-time and independently of the initialization .

Summary: The paper introduces a decentralized framework for adaptive momentum-based gradient descent optimizers, such as ADAM. The proposed method is novel and is among the first works to consider a decentralized communication graph without a master node. The author discovers the divergent properties of the recent work of DADAM (Nazari, 2019) and proposes a way to fix it by adding a similar consensus step for the adaptive learning rates of agents. The mathematical derivation seems to be correct to the best of my knowledge. Finally, the author tests their method on a simple CNN and show their superiority compared to DADAM to achieve a close to the centralized performance.