Title: DADAM: A CONSENSUS-BASED DISTRIBUTED ADAP-TIVE GRADIENT METHOD FOR ONLINE OPTIMIZATION Anonymous

Abstract: Online and stochastic optimization methods such as SGD, ADAGRAD and ADAM are key algorithms in solving large-scale machine learning problems including deep learning. A number of schemes that are based on communications of nodes with a central server have been recently proposed in the literature to parallelize them. A bottleneck of such centralized algorithms lies on the high communication cost incurred by the central node. In this paper, we present a new consensus-based distributed adaptive moment estimation method (DADAM) for online optimization over a decentralized network that enables data parallelization, as well as decentralized computation. Such a framework not only can be extremely useful for learning agents with access to only local data in a communication constrained environment, but as shown in this work also outperform centralized adaptive algorithms such as ADAM for certain realistic classes of loss functions. We analyze the convergence properties of the proposed algorithm and provide a dynamic regret bound on the convergence rate of adaptive moment estimation methods in both stochastic and deterministic settings. Empirical results demonstrate that DADAM works well in practice and compares favorably to competing online optimization methods.

1. INTRODUCTION:

Online optimization is a fundamental procedure for solving a wide range of machine learning problems Shalev-Shwartz et al. (2012) ; Hazan et al. (2016) . It can be formulated as a repeated game between a learner (algorithm) and an adversary. The learner receives a streaming data sequence, sequentially selects actions, and the adversary reveals the convex or nonconvex losses to the learner. A standard performance metric for an online algorithm is regret, which measures the performance of the algorithm versus a static benchmark Zinkevich (2003) ; Hazan et al. (2016) . For example, the benchmark could be an optimal point of the online average of the loss (local cost) function, had the learner known all the losses in advance. In a broad sense, if the benchmark is a fixed sequence, the regret is called static. Recent work on online optimization has investigated the notion of dynamic regret Zinkevich (2003) ; Hall & Willett (2015) ; Besbes et al. (2015) . Dynamic regret can take the form of the cumulative difference between the instantaneous loss and the minimum loss. For convex functions, previous studies have shown that the dynamic regret of online gradient-based methods can be upper bounded by O( √ T D T ), where D T is a measure of regularity of the comparator sequence or the function sequence Zinkevich (2003) ; Hall & Willett (2015) . This bound can be improved to O(D T ) Mokhtari et al. (2016) ; Zhang et al. (2017) when all the cost functions are strongly convex and smooth.

Decentralized nonlinear programming has received a great deal of interest in diverse scientific and engineering fields Tsitsiklis et al. (1986) ; Li et al. (2002) ; Rabbat & Nowak (2004) ; Lesser et al. (2012) . The key problem involves optimizing a cost function f (x) = 1 n ∑ n i=1 f i (x), where x ∈ R p and each f i is only known to the individual agent i in a connected network of n agents. The agents collaborate by successively sharing information with other agents located in their neighborhood with the goal of jointly converging to the network-wide optimal argument Nedic & Ozdaglar (2009) . Compared to optimization procedures involving a fusion center that collects data and performs the computation, decentralized nonlinear programming enjoys the advantage of scalability to network sizes, robustness to dynamic topologies, and privacy preservation in data-sensitive applications Yuan et al. (2016) ; Shi et al. (2015) ; Jiang et al. (2017) ; Lian et al. (2017) .

Appropriately choosing the learning rate that scale coordinates of the gradient and the way of updating them are crucial issues driving the performance of first Duchi et al. (2011) ; Kingma & Ba (2014) and second order optimization procedures Peyghami & Tarzanagh (2015) . Indeed, the understanding that adaptation of the learning rate is advantageous, particularly on a per parameter basis dynamically, led to the development of a family of widely-used adaptive gradient methods including ADAGRAD Duchi et al. (2011) , RMSPROP Tieleman & Hinton, and ADAM Kingma & Ba (2014) . The ADAM optimizer computes adaptive learning rates for different parameters from estimates of first and second moments of the gradients and performs a local optimization. Numerical results show that ADAM can achieve significantly better performance compared to ADAGRAD, RMSPROP and other gradient descent procedures when the gradients are sparse, or in general small in magnitude. However, its performance has been observed to deteriorate in settings where the loss functions are nonconvex and gradients are dense. Further, there is currently a gap in the theoretical understanding of these methods, especially in the nonconvex and stochastic setting Ward et al. (2018) ; Zeng & Yin (2018) .

1.1. CONTENTS AND CONTRIBUTIONS:

In this paper, we develop and analyze a new consensus-based distributed adaptive moment estimation (DADAM) method that incorporates decentralized optimization and uses a variant of the adaptive moment estimation methods Duchi et al. (2011) ; Kingma & Ba (2014) ; Tieleman & Hinton; McMahan & Streeter (2010) . Existing distributed stochastic and adaptive gradient methods for deep learning are mostly designed for central network topology Dean et al. (2012) ; Li et al. . The main bottleneck of such a topology lies on the communication overload on the central node, since all nodes need to concurrently communicate with it. Hence, performance can be significantly degraded when network bandwidth is limited. These considerations motivate us to study an adaptive algorithm for network topologies, where all nodes can only communicate with their neighbors and none of the nodes is designated as "central". Therefore, the proposed method is suitable for large scale machine learning problems, since it enables both data parallelization and decentralized computation.

Next, we briefly summarize the main technical contributions of the work.

-Our first main result (Theorem 4) provides guarantees of DADAM for constrained convex minimization problems defined over a closed convex set X . We provide the convergence bound in terms of dynamic regret and show that when the data features are sparse and have bounded gradients, our algorithm's regret bound can be considerably better than the ones provided by standard mirror descent and gradient descent methods Nedic & Ozdaglar (2009) ; Hall & Willett (2015) ; Besbes et al. (2015) . It is worth mentioning that the regret bounds provided for adaptive gradient methods Duchi et al. (2011) are static and our results generalize them to dynamic settings.

-In Theorem 5, we give a new local regret analysis for distributed gradient-based algorithms for nonconvex minimization problems computed over a network of agents. Specifically, we prove that under certain regularity conditions, DADAM can achieve a local regret bound of order Õ( 1 T ) for nonconvex distributed optimization. To the best of our knowledge, rigorous extensions of existing adaptive gradient methods to the distributed nonconvex setting considered in this work do not seem to be available.

-In this paper, we also present regret analysis for distributed stochastic optimization problems computed over a network of agents. 

√:

nT + 1 T ) for nonconvex distributed stochastic optimization where ξ is an upper bound on the variance of the stochastic gradient. Hence, DADAM outperforms centralized adaptive algorithms such as ADAM for certain realistic classes of loss functions when T is sufficiently large.

The remainder of the paper is organized as follows. Section 2 gives a detailed description of DADAM, while Section 3 establishes its theoretical results. Section 4 explains a network correction technique for our proposed algorithm. Section 5 illustrates the proposed framework on a number of synthetic and real data sets. Finally, Section 6 concludes the paper.

The detailed proofs of the main results established are delegated to the Appendix.

1.2. MATHEMATICAL PRELIMINARIES AND NOTATIONS.:

Throughout the paper, R p denotes the real coordinate space of p dimensions. For any pair of vectors x, y ∈ R p , x, y indicates the standard Euclidean inner product. We denote the 1 norm by X 1 = ∑ i j |x i j |, the infinity norm by X ∞ = max i j |x i j |, and the Euclidean norm by X = ∑ i j |x i j | 2 . The above norms reduce to the vector norms if X is a vector. The diameter of the set X is given by

γ ∞ = sup x,y∈X

xy ∞ .

(1)

Let S p + be the set of all positive definite p × p matrices. Π X ,A [x] denotes the Euclidean projection of a vector x onto X for A ∈ S p + :

Π X ,A x = arg min y∈X A 1 2 (x -y) .

The subscript t is often used to denote the time step while y i,t,d stands for the d-th element of y i,t . Further, y i,1:t,d ∈ R t is given by

y i,1:t,d = [y i,1,d , y i,2,d , . . . , y i,t,d ] .

We let g i,t to denote the gradient of f at x i,t . The i-th largest singular value of matrix X is denoted by σ i (X). We denote the element in the i-th row and j-th column of matrix X by [X] i j . In several theorems, we consider a connected undirected graph G = (V , E ) with nodes V = {1, 2, . . . , n} and edges E . The matrix W ∈ R n×n is often used to denote a symmetric mixing matrix Shi et al. (2015) of graph G . Indeed, for any agent i, we assign a positive weight [W ] i j for the information received from agent j = i so that

[W ] i j > 0 if and only if (i, j) ∈ E and n ∑ i=1 [W ] i j = n ∑ j=1 [W ] i j = 1. (2)

The role of W is the similar as that in decentralized gradient descent methods Yuan et al. ( 2016) and average consensus Xiao et al. (2007) . It has a few common choices, which can significantly affect performance (see, Shi et al. (2015) ). Throughout this paper, we consider Metropolis constant edge weight matrix Xiao & Boyd (2004) ; Xiao et al. (2007) defined as follows

[ Ŵ ] i j =      1 max{deg(i),deg( j)}+ε , if (i, j) ∈ E , 0, if (i, j) / ∈ E and i = j, 1 -∑ k∈V [ Ŵ ] ik , if i = j,

for some small positive ε > 0. When Ŵ is chosen according this scheme, W = I+ Ŵ 2 is found to be very efficient. Note, the above doubly stochastic matrix implies uniqueness of σ 1 (W ) = 1 and warrants that other singular value of W are strictly less than one in magnitude.

The Hadamard (entrywise) and Kronecker product are denoted by and ⊗, respectively. Finally, the expectation operator is denoted by E.

2. PROBLEM FORMULATION AND ALGORITHM:

In this section, we propose a new optimization method for online optimization that employs data parallelization and decentralized computation over a network of agents. In our new structure, given a connected undirected graph G = (V , E ), we let each node i ∈ V at time t ∈ {1, . . . , T } hold its own measurement and training data m i , and set x) . We also let each agent i hold a local copy of the global variable x at time t ∈ {1, . . . , T }, which is denoted by x i,t ∈ R p . With this setup, we present a distributed adaptive gradient method for solving the minimization problem

EQUATION

EQUATION

where f i,t : X → R is a continuously differentiable mapping on the convex set X .

It is known that it is impossible to achieve a sub-linear dynamic regret bound Zhang et al. (2017) , because of the arbitrary fluctuations in problem equation 3. We want to characterize the hardness of the problem via a complexly measure that captures the pattern of the minimizer sequence {x * t } T t=1 , where x * t = arg min x∈X f t (x). Subsequently, we would like to provide a regret bound in terms of

EQUATION

which represents the variations in {x * t } T t=1 . DADAM uses a new distributed adaptive gradient method in which a group of n agents seeking to solve a sequential version of problem equation 3. Here, we assume that each component function f i,t : X → R becomes only available to agent i ∈ V , after having made its decision at time t ∈ {1, . . . , T }. In the t-th, the i-th agent chooses a point x i,t corresponding to what it considers the network as a whole should have selected. After committing to this choice, the agent has access to a cost function f i,t : X → R and the network cost is then given by f t (x) = 1 n ∑ n i=1 f i,t (x). Note that this function is not known to any of the agents and is not available at any single location.

The procedure of our proposed method is outlined in Algorithm 1.

Algorithm 1: A distributed adaptive moment estimation method (DADAM) input : x 1 ∈ X , a positive sequence {α t } T t=1 , exponential decay rates for the moment estimate β 1 , β 2 , β 3 ∈ [0, 1), and a mixing matrix W satisfying equation 2; 

1 for all i ∈ V , initialize moment vectors m i,0 = υ i,0 = υi,0 = 0 and x i,1 = x 1 ; 2 for i ∈ V do 3 for t ← 1 to T do 4 g i,t = ∇ f i,t (x i,t ); 5 m i,t = β 1 m i,t-1 + (1 -β 1 )g i,t ; 6 υ i,t = β 2 υ i,t-1 + (1 -β 2 )g i,t g i,t ; 7 υi,t = β 3 υ i,t + (1 -β 3 ) max( υi,t-1 , υ i,t ); 8 x i,t+ 1 2 = ∑ n j=1 [W ] i j x j,t ; 9 x i,t+1 = Π X , √ diag( υi,t ) x i,

υ i,t + (1 -β 3 ) max( υi,t-1 , υ i,t ),

for normalizing the running average of the gradient instead of υ i,t in ADAM and max( υi,t-1 , υ i,t ) in AMSGrad. The learning rate υi,t is an important component of the DADAM framework, since it enables us to develop a convergent adaptive method similar to AMSGrad while maintaining the efficiency of ADAM.

Next, we provide the notion of regret which measures the performance of DADAM against a sequence of local minimizers.

In the framework of online convex optimization, the performance of algorithms is assessed by regret that measures how competitive the algorithm is with respect to the best fixed solution Mateos-Núnez & Cortés (2014); Hazan et al. (2016) . However, the notion of regret fails to illustrate the performance of online algorithms in a dynamic setting. To overcome this issue, we consider a more stringent metric-dynamic regret Hall & Willett (2015); Besbes et al. (2015) ; Zinkevich (2003) , in which the cumulative loss of the learner is compared against the minimizer sequence {x * t } T t=1 , i.e.,

Reg C T := 1 n n ∑ i=1 T ∑ t=1 f i,t (x i,t ) - T ∑ t=1 f t (x * t ),

where x * t = arg min x∈X f t (x). On the other hand, in the framework of nonconvex optimization, it is usual to state convergence guarantees of an algorithm towards an ε-approximate stationary point-that is, there exist some iterates x i,t for which ∇ f t (x i,t ) ≤ ε. Inspired by Hazan et al. (2017) , we next provide the definition of projected gradient and introduce local regret, a new notion of regret which quantifies the moving average of gradients over network.

EQUATION

where

EQUATION

Then, the local regret of an online algorithm is given by

Reg N T := 1 n n ∑ i=1 min t∈{1,...,T } G X (x i,t , fi,t , α t ) 2 ,

where fi,t (x i,t ) = 1 t ∑ t s=1 f i,s (x i,t ) is an aggregate loss.

We analyze the convergence of DADAM as applied to minimization problem equation 3 using regrets Reg C T and Reg N T . It is worth to mention that DADAM is initialized at x i,t = 0 to keep the presentation of the convergence analysis clear. In general, any initialization can be selected for implementation purposes.

3. CONVERGENCE ANALYSIS:

In this section, our aim is to establish convergence properties of DADAM under the following assumptions: Assumption 2. For all i ∈ V and t ∈ {1, . . . , T }, the function f i,t (x) is continuously differentiable over X , and has Lipschitz continuous gradient on this set, i.e. there exists a constant ρ ≥ 0 so that

EQUATION

There are many cost functions f i,t that satisfy this type of Lipschitz condition. For example, it holds for any convex function on a compact set X , or for any polyhedral function on an arbitrary domain Duchi et al. (2011) .

Assumption 3. For all i ∈ V and t ∈ {1, . . . , T }, the stochastic gradient, as notation we use bold letters g = ∇ f i,t (x i,t ), satisfies

E ∇ f i,t (x i,t ) F t-1 = ∇ f i,t (x i,t ), E ∇ f i,t (x i,t ) 2 F t-1 ≤ ξ 2 ,

where F t is the ξ -field containing all information prior to the outset of round t + 1.

3.1. CONVEX CASE:

Next, we focus on the case where for all i ∈ V and t ∈ {1, . . . , T }, the agent i at time t has access to the exact gradient g i,t = ∇ f i,t (x i,t ). The following results apply to convex problems, as well as their stochastic variants in a dynamic environment.

Theorem 4. Suppose that the parameters

β 1 , β 2 ∈ [0, 1) satisfy η = β 1 √ β 2 < 1. Let β 1,t = β 1 λ t-1 , λ ∈ (0, 1) and ∇ f i,t (x) ∞ ≤ G ∞ for all t ∈ {1, . . . , T }.

Then, using a step-size α t = α √ t for the sequence x i,t generated by Algorithm 1, we have

Reg C T ≤ (1 -β 1 )α √ 1 + log T 2 √ n (1 -β 2 ) p ∑ d=1 g 1:T,d + p ∑ d=1 G ∞ γ ∞ (1 -β 1 )(1 -λ ) + p ∑ d=1 γ ∞ (γ ∞ + D T,d ) √ n(1 -β 1 )α T υT,d + 4α √ 1 + log T ∑ p d=1 g 1:T,d (1 -σ 2 (W )) (1 -β 1 ) (1 -η) (1 -β 2 )

.

Next, we analyze the stochastic convex setting and extend the result of Theorem 4 to the noisy case where agents have access to stochastic gradients of the objective function equation 3.

Theorem 5. Suppose that Assumption 3 holds. Further, the parameters

β 1 , β 2 ∈ [0, 1) satisfy η = β 1 √ β 2 < 1. Let β 1,t = β 1 λ t-1 , λ ∈ (0, 1).

Then, using a step-size α t = α √ t for the sequence x i,t generated by Algorithm 1, we have

E Reg C T ≤ (1 -β 1 )α √ 1 + log T 2 √ n (1 -β 2 ) p ∑ d=1 E g 1:T,d + p ∑ d=1 ξ γ ∞ (1 -β 1 )(1 -λ ) + p ∑ d=1 γ ∞ (γ ∞ + D T,d ) √ n(1 -β 1 )α √ T E υT,d + 4α √ 1 + log T ∑ p d=1 E g 1:T,d (1 -σ 2 (W )) (1 -β 1 ) (1 -η) (1 -β 2 )

. ) ) and the bound presented in Theorems 4 and 5, and thus the worst case dependence on T is √ T rather than √ T log T .

,d = p ∑ d=1 g 2 1,d + g 2 2,d + • • • + g 2 T,d ≤ p ∑ d=1 G 2 ∞ + G 2 ∞ + • • • + G 2 ∞ = p ∑ d=1 √ T G ∞ = pG ∞ √ T , p ∑ d=1 T υT,d ≤ p ∑ d=1 √ T G ∞ ≤ pG ∞ √ T . It is easy to show that the regret of DADAM is upper bounded by O( G ∞ D T √ T 1-σ 2 (W ) ) where D T = max d∈{1,...,p} D T,d . Indeed, the term ∑ T t=1 |g i,t,d |/ √ t in the proof of Lemma 14 can be bounded by O(G ∞ √ T ) instead of O(G ∞ √ T log T ). Hence, the regret of DADAM is upper bounded by minimum of O( G ∞ D T √ T 1-σ 2 (W

Remark 7. We note that in the static setting, i.e. D T = 0, the regret of DADAM is upper bounded by

O( G ∞ √ T 1 -σ 2 (W ) ),

where 1σ 2 (W ) is the spectral gap of the network.

3.2. NONCONVEX CASE:

In this section, we provide convergence guarantees for DADAM for the nonconvex minimization problem equation 3 defined over a closed convex set X . To do so, we use the projection map Π X instead of Π X , √ diag( vi,t ) for updating parameters x i,t for t ∈ {1, . . . , T }, i ∈ V (see, Algorithm 1 for details).

Next, we derive an upper and lower bound for the second moment of gradient, υ i,t . Assume that there exists a positive number ῡ such that max d∈{1,...,p} g i,t,d ≤ ῡ.

(8)

Now, using the update rule of υ i,t , we have

υ i,t = (1-β 2 ) ∑ t l=1 β t-l 2 g 2 i,l

. This together with equation 8 imply that

EQUATION

)

On the other hand, since υ i,t,d is non-decreasing and g i,t,d is required to be element-wise non-zero, we have

EQUATION

for some constant υ. Now, using the update rule of υi,t along with equation 10 and equation 9, we have

EQUATION

)

Theorem 8. Suppose that Assumption 2 holds. Further, the parameters

β 1 , β 2 ∈ [0, 1) satisfy η = β 1 √ β 2 < 1. Choose the positive sequence {α t } T t=1 such that 0 < α t ≤ (2-β 1 )υ 2 ρ ῡ with α t < (2-β 1 )υ 2 ρ ῡ

for at least one t. Then, for the sequence x i,t generated by Algorithm 1, we have

EQUATION

where

ϑ t = ∑ T t=1 [ (2-β 1 )α t 2 ῡ -ρα 2 t 2υ 2 ].

The following corollary shows that DADAM using a certain step-size leads to a near optimal regret bound for nonconvex functions.

Corollary 9. Under the same conditions of Theorem 8, using the step-sizes

α t = (2-β 1 )υ 2 2ρ ῡ

and

EQUATION

)

To complete the analysis of our algorithm in the nonconvex setting, we provide the regret bound for DADAM, when stochastic gradients are accessible to the learner.

Theorem 10. Suppose that Assumption 2 holds. Further, the parameters

β 1 , β 2 ∈ [0, 1) satisfy η = β 1 √ β 2 < 1. Choose the positive sequence {α t } T t=1 such that 0 < α t ≤ (2-β 1 )υ 2 ρ ῡ with α t < (2-β 1 )υ 2 ρ ῡ

for at least one t. Then, for the sequence x i,t generated by Algorithm 1, we have

EQUATION

where

ϑ t = ∑ T t=1 [ (2-β 1 )α t 2 ῡ -ρα 2 t 2υ 2 ].

3.2.1. WHEN DOES DADAM OUTPERFORM ADAM?:

We next theoretically justify the potential advantage of the proposed decentralized algorithm DADAM over centralized adaptive moment estimation methods such as ADAM . More specifically, the following corollary shows that when T is sufficiently large, the 1 T term will be dominated by the 1 √ nT term which leads to a 1 √ nT convergence rate. Corollary 11. Suppose that Assumptions 2 and 3 hold. Moreover, the parameters

β 1 , β 2 ∈ [0, 1) satisfy η = β 1 √ β 2 < 1 and β 1,t = β 1 λ t-1 , λ ∈ (0, 1). Choose the step-size sequence as α t = α √ nT with α = (2-β 1 )υ 2 ρ ῡ

. Then, for the sequence x i,t generated by Algorithm 1, we have

EQUATION

if the total number of time steps T satisfies

EQUATION

EQUATION

where

I 1 = υ 2 2(1 -η) 2 (1 -β 2 )(1 -λ )ξ 2 , I 2 = 16nL 2 υ 4 (1 -β 1 ) 2 (1 -η) 2 (1 -β 2 )(1 -σ 2 (W )) 2 ῡ2 ξ 4 .

Let ε-approximation solution of equation 3 be defined by

E[Reg N T ] T ≤ ε.

Corollary 11 indicates that the total computational complexity of DADAM to achieve an ε-approximation solution is bounded by O( 1 ε 2 ). The key setup which leads to this regret bound is that we do not use the boundedness assumption for domain or gradient. These assumptions can simplify the proof but lose some of the sophisticated structures in the problem.

4. AN EXTENSION OF DADAM WITH A CORRECTED UPDATE RULE:

Compared to classical centralized algorithms, decentralized algorithms encounter more restrictive assumptions and typically worse convergence rates. In order to improve the convergence rate, Shi et al. (2015) introduced a corrected decentralized gradient method called EXTRA, which has a linear rate of convergence if the objective function is strongly convex. Similar to EXTRA, we provide next a corrected update rule for i ∈ V , given by

EQUATION

)

where x i,t+1 is generated by Algorithm 1.

Our numerical results on some test problems show the efficiency and effectiveness of the corrected DADAM (C-DADAM) in practice. Note that a C-DADAM update is a DADAM update with a cumulative correction term. The summation in equation 17 is necessary, since each individual term ∑ n j=1 [W -Ŵ ] i j x j,s is asymptotically vanishing and the terms must work cumulatively.

5. NUMERICAL RESULTS:

In this section, we present numerical results for Algorithm 1 (DADAM), on both synthetic and real data sets. The employed data sets and the code will be made available on github upon acceptance of this manuscript. We first present numerical results on a sparse binary classification problem, comparing the performance of DADAM to that of standard decentralized gradient descent (DGD) Nedic & Ozdaglar (2009) ; Yuan et al. ( 2016) and its corrected version EXTRA Shi et al. (2015) .

In Section 5.2, we demonstrate the efficacy of DADAM in comparison with a decentralized ADAM (Algorithm 1 with β 3 = 1) and the recently proposed distributed federated averaging SGD (FedAvg) algorithm on benchmark datasets such as MNIST and CIFAR-10. All algorithms have been implemented in a Mac machine equipped with a 1.8 GHz Intel Core i5 processor and 8 GB 1600 MHz DDR3 of memory.

5.1. BINARY CLASSIFICATION BASED ON SYNTHETIC DATA IN THE STATIC ENVIRONMENT:

Consider the following online distributed learning setting: at each time t, m i randomly generated data points are given to every agent i in the form of (y t,i, j , x t,i, j ), where y t,i, j ∈ {±1} is a binary label and x t,i, j ∈ R p is a feature vector for j = 1, . . . , m i . Our goal is to train a classifier θ ∈ R p such that for an arbitrary new feature vector x it assigns ŷ = sign( θ , x ). We design decentralized convex logistic and nonconvex sigmoid loss functions with 2 regularization as follows:

EQUATION

)

where ν = 1/m i and m i is the size of the training data on node i.

For X , we consider the 1 ball X 1 = {θ ∈ R p : θ 1 ≤ r}, when a sparse classifier is preferred.

We show results for the decentralized problem solved by DADAM, C-DADAM, DGD and EXTRA over a medium-scale network. We have implemented EXTRA and DGD with their default settings. The connected network is randomly generated with n = 50 agents and connectivity ratio r = 0.2. Each agent holds 10 samples, i.e., m i = 10, ∀i ∈ V . The agents shall collaboratively obtain p coefficients via loss functions equation 18. All samples are randomly generated, and the reference classifier θ * is pre-computed using a centralized method. As it is easy to implement in practice, we We compare the accuracy of DADAM with that of the decentralized ADAM (Algorithm 1 with β 3 = 1 ) and the Federated Averaging (FedAvg) algorithm McMahan et al. (2016) which also performs data parallelization without decentralized computation. The connected network is generated with n = 5 agents and connectivity ratio r = 0.8. The parameters for DADAM and ADAM are selected in a way similar to the previous experiments. In our implementation, we use same number of agents and choose E = C = 1 as the parameters in the FedAvg algorithm since it is close to a connected topology scenario as considered in the DADAM and ADAM. It can be easily seen from Figure 2 that DADAM and the decentralized ADAM (β 3 = 1) outperform FedAvg. Further, DADAM can achieve high accuracy in comparison with the decentralized ADAM and FedAvg.

6. CONCLUSION:

A decentralized adaptive moment estimation method DADAM was proposed for the distributed learning of deep networks based on adaptive moment of first and second moment of estimations. Convergence properties of the proposed algorithm were established for convex (and dynamic) and nonconvex functions in both stochastic and deterministic settings. Numerical results on some datasets show the efficiency and effectiveness of the new proposed method in practice. Lijun Zhang, Tianbao Yang, Jinfeng Yi, Jing Rong, and Zhi-Hua Zhou. Improved dynamic regret for non-degenerate functions. In Advances in Neural Information Processing Systems, pp. 732-741, 2017.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. 2003.

Next, we establish a series of lemmas used in the proof of main theorems.

6.1. PROPERTIES OF DADAM:

Lemma 12. Beck & Teboulle (2003) Let X be a nonempty closed convex set in R p . Then, for any d ∈ X , we have

x * -d, a ≤ 1 2 d -c 2 - 1 2 d -x * 2 - 1 2 x * -c 2 , where x * = arg min x∈X { a, x + 1 2 x -c 2 }. Lemma 13. McMahan & Streeter (2010) For any A ∈ S p + and convex feasible set C ⊂ R p , suppose a 1 = Π C,A [b 1 ], a 2 = Π C,A [b 2 ], we have A 1 2 (a 1 -a 2 ) ≤ A 1 2 (b 1 -b 2 ) . Lemma 14. For all i ∈ V if β 1 , β 2 ∈ [0, 1) satisfy η = β 1 √ β 2 < 1, then we have T ∑ t=1 p ∑ d=1 α t m 2 i,t,d υi,t,d ≤ α √ 1 + log T (1 -β 1 )(1 -η) (1 -β 2 ) p ∑ d=1 g i,1:T,d ,

where α t = α √ t for all t ∈ {1, . . . , T }.

Proof. Using the update rules in Algorithm 1, we have

T ∑ t=1 αm 2 i,t,d t υi,t,d = T -1 ∑ t=1 αm 2 i,t,d t υi,t,d + α T m 2 i,T,d (1 -β 3 ) max{ υi,T-1,d , υ i,T,d } + β 3 υ i,T,d ≤ T -1 ∑ t=1 αm 2 i,t,d t υi,t,d + α T m 2 i,T,d (1 -β 3 )υ i,T,d + β 3 υ i,T,d (i) = T -1 ∑ t=1 αm 2 i,t,d t υi,t,d + α(∑ T l=1 (1 -β 1 )β T -l 1 g i,l,d ) 2 T ∑ T l=1 (1 -β 2 )β T -l 2 g 2 i,l,d (ii) ≤ T -1 ∑ t=1 αm 2 i,t,d t υi,t,d + α T (1 -β 2 ) (∑ T l=1 β T -l 1 )(∑ T l=1 β T -l 1 g 2 i,l,d ) ∑ T l=1 β T -l 2 g 2 i,l,d (iii) ≤ T -1 ∑ t=1 αm 2 i,t,d t υi,t,d + α (1 -β 1 ) T (1 -β 2 ) T ∑ l=1 β T -l 1 g 2 i,l,d β T -l 2 g 2 i,l,d ≤ T -1 ∑ t=1 αm 2 i,t,d t υi,t,d + α (1 -β 1 ) T (1 -β 2 ) T ∑ l=1 η T -l |g i,l,d |,

where (i) follows from the fact that the update rules of m T and υ T can be written as

m T = (1 - β 1 ) ∑ T l=1 β T -l 1 g l and υ T = (1 -β 2 ) ∑ T l=1 β T -l 2

g 2 l , respectively. (ii) follows from Cauchy-Schwarz inequality and the fact that 0

≤ β 1 < 1. Inequality (iii) follows since ∑ T l=1 β T -l 1 ≤ 1 1-β 1

. Hence, we have

T ∑ t=1 αm 2 i,t,d (1 -β 1 ) t υi,t,d ≤ T ∑ t=1 α (1 -β 1 ) t(1 -β 2 ) t ∑ l=1 η t-l |g i,l,d | = α (1 -β 1 ) (1 -β 2 ) T ∑ t=1 1 √ t t ∑ l=1 η t-l |g i,l,d | = α (1 -β 1 ) (1 -β 2 ) T ∑ t=1 |g i,t,d | T ∑ l=t η l-t √ l ≤ α (1 -β 1 ) (1 -β 2 ) T ∑ t=1 |g i,t,d | T ∑ l=t η l-t √ t (i) ≤ α (1 -β 1 ) (1 -β 2 ) T ∑ t=1 |g i,t,d | 1 (1 -η) √ t (ii) ≤ α (1 -β 1 )(1 -η) (1 -β 2 ) g i,1:T,d T ∑ t=1 1 t (iii) ≤ α √ 1 + log T (1 -β 1 )(1 -η) (1 -β 2 ) g i,1:T,d ,

where inequality (i) follows since ∑ T l=t η l-t ≤ 1 (1-η) . Inequality (ii) follows from Cauchy-Schwarz inequality. The inequality (iii) follows since

EQUATION

Next, we provide an upper bound on the deviation of the local estimates at each iteration from their consensual value. A similar result has been proven in Shahrampour & Jadbabaie (2017) for online decentralized mirror descent; however, the following lemma extends that of Shahrampour & Jadbabaie (2017) to the online adaptive setting and takes into account the sparsity of gradient vector.

Lemma 15. ( Network Error with Sparse Data) If

β 1 , β 2 ∈ [0, 1) satisfy η = β 1 √ β 2

< 1, then the sequence x i,t generated by Algorithm 1 satisfies

T ∑ t=1 n ∑ i=1 1 α t V 1 4 i,t ( xt -x i,t ) 2 ≤ n √ nα √ 1 + log T ∑ p d=1 g 1:T,d (1 -σ 2 (W )) 2 (1 -β 1 )(1 -η) (1 -β 2 ) ,

where Vi,t = diag( υi,t ) and xt = 1 n ∑ n i=1 x i,t .

Proof. Let e i,t := x i,t+1 -∑ n j=1 [W ] i j x j,t , where W satisfies equation 2. Using the update rule of x i,t+1 in Algorithm 1, we have

EQUATION

where the first inequality follows from Lemma 13.

Further, from the definition of e i,t , we have

EQUATION

Now, from equation 2 and equation 21, we have

xt+1 = 1 n n ∑ i=1 x i,t+1 = 1 n n ∑ i=1 n ∑ j=1 [W ] i j x j,t + 1 n n ∑ i=1 e i,t = 1 n n ∑ j=1 ( n ∑ i=1 [W ] i j )x j,t + 1 n n ∑ i=1 e i,t = xt + ēt ,

where ēt = 1 n ∑ n i=1 e i,t . Hence,

EQUATION

)

It follows from equation 21 that

EQUATION

Now, using equation 23 and equation 22, we have

EQUATION

Now, taking the Euclidean norm of equation 24 and summing over t ∈ {1, . . . , T }, one has:

EQUATION

where step (i) follows from ∑ n i=1 a i 2 ≤ n ∑ n i=1 a i 2 , step (ii) follows from the following property of mixing matrix W Horn et al. (1990) ,

EQUATION

step (iii) follows from ∑ T t=1 tσ t 2 (W ) < 1

(1-σ 2 (W )) 2 , step (iv) follows from equation 20 and step (v) follows from Lemma 14. Now, summing the equation 25 over i ∈ V and using

EQUATION

we complete the proof.

Lemma 16. For the sequence x i,t generated by Algorithm 1, we have

EQUATION

)

Proof. From the left side of equation 28, we have υi,t

EQUATION

)

By construction of equation 29, we have υi,t+1

EQUATION

where the last inequality holds due to equation 1.

Summing equation 30 over t ∈ {1, . . . , T }, the first term telescopes, while the second is handled with equation 31. Hence,

EQUATION

where the last inequality follows since from definition of υi,t , we have υi,t+1,d

EQUATION

Now, summing equation 32 over i ∈ V and using the inequality ∑ n i=1 υi,t ≤ √ n √ υt , the claim in equation 28 follows.

Lemma 17. Suppose that the parameters

β 1 , β 2 ∈ [0, 1) satisfy η = β 1 √ β 2 < 1. Let β 1,t = β 1 λ t-1 , λ ∈ (0, 1) and ∇ f i,t (x) ∞ ≤ G ∞

for all t ∈ {1, . . . , T }. Then, using a step-size α t = α √ t for the sequence x i,t generated by Algorithm 1, we have

1 n n ∑ i=1 T ∑ t=1 f i,t (x i,t ) -f i,t (x * t ) ≤ (1 -β 1 )α √ 1 + log T 2 (1 -β 2 ) p ∑ d=1 g 1:T,d + p ∑ d=1 G ∞ γ ∞ (1 -β 1 )(1 -λ ) + 2α √ 1 + log T ∑ p d=1 g 1:T,d (1 -σ 2 (W )) (1 -β 1 ) (1 -η) (1 -β 2 ) + γ 2 ∞ √ n p ∑ d=1 T υT,d (1 -β 1 )α + γ ∞ √ n p ∑ d=1 T υT,d (1 -β 1 )α T -1 ∑ t=1 |x * t+1,d -x * t,d |.

Proof. From convexity of f i,t (•), we have

EQUATION

)

Individual terms in equation 34 can be bounded in the following way. From the Young's inequality for products 3 , we have

EQUATION

)

Note also that:

T ∑ t=1 α t g 2 i,t,d υi,t,d = T -1 ∑ t=1 α t g 2 i,t,d υi,t,d + α T g 2 i,T,d υi,T,d = T -1 ∑ t=1 α t g 2 i,t,d υi,t,d + αg 2 i,T,d T (1 -β 2 ) ∑ T l=1 β T -l 2 g 2 i,l,d ≤ T -1 ∑ t=1 α t g 2 i,t,d υi,t,d + α|g i,T,d | T (1 -β 2 ) .

3 An elementary case of Young's inequality is ab

≤ a 2 2 + b 2 2 .

Under review as a conference paper at ICLR 2019 Using Cauchy-Schwarz inequality, equation 19 and equation 27, we have

EQUATION

Using equation 35 and equation 36, we have

EQUATION

In addition, we have

I 2 = 1 n n ∑ i=1 T ∑ t=1 g i,t , x i,t - n ∑ j=1 [W ] i j x j,t = 1 n n ∑ i=1 T ∑ t=1 g i,t , x i,t -xt + xt - n ∑ j=1 [W ] i j x j,t = 1 n n ∑ i=1 T ∑ t=1 g i,t , x i,t -xt + 1 n n ∑ i=1 T ∑ t=1 n ∑ j=1 [W ] i j g i,t , xt -x j,t = 1 n n ∑ i=1 T ∑ t=1 √ α t V -1 4 i,t g i,t , V 1 4 i,t √ α t (x i,t -xt ) + 1 n n ∑ i=1 T ∑ t=1 n ∑ j=1 [W ] i j √ α t V -1 4 i,t g i,t , V 1 4 i,t √ α t ( xt -x j,t ) ≤ 1 n n ∑ i=1 T ∑ t=1 α t V -1 2 i,t g 2 i,t T ∑ t=1 V 1 2 i,t α t (x i,t -xt ) 2 + 1 n n ∑ i=1 n ∑ j=1 [W ] i j T ∑ t=1 α t V -1 2 i,t g 2 i,t T ∑ t=1 V 1 2 i,t α t (x j,t -xt ) 2 (38)

where equation 38 follows from Cauchy-Schwarz inequality. Now, using equation 38 we obtain

EQUATION

)

where equation 39 utilize Cauchy-Schwarz inequality and equation 40 follows from equation 2, Lemma 15 and equation 36.

To bound I 1 , using the update rule of mi,t in Algorithm 1, we have

α t m i,t υi,t , x i,t+1 -x * t = α t ( β 1,t υi,t m i,t-1 + (1 -β 1,t ) υi,t ∇ f i,t (x i,t )), x i,t+1 -x * t = α t β 1,t υi,t m i,t-1 , x i,t+1 -x * t + α t (1 -β 1,t ) υi,t ∇ f i,t (x i,t ), x i,t+1 -x * t .

Now, by rearranging the above equality, we obtain:

EQUATION

where (II 1 ) follows from Lemma 12. The term (II 2 ) is obtained by induction as follows: using the assumption, g i,t ∞ ≤ G ∞ ; now, using the update rule of m i,t in Algorithm 1, we have

EQUATION

where m i,t-1 ∞ ≤ G ∞ by induction hypothesis.

Note that from our assumption, we have β 1,t = β 1 λ t-1 , λ ∈ (0, 1) and β 1,t ≤ β 1 . Substituting this into equation 41, we get

EQUATION

)

where the last line comes from equation 1.

Plugging equation 37, equation 40, and equation 43 into equation 34, we obtain

EQUATION

Finally, using Lemma 16 and equation 44, we obtain the desired result.

6.1.1 PROOF OF THEOREM 4

Proof. From convexity of f i,t (•) it follows that

EQUATION

EQUATION

Summing over t ∈ {1, . . . , T } and i ∈ V , and applying Lemma 17 and equation 40 gives the desired result.

6.1.2. PROOF OF THEOREM 5:

Proof. We just need to rework the proof of Lemma 17 and Theorem 4 using stochastic gradients by tracking the changes. Indeed, using stochastic gradient at the beginning of Lemma 17, we have

T ∑ t=1 f i,t (x i,t ) -f i,t (x * t ) ≤ T ∑ t=1 ∇ f i,t (x i,t ), x i,t -x * t = T ∑ t=1 ∇ f i,t (x i,t ), x i,t -x * t + T ∑ t=1 ∇ f i,t (x i,t ) -∇ f i,t (x i,t ), x i,t -x * t = T ∑ t=1 ∇ f i,t (x i,t ), x i,t+1 -x * t + T ∑ t=1 ∇ f i,t (x i,t ), x i,t - n ∑ j=1 [W ] i j x j,t + T ∑ t=1 ∇ f i,t (x i,t ), n ∑ j=1 [W ] i j x j,t -x i,t+1 + T ∑ t=1 ∇ f i,t (x i,t ) -∇ f i,t (x i,t ), x i,t -x * t .

Further, if we replace any bound involving G ∞ which is an upper bound on the exact gradient with the norm of stochastic gradient, we obtain

EQUATION

Note that using Assumption 3, we have

EQUATION

E [ ∇ f i,t (x i,t ) ] ≤ E ∇ f i,t (x i,t ) 2 1 2 ≤ ξ .

Hence taking expectation from equation 46, using the above inequality and equation 42, we have

1 n n ∑ i=1 T ∑ t=1 E [ f i,t (x i,t )] -f i,t (x * t ) ≤ (1 -β 1 )α √ 1 + log T 2 √ n (1 -β 2 ) p ∑ d=1 E g 1:T,d + γ ∞ ξ T ∑ t=1 p ∑ d=1 β 1,t (1 -β 1,t ) + 2α √ 1 + log T ∑ p d=1 E g 1:T,d (1 -σ 2 (W )) (1 -β 1 ) (1 -η) (1 -β 2 ) + 1 n n ∑ i=1 1 2(1 -β 1 ) E υi,t α t x * t - n ∑ j=1 [W ] i j x j,t 2 - υi,t α t x * t -x i,t+1 2 .

According to the result in Lemma 16 and equation 44, we have

1 n n ∑ i=1 T ∑ t=1 E [ f i,t (x i,t )] -f i,t (x * t ) ≤ (1 -β 1 )α √ 1 + log T 2 √ n (1 -β 2 ) p ∑ d=1 E g 1:T,d + p ∑ d=1 ξ γ ∞ (1 -β 1 )(1 -λ ) + 2α √ 1 + log T ∑ p d=1 E g 1:T,d (1 -σ 2 (W )) (1 -β 1 ) (1 -η) (1 -β 2 ) + γ 2 ∞ √ n p ∑ d=1 √ T E υT,d (1 -β 1 )α + γ ∞ √ n p ∑ d=1 √ T E υT,d (1 -β 1 )α T -1 ∑ t=1 |x * t+1,d -x * t,d |.

Finally, using equation 45 we complete the proof.

Next, we establish a series of lemmas used in the proof of Theorems 8 and 10. Lemma 18. Attouch et al. ( 2013) Let G X be the projected gradient defied in equation 5. Then, G X (ϖ, f i , α) = 0 if and only if ϖ is a critical point of equation 3.

Next, we show that the projection map G X in Definition 1 is Lipschitz continuous. Lemma 19. Suppose the second moment υ satisfies equation 11. Let x + 1,i and x + 2,i be given in Definition 1 with m i √ υi replaced by

m 1 i √ υ1 i and m 2 i √ υ2 i , respectively. Then, G 1 X (x i , fi , α) -G 2 X (x i , fi , α) ≤ ῡ υ m 1 i -m 2 i , ∀i ∈ V ,

where G 1 X and G 2 X are the projection maps corresponding to x + 1,i and x + 2,i , respectively.

Proof. Consider the optimality condition of equation 6, for any u ∈ X . For each i ∈ V , observe that

EQUATION

EQUATION

)

Taking u = x + 2,i in equation 47, we have

EQUATION

Likewise, setting u = x + 1,i in equation 48, we get

EQUATION

Now, using equation 49, equation 50 and equation 11, we obtain

EQUATION

Without loss of generality, assumes that υ1 i ≤ υ2 i . Then, using equation 5, we have

G 1 X (x i , fi , α) -G 2 X (x i , fi , α) = υ1 i α (x -x + 1,i ) - υ2 i α (x -x + 2,i ) ≤ υ1 i α (x -x + 1,i ) - υ1 i α (x -x + 2,i ) (i) ≤ ῡ α x + 2,i -x + 1,i (ii) ≤ ῡ υ m 1 i -m 2 i ,

where (i) follows from equation 11 and (ii) follows from equation 51.

Lemma 20. Let β 1 , β 2 ∈ [0, 1) satisfy η = β 1 √ β 2 < 1.

Then, for the sequence x i,t generated by

Algorithm 1, we have

1 n n ∑ i=1 x i,t -x * t ≤ 2 √ n (1 -η) (1 -β 2 ) t-1 ∑ s=0 α s σ t-s-1 2 (W ) := B t .

Proof. In the proof of Lemma 15 we showed that

x i,t+1 -xt+1 = t ∑ s=0 n ∑ j=1 ([W t-s ] i j - 1 n )e i,s .

Now, using the above equality, we have

EQUATION

Also, using Lemma 14, we get

EQUATION

)

The above inequality together with the update rule of x i,t+1 , imply that

EQUATION

where the first inequality follows from the nonexpansiveness property of the Euclidean projection4 .

Substituting equation 26 and equation 54 into equation 52, we get

x i,t -xt ≤ √ n (1 -η) (1 -β 2 ) t-1 ∑ s=0 α s σ t-s-1 2 (W ).

Summing the above inequality over:

i ∈ V , we conclude that n ∑ i=1 x i,t -x * t ≤ n ∑ i=1 x i,t -xt + n ∑ i=1 xt -x * t ≤ 2n √ n (1 -η) (1 -β 2 ) t-1 ∑ s=0 α s σ t-s-1 2 (W ).

Lemma 21. For the sequence x i,t generated by Algorithm 1, we have

∇ fi,t , G X (x i,t , fi,t , α t ) ≥ (2 -β 1,t ) 2(1 -β 1,t ) G X (x i,t , fi,t , α t ) 2 - β 1,t υi,t 2(1 -β 1,t )(1 -η) 2 (1 -β 2 )

.

Proof. A quick look at optimality condition of equation 6 verifies that

m i,t υi,t + 1 α t (x i,t+1 - n ∑ j=1

[W ] i j x j,t ), zx i,t+1 ≥ 0, ∀z ∈ X .

Substituting z = x i,t into the above inequality, we get

m i,t υi,t , x i,t -x i,t+1 ≥ 1 α t x i,t+1 - n ∑ j=1

[W ] i j x j,t , x i,t+1x i,t .

Now using equation 2, we have υi,t

α t x i,t -x i,t+1 2 ≤ m i,t , x i,t -x i,t+1 = β 1,t m i,t-1 + (1 -β 1,t )∇ fi,t , x i,t -x i,t+1 = β 1,t υi,t-1 α t α t m i,t-1 υi,t-1 , x i,t -x i,t+1 + (1 -β 1,t ) ∇ fi,t , x i,t -x i,t+1 ≤ β 1,t υi,t 2α t ( α 2 t (1 -η) 2 (1 -β 2 ) + x i,t -x i,t+1 2 ) + (1 -β 1,t ) ∇ fi,t , x i,t -x i,t+1 ,

where the first equality follows from the update rule of m i,t . The last inequality is valid since Cauchy-Schwarz inequality and inequality υi,t-1 β 1,t ≤ υi,t β 1 . The claim then follows after using Definition 1.

6.1.3. PROOF OF THEOREM 8:

Proof. Let fi,t (x i,t ) = 1 t ∑ t s=1 f i,s (x i,t ). Using Assumption 2, Taylor's expansion and Definition 1, we have fi,t (x i,t+1 ) ≤ fi,t (x i,t ) + ∇ fi,t (x i,t ), x i,t+1 -

x i,t + ρ 2 x i,t+1 -x i,t 2 ≤ fi,t (x i,t ) - α t υi,t ∇ fi,t (x i,t ), G X (x i,t , fi,t , α t ) + ρα 2 t 2 υi,t G X (x i,t , fi,t , α t ) 2 .

The above inequality together with Lemma 21, equation 11 and

EQUATION

)

Let ∆ i,t = fi,t (x i,t ) -fi,t (x * t ) denotes the instantaneous loss at round t. We have

EQUATION

Note that (I) can be bounded as follows:

EQUATION

where the first inequality is due to the optimality of x * t and the second inequality follows from equation 55. Now, combining equation 57 and equation 56 gives

∆ i,t+1 ≤ t t + 1 (∆ i,t -[ (2 -β 1 )α t 2 ῡ - ρα 2 t 2υ 2 ] G X (x i,t , fi,t , α t ) 2 + α t β 1,t ῡ 2(1 -β 1,t )(1 -η) 2 (1 -β 2 ) ) + 1 t + 1 ( f i,t+1 (x i,t+1 ) -f i,t+1 (x * t+1 )).

By rearranging the above inequality, we have

t t + 1 [ (2 -β 1 )α t 2 ῡ - ρα 2 t 2υ 2 ] G X (x i,t , fi,t , α t ) 2 ≤ t t + 1 (∆ i,t + α t β 1,t ῡ 2(1 -β 1,t )(1 -η) 2 (1 -β 2 ) ) + 1 t + 1 ( f i,t+1 (x i,t+1 ) -f i,t+1 (x * t+1 )) -∆ i,t+1 . (58)

Using the definition of ∆ i,t+1 , we get 1

EQUATION

)). Hence, using equation 58 and simplifying terms, we obtain

EQUATION

Note that:

EQUATION

where the first equality uses fi,t (x i,t ) -fi,t-1

(x i,t ) = t -1 ( f i,t (x i,t ) -fi,t-1 (x i,t )).

The first inequality follows from Assumption 2. The second inequality follows from Lemma 20 and equation 19.

Summing equation 59 over i ∈ V , t ∈ {1, . . . , T } and using equation 62, we have

EQUATION

) Note that ∑ T t=1 [ (2-β 1 )α t 2 ῡ -ρα 2 t 2υ 2 ] > 0.

Therefore, dividing both sides of the equation 63 by

∑ T t=1 [ (2-β 1 )α t 2 ῡ -ρα 2 t

2υ 2 ], we obtain equation 14.

6.1.4 PROOF OF COROLLARY 9

Proof. With the constant step-sizes α t = (2-β 1 )υ 2 2ρ ῡ for all t ∈ {1, . . . , T }, we have

EQUATION

Therefore, using the above equality and equation 12 together with α t = (2-β 1 )υ 2 2ρ ῡ for all t ∈ {1, . . . , T }, we obtain 1

EQUATION

and

EQUATION

)

Combine the results in equation 64 and equation 65, we have equation 13.

6.1.5. PROOF OF THEOREM 10:

Proof. Let

δ i,t = ∇ fi,t (x i,t ) -∇ fi,t (x i,t ), ∀t ≥ 1, where ∇ fi,t (x i,t ) = 1 t ∑ t s=1 ∇ f i,s (x i,t

) denotes the minibatch stochastic gradient on node i. Since f i,t is ρ-smooth, it follows that fi,t is also ρ-smooth. Hence for every i ∈ V and t ∈ {1, . . . , T }, we have fi

,t (x i,t+1 ) ≤ fi,t (x i,t ) + ∇ fi,t (x i,t ), x i,t+1 -x i,t + ρ 2 x i,t+1 -x i,t 2 ≤ fi,t (x i,t ) - α t υi,t ∇ fi,t (x i,t ), G X (x i,t , fi,t , α t ) + ρα 2 t 2 υi,t G X (x i,t , fi,t , α t ) 2 = fi,t (x i,t ) - α t υi,t ∇ fi,t (x i,t ), G X (x i,t , fi,t , α t ) + ρα 2 t 2 υi,t G X (x i,t , fi,t , α t ) 2 + α t υi,t δ i,t , G X (x i,t , fi,t , α t ) ,

where G X (x i,t , fi,t , α t ) denotes the projected stochastic gradient on node i.

The above inequality together with Lemma 21, equation 11 and

EQUATION

where the second inequality follows from the Cauchy-Schwartz inequality. The last inequality follows from the fact that β

1,t = β 1 λ t-1 , λ ∈ (0, 1) and Lemma 19 if we set G 1 X (x i , fi , α) = G X (x i,t , fi,t , α t ), and G 2 X (x i , fi , α) = G X (x i,t , fi,t , α t ). Let ∆ i,t = fi,t (x i,t ) -fi,t (x *

t ) denotes the instantaneous stochastic loss at time t. We have

EQUATION

Observe that (I) can be bounded as follows:

EQUATION

where the first inequality is due to x * t+1 ∈ X and the optimality of x * t and the second inequality is due to equation 66 and the fact that 0 ≤ β 1,r < 1. Thus, using equation 67 and equation 68, we get

∆ i,t+1 ≤ t t + 1 ∆ i,t -[ (2 -β 1 )α t 2 ῡ - ρα 2 t 2υ 2 ] G X (x i,t , fi,t , α t ) 2 + α t β 1,t ῡ 2(1 -β 1,t )(1 -η) 2 (1 -β 2 ) + α t υi,t δ i,t , G X (x i,t , fi,t , α t ) + ῡα t υ 2 t ∑ r=1 β t-r 1 δ i,r δ i,t + 1 t + 1 ( f i,t+1 (x i,t+1 ) -f i,t+1 (x * t+1 )).

By rearranging the above inequality, we have

EQUATION

Now, using definition of ∆

EQUATION

)), together with equation 69, we obtain

EQUATION

Note that from Assumption 3, we have E δ i,t , G X (x i,t , fi,t , α t ) = 0 and 

E ∇ f i,t (x i,t ) -∇ f i,t (x i,t ) 2 = E ∇ f i,t (x i,t ) 2 -E [∇ f i,t (x i,t )] 2 ≤ E ∇ f i,

EQUATION

where the last inequality follows from equation 60, equation 61 and equation 71.

We proceed to bound RHS of equation 75. By substituting α t = α √ nT and β 1,t = β 1 λ t-1 , λ ∈ (0, 1) into the last two terms of equation 75, we have

EQUATION

EQUATION

Further, using equation 75 and Lemma 20, we get

EQUATION

)

Similarly, using equation 75 and Lemma 20, we have

EQUATION

where the second inequality is due to Cauchy-Schwarz inequality and the last inequality follows because ∑ T t=2 1 t 2 ≤ 1. Using equation 16a, equation 77, equation 78 and equation 79 are bounded by equation 76. This completes the proof.

6.2. SENSITIVITY OF DADAM TO ITS PARAMETERS:

Next, we examine the sensitivity of DADAM on the parameters related to the network connection and update of the moment estimate. As it is clear from Figure 3 the convergence of training accuracy value happens faster for sparser networks (higher σ 2 (W )). This is similar to the trend observed for FedAvg algorithm while reducing parameter C which makes the agent interaction matrix sparser. This is also expected as discussed in Theorems 4 and 5. Note that with the availability of a central parameter server (as in FedAvg algorithm), sparser topology may be useful for a faster convergence, however, topology density of graph is important for a distributed learning scheme with decentralized computation on a network. We also empirically evaluate the effect of the β 3 in Algorithm 1. We consider a range of hyperparameter choices, i.e. β 3 ∈ {0, 0.9, 0.99}. From Figure 4 it can be easily seen that DADAM per- 

:

http://www.math.ucla.edu/ wotaoyin/software.html

http://www.tensorflow.org/tutorials

Π X [x 1 ] -Π X [x 2 ] ≤ x 1x 2 , ∀x 1 , x 2 ∈ R p .

Figures and Tables:

Theorems 4 and 5 show that the regret bound of DADAM can be considerably better than the ones provided by standard mirror descent and gradient descent methods for both centralized and decentralized settings Zinkevich (2003); Hall & Willett (2015); Besbes et al. (2015

use the Metropolis constant edge weight matrixW Shi et al. (2015)  in our experimental evaluation 1 . We have tested the DADAM and C-DADAM with β 1 = β 2 = β 3 = 0.9. The numerical results are illustrated in Figure1. It can be seen that DADAM and C-DADAM outperform DGD and EXTRA, showing almost linear convergence in the convex setting to the reference logistic classifier x * .5.2 CIFAR AND MNIST EXPERIMENTSNext, we present the experimental results using the CIFAR-10 image recognition dataset and the MNIST digit recognition task.The model architecture for training CNN to classify CIFAR10 dataset was taken from the TensorFlow tutorial 2 . Also, the model for training a simple multilayer percepton (MLP) on the MNIST dataset was taken from https://github.com/keras-team/keras/blob/master.

Figure 1: The residual θ t -θ * θ 0 -θ * on iteration (time step t ∈ {1, . . . , T }) for binary classification problem using convex and nonconvex (with α/ √ t) loss functions. Constant α = σ n /ρ is the theoretical critical step-size given for DGD Yuan et al. (2016). (a) classification with p = 10 and ν = 0. (b) classification with p = 20 and ν = 0. (c) classification with p = 20 and ν = 1/m i .

Figure 2: Training loss and accuracy over 15 epochs. (a) MNIST digit recognition task (b) CIFAR10 image recognition dataset.

Figure 3: Performance of the DADAM algorithm with varying network topology: training loss and accuracy over 30 epochs based on the MNIST digit recognition library.

Figure 4: Performance of the DADAM algorithm with varying decay rate β 3 . DADAM1 (β 3 = 0), DADAM2 (β 3 = 0.9), and DADAM3 (β 3 = 0.99) for training loss and accuracy over 30 epochs based on the MNIST digit recognition library. forms equal or better than AMSGrad (β 3 = 0), regardless of the hyper-parameter setting for β 1 and β 2 .

Theorems 8 and 10 provide regret bounds of DADAM for minimization problem equation 3 with stochastic gradients and indicate that the result of Theorems 4 and 5 holds true in expectation. Further, in Corollary 11 we show that DADAM can achieve a local regret bound of order O( ξ 2

Table content: <table/>

t (x i,t ) 2 ≤ ξ 2 , which implies that E δ i,t 2 = E ∇ fi,t (x i,t ) -∇ fi,t (x i,t ) 2 i,s (x i,t ) -∇ f i,s (x i,t )) 2 β 1 )α t := A 1 + A 2 . Now, if A 1 , A 2 ≤ 1 2 , then α t ≤ υ 2 (2-β 1 )2ρ ῡ , and α t ≥ 2 ῡ T (2-β 1 ) , respectively. These bounds together with our assumption on α t give equation 74. Now, let equation 74 holds. Using equation 70, and since ϑ t ≥ 1/2, we haveE G X (x i,t , fi,t , α t ) 2 E G X (x i,t , fi,t , α t ) 2 X (x i,t , fi,t , α t ) 2 (x i,1 )f i,1 (x * 1 ) + L x i,T +1x * T +1 +

Table content: <table><tr><td colspan="13">By rearranging the above inequality, we obtain</td><td/></tr><tr><td colspan="15">1 ≥ T (2 1 ρα t ῡ υ 2 (2 -β 1 ) + n T n ∑ i=1 min t∈{1,...,T }</td><td>ῡ</td></tr><tr><td>≤</td><td>2 T n</td><td colspan="2">n ∑ i=1</td><td colspan="4">min t∈{1,...,T }</td><td/><td/><td/><td/><td/><td/><td>T ∑ t=1 [</td><td>(2 -β 1 )α t 2 ῡ</td><td>-</td><td>ρα 2 t 2υ 2 ]</td></tr><tr><td>≤ ≤</td><td colspan="14">1 ρα 2 t 2υ 2 ] = E -t t ≤ 1 f i,1 T (2 -β 1 )α t 2 ῡ 1 n i=1 T ∑ t=1 [ T n 2 T 2 n ∑</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>T ∑ t=1 [</td><td>(2 -β 1 )α t 2 ῡ</td><td>-</td><td>ρα 2 t 2υ 2 ]</td></tr><tr><td/><td colspan="2">≤</td><td colspan="3">T ∑ t=1 [</td><td colspan="9">(2 -β 1 )α t 2 ῡ t∈{2,...,T } -ρα 2 t 2υ 2 ] (1 -η) (1 -β 2 ) 1 n 2 √ n</td><td>t-1 s=0 ∑</td><td>α s σ t-s-1 2</td><td>(W )</td></tr><tr><td/><td colspan="2">+</td><td colspan="2">T ∑ t=1</td><td colspan="10">α t β 1,t 2(1 -β 1,t )(1 -η) 2 (1 -β 2 ) ῡ</td><td>+</td><td>ῡξ 2 υ 2</td><td>T ∑ t=1</td><td>α t</td><td>t r=1 ∑</td><td>β t-r 1 .</td><td>(72)</td></tr><tr><td colspan="5">Note that ∑ T t=1 [ (2-3β 1 )α t 2 ῡ</td><td colspan="5">-ρα 2 t 2υ 2 ] &gt; 0, and</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td>T ∑ t=1</td><td>α t</td><td>t ∑ r=1</td><td>β t-r 1</td><td colspan="2">=</td><td>T ∑ t=1</td><td>α t</td><td>T ∑ r=t</td><td>β r-t 1</td><td>≤</td><td>T ∑ t=1</td><td>α t (1 -β 1 )</td><td>.</td><td>(73)</td></tr><tr><td colspan="15">Therefore, dividing both sides of the equation 75 by ∑ T t=1 [ (2-3β 1 )α t 2 ῡ</td><td>-ρα 2 t 2υ 2 ], using equation 73, we</td></tr><tr><td colspan="2">complete the proof.</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="10">6.1.6 PROOF OF COROLLARY 11</td><td/><td/><td/><td/></tr><tr><td colspan="7">Proof. Let ϑ t = ∑ T t=1 [ (2-β 1 )α t 2 ῡ</td><td colspan="8">-ρα 2 t 2υ 2 ]. We claim that if</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td colspan="4">T ≥ max{</td><td colspan="4">4ρ 2 ῡ2 nυ 4 (2 -β 1 ) 2 ,</td><td>4 ῡ2 n (2 -β 1 ) 2 },</td><td>(74)</td></tr><tr><td colspan="15">then ϑ 2 ῡ</td><td>-ρα 2 t 2υ 2 ≥ 1 2T which results in</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td colspan="3">-T ρα 2</td><td/><td/><td/><td/></tr></table>

Peer review (iclr2019):
 This paper presents a consensus-based decentralized version of the Adam algorithm for online optimization. The authors consider an empirical risk minimization objective, which they split into different components, and propose running a separate online optimization algorithm for each component, with a consensus synchronization step that involves taking a linear combination of the parameters from each component before applying each component*s individual parameter update. The final output is a simple average of the parameters from each component. The authors study the important problem of distributed optimization and focus on adapting existing state-of-the-art methods to this setting. The algorithm is clearly presented, and to the best of my knowledge, original. The fact that this work includes both theoretical guarantees for the convex and non-convex settings as well as numerical experiments strengthens the contribution. On the other hand, I didn*t find the actual method presented by the authors to be motivated very well. The main innovation with respect to the standard Adam/AMSGrad algorithm is the use of a mixing matrix W, but the authors do not discuss how the choice of this matrix influences the performance of the algorithm or how one should specify this input in practice. This seems like an important issue, especially since all of the bounds depend on the second singular value of this matrix. Moreover, arguments such as Corollary 10 do not actually imply that DADAM outperforms ADAM when this singular value is large, making it difficult to assess the impact of this work. The numerical experiments also do not test for the statistical significance of the results. There are also many typos that make the submission seem relatively unpolished. Specific comments: 1. page 1: "note only". Typo. 2. page 2: "decentalized". Typo. 3. page 2: ".Pi_X[x]. If .Pi_X(x)...." Inconsistent notation. 4. page 3: "largest singular of matrix". Typo. 5. page 3: "x_t* = arg min_{x .in X} f_t(x)". f_t isn*t defined in up to this point. 6. page 4: "network cost is then given by f_t(x) = .frac{1}{n} .sum_{i=1}^n f_{i,t}(x)" Should the cost be .frac{1}{n} .sum_{i=1}^n f_{i,t}(x_{i,t})? That would be more consistent with the definition of regret presented in Reg_T^C. 7. page 4: "assdessed". Typo. 8. page 4: " Reg_T^C := .frac{1}{n} .sum_{i=1}^n .sum)_{t=1}^T f_t(x_{i,t})..." Why is this f_t and not f_{i,t}? 9. page 4: ".hat{v}_{i,t} = v_3 ..." You should reference how this assignment in the algorithm relates to the AMSGrad algorithm. Moreover, you should explain why you chose to use a convex combination in the assignment instead of just the max. 10. page 5: Definition 1. This calculation should be derived and presented somewhere (e.g. in the appendix). 11. page 5: Assumption 3. The notation for the stochastic gradient is not very clear and easily distinguishable from the notation for the deterministic gradient. 12. page 5: Theorem 4. D_T can be very large in the bound, which would make the upper bound meaningless. Can you set hyperparameters in such a way to minimize it? Also, what is the typical size of .sigma_2(W) that one would incur? 13. page 6: Remark 6. This remark seems misleading. It ignores the log(T) and D_T terms, both of which may dominate the data dependent arguments. 14. page 6: "The update rules .tilde{v}_{i,t}...". .tilde{v}_{i,t} is introduced but never defined. 15. page 6: Last display equation. The first inequality seems like it can be an equality. 16. page 7: Equation (14). Doesn*t the presence of .sigma_2(W) imply that the O(1/T) term may not be negligible? It would also be helpful to give some examples of how large T needs to be in (15a) and (15b) in order for this statement to take effect. 17. page8: "distributed federated averaging SGD (FedAvg)". What is the reference for this? It should be included here. It should probably also be mentioned in the introduction as related work. 18. page 9: Figure 1. Without error bars, it is impossible to tell the statistical significance of these results. Moreover, how sensitive are these results to different choices of hyperparameters? 19. page 9: "obtain p coefficients". What is p in these experiments? 20. page 9: Metropolis constant edge weight matrix W". What is .sigma_2(W) in this case? 21. page 10: Acknowledgements. This shouldn*t be included in the submission.