Title: B-ary Tree Push-Pull Method is Provably Efficient for Distributed Learning on Heterogeneous Data
Venue: NeurIPS 2024
Accepted: yes
Keywords: Distributed Learning

Distributed Optimization

Stochastic Optimization
Abstract: This paper considers the distributed learning problem where a group of agents cooperatively minimizes the summation of their local cost functions based on peer-to-peer communication. Particularly, we propose a highly efficient algorithm, termed ``B-ary Tree Push-Pull'' (BTPP), that employs two B-ary spanning trees for distributing the information related to the parameters and stochastic gradients across the network. The simple method is efficient in communication since each agent interacts with at most $(B+1)$ neighbors per iteration. More importantly, BTPP achieves linear speedup for smooth nonconvex objective functions with only $\tilde{O}(n)$ transient iterations, significantly outperforming the state-of-the-art results to the best of our knowledge.

B-ary Tree Push-Pull Method is Provably Efficient for Distributed Learning on Heterogeneous Data

Abstract

This paper considers the distributed learning problem where a group of agents cooperatively minimizes the summation of their local cost functions based on peer-to-peer communication. Particularly, we propose a highly efficient algorithm, termed 'B-ary Tree Push-Pull' (BTPP), that employs two B-ary spanning trees for distributing the information related to the parameters and stochastic gradients across the network. The simple method is efficient in communication since each agent interacts with at most ( B +1) neighbors per iteration. More importantly, BTPP achieves linear speedup for smooth nonconvex and strongly convex objective functions with only ˜ O ( n ) and ˜ O (1) transient iterations, respectively, significantly outperforming the state-of-the-art results to the best of our knowledge. Our code is available at https://github.com/ryou98/BTPP.

1 Introduction

In this paper, we consider a group of agents, labeled as N := { 1 , 2 , . . . , n } , in which each agent i holds its own local cost function f i : R p → R and communicates only within its direct neighborhood. We investigate how the agents collaborate to locate x ∈ R p that minimizes the average of all the cost functions:

where f i ( x ) := E ξ i ∼D i [ F i ( x ; ξ i )] . Here ξ i denotes the local data of agent i that follows the local distribution D i . Data heterogeneity exists if {D i } n i =1 are not identical.

To solve problem (1), we assume each agent i queries a stochastic oracle ( SO ) to obtain noisy gradient samples. Stochastic gradients appear in many areas including online distributed learning [23, 3], reinforcement learning [17, 15], generative modeling [5, 6], and parameter estimation [2, 27]. Assumption 1.1 ensures that the gradient estimator g i ( x ; ξ i ) remains unbiased with a bounded variance for any given x , while independent samples ξ i are gathered continuously over time. In addition, the assumption is critical in simulation-based optimization as gradient estimation often encounters noise from multiple sources, such as modeling and discretization errors, or limitations due to finite sample sizes in Monte-Carlo methods [7].

Modern optimization and machine learning typically involve tremendous data samples and model parameters. The scale of these problems calls for efficient distributed algorithms across multiple computing nodes. Recently, distributed algorithms dealing with problem (1) have been studied extensively in the literature; see, e.g., [19, 14, 4, 34]. Traditional distributed learning approaches typically follow a centralized master-worker setup, where each worker node communicates with a (virtual) central server [12]. However, such a communication pattern incurs significant communication overheads and long latency, especially when the training requires a large number of computing nodes.

Decentralized learning is an emerging paradigm to save communication costs, where the computing nodes are connected through a certain network topology (e.g., ring, grid, hypercube). Decentralized algorithms do not rely on central servers: the agents maintain the similarity among their copies of model parameters through peer-to-peer messages passing by communicating locally with immediate neighbors in the network. Such a setup allows each node to communicate with only a few peers and hence incurs much lower communication overhead [1]. Moreover, it offers strong promise for new applications, allowing agents to collaboratively train a model while respecting the data locality and privacy of each contributor.

Specifically, in decentralized stochastic gradient methods, the agents share their local stochastic gradient updates through gossip communication [32]. At every iteration, the local updates are sent to the neighbors of each agent who iteratively propagate the information through the network. Typically, the agents employ iterative gossip averaging of their neighbors' models with their own, where the averaging weights are chosen to ensure asymptotic uniform distribution of each update across the network. However, local averaging is less effective in 'mixing' information which makes decentralized algorithms converge slower than their centralized counterparts. Generally speaking, the network topology determines both the number of per-iteration communications and the convergence rates of decentralized algorithms, leading to a trade-off. For example, a densely-connected graph enables decentralized methods to converge faster but results in less efficient communication since each node needs to communicate with more neighbors. By contrast, a sparsely-connected topology results in a slower convergence rate but also reduces the per-iteration communication cost [19, 21, 35]. In particular, for smooth and non-convex objective functions, it has been shown that decentralized stochastic gradient methods (with arbitrary topology) can achieve the same convergence rate as the centralized SGD method, but only after an initial period of iterations has passed [14, 34, 20]. The number of transient iterations (transient time) heavily depends on the network topology, and thus a practical decentralized stochastic gradient algorithm should aim to minimize the transient time while keeping the number of per-iteration communications small (e.g., over a a sparselyconnected topology). Such an observation has motivated several recent works, which consider network topologies with Θ(1) per-iteration communications (or degree) for each node; see, e.g., [34, 26].

This work considers an alternative mechanism to gossip averaging, called 'B-ary Tree Push-Pull' (BTPP), inherited from the Push-Pull method in [22, 33]. Rather than relaying the messages over one graph at every iteration, BTPP uses two B-ary trees ( G R and G C ) to spread the information about the parameters and the stochastic gradients, respectively. Each agent assigned in the B-ary tree acts as a worker on an assembly line. The model parameters are transmitted through the graph G R from the parent nodes to the child nodes. Meanwhile, the stochastic gradients are computed under the current model parameters and accumulated through the inverse graph of G R denoted as G C . BTPP can be viewed as a semi-(de)centralized approach given the critical role of node 1 . Notably, the corresponding mixing matrices of G R and G C only consist of 0 's and 1 's, which together with the B-ary Tree topology design, results in high algorithmic efficiency. We show BTPP achieves an ˜ O ( n ) transient time under smooth nonconvex objective functions with Θ(1) per-iteration communications for each agent. By comparison, the state-of-the-art transient time is O ( n 3 ) (see Table 1).

1.1 Related Works

Decentralized Learning Decentralized Stochastic Gradient Descent (DSGD) type algorithms are increasingly popular for accelerating the training of large-scale machine learning models [14, 34, 10] . These algorithms have been adapted under a range of practical settings, including those discussed in [1, 16]. However, DSGD suffers from data heterogeneity [9], which triggers more advanced techniques such as EXTRA [25], Exact-Diffusion/D 2 [13], and gradient tracking [19]. The Push-Pull method [22, 33] which enjoys broad topological requirements was introduced for deterministic decentralized optimization under strongly convex objectives. This work particularly takes advantage of the flexibility in the network design of Push-Pull, utilizing the B-ary tree family, while considering stochastic gradients for minimizing smooth nonconvex objectives.

Topology Design Decentralized stochastic gradient algorithms often rely on gossip averaging over various topologies such as rings, grids, and tori [18]. The hypercube graph [30] strikes a balance between the communication efficiency and the consensus rate, but the network size is constrained to be the power of two. The work in [34] re-examined the static exponential graph with Θ(ln( n ))

degree and introduced a one-peer exponential graph with Θ(1) degree while preserving the consensus properties under the specific requirement of n . The paper [28] proposed a base-( k +1 ) graph as an enhancement that achieves similar convergence rate as in [34] under arbitrary network size by sequentially employing multiple graph topologies (splitting an all-connected graph into Θ(ln( n )) different subgraphs). DSGD-CECA [4] requires roughly ⌈ log 2 ( n ) ⌉ rounds of message passing for global averaging with Θ( n ) network topologies. OD(OU)-EquiDyn [26] introduces algorithms that employ various topologies to achieve network-size independent consensus rates. RelaySGD [31] offers a relay-based algorithm that ensures Θ(1) per-iteration communication across different topologies.

The above-mentioned methods all enjoy comparable convergence rates with centralized SGD (and thus achieves linear speedup) when the number of iterations T is large enough. The transient times are generally in the order of ˜ O ( n 3 ) under smooth nonconvex objectives (see Table 1) and ˜ O ( n ) under smooth strongly convex objectives (see Table 2).

Note that the above works and this paper generally consider training machine learning modes within high-performance data-center clusters, in which the network topology can be fully controlled: any two nodes can directly communicate over the network when necessary. By comparison, in some other scenarios, the underlying network topology is fixed, and the communication between two nodes is constrained (e.g., in wireless sensor networks, internet of vehicles, etc).

1.2 Main Contribution

This paper introduces a novel distributed stochastic gradient algorithm, termed 'B-ary Tree Push-Pull' (BTPP), which is provably efficient for solving the distributed learning problem (1) under arbitrary network size. The main contribution is summarized as follows:

BTPP incurs a Θ(1) communication overhead per-iteration for each agent. Specifically, any agent in the network communicates with at most ( B +1) neighbors, where B can be freely chosen to fit different settings. Generally speaking, larger B increases the per-iteration communication cost but reduces the transient time at the same time.

We show BTPP enjoys an ˜ O ( n ) transient time or iteration complexity under smooth nonconvex objectives and an ˜ O (1) transient time or iteration complexity under smooth strongly convex objectives. Such results outperform the baselines: see Table 1 and Table 2. The improvement is significant since the transient time greatly impacts the algorithmic performance, especially under large n .

The convergence analysis for BTPP is non-trivial, partly due to the fact that the algorithm admits two different network topologies for communicating the model parameters and the (stochastic) gradient trackers respectively. Instead of constructing the induced matrix norms ∥ · ∥ R and ∥ · ∥ C as in [22], the analysis is performed under ∥·∥ 2 and ∥·∥ F only by carefully treating the matrix products and related terms.

1.3 Notation and Preliminaries

Throughout the paper, vectors default to columns if not otherwise specified. Let each agent i hold a local copy x i ∈ R p of the decision variable and an auxiliary variable y i ∈ R p . Their values at iteration k are denoted by x ( k ) i and y ( k ) i , respectively. We let X = [ x 1 , x 2 , · · · , x n ] ⊤ ∈ R n × p , Y = [ y 1 , y 2 , · · · , y n ] ⊤ ∈ R n × p , and 1 denotes the column vector with all entries equal to 1 . Wealso define the aggregated gradients at the local variables as ∇ F ( X ) := [ ∇ f 1 ( x 1 ) , ∇ f 2 ( x 2 ) , · · · , ∇ f n ( x n )] ⊤ ∈ R n × p , where F ( X ) := ∑ n i =1 f i ( x i ) . In addition, denote ξ := [ ξ 1 , ξ 2 , · · · , ξ n ] ⊤ , G ( X , ξ ) := [ g 1 ( x 1 , ξ 1 ) , g 2 ( x 2 , ξ 2 ) , · · · , g n ( x n , ξ n )] ⊤ ∈ R n × p . For the conciseness of presentation, we also use G ( t ) to represent G ( X ( t ) , ξ ( t ) ) . The term ⟨ a, b ⟩ stands for the inner product of two vectors a, b ∈ R p . For matrices, ∥·∥ 2 and ∥·∥ F represent the spectral norm and the Frobenius norm respectively, which degenerate to the Euclidean norm for vectors. For simplicity, any square matrix with power 0 is the unit matrix I with the same dimension if not otherwise specified.

We assume each agent i is able to obtain noisy gradient samples of the form g i ( x, ξ i ) that satisfies the following assumption.

Assumption 1.1. For all i ∈ N and x ∈ R p , each random vector ξ i is independent and

for some σ 2 > 0 .

Regarding the individual objective functions f i , we make the following standard assumption.

Assumption 1.2. Each f i ( x ) : R p → R is lower bounded with L -Lipschitz continuous gradients, i.e., for any x, x ′ ∈ R p ,

We also consider the following standard assumption regarding strongly convexity.

Assumption 1.3. For any x, y ∈ R p ,

Denote f ∗ := min x ∈ R p f ( x ) . Let x ∗ = arg min x f ( x ) if Assumption 1.3 holds.

A directed graph G ( N , E ) consists of a set of n nodes N and a set of directed edges E ⊆ N × N , where an edge ( j, i ) ∈ E indicates that node j can directly send information to node i . To facilitate the local averaging procedure, each graph can be associated with a non-negative weight matrix W = [ w ij ] ∈ R n × n , whose element w ij is non-zero only if ( j, i ) ∈ E . Similarly, a non-negative weight matrix W corresponds to a directed graph denoted by G W . For a given graph G W , the inneighborhood and out-neighborhood of node i ∈ N are given by N in W ,i := { j ∈ N : ( j, i ) ∈ E} and N out W ,i := { j ∈ N : ( i, j ) ∈ E} , respectively. The degree of node i is the number of its in-neighbors or out-neighbors. For example, in a one-peer graph, the degree of each node is at most 1 .

1.4 Organization of the Paper

The rest of this paper is organized as follows. In Section 2, we introduce the B-ary Tree Push-Pull algorithm and present its main convergence results. The sketch of analysis is presented in Section 3, and numerical experiments are provided in Section 4. We conclude the paper in Section 5.

2 B-ary Tree Push-Pull Method

2.1 Communication Graphs

The proposed B-ary Tree Push-Pull method makes use of two spanning trees as communication graphs: G R and G C , which correspond to two mixing matrices R and C , respectively. Specifically, we consider B-ary tree graphs with arbitrary number of nodes n and depth d . The root node is labeled as 1 for convenience, and we index the nodes layer-by-layer. The additional nodes are placed at the last layer if the tree is not full. Figure 1 illustrates the assignment of 10 nodes when B = 2 . In the Pull Tree G R (the left ones), each node has 1 parent node and B child nodes (except the ones in the last layer). The root node 1 has no parent node. In the Push Tree G C (the right ones), each node has 1 child node and B parent nodes (except the ones in the last layer). It can be seen that the tree G C is identical to G R with all the edges reversing directions. Note that only node 1 has a self-loop.

2

3

4

5

6

7

4

5

6

7

8

9

10

8

9

10

2.2 Algorithm

We consider the following distributed stochastic gradient method (Algorithm 1) for solving problem (1). At every iteration t , each agent i pulls the state information from its in-neighborhood N in R ,i , pushes its (stochastic) gradient tracker y i to the out-neighborhood N out C ,i , and updates its local variables x i and y i based on the received information. The agents aim to find the ϵ -stationary point jointly by performing local computation and exchanging information through two spanning trees.

More specifically, in the pull tree G R , each node i pulls the updated model from its parent node along the tree. Note that N in R ,i consists of only one node, the parent node. The Push Tree G C is the inverse of the Pull Tree, in which each node collects and aggregates the gradient trackers from its parent nodes. Due to the tree structure, only y t 1 aggregates and tracks the stochastic gradients across the entire network, which will be made clear from the analysis. The implementation of the algorithm is rather simple. Taking node 2 in Figure 1 as an example, we have x ( t +1) 2 = x ( t ) 1 -γy ( t ) 1 and y ( t +1) 2 = y ( t ) 4 + y ( t ) 5 + g 2 ( x ( t +1) 2 ; ξ ( t +1) 2 ) -g 2 ( x ( t ) 2 ; ξ ( t ) 2 ) .

We can write Algorithm 1 in the following compact form:

where Y (0) = G ( X (0) , ξ (0) ) , and R , C ∈ R n × n are non-negative matrices whose elements are given by

and C = R ⊤ which corresponds to G C , the inverse tree of G R . It can be seen that R is a row-stochastic matrix that only consists of 0 's and 1 's, and C is column stochastic. For example, the mixing matrices corresponding to the graphs in Figure 1 are given by

Algorithm 1 B-ary Tree Push-Pull Method (BTPP)

1: Each agent i initializes with any arbitrary but identical x (0) i = x (0) ∈ R p , y (0) i = g i ( x (0) i , ξ (0) i ) ∈ R p after drawing a random sample ξ (0) i , stepsize γ > 0 and number of iterations T .

2: for iteration t = 0 , 1 , 2 , . . . , T -1 do

3: for agent i in parallel do

4: Pull x ( t ) j -γy ( t ) j from each j ∈ N in R ,i

5: Push y ( t ) i to each j ∈ N out C ,i

6: Independently draw a random sample ξ ( t +1) i

7: Update parameters through

8: end for

9: end for

10: Output x ( T ) 1 .

where the unspecified elements are zeros.

2.3 Main Results

The main convergence properties of BTPP are summarized in the following two theorems, where the second result assumes strongly convexity on f .

Theorem 2.1. For the BTPP algorithm outlined in Algorithm 1 implemented on B-ary tree graphs G R and G C , assume Assumption 1.1 and Assumption 1.2 hold. Let γ = min { ( ∆ f 3 σ 2 Ln ( T +1) ) 1 2 , ( ∆ f 1500 n 2 d 6 σ 2 L 2 ( T +1) ) 1 3 , 1 100 nd 3 L } . The following convergence result holds:

where ∆ f := f ( x (0) 1 ) -f ∗ and d = ⌊ log B ( n ) ⌋ represents the diameter of the graphs.

Remark 2.2 . Based on the convergence rate in (3) of BTPP, we can derive that when T = Θ( n log 12 ( n )) , the term O ( 1 √ nT ) dominates the remaining terms up to a constant scalar. This implies that BTPP achieves linear speedup after O ( n log 12 ( n ) ) transient iterations.

Remark 2.3 . The convergence rate in (3) is related to the branch size B . For larger B , the diameter d = ⌊ log B ( n ) ⌋ becomes smaller, which results in more efficient transmission of information and fewer transient iterations. However, the per-iteration communication cost is relatively larger. When B is smaller, the communication burden for each agent at every iteration is lighter, but the transient time is larger. Therefore, in practice, the communication cost and convergence rate can be balanced by considering a proper B .

Theorem 2.4. For the BTPP algorithm outlined in Algorithm 1 implemented on B-ary tree graphs G R and G C , assume Assumption 1.1, Assumption 1.2 and Assumption 1.3 hold. Let γ = min { 1 100 nd 2 κL , 16 log( n ( T +1) 2 ) n ( T +1) µ } and T ≥ 2 d . The following convergence result holds:

Remark 2.5 . The convergence rate in (4) implies that E ∥ ∥ ∥ x ( T ) 1 -x ∗ ∥ ∥ ∥ 2 ≤ ˜ O ( 1 nT + 1 nT 2 +exp( -T ) ) , where ˜ O hides the constants and polylogarithmic factors. The transient time is thus ˜ O (1) , i.e., the number of iterations before the term O ( 1 nT ) dominates the remaining terms. Such a transient time also outperforms the state-of-the-art results.

3 Analysis of B-ary Tree Push-Pull

In this section, we study the convergence of BTPP and prove Theorem 2.1 by analyzing the properties of the weight matrices R and C , the evolution of the aggregated consensus error ∑ T t =0 E ∥ ∥ Π u X ( t ) ∥ ∥ 2 F , and the expected inner products of the stochastic gradients between different layers. The approach is different from those employed in [22, 19, 26], where the analysis considers two special matrix norms related to R and C , respectively. Such a distinction is because BTPP works with two B-ary trees and iterates in a layer-wise manner, while most other works consider connected graphs.

Our analysis starts with characterizing the weight matrices R and C , as delineated in the following lemmas. It is important to note that for any given n and a specific integer B , we can determine an integer d satisfying B d -1 B -1 < n ≤ B d +1 -1 B -1 which is the diameter of the graphs.

Notice that R has a unique non-negative left eigenvector u ⊤ (w.r.t. eigenvalue 1) with u ⊤ 1 = n . More specifically, u = [ n, 0 , · · · , 0] ⊤ , which is also the unique right eigenvector of C (w.r.t. eigenvalue 1), denoted by v for the clarity of presentation. Following the above observations, it is revealed in Lemma 3.1 that the 2 -norm of the matrix R1 n 1u ⊤ with exponent k remains bounded by √ n and equals zero when k exceeds d -1 .

Lemma 3.1. Given a positive integer k , the 2-norm of the matrix R k -1 n 1u ⊤ satisfies

Similar result applies to the matrix C k -1 n v1 ⊤ . Consequently, we introduce the mixing matrices Π u , Π v based on the eigenvectors u , v , which play a crucial role in the follow-up analysis.

The following lemmas delineate the critical elements for constraining the average expected norms of the objective function as formulated in (1), i.e., 1 T +1 ∑ T t =0 E ∥ ∥ ∥ ∇ f ( x ( t ) 1 ) ∥ ∥ ∥ 2 2 . Lemma 3.2 and Lemma 3.3 provide bounds on the expressions ∑ T t =0 E ∥ ¯ X ( t +1) -¯ X ( t ) ∥ 2 F and ∑ T t =0 ∥ ∥ Π u X ( t ) ∥ ∥ 2 F , where ¯ X ( t ) := 1 n 1u ⊤ X ( t ) .

Lemma 3.2. Suppose Assumption 1.1 holds and γ ≤ 1 10 ndL , we have the following inequality:

Lemma 3.3. Suppose Assumption 1.1 holds and γ ≤ 1 40 nd 2 L , we have for d ≥ 2 that

From the design of BTPP, there is an inherent delay in the transmission of information from layer k to layer 1 . As information traverses through the B-ary trees, the delay becomes evident. Specifically, for nodes at layer k , their information requires an additional k iterations to successfully reach and impact node 1 , as demonstrated in Lemma 3.4.

Lemma 3.4. For any integer t > 1 , we have

Building on the preceding lemmas, we are in a position to establish the main convergence result for the BTPP algorithm. This involves upper bounding the expected norm for the gradient of the objective function evaluated at x ( t ) 1 . To show the result, we integrate the findings from Lemma 3.2, Lemma 3.3, and Lemma 3.4, as detailed in Lemma 3.5.

Lemma 3.5. Suppose Assumption 1.1 and Assumption 1.2 hold and γ ≤ 1 100 nd 3 L , we have

Remark 3.6 . Lemma 3.5 implies that the transient time of BTPP under Assumption 1.2 is influenced by the fourth term in the upper bound: 400 d 3 L 2 ∥ Π u X (0) ∥ 2 F T +1 which is related to the initial consensus error. Therefore, we initialize all the agents with the same solution x (0) .

Under strong convexity of f , we have the following key lemma.

Lemma 3.7. Suppose Assumption 1.1, Assumption 1.2 and Assumption 1.3 hold, and γ ≤ 1 100 nd 2 κL , we have

where κ := L/µ is the conditional number.

4 Numerical Results

This section presents experimental results to compare the B-ary Tree Push-Pull method with other popular algorithms on logistic regression with synthetic data and deep learning tasks with real data.

4.1 Logistic Regression

We compare the performance of BTPP against other algorithms listed in Table 1 for logistic regression with non-convex regularization [26]. The objective functions f i : R p → R are given by

where x [ k ] is the k -th element of x , and { ( h i,j , y i,j ) } J j =1 represent the local data kept by node i . To control the data heterogeneity across the nodes, we first let each node i be associated with a local logistic regression model with parameter ˜ x i generated by ˜ x i = ˜ x + v i , where ˜ x ∼ N (0 , I p ) is a common random vector, and v i ∼ N (0 , σ 2 h I p ) are random vectors generated independently. Therefore, { v i } decide the dissimilarities between ˜ x i , and larger σ h generally amplifies the difference. After fixing { ˜ x i } , local data samples are generated that follow distinct distributions. For node i , the feature vectors are generated as h i,j ∼ N (0 , I p ) , and z i,j ∼ U (0 , 1) . Then, the labels y i,j ∈ {-1 , 1 } are set to satisfy z i,j ≤ 1 + exp( -y i,j h ⊤ i,j ˜ x i ) .

In the simulations, the parameters are set as follows: n = 100 , p = 500 , J = 1000 , R = 0 . 01 , and σ h = 0 . 8 . All the algorithms initialize with the same stepsize γ = 0 . 3 , except BTPP, which employs a modified stepsize γ/n . Such an adjustment is due to BTPP's update mechanism, which incorporates a tracking estimator that effectively accumulates n times the averaged stochastic gradients as the number of iterations increases. This can also be seen from the stepsize choice in Theorem 2.1. 2 Additionally, we implement a stepsize decay of 60% every 100 iterations to facilitate convergence.

In Figure 2, the gradient norm is used as a metric to gauge the algorithmic performance of each algorithm. The left panel of Figure 2 illustrates the comparative performance of various algorithms, highlighting that BTPP (in red) achieves faster convergence than the other algorithms with Θ(1) degree and closely approximates the performance of the centralized SGD algorithm (i.e., DSGDFullyConnected). The right panel of Figure 2 demonstrates the behavior of BTPP when increasing the branch size B . It is observed that with larger B , the convergence trajectory of BTPP more closely aligns with that of centralized SGD, corroborating the prediction of the theoretical analysis.

4 × 10-1

BTPP

DSGD-Ring

4 × 10-1

BTPP (B=2)

BTPP (B=3)

DSGT-Ring

BTPP (B=4)

DSGD-Exponential

BTPP (B=8)

DSGD-Grid

BTPP (B=16)

DSGD-FullyConnected

BTPP (B=32)

3 ×10-1

DSGD-OnePeerExp

3 × 10-1

BTPP (B=64)

DSGT-ODEquiDyn

BTPP (B=100)

DSGD-Base-(k+1)

DSGD-Ring

DSGT-Ring

Gradient Norm

DSGD-CECA-2p

Gradient Norm

DSGD-Exponential

RelaySGD-B-Tree

DSGD-Grid

DSGD-FullyConnected

2 × 10-1

2 × 10-1

DSGD-OnePeerExp

DSGT-ODEquiDyn

DSGD-Base-(k+1)

DSGD-CECA-2p

RelaySGD-B-Tree

100

200

300

Iteration

400

500

600

700

800

0

100

200

300

Iteration

400

500

600

700

800

4.2 Deep Learning

We apply BTPP and the other algorithms to solve the image classification task with CNN over MNIST dataset [11]. We run all experiments on a server with eight Nvidia RTX 3090 GPUs. The network contains two convolutional layers with max pooling and ReLu and two feed-forward layers. In particular, we consider a heterogeneous data setting, where data samples are sorted based on their labels and partitioned among the agents. The local batch size is set to 8 with 24 agents in total. The learning rate is 0 . 01 for all the algorithms except BTPP (which employs a modified stepsize γ/n ) for fairness. Additionally, the starting model is enhanced by pre-training using the SGD optimizer on the entire MNIST dataset for several iterations. Figure 3 illustrates the training loss and the test accuracy curves. Comparing the performance of different algorithms, it can be seen that BTPP (in red) and DSGT with ODEquiDyn (based on Θ( n ) graphs) achieve faster convergence than the

other algorithms with Θ(1) degree and closely approximate the performance of centralized SGD(i.e., DSGD-FullyConnected).

Train Loss

Iteration

2.2

0.95

BTPP

2

DSGD-Ring

0.9

DSGT-Ring

1.8

DSGD-OnePeerExp

0.85

DSGD-CECA-2p

1.6

DSGD-Base-(k+1)

DSGT-ODEquiDyn

0.8

1.4

RelaySGD-B-Tree

BTPP

LOSS

1.2

DSGD-Exponential

0.75

DSGT-Ring

DSGD-Ring

DSGD-FullyConnected

DSGD-Grid

0.7

DSGD-OnePeerExp

DSGD-CECA-2p

0.8

0.65

DSGD-Base-(k+1)

DSGT-ODEquiDyn

0.6

0.6

RelaySGD-B-Tree

-DSGD-Exponential

0.4

0.55

DSGD-FullyConnected

DSGD-Grid

0.2

0.5

0

2000

4000

6000

8000

10000

12000

14000

0

2000

4000

6000

8000

10000

12000

14000

TestAccuracy

Iteration

Remark 4.1 . Higher accuracy can be achieved for BTPP and other methods when using the momentum technique, or when the data heterogeneity is removed, meaning that samples are randomly assigned to each agent. Additional experiments demonstrating the performance of various algorithms across different tasks and scenarios are provided in Appendix B.

5 Conclusions

This paper proposes a novel algorithm for distributed learning over heterogeneous data, named BTPP. The convergence is theoretically analyzed for smooth non-convex stochastic optimization. The results demonstrate that, at the minimal communication cost per iteration, BTPP achieves linear speedup in the number of nodes n , and the transient times behaves as ˜ O ( n ) and ˜ O (1) respectively for smooth nonconvex and strongly convex objectives, outperforming the state-of-the-art results. Numerical experiments further validate the efficiency of BTPP.

References

M. ASSRAN, N. LOIZOU, N. BALLAS, AND M. RABBAT, Stochastic gradient push for distributed deep learning , in International Conference on Machine Learning, PMLR, 2019, pp. 344-353.

L. BOTTOU, Stochastic gradient descent tricks , in Neural Networks: Tricks of the Trade: Second Edition, Springer, 2012, pp. 421-436.

J. DEAN, G. CORRADO, R. MONGA, K. CHEN, M. DEVIN, M. MAO, M. RANZATO, A. SENIOR, P. TUCKER, K. YANG, ET AL., Large scale distributed deep networks , Advances in neural information processing systems, 25 (2012).

L. DING, K. JIN, B. YING, K. YUAN, AND W. YIN, Dsgd-ceca: Decentralized sgd with communication-optimal exact consensus algorithm , arXiv preprint arXiv:2306.00256, (2023).

I. GOODFELLOW, J. POUGET-ABADIE, M. MIRZA, B. XU, D. WARDE-FARLEY, S. OZAIR, A. COURVILLE, AND Y. BENGIO, Generative adversarial nets , Advances in neural information processing systems, 27 (2014).

D. P. KINGMA, M. WELLING, ET AL., An introduction to variational autoencoders , Foundations and Trends® in Machine Learning, 12 (2019), pp. 307-392.

J. P. KLEIJNEN, Design and analysis of simulation experiments , Springer, 2018.

A. KOLOSKOVA, T. LIN, AND S. U. STICH, An improved analysis of gradient tracking for decentralized machine learning , Advances in Neural Information Processing Systems, 34 (2021), pp. 11422-11435.

A. KOLOSKOVA, N. LOIZOU, S. BOREIRI, M. JAGGI, AND S. STICH, A unified theory of decentralized sgd with changing topology and local updates , in International Conference on Machine Learning, PMLR, 2020, pp. 5381-5393.

A. KOLOSKOVA, S. STICH, AND M. JAGGI, Decentralized stochastic optimization and gossip algorithms with compressed communication , in International Conference on Machine Learning, PMLR, 2019, pp. 3478-3487.

Y. LECUN, C. CORTES, C. BURGES, ET AL., Mnist handwritten digit database , 2010.

M. LI, D. G. ANDERSEN, J. W. PARK, A. J. SMOLA, A. AHMED, V. JOSIFOVSKI, J. LONG, E. J. SHEKITA, AND B.-Y. SU, Scaling distributed machine learning with the parameter server , in 11th USENIX Symposium on operating systems design and implementation (OSDI 14), 2014, pp. 583-598.

Z. LI, W. SHI, AND M. YAN, A decentralized proximal-gradient method with network independent step-sizes and separated convergence rates , IEEE Transactions on Signal Processing, 67 (2019), pp. 4494-4506.

X. LIAN, C. ZHANG, H. ZHANG, C.-J. HSIEH, W. ZHANG, AND J. LIU, Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent , Advances in neural information processing systems, 30 (2017).

T. P. LILLICRAP, J. J. HUNT, A. PRITZEL, N. HEESS, T. EREZ, Y. TASSA, D. SILVER, AND D. WIERSTRA, Continuous control with deep reinforcement learning , arXiv preprint arXiv:1509.02971, (2015).

T. LIN, S. P. KARIMIREDDY, S. U. STICH, AND M. JAGGI, Quasi-global momentum: Accelerating decentralized deep learning on heterogeneous data , arXiv preprint arXiv:2102.04761, (2021).

V. MNIH, K. KAVUKCUOGLU, D. SILVER, A. GRAVES, I. ANTONOGLOU, D. WIERSTRA, AND M. RIEDMILLER, Playing atari with deep reinforcement learning , arXiv preprint arXiv:1312.5602, (2013).

A. NEDI ´ C, A. OLSHEVSKY, AND M. G. RABBAT, Network topology and communicationcomputation tradeoffs in decentralized optimization , Proceedings of the IEEE, 106 (2018), pp. 953-976.

S. PU AND A. NEDI ´ C, Distributed stochastic gradient tracking methods , Mathematical Programming, 187 (2021), pp. 409-457.

S. PU, A. OLSHEVSKY, AND I. C. PASCHALIDIS, Asymptotic network independence in distributed stochastic optimization for machine learning: Examining distributed and centralized stochastic gradient descent , IEEE signal processing magazine, 37 (2020), pp. 114-122.

S. PU, A. OLSHEVSKY, AND I. C. PASCHALIDIS, A sharp estimate on the transient time of distributed stochastic gradient descent , IEEE Transactions on Automatic Control, 67 (2021), pp. 5900-5915.

S. PU, W. SHI, J. XU, AND A. NEDI ´ C, Push-pull gradient methods for distributed optimization in networks , IEEE Transactions on Automatic Control, 66 (2020), pp. 1-16.

B. RECHT, C. RE, S. WRIGHT, AND F. NIU, Hogwild!: A lock-free approach to parallelizing stochastic gradient descent , Advances in neural information processing systems, 24 (2011).

S. RESNICK, A probability path , Springer, 2019.

W. SHI, Q. LING, G. WU, AND W. YIN, Extra: An exact first-order algorithm for decentralized consensus optimization , SIAM Journal on Optimization, 25 (2015), pp. 944-966.

Z. SONG, W. LI, K. JIN, L. SHI, M. YAN, W. YIN, AND K. YUAN, Communication-efficient topologies for decentralized learning with o (1) consensus rate , Advances in Neural Information Processing Systems, 35 (2022), pp. 1073-1085.

N. SRIVASTAVA, G. HINTON, A. KRIZHEVSKY, I. SUTSKEVER, AND R. SALAKHUTDINOV, Dropout: a simple way to prevent neural networks from overfitting , The journal of machine learning research, 15 (2014), pp. 1929-1958.

Y. TAKEZAWA, R. SATO, H. BAO, K. NIWA, AND M. YAMADA, Beyond exponential graph: Communication-efficient topologies for decentralized learning via finite-time convergence , arXiv preprint arXiv:2305.11420, (2023).

H. TANG, X. LIAN, M. YAN, C. ZHANG, AND J. LIU, d Θ 2 : Decentralized training over decentralized data , in International Conference on Machine Learning, PMLR, 2018, pp. 48484856.

L. TREVISAN, Lecture notes on graph partitioning, expanders and spectral methods , University of California, Berkeley, https://people. eecs. berkeley. edu/luca/books/expanders-2016. pdf, (2017).

T. VOGELS, L. HE, A. KOLOSKOVA, S. P. KARIMIREDDY, T. LIN, S. U. STICH, AND M. JAGGI, Relaysum for decentralized deep learning on heterogeneous data , Advances in Neural Information Processing Systems, 34 (2021), pp. 28004-28015.

L. XIAO AND S. BOYD, Fast linear iterations for distributed averaging , Systems & Control Letters, 53 (2004), pp. 65-78.

R. XIN AND U. A. KHAN, A linear algorithm for optimization over directed graphs with geometric convergence , IEEE Control Systems Letters, 2 (2018), pp. 315-320.

B. YING, K. YUAN, Y. CHEN, H. HU, P. PAN, AND W. YIN, Exponential graph is provably efficient for decentralized deep training , Advances in Neural Information Processing Systems, 34 (2021), pp. 13975-13987.

K. YUAN, S. A. ALGHUNAIM, AND X. HUANG, Removing data heterogeneity influence enhances network topology dependence of decentralized sgd , Journal of Machine Learning Research, 24 (2023), pp. 1-53.

A Convergence Analysis of BTPP

In this section, we aim to demonstrate the convergence results of BTPP through a three-step process. First, we explore the key properties of matrices R and C , acquainting readers with several operations that will be frequently utilized in the subsequent parts. Then, we introduce various technical tools essential for the analysis. Finally, we delve into proving the convergence results supported by a series of pertinent lemmas.

A.1 Properties of the Weight Matrices

In this part, we first demonstrate that R∈ R n × n possesses a set of properties ( the matrix C = R ⊤ shares similar properties). Then, we utilize the established tools to prove the crucial result presented in Lemma 3.1. Lastly, we provide clarifications on certain matrix operations that will be frequently employed in deriving the convergence results.

It is important to note that for any given n and specific integer B , the diameter of the corresponding B-ary tree graph d (the distance from the last layer node to node 1) satisfies B d -1 B -1 < n ≤ B d +1 -1 B -1 . To investigate the properties of R and C , we will introduce the column vector e I ∈ R n , where each element of e I is equal to 1 for indices i ∈ I and 0 otherwise. Define the index sets

where k 1 : k 2 is the arithmetic progression from k 1 to k 2 with difference 1. We can then define the matrix Z k ∈ R n × n as a composite of several column vectors arranged in the following format:

This closed-form expression of R with any power k is shown in Lemma A.1 which aids in developing further properties.

Lemma A.1. For the pull matrix R corresponding to the B-ary tree G R , given any positive index k , we have

Proof. We prove the lemma by induction. First, it is obvious that R = Z 1 by the definition of R :

Now assume the statement is true for k = j . Then, for k = j +1 , we have

Denote [ Z j · Z 1 ] i as the i -th column of Z j · Z 1 . To establish the result, we only need to demonstrate that the two matrices, R k +1 and Z k +1 , have the same column entries. For i = 1 ,

For i > 1 , we have

k +1

Thus, we conclude that R = Z k +1 .

Corollary A.2 below reveals that when the power k exceeds d , R k transforms into a matrix where the first column is entirely composed of ones, while all the other columns consist of zeros.

Corollary A.2. For k = d , we have

where 0 is the matrix with all entries equal 0.

Proof. From Lemma A.1, we have for the i -th column of R k -1 n 1u ⊤ that

For k = d , the first n elements of all the columns remain 0 , which implies the desired result.

Now, we are ready to prove Lemma 3.1:

Proof of Lemma 3.1. For any integer k ≤ d -1 , define

This ensures that B k +1 -1 B -1 + n 0 B k ≤ n and B k +1 -1 B -1 + ( n 0 + 1) B k > n , so that only the first ( n 0 +2) -th columns of R k consist of non-zero elements. Note that

Then, we focus on the non-zero elements of the matrix R k -1 n 1u ⊤ .

where ˜ x = ( x 1 , · · · , x n 0 +2 ) is the truncated x , and

where the unspecified elements are all zeros. Since Σ is symmetric, all the eigenvalues are real. We show by contradiction that any eigenvalue λ of Σ is upper bounded by n . Otherwise, if there exists λ > n , we denote x as the corresponding eigenvector of λ . Then, we have from Σ x = λ x that

̸

Without loss of generality, assume x 1 = 0 . Then, by substituting the other relations into the first one, we have

With the fact that λ > n , there holds

which is a contradiction. Thus, we have λ ≤ n . It follows that

From the fact that the square root function is monotonically increasing on [0 , ∞ ) , we have

which implies that ∥R k -1 n 1u ⊤ ∥ 2 ≤ √ n for k ≤ d -1 and ∥R k -1 n 1u ⊤ ∥ 2 = 0 otherwise by Corollary A.2.

The transformations described in Corollary A.3 below are straightforward.

Corollary A.3. For any integer m> 0 , we have

To simplify the convergence analysis, we introduce the matrix A i defined as follows:

for i = 1 , 2 , · · · , d . Specifically, A 1 = C I . Consequently, Corollary A.4 below can be directly deduced from Lemma A.1 and Corollary A.3.

Corollary A.4. For i = 1 , · · · , d , we have

Intuitively, Corollary A.4 illustrates that ( u ⊤ n -1 ⊤ ) A i serves as an indicator vector representing the ( i +1) -th layer of the graph.

A.2 Supporting Inequalities and Lemmas

Lemma A.5 and Lemma A.6 below are frequently employed for bounding the norms of matrix summations and multiplications. Their proofs rely on the Cauchy-Schwartz inequality and the definitions of matrix norms ∥ · ∥ 2 and ∥ · ∥ F .

Lemma A.5. For an arbitrary set of m matrices { A i } i m =1 with the same size, we have

Proof. By the definition of Frobenius norm, we have

Taking squares on both sides and invoking the Cauchy-Schwarz inequality, we have

Lemma A.6. Let A , B be two real matrices whose sizes match. Then,

Proof. Let A = U Σ V H be the singular value decomposition of A , with the largest singular value σ max and hence ∥ A ∥ 2 = σ max . Then, we have

which implies the desired result.

Lemma A.7 below will be used in the last step for deriving the convergence rate of BTPP.

Lemma A.7. Let A,B,C and α be positive constants and T be a positive integer. Define function

Then,

where the upper bound can be achieved by choosing γ = min { ( A B ( T +1) ) 1 2 , ( A C ( T +1) ) 1 3 , 1 α } .

Proof. See Lemma 26 in [8] for a reference.

Lemma A.8 is a technical result related to random variables.

Lemma A.8. Consider three random variables X , Y , and Z . Assume that Z is independent with ( X,Y ) . Let h and g be functions such that the conditional expectation E [ g ( Y, Z ) | Y ] = 0 . We have

Proof. It implies by the condition Z ⊥ ⊥ ( X,Y ) that σ ( Z ) ⊥ ⊥ σ ( X,Y ) . Then,

It suffices to show

which follows directly from (10.17) in [24]. Thus, by the Tower Rule, we reach the statement as follows:

A.3 Proofs of Key Lemmas

In this section, we prove several key lemmas for proving the main convergence result of BTPP.

A.3.1 Preparation

Algorithm 1, as encapsulated by the equations in (2), can be succinctly expressed in the following matrix form:

By repeatedly applying equation (5) starting from time step t and going back to time step 0 , we arrive at the following relation:

For the sake of clarity, we start with introducing some simple definitions. Any matrix raised to the power of 0 is defined as the identity matrix I , which matches the original matrix in dimension. The only exceptions are ( R1 n 1u ⊤ ) 0 := Π u and ( C 1 n v1 ⊤ ) 0 := Π v for convenience. Furthermore, we introduce the following terms:

Note that, for any given integer m> 0 ,

As a result, given the initial condition Y (0) = G (0) , we can deduce the outcomes of X ( t ) and Y ( t ) as follows.

Then, after multiplying Π u and Π v to equation (6) and equation (7) respectively, and invoking Corollary A.3, we have

A.3.2 Analysis of the Variance

Denote by F t the σ -algebra generated by { ξ 0 , · · · , ξ t -1 } , and define E [ ·|F t ] as the conditional expectation given F t . Lemma A.9 provides an estimate for the variance of the gradient estimator G ( X ( t ) , ξ ( t ) ) .

Lemma A.9. Under Assumption 1.1, for any given power k ≤ d -1 , we have for all t ≥ 0 that

̸

Proof. For any given t and i = j , due to the independently drawn sample ξ ( t ) i , we have that ξ ( t ) i is independent of ( F t , ξ ( t ) j ) , and thus ξ ( t ) i is independent of σ ( x ( t ) i , x ( t ) j , ξ ( t ) j ) . Hence, invoking Lemma A.8 and Assumption 1.1 yields

Then, for any index set I ⊆ { 1 , 2 , · · · , n } , we have E ∥ e ⊤ I ( G ( t ) -∇ F ( X ( t ) ) ) ∥ 2 2 ≤ |I| σ 2 . Notice that

Thus, we obtain the desired result by invoking Lemma A.1 and Corollary A.2 after taking expectation on both sides of the above relation:

Under Assumption 1.1 and the randomly selected samples, Lemma A.9 and Corollary A.10 below provide an initial estimation for the variance terms. The proof of Corollary A.10 is directly from the analysis in Appendix A.3.2 and Corollary A.4.

Corollary A.10. Under Assumption 1.1, we have for all t ≥ 0 that

A.3.3 Proof of Lemma 3.2

Proof. Notice that

Multiplying 1 ⊤ on both sides of equation (7), we have 1 ⊤ Y ( t ) = 1 ⊤ G ( t ) for any integer t . Thus, in light of equation (9), we have

Hence, taking the F-norm and expectation on both sides, we have from Lemma A.5 that

Note that, invoking Lemma A.9 and Corollary A.10 yields

From Assumption 1.2, Lemma A.6 and Lemma A.5, we have

Thus, summing over t in (10) from 0 to T , combining all the inequalities above, and invoking Assumption 1.1 and Assumption 1.2, we have

Since γ ≤ 1 10 ndL , we have 15 γ 2 n 2 d 2 L 2 ≤ 1 6 , and the desired result follows.

A.3.4 Proof of Lemma 3.3

Proof. We show the upper bound for E ∥ ∥ Π u X ( t ) ∥ ∥ 2 F by studying equation (8) and bound the F-norm of each term respectively. From Corollary A.2, we can change the power of R1 n 1u ⊤ to at most

d -2 :

Then, we derive the following decomposition by pairing the gradients with each of the stochastic gradients in order to use Assumption 1.1.

where the terms from Q 0 ,t, 1 to Q 0 ,t, 4 and from Q 1 ,t to Q 4 ,t correspond to each term following ( R1 n 1u ⊤ ) ⊤ X (0) one-by-one.

We assume that d ≥ 2 , since d = 1 makes the summation illegal (summing over j from a positive number to a non-positive number), in which case Π u X ( t ) degenerates to ( R t -1 n 1u ⊤ ) X (0) and hence by Corollary A.2, there is no consensus error, i.e.

For d ≥ 2 , Lemma A.11 - Lemma A.13 below introduce the upper bounds for the F-norms of Q 1 ,t , Q 2 ,t and Q 3 ,t + Q 0 ,t, 2 , summing from t = 0 to T . Lemma A.14 establishes a similar upper bound for the F-norm of Q 0 ,t, 1 + Q 0 ,t, 3 . Furthermore, Lemma A.15 provides the upper bound for the F-norm of Q 0 ,t, 4 + Q 4 ,t .

Lemma A.11. For any iteration number T , we have

Proof. To make the summation legal given d ≥ 2 , we need d -1 ≥ t -m -d , which implies that m ≥ t +1 -2 d . Then,

Invoking Lemma A.5, Lemma 3.1 and Lemma A.9, we have

Summing over t from 0 to T , we get the desired result.

Lemma A.12. For any γ ≤ 1 20 ndL , we have

Proof. Similar to the proof of Lemma A.11, we have

Invoking Lemma A.5 and Assumption 1.2, we obtain

It follows by summing over t from 0 to T and applying Lemma 3.2 that

Hence, under the condition that γ ≤ 1 20 ndL , there holds 1200 γ 2 n 4 ( d -1) 6 L 4 ≤ 4 n 2 ( d -1) 4 L 2 , which implies the desired result.

Proof. By definition, we have

Thus, invoking Lemma A.5 and Lemma 3.1, we have

After summing over t from 0 to T , we get the desired result.

Proof. Note that

We show the upper bounds for ∑ T t =0 ∥ Q 0 ,t,i ∥ 2 F , where i = 1 , 3 respectively. Based on Corollary A.2, Lemma A.5 and Lemma A.9, we have the following result:

Then, recall that the summation is legal only when t ≤ 2( d -2) . We have

Combining the above upper bounds leads to the final result.

Proof. Note that

where the last equality comes from extending the summation in the first line and telescoping the summation. Consequently, we have

Then, taking the F-norm on both sides and invoking Lemma A.6, Lemma A.5 and Lemma 3.2 as before, we have

Taking expectation on both sides and summing over t from 0 to T , we get

Back to equation (13), note that

Taking full expectation on both sides, summing over t from 0 to T and combining Lemma A.11 to Lemma A.14, we have

For γ ≤ 1 40 nd 2 L , which implies that 312 γ 2 n 2 d 4 L 2 +12 γ 2 n 2 d 2 L 2 ≤ 1 4 , we can simplify equation (14) as follows:

which implies the desired result.

A.3.5 Proof of Lemma 3.4

Proof. Notice that

̸

Therefore, x ( t ) 1 does not depend on ξ t -1 i for i = 1 . We iterate the above procedure to get

̸

Similar to x ( t ) 1 , we know that x ( t -1) 1 does not depend on ξ ( t -2) i , i = 1 . Hence x ( t ) 1 is independent with ξ ( t -2) i for i / ∈ I 1 , 1 . By iterating the procedure, we conclude that x ( t ) 1 is independent with ξ ( t -k ) i for i / ∈ I 1 ,k .

Consequently, by choosing Z = { ξ ( t -k ) i , i ∈ I 1 ,k , i / ∈ I 1 ,k -1 } , X = x ( t ) 1 , Y = { x ( t -k ) i , i ∈ [ n ] } in Lemma A.8, Z is independent with ( X,Y ) , we get

Then, invoking Corollary A.4, we have

A.3.6 Proof of Lemma 3.5

Proof. By Assumption 1.2, the function f := 1 n ∑ n i =1 f i is L -smooth. Then,

For the last term, we have

For the second last term, we have

We now bound the two terms in the above equation. Firstly,

Recall that by equation (9),

We bound the four terms above one by one. For the first term,

For the second one, invoking Lemma 3.4, we have

For the last two terms in (17), we have as:

By the Cauchy-Schwartz inequality, we have

Thus, combining the above inequalities together yields

Secondly, for the second term in (16),

Summing over t from 0 to T , we have

It yields by summing over t from 0 to T on both sides of equation (15) that

With the results above, given ∆ f = f ( x 0 ) -f ∗ , we have

Invoking Lemma 3.2, we have for γ ≤ 1 100 nd 3 L ( ≤ 1 10 ndL ) that

where it holds that 150 γ 3 n 2 d 4 L 4 +25 γ 2 nd 2 L 3 +5 γd 2 L 2 ≤ 6 γd 2 L 2 . Invoking Lemma 3.3, we have for γ ≤ 1 3 ( ≤ 1 2 ) that

100 nd L 40 nd L

Thus, for γ ≤ 1 100 nd 3 L , we have

After re-arranging the terms, we conclude that

A.3.7 Proof of Lemma 3.7

Let x ∗ = arg min x f ( x ) . We start with analyzing the behavior of ∥ x ( t ) 1 -x ∗ ∥ 2 after obtaining Lemma 3.4. It holds that

To deal with the critical inner product, similar to the decomposition in Equation (16), we have, by replacing ∇ f ( x ( t ) 1 ) with x ( t ) 1 -x ∗ in Equation (17), and invoking Lemma 3.4 as we have done in Lemma 3.5, that

and

Notice that, first by strong convexity of f (Assumption 1.3) and then by L -smoothness (Assumption 1.2), there holds

Then,

Thus, plugging the above results into Equation (18), with κ := L/µ as the conditional number, it holds that

We derive the convergence result by several standard steps as follows. Step 1 . Unwinding the above recursion, it follows by 1 2 ≤ 1 -nγµ 4 ≤ 1 for γ ≤ 1 10 nd 2 κL that

Step 2 . To refine Lemma 3.2, we start with Equation (11) multiplied by the coefficient ( 1 -nγµ 4 ) T -t before summing over t in Equation (10) from 0 to T . Then, we have, for γ ≤ 1 10 nd 2 κL ( 1 2 ≤ 1 -nγµ 4 ≤ 1 for γ ≤ 1 10 nd 2 κL ), that

where we get the result which only modifies the coefficient of the term ∥ ∥ ∇ F ( X (0) ) ∥ ∥ 2 F . Implementing the above result, we have, for γ ≤ 1 10 nd 2 κL ( ≤ 1 10 ndL ) ,

Step 3 . Similarly, we refine Lemma 3.3 as follows. By multiplying ( 1 -nγµ 4 ) T -t before summing over t in Equation (14), we have

Implementing the above result, we have for γ ≤ 1 40 nd 2 κL that

and

Step 4 . Note that the coefficient of the term ∑ T t =0 ∥ ∥ ∥ ∇ f ( x ( t ) 1 ) ∥ ∥ ∥ 2 is smaller than 0 for γ ≤ 1 100 nd 2 κL , derived as follows:

Thus, it holds that

A.4 Proof of the Convergence Results

A.4.1 Proof of Theorem 2.1

Invoking Lemma 3.5, with identical initial values x (0) i that implies ∥ ∥ Π u X (0) ∥ ∥ 2 F = 0 , we have

Referring to Lemma 26 in [8], as stated in Lemma A.7, by taking A = 8∆ f n , B = 24 σ 2 L , C = 20000 nd 6 σ 2 L 2 and α = 100 nd 3 L , when considering γ = min { ( ∆ f 3 σ 2 Ln ( T +1) ) 1 2 , ( ∆ f 1500 n 2 d 6 σ 2 L 2 ( T +1) ) 1 3 , 1 100 nd 3 L } , we have

A.4.2 Proof of Theorem 2.4

Invoking Lemma 3.7, with identical values x (0) i that implies ∥ ∥ Π u X (0) ∥ ∥ 2 F = 0 , we have

Considering γ = min { 1 100 nd 2 κL , 16 log( n ( T +1) 2 ) n ( T +1) µ } , for T ≥ 2 d we get that

and

where we use the fact (1 -1 x ) x ≤ e -1 and 1 -x ≤ exp( -x ) for any x ∈ R + . Then,

B Additional Experiments

For the problem of training a CNN on the MNIST dataset, we have further compared the real-time performance of BTPP with other representative methods. The experiments are conducted on a server equipped with eight Nvidia RTX 3090 GPUs and two Intel Xeon Gold 4310 CPUs, where the

Train Loss

Time

2.2

0.95

BTPP,B=2

BTPP, B=4

BTPP, B=8

0.9

1.8

BTPP, B=16

DSGD-Base-(k+1)

1.6

1.4

DSGT-Ring

DSGD-OnePeerExp

DSGD-Exponential

Accuracy (%)

0.85

LoSS

DSGD-FullyConnected

1.2

0.8

BTPP,B=2

1

BTPP, B=4

0.75

BTPP, B=8

0.8

BTPP,B=16

DSGD-Base-(k+1)

0.6

DSGD-OnePeerExp

0.7

DSGT-Ring

0.4

DSGD-Exponential

DSGD-FullyConnected

0.2

0.65

0

50

100

150

200

250

0

50

100

150

200

250

TestAccuracy

Time

communication between GPUs follows the topology requirement of each algorithm. We measure the running time including GPU computation and communication for 13,000 iterations. The experimental settings are consistent with those described in Section 4.2. From Figure 4, BTPP outperforms the other algorithms concerning the running time. Additionally, we evaluate BTPP with various branch sizes B , concluding that for relatively small values of n , a branch size of B = 2 is most effective.

Furthermore, we consider training VGG13 on the CIFAR10 dataset, with n = 8 and a batch size of 16. The learning rate and topology configurations are consistent with those described in Section 4.2. Additionally, the case of BTPP with B = 8 is equivalent to DSGD in a fully connected setting, meaning that they produce identical outputs when using the same random seed. Figure 5 and Figure 6 illustrate that BTPP beats competing algorithms in terms of the convergence rate (against iteration number) and running time. Moreover, a branch size of B = 2 is optimal.

Train Loss

TestAccuracy

2.2

0.85

BTPP,B=2

2

BTPP, B=4

0.8

DSGD-Ring

1.8

DSGT-Ring

DSGD-OnePeerExp

0.75

1.6

DSGD-Base-(k+1)

DSGD-Exponential

0.7

DSGD-FullyConnected

0.65

LoSS

1.4

0.6

1.2

0.55

BTPP,B=2

BTPP,B=4

0.5

DSGD-Ring

DSGT-Ring

0.8

0.45

DSGD-OnePeerExp

DSGD-Base-(k+1)

0.6

0.4

DSGD-Exponential

DSGD-FullyConnected

0.4

0.35

0

2000

4000

6000

8000

10000

12000

14000

0

2000

4000

6000

8000

10000

12000

14000

Iteration

Iteration

We further demonstrate that the performance of BTPP can be improved by incorporating a momentum term (with momentum parameter set to 0.9) when data heterogeneity exists or by removing the data heterogeneity, which involves randomly assigning samples to each agent; see Figure 7 and Figure 8.

Train Loss

TestAccuracy

2.2

0.85

BTPP,B=2

2

BTPP, B=4

0.8

BTPP, B=8

1.8

DSGD-Ring

0.75

DSGT-Ring

1.6

DSGD-OnePeerExp

DSGD-Base-(k+1)

0.7

DSGD-Exponential

0.65

LOSS

1.4

DSGD-FullyConnected

0.6

1.2

BTPP,B=2

0.55

BTPP, B=4

BTPP,B=8

0.5

DSGD-Ring

DSGT-Ring

0.8

0.45

DSGD-OnePeerExp

DSGD-Base-(k+1)

0.6

0.4

DSGD-Exponential

DSGD-FullyConnected

0.4

0.35

0

0.5

1

1.5

2

2.5

3

3.5

0

0.5

1

1.5

2

2.5

3

3.5

Time

X104

Time

X104

2.5

BTPP

BTPP with Momentum

BTPP without Data Heterogeneity

.5

0.5

=0=0=0

0

2000

4000

6000

8000

10000

12000

14000

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0

2000

4000

6000

BTPP

BTPP with Momentum

BTPP without Data Heterogeneity

8000

10000

12000

14000

2.5

BTPP

BTPP with Momentum

BTPPwithout Data Heterogeneity

1.5

0.5

0

0

2000

4000

6000

8000

10000

12000

14000

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

2000

4000

6000

BTPP

BTPP with Momentum

BTPP without Data Heterogeneity

8000

10000

12000

14000

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our paper meticulously delineates its primary contributions in both the abstract (lines 8-9) and the introduction (subsection 1.2, lines 105-121).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: At the end of section 1.1, lines 100-104.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions are in subsection 1.3, lines 129-132. The proofs of all theoretical results are in the Appendix and subsection 2.1.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We present the experiment details in section 4 and upload our code in the link shown in the abstract.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide our code with the link shown in lines 10-11.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all the experimental details and settings in Section 4 and our code link.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We have repeated some experiments with different seeds and reported the averaged performance. See Section 4 for details.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See line 255 for reference.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the original paper that produced the code package or dataset.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer:[NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper proposes the B-ary Tree Push-Pull (BTPP) algorithm for distributed learning over heterogeneous data. The BTPP algorithm introduces 2 spanning directed trees as communication graphs: Pull Tree $G_R$ and Push Tree $G_C$. BTPP has only $\Theta(1)$ communication overhead per iteration and $O(n)$ transient time complexity but outperforms the baseline. The paper also theoretically shows that BKPP is of sublinear convergence rate. The experimental results shown in the paper also support that BTPP outperforms the SOTA baseline.

STRENGTHS:
1. BTPP combines two directed trees, the Pull Tree and the Push Tree, for distributed learning. 
2. Since the degree for the Tree is at most $O(n)$ with n nodes, the communication time complexity for BTPP is lower than other methods. At the same time, the result outperforms other methods.
3. The paper provides theoretical guarantees for the BTPP algorithm.

WEAKNESSES:
1. BTPP algorithm is derived from the Push-Pull gradient method in [1] by defining the communication graph as B-ary trees. The mathematical formulation and pseudocode Algorithm 1 looks the same as that in [1]. What is the main difference between the two algorithms? 

2. In the paper, the authors state that the Pull graph $G_R$ is the inverse of the Push graph $G_C$. What is the reason that the two graphs must have the same structure?

3. The authors claim that the advantage of BTPP is that it could reach better performance than the SOTA with lower communication costs. However, in section 4, the paper only shows iteration-wise test performance rather than the time-wise performance.

[1] Push-Pull Gradient Methods for Distributed Optimization in Networks

QUESTIONS:
See weaknesses.

LIMITATIONS:
See weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper reduces transient time in decentralized optimization algorithms by introducing the B-ary Tree Push-Pull (BTPP) algorithm, derived from the Push-Pull method. The authors assume a setup where any type of communication network can be established among the nodes. They employ two distinct communication networks: the network represented by graph $R$ for pulling parameter information, and the network represented by graph $C$ for pushing stochastic gradient information towards node 1. The method achieves linear speedup for smooth nonconvex objective functions with only $O(n)$ transient iterations, significantly outperforming existing state-of-the-art methods.\n\nSTRENGTHS:\n- Reduction of transient time in decentralized optimization\n - Novel convergence approach\n - Demonstrating the use of B-ary trees to balance between communications per iteration and transient time\n\nWEAKNESSES:\n1. The introduction lacks cohesion and focus, with unnecessary information in the second paragraph. The significance of the transient time problem, a key contribution of the paper, could be better defined to highlight its importance.\n\n 2. The paper demonstrates that only node 1 converges to the local solution. However, it is not entirely clear what happens to the variables at other nodes or whether the nodes achieve consensus similar to classical decentralized algorithms. \n\n 3. The proposed setup appears to be limited to high-performance data-center clusters where a B-ary spanning tree communication network can be established among nodes. This might limit the applicability of the algorithm in other environments.\n\n 4. Additionally, the analysis is focused on smooth nonconvex objective functions, and there is no discussion on strongly convex functions.\n\nQUESTIONS:\n1. Refer to weaknesses. \n\n2. Could you please elaborate on how the BTPP algorithm addresses data heterogeneity problems? Specifically, which term in equation (3) demonstrates that data heterogeneity decreases as the algorithm progresses?\n\n3. I am interested in the fixed point analysis of BTPP. The Push-Pull algorithm has a comprehensive fixed point analysis. In your algorithm, if all nodes converge to a stationary solution, will the algorithm maintain that state? Additionally, to which points do the other nodes converge, apart from node 1?\n\nLIMITATIONS:\nRefer to Weaknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a method for distributed learning, for the common scenario that various \"agents\" each hold part of a data set locally, and aim to coordinate to find the minimizer of some criterion. Here, the criterion consists of the average of local cost functions, which are assumed to be smooth but non-convex, and each agent is assumed to able to compute a \"gradient estimator\". By communicating information concerning these gradient estimators, the agents together minimize global criterion (i.e. averaged local criteria).\n\nThe method consists of both a specific network architecture/topology, combined with an algorithm for what each agent computes and communicates throughout the network. Unbiased estimators of the local gradients are generally assumed to be available. The authors consider possibly heterogenous data, but this plays no role beyond just extending generality it seems.\n\nThe analysis of the authors focusses mainly on the number of transient iterations required by the algorithm. Communication cost does not take into account the actual size of the gradient estimator, but concerns the number of neighbors that a message is to be passed onto by a node, for every node and at every iteration.\n\nThe authors provide a main theorem describing the convergence properties of the algorithm and simulations. Both are used for comparison with existing methods in the literature.\n\nSTRENGTHS:\n* The paper provides a rigorous theoretical analysis, providing clear convergence guarantees\n\n* The method described by the paper has clear advantages in comparison to various existing algorithms in the literature, in terms of the number of transient iterations and in terms of total communication cost.\n\n* The method is practically relevant in distributed settings where one has control over the design of the network nodes.\n\n* The numerical results are well-chosen, and provide convincing support for the theoretical guarantees in small samples / finite iterations, in comparison to many existing methods in the literature.\n\n* The paper is well written (up to some punctuation mistakes), from the introduction lasting into the appendix, and its analysis appears sound to me.\n\nWEAKNESSES:\nMy main grief with setup considered by the authors is the (in my opinion very strong assumption) that there is no bias in the gradient estimator. Since the authors seem to target high-dimensional setups, this is often not case, unless one has access to the original gradient. Furthermore, it is difficult to verify whether one has zero bias. \n\nIt would be a great improvement if the authors captured the importance having (very) small bias in their theory. Is 0 bias strictly necessary? How much bias can one get away with here?\n\nSimilarly, the uniform bound on the variance of the gradient estimator seems rather strong. Going through the analysis provided by the authors, I also wonder whether uniformity is generally necessary here. Since this appears as a constant in the main theorem (i.e. in (3)), it might in fact significantly impact the convergence of the algorithm, and I think it would be wise for the authors to comment on this further.\n\nQUESTIONS:\n* The role of $n$ is somewhat surprising in (3), making many of the terms smaller as the data gets distributed across more servers. Do the authors have an interpretation for these terms / an intuition as to why this is? In principle, I would say that the more servers one considers, the more difficult the problem should get in principle. Is the appearance of $n$ in the dominator a consequence of total communication increasing as $n$ increases?\n\n* The authors comment on the choice of B in Remark 2.3 in terms of the upper bound given in (3); ending with the sentence \"..the communication cost and convergence rate can be balanced by considering a proper B\". How should this be done in practice? Is there a practical guideline that the authors have here? The insight provided by (3) seem rather limited here, as it is only an upper bound.\n\n* If one does not have full control when designing the network topology, what are the take-aways from the current design?\n\nLIMITATIONS:\n* The assumptions on the gradient estimator (mean 0, uniformly bounded variance) are very strong.\n* The interpretation of the quantities appearing could be somewhat more extensive.\n* The attention spent on //why// this network design considered by the authors leads to such an improvement in terms of transient times is limited.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces B-ary Tree Push-Pull (BTPP), an extension of the push-pull framework for distributed learning across a network of agents. The algorithm employs two B-ary tree communication graphs - one for distributing model parameters and one for aggregating gradients. This approach allows each agent to communicate with at most B+1 neighbors per iteration. The main contribution is an analysis demonstrating that BTPP achieves linear speedup with O(n) transient iterations for smooth nonconvex objectives, improving upon previous results for related methods while maintaining lower per-iteration communication.\n\nSTRENGTHS:\n- Novel use of B-ary tree topologies within the push-pull framework\n- Improved theoretical results showing O(n) transient time\n\nWEAKNESSES:\n- The experimental evaluation could be more comprehensive, particularly given that the analysis extends to smooth non-convex problems. Empirical evaluations on deep learning tasks or other practical non-convex optimization problems would strengthen the paper's claims and demonstrate real-world applicability.\n- While the improvement in transient time is theoretically significant, the paper could benefit from a more thorough discussion of how this translates to practical advantages in real-world distributed learning scenarios, especially for large-scale or long-running optimization tasks.\n- The approach is primarily applicable to controlled network environments like high-performance data-center clusters, limiting its relevance to scenarios with fixed or constrained network topologies (e.g., wireless sensor networks, internet of vehicles).\n\nQUESTIONS:\n- How does BTPP compare to other extensions of push-pull in the literature?\n- Can this B-ary tree approach be applied to other distributed optimization algorithms? What are the main limitations?\n- How sensitive is the performance to the choice of B? Is there an optimal B value?\n- Can the analysis be extended to strongly convex or convex objective functions?\n\nLIMITATIONS:\nThe authors acknowledge a fundamental limitation common in this area of research: the assumption of a controlled network environment typical of high-performance data-center clusters. This assumption, where network topology can be fully manipulated, is inherent to the proposed BTPP algorithm and many related works in the literature. It's recognized that this limits applicability to scenarios with fixed or constrained network topologies, such as wireless sensor networks or IOT applications.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}