Title: Beyond Exponential Graph: Communication-Efficient Topologies for Decentralized Learning via Finite-time Convergence
Venue: NeurIPS 2023
Accepted: yes
Keywords: decentralized learning

distributed optimization

network topology

consensus rate
Abstract: Decentralized learning has recently been attracting increasing attention for its applications in parallel computation and privacy preservation. Many recent studies stated that the underlying network topology with a faster consensus rate (a.k.a. spectral gap) leads to a better convergence rate and accuracy for decentralized learning. However, a topology with a fast consensus rate, e.g., the exponential graph, generally has a large maximum degree, which incurs significant communication costs. Thus, seeking topologies with both a fast consensus rate and small maximum degree is important. In this study, we propose a novel topology combining both a fast consensus rate and small maximum degree called the Base-$\left(k+1\right)$ Graph. Unlike the existing topologies, the Base-$\left(k+1\right)$ Graph enables all nodes to reach the exact consensus after a finite number of iterations for any number of nodes and maximum degree $k$. Thanks to this favorable property, the Base-$\left(k+1\right)$ Graph endows Decentralized SGD (DSGD) with both a faster convergence rate and more communication efficiency than the exponential graph. We conducted experiments with various topologies, demonstrating that the Base-$\left(k+1\right)$ Graph enables various decentralized learning methods to achieve higher accuracy with better communication efficiency than the existing topologies. Our code is available at https://github.com/yukiTakezawa/BaseGraph.

Beyond Exponential Graph: Communication-Efficient Topologies for Decentralized Learning via Finite-time Convergence

Abstract

Decentralized learning has recently been attracting increasing attention for its applications in parallel computation and privacy preservation. Many recent studies stated that the underlying network topology with a faster consensus rate (a.k.a. spectral gap) leads to a better convergence rate and accuracy for decentralized learning. However, a topology with a fast consensus rate, e.g., the exponential graph, generally has a large maximum degree, which incurs significant communication costs. Thus, seeking topologies with both a fast consensus rate and small maximum degree is important. In this study, we propose a novel topology combining both a fast consensus rate and small maximum degree called the BASE-( k +1) GRAPH. Unlike the existing topologies, the BASE-( k + 1) GRAPH enables all nodes to reach the exact consensus after a finite number of iterations for any number of nodes and maximum degree k . Thanks to this favorable property, the BASE-( k +1) GRAPH endows Decentralized SGD (DSGD) with both a faster convergence rate and more communication efficiency than the exponential graph. We conducted experiments with various topologies, demonstrating that the BASE-( k +1) GRAPH enables various decentralized learning methods to achieve higher accuracy with better communication efficiency than the existing topologies. Our code is available at https://github.com/yukiTakezawa/BaseGraph .

1 Introduction

Distributed learning, which allows training neural networks in parallel on multiple nodes, has become an important paradigm owing to the increased utilization of privacy preservation and largescale machine learning. In a centralized fashion, such as All-Reduce and Federated Learning [8, 9, 16, 26, 27], all or some selected nodes update their parameters by using their local dataset and then compute the average parameter of these nodes, although computing the average of many nodes is the major bottleneck in the training time [18, 19, 23]. To reduce communication costs, decentralized learning gains significant attention [11, 18, 24]. Because decentralized learning allows nodes to exchange parameters only with a few neighbors in the underlying network topology, decentralized learning is more communication efficient than All-Reduce and Federated Learning.

While decentralized learning can improve communication efficiency, it may degrade the convergence rate and accuracy due to its sparse communication characteristics [11, 48]. Specifically, the smaller the maximum degree of an underlying network topology is, the fewer the communication cost becomes [33, 39]; meanwhile, the faster the consensus rate (a.k.a. spectral gap) of a topology is, the faster the convergence rate of decentralized learning becomes [11]. Thus, developing a topology with both a fast consensus rate and small maximum degree is essential for decentralized learning. Table 1 summarizes the properties of various topologies. For instance, the ring and exponential graph are

commonly used [1, 3, 12, 23]. The ring is a communication-efficient topology because its maximum degree is two but its consensus rate quickly deteriorates as the number of nodes n increases [28]. The exponential graph has a fast consensus rate, which does not deteriorate much as n increases, but it incurs significant communication costs because its maximum degree increases as n increases [43]. Thus, these topologies sacrifice either communication efficiency or consensus rate.

Recently, the 1 -peer exponential graph [43] and 1 -peer hypercube graph [31] were proposed as topologies that combine both a small maximum degree and fast consensus rate (see Sec. C.4 for examples). As Fig. 1 shows, in the ring and exponential graph, node parameters only reach the consensus asymptotically by repeating exchanges of parameters with neighbors. Contrarily, in the 1 -peer exponential and 1 -peer hypercube graphs, parameters reach the exact consensus after a finite number of iterations when n is a power of 2 (see Fig. 21 in Sec. F.2). Thanks to this property of finite-time convergence, the 1 -peer exponential and 1 -peer hypercube graphs enable Decentralized SGD (DSGD) [18] to converge at the same convergence rate as the exponential graph when n is a power of 2 , even though the maximum degree of the 1 -peer exponential and 1 -peer hypercube graphs is only one [43].

However, this favorable property only holds when n is a power of 2 . When n is not a power of 2 , the 1 -peer hypercube graph cannot be constructed, and the 1 -peer exponential graph only reaches the consensus asymptotically as well as the ring and exponential graph, as Fig. 1 illustrates. Thus, the 1 -peer exponential and 1 -peer hypercube graphs cannot enable DSGD to converge as fast as the exponential graph when n is not a power of 2 . Moreover, even if n is a power of 2 , the 1 -peer hypercube and 1 -peer exponential graphs still cannot enable DSGD to converge faster than the exponential graph.

0

10

20

30

40

50

Iterations

10

25

10

21

10

17

10

13

10

9

10

5

10

1

Consensus error

n

=1500

Exp. (11)

Ring (2)

Torus (4)

1-peer Exp. (1)

1-peer D-EquiDyn (1)

1-peer U-EquiDyn (1)

Base

-

2

Graph

(1)

Base

-

3

Graph

(2)

Base

-

4

Graph

(3)

Base

-

5

Graph

(4)

In this study, we ask the following question: Can we construct topologies that provide DSGD with both a faster convergence rate and better communication efficiency than the exponential graph for any number of nodes? Our work provides the affirmative answer by proposing the BASE-( k +1) GRAPH, 2 which is finite-time convergence for any number of nodes n and maximum degree k (see Fig. 1). Thanks to this favorable property, the BASE2 GRAPH enables DSGD to converge faster than the ring and torus and as fast as the exponential graph for any n , while the BASE2 GRAPH is more communication-efficient than the ring, torus, and exponential graph because its maximum degree is only one. Furthermore, when 2 ≤ k < ⌈ log 2 ( n ) ⌉ , the BASE-( k +1) GRAPH enables DSGD to converge faster with fewer communication costs than the exponential graph because the maximum degree of the BASE-( k +1) GRAPH is still less than that of the exponential graph. Experimentally, we compared the BASE-( k +1) GRAPH with various existing topologies, demonstrating that the BASE-( k +1) GRAPH enables various decentralized learning methods to more successfully reconcile accuracy and communication efficiency than the existing topologies.

2 Related Work

Decentralized Learning. The most widely used decentralized learning methods are DSGD [18] and its adaptations [1, 2, 19]. Many researchers have improved DSGD and proposed DSGD with momentum [4, 20, 44, 45], communication compression methods [6, 10, 23, 35, 38], etc. While DSGD is a simple and efficient method, DSGD is sensitive to data heterogeneity [36]. To mitigate this issue, various methods have been proposed, which can eliminate the effect of data heterogeneity on the convergence rate, including gradient tracking [21, 29, 30, 34, 42, 47], D 2 [36], etc. [17, 37, 46].

Effect of Topologies. Many prior studies indicated that topologies with a fast consensus rate improve the accuracy of decentralized learning [11, 13, 25, 39]. For instance, DSGD and gradient tracking converge faster as the topology has a faster consensus rate [18, 34]. Zhu et al. [48] revealed that topology with a fast consensus rate improves the generalization bound of DSGD. Especially when the data distributions are statistically heterogeneous, the topology with a fast consensus rate prevents the parameters of each node from drifting away and can improve accuracy [11, 34]. However, communication costs increase as the maximum degree increases [33, 43]. Thus, developing a topology with a fast consensus rate and small maximum degree is important for decentralized learning.

3 Preliminary and Notation

Notation. A graph G is represented by ( V, E ) where V is a set of nodes and E is a set of edges. If G is a graph, V ( G ) (resp. E ( G ) ) denotes the set of nodes (resp. edges) of G . For any a 1 , · · · , a n , ( a 1 , · · · , a n ) denotes the ordered set. An empty (ordered) set is denoted by ∅ . For any n ∈ N , let [ n ] := { 1 , · · · , n } . For any n, a ∈ N , mod ( a, n ) is the remainder of dividing a by n . ∥ · ∥ F denotes Frobenius norm, and 1 n denotes an n -dimensional vector with all ones.

Topology. Let G be an underlying network topology with n nodes, and W ∈ [0 , 1] n × n be a mixing matrix associated with G . That is, W ij is the weight of the edge ( i, j ) , and W ij > 0 if and only if ( i, j ) ∈ E ( G ) . Most of the decentralized learning methods require W to be doubly stochastic (i.e., W 1 n = 1 n and W ⊤ 1 n = 1 n ) [18, 20, 29, 36]. Then, the consensus rate of G is defined below.

Definition 1. Let W be a mixing matrix associated with a graph G with n nodes. Let x i ∈ R d be a parameter that node i has. Let X := ( x 1 , · · · , x n ) ∈ R d × n and ¯ X := 1 n X 1 n 1 ⊤ n . The consensus rate β ∈ [0 , 1) is the smallest value that satisfies ∥ ∥ XW -¯ X ∥ ∥ 2 F ≤ β 2 ∥ ∥ X -¯ X ∥ ∥ 2 F for any X .

Thanks to β ∈ [0 , 1) , x i asymptotically converge to consensus 1 n ∑ n i =1 x i by repeating parameter exchanges with neighbors. However, this does not mean that all nodes reach the exact consensus within a finite number of iterations except when β = 0 , that is, when G is fully connected. Then, utilizing time-varying topologies, Ying et al. [43] and Shi et al. [31] aimed to obtain sequences of graphs that can make all nodes reach the exact consensus within finite iterations and proposed the 1 -peer exponential and 1 -peer hypercube graphs respectively (see Sec. C.4 for illustrations).

Definition 2. Let ( G (1) , · · · , G ( m ) ) be a sequence of graphs with the same set of nodes (i.e., V ( G (1) ) = · · · = V ( G ( m ) ) ). Let n be the number of nodes. Let W (1) , · · · , W ( m ) be mixing matrices associated with G (1) , · · · , G ( m ) , respectively. Suppose that W (1) , · · · , W ( m ) satisfy XW (1) W (2) · · · W ( m ) = ¯ X for any X ∈ R d × n , where ¯ X = 1 n X 1 n 1 ⊤ n . Then, ( G (1) , · · · , G ( m ) ) is called m -finite time convergence or an m -finite time convergent sequence of graphs.

Because Definition 2 assumes that V ( G (1) ) = · · · = V ( G ( m ) ) holds, we often write a sequence of graphs ( G (1) , · · · , G ( m ) ) as ( E ( G (1) ) , · · · , E ( G ( m ) )) using a slight abuse of notation. Additionally, in the following section, we often abbreviate the weights of self-loops because they are uniquely determined due to the condition that the mixing matrix is doubly stochastic.

4 Construction of Finite-time Convergent Sequence of Graphs

In this section, we propose the BASE-( k + 1) GRAPH, which is finite-time convergence for any number of nodes n ∈ N and maximum degree k ∈ [ n -1] . Specifically, we consider the setting where node i has a parameter x i and propose a graph sequence whose maximum degree is at most k that makes all nodes reach the exact consensus 1 n ∑ n i =1 x i . To this end, we first propose the k -PEER HYPER-HYPERCUBE GRAPH, which is finite-time convergence when n does not have prime factors larger than k +1 . Using it, we propose the SIMPLE BASE-( k +1) GRAPH and BASE-( k +1) GRAPH, which are finite-time convergence for any n .

1

2

3

4

5

6

1

2

3

4

5

6

(a)

n

= 6(= 2

×

3)

1

2

3

4

5

6

7

8

9

10

11

12

1

2

3

4

5

6

7

8

9

10

11

12

1

2

3

4

5

6

7

8

9

10

11

12

(b)

n

= 12(= 2

×

2

×

3)

4.1 k -peer Hyper-hypercube Graph

Before proposing the BASE-( k +1) GRAPH, we first extend the 1 -peer hypercube graph [31] to the k -peer setting and propose the k -PEER HYPER-HYPERCUBE GRAPH, which is finite-time convergence when the number of nodes n does not have prime factors larger than k +1 and is used as a component in the BASE-( k +1) GRAPH. Let V be a set of n nodes. We assume that all prime factors of n are less than or equal to k +1 . That is, there exists n 1 , n 2 , · · · , n L ∈ [ k +1] such that n = n 1 ×···× n L . In this case, we can construct the L -finite time convergent sequence of graphs whose maximum degree is at most k . Using Fig. 2a, we explain how all nodes reach the exact consensus. Let G (1) and G (2) denote the graphs in Fig. 2a from left to right, respectively. After the nodes exchange parameters with neighbors in G (1) , nodes 1 and 2 , nodes 3 and 4 , and nodes 5 and 6 have the same parameter respectively. Then, after exchanging parameters in G (2) , all nodes reach the exact consensus. We present the complete algorithms for constructing the k -PEER HYPER-HYPERCUBE GRAPH in Alg. 1.

4.2 Simple Base-( k +1) Graph

As described in Sec. 4.1, when n do not have prime factors larger than k +1 , we can easily make all nodes reach the exact consensus by the k -PEER HYPER-HYPERCUBE GRAPH. However, when n has prime factors larger than k +1 , e.g., when ( k, n ) = (1 , 5) , the k -PEER HYPER-HYPERCUBE GRAPH cannot be constructed. In this section, we extend the k -PEER HYPER-HYPERCUBE GRAPH and propose the SIMPLE BASE-( k +1) GRAPH, which is finite-time convergence for any number of nodes n and maximum degree k . Note that the maximum degree of the SIMPLE BASE-( k +1) GRAPH is not k +1 , but at most k .

We present the pseudo-code for constructing the SIMPLE BASE-( k + 1) GRAPH in Alg. 2. For simplicity, here we explain only the case when the maximum degree k is one. The case with k ≥ 2 is explained in Sec. B. The SIMPLE BASE-( k +1) GRAPH mainly consists of the following four steps. The key idea is that splitting V into disjoint subsets to which the k -PEER HYPER-HYPERCUBE GRAPH is applicable.

Step 1. As in the base2 number of n , we decompose n as n = 2 p 1 + · · · +2 p L in line 1, and then split V into disjoint subsets V 1 , · · · , V L such that | V l | = 2 p l for all l ∈ [ L ] in line 3 . 3

Step 2. For all l ∈ [ L ] , we make all nodes in V l obtain the average of parameters in V l using the 1 -PEER HYPER-HYPERCUBE GRAPH H 1 ( V l ) in line 11. Then, we initialize l ′ as one.

Step 3. Each node in V l ′ +1 ∪ · · · ∪ V L exchanges parameters with one node in V l ′ such that the average in V l ′ becomes equivalent to the average in V . We increase l ′ by one and repeat step 3 until l ′ = L . This procedure corresponds to line 15.

Step 4. For all l ∈ [ L ] , we make all nodes in V l obtain the average in V l using the 1 -PEER HYPERHYPERCUBE GRAPH H 1 ( V l ) . Because the average in V l is equivalent to the average in V after step 3 , all nodes can reach the exact consensus. This procedure corresponds to line 25.

Using the example presented in Fig. 3, we explain Alg. 2 in more detail. Let G (1) , · · · , G (5) denote the graphs in Fig. 3 from left to right, respectively. In step 1 , we split V := { 1 , · · · , 5 } into

V 1 := { 1 , · · · , 4 } and V 2 := { 5 } . In step 2 , nodes in V 1 obtain the average in V 1 by exchanging parameters in G (1) and G (2) . In step 3 , the average in V 1 becomes equivalent to the average in V by exchanging parameters in G (3) . In step 4 , nodes in V 1 can get the average in V by exchanging parameters in G (4) and G (5) . Because node 5 also obtains the average in V after exchanging parameters in G (3) , all nodes reach the exact consensus after exchanging parameters in G (5) .

Note that edges added in lines 20 and 27 are not necessary if we only need to make all nodes reach the exact consensus. Nonetheless, these edges are effective in keeping the parameters of nodes close in value to each other in decentralized learning because the parameters are updated by gradient descent before the parameter exchange with neighbors. For instance, edge (1 , 2) in G (3) , which is added in line 20, is not necessary for finite-time convergence because nodes 1 and 2 already have the same parameter after exchanging parameters in G (1) and G (2) . We provide more examples in Sec. C.

1

2

1

2

1

2

3

4

5

1

2

1

2

1

2

3

4

5

1

2

4

5

1

2

3

4

5

1

2

1

2

1

2

3

4

5

1

2

1

2

1

2

3

4

5

Algorithm 2 simple

̸

̸

.

Algorithm 3 BASE-( k +1) GRAPH A k ( V )

1: Input: the set of nodes V and number of nodes n .

2: Decompose n as n = p × q such that p is a multiple of 2 , 3 , · · · , ( k + 1) and q is prime to 2 , 3 , · · · , ( k +1) .

3: Split V into disjoint subsets V 1 , · · · , V p such that | V l | = q for all l .

4: Construct SIMPLE BASE-( k +1) GRAPH A simple k ( V l ) for all l ∈ [ p ] .

6: E ( m ) ← ⋃ l ∈ [ p ] E ( A simple k ( V l ) ( m ) ) .

5: for m ∈ { 1 , 2 , · · · , |A simple k ( V 1 ) |} do

7: Split V into disjoint subsets U 1 , · · · , U q such that | U l | = p and | U l ∩ V l ′ | = 1 for all l, l ′ .

8: Construct k -PEER HYPER-HYPERCUBE GRAPH H k ( U l ) for all l ∈ [ q ] .

9: for m ∈ { 1 , 2 , · · · , |H k ( U 1 ) |} do

10: E ( m + |A simple k ( V 1 ) | ) ← ⋃ l ∈ [ q ] E ( H k ( U l ) ( m ) ) .

11: E ← ( E (1) , E (2) , · · · , E ( |A simple k ( V 1 ) | + |H k ( U 1 ) | ) ) .

12: If |A simple k ( V ) | < |E| then return A simple k ( V ) else return E .

1

2

1

2

1

2

3

4

5

6

2

3

2

3

1

2

3

4

5

6

1

2

1

2

1

2

3

4

5

6

1

2

1

2

1

2

1

2

3

4

5

6

(a) BASE-

2

GRAPH with

n

= 6(= 2

×

(2 + 1))

1

2

1

2

1

2

1

2

3

4

5

6

1

2

1

2

1

2

1

2

3

4

5

6

1

2

2

3

2

3

1

2

3

4

5

6

1

2

1

2

1

2

1

2

3

4

5

6

1

2

1

2

1

2

1

2

3

4

5

6

(b) SIMPLE BASE-

2

GRAPH with

n

= 6(= 2

2

+2)

4.3 Base-( k +1) Graph

The SIMPLE BASE-( k +1) GRAPH is finite-time convergence for any n and k , while the SIMPLE BASE-( k +1) GRAPH contains graphs that are not necessary for the finite-time convergence and becomes a redundant sequence of graphs in some cases, e.g., ( k, n ) = (1 , 6) (see the example in Fig. 4b and a detailed explanation in Sec. C.2). To remove this redundancy, this section proposes the BASE-( k +1) GRAPH that can make all nodes reach the exact consensus after fewer iterations than the SIMPLE BASE-( k +1) GRAPH.

The pseudo-code for constructing the BASE-( k +1) GRAPH is shown in Alg. 3. The BASE-( k +1) GRAPH consists of the following three steps.

Step 1. We decompose n as p × q such that p is a multiple of 2 , · · · , ( k + 1) and q is prime to 2 , · · · , ( k +1) , and split V into disjoint subsets V 1 , · · · , V p such that | V l | = q for all l ∈ [ p ] .

Step 2. For all l ∈ [ p ] , we make all nodes in V l reach the average in V l by the SIMPLE BASE-( k +1) GRAPH A simple k ( V l ) . Then, we take p nodes from V 1 , · · · , V p respectively and construct a set U 1 . Similarly, we construct U 2 , · · · , U q such that U 1 , · · · , U q are disjoint sets.

Step 3. For all l ∈ [ q ] , we make all nodes in U l reach the average in U l by the k -PEER HYPERHYPERCUBE GRAPH H k ( U l ) . Because the average in U l is equivalent to the average in V after step 2 , all nodes reach the exact consensus.

Using the example in Fig. 4a, we explain the BASE-( k +1) GRAPH in more detail. Let G (1) , · · · , G (4) denote the graphs in Fig. 4a from left to right, respectively. In step 1 , we split V into V 1 := { 1 , 2 , 3 } and V 2 := { 4 , 5 , 6 } . In step 2 , nodes in V 1 and nodes in V 2 have the same parameter respectively by

exchanging parameters on G (1) , · · · , G (3) because the subgraphs composed on V 1 and V 2 are same as the SIMPLE BASE-( k +1) GRAPH (see Fig. 14a). Then, we construct U 1 := { 1 , 4 } , U 2 := { 2 , 5 } , and U 3 := { 3 , 6 } . Finally, in step 3 , all nodes reach the exact consensus by exchanging parameters in G (4) .

Fig. 5 and Sec. F.1 compare the BASE-( k +1) GRAPH with the SIMPLE BASE-( k +1) GRAPH, demonstrating that the length of the BASE-( k +1) GRAPH is less than that of the SIMPLE BASE-( k +1) GRAPH in many cases. Moreover, Theorem 1 show the upper bound of the length of the SIMPLE BASE-( k +1) GRAPH and BASE-( k +1) GRAPH. The proof is provided in Sec. D.

Theorem 1. For any number of nodes n ∈ N and maximum degree k ∈ [ n -1] , the length of the SIMPLE BASE-( k +1) GRAPH and BASE-( k + 1) GRAPH is less than or equal to 2 log k +1 ( n ) + 2 .

0

50

100

150

200

Number of nodes

2

4

6

8

10

12

14

Length

Simple Base-3 Graph (2)

Corollary 1. For any number of nodes n ∈ N and maximum degree k ∈ [ n -1] , the SIMPLE BASE-( k +1) GRAPH and BASE-( k +1) GRAPH are O (log k +1 ( n )) -finite time convergence.

Therefore, the BASE-( k + 1) GRAPH is a powerful extension of the 1 -peer exponential [43] and 1 -peer hypercube graphs [31] because they are O (log 2 ( n )) -finite time convergence only if n is a power of 2 and their maximum degree cannot be set to any number other than 1 .

5 Decentralized SGD on Base-( k +1) Graph

In this section, we verify the effectiveness of the BASE-( k +1) GRAPH for decentralized learning, demonstrating that the BASE-( k +1) GRAPH can endow DSGD with both a faster convergence rate and fewer communication costs than the existing topologies, including the ring, torus, and exponential graph. We consider the following decentralized learning problem:

where n is the number of nodes, f i is the loss function of node i , D i is the data distribution held by node i , F i ( x ; ξ i ) is the loss of node i at data sample ξ i , and ∇ F i ( x ; ξ i ) denotes the stochastic gradient. Then, we assume that the following hold, which are commonly used for analyzing decentralized learning methods [18, 20, 23, 43].

Assumption 1. There exists f ⋆ > -∞ that satisfies f ( x ) ≥ f ⋆ for any x ∈ R d .

Assumption 2. f i is L -smooth for all i ∈ [ n ] .

Assumption 3. There exists σ 2 that satisfies E ξ i ∼D i ∥∇ F i ( x ; ξ i ) -∇ f i ( x ) ∥ 2 ≤ σ 2 for all x ∈ R d .

Assumption 4. There exists ζ 2 that satisfies 1 n ∑ n i =1 ∥∇ f i ( x ) -∇ f ( x ) ∥ 2 ≤ ζ 2 for all x ∈ R d .

We consider the case when DSGD [18], the most widely used decentralized learning method, is used as an optimization method. Let W (1) , · · · , W ( m ) be mixing matrices of the BASE-( k +1) GRAPH. In DSGD on the BASE-( k +1) GRAPH, node i updates its parameter x i as follows:

where η is the learning rate. In this case, thanks to the property of finite-time convergence, DSGD on the BASE-( k +1) GRAPH converges at the following convergence rate.

Theorem 2. Suppose that Assumptions 1-4 hold. Then, for any number of nodes n ∈ N and maximum degree k ∈ [ n -1] , there exists η such that ¯ x := 1 n ∑ n i =1 x i generated by Eq. (1) satisfies 1 R +1 ∑ R r =0 E ∥ ∥ ∇ f (¯ x ( r ) ) ∥ ∥ 2 ≤ ϵ after

iterations, where F 0 := f (¯ x (0) ) -f ⋆ .

The above theorem follows immediately from Theorem 2 stated in Koloskova et al. [11] and Corollary 1. The convergence rates of DSGD over commonly used topologies are summarized in Sec. E. From Theorem 2 and Sec. E, we can conclude that for any number of nodes n , the BASE2 GRAPH enables DSGD to converge faster than the ring and torus and as fast as the exponential graph, although the maximum degree of the BASE2 GRAPH is only one. Moreover, if we set the maximum degree k to the value between 2 to ⌈ log 2 ( n ) ⌉ , the BASE-( k +1) GRAPH enables DSGD to converge faster than the exponential graph, even though the maximum degree of the BASE-( k +1) GRAPH remains less than that of the exponential graph. It is worth noting that if we increase the maximum degree of the 1 -peer exponential and 1 -peer hypercube graphs (i.e., k -peer exponential and k -peer hypercube graphs with k ≥ 2 ), these topologies cannot enable DSGD to converge faster than the exponential graph because these topologies are no longer finite-time convergence even when the number of nodes is a power of 2 .

6 Experiments

In this section, we validate the effectiveness of the BASE-( k +1) GRAPH. First, we experimentally verify that the BASE-( k +1) GRAPH is finite-time convergence for any number of nodes in Sec. 6.1, and we verify the effectiveness of the BASE-( k +1) GRAPH for decentralized learning in Sec. 6.2.

0

10

20

30

40

50

Iterations

10

25

10

21

10

17

10

13

10

9

10

5

10

1

Consensus error

n

=100

0

10

20

30

40

50

Iterations

10

25

10

21

10

17

10

13

10

9

10

5

10

1

Consensus error

n

=500

0

10

20

30

40

50

Iterations

10

25

10

21

10

17

10

13

10

9

10

5

10

1

Consensus error

n

=1000

Exp. (7,9,10)

Ring (2)

Torus (4)

1-peer Exp. (1)

1-peer D-EquiDyn (1)

1-peer U-EquiDyn (1)

Base

-

2

Graph

(1)

Base

-

3

Graph

(2)

Base

-

4

Graph

(3)

Base

-

5

Graph

(4)

0

50

100

150

200

Epoch

50

60

70

80

90

Fashion MNIST

180

190

200

90

90.5

91

0

100

200

300

400

500

Epoch

40

50

60

70

80

90

CIFAR-10

450

475

500

89

89.5

90

0

100

200

300

400

500

Epoch

10

20

30

40

50

60

CIFAR-100

450

475

500

58

59

60

61

Exp. (5)

Ring (2)

Torus (4)

1-peer Exp. (1)

1-peer U-EquiDyn (1)

1-peer D-EquiDyn (1)

Base

-

2

Graph

(1)

Base

-

3

Graph

(2)

Base

-

4

Graph

(3)

Base

-

5

Graph

(4)

(a)

α

= 10

0

50

100

150

200

Epoch

50

60

70

80

90

Fashion MNIST

180

190

200

83

87

91

0

100

200

300

400

500

Epoch

40

50

60

70

80

90

CIFAR-10

450

475

500

84

86

88

90

0

100

200

300

400

500

Epoch

10

20

30

40

50

60

CIFAR-100

450

475

500

54

58

62

Exp. (5)

Ring (2)

Torus (4)

1-peer Exp. (1)

1-peer U-EquiDyn (1)

1-peer D-EquiDyn (1)

Base

-

2

Graph

(1)

Base

-

3

Graph

(2)

Base

-

4

Graph

(3)

Base

-

5

Graph

(4)

(b)

α

= 0

.

1

6.1 Consensus Rate

Setup. Let x i ∈ R be the parameter that node i has, and let ¯ x := 1 n ∑ n i =1 x i . For each i , the initial value of x i was drawn from Gaussian distribution with mean 0 and standard variance 1 . Then, we evaluated how the consensus error 1 n ∑ n i =1 ( x i -¯ x ) 2 decreases when x i is updated as x i ← ∑ n j =1 W ij x j where W is the mixing matrix associated with a given topology.

Results. Figs. 1 and 6 present how the consensus errors decrease on various topologies. The results indicate that the BASE-( k +1) GRAPH reaches the exact consensus after a finite number of iterations, while the other topologies only reach the consensus asymptotically. Moreover, as the maximum degree k increases, the BASE-( k +1) GRAPH reaches the exact consensus with fewer iterations. We also present the results when n is a power of 2 in Sec. F.2, demonstrating that the 1 -peer exponential graph can reach the exact consensus as well as the BASE2 GRAPH, but requires more iterations than the BASE4 GRAPH.

6.2 Decentralized Learning

Next, we examine the effectiveness of the BASE-( k +1) GRAPH in decentralized learning.

Setup. We used three datasets, Fashion MNIST [41], CIFAR-{ 10 , 100 } [14], and used LeNet [15] for Fashion MNIST and VGG-11 [32] for CIFAR-{ 10 , 100 } . Additionally, we present the results using ResNet-18 [5] in Sec. G. The learning rate was tuned by the grid search and we used the cosine learning rate scheduler [22]. We distributed the training dataset to nodes by using Dirichlet distributions with hyperparameter α [7], conducting experiments in both homogeneous and heterogeneous data distribution settings. As α approaches zero, the data distributions held by each node become more heterogeneous. We repeated all experiments with three different seed values and reported their averages. See Sec. H for more detailed settings.

Results of DSGD on Various Topologies. We compared various topologies combined with the DSGD with momentum [4, 18], showing the results in Fig. 7. From Fig. 7, the accuracy differences among topologies are larger as the data distributions are more heterogeneous. From Fig. 7b, the BASE-{ 2 , 3 , 4 , 5 } GRAPH reach high accuracy faster than the other topologies. Furthermore, comparing the final accuracy, the final accuracy of the BASE2 GRAPH is comparable to or higher than that of the existing topologies, including the exponential graph.

Moreover, the final accuracy of the BASE-{ 3 , 4 , 5 } GRAPH is higher than that of all existing topologies. From Fig. 7a, the accuracy differences among topologies become small when α = 10 ; however, the BASE5 GRAPH still outperforms the other topologies. In Fig. 8, we present the results in cases other than n = 25 , demonstrating that the BASE2 GRAPH outperforms the 1 -peer exponential graph and the BASE-{ 3 , 4 , 5 } GRAPH can consistently outperform the exponential and 1 -peer exponential graphs for all n . In Sec. F.3.2, we show the learning curves and the comparison of the consensus rate when n is 21 , 22 , 23 , 24 , and 25 .

21

22

23

24

25

Number of nodes

84

85

86

87

88

89

90

Exp. (5)

1-peer Exp. (1)

Base

-

2

Graph

(1)

Base

-

3

Graph

(2)

Base

-

4

Graph

(3)

Base

-

5

Graph

(4)

Base

-

6

Graph

(5)

Results of D 2 and QG-DSGDm on Various Topologies. The above results demonstrated that the BASE-( k +1) GRAPH outperforms the existing topologies, especially when the data distributions are heterogeneous. Hence, we next compared the BASE-( k +1) GRAPH with the existing topologies in the case where D 2 [36] and QG-DSGDm [20], which are robust to data heterogeneity, are used as decentralized learning methods. From Fig. 9, the BASE2 GRAPH can achieve comparable or higher accuracy than the 1 -peer exponential graph, and the BASE5 GRAPH consistently out-

0

100

200

300

400

500

Epoch

40

50

60

70

80

90

D 2

450

475

500

84

86

88

0

100

200

300

400

500

Epoch

40

50

60

70

80

90

QG-DSGDm

450

475

500

84

86

88

Exp. (5)

1-peer Exp. (1)

Base

-

2

Graph

(1)

Base

-

3

Graph

(2)

Base

-

4

Graph

(3)

Base

-

5

Graph

(4)

performs the exponential graph. Thus, the BASE-( k +1) GRAPH is useful not only for DSGD but also for D 2 and QG-DSGDm and then enables these methods to achieve a reasonable balance between accuracy and communication efficiency.

7 Conclusion

In this study, we propose the BASE-( k +1) GRAPH, a novel topology with both a fast consensus rate and small maximum degree. Unlike the existing topologies, the BASE-( k + 1) GRAPH is finite-time convergence for any number of nodes and maximum degree k . Thanks to this favorable property, the BASE-( k +1) GRAPH enables DSGD to obtain both a faster convergence rate and more communication efficiency than the existing topologies, including the ring, torus, and exponential graph. Through experiments, we compared the BASE-( k +1) GRAPH with various existing topologies, demonstrating that the BASE-( k +1) GRAPH enables various decentralized learning methods to more successfully reconcile accuracy and communication efficiency than the existing topologies.

Acknowledgments

Yuki Takezawa, Ryoma Sato, and Makoto Yamada were supported by JSPS KAKENHI Grant Number 23KJ1336, 21J22490, and MEXT KAKENHI Grant Number 20H04243, respectively.

References

Assran, M., Loizou, N., Ballas, N., and Rabbat, M. (2019). Stochastic gradient push for distributed deep learning. In International Conference on Machine Learning .

Chen, Y., Yuan, K., Zhang, Y., Pan, P., Xu, Y., and Yin, W. (2021). Accelerating gossip sgd with periodic global averaging. In International Conference on Machine Learning .

Cyffers, E., Even, M., Bellet, A., and Massoulié, L. (2022). Muffliato: Peer-to-peer privacy amplification for decentralized optimization and averaging. In Advances in Neural Information Processing Systems .

Gao, H. and Huang, H. (2020). Periodic stochastic gradient descent with momentum for decentralized training. In arXiv .

He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition .

Horváth, S. and Richtarik, P. (2021). A better alternative to error feedback for communicationefficient distributed learning. In International Conference on Learning Representations .

Hsu, T.-M. H., Qi, and Brown, M. (2019). Measuring the effects of non-identical data distribution for federated visual classification. In arXiv .

Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., D'Oliveira, R. G. L., Eichner, H., Rouayheb, S. E., Evans, D., Gardner, J., Garrett, Z., Gascón, A., Ghazi, B., Gibbons, P. B., Gruteser, M., Harchaoui, Z., He, C., He, L., Huo, Z., Hutchinson, B., Hsu, J., Jaggi, M., Javidi, T., Joshi, G., Khodak, M., Konecný, J., Korolova, A., Koushanfar, F., Koyejo, S., Lepoint, T., Liu, Y., Mittal, P., Mohri, M., Nock, R., Özgür, A., Pagh, R., Qi, H., Ramage, D., Raskar, R., Raykova, M., Song, D., Song, W., Stich, S. U., Sun, Z., Suresh, A. T., Tramèr, F., Vepakomma, P., Wang, J., Xiong, L., Xu, Z., Yang, Q., Yu, F. X., Yu, H., and Zhao, S. (2021). Advances and open problems in federated learning. In Foundations and Trends in Machine Learning .

Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. (2020). SCAFFOLD: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning .

Koloskova, A., Lin, T., Stich, S. U., and Jaggi, M. (2020a). Decentralized deep learning with arbitrary communication compression. In International Conference on Learning Representations .

Koloskova, A., Loizou, N., Boreiri, S., Jaggi, M., and Stich, S. (2020b). A unified theory of decentralized SGD with changing topology and local updates. In International Conference on Machine Learning .

Koloskova, A., Stich, S., and Jaggi, M. (2019). Decentralized stochastic optimization and gossip algorithms with compressed communication. In International Conference on Machine Learning .

Kong, L., Lin, T., Koloskova, A., Jaggi, M., and Stich, S. (2021). Consensus control for decentralized deep learning. In International Conference on Machine Learning .

Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Technical report.

LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. In IEEE .

Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. (2020). Federated optimization in heterogeneous networks. In arXiv .

Li, Z., Shi, W., and Yan, M. (2019). A decentralized proximal-gradient method with network independent step-sizes and separated convergence rates. In IEEE Transactions on Signal Processing .

Lian, X., Zhang, C., Zhang, H., Hsieh, C.-J., Zhang, W., and Liu, J. (2017). Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. In Advances in Neural Information Processing Systems .

Lian, X., Zhang, W., Zhang, C., and Liu, J. (2018). Asynchronous decentralized parallel stochastic gradient descent. In International Conference on Machine Learning .

Lin, T., Karimireddy, S. P., Stich, S., and Jaggi, M. (2021). Quasi-global momentum: Accelerating decentralized deep learning on heterogeneous data. In International Conference on Machine Learning .

Lorenzo, P. D. and Scutari, G. (2016). Next: In-network nonconvex optimization. In IEEE Transactions on Signal and Information Processing over Networks .

Loshchilov, I. and Hutter, F. (2017). SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations .

Lu, Y. and De Sa, C. (2020). Moniqua: Modulo quantized communication in decentralized SGD. In International Conference on Machine Learning .

Lu, Y. and De Sa, C. (2021). Optimal complexity in decentralized training. In International Conference on Machine Learning .

Marfoq, O., Xu, C., Neglia, G., and Vidal, R. (2020). Throughput-optimal topology design for cross-silo federated learning. In Advances in Neural Information Processing Systems .

McMahan, B., Moore, E., Ramage, D., Hampson, S., and Arcas, B. A. y. (2017). Communication-efficient learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics .

Murata, T. and Suzuki, T. (2021). Bias-variance reduced local sgd for less heterogeneous federated learning. In International Conference on Machine Learning .

Nedi´ c, A., Olshevsky, A., and Rabbat, M. G. (2018). Network topology and communicationcomputation tradeoffs in decentralized optimization. In IEEE .

Nedi´ c, A., Olshevsky, A., and Shi, W. (2017). Achieving geometric convergence for distributed optimization over time-varying graphs. In SIAM Journal on Optimization .

Pu, S. and Nedi´ c, A. (2021). Distributed stochastic gradient tracking methods. In Mathematical Programming .

Shi, G., Li, B., Johansson, M., and Johansson, K. H. (2016). Finite-time convergent gossiping. In IEEE Transactions on Networking .

Simonyan, K. and Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations .

Song, Z., Li, W., Jin, K., Shi, L., Yan, M., Yin, W., and Yuan, K. (2022). Communicationefficient topologies for decentralized learning with O (1) consensus rate. In Advances in Neural Information Processing Systems .

Takezawa, Y., Bao, H., Niwa, K., Sato, R., and Yamada, M. (2023). Momentum tracking: Momentum acceleration for decentralized deep learning on heterogeneous data. In Transactions on Machine Learning Research .

Tang, H., Gan, S., Zhang, C., Zhang, T., and Liu, J. (2018a). Communication compression for decentralized training. In Advances in Neural Information Processing Systems .

Tang, H., Lian, X., Yan, M., Zhang, C., and Liu, J. (2018b). D 2 : Decentralized training over decentralized data. In International Conference on Machine Learning .

Vogels, T., He, L., Koloskova, A., Karimireddy, S. P., Lin, T., Stich, S. U., and Jaggi, M. (2021). Relaysum for decentralized deep learning on heterogeneous data. In Advances in Neural Information Processing Systems .

Vogels, T., Karimireddy, S. P., and Jaggi, M. (2020). Practical low-rank communication compression in decentralized deep learning. In Advances in Neural Information Processing Systems .

Wang, J., Sahu, A. K., Yang, Z., Joshi, G., and Kar, S. (2019). Matcha: Speeding up decentralized sgd via matching decomposition sampling. In Indian Control Conference .

Wu, Y. and He, K. (2018). Group normalization. In European Conference on Computer Vision .

Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. In arXiv .

Xin, R. and Khan, U. A. (2020). Distributed heavy-ball: A generalization and acceleration of first-order methods with gradient tracking. In IEEE Transactions on Automatic Control .

Ying, B., Yuan, K., Chen, Y., Hu, H., Pan, P., and Yin, W. (2021). Exponential graph is provably efficient for decentralized deep training. In Advances in Neural Information Processing Systems .

Yu, H., Jin, R., and Yang, S. (2019). On the linear speedup analysis of communication efficient momentum SGD for distributed non-convex optimization. In International Conference on Machine Learning .

Yuan, K., Chen, Y., Huang, X., Zhang, Y., Pan, P., Xu, Y., and Yin, W. (2021). DecentLaM: Decentralized momentum sgd for large-batch deep training. In International Conference on Computer Vision .

Yuan, K., Ying, B., Zhao, X., and Sayed, A. H. (2019). Exact diffusion for distributed optimization and learning-part i: Algorithm development. In IEEE Transactions on Signal Processing .

Zhao, H., Li, B., Li, Z., Richtárik, P., and Chi, Y. (2022). BEER: Fast O (1 /T ) rate for decentralized nonconvex optimization with communication compression. In Advances in Neural Information Processing Systems .

Zhu, T., He, F., Zhang, L., Niu, Z., Song, M., and Tao, D. (2022). Topology-aware generalization of decentralized SGD. In International Conference on Machine Learning .

A Detailed Explanation of k -peer Hyperhypercube Graph

In this section, we explain Alg. 1 in more detail. The k -PEER HYPER-HYPERCUBE GRAPH mainly consists of the following five steps.

Step 1. Decompose n as n = n 1 ×··· × n L with minimum L such that n l ∈ [ k +1] for all l ∈ [ L ] .

Step 2. If L = 1 , we make all nodes obtain the average of parameters in V by using the complete graph. If L ≥ 2 , we split V into disjoint subsets V 1 , · · · , V n L such that | V l | = n n L for all l ∈ [ n L ] and continue to step 3 .

Step 3. For all l ∈ [ n L ] , we make all nodes in V l obtain the average of parameters in V l by using the k -PEER HYPER-HYPERCUBE GRAPH H k ( V l ) .

Step 4. We take n L nodes from V 1 , · · · , V n L respectively and construct a set U 1 . Similarly, we construct U 2 , · · · , U n L such that U 1 , · · · , U n L are disjoint sets.

Step 5. For all l ∈ [ n L ] , we make all nodes in U l obtain the average of parameters in U l by using the complete graph. Because the average of parameter U l is equivalent to the average in V after step 4 , all nodes reach the exact consensus.

When n ≤ k +1 , the k -PEER HYPER-HYPERCUBE GRAPH becomes the complete graph because of step 2 . When n > k +1 , we decompose n in step 1 and construct the k -PEER HYPER-HYPERCUBE GRAPH recursively in step 3 . Thus, the k -PEER HYPER-HYPERCUBE GRAPH can make all nodes reach the exact consensus by the sequence of L graphs.

Using the example provided in Fig. 10, we explain the k -PEER HYPER-HYPERCUBE GRAPH in a more detailed manner. When n = 12 , we decompose 12 as 2 × 2 × 3 . In step 2 , we split V := { 1 , · · · , 12 } into V 1 := { 1 , · · · , 4 } , V 2 := { 5 , · · · , 8 } , and V 3 := { 9 , · · · , 12 } . Step 3 corresponds to the first two graphs in Fig. 10b. As shown in Fig. 10a, the subgraphs consisting of V 1 , V 2 , and V 3 in the first two graphs in Fig. 10b are equivalent to the k -PEER HYPER-HYPERCUBE GRAPH with the number of nodes 4 . Thus, all nodes reach the exact consensus by exchanging parameters in Fig. 10b.

1

2

3

4

1

2

3

4

(a)

n

= 4(= 2

×

2)

1

2

3

4

5

6

7

8

9

10

11

12

1

2

3

4

5

6

7

8

9

10

11

12

1

2

3

4

5

6

7

8

9

10

11

12

(b)

n

= 12(= 2

×

2

×

3)

Figure 10: Illustration of the 2 -PEER HYPER-HYPERCUBE GRAPH. In Fig. 10a, all edge weights are 1 2 . In Fig. 10b, edge weights are 1 2 in the first two graphs and 1 3 in the last graph.

B Detailed Explanation of Simple Base-( k +1) Graph with k ≥ 2

In Sec. 4.2, we explain Alg. 2 only in the case where maximum degree k is one. In this section, we explain the details of Alg. 2 in the case with k ≥ 2 .

The SIMPLE BASE-( k +1) GRAPH mainly consists of the following five steps.

Step 1. As in the base-( k +1) number of n , we decompose n as n = a 1 ( k +1) p 1 + · · · + a L ( k +1) p L in line 1, and then split V into disjoint subsets V 1 , · · · , V L such that | V l | = a l ( k +1) p l for all l ∈ [ L ] .

Step 2. For all l ∈ [ L ] , we split V l into disjoint subsets V l, 1 , · · · , V l,a l such that | V l,a | = ( k +1) p l for all a ∈ [ a l ] in line 3 .

Step 3. For all l ∈ [ L ] , we make all nodes in V l obtain the average of parameters in V l using the k -PEER HYPER-HYPERCUBE GRAPH H k ( V l ) in line 11. Then, we initialize l ′ as one.

Step 4. Each node in V l ′ +1 ∪· · ·∪ V L exchange parameters with a l ′ nodes in V l ′ (= V l ′ , 1 ∪· · ·∪ V l ′ ,a l ′ ) such that the average in V l ′ ,a becomes equivalent to the average in V for all a ∈ [ a l ′ ] . We increase l ′ by one and repeat step 4 until l ′ = L . This procedure corresponds to line 15.

Step 5. For all l ∈ [ L ] and a ∈ [ a l ] , we make all nodes in V l,a obtain the average in V l,a using the k -PEER HYPER-HYPERCUBE GRAPH H k ( V l,a ) . Since the average in V l,a is equivalent to the average in V after step 4 , all nodes reach the exact consensus. This procedure corresponds to line 25.

The major difference compared with the case where k = 1 is step 4 . In the case where k = 1 , each node in V l ′ +1 ∪· · ·∪ V L exchange parameters with one node in V l ′ such that the average in V l ′ becomes equivalent to that in V , while in the case where k ≥ 2 , each node in V l ′ +1 ∪ · · · ∪ V L exchange parameters such that the average in V l ′ ,a becomes equivalent to that in V for all a ∈ [ a l ] . Thanks to this step, we can make all nodes reach the exact consensus using k -PEER HYPER-HYPERCUBE GRAPH H k ( V l,a ) instead of H k ( V l ) in step 5 , and we can reduce the length of a graph sequence.

Using the example provided in Fig. 11, we explain Alg. 2 in a more detailed manner. Let G (1) , · · · , G (4) denote the graphs depicted in Fig. 11 from left to right, respectively. First, we split V := { 1 , · · · , 7 } into V 1 := { 1 , · · · , 6 } and V 2 := { 7 } , and then split V 1 into V 1 , 1 := { 1 , 2 , 3 } and V 1 , 2 := { 4 , 5 , 6 } . In step 3 , all nodes in V 1 obtain the same parameter by exchanging parameters in G (1) and G (2) . In step 4 , the average in V 1 , 1 , that in V 1 , 2 , and that in V 2 become the same as the average in all nodes V by exchanging parameters in G (3) . Thus, in step 5 , all nodes reach the exact consensus by exchanging parameters in G (4) .

1

2

1

2

1

2

1

2

3

4

5

6

7

1

2

3

4

5

6

7

3

7

3

7

1

2

3

4

5

6

7

1

2

3

4

5

6

7

C Illustration of Topologies

C.1 Examples

Fig. 12 shows the examples of the SIMPLE BASE-( k +1) GRAPH. Using these examples, we explain how all nodes reach the exact consensus.

We explain the case depicted in Fig. 12a. Let G (1) , G (2) , G (3) denote the graphs depicted in Fig. 12a from left to right, respectively. First, we split V := { 1 , · · · , 5 } into V 1 := { 1 , 2 , 3 } and V 2 := { 4 , 5 } , and then split V 2 into V 2 , 1 := { 4 } and V 2 , 2 := { 5 } . After exchanging parameters in G (1) , nodes in V 1 and nodes in V 2 have the same parameter respectively. Then, after exchanging parameters in G (2) , the average in V 1 , that in V 2 , 1 , and that in V 2 , 2 become same as the average in V . Thus, by exchanging parameters in G (3) , all nodes reach the exact consensus. Note that edge (4 , 5) in G (3) , which is added in line 27 in Alg. 2, is not necessary for all nodes to reach the exact consensus because nodes 4 and 5 already have the same parameter after exchanging parameters in G (2) ; however, it is effective in decentralized learning as we explained in Sec. 4.2.

1

2

1

2

3

4

5

4

7

4

7

1

2

3

4

5

1

2

1

2

3

4

5

(a)

k

= 2

, n

= 5(= 3 + 2)

1

2

1

2

1

2

1

2

3

4

5

6

7

1

2

1

2

1

2

1

2

3

4

5

6

7

4

7

4

7

4

7

1

2

3

4

5

6

7

1

2

1

2

2

3

1

2

3

4

5

6

7

1

2

1

2

1

2

1

2

3

4

5

6

7

(b)

k

= 1

, n

= 7(= 2

2

+2+1)

C.2 Illustrative Comparison between Simple Base-( k +1) and Base-( k +1) Graphs

In this section, we provide an example of the SIMPLE BASE-( k +1) GRAPH, explaining the reason why the length of the BASE-( k +1) GRAPH is less than that of the SIMPLE BASE-( k +1) GRAPH.

Let G (1) , · · · , G (5) denote the graphs depicted in Fig. 13 from left to right, respectively. ( G (1) , G (2) , G (3) , G (4) , G (5) ) is finite-time convergence, but ( G (1) , G (2) , G (3) , G (5) ) is also finitetime convergence because after exchanging parameters in G (3) , nodes 3 and 4 already have the same parameters. Then, using the technique proposed in Sec. 4.3, we can remove such unnecessary graphs contained in the SIMPLE BASE-( k + 1) GRAPH (see Fig. 4a). Consequently, the BASE-( k + 1) GRAPH can make all nodes reach the exact consensus faster than the SIMPLE BASE-( k +1) GRAPH.

1

2

1

2

1

2

1

2

3

4

5

6

1

2

1

2

1

2

1

2

3

4

5

6

1

2

2

3

2

3

1

2

3

4

5

6

1

2

1

2

1

2

1

2

3

4

5

6

1

2

1

2

1

2

1

2

3

4

5

6

C.3 Additional Examples

C.3.1 Simple Base-( k +1) Graph

1

2

3

1

2

3

1

2

3

(a)

n

= 3

1

2

3

4

1

2

3

4

(b)

n

= 4

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

(c)

n

= 5

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

(d)

n

= 6

1

2

3

4

5

6

7

1

2

3

4

5

6

7

1

2

3

4

5

6

7

1

2

3

4

5

6

7

1

2

3

4

5

6

7

(e)

n

= 7

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

(f)

n

= 8

1

2

3

4

5

6

7

8

9

1

2

3

4

5

6

7

8

9

1

2

3

4

5

6

7

8

9

1

2

3

4

5

6

7

8

9

1

2

3

4

5

6

7

8

9

1

2

3

4

5

6

7

8

9

1

2

3

4

5

6

7

8

9

(g)

n

= 9

1

2

3

4

5

6

7

8

9

10

1

2

3

4

5

6

7

8

9

10

1

2

3

4

5

6

7

8

9

10

1

2

3

4

5

6

7

8

9

10

1

2

3

4

5

6

7

8

9

10

1

2

3

4

5

6

7

8

9

10

1

2

3

4

5

6

7

8

9

10

(h)

n

= 10

1

2

3

(a)

n

= 3

1

2

3

4

1

2

3

4

(b)

n

= 4

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

(c)

n

= 5

1

2

3

4

5

6

1

2

3

4

5

6

(d)

n

= 6

1

2

3

4

5

6

7

1

2

3

4

5

6

7

1

2

3

4

5

6

7

1

2

3

4

5

6

7

(e)

n

= 7

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

(f)

n

= 8

1

2

3

4

5

6

7

8

9

1

2

3

4

5

6

7

8

9

(g)

n

= 9

1

2

3

4

5

6

7

8

9

10

1

2

3

4

5

6

7

8

9

10

1

2

3

4

5

6

7

8

9

10

1

2

3

4

5

6

7

8

9

10

1

2

3

4

5

6

7

8

9

10

(h)

n

= 10

C.3.2 Base-( k +1) Graph

1

2

3

1

2

3

1

2

3

(a)

n

= 3

1

2

3

4

1

2

3

4

(b)

n

= 4

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

(c)

n

= 5

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

(d)

n

= 6

1

2

3

4

5

6

7

1

2

3

4

5

6

7

1

2

3

4

5

6

7

1

2

3

4

5

6

7

1

2

3

4

5

6

7

(e)

n

= 7

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

(f)

n

= 8

1

2

3

4

5

6

7

8

9

1

2

3

4

5

6

7

8

9

1

2

3

4

5

6

7

8

9

1

2

3

4

5

6

7

8

9

1

2

3

4

5

6

7

8

9

1

2

3

4

5

6

7

8

9

1

2

3

4

5

6

7

8

9

(g)

n

= 9

1

2

3

4

5

6

7

8

9

10

1

2

3

4

5

6

7

8

9

10

1

2

3

4

5

6

7

8

9

10

1

2

3

4

5

6

7

8

9

10

1

2

3

4

5

6

7

8

9

10

1

2

3

4

5

6

7

8

9

10

(h)

n

= 10

1

2

3

(a)

n

= 3

1

2

3

4

1

2

3

4

(b)

n

= 4

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

(c)

n

= 5

1

2

3

4

5

6

1

2

3

4

5

6

(d)

n

= 6

1

2

3

4

5

6

7

1

2

3

4

5

6

7

1

2

3

4

5

6

7

1

2

3

4

5

6

7

(e)

n

= 7

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

(f)

n

= 8

1

2

3

4

5

6

7

8

9

1

2

3

4

5

6

7

8

9

(g)

n

= 9

1

2

3

4

5

6

7

8

9

10

1

2

3

4

5

6

7

8

9

10

1

2

3

4

5

6

7

8

9

10

1

2

3

4

5

6

7

8

9

10

(h)

n

= 10

C.4 1 -peer Hypercube Graph and 1 -peer Exponential Graph

For completeness, we provide examples of the 1 -peer hypercube [31] and 1 -peer exponential graphs [43] in Figs. 19 and 18, respectively.

1

2

3

4

1

2

3

4

(a)

n

= 4

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

(b)

n

= 8

1

2

3

4

1

2

3

4

(a)

n

= 4

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

(b)

n

= 5

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

(c)

n

= 6

1

2

3

4

5

6

7

1

2

3

4

5

6

7

1

2

3

4

5

6

7

(d)

n

= 7

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

(e)

n

= 8

D Proof of Theorem 1

Lemma 1 (Length of k -PEER HYPER-HYPERCUBE GRAPH) . Suppose that all prime factors of the number of nodes n are less than or equal to k + 1 . Then, for any number of nodes n ∈ N and maximum degree k ∈ [ n -1] , the length of the k -PEER HYPER-HYPERCUBE GRAPH is less than or equal to max { 1 , 2 log k +2 ( n ) } .

̸

Proof. We assume that n is decomposed as n = n 1 ×··· × n L with minimum L where n l ∈ [ k +1] for all l ∈ [ L ] . Without loss of generality, we suppose n 1 ≤ n 2 ≤ · · · ≤ n L . Then, for any i = j , it holds that n i × n j ≥ k + 2 because if n i × n j ≤ k + 1 for some i and j , this contradicts the assumption that L is minimum.

When L is even, we have

Then, we get L ≤ 2 log k +2 ( n ) .

Next, we discuss the case when L is odd. When L ≥ 3 , n L ≥ √ k +2 holds because n L -2 × n L -1 ≥ k +2 . Thus, we get

Then, we get L ≤ 2 log k +2 ( n ) when L ≥ 3 .

Thus, given the case when L = 1 , the length of the k -PEER HYPER-HYPERCUBE GRAPH is less than or equal to max { 1 , 2 log k +2 ( n ) } .

Lemma 2 (Length of SIMPLE BASE-( k + 1) GRAPH) . For any number of nodes n ∈ N and maximum degree k ∈ [ n -1] , the length of the SIMPLE BASE-( k +1) GRAPH is less than or equal to 2 log k +1 ( n ) + 2 .

Proof. When all prime factors of n are less than or equal to k +1 , the SIMPLE BASE-( k +1) GRAPH is equivalent to the k -PEER HYPER-HYPERCUBE GRAPH and the statement holds from Lemma 1. In the following, we consider the case when there exists a prime factor of n that is larger than k +1 . Note that because when L = 1 (i.e., n = a 1 × ( k +1) p 1 ), all prime factors of n are less than or equal to k +1 , we only need to consider the case when L ≥ 2 . We have the following inequality:

Then, because | V 1 | = a 1 × ( k + 1) p 1 , it holds that m 1 = |H k ( V 1 ) | ≤ 1 + p 1 ≤ log k +1 ( n ) + 1 . Similarly, it holds that |H k ( V 1 , 1 ) | = p 1 ≤ log k +1 ( n ) because | V 1 , 1 | = ( k + 1) p 1 . In Alg. 2, the update rule b 1 ← b 1 +1 in line 22 is executed for the first time when m = m 1 +2 because L ≥ 2 . Thus, the length of the SIMPLE BASE-( k +1) GRAPH is at most m 1 + |H k ( V 1 , 1 ) | +1 ≤ 2 log k +1 ( n ) + 2 . This concludes the statement.

Lemma 3 (Length of BASE-( k +1) GRAPH) . For any number of nodes n ∈ N and maximum degree k ∈ [ n -1] , the length of the BASE-( k +1) GRAPH is less than or equal to 2 log k +1 ( n ) + 2 .

Proof. The statement follows immediately from Lemma 2 and line 12 in Alg. 3.

E Convergence Rate of DSGD over Various Topologies

Table 2 lists the convergence rates of DSGD over various topologies. These convergence rates can be immediately obtained from Theorem 2 stated in Koloskova et al. [11] and consensus rate of the topology. As seen from Table 2, the BASE2 GRAPH enables DSGD to converge faster than the ring and torus and as fast as the exponential graph for any number of nodes, although the maximum degree of the BASE2 GRAPH is only one. Moreover, for any number of nodes, the BASE-( k +1) GRAPH with 2 ≤ k < ⌈ log 2 ( n ) ⌉ enables DSGD to converge faster than the exponential graph, even though the maximum degree of the BASE-( k +1) GRAPH remains to be less than that of the exponential graph.

F Additional Experiments

F.1 Comparison of Base-( k +1) and Simple Base-( k +1) Graphs

Fig. 20 shows the length of the SIMPLE BASE-( k +1) GRAPH and BASE-( k +1) GRAPH. The results indicate that for all k , the length of the BASE-( k +1) GRAPH is less than the length of the SIMPLE BASE-( k +1) GRAPH in many cases.

0

25

50

75

100

125

150

175

200

Number of nodes

2

4

6

8

10

12

14

16

Length

0

25

50

75

100

125

150

175

200

Number of nodes

2

4

6

8

10

12

14

16

Length

Simple Base-3 Graph (2)

0

25

50

75

100

125

150

175

200

Number of nodes

2

4

6

8

10

12

14

16

Length

Simple Base-4 Graph (3)

0

25

50

75

100

125

150

175

200

Number of nodes

2

4

6

8

10

12

14

16

Length

0

25

50

75

100

125

150

175

200

Number of nodes

2

4

6

8

10

12

14

16

Length

0

25

50

75

100

125

150

175

200

Number of nodes

2

4

6

8

10

12

14

16

Length

0

25

50

75

100

125

150

175

200

Number of nodes

2

4

6

8

10

12

14

16

Length

Simple Base-8 Graph (7)

Base-8 Graph (7)

0

25

50

75

100

125

150

175

200

Number of nodes

2

4

6

8

10

12

14

16

Length

0

25

50

75

100

125

150

175

200

Number of nodes

2

4

6

8

10

12

14

16

Length

Simple Base-10 Graph (9)

Base-10 Graph (9)

F.2 Consensus Rate

In Fig. 21, we demonstrate how consensus error decreases on various topologies when the number of nodes n is a power of 2 . The results indicate that the BASE2 GRAPH and 1 -peer exponential graph can reach the exact consensus after the same finite number of iterations and reach the consensus faster than other topologies. Note that the BASE2 GRAPH is equivalent to the 1 -peer hypercube graph when n is a power of 2 .

0

10

20

30

40

50

Iterations

10

25

10

21

10

17

10

13

10

9

10

5

10

1

Consensus error

n

=128

0

10

20

30

40

50

Iterations

10

25

10

21

10

17

10

13

10

9

10

5

10

1

Consensus error

n

=512

0

10

20

30

40

50

Iterations

10

25

10

21

10

17

10

13

10

9

10

5

10

1

Consensus error

n

=1024

Exp. (7,9,10)

Ring (2)

Torus (4)

1-peer Exp. (1)

1-peer D-EquiDyn (1)

1-peer U-EquiDyn (1)

F.3 Decentralized Learning

F.3.1 Comparison of Base-( k +1) Graph and EquiStatic

In this section, we compared the BASE-( k +1) GRAPH with the {U, D}-EquiStatic [33]. The {U, D}-EquiStatic are dense variants of the 1 -peer {U, D}-EquiDyn, and their maximum degree can be set as hyperparameters. We evaluated the {U, D}-EquiStatic varying their maximum degrees; the results are presented in Fig. 22. In both cases with α = 10 and α = 0 . 1 , the BASE2 GRAPH can achieve comparable or higher final accuracy than all {U, D}-EquiStatic, and the BASE-{ 3 , 4 , 5 } GRAPH outperforms all {U, D}-EquiStatic. Thus, the BASE-( k +1) GRAPH is superior to the {U, D}-EquiStatic from the perspective of achieving a balance between accuracy and communication efficiency.

0

100

200

300

400

500

Epoch

40

50

60

70

80

90

=10.0

450

475

500

88

89

90

0

100

200

300

400

500

Epoch

40

50

60

70

80

90

=0.1

450

475

500

80

85

90

D-EquiStatic (2)

D-EquiStatic (3)

D-EquiStatic (4)

U-EquiStatic (2)

U-EquiStatic (4)

Base

-

2

Graph

(1)

Base

-

3

Graph

(2)

Base

-

4

Graph

(3)

Base

-

5

Graph

(4)

F.3.2 Comparison with Various Number of Nodes

In this section, we evaluated the effectiveness of the BASE-( k +1) GRAPH when varying the number of nodes n . Fig. 24 presents the learning curves, and Fig 23 shows how consensus error decreases when n is 21 , 22 , 23 , 24 , and 25 . From Fig. 24, the BASE2 GRAPH consistently outperforms the 1 -peer exponential graph and can achieve a final accuracy comparable to that of the exponential graph. Furthermore, the BASE-{ 3 , 4 , 5 } GRAPH can consistently outperform the exponential graph, even though the maximum degree of the BASE-{ 3 , 4 , 5 } GRAPH is less than that of the exponential graph.

In Fig. 25 presents the learning curve for n = 16 . When the number of nodes is a power of two, the 1 -peer exponential graph is also finite-time convergence, and the 1 -peer exponential graph and BASE2 GRAPH achieve competitive accuracy.

0

10

20

30

40

50

Iterations

10

25

10

21

10

17

10

13

10

9

10

5

10

1

Consensus error

n

=21

0

10

20

30

40

50

Iterations

10

25

10

21

10

17

10

13

10

9

10

5

10

1

Consensus error

n

=22

0

10

20

30

40

50

Iterations

10

25

10

21

10

17

10

13

10

9

10

5

10

1

Consensus error

n

=23

0

10

20

30

40

50

Iterations

10

25

10

21

10

17

10

13

10

9

10

5

10

1

Consensus error

n

=24

0

10

20

30

40

50

Iterations

10

25

10

21

10

17

10

13

10

9

10

5

10

1

Consensus error

n

=25

Exp. (5)

1-peer Exp. (1)

Base

-

2

Graph

(1)

Base

-

3

Graph

(2)

Base

-

4

Graph

(3)

Base

-

5

Graph

(4)

0

100

200

300

400

500

Epoch

40

50

60

70

80

90

n=21

450

475

500

84

86

88

90

0

100

200

300

400

500

Epoch

40

50

60

70

80

90

450

475

500

84

86

88

90

Exp. (5)

1-peer Exp. (1)

Base

-

2

Graph

(1)

Base

-

3

Graph

(2)

Base

-

4

Graph

(3)

Base

-

5

Graph

(4)

Base

-

6

Graph

(5)

0

100

200

300

400

500

Epoch

40

50

60

70

80

90

n=23

450

475

500

84

86

88

90

0

100

200

300

400

500

Epoch

40

50

60

70

80

90

n=24

450

475

500

84

86

88

90

0

100

200

300

400

500

Epoch

40

50

60

70

80

90

n=25

450

475

500

84

86

88

90

0

100

200

300

400

500

Epoch

40

50

60

70

80

90

n=16

450

475

500

86

88

90

92

Exp. (5)

1-peer Exp. (1)

Base

-

2

Graph

(1)

Base

-

4

Graph

(3)

G Results with Other Neural Network Architecture

In this section, we evaluate the effectiveness of the BASE-( k +1) GRAPH with other neural network architecture. Fig. 26 shows the accuracy when we use ResNet-18 [5]. The results are consistent with the results with VGG-11 shown in Sec. 6.

0

100

200

300

400

500

Epoch

40

50

60

70

80

90

=10.0

450

475

500

91

92

93

94

0

100

200

300

400

500

Epoch

40

50

60

70

80

90

=0.1

450

475

500

86

90

94

Exp. (5)

1-peer Exp. (1)

Base

-

2

Graph

(1)

Base

-

3

Graph

(2)

Base

-

4

Graph

(3)

Base

-

5

Graph

(4)

H Hyperparameter Setting

Tables 3 and 4 list the detailed hyperparameter settings used in Secs. 6 and F.3. We ran all experiments on a server with eight Nvidia RTX 3090 GPUs.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies communication-efficient topologies for decentralized learning. The proposed topology has four merits: (1) finite-time exact consensus; (2) fast consensus rate (having exact consensus using $O(\\log_{k+1} n)$ graphs); (3) arbitrary maximum degree ($1 \\leq k \\leq n-1$); (4) arbitrary number of nodes. The most related works to this paper are 1-peer exponential graphs, 1-peer hypercube graphs.  Given the existing algorithms, to the best of my knowledge, this paper is the first to get exact consensus using graphs of length $O(\\log_{k+1} n)$ instead of $O(\\log_2 n)$ with arbitrary number of nodes, where $k$ is the maximum degree. The main algorithm relies on k-peer hyper-hypercube graph. and simple base-(k+1) graph. The k-peer hyper-hypercube graph is a generalization of the 1-peer hypercube graph.  The simple base-(k+1) graph is constructed by dividing the graph into subgraphs with node numbers being a power of k, and then applying the 1-peer hypercube graphs, and using the communication between subgraphs.  The base-(k+1) graphs are constructed by decomposing $n = p\\times q$, where $p$ is prime to $q$ and the factors of $q$ are larger than k+1, and then, apply the sub-algorithms respectively.\n\nThe paper is generally well-written. The authors provide sufficient illustrations to show how the algorithms work and some numerical experiments to demonstrate the efficiency of their topologies.\n\nSTRENGTHS:\n* the length of graphs needed to have exact consensus is $O(\\log_{k+1} n)$, which improves upon the length $O(\\log_2 n)$ of the 1-peer exponential graphs. \n\n* the authors use lots of illustrations to show the implementation of the algorithms, which are very clear.\n\nWEAKNESSES:\n* The proof for the exact consensus of Algorithm 1 (k-peer hyper hypercube graph) is lacking.\n\n* Considering that the algorithm often divides the graph into many subgraphs, while the nodes do not have equal status, the topologies proposed in this paper are relatively complicated. The intricate graph structures could possibly limit its applications in practice, like delayed communication, compressed communication, and information loss. \n\n* The descriptions of the algorithm are only from a panoramic view. It is not clear whether there is a simple way for each node to know its neighbor in each graph efficiently.\n\nQUESTIONS:\n* Is it possible to give theoretical guarantees for the claim that base-(k+1) graphs have shorter lengths than simple base-(k+1) graphs?\n\n* Are there efficient methods for each node to know its neighbors in each graph instead of generating the entire graph by each node?\n\n* Is it possible to extend the algorithm to more restricted topologies where each node can only communicate with a subset of nodes directly?\n\nLIMITATIONS:\nI would suggest the authors provide formal proof, or at least a sketch of proof, for the exact consensus of algorithm 1 (k-peer hyper hypercube graph).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces time-varying communication topologies to tackle the problem of communication efficiency in decentralized learning. These topologies, inspired by the idea of the 1-peer exponential graph, achieve finite-time exact consensus, meaning that the exact average of values is obtained after a finite number of steps. \n\nThe graph is obtained through an iterative process: starting from the k-peer hyper-hypercube graph (an extension of the 1-peer hypercube graph), the simple-(k+1) graph is introduced to handle the case when $n$ has large prime factors, using k-peer hyper-hypercube graph as a subroutine. Finally, the base-(k+1) algorithm is developed based on the two previous graphs to remove redundancy and obtain exact consensus in fewer iterations. \n\nThen, a convergence result is given based on the bound from Koloskova et al. (2020b), and experimental results suggest that the base-(k+1) graph improves over the (1-peer) exponential graph indeed. Unsurprisingly, more connected topologies lead to better accuracies in heterogeneous settings.\n\nSTRENGTHS:\n- Finite-time consensus for any number of nodes. \n- The bandwidth / speed trade-off can be controlled through parameter k (max degree at each iteration). \n- Great efforts are made to make the constructions understandable (though they still remain quite hard to grasp), through figures and step-by-step introduction.\n\nWEAKNESSES:\n- I find the state of the art is not discussed in enough details. While decentralized methods are discussed in details, the finite-time consensus aspect calls to comparing with efficient allreduce methods as well, that are used to scale modern deep learning algorithms. I am not an expert in this domain, but a quick search led me to, e.g., refs [A,B,C] (but there are probably many relevant others). \n\nThe advantage of finite-time consensus is that updates can be done after each iteration, which is not necessarily the case with allreduce methods. At the same time, there are no guarantee on how close to the mean the iterates are before finite-time convergence, so several SGD steps without communication could also be performed between allreduce SGD for instance (local SGD style), which would not be too different. This could / should also be compared against. \n\nMinor comment: One can always have a number of nodes equal to a power of 2 by introducing some nodes with value 0 (and upweighing the other nodes), and making it so that each actual node owns at most 1 virtual node, thus at most doubling the effective degree. It's an easy way to use 1-peer exponential graphs for all n. \n\n[A] Rabenseifner, Rolf. \"Optimization of collective reduction operations.\" Computational Science-ICCS 2004: 4th International Conference, Kraków, Poland, June 6-9, 2004, Proceedings, Part I 4. Springer Berlin Heidelberg, 2004.\n\n[B] Mikami, Hiroaki, et al. \"Massively distributed SGD: ImageNet/ResNet-50 training in a flash.\" arXiv preprint arXiv:1811.05233 (2018).\n\n[C] Ueno, Yuichiro, and Rio Yokota. \"Exhaustive study of hierarchical allreduce patterns for large messages between gpus.\" 2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID). IEEE, 2019.\n\nQUESTIONS:\n- Can you clarify how your method compares with optimized allreduce methods? For instance if you use it until exact consensus each time. What are the bandwidth/latency trade-offs achieved? How do they compare to existing results?   \n- How robust is the Base-(k+1) graph approach to failures/packet drops? Finite-time exact consensus would be lost in this case, but can we recover good approximation properties or is everything lost in this case?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the BASE-(k+1) GRAPH, a new way of designing topology for decentralized learning that is shown to enjoy a fast consensus rate while maintaining small maximum degree. The authors validate the effectiveness of the BASE-(k+1) GRAPH through experiments, demonstrating its finite-time convergence for certain number of nodes and its effectiveness for decentralized learning. Overall, the BASE-(k + 1) GRAPH offers improved communication efficiency and convergence rate for decentralized learning methods.\n\nSTRENGTHS:\n- The paper proposes a new way of designing topology for decentralized learning, the BASE-(k + 1) GRAPH, which allows for a fast consensus rate while maintaining small maximum degree to enable more communication-efficient decentralized learning. \n\n- The authors provide a theoretical analysis of the proposed method which ensures the property of small maximum degree and several experiments are conducted to validate their claims.\n\n- The paper is well written and organized, with a logical flow of ideas. The authors provide a clear explanation of the proposed approach, and also includes helpful figures and tables to illustrate the concepts and results.\n\nWEAKNESSES:\n- The numerical experiments are still not sufficient; it would be more convincing if they compared the BASE-(k + 1) GRAPH with a more comprehensive set of topologies to demonstrate its superiority. As far as the reviewer know, there are several other topologies that can also achieve fast consensus rate and small maximum degree, such as Ring All-Reduce where maximum degree is 1 [a] , Tree All-Reduce where maximum degree is 2 [b] which and optimization-based topology [c].\n\n- The paper does not explain how to decompose n, whether different decomposition methods will produce different results and how to decompose n to get the best results.\n\n- The paper assumes a homogeneous network, which may not be practical in some real-world scenarios, where communication bandwidth and underlying topology constraints may affect the communication between nodes.\n\n[a] Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in tensorflow. arXiv preprint arXiv:1802.05799, 2018.\n\n[b] https://developer.nvidia.com/blog/massively-scale-deep-learning-training-nccl-2-4/\n\n[c] Sun, Chuangchuang, Ran Dai, and Mehran Mesbahi. \"Weighted network design with cardinality constraints via alternating direction method of multipliers.\" IEEE Transactions on Control of Network Systems 5.4 (2018): 2073-2084.\n\nQUESTIONS:\n- The reviewer would also like to know how the method can be extended to heterogeneous networks where the topology may not exhibit regular structures. \n\n- There are several other topologies that can also achieve fast consensus rate and small maximum degree; how the proposed method compares with them?\n\n- For a given number of nodes n, there exists several ways of decomposition, and different way of decomposition will lead to different maximum degree and consensus rate. The reviewer would like to know how one can find the optimal decomposition method to properly balance the maximum degree and consensus rate.\n\nLIMITATIONS:\n- The paper could benefit from a more detailed discussion of the limitations of the proposed approach, such as its sensitivity to certain types of data heterogeneity or its applicability to heterogeneous networks.\n\n- A more detailed description of the algorithm and its implementation (such as the decomposition method of node number n) would be helpful for readers who want to replicate the experiments or apply the method to their own problems.\n\n- The BASE-(k + 1) GRAPH should be compared with a wider range of existing topologies.\n\n- The derivation of Theorem 2 should be shown to help readers better understand the influence of BASE-(k + 1) GRAPH on the convergence of decentralized SGD.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of decentralized optimization and proposes a novel time-varying communication topology that has better properties than prior topologies, especially when the number of nodes n is not a power of 2 i.e. (i) it is sparse, and (ii) has good spectral properties, translating to the good optimization behavior. Paper provides a description of the proposed topology, its properties, and evaluates its effectiveness in practice.\n\nSTRENGTHS:\n- The paper is very well written and clear\n- Proposed topology clearly improves over the previously known topologies, especially when the number of nodes n is not a power of 2.\n\nWEAKNESSES:\n1. Paper gives consensus rate (in e.g. Table 1) in O-notation, i.e. hiding the constants. However, actual constants do matter a lot. If we allow the use of O-notation, then expander graphs would be the best theoretical graphs, as they have constant consensus rate, as well as constant maximum degree. However, expanders do not work well in practice for relatively small graphs as the constants are quite large. Therefore, I believe it is important to get good constants while designing a new communication topology. \n\n2. Description of the algorithm for Base-(k + 1) graph is not fully clear. In step 1. of the algorithm (line 181), it was not clear why n can always be decomposed in this way? What if n is a prime number ?\n\nQUESTIONS:\n1. If the number of nodes n is a power of two, is it still beneficial to use your proposed topologies ? Including the larger range of n in figure 8 that covers power of 2 might be good. \n2. Why did you not include the gradient tracking in comparison in figure 9? \n3. When reading Figure 3, I got confused why the number of averaging steps is 5, while log_2(5) < 3.\n4. In the Experiments section (Fig. 6, 7, 8, 9), having a Base-6 graph might still potentially be better than the Exponential graph, as its maximum degree would be the same, but it would lead to the faster convergence. Why did you not include the Base-6 graph in the comparison? \n\nMinor suggestions: \n\n5. It might be good to provide the open-source code of generating proposed topologies.\n6. Might be helpful for the reader to indicate how large is log_2(n) for each of the experiments.\n7. When giving an example in Figures 3 & 4, I felt it might be better to have the same number of nodes n in both cases, so that the reader can see how is the Base-(k + 1) graph can reduce the number of steps from the Simple Base-(k + 1) graph\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}