Title: Epidemic Learning: Boosting Decentralized Learning with Randomized Communication
Venue: NeurIPS 2023
Accepted: yes
Keywords: Epidemic

Decentralized Learning

Randomized Communication

Peer sampling
Abstract: We present Epidemic Learning (EL), a simple yet powerful decentralized learning (DL) algorithm that leverages changing communication topologies to achieve faster model convergence compared to conventional DL approaches. At each round of EL, each node sends its model updates to a random sample of $s$ other nodes (in a system of $n$ nodes). We provide an extensive theoretical analysis of EL, demonstrating that its changing topology culminates in superior convergence properties compared to the state-of-the-art (static and dynamic) topologies. Considering smooth non-convex loss functions, the number of transient iterations for EL, i.e., the rounds required to achieve asymptotic linear speedup, is in $O(n^3/s^2)$ which outperforms the best-known bound $O(n^3)$ by a factor of $s^2$, indicating the benefit of randomized communication for DL. We empirically evaluate EL in a 96-node network and compare its performance with state-of-the-art DL approaches. Our results illustrate that EL converges up to  $ 1.7\times$ quicker than baseline DL algorithms and attains $2.2 $\% higher accuracy for the same communication volume.

Epidemic Learning: Boosting Decentralized Learning with Randomized Communication

Abstract

We present Epidemic Learning (EL), a simple yet powerful decentralized learning (DL) algorithm that leverages changing communication topologies to achieve faster model convergence compared to conventional DL approaches. At each round of EL, each node sends its model updates to a random sample of s other nodes (in a system of n nodes). We provide an extensive theoretical analysis of EL, demonstrating that its changing topology culminates in superior convergence properties compared to the state-of-the-art (static and dynamic) topologies. Considering smooth nonconvex loss functions, the number of transient iterations for EL, i . e ., the rounds required to achieve asymptotic linear speedup, is in O ( n 3 / s 2 ) which outperforms the best-known bound O ( n 3 ) by a factor of s 2 , indicating the benefit of randomized communication for DL. We empirically evaluate EL in a 96-node network and compare its performance with state-of-the-art DL approaches. Our results illustrate that EL converges up to 1.7 × quicker than baseline DL algorithms and attains 2.2% higher accuracy for the same communication volume.

1 Introduction

In Decentralized Learning (DL), multiple machines (or nodes) collaboratively train a machine learning model without any central server [32, 37, 42]. Periodically, each node updates the model using its local data, sends its model updates to other nodes, and averages the received model updates, all without sharing raw data. Compared to centralized approaches [28], DL circumvents the need for centralized control, ensures scalability [21, 32], and avoids imposing substantial communication costs on a central server [58]. However, DL comes with its own challenges. The exchange of model updates with all nodes can become prohibitively expensive in terms of communication costs as the network size grows [25]. For this reason, nodes in DL algorithms usually exchange model updates with only a small number of other nodes in a particular round, i . e ., they perform partial averaging instead of an All-Reduce (network-wide) averaging of local model updates [57].

A key element of DL algorithms is the communication topology, governing how model updates are exchanged between nodes. The properties of the communication topology are critical for the performance of DL approaches as it directly influences the speed of convergence [27, 54, 56, 52]. The seminal decentralized parallel stochastic gradient descent (D-PSGD) algorithm and many of its proposed variants rely on a static topology, i . e ., each node exchanges its model with a set of neighboring nodes that remain fixed throughout the training process [32]. More recent approaches study changing topologies, i . e ., topologies that change during training, with notable examples being time-varying graphs [24, 36, 51], one-peer exponential graphs [57], EquiTopo [53], and Gossip Learning [16, 17, 20]. We discuss these works in more detail in Section 5.

This paper investigates the benefits of randomized communication for DL. Randomized communication, extensively studied in distributed computing, has been proven to enhance the performance of fundamental algorithms, including consensus and data dissemination protocols [22, 5, 8]. In the case of DL, randomized communication can reduce the convergence time and therefore communication overhead [11]. In this work, we specifically consider the setting where each node communicates with a random subset of other nodes that changes at each round, as with epidemic interaction schemes [12, 38]. We also focus on the scenario where data is unevenly distributed amongst nodes, i . e ., non independent and identically distributed (non-IID) settings, a common occurrence in DL [18].

To illustrate the potential of randomized communication, we empirically compare model convergence in Figure 1 for static s -regular topologies (referred to as Static-Topo ) and randomized topologies (referred to as Rand.-Topo (EL), our work) in 96-node networks on the CIFAR-10 learning task. As particular static topologies can lead to sub-optimal convergence, we also experiment with the setting in which nodes are stuck in an unlucky static s -regular topology (referred to as Static-Topo-Unlucky ). This unlucky topology consists of two network partitions, connected by just two edges. Figure 1 reveals that dynamic topologies converge quicker than static topologies. After 3000 communication rounds, Rand.-Topo (EL, our work) achieves 65.3% top-1 test accuracy, compared to 63.4% and 61.8% for Static-Random and Static-Unlucky, respectively. Additional experiments can be found in Section 4, and their setup is elaborated in Appendix C.

1

2

3

20

40

60

Communication round (

10 3

)

Test accuracy [

%

]

Rand.-Topo (EL)

Static-Topo

Static-Topo-Unlucky

2

2.5

3

60

62

64

66

×

Our contributions Our paper makes the following four contributions:

We formulate, design, analyze, and experimentally evaluate Epidemic Learning (EL) , a novel DL algorithm in which nodes collaboratively train a machine learning model using a dynamically changing, randomized communication topology. More specifically, in EL, at each round, each node sends its model update to a random sample of s other nodes (out of n total nodes). This process results in a randomized topology that changes every round.

We first analyze an EL variant, named EL-Oracle , where the union of the random samples by all nodes forms an s -regular random graph in each round. EL-Oracle ensures a perfectly balanced communication load among the participating nodes as each node sends and receives exactly s model updates every round. Nevertheless, achieving an s -regular graph necessitates coordination among the nodes, which is not ideal in a decentralized setting. To address this challenge, we also analyze another EL variant, named EL-Local . In EL-Local, each node independently and locally draws a uniformly random sample of s other nodes at each round and sends its model update to these nodes. We demonstrate that EL-Local enjoys a comparable convergence guarantee as EL-Oracle without requiring any coordination among the nodes and in a fully decentralized manner.

Our theoretical analysis in Section 3 shows that EL surpasses the best-known static and randomized topologies in terms of convergence speed. More precisely, we prove that EL converges with the rate O ( 1 / √ nT + 1 / 3 √ sT 2 + 1 / T ) , where T is the number of learning rounds. Similar to most stateof-the-art algorithms for decentralized optimization [30, 24] and centralized stochastic gradient descent (SGD) ( e . g ., with a parameter server) [29], our rate asymptotically achieves linear speedup , i . e ., when T is sufficiently large, the first term in the convergence rate O ( 1 / √ nT ) becomes dominant and improves with respect to the number of nodes.

Even though linear speedup is a very desirable property, DL algorithms often require many more rounds to reach linear speedup compared to centralized SGD due to the additional error (the second term in the above convergence rate) arising from partial averaging of the local updates. To capture this phenomenon and to compare different decentralized learning algorithms, previous works [10, 50, 53, 57] adopt the concept of transient iterations which are the number of rounds before a decentralized algorithm reaches its linear speedup stage, i . e ., when T is relatively small such that the second term of the convergence rate dominates the first term.

Wederive that EL requires O ( n 3 / s 2 ) transient iterations, which improves upon the best known bound by a factor of s 2 . We also show this result in Table 1. We note that while EL matches the state-of-

1 Semi-dynamic topologies remain fixed throughout the learning process but nodes select subsets of adjacent (neighboring) nodes each round to communicate with.

the-art bounds when s ∈ O (1) , it offers additional flexibility over other methods through parameter s that provably improves the theoretical convergence speed depending on the communication capabilities of the nodes. For instance, when s ∈ O (log n ) as in Erd˝ os-Rényi and EquiStatic topologies, the number of transient iterations for EL reduces to O ( n 3 / log 2 n ) , outperforming other methods. This improvement comes from the fact that the second term in our convergence rate is superior to the corresponding term O ( 1 / 3 √ p 2 T 2 ) in the rate of D-PSGD, where p ∈ (0, 1] is the spectral gap of the mixing matrix. We expound more on this in Section 3.

We present in Section 4 our experimental findings. Using two standard image classification datasets, we compare EL-Oracle and EL-Local against static regular graphs and the state-of-the-art EquiTopo topologies. We find that EL-Oracle and EL-Local converge faster than the baselines and save up to 1.7 × communication volume to reach the highest accuracy of the most competitive baseline.

2 Epidemic Learning

In this section, we first formally define the decentralized optimization problem. Then we outline our EL algorithm and its variants in Section 2.2.

2.1 Problem statement

We consider a system of n nodes [ n ] := { 1, . . . , n } where the nodes can communicate by sending messages. Similar to existing work in this domain [53], we consider settings in which a node can communicate with all other nodes. The implications of this assumption are further discussed in Appendix D. Consider a data space Z and a loss function f : R d ×Z → R . Given a parameter x ∈ R d , a data point ξ ∈ Z incurs a loss of value f ( x , ξ ) . Each node i ∈ [ n ] has a data distribution D ( i ) over Z , which may differ from the data distributions of other nodes. We define the local loss function of i over distribution D ( i ) as f ( i ) ( x ) := E ξ ∼D ( i ) [ f ( x , ξ )] . The goal is to collaboratively minimize the global average loss by solving the following optimization problem:

j

Round k

j

Round k + 1

j

j

EL-Oracle (forming a

s

-regular graph)

EL-Local (forming a

s

-out graph)

Round k + 1

Round k

2.2 Description of EL

We outline EL, executed by node i , in Algorithm 1. We define the initial model of node i as x ( i ) 0 and a step-size γ used during the local model update. The EL algorithm runs for T rounds. Each round consists of two phases: a local update phase (line 3-5) in which the local model is updated using the local dataset of node i , and a random communication phase (line 6-9) in which model updates are sent to other nodes chosen randomly. In the local update phase, node i samples a data point ξ ( i ) t from its local data distribution D ( i ) (line 3), computes the stochastic gradient g ( i ) t (line 4) and partially updates its local model x ( i ) t +1 / 2 using step-size γ and gradient g ( i ) t (line 5).

The random communication phase follows, where node i first selects s of other nodes from the set of all nodes excluding itself: [ n ] \ { i } (line 6). This sampling step is the innovative element of EL, and we present two variants later. It then sends its recently updated local model x ( i ) t + 1 / 2 to the selected nodes and waits for model updates from other nodes. Subsequently, each node i updates its model based on the models it receives from other nodes according to Equation (2). The set of nodes that send their models to node i is denoted by S ( i ) t . The new model for node i is computed as a weighted average of the models received from the other nodes and the local model of node i , where the weights are inversely proportional to the number of models received plus one.

We now describe two approaches to sample s other nodes (line 6), namely EL-Oracle and EL-Local:

EL-Oracle With EL-Oracle, the union of selected communication links forms a s -regular topology in which every pair of nodes has an equal probability of being neighbors. Moreover, if node i samples node j , j will also sample i (communication is undirected). Figure 2 (left) depicts EL-Oracle sampling from the perspective of node j in two consecutive iterations. One possible way to generate such a dynamic graph is by generating an s -regular structure and then distributing a random permutation of

nodes at each round. Our implementation (see Section 4) uses a central coordinator to randomize and synchronize the communication topology each round.

EL-Local Constructing the s -regular topology in EL-Oracle every round can be challenging in a fully decentralized manner as it requires coordination amongst nodes to ensure all nodes have precisely s incoming and outgoing edges. This motivates us to introduce EL-Local, a sampling approach where each node i locally and independently samples s other nodes and sends its model update to them, without these s nodes necessarily sending their model back to i . The union of selected nodes now forms a s -out topology. Figure 2 (right) depicts EL-Local sampling from the perspective of node j in two consecutive iterations. In practice, EL-Local can be realized either by exchanging peer information before starting the learning process or using a decentralized peer-sampling service that provides nodes with (partial) views on the network [20, 41, 55]. While both the topology construction in EL-Oracle as well as peer sampling in EL-Local add some communication and computation overhead, this overhead is minimal compared to the resources used for model exchange and training.

Even though each node sends s messages for both ELOracle and EL-Local, in EL-Local, different nodes may receive different numbers of messages in each training round as each node selects the set of its out-neighbors locally and independent from other nodes. While this might cause an imbalance in the load on individual nodes, we argue that this does not pose a significant issue in practice. To motivate this statement, we run an experiment with n = 100 , 1000 and 10 000 and for each value of n set s = ⌈ log 2 ( n ) ⌉ . We simulate up to 5000 rounds for each configuration. We show in Figure 3 a CDF with the number of incoming models each round and observe that the distribution of the number of models received by each node is very light-tailed. In a 10 000 node network (and s =13 ), nodes receive less than 22 models in 99 %of all rounds. As such, it is improbable that a node receives

0

10

20

30

40

0

50

100

13

18

22

7

10 13

99th percentile

50th percentile

CDF [

%

]

s

= 7, 100 nodes

s

= 10, 1000 nodes

s

= 13, 10000 nodes

a disproportionally large number of models in a given round. In practice, we can alleviate this imbalance issue by adopting a threshold value k on the number of models processed by each node in a particular round and ignoring incoming models after having received k models already. The sender node can then retry model exchange with another random node that is less occupied.

3 Theoretical Analysis

In this section, we first present our main theoretical result demonstrating the finite-time convergence of EL. We then compare our result with the convergence rate of D-PSGD on different topologies.

3.1 Convergence of EL

In our analysis, we consider the class of smooth loss functions, and we assume that the variance of the noise of stochastic gradients is bounded. We use of the following assumptions that are classical to the analysis of stochastic first-order methods and hold for many learning problems [4, 14].

Assumption 1 (Smoothness) . For all i ∈ [ n ] , the function f ( i ) : R d → R is differentiable and there exists L < ∞ , such that for all x , y ∈ R d ,

Assumption 2 (Bounded stochastic noise) . There exists σ < ∞ such that for all i ∈ [ n ] , and x ∈ R d ,

Moreover, we assume that the heterogeneity among the local loss functions measured by the average distance between the local gradients is bounded.

Assumption 3 (Bounded heterogeneity) . There exists H < ∞ , such that for all x ∈ R d ,

We note that this assumption is standard in heterogeneous (a.k.a. non-i.i.d) settings, i . e ., when nodes have different data distributions [32, 57]. In particular, H can be bounded based on the closeness of the underlying local data distributions [13]. We now present our main theorem.

Theorem 1. Consider Algorithm 1. Suppose that assumptions 1, 2 and 3 hold true. Let ∆ 0 be a real value such that F ( x 0 ) -min x ∈ R d F ( x ) ≤ ∆ 0 . Then, for any T ≥ 1 , n ≥ 2 , and s ≥ 1 : a) For EL-Oracle, setting

we have

where b) For EL-Local , setting

we have

where

To check the tightness of this result, we consider the special case when s = n -1 . Then, by Theorem 1, we have α s = β s = 0 , and thus both of the convergence rates become O ( √ L ∆ 0 σ 2 / nT + L ∆ 0 / T ) , which is the same as the convergence rate of (centralized) SGD for non-convex loss functions [14]. This is expected as, in this case, every node sends its updated model to all other nodes, corresponding to all-to-all communication in a fully-connected topology and thus perfectly averaging the stochastic gradients without any drift between the local models.

The proof of Theorem 1 is given in Appendix A, where we obtain a tighter convergence rate than existing methods. It is important to note that as the mixing matrix of a regular graph is doubly stochastic, for EL-Oracle, one can use the general analysis of D-PSGD with (time-varying) doubly stochastic matrices [24, 30] to obtain a convergence guarantee. However, the obtained rate would not be as tight and would not capture the O ( 1 / 3 √ s ) improvement in the second term, which is the main advantage of randomization (see Section 3.2). Additionally, it is unclear how these analyses can be generalized to the case where the mixing matrix is not doubly stochastic, which is the case for our EL-Local algorithm. Furthermore, another line of work [1, 43, 44] provides convergence guarantees for decentralized optimization algorithms based on the PushSum algorithm [22], with communicating the mixing weights. It may be possible to leverage this proof technique to prove the convergence of EL-Local. However, this approach yields sub-optimal dimension-dependent convergence guarantees ( e . g ., see parameter C in Lemma 3 of [1]) and does not capture the benefit of randomized communication.

Remark 1. Most prior work [24, 32] provides the convergence guarantee on the average of the local models ¯ x t = 1 n ∑ i ∈ [ n ] x ( i ) t . However, as nodes cannot access the global averaged model, we provide the convergence rate directly on the local models. Nonetheless, the same convergence guarantee as Theorem 1 also holds for the global averaged model.

3.2 Discussion and comparison to prior results

To provide context for the above result, we note that the convergence rate of decentralized SGD with non-convex loss functions and a doubly stochastic mixing matrix [24, 30] is

where p ∈ (0, 1] is the spectral gap of the mixing matrix and 1 / p is bounded by O ( n 2 ) for ring [23], O ( n ) for torus [23], O (1) for Erd˝ os-Rényi random graph [45], O (log n ) for exponential graph [57], and O (1) for EquiTopo [53]. We now compare the convergence of EL against other topologies across two key properties: linear speed-up and transient iterations.

̸

Linear speed-up Both of our convergence rates preserve a linear speed-up of O ( 1 / √ nT ) in the first term. For EL-Oracle, this term is the same as (3). However, in the case of EL-Local, in addition to the stochastic noise σ , this term also depends on the heterogeneity parameter H that vanishes when increasing the sample size s . This comes from the fact that, unlike EL-Oracle, the communication phase of EL-Local does not preserve the exact average of the local models ( i . e ., ∑ i ∈ [ n ] x ( i ) t +1 = ∑ i ∈ [ n ] x ( i ) t + 1 / 2 ), and it only preserves the average in expectation. This adds an error term to the rate of EL-Local. However, as the update vector remains an unbiased estimate of the average gradient, this additional term does not violate the linear speed-up property. Our analysis suggests that setting s ≈ H 2 σ 2 can help mitigate the effect of heterogeneity on the convergence of EL-Local. Intuitively, more data heterogeneity leads to more disagreement between the nodes, which requires more communication rounds to converge.

Transient iterations Our convergence rates offer superior second and third terms compared to those in (3). This is because first, p can take very small values, particularly when the topology connectivity is low ( e . g ., 1 p ∈ O ( n 2 ) for a ring) and second, even when the underlying topology is well-connected and 1 p ∈ O (1) , such as in EquiTopo [53], the second term in our rates still outperforms the one in (3) by a factor of 3 √ s . This improvement is reflected in the number of transient iterations before the linear speed-up stage, i . e ., the number of rounds required for the first term of the convergence rate to dominate the second term [57]. In our rates, the number of transient iterations is in O ( n 3 / s 2 ) , whereas in (3), it is O ( n 3 / p 2 ) for the homogeneous case and O ( n 3 / p 4 ) for the heterogeneous case. We remark that p ∈ (0, 1] , but s ≥ 1 is an integer; therefore, when s ∈ O (1) the number of transient iterations for EL matches the state-of-the-art bound. However, it can be provably improved by increasing s depending on the communication capabilities of the nodes, which adds more flexibility to EL with theoretical guarantees compared to other methods. For instance, for s ∈ O (log n ) as in Erd˝ os-Rényi and EquiStatic topologies, the number of transient iterations for EL becomes O ( n 3 / log 2 n ) which outperforms other methods (also see Table 1). Crucially, a key implication of this result is that our algorithm requires fewer rounds and, therefore, less communication to converge. We empirically show the savings in the communication of EL in Section 4.

4 Evaluation

We present here the empirical evaluation of EL and compare it with state-of-the-art DL baselines. We first describe the experimental setup and then show the performance of EL-Oracle and EL-Local.

4.1 Experimental setup

Network setup and implementation We deploy 96 DL nodes for each experiment, interconnected according to the evaluated topologies. When experimenting with s -regular topologies, each node maintains a fixed degree of ⌈ log 2 ( n ) ⌉ , i . e ., each node has 7 neighbors. For EL-Oracle we introduce a centralized coordinator (oracle) that generates a random 7 -Regular topology at the start of each

1

2

3

20

40

60

+

2.2

%accuracy

Communication round (

×

10 3

)

Test accuracy [

%

]

1

2

3

1

1.5

2

Test loss

Fully connected

EL-Local

EL-Oracle

7-Regular static

8-U-EquiStatic

100MiB

1GiB

10GiB

100GiB

20

40

60

Communication volume per node (log scale)

Test accuracy [

%

]

4GiB

6GiB

8GiB

62

64

66

-

2.9GiB

(

1.7

×

)

Test accuracy [

%

]

2

2.5

3

61

65

round and informs all nodes about their neighbors for the upcoming round. For EL-Local we make each node aware of all other nodes at the start of the experiment. To remain consistent with other baselines, we fix s = ⌈ log 2 ( n ) ⌉ = 7 when experimenting with EL, i . e ., each node sends model updates to 7 other nodes each round. Both EL-Oracle and EL-Local were implemented using the DecentralizePy framework [11] and Python 3.8 3 . For reproducibility, a uniform seed was employed for all pseudo-random generators within each node.

Baselines We compare the performance of EL-Oracle and EL-Local against three variants of D-PSGD. Our first baseline is a fully-connected topology (referred to as Fully connected ), which presents itself as the upper bound for performance given its optimal convergence rate [1]. We also compare with a s -regular static topology, the non-changing counterpart of EL-Oracle (referred to as 7-Regular static ). This topology is randomly generated at the start of each run according to the random seed, but is kept fixed during the learning. Finally, we compare EL against the communication-efficient topology U-EquiStatic [53]. Since U-EquiStatic topologies can only have even degrees, we generate U-EquiStatic topologies with a degree of 8 to ensure a fair comparison. We refer to this setting as 8-U-EquiStatic .

Learning task and partitioning We evaluate the baseline algorithms using the CIFAR-10 image classification dataset [26] and the FEMNIST dataset, the latter being part of the LEAF benchmark [7]. In this section we focus on the results for CIFAR-10 and present the results for FEMNIST in Appendix C.4. We employ a non-IID data partitioning using the Dirichlet distribution function [19], parameterized with α = 0.1 . We use a GN-LENET convolutional neural network [18]. Full details on our experimental setup and hyperparameter tuning can be found in Appendix C.1.

Metrics We measure the average top-1 test accuracy and test loss of the model on the test set in the CIFAR-10 learning task every 20 communication rounds. Furthermore, we present the average top-1 test accuracy against the cumulative outgoing communication per node in bytes. We also emphasize the number of communication rounds taken by EL to reach the best top-1 test accuracy of static 7-Regular topology. We run each experiment five times with different random seeds, and we present the average metrics with a 95% confidence interval.

4.2 EL against baselines

Figure 4 shows the performance of EL-Oracle and EL-Local against the baselines for the CIFAR-10 dataset. D-PSGD over a fully-connected topology achieves the highest accuracy, as expected, but incurs more than an order of magnitude of additional communication. EL-Oracle converges faster than its static counterparts of 7-Regular static and 8-U-EquiStatic . After 3000 communication rounds, EL-Oracle and EL-Local achieve up to 2.2% higher accuracy compared to 7-Regular static (the most competitive baseline). Moreover, EL-Oracle takes up to 1.7 × fewer communication rounds and saves 2.9 GiB of communication to reach the best accuracy attained by 7-Regular static. Surprisingly, 8-U-EquiStatic shows worse performance compared to 7-Regular static . The plots further highlight that the less constrained EL-Local variant has a very competitive performance compared to EL-Oracle: there is negligible utility loss when sampling locally compared to generating a s -Regular graph every round. We provide additional observations and results in Appendix C.

1

2

3

30

40

50

60

Communication round (

×

10 3

)

Test accuracy [

%

]

EL-Oracle

7-regular static

s

= 7

s

= 4

s

= 14

1

2

3

30

40

50

60

EL-Local

10MiB

100MiB

1GiB

10GiB

20

40

60

Communication volume per node (log scale)

EL-Oracle

3GiB

5GiB

10GiB

60

65

We summarize our main experimental findings in Table 2, which outlines the highest achieved top-1 test accuracy, lowest top-1 test loss, and communication cost to a particular target accuracy for our evaluated baselines. This target accuracy is chosen as the best top-1 test accuracy achieved by the 7-Regular static topology (64.32%), and the communication cost to reach this target accuracy is presented for all the topologies in the right-most column of Table 2. In summary, EL-Oracle and EL-Local converge faster and to higher accuracies compared to 7-Regular static and 8-UEquiStatic topologies, and require 1.7 × and 1.6 × less communication cost to reach the target accuracy, respectively.

4.3 Sensitivity Analysis of sample size s

The sample size s determines the number of outgoing neighbours of each node at each round of EL-Oracle and EL-Local. We show the impact of this parameter on the test accuracy in the two left-most plots in Figure 5, for varying values of s , against the baseline of a 7-regular static graph. We chose the values of s as ⌈ log 2 ( n ) 2 ⌉ = 4 , and ⌈ 2 log 2 ( n ) ⌉ = 14 , over and above ⌈ log 2 ( n ) ⌉ = 7 . We observe that, as expected, increasing s leads to quicker convergence for both EL-Oracle and EL-Local (see Theorem 1). Increasing s also directly increases the communication volume. The two right-most plots in Figure 5 show the test accuracy of EL-Oracle when the communication volume increases. After 3000 communication rounds, EL-Oracle with s = 14 has incurred a communication volume of 15.1 GiB , compared to 4.3 GiB for s = 4 . An optimal value of s depends on the network of the environment where EL is deployed. In data center settings where network links usually have high capacities, one can employ a high value of s . In edge settings with limited network capacities, however, the value of s should be smaller to avoid network congestion.

5 Related Work

Decentralized Parallel Stochastic Gradient Descent (D-PSGD) Stochastic Gradient Descent is a stochastic variant of the gradient descent algorithm and is widely used to solve optimization problems at scale [14, 39, 46]. Decentralized algorithms using SGD have gained significant adoption as a method to train machine learning models, with Decentralized Parallel SGD (D-PSGD) being the most well-known DL algorithm [15, 31-33, 49]. D-PSGD avoids a server by relying on a communication topology describing how peers exchange their model updates [32].

Static topologies In a static topology, all nodes and edges remain fixed throughout the training process. The convergence speed of decentralized optimization algorithms closely depends on the mixing properties of the underlying topologies. There is a large body of work [2, 3, 6, 40] studying the mixing time of different random graphs such as the Erdos-Renyi graph and the geometric random graph. As the Erdos-Renyi graphs have better mixing properties [45], we compare EL with this family in Table 1. In [57], the authors prove that static exponential graphs in which nodes are connected to O (log n ) neighbors are an effective topology for DL. EquiStatic is a static topology family whose consensus rate is independent of the network size [53].

Semi-dynamic topologies Several DL algorithms impose a fixed communication topology at the start of the training but have nodes communicate with random subsets of their neighbors each round. We classify such topologies as semi-dynamic topologies. The one-peer exponential graph has each node cycling through their O (log n ) neighbors and has a similar convergence rate to static exponential graphs [57]. In the AD-PSGD algorithm, each node independently trains and averages its model with the model of a randomly selected neighbour [33]. EquiDyn is a dynamic topology family whose consensus rate is independent of the network size [53].

Time-varying and randomized topologies A time-varying topology is a topology that changes throughout the training process [9]. The convergence properties of time-varying graphs in distributed optimization have been studied by various works [23, 35, 36, 43, 45]. While these works provide convergence guarantees for decentralized optimization algorithms over time-varying (or random) topologies, they do not show the superiority of randomized communication, and they do not prove a convergence rate that cannot be obtained with a static graph [53]. Another work [34] considers client subsampling in decentralized optimization where at each round, only a fraction of nodes participate in the learning procedure. This approach is orthogonal to the problem we consider in this paper.

Gossip Learning Closely related to EL-Local is gossip learning (GL), a DL algorithm in which each node progresses through rounds independently from other peers [48]. In each round, a node sends their model to another random node and aggregates incoming models received by other nodes, weighted by age. While GL shows competitive performance compared to centralized approaches [16, 17], its convergence on non-convex loss functions has not been theoretically proven yet [47]. While at a high level, EL-Local with s = 1 may look very similar to GL, there are some subtle differences. First, GL operates in an asynchronous manner whereas EL proceeds in synchronous rounds. Second, GL applies weighted averaging when aggregating models, based on model age, and EL aggregates models unweighted. Third, if a node in GL receives multiple models in a round, this node will aggregate the received model with its local model for each received model separately, whereas in EL, there will be a single model update per round, and all the received models from that round are aggregated together.

In contrast to the existing works, EL leverages a fully dynamic and random topology that changes each round. While static and semi-dynamic topologies have shown to be effective in certain settings, EL surpasses their performance by enabling faster convergence, both in theory and in practice.

6 Conclusions

We have introduced Epidemic Learning (EL), a new DL algorithm that accelerates model convergence and saves communication costs by leveraging randomized communication. The key idea of EL is that in each round, each node samples s other nodes in a n -node network and sends their model updates to these sampled nodes. We introduced two variants of the sampling approach: EL-Oracle in which the communication topology forms a s -regular graph each round, and EL-Local which forms a s -out graph. We theoretically proved the convergence of both EL variants and derived that the number of transient iterations for EL is in O ( n 3 / s 2 ) , outperforming the best-known bound O ( n 3 ) by a factor of s 2 for smooth non-convex functions. Our experimental evaluation on the CIFAR-10 learning task showcases the 1.7 × quicker convergence of EL compared to baselines and highlights the reduction in communication costs.

References

Mahmoud Assran, Nicolas Loizou, Nicolas Ballas, and Mike Rabbat. Stochastic gradient push for distributed deep learning. In International Conference on Machine Learning , pages 344-353. PMLR, 2019.

Itai Benjamini, Gady Kozma, and Nicholas Wormald. The mixing time of the giant component of a random graph. Random Structures & Algorithms , 45(3):383-407, 2014.

Andrew Beveridge and Jeanmarie Youngblood. The best mixing time for random walks on trees. Graphs and Combinatorics , 32:2211-2239, 2016.

Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review , 60(2):223-311, 2018.

Stephen Boyd, Arpita Ghosh, Balaji Prabhakar, and Devavrat Shah. Randomized gossip algorithms. IEEE transactions on information theory , 52(6):2508-2530, 2006.

Stephen P Boyd, Arpita Ghosh, Balaji Prabhakar, and Devavrat Shah. Mixing times for random walks on geometric random graphs. In ALENEX/ANALCO , pages 240-249, 2005.

Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Koneˇ cn` y, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097 , 2018.

Daniel Cason, Nenad Milosevic, Zarko Milosevic, and Fernando Pedone. Gossip consensus. In Proceedings of the 22nd International Middleware Conference , pages 198-209, 2021.

Arnaud Casteigts, Paola Flocchini, Walter Quattrociocchi, and Nicola Santoro. Time-varying graphs and dynamic networks. International Journal of Parallel, Emergent and Distributed Systems , 27(5):387-408, 2012.

Yiming Chen, Kun Yuan, Yingya Zhang, Pan Pan, Yinghui Xu, and Wotao Yin. Accelerating gossip sgd with periodic global averaging. In International Conference on Machine Learning , pages 1791-1802. PMLR, 2021.

Akash Dhasade, Anne-Marie Kermarrec, Rafael Pires, Rishi Sharma, and Milos Vujasinovic. Decentralized learning made easy with DecentralizePy. In 3rd Workshop on Machine Learning and Systems , 2023. doi: 10.1145/3578356.3592587.

Patrick T Eugster, Rachid Guerraoui, A-M Kermarrec, and Laurent Massoulié. Epidemic information dissemination in distributed systems. Computer , 37(5):60-67, 2004.

Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems , 33:3557-3568, 2020.

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

Saeed Ghadimi, Guanghui Lan, and Hongchao Zhang. Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming , 155 (1-2):267-305, 2016.

István Heged˝ us, Gábor Danner, and Márk Jelasity. Gossip learning as a decentralized alternative to federated learning. In Distributed Applications and Interoperable Systems: 19th IFIP WG 6.1 International Conference, DAIS 2019, Held as Part of the 14th International Federated Conference on Distributed Computing Techniques, DisCoTec 2019, Kongens Lyngby, Denmark, June 17-21, 2019, Proceedings 19 , pages 74-90. Springer, 2019.

István Heged˝ us, Gábor Danner, and Márk Jelasity. Decentralized learning works: An empirical comparison of gossip learning and federated learning. Journal of Parallel and Distributed Computing , 148:109-124, 2021.

Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Conference on Machine Learning , pages 4387-4398. PMLR, 2020.

Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 , 2019.

Márk Jelasity, Spyros Voulgaris, Rachid Guerraoui, Anne-Marie Kermarrec, and Maarten Van Steen. Gossip-based peer sampling. ACM Transactions on Computer Systems (TOCS) , 25 (3):8-es, 2007.

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210, 2021.

David Kempe, Alin Dobra, and Johannes Gehrke. Gossip-based computation of aggregate information. In 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. Proceedings. , pages 482-491. IEEE, 2003.

Anastasia Koloskova, Sebastian Stich, and Martin Jaggi. Decentralized stochastic optimization and gossip algorithms with compressed communication. In International Conference on Machine Learning , pages 3478-3487. PMLR, 2019.

Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning , pages 5381-5393. PMLR, 2020.

Lingjing Kong, Tao Lin, Anastasia Koloskova, Martin Jaggi, and Sebastian Stich. Consensus control for decentralized deep learning. In International Conference on Machine Learning , pages 5686-5696. PMLR, 2021.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 dataset. 55(5), 2014. URL https://www.cs.toronto.edu/~kriz/cifar.html .

Batiste Le Bars, Aurélien Bellet, Marc Tommasi, Erick Lavoie, and Anne-Marie Kermarrec. Refined convergence and topology learning for decentralized sgd with heterogeneous data. In International Conference on Artificial Intelligence and Statistics , pages 1672-1702. PMLR, 2023.

Mu Li, Li Zhou, Zichao Yang, Aaron Li, Fei Xia, David G Andersen, and Alexander Smola. Parameter server for distributed machine learning. In Big learning NIPS workshop , volume 6, 2013.

Mu Li, David G Andersen, Jun Woo Park, Alexander J Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J Shekita, and Bor-Yiing Su. Scaling distributed machine learning with the parameter server. In 11th { USENIX } Symposium on Operating Systems Design and Implementation ( { OSDI } 14) , pages 583-598, 2014.

Xiang Li, Wenhao Yang, Shusen Wang, and Zhihua Zhang. Communication-efficient local decentralized sgd methods. arXiv preprint arXiv:1910.09126 , 2019.

Xiangru Lian, Yijun Huang, Yuncheng Li, and Ji Liu. Asynchronous parallel stochastic gradient for nonconvex optimization. Advances in neural information processing systems , 28, 2015.

Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in neural information processing systems , 30, 2017.

Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. Asynchronous decentralized parallel stochastic gradient descent. In International Conference on Machine Learning , pages 3043-3052. PMLR, 2018.

Ziwei Liu, Anastasia Koloskova, Martin Jaggi, and Tao Lin. Decentralized stochastic optimization with client sampling. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop) .

Ilan Lobel and Asuman Ozdaglar. Distributed subgradient methods for convex optimization over random networks. IEEE Transactions on Automatic Control , 56(6):1291-1306, 2010.

Songtao Lu and Chai Wah Wu. Decentralized stochastic non-convex optimization over weakly connected time-varying digraphs. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5770-5774. IEEE, 2020.

Yucheng Lu and Christopher De Sa. Optimal complexity in decentralized training. In International Conference on Machine Learning , pages 7111-7123. PMLR, 2021.

Alberto Montresor. Gossip and epidemic protocols. Wiley encyclopedia of electrical and electronics engineering , 1, 2017.

Eric Moulines and Francis Bach. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. Advances in neural information processing systems , 24, 2011.

Asaf Nachmias and Yuval Peres. Critical random graphs: diameter and mixing time. 2008.

Brice Nédelec, Julian Tanke, Davide Frey, Pascal Molli, and Achour Mostéfaoui. An adaptive peer-sampling protocol for building networks of browsers. World Wide Web , 21(3):629-661, 2018.

Angelia Nedic. Distributed gradient methods for convex machine learning problems in networks: Distributed optimization. IEEE Signal Processing Magazine , 37(3):92-101, 2020.

Angelia Nedi´ c and Alex Olshevsky. Distributed optimization over time-varying directed graphs. IEEE Transactions on Automatic Control , 60(3):601-615, 2014.

Angelia Nedi´ c and Alex Olshevsky. Stochastic gradient-push for strongly convex functions on time-varying directed graphs. IEEE Transactions on Automatic Control , 61(12):3936-3947, 2016.

Angelia Nedi´ c, Alex Olshevsky, and Michael G Rabbat. Network topology and communicationcomputation tradeoffs in decentralized optimization. Proceedings of the IEEE , 106(5):953-976, 2018.

Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization , 19(4): 1574-1609, 2009.

Noa Onoszko, Gustav Karlsson, Olof Mogren, and Edvin Listo Zec. Decentralized federated learning of deep neural networks on non-iid data. arXiv preprint arXiv:2107.08517 , 2021.

Róbert Ormándi, István Heged˝ us, and Márk Jelasity. Gossip learning with linear models on fully distributed data. Concurrency and Computation: Practice and Experience , 25(4):556-571, 2013.

Pitch Patarasuk and Xin Yuan. Bandwidth optimal all-reduce algorithms for clusters of workstations. Journal of Parallel and Distributed Computing , 69(2):117-124, 2009.

Shi Pu, Alex Olshevsky, and Ioannis Ch Paschalidis. A sharp estimate on the transient time of distributed stochastic gradient descent. IEEE Transactions on Automatic Control , 67(11): 5900-5915, 2021.

Alexander Rogozin, Alexander Gasnikov, Aleksander Beznosikov, and Dmitry Kovalev. Decentralized optimization over time-varying graphs: a survey. arXiv preprint arXiv:2210.09719 , 2022.

Kevin Scaman, Francis Bach, Sébastien Bubeck, Laurent Massoulié, and Yin Tat Lee. Optimal algorithms for non-smooth distributed optimization in networks. Advances in Neural Information Processing Systems , 31, 2018.

Zhuoqing Song, Weijian Li, Kexin Jin, Lei Shi, Ming Yan, Wotao Yin, and Kun Yuan. Communication-efficient topologies for decentralized learning with $o(1)$ consensus rate. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id= AyiiHcRzTd .

Thijs Vogels, Hadrien Hendrikx, and Martin Jaggi. Beyond spectral gap (extended): The role of the topology in decentralized learning. arXiv preprint arXiv:2301.02151 , 2023.

Spyros Voulgaris, Daniela Gavidia, and Maarten Van Steen. Cyclon: Inexpensive membership management for unstructured p2p overlays. Journal of Network and systems Management , 13: 197-217, 2005.

Jianyu Wang, Anit Kumar Sahu, Zhouyi Yang, Gauri Joshi, and Soummya Kar. Matcha: Speeding up decentralized sgd via matching decomposition sampling. In 2019 Sixth Indian Control Conference (ICC) , pages 299-300. IEEE, 2019.

Bicheng Ying, Kun Yuan, Yiming Chen, Hanbin Hu, Pan Pan, and Wotao Yin. Exponential graph is provably efficient for decentralized deep training. Advances in Neural Information Processing Systems , 34:13975-13987, 2021.

Bicheng Ying, Kun Yuan, Hanbin Hu, Yiming Chen, and Wotao Yin. Bluefog: Make decentralized algorithms practical for optimization and deep learning. arXiv preprint arXiv:2111.04287 , 2021.

Organization

The appendices are organized as follows:

Appendix A proves the convergence guarantee of EL.

In Appendix B, we prove the key lemmas that are used in the convergence proof.

Appendix C provides experimental details, computational resources used, and further evaluation of EL with independent and identically distributed (IID) data distributions and the FEMNIST dataset.

In Appendix D, we discuss the effect of network connectivity on the performance of EL.

A Convergence Proof

In this section, we prove Theorem 1, by setting

for EL-Oracle, and

for EL-Local.

Notation: For any set of n vectors { x (1) , . . . , x ( n ) } , we denote their average by ¯ x := 1 n ∑ i ∈ [ n ] x ( i ) .

A.1 Proof steps

We outline here the critical elements for proving Theorem 1.

Mixing efficiency of EL. First, we analyze the mixing properties of the communication phase of both variants of the algorithm, and we observe that:

(a) while EL-Oracle preserves the exact average of the local vectors, i.e., ¯ x t +1 = ¯ x t + 1 / 2 , ELLocal only does so in expectation. This property is critical for ensuring that the global update is an unbiased estimator of the average of local gradients, which is necessary for obtaining convergence guarantees with linear speed-up.

(b) The communication phase of both EL-Local and EL-Oracle reduce the drift among the local models by a factor of O ( 1 / s ) .

(c) The variance of the averaged model after the communication phase, in EL-Local is in O ( 1 / ns ) .

More formally, we have the following lemma.

Appendix

Lemma 1. Consider Algorithm 1. Let n ≥ 2 , s ≥ 1 , T ≥ 1 , and t ∈ { 0, . . . , T -1 } .

For EL-Oracle , we have

where

For EL-Local , we have:

Uniform bound on model drift and gradient drift. Next, using the previous lemma, we prove that, for a careful choice of the step-size γ , the local models stay close to each other during the learning process.

Lemma 2. Suppose that assumptions 1, 2, and 3 hold true. Consider Algorithm 1. Consider a step-size γ such that γ ≤ 1 20 L . For any t ≥ 0 , we obtain that

and

where η s = α s for EL-Oracle and η s = β s for EL-Local as defined in Lemma 1.

Bound on the gradient norm. Next, we obtain a bound on the gradient of the loss function F ( x ) by analyzing the growth of F ( x ) , over the trajectory of the average of local models in Algorithm 1.

Lemma 3. Suppose that assumptions 1 and 2 hold true. Consider Algorithm 1 with γ ≤ 1 2 L . For any t ∈ { 0, . . . , T -1 } , we obtain that

A.2 Proof of Theorem 1

We can now prove the theorem using the above lemmas.

Proof of Theorem 1. By Young's inequality, for any i ∈ [ n ] , we have

where in the second inequality we used Assumption 1. Then, averaging over i ∈ [ n ] , we obtain that

where we used Lemma 5. Combining this with Lemma 3, we obtain that 4

We now analyze the two variants of the algorithm separately.

EL-Oracle:

In this case, by Property (2a) of Lemma 1, we have ¯ x t +1 -¯ x t + 1 / 2 = 0 , therefore, by (6), we have

Note also that by Lemma 1, we have

Moreover, as shown in Remark 2, we have α s ≤ 1 / 2 , and thus, 20 1+3 α s (1 -α s ) 2 ≤ 200 . Therefore,

Combining this with (7), we obtain that

Averaging over t ∈ { 0, . . . , T -1 } , we obtain that

where we used the fact that F (¯ x 0 ) -F (¯ x T ) ≤ F (¯ x 0 ) -min x ∈ R d F ( x ) ≤ ∆ 0 . Setting

Plugging (9) and (10) in (8), we obtain that

where we used the fact that 800 100 (2 / 3) ≤ 38 .

EL-Local:

The proof is similar to EL-Oracle, up to some additional error terms. By Property (2c) of Lemma 1, we obtain that

where in the last inequality we use Young's inequality. Combining this with (6), we obtain that

Note that by Remark 2, for any s ≥ 1 , and n ≥ 2 we have β s ≤ 1 -1 e , where e is Euler's number. Therefore,

Using this, the first bound in Lemma 2 can be simplified to

Combining this with (13), and the second bound of Lemma 2, we have

Taking the average over t ∈ { 0, . . . , T -1 } , we obtain that

Noting that β s < 1 -1 e (Remark 2), we have

Now, setting

we have

Plugging (17), and (16) in (15) we obtain that

where we used the fact that 2000 250 (2 / 3) ≤ 51 . This concludes the proof.

B Proof of the main lemmas

Notation: Let P t represent the history from step 0 to t . More precisely, we define

Moreover, we use the notation E t [ · ] := E [ · P t ] to denote the conditional expectation given the history P t and E [ · ] to denote the total expectation over the randomness of the algorithm; therefore we have, E [ · ] := E 0 [ · · · E T [ · ]] .

We first prove two simple useful lemmas.

Lemma 4. for any set of k vectors x (1) , . . . , x ( k ) , we have

Proof. By the triangle inequality, we have

The result then follows by noting that using Jensen's inequality, we have

Lemma 5. For any set { x ( i ) } i ∈ [ n ] of n vectors, we have

Proof.

Noting that ∑ j ∈ [ n ] ( x ( j ) -¯ x ) = 0 , yields the desired result.

B.1 Proof of Lemma 1

Lemma 1. Consider Algorithm 1. Let n ≥ 2 , s ≥ 1 , T ≥ 1 , and t ∈ { 0, . . . , T -1 } .

2. For EL-Local , we have:

Proof. For simplicity of the notation, we make the dependence on t implicit, and we denote by x ( i ) := x ( i ) t + 1 / 2 , the input vector and y ( i ) := x ( i ) t +1 the output vector of node i for the communication phase, and S ( i ) := S ( i ) t . Therefore, we have

Moreover, in the proof of this lemma, all the expectations E [ · ] are only with respect to the randomness of the communication phase, i.e., we compute the expectations assuming x ( i ) 's are given for all i ∈ [ n ] , and y ( i ) 's are computed by a random communication phase. Taking the total expectation of the final expression provided for each property then proves it as stated in the lemma.

First, we consider EL-Oracle :

Recall that in EL-Oracle, the communication topology forms a random undirected s -regular graph, and for each node i , all other nodes have the same probability of being i 's neighbor.

Property (a):

We have

where we used the fact that in EL-Oracle, each node receives exactly s models. Now as each node also sends its model to s other nodes, we have

Property (b):

We have

Let us define I ( i ) j the indicator function showing whether node j and node i are neighbors, i.e., I ( i ) j = 1 if i and j are neighbors and I ( i ) j = 0 otherwise. We then have

̸

̸

̸

̸

̸

̸

Now note that by symmetry (all nodes have the same probability of being a neighbor of node i ).

Similarly I ( i ) j I ( i ) k is only 1 when both j and k are neighbors of i , thus

̸

̸

Therefore,

Also,

Combining (18), (19), and (20) we obtain that

Also note that

̸

Combining (21) and (22), we obtain that

̸

̸

̸

̸

̸

̸

Now note that as ¯ y is the minimizer of g ( z ) := 1 n ∑ i ∈ [ n ] E [ ∥ ∥ y ( i ) -z ∥ ∥ 2 ] , we have

Combining this with (23), and using Lemma 5 yields

which is the desired result.

Now, we consider EL-Local :

Recall that in EL-Local, each node i , locally samples uniformly at random s other nodes and sends its model to them. Denote by A ( i ) := ∣ ∣ S ( i ) ∣ ∣ the number of nodes that send their vectors to node i , and I ( i ) j the indicator showing whether node j sends its vector to node i or not ( I ( i ) j = 1 if j sends its model to i , and I ( i ) j = 0 otherwise). We then have

and

Property (a):

For any node i , we have

where (a) uses the fact that E [ I ( i ) j | A ( i ) ] = A ( i ) n -1 as all other n -1 nodes have the same probability of sending their model to i . Denoting p = E [ A ( i ) A ( i ) +1 ] , we obtain that

Averaging over all i ∈ [ n ] yields the result.

Property (b):

We have

̸

Taking the expectation inside, we obtain that

̸

̸

̸

̸

Now note that E [ I ( i ) j | A ( i ) ] is the probability that j selects i , given that in total A ( i ) nodes select i , thus

Similarly I ( i ) j I ( i ) k is only 1 when both j and k select i , thus

Also, note that

and

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

Combining all yields

̸

Now note that by symmetry, the distribution of A ( i ) is the same as A ( j ) for any i , j ∈ [ n ] . Therefore,

Now note that

thus

̸

Now note that as each node j = 1 , independently and uniformly selects a set of s nodes, A (1) has a binomial distribution with parameters n -1 , and s n -1 . Therefore, for s > 0 , we have

Also noting that we obtain that

̸

̸

Noting that 1 n ∑ i ∈ [ n ] E [ ∥ ∥ y ( i ) -¯ y ∥ ∥ 2 ] ≤ 1 n ∑ i ∈ [ n ] E [ ∥ ∥ y ( i ) -¯ x ∥ ∥ 2 ] , from (24), and using Lemma 5, we obtain that

which is the desired result.

Property (c):

Note that

Now recall that

This implies that

̸

̸

̸

̸

̸

̸

̸

̸

Now note that by symmetry, for any i , j ∈ [ n ] , we have

Similarly, and

This implies that all three terms in (27) can be written as

̸

where c is a positive constant. We also have

̸

̸

̸

̸

̸

̸

Therefore all the terms in (27) are non-positive. Combining this with (26), we obtain that

where the second inequality uses (25). Combining this with Lemma 5 then concludes the proof.

Remark 2. Note that β s computed in Lemma 1 is decreasing in s and increasing in n , therefore, for any s ≥ 1 , and n ≥ 2 , we have

where e is Euler's Number and we used the fact that lim n →∞ ( 1 -1 n ) n = 1 e . Similarly, for any s ≥ 1 , and n ≥ 2 , we have

B.2 Proof of Lemma 2

Lemma 2. Suppose that assumptions 1, 2, and 3 hold true. Consider Algorithm 1. Consider a step-size γ such that γ ≤ 1 20 L . For any t ≥ 0 , we obtain that

and

where η s = α s for EL-Oracle and η s = β s for EL-Local as defined in Lemma 1.

Proof. First note that for any i ∈ [ n ] , we have

Thus, using Lemma 4, we have

Taking the conditional expectation, we have

Now by Assumption 2, we have

By Assumption 1, we have

Thus, by Assumption 3, and Lemma 5, we obtain that

Combining (28), (29), (30), and (31), and taking total expectation from both sides, we obtain that

We now analyze 1 n 2 ∑ i , j ∈ [ n ] E [ ∥ ∥ ∥ x ( i ) t -x ( j ) t ∥ ∥ ∥ 2 ] . From Algorithm 1, recall that for all i ∈ [ n ] , we have x ( i ) t + 1 / 2 = x ( i ) t -γg ( i ) t . We obtain for all i , j ∈ [ n ] , that

where we used the fact that ( x + y ) 2 ≤ (1 + c ) x 2 + (1 + 1 / c ) y 2 for any c > 0 . Now recall that η s = α s for EL-Oracle, and η s = β s for EL-Local. Therefore, by Property ( b ) of Lemma 1, for both variant of the algorithm, we have

Then, applying Lemma 5, we have

Combining this with (34), we obtain that

Combining this with (33), we obtain that

Now note that for by Remark 2, for both variants of the algorithm, we have η s ≤ 1 -1 e , which implies that

Therefore,

Unrolling the recursion, we obtain that

Combining this with (32), we obtain that

This is the desired result.

B.3 Proof of Lemma 3

Lemma 3. Suppose that assumptions 1 and 2 hold true. Consider Algorithm 1 with γ ≤ 1 2 L . For any t ∈ { 0, . . . , T -1 } , we obtain that

Proof. Consider an arbitrary t ∈ [ T ] . Note that, we do not necessarily have ¯ x t +1 = ¯ x t + 1 / 2 . But by Lemma 1, we have E [¯ x t +1 ] = E [ ¯ x t + 1 / 2 ] . Now by the smoothness of the loss function (Assumption 1), we have

Now denoting ∇ F t = 1 n ∑ i ∈ [ n ] ∇ f ( i ) ( x ( i ) t ) , g t = 1 n ∑ i ∈ [ n ] g ( i ) t and taking the conditional expectation, we have

where (a) uses Young's inequality and (b) is based on the facts that by Assumption 2, we have E t [ g t ] = ∇ F t , and E t [ ∥ ∥ g t -∇ F t ∥ ∥ 2 ] ≤ σ 2 n . Now note that using the fact that γ ≤ 1 2 L , we have

Combining (35), and (36), we obtain that

Now, note that

where (a) uses Assumption 1, in (b), we used Lemma 5. Combining this with (37), we obtain that

Rearranging the terms we have

EL-Oracle

7-Regular static

8-U-EquiStatic

0

1

2

3

20

40

60

Test accuracy [

%

]

0

1

2

3

20

40

60

Communication round (

10 3

)

Test accuracy [

%

]

0

1

2

3

20

40

60

Test accuracy [

%

]

γ

= 0.01

γ

= 0.05

γ

= 0.1

γ

= 0.5

×

which is the lemma.

C Experimental Details and Further Evaluation

C.1 Experiment setup and learning rate tuning

Table 3 provides a summary of the learning parameters, model, and dataset used in the experiments. The models were evaluated on samples from the test set that were unseen during training. The step size ( γ ) was tuned by running each baseline on a range of values and taking the setup with the best validation accuracy. The plots in Figure 6 show the CIFAR-10 convergence plots for EL-Oracle, 7-Regular static , and 8-U-EquiStatic (most prominent topologies) over different step sizes. The optimal step-sizes are γ = 0.1 for Fully connected and 8-U-EquiStatic , and γ = 0.05 for the remaining algorithms and topologies over CIFAR-10. For FEMNIST, the optimal step size is γ = 0.1 for all algorithms and topologies. All experiments in Section 4 were conducted for a total of 3000 communication rounds and the experiments for homogeneous data distributions of CIFAR-10 (Figure 7) were conducted for a total of 1000 communication rounds. The experiments over the FEMNIST dataset were conducted for a total of 1440 communication rounds.

C.2 Computational resources

We perform experiments on 6 hyperthreading-enabled machines with dual Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz of 8 cores. All algorithms were implemented as part of Decentralizepy [11]. Each DL node was assigned 2 virtual cores. An experiment took at most 3 hours in wall-clock time for experiments in Section 4, 2 hours in wall-clock time for FEMNIST experiments, and 1 hour in wall-clock time for CIFAR-10 experiments in Appendix C, summing to a total of 1152 virtual CPU hours. Across all experiments that are presented in this article, including different seeds and learning rate tuning, the total virtual CPU time goes up to approximately 54 700 hours.

C.3 Experiments with homogeneous data distributions

In this experiment, we focused on evaluating the performance of different communication topologies under IID data. The data was generated using a Dirichlet distribution with α = 1.0 . As a reference

2

4

6

8

10

20

40

60

Communication round (

×

10 2

)

Test accuracy [

%

]

2

4

6

8

10

1

1.5

2

Test loss

Fully connected

EL-Local

EL-Oracle

7-Regular static

8-U-EquiStatic

100MiB

1GiB

10GiB

20

40

60

Communication volume per node (log scale)

Test accuracy [

%

]

1.5 GiB

2GiB

3GiB

68

69

70

71

Test accuracy [

%

]

68

70

72

0.5

1

20

40

60

80

Communication round (

×

10 3

)

Test accuracy [

%

]

0.5

1

1

2

3

Test loss

Fully connected

EL-Local

EL-Oracle

7-Regular static

8-U-EquiStatic

1GiB

10GiB

100GiB

1TiB

0

20

40

60

80

Communication volume per node (log scale)

Test accuracy [

%

]

50GiB

500GiB

1TiB

86.6

86.8

87

87.2

Test accuracy [

%

]

1

1.2

1.4

86

87

baseline, we established a Fully connected topology. Additionally, we examined EL (EL-Oracle and EL-Local) in comparison to 7-Regular static and 8-U-EquiStatic .

We observed that the results were not particularly exciting (Figure 7), primarily due to the nature of the data distribution, which leaves limited room for improvement across the different topologies. In terms of accuracy, the Fully connected topology achieved roughly 2 % better accuracy than the sparser topologies. EL-Oracle displayed a marginal advantage over the other approaches. However, the convergence behavior for EL-Local, 7-Regular static , and EL-Oracle was nearly overlapping, both in terms of the number of rounds and communication usage. On the other hand, the 8-U-EquiStatic approach lagged behind, exhibiting 1 % lower accuracy.

Although significant improvements were not observed in IID data settings, it is worth noting that EL did not hinder convergence. In more realistic non-IID data distributions, EL demonstrated improved accuracy and reduced the network utilization, as shown in Figure 4, Section 4.

C.4 Performance of EL and baselines on FEMNIST

As an additional dataset, we employed FEMNIST [7] to evaluate EL against D-PSGD over other topologies. Figure 8 shows the convergence plots for the baselines against both EL-Oracle and EL-Local. We observe the same trend with Fully connected topology performing the best in terms of achieved top-1 accuracy. EL-Oracle outperforms EL-Local and both achieve higher top-1 accuracy compared to the static topologies 7-Regular static and 8-U-EquiStatic . The improvement in accuracy with EL ( 0.2 % between EL-Oracle and 7-Regular static ) does not look as evident as with the CIFAR-10 dataset because the room for improvement (difference between 7-Regular static and Fully connected ) is quite small. This can be attributed to the homogeneity of the FEMNIST dataset and the complexity of the digit recognition task.

D Notes on Network Connectivity and EL Performance

The implementation of our decentralized scheme is built around the condition that all nodes can communicate with each other. This is similar to the assumptions of the EquiTopo topologies, a

competitor baseline [53]. However, we argue that the connectivity requirement is a bit more lenient, allowing our EL approach to function in a wide range of practical scenarios. In data center settings, it is common to train on clusters of highly interconnected GPUs, and all-to-all communication should be achievable in these settings. In edge settings, e . g ., a network of mobile devices collaboratively training a model while keeping private datasets, the communication barrier might appear more substantial. Nonetheless, Internet networks are generally well-connected, which mitigates this concern. More importantly, from a practical point of view, even if pairwise communications encounter some barriers, the decentralized and randomized nature of EL-Oracle and EL-Local should still allow for effective model learning and convergence. The occasional lack of communication between specific nodes should not significantly impact the algorithm's performance, as model updates are still propagated through other communicating nodes, as long as the network is not partitioned.

EL is most useful in scenarios where every pair of nodes can communicate, but the total communication budget is limited. Our randomized communication scheme allows for efficient use of the limited resources while ensuring faster model convergence than conventional decentralized learning approaches.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the benefits of using randomized communication topologies for decentralized optimization of non-convex functions. Critically, the communication protocols studied are not based on picking/sampling a communication graph that remains fixed throughout learning. Instead the authors study the case where the nodes randomize their communication patterns with the rest of the nodes in each round. Based on this premise they propose two algorithms, EL Oracle and EL local and study their convergence rate. The prosed algorithms are shown to converge asymptotically faster than state of the art. Experiments verify the theoretical advantage empirically.\n\nSTRENGTHS:\nThe paper communicates clearly and precisely its contribution. Despite extensive work on decentralized algorithms, including randomized and fixed topologies that can vary over time, to the best of my knowledge the proposed algorithms are novel. Importantly, the authors note that time varying randomized topologies have not been shown to have an advantage over their static counterparts.\n\nWEAKNESSES:\nI am not 100% sure which quantity is bounded in (3) and if it is supposed to be comparable to the rates provided in Theorem 1. The paragraphs after (3) list various ways EL Oracle and Local improve over (3) but I cannot say I follow them. For example, the first term in EL Oracle in Theorem 1 and (3) are identical but the comments mention an advantage in the first term. Clarifying this would help.\n\nIn its current state, the paper is not adequately explaining which part of the analysis unlocks the potential of non-static randomized communication graphs. I think highlighting which parts of the analysis are critical would help. Intuitively, I would expect that improving communication would have an effect similar to decreasing the variance $\\sigma$ or improving $p$ but it seems like this is not true.\n\nQUESTIONS:\nAnother thing that was unclear to me is how is this improvement affected by the fact that we are averaging iterates $x$ instead of gradients. Does the advantage persist in this case? In the case where we are averaging gradients I have the intuition that it would be harder to show an advantage but I am not sure.\n\nLIMITATIONS:\nNo limitations to address.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a decentralized learning algorithm based on random communication, i.e., each node sends its model to a random set (with a fixed size) of other nodes at each round. This paper theoretically shows the superiority of random communication in terms of transient iterations over other decentralized algorithms, which is further validated by experiments.\n\nSTRENGTHS:\n1. This paper provides a simple yet effective scheme of communication for decentralized learning.\n\n2. This paper is technically sound. The technique developed for convergence analysis is interesting.\n\nWEAKNESSES:\n1. In the proposed algorithm, each node is required to send message to a fixed number $k$ of random neighbors at each round. In practice, this may only be applied to networks with high connectivity (e.g., fully connected), and other application scenarios need to be motivated. In addition, the theoretical superiority of convergence directly depends on $k$, so it might be unfair comparing with other algorithms which conventionally works on arbitrary connected graphs. Can the restriction on $k$ or uniformly random selection be relaxed a little bit, say, allow to communicate with varying number of neighbors?\n\n2. Although the authors conduct an experiment to explain the imbalanced load on different nodes, I still have a concern that the balanced pattern reported in this paper may rely on the uniformly random selection. Such an assumption may not be aligned with the real cases, e.g., the degrees of different nodes vary, then some nodes may have higher load than others under random selection. This problem may arise from the non-doubly stochasticity of the mixing matrices. Could other methods handling asymmetric communication, such as push-sum, be helpful for load balancing?\n\n3. Figure 4 shows that EL outperforms the baseline algorithm throughout the learning process, which seems inconsistent with the theoretical statement that the superiority of convergence lies in transient iterations, i.e., the early stage. As the learning proceeds, the convergence will be dominated by the first term. So I suppose the test accuracy of the various methods may get closer with large $T$?\n\nQUESTIONS:\nSee the weaknesses.\n\nLIMITATIONS:\nLimitations have been properly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers Epidemic Learning, a framework for distributed optimization where each node in a network pushes gradient-descent updates to a uniform random subset of $s$ nodes in the network. Theoretical bounds on the rate of convergence are derived as well as the number of ``transient iterations,\" showing that this scheme improves upon existing fixed and randomized topologies. These results are supplemented by some empirical evaluations that show gains in communication and iterations to convergence compared to complete or fixed sparse topologies.\n\nSTRENGTHS:\nThe technical results generalize existing convergence rates of complete graphs (i.e. centralized SGD). The analysis is fairly clean and logical, and the paper is fairly well-written. This work also provides some experimental components to support their findings.\n\nWEAKNESSES:\nThe practical benefits of this kind of decentralized optimization could be better contextualized (see below). The analysis, while clean, does not seem particularly conceptually surprising.\n\nQUESTIONS:\n---As I am not really an expert on these kinds of methods, I'd defer to the other reviewers as to the novelty of this work's approach. I'd be happy to revise my score in light of any such discussions.\n\n---Is there a succinct summary of where the convergence benefits arise compared to, say, fast mixing but fixed network topologies? Intuitively, it seems to be because the convergence is dominated by the second-largest eigenspace, while random averaging avoids any particular bad eigenspace in expectation.\n\n---To elaborate on the above ``Weaknesses,\" one aspect that was not clear to me (and perhaps could be elaborated more) is the relevant tradeoffs in implementing such a decentralized scheme. More concretely, in what scenarios should this kind of randomized scheme that nonetheless must permit communication between any pair of nodes be employed? If the communication architecture is itself a limiting bottleneck, then this scheme cannot work due to the need for pairwise communications. Is the proposed use case meant primarily as a way to speed up/reduce optimization problems that already employ complete topologies?\n\n---While I am not a systems expert either, are there other tradeoffs one must consider with respect to routing/scheduling these randomized communications? For instance, it seems fairly intuitive how to easily route messages in, say, a ring topology --- when each node now must route to $s$ random nodes and received messages from $\\Omega(s)$ random nodes, is there any ``scheduling'' overhead in practice?\n\n---While I did not get the chance to carefully check the analysis in the Supplementary Material, the analysis seemed quite clean and logical, i.e. write out the various expected quantities in the natural way and derive relatively standard-looking GD-type recurrences. Conversely, this could mean that the analysis itself may not lead to new technical insights for others in the community.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a decentralized learning algorithm in which each node\nupdates its model from a set of s random nodes in a system with n > s\nnodes. The authors provide a theoretical analysis of the convergence speed\nand the number of transient iterations, i.e., the number of rounds required\nto reach linear speedup stage. Experiments are performed for the CIFAR-10\ndataset comparing the proposed two methods EL-Oracle and EL-Local with a\nnumber of baselines that are static topologies. The comparison metrics are\naccuracy, test loss, and communication volume with increasing number of\ncommunication rounds.\n\nSTRENGTHS:\nA simple yet effective solution for decentralized learning. It is easily\nimplementable and can be easily adopted for any decentralized learning\ntask.\n\nThe theoretical analysis has some novel approach and is quite technical.\n\nThe paper is generally well written.\n\nWEAKNESSES:\nTitle: Epidemic learning is a very confusing title. The reviewer is of the\nopinion that it is an overloaded term. For example, there are many\ninference problems in the context of contagions (like infectious diseases)\nwhere the objective is to estimate information about an outbreak. Also, the\nreviewer is of the opinion that this title might fail to gain the attention\nof readers from the decentralized learning community. Thirdly, it is very\nshort and non-informative.\n\nTheorem 1 mentions step size \\gamma, which does not feature in any\nexpressions that appear in the statement. The expressions for \\gamma in\nequations (4) and (5) should be included in the statement.\n\nRemark 1: The authors mention that they \"provide convergence rate directly\nfor the local models\". But wouldn't the expected convergence rate of the\nentire global averaged model depend on the maximum of (worst case) the\nconvergence rates among the local models? This is not addressed.\n\nExperiments section is weak: All the inferences are based on a single\nlearning problem on a single dataset. The gain in accuracy is not\nsignificant, but again the reviewer agrees that there are benefits such as\nreduction in communication rounds and volume. Yet, the gain in accuracy is\nlow enough to wonder what would happen if the same was used for, say a\nclassification task on ImageNet.\n\nThe performance for varying s: The authors only use s=7. They mention that\nthis is consistent with baselines. However, they could have performed a\nseparate experiment to study how their method performs under different\nmetrics for varying s. Suppose, they get very good performance for s=5\nitself, wouldn't it be an interesting statement?\n\nDisconnect between theory and experiments: \n\n1. The authors do not make an attempt to connect theoretical results to\nexperiments. For example, they mention that the number of transient\niterations is an inverse function of s. There are no experiments (like the\none suggested above) to test the tightness of the theoretical bounds. A\ntable analogous to Table 1 for experiments would be very helpful.\n\n2. Step size: From a theoretical perspective, the step size \\gamma in\nequations (4) and (5) depends on s. However, in the experiments (Appendix\nD), it is obtained by only running the fully-connected topology. Some\nanalysis of how convergence depends on s and \\gamma (as a function of\ns) would be useful.\n\n7-regular static: The authors consider only one instance of 7-regular\ntopology. There are many possibilities for a 96-node graph. The authors\ncould have considered 5-10 instances of 7-regular graphs and provided\nresults averaged over these topologies.\n\nQUESTIONS:\nThere are several doubts raised in the Weaknesses section.\n\nLIMITATIONS:\nThe authors do not bring up any limitations. No negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores decentralized learning algorithms with the aim of faster model convergence while comparable accuracy compared with conventional DL methods. The new proposed algorithm - epidemic learning (EL) - leverages a dynamically changing, randomized communication topology to train a machine learning model in DL environment. The paper provides theoretical analysis which shows that the EL algorithm surpasses the best-known static and randomized topologies in terms of convergence speed, w.r.t. two key properties: linear speed-up and transient iterations. The experimental results show that the proposed EL-oracle and EL-local achieve quicker convergence than baselines.\n\nSTRENGTHS:\nThis paper is studying a very interesting and important problem - DL algorithms. The paper is well written, which is easy to follow. The proposed EL algorithm is technically sound. The theoretical analysis and experimental results show its effectiveness compared to baselines.\n\nWEAKNESSES:\nA major concern is that the proposed EL algorithm is not significantly different with semi-dynamic and time-varying and randomized topologies (they are introduced in related work section). Especially, gossip learning (GL) is like a special case of EL-local with s = 1. \n\nA minor comment: Figure 3 should be improved by giving label for x-axis. Also there is no full words for the first abbr \"CDF\".\n\nQUESTIONS:\n1. Compared with semi-dynamic topologies, and time-varying and randomized topologies, what's the signaficance of EL? Have any of the previous works provided theoretical analysis on convergence rate? \n2. What does \"the convergence of GL on non-convex problems remains uncertain\" mean? \n3. Is GL a special case of EL-local with s=1? In the experiment, have you done sensitivity analysis on s values?\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}