SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Martijn de Vos* Sadegh Farhadkhani† Rachid Guerraoui Anne-Marie Kermarrec Rafael Pires Rishi Sharma
We present Epidemic Learning (EL), a simple yet powerful decentralized learning (DL) algorithm that leverages changing communication topologies to achieve faster model convergence compared to conventional DL approaches. At each round of EL, each node sends its model updates to a random sample of  $s$  other nodes (in a system of  $n$  nodes). We provide an extensive theoretical analysis of EL, demonstrating that its changing topology culminates in superior convergence properties compared to the state-of-the-art (static and dynamic) topologies. Considering smooth nonconvex loss functions, the number of transient iterations for EL, i.e., the rounds required to achieve asymptotic linear speedup, is in  $\mathcal{O}(n^3 / s^2)$  which outperforms the best-known bound  $\mathcal{O}(n^3)$  by a factor of  $s^2$ , indicating the benefit of randomized communication for DL. We empirically evaluate EL in a 96-node network and compare its performance with state-of-the-art DL approaches. Our results illustrate that EL converges up to  $1.7 \times$  quicker than baseline DL algorithms and attains  $2.2\%$  higher accuracy for the same communication volume.
In Decentralized Learning (DL), multiple machines (or nodes) collaboratively train a machine learning model without any central server [32, 37, 42]. Periodically, each node updates the model using its local data, sends its model updates to other nodes, and averages the received model updates, all without sharing raw data. Compared to centralized approaches [28], DL circumvents the need for centralized control, ensures scalability [21, 32], and avoids imposing substantial communication costs on a central server [58]. However, DL comes with its own challenges. The exchange of model updates with all nodes can become prohibitively expensive in terms of communication costs as the network size grows [25]. For this reason, nodes in DL algorithms usually exchange model updates with only a small number of other nodes in a particular round, i.e., they perform partial averaging instead of an All-Reduce (network-wide) averaging of local model updates [57].
A key element of DL algorithms is the communication topology, governing how model updates are exchanged between nodes. The properties of the communication topology are critical for the performance of DL approaches as it directly influences the speed of convergence [27, 54, 56, 52]. The seminal decentralized parallel stochastic gradient descent (D-PSGD) algorithm and many of its proposed variants rely on a static topology, i.e., each node exchanges its model with a set of neighboring nodes that remain fixed throughout the training process [32]. More recent approaches study changing topologies, i.e., topologies that change during training, with notable examples being time-varying graphs [24, 36, 51], one-peer exponential graphs [57], EquiTopo [53], and Gossip Learning [16, 17, 20]. We discuss these works in more detail in Section 5.
†Corresponding author <sadegh.farhadkhani@epfl.ch>.
This paper investigates the benefits of randomized communication for DL. Randomized communication, extensively studied in distributed computing, has been proven to enhance the performance of fundamental algorithms, including consensus and data dissemination protocols [22, 5, 8]. In the case of DL, randomized communication can reduce the convergence time and therefore communication overhead [11]. In this work, we specifically consider the setting where each node communicates with a random subset of other nodes that changes at each round, as with epidemic interaction schemes [12, 38]. We also focus on the scenario where data is unevenly distributed amongst nodes, i.e., non independent and identically distributed (non-IID) settings, a common occurrence in DL [18].
To illustrate the potential of randomized communication, we empirically compare model convergence in Figure 1 for static  $s$ -regular topologies (referred to as Static-Topo) and randomized topologies (referred to as Rand. -Topo (EL), our work) in 96-node networks on the CIFAR-10 learning task. As particular static topologies can lead to sub-optimal convergence, we also experiment with the setting in which nodes are stuck in an unlucky static  $s$ -regular topology (referred to as Static-Topo-Unlucky). This unlucky topology consists of two network partitions, connected by just two edges. Figure 1 reveals that dynamic topologies converge quicker than static topologies. After 3000 communication rounds, Rand. -Topo (EL, our work) achieves  $65.3\%$  top-1 test accuracy, compared to  $63.4\%$  and  $61.8\%$  for Static-Random and Static-Unlucky, respectively. Additional experiments can be found in Section 4, and their setup is elaborated in Appendix C.
Figure 1: Randomized topologies can converge quicker than static ones.
Our contributions Our paper makes the following four contributions:
- We formulate, design, analyze, and experimentally evaluate Epidemic Learning (EL), a novel DL algorithm in which nodes collaboratively train a machine learning model using a dynamically changing, randomized communication topology. More specifically, in EL, at each round, each node sends its model update to a random sample of  $s$  other nodes (out of  $n$  total nodes). This process results in a randomized topology that changes every round.
- We first analyze an EL variant, named EL-Oracle, where the union of the random samples by all nodes forms an  $s$ -regular random graph in each round. EL-Oracle ensures a perfectly balanced communication load among the participating nodes as each node sends and receives exactly  $s$  model updates every round. Nevertheless, achieving an  $s$ -regular graph necessitates coordination among the nodes, which is not ideal in a decentralized setting. To address this challenge, we also analyze another EL variant, named EL-Local. In EL-Local, each node independently and locally draws a uniformly random sample of  $s$  other nodes at each round and sends its model update to these nodes. We demonstrate that EL-Local enjoys a comparable convergence guarantee as EL-Oracle without requiring any coordination among the nodes and in a fully decentralized manner.
- Our theoretical analysis in Section 3 shows that EL surpasses the best-known static and randomized topologies in terms of convergence speed. More precisely, we prove that EL converges with the rate  $\mathcal{O}\left(1 / \sqrt{nT} +\frac{1}{3\sqrt{sT^2}} +\frac{1}{T}\right)$ , where  $T$  is the number of learning rounds. Similar to most state-of-the-art algorithms for decentralized optimization [30, 24] and centralized stochastic gradient descent (SGD) (e.g., with a parameter server) [29], our rate asymptotically achieves linear speedup, i.e., when  $T$  is sufficiently large, the first term in the convergence rate  $\mathcal{O}\left(1 / \sqrt{nT}\right)$  becomes dominant and improves with respect to the number of nodes.
Even though linear speedup is a very desirable property, DL algorithms often require many more rounds to reach linear speedup compared to centralized SGD due to the additional error (the second term in the above convergence rate) arising from partial averaging of the local updates. To capture this phenomenon and to compare different decentralized learning algorithms, previous works [10, 50, 53, 57] adopt the concept of transient iterations which are the number of rounds before a decentralized algorithm reaches its linear speedup stage, i.e., when  $T$  is relatively small such that the second term of the convergence rate dominates the first term.
We derive that EL requires  $\mathcal{O}(n^3 / s^2)$  transient iterations, which improves upon the best known bound by a factor of  $s^2$ . We also show this result in Table 1. We note that while EL matches the state-of-
Table 1: Comparison of EL with state-of-the-art DL approaches (grouped by topology family). We compare EL-Oracle and EL-Local to ring, torus, Erdős-Rényi, exponential and EquiTopo topologies.
Method	Per-Iter Out Msgs.	Transient Iterations	Topology	Communication
Ring [23]	2	O(n11)	static	undirected
Torus [23]	4	O(n7)	static	undirected
E.-R. Rand [45]	O(log n)	O(n3)	static	undirected
Static Exp. [57]	log n	O(n3 log4 n)	static	directed
One-Peer Exp. [57]	1	O(n3 log4 n)	semi-dynamic1	directed
D-EquiStatic [53]	log n	O(n3)	static	directed
U-EquiStatic [53]	log n	O(n3)	static	undirected
OD-EquiDyn [53]	1	O(n3)	semi-dynamic1	directed
OU-EquiDyn [53]	1	O(n3)	semi-dynamic1	undirected
EL-Oracle (ours)	s	O(n3/s2)	rand.-dynamic2	undirected
EL-Local (ours)	s	O(n3/s2)	rand.-dynamic2	directed
<sup>1</sup> Semi-dynamic topologies remain fixed throughout the learning process but nodes select subsets of adjacent (neighboring) nodes each round to communicate with.
2 In a randomized-dynamic topology, the topology is replaced each round.
the-art bounds when  $s \in \mathcal{O}(1)$ , it offers additional flexibility over other methods through parameter  $s$  that provably improves the theoretical convergence speed depending on the communication capabilities of the nodes. For instance, when  $s \in \mathcal{O}(\log n)$  as in Erdős-Rényi and EquiStatic topologies, the number of transient iterations for EL reduces to  $\mathcal{O}\left(n^{3} / \log^{2}n\right)$ , outperforming other methods. This improvement comes from the fact that the second term in our convergence rate is superior to the corresponding term  $\mathcal{O}\left(1 / \sqrt[3]{p^2 T^2}\right)$  in the rate of D-PSGD, where  $p \in (0,1]$  is the spectral gap of the mixing matrix. We expound more on this in Section 3.
- We present in Section 4 our experimental findings. Using two standard image classification datasets, we compare EL-Oracle and EL-Local against static regular graphs and the state-of-the-art EquiTopo topologies. We find that EL-Oracle and EL-Local converge faster than the baselines and save up to  $1.7 \times$  communication volume to reach the highest accuracy of the most competitive baseline.
2 Epidemic Learning
In this section, we first formally define the decentralized optimization problem. Then we outline our EL algorithm and its variants in Section 2.2.
We consider a system of  $n$  nodes  $[n] := \{1, \dots, n\}$  where the nodes can communicate by sending messages. Similar to existing work in this domain [53], we consider settings in which a node can communicate with all other nodes. The implications of this assumption are further discussed in Appendix D. Consider a data space  $\mathcal{Z}$  and a loss function  $f: \mathbb{R}^d \times \mathcal{Z} \to \mathbb{R}$ . Given a parameter  $x \in \mathbb{R}^d$ , a data point  $\xi \in \mathcal{Z}$  incurs a loss of value  $f(x, \xi)$ . Each node  $i \in [n]$  has a data distribution  $\mathcal{D}^{(i)}$  over  $\mathcal{Z}$ , which may differ from the data distributions of other nodes. We define the local loss function of  $i$  over distribution  $\mathcal{D}^{(i)}$  as  $f^{(i)}(x) := \mathbb{E}_{\xi \sim \mathcal{D}^{(i)}}[f(x, \xi)]$ . The goal is to collaboratively minimize the global average loss by solving the following optimization problem:
$$
\min  _ {x \in \mathbb {R} ^ {d}} \left[ F (x) := \frac {1}{n} \sum_ {i \in [ n ]} f ^ {(i)} (x) \right]. \tag {1}
$$
Figure 2: EL-Oracle (left) and EL-Local (right), from the perspective of node  $j$ , with  $s = 3$  and for two rounds. We show both outgoing model updates (solid line) and incoming ones (dashed line).
2.2 Description of EL
We outline EL, executed by node  $i$ , in Algorithm 1. We define the initial model of node  $i$  as  $x_0^{(i)}$  and a step-size  $\gamma$  used during the local model update. The EL algorithm runs for  $T$  rounds. Each round consists of two phases: a local update phase (line 3-5) in which the local model is updated using the local dataset of node  $i$ , and a random communication phase (line 6-9) in which model updates are sent to other nodes chosen randomly. In the local update phase, node  $i$  samples a data point  $\xi_t^{(i)}$  from its local data distribution  $\mathcal{D}^{(i)}$  (line 3), computes the stochastic gradient  $g_t^{(i)}$  (line 4) and partially updates its local model  $x_{t+1/2}^{(i)}$  using step-size  $\gamma$  and gradient  $g_t^{(i)}$  (line 5).
The random communication phase follows, where node  $i$  first selects  $s$  of other nodes from the set of all nodes excluding itself:  $[n] \setminus \{i\}$  (line 6). This sampling step is the innovative element of EL, and we present two variants later. It then sends its recently updated local model  $x_{t+1/2}^{(i)}$  to the selected nodes and waits for model updates from other nodes. Subsequently, each node  $i$  updates its model based on the models it receives from other nodes according to Equation (2). The set of nodes that send their models to node  $i$  is denoted by  $S_t^{(i)}$ . The new model for node  $i$  is computed as a weighted average of the models received from the other nodes and the local model of node  $i$ , where the weights are inversely proportional to the number of models received plus one.
$$
x _ {t + 1} ^ {(i)} := \frac {1}{\left| \mathcal {S} _ {t} ^ {(i)} \right| + 1} \left(x _ {t + 1 / 2} ^ {(i)} + \sum_ {j \in \mathcal {S} _ {t} ^ {(i)}} x _ {t + 1 / 2} ^ {(j)}\right). \tag {2}
$$
We now describe two approaches to sample  $s$  other nodes (line 6), namely EL-Oracle and EL-Local:
EL-Oracle With EL-Oracle, the union of selected communication links forms a  $s$ -regular topology in which every pair of nodes has an equal probability of being neighbors. Moreover, if node  $i$  samples node  $j$ ,  $j$  will also sample  $i$  (communication is undirected). Figure 2 (left) depicts EL-Oracle sampling from the perspective of node  $j$  in two consecutive iterations. One possible way to generate such a dynamic graph is by generating an  $s$ -regular structure and then distributing a random permutation of
Algorithm 1 Epidemic Learning as executed by a node  $i$
1: Require: Initial model  $x_0^{(i)} = x_0 \in \mathbb{R}^d$ , number of rounds  $T$ , step-size  $\gamma$ , sample size  $s$ .
2: for  $t = 0,\dots ,T - 1$  do  $\triangleright$  Line 3-5: Local training phase
3: Randomly sample a data point  $\xi_t^{(i)}$  from the local data distribution  $\mathcal{D}^{(i)}$
4: Compute the stochastic gradient  $g_{t}^{(i)}\coloneqq \nabla f(x_{t}^{(i)},\xi_{t}^{(i)})$
5: Partially update local model  $x_{t + 1 / 2}^{(i)}\coloneqq x_t^{(i)} - \gamma g_t^{(i)}$ $\triangleright$  Line 6-9: Random communication phase
6: Sample  $s$  other nodes from  $[n] \setminus \{i\}$  using EL-Oracle or EL-Local
7: Send  $x_{t + 1 / 2}^{(i)}$  to the selected nodes
8: Wait for the set of updated models  $S_{t}^{(i)}$ $\triangleright S_t^{(i)}$  is the set of received models by node  $i$  in round  $t$
9: Update  $x_{t + 1}^{(i)}$  to the average of available updated models according to (2)
10: end for
nodes at each round. Our implementation (see Section 4) uses a central coordinator to randomize and synchronize the communication topology each round.
EL-Local Constructing the  $s$ -regular topology in EL-Oracle every round can be challenging in a fully decentralized manner as it requires coordination amongst nodes to ensure all nodes have precisely  $s$  incoming and outgoing edges. This motivates us to introduce EL-Local, a sampling approach where each node  $i$  locally and independently samples  $s$  other nodes and sends its model update to them, without these  $s$  nodes necessarily sending their model back to  $i$ . The union of selected nodes now forms a  $s$ -out topology. Figure 2 (right) depicts EL-Local sampling from the perspective of node  $j$  in two consecutive iterations. In practice, EL-Local can be realized either by exchanging peer information before starting the learning process or using a decentralized peer-sampling service that provides nodes with (partial) views on the network [20, 41, 55]. While both the topology construction in EL-Oracle as well as peer sampling in EL-Local add some communication and computation overhead, this overhead is minimal compared to the resources used for model exchange and training.
Even though each node sends  $s$  messages for both EL-Oracle and EL-Local, in EL-Local, different nodes may receive different numbers of messages in each training round as each node selects the set of its out-neighbors locally and independent from other nodes. While this might cause an imbalance in the load on individual nodes, we argue that this does not pose a significant issue in practice. To motivate this statement, we run an experiment with  $n = 100$ , 1000 and 10000 and for each value of  $n$  set  $s = \lceil \log_2(n) \rceil$ . We simulate up to 5000 rounds for each configuration. We show in Figure 3 a CDF with the number of incoming models each round and observe that the distribution of the number of models received by each node is very light-tailed. In a 10000 node network (and  $s = 13$ ), nodes receive less than 22 models in  $99\%$  of all rounds. As such, it is improbable that a node receives
a disproportionately large number of models in a given round. In practice, we can alleviate this imbalance issue by adopting a threshold value  $k$  on the number of models processed by each node in a particular round and ignoring incoming models after having received  $k$  models already. The sender node can then retry model exchange with another random node that is less occupied.
Figure 3: The distribution of incoming models, for different values of  $n$  and  $s$ .
In this section, we first present our main theoretical result demonstrating the finite-time convergence of EL. We then compare our result with the convergence rate of D-PSGD on different topologies.
3.1 Convergence of EL
In our analysis, we consider the class of smooth loss functions, and we assume that the variance of the noise of stochastic gradients is bounded. We use of the following assumptions that are classical to the analysis of stochastic first-order methods and hold for many learning problems [4, 14].
Assumption 1 (Smoothness). For all  $i \in [n]$ , the function  $f^{(i)}: \mathbb{R}^d \to \mathbb{R}$  is differentiable and there exists  $L < \infty$ , such that for all  $x, y \in \mathbb{R}^d$ ,
$$
\left\| \nabla f ^ {(i)} (y) - \nabla f ^ {(i)} (x) \right\| \leq L \| y - x \|.
$$
Assumption 2 (Bounded stochastic noise). There exists  $\sigma < \infty$  such that for all  $i \in [n]$ , and  $x \in \mathbb{R}^d$ ,
$$
\mathbb {E} _ {\xi \sim \mathcal {D} ^ {(i)}} \left[ \left\| \nabla f (x, \xi) - f ^ {(i)} (x) \right\| ^ {2} \right] \leq \sigma^ {2}.
$$
Moreover, we assume that the heterogeneity among the local loss functions measured by the average distance between the local gradients is bounded.
Assumption 3 (Bounded heterogeneity). There exists  $\mathcal{H} < \infty$ , such that for all  $x \in \mathbb{R}^d$
$$
\frac {1}{n} \sum_ {i \in [ n ]} \left\| \nabla f ^ {(i)} (x) - \nabla F (x) \right\| ^ {2} \leq \mathcal {H} ^ {2}.
$$
We note that this assumption is standard in heterogeneous (a.k.a. non-i.i.d) settings, i.e., when nodes have different data distributions [32, 57]. In particular,  $\mathcal{H}$  can be bounded based on the closeness of the underlying local data distributions [13]. We now present our main theorem.
Theorem 1. Consider Algorithm 1. Suppose that assumptions 1, 2 and 3 hold true. Let  $\Delta_0$  be a real value such that  $F(x_0) - \min_{x\in \mathbb{R}^d}F(x)\leq \Delta_0$ . Then, for any  $T\geq 1$ ,  $n\geq 2$ , and  $s\geq 1$ : a) For EL-Oracle, setting
$$
\gamma \in \Theta \left(\min \left\{\sqrt {\frac {n \Delta_ {0}}{T L \sigma^ {2}}}, \sqrt [ 3 ]{\frac {\Delta_ {0}}{T \alpha_ {s} L ^ {2} (\sigma^ {2} + \mathcal {H} ^ {2})}}, \frac {1}{L} \right\}\right),
$$
$$
\frac {1}{n} \sum_ {i \in [ n ]} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left\| \nabla F (x _ {t} ^ {(i)}) \right\| ^ {2} \right] \in \mathcal {O} \left(\sqrt {\frac {L \Delta_ {0} \sigma^ {2}}{n T}} + \sqrt [ 3 ]{\frac {\alpha_ {s} L ^ {2} \Delta_ {0} ^ {2} (\sigma^ {2} + \mathcal {H} ^ {2})}{T ^ {2}}} + \frac {L \Delta_ {0}}{T}\right),
$$
$$
\alpha_ {s} := \frac {1}{s + 1} \left(1 - \frac {s}{n - 1}\right) \in \mathcal {O} (\frac {1}{s}).
$$
b) For EL-Local, setting
$$
\gamma \in \Theta \left(\min \left\{\sqrt {\frac {n \Delta_ {0}}{T (\sigma^ {2} + \beta_ {s} \mathcal {H} ^ {2}) L}}, \sqrt [ 3 ]{\frac {\Delta_ {0}}{T \beta_ {s} L ^ {2} (\sigma^ {2} + \mathcal {H} ^ {2})}}, \frac {1}{L} \right\}\right),
$$
$$
\frac {1}{n} \sum_ {i \in [ n ]} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left\| \nabla F (x _ {t} ^ {(i)}) \right\| ^ {2} \right] \in \mathcal {O} \left(\sqrt {\frac {L \Delta_ {0} (\sigma^ {2} + \beta_ {s} \mathcal {H} ^ {2})}{n T}} + \sqrt [ 3 ]{\frac {\beta_ {s} L ^ {2} \Delta_ {0} ^ {2} (\sigma^ {2} + \mathcal {H} ^ {2})}{T ^ {2}}} + \frac {L \Delta_ {0}}{T}\right),
$$
$$
\beta_ {s} := \frac {1}{s} \left(1 - \left(1 - \frac {s}{n - 1}\right) ^ {n}\right) - \frac {1}{n - 1} \in \mathcal {O} (\frac {1}{s}).
$$
To check the tightness of this result, we consider the special case when  $s = n - 1$ . Then, by Theorem 1, we have  $\alpha_{s} = \beta_{s} = 0$ , and thus both of the convergence rates become  $\mathcal{O}\left(\sqrt{L\Delta_0\sigma^2 / nT} + L\Delta_0 / T\right)$  which is the same as the convergence rate of (centralized) SGD for non-convex loss functions [14]. This is expected as, in this case, every node sends its updated model to all other nodes, corresponding to all-to-all communication in a fully-connected topology and thus perfectly averaging the stochastic gradients without any drift between the local models.
The proof of Theorem 1 is given in Appendix A, where we obtain a tighter convergence rate than existing methods. It is important to note that as the mixing matrix of a regular graph is doubly stochastic, for EL-Oracle, one can use the general analysis of D-PSGD with (time-varying) doubly stochastic matrices [24, 30] to obtain a convergence guarantee. However, the obtained rate would not be as tight and would not capture the  $\mathcal{O}(1 / \sqrt[3]{s})$  improvement in the second term, which is the main advantage of randomization (see Section 3.2). Additionally, it is unclear how these analyses can be generalized to the case where the mixing matrix is not doubly stochastic, which is the case for our EL-Local algorithm. Furthermore, another line of work [1, 43, 44] provides convergence guarantees for decentralized optimization algorithms based on the PushSum algorithm [22], with communicating the mixing weights. It may be possible to leverage this proof technique to prove the convergence of EL-Local. However, this approach yields sub-optimal dimension-dependent convergence guarantees (e.g., see parameter  $C$  in Lemma 3 of [1]) and does not capture the benefit of randomized communication.
Remark 1. Most prior work [24, 32] provides the convergence guarantee on the average of the local models  $\bar{x}_t = \frac{1}{n}\sum_{i\in [n]}x_t^{(i)}$ . However, as nodes cannot access the global averaged model, we provide the convergence rate directly on the local models. Nonetheless, the same convergence guarantee as Theorem 1 also holds for the global averaged model.
3.2 Discussion and comparison to prior results
To provide context for the above result, we note that the convergence rate of decentralized SGD with non-convex loss functions and a doubly stochastic mixing matrix [24, 30] is
$$
\mathcal {O} \left(\sqrt {\frac {L \Delta_ {0} \sigma^ {2}}{n T}} + \sqrt [ 3 ]{\frac {L ^ {2} \Delta_ {0} ^ {2} (p \sigma^ {2} + \mathcal {H} ^ {2})}{p ^ {2} T ^ {2}}} + \frac {L \Delta_ {0}}{p T}}\right), \tag {3}
$$
where  $p \in (0,1]$  is the spectral gap of the mixing matrix and  $1/p$  is bounded by  $\mathcal{O}(n^2)$  for ring [23],  $\mathcal{O}(n)$  for torus [23],  $\mathcal{O}(1)$  for Erdős–Rényi random graph [45],  $\mathcal{O}(\log n)$  for exponential graph [57], and  $\mathcal{O}(1)$  for EquiTopo [53]. We now compare the convergence of EL against other topologies across two key properties: linear speed-up and transient iterations.