SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Linear speed-up Both of our convergence rates preserve a linear speed-up of  $\mathcal{O}(1 / \sqrt{nT})$  in the first term. For EL-Oracle, this term is the same as (3). However, in the case of EL-Local, in addition to the stochastic noise  $\sigma$ , this term also depends on the heterogeneity parameter  $\mathcal{H}$  that vanishes when increasing the sample size  $s$ . This comes from the fact that, unlike EL-Oracle, the communication phase of EL-Local does not preserve the exact average of the local models (i.e.,  $\sum_{i\in [n]}x_{t + 1}^{(i)}\neq \sum_{i\in [n]}x_{t + 1 / 2}^{(i)})$ , and it only preserves the average in expectation. This adds an error term to the rate of EL-Local. However, as the update vector remains an unbiased estimate of the average gradient, this additional term does not violate the linear speed-up property. Our analysis suggests that setting  $s\approx \frac{\mathcal{H}^2}{\sigma^2}$  can help mitigate the effect of heterogeneity on the convergence of EL-Local. Intuitively, more data heterogeneity leads to more disagreement between the nodes, which requires more communication rounds to converge.
Transient iterations Our convergence rates offer superior second and third terms compared to those in (3). This is because first,  $p$  can take very small values, particularly when the topology connectivity is low (e.g.,  $\frac{1}{p} \in \mathcal{O}(n^2)$  for a ring) and second, even when the underlying topology is well-connected and  $\frac{1}{p} \in \mathcal{O}(1)$ , such as in EquiTopo [53], the second term in our rates still outperforms the one in (3) by a factor of  $\sqrt[3]{s}$ . This improvement is reflected in the number of transient iterations before the linear speed-up stage, i.e., the number of rounds required for the first term of the convergence rate to dominate the second term [57]. In our rates, the number of transient iterations is in  $\mathcal{O}(n^3 / s^2)$ , whereas in (3), it is  $\mathcal{O}(n^3 / p^2)$  for the homogeneous case and  $\mathcal{O}(n^3 / p^4)$  for the heterogeneous case. We remark that  $p \in (0, 1]$ , but  $s \geq 1$  is an integer; therefore, when  $s \in \mathcal{O}(1)$  the number of transient iterations for EL matches the state-of-the-art bound. However, it can be provably improved by increasing  $s$  depending on the communication capabilities of the nodes, which adds more flexibility to EL with theoretical guarantees compared to other methods. For instance, for  $s \in \mathcal{O}(\log n)$  as in Erdős-Rényi and EquiStatic topologies, the number of transient iterations for EL becomes  $\mathcal{O}(n^3 / \log^2 n)$  which outperforms other methods (also see Table 1). Crucially, a key implication of this result is that our algorithm requires fewer rounds and, therefore, less communication to converge. We empirically show the savings in the communication of EL in Section 4.
We present here the empirical evaluation of EL and compare it with state-of-the-art DL baselines. We first describe the experimental setup and then show the performance of EL-Oracle and EL-Local.
Network setup and implementation We deploy 96 DL nodes for each experiment, interconnected according to the evaluated topologies. When experimenting with  $s$ -regular topologies, each node maintains a fixed degree of  $\lceil \log_2(n) \rceil$ , i.e., each node has 7 neighbors. For EL-Oracle we introduce a centralized coordinator (oracle) that generates a random 7-Regular topology at the start of each
Figure 4: Communication rounds vs. top-1 test accuracy and (left) and communication volume per node vs. test accuracy (right) for the CIFAR-10 dataset.
round and informs all nodes about their neighbors for the upcoming round. For EL-Local we make each node aware of all other nodes at the start of the experiment. To remain consistent with other baselines, we fix  $s = \lceil \log_2(n) \rceil = 7$  when experimenting with EL, i.e., each node sends model updates to 7 other nodes each round. Both EL-Oracle and EL-Local were implemented using the DecentralizePy framework [11] and Python  $3.8^3$ . For reproducibility, a uniform seed was employed for all pseudo-random generators within each node.
Baselines We compare the performance of EL-Oracle and EL-Local against three variants of D-PSGD. Our first baseline is a fully-connected topology (referred to as Fully connected), which presents itself as the upper bound for performance given its optimal convergence rate [1]. We also compare with a  $s$ -regular static topology, the non-changing counterpart of EL-Oracle (referred to as 7-Regular static). This topology is randomly generated at the start of each run according to the random seed, but is kept fixed during the learning. Finally, we compare EL against the communication-efficient topology U-EquiStatic [53]. Since U-EquiStatic topologies can only have even degrees, we generate U-EquiStatic topologies with a degree of 8 to ensure a fair comparison. We refer to this setting as 8-U-EquiStatic.
Learning task and partitioning We evaluate the baseline algorithms using the CIFAR-10 image classification dataset [26] and the FEMNIST dataset, the latter being part of the LEAF benchmark [7]. In this section we focus on the results for CIFAR-10 and present the results for FEMNIST in Appendix C.4. We employ a non-IID data partitioning using the Dirichlet distribution function [19], parameterized with  $\alpha = 0.1$ . We use a GN-LENET convolutional neural network [18]. Full details on our experimental setup and hyperparameter tuning can be found in Appendix C.1.
Metrics We measure the average top-1 test accuracy and test loss of the model on the test set in the CIFAR-10 learning task every 20 communication rounds. Furthermore, we present the average top-1 test accuracy against the cumulative outgoing communication per node in bytes. We also emphasize the number of communication rounds taken by EL to reach the best top-1 test accuracy of static 7-Regular topology. We run each experiment five times with different random seeds, and we present the average metrics with a  $95\%$  confidence interval.
4.2 EL against baselines
Figure 4 shows the performance of EL-Oracle and EL-Local against the baselines for the CIFAR-10 dataset. D-PSGD over a fully-connected topology achieves the highest accuracy, as expected, but incurs more than an order of magnitude of additional communication. EL-Oracle converges faster than its static counterparts of 7-Regular static and 8-U-EquiStatic. After 3000 communication rounds, EL-Oracle and EL-Local achieve up to  $2.2\%$  higher accuracy compared to 7-Regular static (the most competitive baseline). Moreover, EL-Oracle takes up to  $1.7\times$  fewer communication rounds and saves 2.9 GiB of communication to reach the best accuracy attained by 7-Regular static. Surprisingly, 8-U-EquiStatic shows worse performance compared to 7-Regular static. The plots further highlight that the less constrained EL-Local variant has a very competitive performance compared to EL-Oracle: there is negligible utility loss when sampling locally compared to generating a  $s$ -Regular graph every round. We provide additional observations and results in Appendix C.
<sup>3</sup>Source code can be found at https://github.com/sacs-epfl/decentralizepy/releases/tag/epidemic-neurips-2023.
Table 2: A summary of key experimental findings for the CIFAR-10 dataset.
Topology	Top-1 Test Accuracy (%)	Top-1 Test Loss	Communication to Target Accuracy (GiB)
Fully connected	68.67	0.98	30.54
7-Regular static	64.32	1.21	7.03
8-U-EquiStatic1	62.72	1.28	-
EL-Oracle	66.56	1.10	4.12
EL-Local	66.14	1.12	4.37
1 The 8-U-EquiStatic topology did not reach the  $64.32\%$  target accuracy.
Figure 5: The test accuracy of EL-Oracle and EL-Local (left) and communication volume per node of EL-Oracle (right) for different value of sample size  $s$  and a 7-Regular static topology.
We summarize our main experimental findings in Table 2, which outlines the highest achieved top-1 test accuracy, lowest top-1 test loss, and communication cost to a particular target accuracy for our evaluated baselines. This target accuracy is chosen as the best top-1 test accuracy achieved by the 7-Regular static topology (64.32%), and the communication cost to reach this target accuracy is presented for all the topologies in the right-most column of Table 2. In summary, EL-Oracle and EL-Local converge faster and to higher accuracies compared to 7-Regular static and 8-U-EquiStatic topologies, and require  $1.7 \times$  and  $1.6 \times$  less communication cost to reach the target accuracy, respectively.
4.3 Sensitivity Analysis of sample size  $s$
The sample size  $s$  determines the number of outgoing neighbours of each node at each round of EL-Oracle and EL-Local. We show the impact of this parameter on the test accuracy in the two left-most plots in Figure 5, for varying values of  $s$ , against the baseline of a 7-regular static graph. We chose the values of  $s$  as  $\lceil \frac{log_2(n)}{2} \rceil = 4$ , and  $\lceil 2\log_2(n) \rceil = 14$ , over and above  $\lceil \log_2(n) \rceil = 7$ . We observe that, as expected, increasing  $s$  leads to quicker convergence for both EL-Oracle and EL-Local (see Theorem 1). Increasing  $s$  also directly increases the communication volume. The two right-most plots in Figure 5 show the test accuracy of EL-Oracle when the communication volume increases. After 3000 communication rounds, EL-Oracle with  $s = 14$  has incurred a communication volume of 15.1 GiB, compared to 4.3 GiB for  $s = 4$ . An optimal value of  $s$  depends on the network of the environment where EL is deployed. In data center settings where network links usually have high capacities, one can employ a high value of  $s$ . In edge settings with limited network capacities, however, the value of  $s$  should be smaller to avoid network congestion.
Decentralized Parallel Stochastic Gradient Descent (D-PSGD) Stochastic Gradient Descent is a stochastic variant of the gradient descent algorithm and is widely used to solve optimization problems at scale [14, 39, 46]. Decentralized algorithms using SGD have gained significant adoption as a method to train machine learning models, with Decentralized Parallel SGD (D-PSGD) being the most well-known DL algorithm [15, 31-33, 49]. D-PSGD avoids a server by relying on a communication topology describing how peers exchange their model updates [32].
Static topologies In a static topology, all nodes and edges remain fixed throughout the training process. The convergence speed of decentralized optimization algorithms closely depends on the mixing properties of the underlying topologies. There is a large body of work [2, 3, 6, 40] studying the mixing time of different random graphs such as the Erdos-Renyi graph and the geometric random graph. As the Erdos-Renyi graphs have better mixing properties [45], we compare EL with this family in Table 1. In [57], the authors prove that static exponential graphs in which nodes are connected to  $\mathcal{O}(\log n)$  neighbors are an effective topology for DL. EquiStatic is a static topology family whose consensus rate is independent of the network size [53].
Semi-dynamic topologies Several DL algorithms impose a fixed communication topology at the start of the training but have nodes communicate with random subsets of their neighbors each round. We classify such topologies as semi-dynamic topologies. The one-peer exponential graph has each node cycling through their  $\mathcal{O}(\log n)$  neighbors and has a similar convergence rate to static exponential graphs [57]. In the AD-PSGD algorithm, each node independently trains and averages its model with the model of a randomly selected neighbour [33]. EquiDyn is a dynamic topology family whose consensus rate is independent of the network size [53].
Time-varying and randomized topologies A time-varying topology is a topology that changes throughout the training process [9]. The convergence properties of time-varying graphs in distributed optimization have been studied by various works [23, 35, 36, 43, 45]. While these works provide convergence guarantees for decentralized optimization algorithms over time-varying (or random) topologies, they do not show the superiority of randomized communication, and they do not prove a convergence rate that cannot be obtained with a static graph [53]. Another work [34] considers client subsampling in decentralized optimization where at each round, only a fraction of nodes participate in the learning procedure. This approach is orthogonal to the problem we consider in this paper.
Gossip Learning Closely related to EL-Local is gossip learning (GL), a DL algorithm in which each node progresses through rounds independently from other peers [48]. In each round, a node sends their model to another random node and aggregates incoming models received by other nodes, weighted by age. While GL shows competitive performance compared to centralized approaches [16, 17], its convergence on non-convex loss functions has not been theoretically proven yet [47]. While at a high level, EL-Local with  $s = 1$  may look very similar to GL, there are some subtle differences. First, GL operates in an asynchronous manner whereas EL proceeds in synchronous rounds. Second, GL applies weighted averaging when aggregating models, based on model age, and EL aggregates models unweighted. Third, if a node in GL receives multiple models in a round, this node will aggregate the received model with its local model for each received model separately, whereas in EL, there will be a single model update per round, and all the received models from that round are aggregated together.
In contrast to the existing works, EL leverages a fully dynamic and random topology that changes each round. While static and semi-dynamic topologies have shown to be effective in certain settings, EL surpasses their performance by enabling faster convergence, both in theory and in practice.
We have introduced Epidemic Learning (EL), a new DL algorithm that accelerates model convergence and saves communication costs by leveraging randomized communication. The key idea of EL is that in each round, each node samples  $s$  other nodes in a  $n$ -node network and sends their model updates to these sampled nodes. We introduced two variants of the sampling approach: EL-Oracle in which the communication topology forms a  $s$ -regular graph each round, and EL-Local which forms a  $s$ -out graph. We theoretically proved the convergence of both EL variants and derived that the number of transient iterations for EL is in  $\mathcal{O}(n^3 / s^2)$ , outperforming the best-known bound  $\mathcal{O}(n^3)$  by a factor of  $s^2$  for smooth non-convex functions. Our experimental evaluation on the CIFAR-10 learning task showcases the  $1.7 \times$  quicker convergence of EL compared to baselines and highlights the reduction in communication costs.
This work has been supported in part by the Swiss National Science Foundation (SNSF) project 200021-200477.
[53] Zhuoqing Song, Weijian Li, Kexin Jin, Lei Shi, Ming Yan, Wotao Yin, and Kun Yuan. Communication-efficient topologies for decentralized learning with  $\$0(1)$ consensus rate. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=AyiiHcRzTd.
[54] Thijs Vogels, Hadrien Hendrikx, and Martin Jaggi. Beyond spectral gap (extended): The role of the topology in decentralized learning. arXiv preprint arXiv:2301.02151, 2023.
[55] Spyros Voulgaris, Daniela Gavidia, and Maarten Van Steen. Cyclon: Inexpensive membership management for unstructured p2p overlays. Journal of Network and systems Management, 13: 197-217, 2005.
[56] Jianyu Wang, Anit Kumar Sahu, Zhouyi Yang, Gauri Joshi, and Soummya Kar. Matcha: Speeding up decentralized sgd via matching decomposition sampling. In 2019 Sixth Indian Control Conference (ICC), pages 299-300. IEEE, 2019.
[57] Bicheng Ying, Kun Yuan, Yiming Chen, Hanbin Hu, Pan Pan, and Wotao Yin. Exponential graph is provably efficient for decentralized deep training. Advances in Neural Information Processing Systems, 34:13975-13987, 2021.
[58] Bicheng Ying, Kun Yuan, Hanbin Hu, Yiming Chen, and Wotao Yin. Bluefog: Make decentralized algorithms practical for optimization and deep learning. arXiv preprint arXiv:2111.04287, 2021.
- Appendix A proves the convergence guarantee of EL.
- In Appendix B, we prove the key lemmas that are used in the convergence proof.
- Appendix C provides experimental details, computational resources used, and further evaluation of EL with independent and identically distributed (IID) data distributions and the FEMNIST dataset.
- In Appendix D, we discuss the effect of network connectivity on the performance of EL.
A Convergence Proof
In this section, we prove Theorem 1, by setting
$$
\gamma := \min  \left\{\sqrt {\frac {n \Delta_ {0}}{T L \sigma^ {2}}}, \sqrt [ 3 ]{\frac {\Delta_ {0}}{1 0 0 T \alpha_ {s} L ^ {2} \left(\sigma^ {2} + \mathcal {H} ^ {2}\right)}}, \frac {1}{2 0 L} \right\}, \tag {4}
$$
for EL-Oracle, and
$$
\gamma := \min  \left\{\sqrt {\frac {n \Delta_ {0}}{T \left(2 1 1 \sigma^ {2} + 3 3 2 \beta_ {s} \mathcal {H} ^ {2}\right) L}}, \sqrt [ 3 ]{\frac {\Delta_ {0}}{2 5 0 T \beta_ {s} L ^ {2} \left(\sigma^ {2} + \mathcal {H} ^ {2}\right)}}, \frac {1}{2 0 L} \right\}, \tag {5}
$$
for EL-Local.
Notation: For any set of  $n$  vectors  $\{x^{(1)},\ldots ,x^{(n)}\}$ , we denote their average by  $\bar{x} := \frac{1}{n}\sum_{i\in [n]}x^{(i)}$ .
A.1 Proof steps
We outline here the critical elements for proving Theorem 1.
Mixing efficiency of EL. First, we analyze the mixing properties of the communication phase of both variants of the algorithm, and we observe that:
(a) while EL-Oracle preserves the exact average of the local vectors, i.e.,  $\bar{x}_{t + 1} = \bar{x}_{t + 1 / 2}$ , EL-Local only does so in expectation. This property is critical for ensuring that the global update is an unbiased estimator of the average of local gradients, which is necessary for obtaining convergence guarantees with linear speed-up.
(b) The communication phase of both EL-Local and EL-Oracle reduce the drift among the local models by a factor of  $\mathcal{O}(1 / s)$ .
(c) The variance of the averaged model after the communication phase, in EL-Local is in  $\mathcal{O}(1 / ns)$ .
More formally, we have the following lemma.
Lemma 1. Consider Algorithm 1. Let  $n \geq 2$ ,  $s \geq 1$ ,  $T \geq 1$ , and  $t \in \{0, \dots, T - 1\}$ .
1. For EL-Oracle, we have
(a)  $\bar{x}_{t + 1} = \bar{x}_{t + 1 / 2},$
(b)  $\frac{1}{n^2}\sum_{i,j\in [n]}\mathbb{E}\left[\left\| x_{t + 1}^{(i)} - x_{t + 1}^{(j)}\right\| ^2\right]\leq \alpha_s\cdot \frac{1}{n^2}\sum_{i,j\in [n]}\mathbb{E}\left[\left\| x_{t + 1 / 2}^{(i)} - x_{t + 1 / 2}^{(j)}\right\| ^2\right],$
2. For EL-Local, we have:
(a)  $\mathbb{E}\left[\bar{x}_{t + 1}\right] = \mathbb{E}\left[\bar{x}_{t + 1 / 2}\right]$  (Note that we do not necessarily have  $\bar{x}_{t + 1} = \bar{x}_{t + 1 / 2})$
(b)  $\frac{1}{n^2}\sum_{i,j\in [n]}\mathbb{E}\left[\left\| x_{t + 1}^{(i)} - x_{t + 1}^{(j)}\right\| ^2\right]\leq \beta_s\cdot \frac{1}{n^2}\sum_{i,j\in [n]}\mathbb{E}\left[\left\| x_{t + 1 / 2}^{(i)} - x_{t + 1 / 2}^{(j)}\right\| ^2\right],$
(c)  $\mathbb{E}\left[\left\| \bar{x}_{t + 1} - \bar{x}_{t + 1 / 2}\right\| ^2\right]\leq \frac{\beta_s}{2n}\cdot \frac{1}{n^2}\sum_{i,j\in [n]}\mathbb{E}\left[\left\| x_{t + 1 / 2}^{(i)} - x_{t + 1 / 2}^{(j)}\right\| ^2\right],$
$$
\beta_ {s} := \frac {1}{s} \left(1 - \left(1 - \frac {s}{n - 1}\right) ^ {n}\right) - \frac {1}{n - 1} \in \mathcal {O} (\frac {1}{s})
$$
Uniform bound on model drift and gradient drift. Next, using the previous lemma, we prove that, for a careful choice of the step-size  $\gamma$ , the local models stay close to each other during the learning process.
Lemma 2. Suppose that assumptions 1, 2, and 3 hold true. Consider Algorithm 1. Consider a step-size  $\gamma$  such that  $\gamma \leq \frac{1}{20L}$ . For any  $t \geq 0$ , we obtain that
$$
\frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] \leq 2 0 \frac {1 + 3 \eta_ {s}}{(1 - \eta_ {s}) ^ {2}} \eta_ {s} \gamma^ {2} (\sigma^ {2} + \mathcal {H} ^ {2}),
$$
$$
\frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| g _ {t} ^ {(i)} - g _ {t} ^ {(j)} \right\| ^ {2} \right] \leq 1 5 \left(\sigma^ {2} + \mathcal {H} ^ {2}\right),
$$
where  $\eta_s = \alpha_s$  for EL-Oracle and  $\eta_s = \beta_s$  for EL-Local as defined in Lemma 1.
Bound on the gradient norm. Next, we obtain a bound on the gradient of the loss function  $F(x)$  by analyzing the growth of  $F(x)$ , over the trajectory of the average of local models in Algorithm 1.
Lemma 3. Suppose that assumptions 1 and 2 hold true. Consider Algorithm 1 with  $\gamma \leq \frac{1}{2L}$ . For any  $t \in \{0, \dots, T - 1\}$ , we obtain that
$$
\begin{array}{l} \mathbb {E} \left[ \| \nabla F (\bar {x} _ {t}) \| ^ {2} \right] \leq \frac {2}{\gamma} \mathbb {E} \left[ F (\bar {x} _ {t}) - F (\bar {x} _ {t + 1}) \right] + \frac {L ^ {2}}{2 n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] \\ + 2 L \gamma \frac {\sigma^ {2}}{n} + \frac {2 L}{\gamma} \mathbb {E} \left[ \left\| \bar {x} _ {t + 1} - \bar {x} _ {t + 1 / 2} \right\| ^ {2} \right]. \\ \end{array}
$$
We can now prove the theorem using the above lemmas.
Proof of Theorem 1. By Young's inequality, for any  $i \in [n]$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \nabla F \left(x _ {t} ^ {(i)}\right) \right\| ^ {2} \right] \leq 2 \mathbb {E} \left[ \left\| \nabla F \left(\bar {x} _ {t}\right) \right\| ^ {2} \right] + 2 \mathbb {E} \left[ \left\| \nabla F \left(\bar {x} _ {t}\right) - \nabla F \left(x _ {t} ^ {(i)}\right) \right\| ^ {2} \right] \\ \leq 2 \mathbb {E} \left[ \left\| \nabla F (\bar {x} _ {t}) \right\| ^ {2} \right] + 2 L ^ {2} \mathbb {E} \left[ \left\| \bar {x} _ {t} - x _ {t} ^ {(i)} \right\| ^ {2} \right], \\ \end{array}
$$
where in the second inequality we used Assumption 1. Then, averaging over  $i \in [n]$ , we obtain that
$$
\begin{array}{l} \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| \nabla F (x _ {t} ^ {(i)}) \right\| ^ {2} \right] \leq 2 \mathbb {E} \left[ \| \nabla F (\bar {x} _ {t}) \| ^ {2} \right] + 2 L ^ {2} \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| \bar {x} _ {t} - x _ {t} ^ {(i)} \right\| ^ {2} \right] \\ = 2 \mathbb {E} \left[ \left\| \nabla F (\bar {x} _ {t}) \right\| ^ {2} \right] + L ^ {2} \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right], \\ \end{array}
$$
where we used Lemma 5. Combining this with Lemma 3, we obtain that<sup>4</sup>
$$
\begin{array}{l} \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| \nabla F \left(x _ {t} ^ {(i)}\right) \right\| ^ {2} \right] \leq \frac {4}{\gamma} \mathbb {E} \left[ F (\bar {x} _ {t}) - F (\bar {x} _ {t + 1}) \right] + \frac {2 L ^ {2}}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] \\ + 4 L \gamma \frac {\sigma^ {2}}{n} + \frac {4 L}{\gamma} \mathbb {E} \left[ \left\| \bar {x} _ {t + 1} - \bar {x} _ {t + 1 / 2} \right\| ^ {2} \right]. \tag {6} \\ \end{array}
$$
We now analyze the two variants of the algorithm separately.
EL-Oracle:
In this case, by Property (2a) of Lemma 1, we have  $\bar{x}_{t + 1} - \bar{x}_{t + 1 / 2} = 0$ , therefore, by (6), we have
$$
\frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| \nabla F \left(x _ {t} ^ {(i)}\right) \right\| ^ {2} \right] \leq \frac {4}{\gamma} \mathbb {E} \left[ F \left(\bar {x} _ {t}\right) - F \left(\bar {x} _ {t + 1}\right) \right] + \frac {2 L ^ {2}}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] + 4 L \gamma \frac {\sigma^ {2}}{n} \tag {7}
$$
Note also that by Lemma 1, we have
$$
\frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] \leq 2 0 \frac {1 + 3 \alpha_ {s}}{(1 - \alpha_ {s}) ^ {2}} \alpha_ {s} \gamma^ {2} (\sigma^ {2} + \mathcal {H} ^ {2}),
$$