SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Moreover, as shown in Remark 2, we have  $\alpha_{s} \leq 1/2$ , and thus,  $20\frac{1 + 3\alpha_{s}}{(1 - \alpha_{s})^{2}} \leq 200$ . Therefore,
$$
\frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] \leq 2 0 0 \alpha_ {s} \gamma^ {2} \left(\sigma^ {2} + \mathcal {H} ^ {2}\right),
$$
Combining this with (7), we obtain that
$$
\frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| \nabla F (x _ {t} ^ {(i)}) \right\| ^ {2} \right] \leq \frac {4}{\gamma} \mathbb {E} [ F (\bar {x} _ {t}) - F (\bar {x} _ {t + 1}) ] + 4 0 0 \alpha_ {s} L ^ {2} \gamma^ {2} (\sigma^ {2} + \mathcal {H} ^ {2}) + 4 L \gamma \frac {\sigma^ {2}}{n}
$$
Averaging over  $t\in \{0,\dots ,T - 1\}$  , we obtain that
$$
\begin{array}{l} \frac {1}{n T} \sum_ {i \in [ n ]} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left\| \nabla F (x _ {t} ^ {(i)}) \right\| ^ {2} \right] \leq \frac {4}{\gamma T} \mathbb {E} [ F (\bar {x} _ {0}) - F (\bar {x} _ {T}) ] + 4 0 0 \alpha_ {s} L ^ {2} \gamma^ {2} (\sigma^ {2} + \mathcal {H} ^ {2}) + 4 L \gamma \frac {\sigma^ {2}}{n} \\ \leq \frac {4}{\gamma T} \Delta_ {0} + 4 0 0 \alpha_ {s} L ^ {2} \gamma^ {2} (\sigma^ {2} + \mathcal {H} ^ {2}) + 4 L \gamma \frac {\sigma^ {2}}{n}, \tag {8} \\ \end{array}
$$
where we used the fact that  $F(\bar{x}_0) - F(\bar{x}_T) \leq F(\bar{x}_0) - \min_{x \in \mathbb{R}^d} F(x) \leq \Delta_0$ . Setting
$$
\gamma := \min  \left\{\sqrt {\frac {n \Delta_ {0}}{T L \sigma^ {2}}}, \sqrt [ 3 ]{\frac {\Delta_ {0}}{1 0 0 T \alpha_ {s} L ^ {2} \left(\sigma^ {2} + \mathcal {H} ^ {2}\right)}}, \frac {1}{2 0 L} \right\}, \tag {9}
$$
In order to obtain a bound on  $\mathbb{E}\left[\| \nabla F(\bar{x}_t)\|^2\right]$ , we can skip this step and directly use Lemma 3.
as  $\frac{1}{\min\{a,b\}} = \max \left\{\frac{1}{a},\frac{1}{b}\right\}$  for any  $a,b\geq 0$  , we have
$$
\frac {1}{\gamma} = \max  \left\{\sqrt {\frac {T L \sigma^ {2}}{n \Delta_ {0}}}, \sqrt [ 3 ]{\frac {1 0 0 T \alpha_ {s} L ^ {2} (\sigma^ {2} + \mathcal {H} ^ {2})}{\Delta_ {0}}}, 2 0 L \right\} \leq \sqrt {\frac {T L \sigma^ {2}}{n \Delta_ {0}}} + \sqrt [ 3 ]{\frac {1 0 0 T \alpha_ {s} L ^ {2} (\sigma^ {2} + \mathcal {H} ^ {2})}{\Delta_ {0}}} + 2 0 L, \tag {10}
$$
Plugging (9) and (10) in (8), we obtain that
$$
\begin{array}{l} \frac {1}{n T} \sum_ {i \in [ n ]} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left\| \nabla F (x _ {t} ^ {(i)}) \right\| ^ {2} \right] \leq 8 \sqrt {\frac {L \Delta_ {0} \sigma^ {2}}{n T}} + 3 8 \sqrt [ 3 ]{\frac {\alpha_ {s} L ^ {2} \Delta_ {0} ^ {2} (\sigma^ {2} + \mathcal {H} ^ {2})}{T ^ {2}}} + \frac {8 0 L \Delta_ {0} \sigma^ {2}}{T} \\ \in \mathcal {O} \left(\sqrt {\frac {L \Delta_ {0} \sigma^ {2}}{n T}} + \sqrt [ 3 ]{\frac {\alpha_ {s} L ^ {2} \Delta_ {0} ^ {2} (\sigma^ {2} + \mathcal {H} ^ {2})}{T ^ {2}}} + \frac {L \Delta_ {0}}{T}\right) \\ \end{array}
$$
where we used the fact that  $\frac{800}{100^{(2/3)}} \leq 38$ .
EL-Local:
The proof is similar to EL-Oracle, up to some additional error terms. By Property (2c) of Lemma 1, we obtain that
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \bar {x} _ {t + 1} - \bar {x} _ {t + 1 / 2} \right\| ^ {2} \right] \leq \frac {\beta_ {s}}{2 n} \cdot \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t + 1 / 2} ^ {(i)} - x _ {t + 1 / 2} ^ {(j)} \right\| ^ {2} \right] \\ = \frac {\beta_ {s}}{2 n} \cdot \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - \gamma g _ {t} ^ {(i)} - x _ {t} ^ {(j)} + \gamma g _ {t} ^ {(j)} \right\| ^ {2} \right] (11) \\ \leq \frac {\beta_ {s}}{n} \cdot \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] + \frac {\gamma^ {2} \beta_ {s}}{n} \cdot \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| g _ {t} ^ {(i)} - g _ {t} ^ {(j)} \right\| ^ {2} \right], (12) \\ \end{array}
$$
where in the last inequality we use Young's inequality. Combining this with (6), we obtain that
$$
\begin{array}{l} \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| \nabla F (x _ {t} ^ {(i)}) \right\| ^ {2} \right] \leq \frac {4}{\gamma} \mathbb {E} [ F (\bar {x} _ {t}) - F (\bar {x} _ {t + 1}) ] + \frac {2 L ^ {2}}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] \\ + 4 L \gamma \frac {\sigma^ {2}}{n} + \frac {4 L}{\gamma} \frac {\beta_ {s}}{n} \cdot \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] + \frac {4 L \gamma \beta_ {s}}{n} \cdot \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| g _ {t} ^ {(i)} - g _ {t} ^ {(j)} \right\| ^ {2} \right] \\ = \frac {4}{\gamma} \mathbb {E} \left[ F (\bar {x} _ {t}) - F (\bar {x} _ {t + 1}) \right] + (2 L ^ {2} + \frac {4 L \beta_ {s}}{n \gamma}) \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] \\ + 4 L \gamma \frac {\sigma^ {2}}{n} + \frac {4 L \gamma \beta_ {s}}{n} \cdot \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| g _ {t} ^ {(i)} - g _ {t} ^ {(j)} \right\| ^ {2} \right] \tag {13} \\ \end{array}
$$
Note that by Remark 2, for any  $s \geq 1$ , and  $n \geq 2$  we have  $\beta_s \leq 1 - \frac{1}{e}$ , where  $e$  is Euler's number. Therefore,
$$
2 0 \frac {1 + 3 \beta_ {s}}{(1 - \beta_ {s}) ^ {2}} \leq 5 0 0.
$$
Using this, the first bound in Lemma 2 can be simplified to
$$
\frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] \leq 5 0 0 \beta_ {s} \gamma^ {2} (\sigma^ {2} + \mathcal {H} ^ {2}) \tag {14}
$$
Combining this with (13), and the second bound of Lemma 2, we have
$$
\begin{array}{l} \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| \nabla F (x _ {t} ^ {(i)}) \right\| ^ {2} \right] \leq \frac {4}{\gamma} \mathbb {E} [ F (\bar {x} _ {t}) - F (\bar {x} _ {t + 1}) ] + 4 L \gamma \frac {\sigma^ {2}}{n} (1 + 1 5 \beta_ {s} + 5 0 0 \beta_ {s} ^ {2}) \\ + 4 L \gamma \frac {\mathcal {H} ^ {2}}{n} \left(1 5 \beta_ {s} + 5 0 0 \beta_ {s} ^ {2}\right) + L ^ {2} 1 0 0 0 \beta_ {s} \gamma^ {2} \left(\sigma^ {2} + \mathcal {H} ^ {2}\right). \\ \end{array}
$$
Taking the average over  $t\in \{0,\dots ,T - 1\}$  , we obtain that
$$
\begin{array}{l} \frac {1}{n T} \sum_ {t = 0} ^ {T - 1} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| \nabla F (x _ {t} ^ {(i)}) \right\| ^ {2} \right] \leq \frac {4}{T \gamma} \Delta_ {0} + 4 L \gamma \frac {\sigma^ {2}}{n} (1 + 1 5 \beta_ {s} + 5 0 0 \beta_ {s} ^ {2}) \\ + 4 L \gamma \frac {\mathcal {H} ^ {2}}{n} \left(1 5 \beta_ {s} + 5 0 0 \beta_ {s} ^ {2}\right) + L ^ {2} 1 0 0 0 \beta_ {s} \gamma^ {2} \left(\sigma^ {2} + \mathcal {H} ^ {2}\right). \\ \end{array}
$$
Noting that  $\beta_{s} < 1 - \frac{1}{e}$  (Remark 2), we have
$$
\frac {1}{n T} \sum_ {t = 0} ^ {T - 1} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| \nabla F \left(x _ {t} ^ {(i)}\right) \right\| ^ {2} \right] \leq \frac {4}{T \gamma} \Delta_ {0} + 4 \frac {L \gamma}{n} (2 1 1 \sigma^ {2} + 3 3 2 \beta_ {s} \mathcal {H} ^ {2}) + 1 0 0 0 \beta_ {s} L ^ {2} \gamma^ {2} (\sigma^ {2} + \mathcal {H} ^ {2}) \tag {15}
$$
Now, setting
$$
\gamma = \min  \left\{\sqrt {\frac {n \Delta_ {0}}{T \left(2 1 1 \sigma^ {2} + 3 3 2 \beta_ {s} \mathcal {H} ^ {2}\right) L}}, \sqrt [ 3 ]{\frac {\Delta_ {0}}{2 5 0 T \beta_ {s} L ^ {2} \left(\sigma^ {2} + \mathcal {H} ^ {2}\right)}}, \frac {1}{2 0 L} \right\}, \tag {16}
$$
$$
\begin{array}{l} \frac {1}{\gamma} = \max  \left\{\sqrt {\frac {T (2 1 1 \sigma^ {2} + 3 3 2 \beta_ {s} \mathcal {H} ^ {2}) L}{n \Delta_ {0}}}, \sqrt [ 3 ]{\frac {2 5 0 T \beta_ {s} L ^ {2} (\sigma^ {2} + \mathcal {H} ^ {2})}{\Delta_ {0}}}, 2 0 L \right\} \\ \leq \sqrt {\frac {T \left(2 1 1 \sigma^ {2} + 3 3 2 \beta_ {s} \mathcal {H} ^ {2}\right) L}{n \Delta_ {0}}} + \sqrt [ 3 ]{\frac {2 5 0 T \beta_ {s} L ^ {2} \left(\sigma^ {2} + \mathcal {H} ^ {2}\right)}{\Delta_ {0}}} + 2 0 L \tag {17} \\ \end{array}
$$
Plugging (17), and (16) in (15) we obtain that
$$
\begin{array}{l} \frac {1}{n T} \sum_ {t = 0} ^ {T - 1} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| \nabla F (x _ {t} ^ {(i)}) \right\| ^ {2} \right] \leq 8 \sqrt {\frac {L \Delta_ {0} (2 1 1 \sigma^ {2} + 3 3 2 \beta_ {s} \mathcal {H} ^ {2})}{n T}} + 5 1 \sqrt [ 3 ]{\frac {\beta_ {s} L ^ {2} \Delta_ {0} ^ {2} (\sigma^ {2} + \mathcal {H} ^ {2})}{T ^ {2}}} + \frac {8 0 L}{T} \Delta_ {0} \\ \in \mathcal {O} \left(\sqrt {\frac {L \Delta_ {0} (\sigma^ {2} + \beta_ {s} \mathcal {H} ^ {2})}{n T}} + \sqrt [ 3 ]{\frac {\beta_ {s} L ^ {2} \Delta_ {0} ^ {2} (\sigma^ {2} + \mathcal {H} ^ {2})}{T ^ {2}}} + \frac {L \Delta_ {0}}{T}\right). \\ \end{array}
$$
where we used the fact that  $\frac{2000}{250^{(2/3)}} \leq 51$ . This concludes the proof.
B Proof of the main lemmas
Notation: Let  $\mathcal{P}_t$  represent the history from step 0 to  $t$ . More precisely, we define
$$
\mathcal {P} _ {t} := \left\{x _ {0} ^ {(i)}, \ldots , x _ {t} ^ {(i)}; i = 1, \ldots , n \right\}.
$$
Moreover, we use the notation  $\mathbb{E}_t[\cdot] \coloneqq \mathbb{E}[\cdot|\mathcal{P}_t]$  to denote the conditional expectation given the history  $\mathcal{P}_t$  and  $\mathbb{E}[\cdot]$  to denote the total expectation over the randomness of the algorithm; therefore we have,  $\mathbb{E}[\cdot] \coloneqq \mathbb{E}_0[\dots \mathbb{E}_T[\cdot]]$ .
We first prove two simple useful lemmas.
Lemma 4. for any set of  $k$  vectors  $x^{(1)},\ldots ,x^{(k)}$  , we have
$$
\left\| \sum_ {i = 1} ^ {k} x ^ {(i)} \right\| ^ {2} \leq k \sum_ {i = 1} ^ {k} \left\| x ^ {(i)} \right\| ^ {2}.
$$
$$
\left\| \sum_ {i = 1} ^ {k} x ^ {(i)} \right\| \leq \sum_ {i = 1} ^ {k} \left\| x ^ {(i)} \right\|.
$$
The result then follows by noting that using Jensen's inequality, we have
$$
\left(\frac {1}{k} \sum_ {i = 1} ^ {k} \left\| x ^ {(i)} \right\|\right) ^ {2} \leq \frac {1}{k} \sum_ {i = 1} ^ {k} \left\| x ^ {(i)} \right\| ^ {2}.
$$
Lemma 5. For any set  $\{x^{(i)}\}_{i\in [n]}$  of  $n$  vectors, we have
$$
\frac {1}{n} \sum_ {i \in [ n ]} \left\| x ^ {(i)} - \bar {x} \right\| ^ {2} = \frac {1}{2} \cdot \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \left\| x ^ {(i)} - x ^ {(j)} \right\| ^ {2}.
$$
$$
\begin{array}{l} \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \left\| x ^ {(i)} - x ^ {(j)} \right\| ^ {2} = \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \left\| (x ^ {(i)} - \bar {x}) - (x ^ {(j)} - \bar {x}) \right\| ^ {2} \\ = \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \left[ \left\| x ^ {(i)} - \bar {x} \right\| ^ {2} + \left\| x ^ {(j)} - \bar {x} \right\| ^ {2} + 2 \left\langle x ^ {(i)} - \bar {x}, x ^ {(j)} - \bar {x} \right\rangle \right] \\ = \frac {2}{n} \sum_ {i, j \in [ n ]} \left\| x ^ {(i)} - \bar {x} \right\| ^ {2} + \frac {2}{n ^ {2}} \sum_ {i \in [ n ]} \left\langle x ^ {(i)} - \bar {x}, \sum_ {j \in [ n ]} (x ^ {(j)} - \bar {x}) \right\rangle . \\ \end{array}
$$
Noting that  $\sum_{j\in [n]}(x^{(j)} - \bar{x}) = 0$ , yields the desired result.
$$
\alpha_ {s} := \frac {1}{s + 1} \left(1 - \frac {s}{n - 1}\right) \in \mathcal {O} \left(\frac {1}{s}\right).
$$
Proof. For simplicity of the notation, we make the dependence on  $t$  implicit, and we denote by  $x^{(i)} \coloneqq x_{t+1/2}^{(i)}$ , the input vector and  $y^{(i)} \coloneqq x_{t+1}^{(i)}$  the output vector of node  $i$  for the communication phase, and  $\mathcal{S}^{(i)} \coloneqq \mathcal{S}_t^{(i)}$ . Therefore, we have
$$
y ^ {(i)} = \frac {1}{\left| \mathcal {S} ^ {(i)} \right| + 1} \left(x ^ {(i)} + \sum_ {j \in \mathcal {S} ^ {(i)}} x ^ {(j)}\right).
$$
Moreover, in the proof of this lemma, all the expectations  $\mathbb{E}[\cdot ]$  are only with respect to the randomness of the communication phase, i.e., we compute the expectations assuming  $x^{(i)}$ 's are given for all  $i\in [n]$ , and  $y^{(i)}$ 's are computed by a random communication phase. Taking the total expectation of the final expression provided for each property then proves it as stated in the lemma.
First, we consider EL-Oracle:
Recall that in EL-Oracle, the communication topology forms a random undirected  $s$ -regular graph, and for each node  $i$ , all other nodes have the same probability of being  $i$ 's neighbor.
Property (a):
$$
\begin{array}{l} \bar {y} = \frac {1}{n} \sum_ {i \in [ n ]} y ^ {(i)} \\ = \frac {1}{n} \sum_ {i \in [ n ]} \frac {1}{\left| \mathcal {S} ^ {(i)} \right| + 1} \left(x ^ {(i)} + \sum_ {j \in \mathcal {S} ^ {(i)}} x ^ {(j)}\right) \\ = \frac {1}{n (s + 1)} \sum_ {i \in [ n ]} \left(x ^ {(i)} + \sum_ {j \in \mathcal {S} ^ {(i)}} x ^ {(j)}\right) \\ = \frac {1}{n (s + 1)} \left(\sum_ {i \in [ n ]} x ^ {(i)} + \sum_ {i \in [ n ]} \sum_ {j \in \mathcal {S} ^ {(i)}} x ^ {(j)}\right), \\ \end{array}
$$
where we used the fact that in EL-Oracle, each node receives exactly  $s$  models. Now as each node also sends its model to  $s$  other nodes, we have
$$
\bar {y} = \frac {1}{n (s + 1)} \left(\sum_ {i \in [ n ]} x ^ {(i)} + \sum_ {i \in [ n ]} s x ^ {(i)}\right) = \frac {1}{n} \sum_ {i \in [ n ]} x ^ {(i)} = \bar {x}.
$$
Property (b):
$$
\begin{array}{l} \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| y ^ {(i)} - \bar {x} \right\| ^ {2} \right] = \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| \frac {1}{\left| \mathcal {S} ^ {(i)} \right| + 1} \left(x ^ {(i)} + \sum_ {j \in \mathcal {S} ^ {(i)}} x ^ {(j)}\right) - \bar {x} \right\| ^ {2} \right] \\ = \frac {1}{n (s + 1) ^ {2}} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| \left(x ^ {(i)} - \bar {x}\right) + \sum_ {j \in \mathcal {S} ^ {(i)}} \left(x ^ {(j)} - \bar {x}\right) \right\| ^ {2} \right] \\ = \frac {1}{n (s + 1) ^ {2}} \sum_ {i \in [ n ]} \left\| x ^ {(i)} - \bar {x} \right\| ^ {2} + \frac {1}{n (s + 1) ^ {2}} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| \sum_ {j \in \mathcal {S} ^ {(i)}} \left(x ^ {(j)} - \bar {x}\right) \right\| ^ {2} \right] \\ + \frac {2}{n (s + 1) ^ {2}} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\langle \left(x ^ {(i)} - \bar {x}\right), \sum_ {j \in \mathcal {S} ^ {(i)}} \left(x ^ {(j)} - \bar {x}\right) \right\rangle \right]. \tag {18} \\ \end{array}
$$
Let us define  $\mathcal{I}_j^{(i)}$  the indicator function showing whether node  $j$  and node  $i$  are neighbors, i.e.,  $\mathcal{I}_j^{(i)} = 1$  if  $i$  and  $j$  are neighbors and  $\mathcal{I}_j^{(i)} = 0$  otherwise. We then have
$$
\begin{array}{l} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| \sum_ {j \in \mathcal {S} ^ {(i)}} \left(x ^ {(j)} - \bar {x}\right) \right\| ^ {2} \right] = \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| \sum_ {j \in [ n ] \backslash \{i \}} \mathcal {I} _ {j} ^ {(i)} \left(x ^ {(j)} - \bar {x}\right) \right\| ^ {2} \right] \\ = \sum_ {i \in [ n ]} \mathbb {E} \left[ \sum_ {j \in [ n ] \backslash \{i \}} \mathcal {I} _ {j} ^ {(i)} \left\| x ^ {(j)} - \bar {x} \right\| ^ {2} \right] + \sum_ {i \in [ n ]} \mathbb {E} \left[ \sum_ {j \neq i} \sum_ {k \neq i, k \neq j} \mathcal {I} _ {j} ^ {(i)} \mathcal {I} _ {k} ^ {(i)} \left\langle x ^ {(j)} - \bar {x}, x ^ {(k)} - \bar {x} \right\rangle \right] \\ = \sum_ {i \in [ n ]} \sum_ {j \in [ n ] \backslash \{i \}} \mathbb {E} \left[ \mathcal {I} _ {j} ^ {(i)} \right] \left\| x ^ {(j)} - \bar {x} \right\| ^ {2} + \sum_ {i \in [ n ]} \sum_ {j \neq i} \sum_ {k \neq i, k \neq j} \mathbb {E} \left[ \mathcal {I} _ {j} ^ {(i)} \mathcal {I} _ {k} ^ {(i)} \right] \left\langle x ^ {(j)} - \bar {x}, x ^ {(k)} - \bar {x} \right\rangle . \\ \end{array}
$$
Now note that by symmetry (all nodes have the same probability of being a neighbor of node  $i$ ).
$$
\mathbb {E} \left[ \mathcal {I} _ {j} ^ {(i)} \right] = \frac {s}{n - 1}.
$$
Similarly  $\mathcal{I}_j^{(i)}\mathcal{I}_k^{(i)}$  is only 1 when both  $j$  and  $k$  are neighbors of  $i$ , thus
$$
\mathbb {E} \left[ \mathcal {I} _ {j} ^ {(i)} \mathcal {I} _ {k} ^ {(i)} \right] = \frac {s (s - 1)}{(n - 1) (n - 2)}.
$$
$$
\begin{array}{l} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| \sum_ {j \in \mathcal {S} ^ {(i)}} (x ^ {(j)} - \bar {x}) \right\| ^ {2} \right] = s \sum_ {i \in [ n ]} \left\| x ^ {(j)} - \bar {x} \right\| ^ {2} + \sum_ {i \in [ n ]} \sum_ {j \neq i} \sum_ {k \neq i, k \neq j} \frac {s (s - 1)}{(n - 1) (n - 2)} \left\langle x ^ {(j)} - \bar {x}, x ^ {(k)} - \bar {x} \right\rangle \\ = s \sum_ {i \in [ n ]} \left\| x ^ {(j)} - \bar {x} \right\| ^ {2} + \sum_ {i \in [ n ]} \sum_ {j \neq i} \frac {s (s - 1)}{n - 1} \left\langle x ^ {(i)} - \bar {x}, x ^ {(j)} - \bar {x} \right\rangle . \tag {19} \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\langle \left(x ^ {(i)} - \bar {x}\right), \sum_ {j \in \mathcal {S} ^ {(i)}} \left(x ^ {(j)} - \bar {x}\right) \right\rangle \right] = \sum_ {i \in [ n ]} \left\langle \left(x ^ {(i)} - \bar {x}\right), \sum_ {j \neq i} \mathbb {E} \left[ \mathcal {I} _ {j} ^ {(i)} \right] \left(x ^ {(j)} - \bar {x}\right) \right\rangle \\ = \frac {s}{n - 1} \sum_ {i \in [ n ]} \sum_ {j \neq i} \left\langle x ^ {(i)} - \bar {x}, x ^ {(j)} - \bar {x} \right\rangle \tag {20} \\ \end{array}
$$
Combining (18), (19), and (20) we obtain that
$$
\begin{array}{l} \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| y ^ {(i)} - \bar {x} \right\| ^ {2} \right] = \frac {1 + s}{n (s + 1) ^ {2}} \sum_ {i \in [ n ]} \left\| x ^ {(i)} - \bar {x} \right\| ^ {2} \tag {21} \\ + \frac {1}{n (s + 1) ^ {2}} \left(\frac {2 s}{n - 1} + \frac {s (s - 1)}{n - 1}\right) \sum_ {i \in [ n ]} \sum_ {j \neq i} \left\langle x ^ {(i)} - \bar {x}, x ^ {(j)} - \bar {x} \right\rangle . \\ \end{array}
$$
Also note that
$$
\sum_ {i \in [ n ]} \sum_ {j \neq i} \left\langle x ^ {(i)} - \bar {x}, x ^ {(j)} - \bar {x} \right\rangle = \sum_ {i \in [ n ]} \left\langle x ^ {(i)} - \bar {x}, \sum_ {j \neq i} x ^ {(j)} - \bar {x} \right\rangle = - \sum_ {i \in [ n ]} \left\| x ^ {(i)} - \bar {x} \right\| ^ {2}. \tag {22}
$$
Combining (21) and (22), we obtain that
$$
\frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| y ^ {(i)} - \bar {x} \right\| ^ {2} \right] = \frac {1}{s + 1} \left(1 - \frac {s}{n - 1}\right) \frac {1}{n} \sum_ {i \in [ n ]} \left\| x ^ {(i)} - \bar {x} \right\| ^ {2} \tag {23}
$$
Now note that as  $\bar{y}$  is the minimizer of  $g(z)\coloneqq \frac{1}{n}\sum_{i\in [n]}\mathbb{E}\left[\left\| y^{(i)} - z\right\| ^2\right]$ , we have
$$
\frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| y ^ {(i)} - \bar {y} \right\| ^ {2} \right] \leq \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| y ^ {(i)} - \bar {x} \right\| ^ {2} \right]. \tag {24}
$$
Combining this with (23), and using Lemma 5 yields
$$
\frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| y ^ {(i)} - y ^ {(j)} \right\| ^ {2} \right] \leq \frac {1}{s + 1} \left(1 - \frac {s}{n - 1}\right) \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x ^ {(i)} - x ^ {(j)} \right\| ^ {2} \right],
$$
Now, we consider EL-Local:
Recall that in EL-Local, each node  $i$ , locally samples uniformly at random  $s$  other nodes and sends its model to them. Denote by  $A^{(i)} \coloneqq \left| S^{(i)} \right|$  the number of nodes that send their vectors to node  $i$ , and  $\mathcal{I}_j^{(i)}$  the indicator showing whether node  $j$  sends its vector to node  $i$  or not ( $\mathcal{I}_j^{(i)} = 1$  if  $j$  sends its model to  $i$ , and  $\mathcal{I}_j^{(i)} = 0$  otherwise). We then have
$$
A ^ {(i)} = \sum_ {j \in [ n ] \backslash \{i \}} \mathcal {I} _ {j} ^ {(i)},
$$
$$
\mathbb {E} \left[ A ^ {(i)} \right] = s.
$$
For any node  $i$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ y ^ {(i)} \right] = \mathbb {E} \left[ \frac {1}{A ^ {(i)} + 1} \left(x ^ {(i)} + \sum_ {j \in [ n ] \backslash \{i \}} \mathcal {I} _ {j} ^ {(i)} x ^ {(j)}\right) \right] \\ = \mathbb {E} \left[ \mathbb {E} \left[ \frac {1}{A ^ {(i)} + 1} \left(x ^ {(i)} + \sum_ {j \in [ n ] \backslash \{i \}} \mathcal {I} _ {j} ^ {(i)} x ^ {(j)}\right) | A ^ {(i)} \right] \right] \\ = \mathbb {E} \left[ \frac {1}{A ^ {(i)} + 1} \left(x ^ {(i)} + \sum_ {j \in [ n ] \backslash \{i \}} \mathbb {E} \left[ \mathcal {I} _ {j} ^ {(i)} | A ^ {(i)} \right] x ^ {(j)}\right) \right] \\ \stackrel {(a)} {=} \mathbb {E} \left[ \frac {1}{A ^ {(i)} + 1} \left(x ^ {(i)} + \frac {A ^ {(i)}}{n - 1} \sum_ {j \in [ n ] \backslash \{i \}} x ^ {(j)}\right) \right] \\ = \mathbb {E} \left[ \frac {1}{A ^ {(i)} + 1} \left(x ^ {(i)} + \frac {A ^ {(i)}}{n - 1} \left(n \bar {x} - x ^ {(i)}\right)\right) \right], \\ \end{array}
$$
where (a) uses the fact that  $\mathbb{E}\left[\mathcal{I}_j^{(i)}|A^{(i)}\right] = \frac{A^{(i)}}{n - 1}$  as all other  $n - 1$  nodes have the same probability of sending their model to  $i$ . Denoting  $p = \mathbb{E}\left[\frac{A^{(i)}}{A^{(i)} + 1}\right]$ , we obtain that
$$
\mathbb {E} \left[ y ^ {(i)} \right] = \frac {p n}{n - 1} \bar {x} + \left(1 - \frac {p n}{n - 1}\right) x ^ {(i)}.
$$
Averaging over all  $i \in [n]$  yields the result.
$$
\begin{array}{l} \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| y ^ {(i)} - \bar {x} \right\| ^ {2} \right] = \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| \frac {1}{A ^ {(i)} + 1} \left(x ^ {(i)} + \sum_ {j \in [ n ] \backslash \{i \}} \mathcal {I} _ {j} ^ {(i)} x ^ {(j)}\right) - \bar {x} \right\| ^ {2} \right] \\ = \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \mathbb {E} \left[ \left\| \frac {1}{A ^ {(i)} + 1} \left(x ^ {(i)} + \sum_ {j \in [ n ] \backslash \{i \}} \mathcal {I} _ {j} ^ {(i)} x ^ {(j)}\right) - \bar {x} \right\| ^ {2} | A ^ {(i)} \right] \right] \\ = \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \mathbb {E} \left[ \left\| \frac {1}{A ^ {(i)} + 1} \left((x ^ {(i)} - \bar {x}) + \sum_ {j \in [ n ] \backslash \{i \}} \mathcal {I} _ {j} ^ {(i)} (x ^ {(j)} - \bar {x})\right) \right\| ^ {2} | A ^ {(i)} \right] \right] \\ = \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \frac {1}{\left(A ^ {(i)} + 1\right) ^ {2}} \mathbb {E} \left[ \left\| x ^ {(i)} - \bar {x} \right\| ^ {2} + \sum_ {j \neq i} \mathcal {I} _ {j} ^ {(i)} \left\| x ^ {(j)} - \bar {x} \right\| ^ {2} \right] | A ^ {(i)} \right] \\ + \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \frac {1}{(A ^ {(i)} + 1) ^ {2}} \mathbb {E} \left[ 2 \sum_ {j \neq i} \mathcal {I} _ {j} ^ {(i)} \left\langle x ^ {(i)} - \bar {x}, x ^ {(j)} - \bar {x} \right\rangle + \sum_ {j \neq i} \sum_ {k \neq i, k \neq j} \mathcal {I} _ {j} ^ {(i)} \mathcal {I} _ {k} ^ {(i)} \left\langle x ^ {(j)} - \bar {x}, x ^ {(k)} - \bar {x} \right\rangle \right] | A ^ {(i)} \right] \\ \end{array}
$$
Taking the expectation inside, we obtain that
$$
\begin{array}{l} \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| y ^ {(i)} - \bar {x} \right\| ^ {2} \right] = \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \frac {1}{(A ^ {(i)} + 1) ^ {2}} \left(\left\| x ^ {(i)} - \bar {x} \right\| ^ {2} + \sum_ {j \neq i} \mathbb {E} \left[ \mathcal {I} _ {j} ^ {(i)} | A ^ {(i)} \right] \left\| x ^ {(j)} - \bar {x} \right\| ^ {2}\right) \right] \\ + \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \frac {1}{\left(A ^ {(i)} + 1\right) ^ {2}} \left(2 \sum_ {j \neq i} \mathbb {E} \left[ \mathcal {I} _ {j} ^ {(i)} \mid A ^ {(i)} \right] \left\langle x ^ {(i)} - \bar {x}, x ^ {(j)} - \bar {x} \right\rangle\right) \right] \\ + \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \frac {1}{(A ^ {(i)} + 1) ^ {2}} \left(\sum_ {j \neq i} \sum_ {k \neq i, k \neq j} \mathbb {E} \left[ \mathcal {I} _ {j} ^ {(i)} \mathcal {I} _ {k} ^ {(i)} | A ^ {(i)} \right] \left\langle x ^ {(j)} - \bar {x}, x ^ {(k)} - \bar {x} \right\rangle\right) \right]. \\ \end{array}
$$
Now note that  $\mathbb{E}\left[\mathcal{I}_j^{(i)}|A^{(i)}\right]$  is the probability that  $j$  selects  $i$ , given that in total  $A^{(i)}$  nodes select  $i$ , thus
$$
\mathbb {E} \left[ \mathcal {I} _ {j} ^ {(i)} | A ^ {(i)} \right] = \frac {A ^ {(i)}}{n - 1}.
$$
Similarly  $\mathcal{I}_j^{(i)}\mathcal{I}_k^{(i)}$  is only 1 when both  $j$  and  $k$  select  $i$ , thus
$$
\mathbb {E} \left[ \mathcal {I} _ {j} ^ {(i)} \mathcal {I} _ {k} ^ {(i)} | A ^ {(i)} \right] = \frac {A ^ {(i)} (A ^ {(i)} - 1)}{(n - 1) (n - 2)}
$$
Also, note that