SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {j \neq i} \left\langle x ^ {(i)} - \bar {x}, x ^ {(j)} - \bar {x} \right\rangle = \left\langle x ^ {(i)} - \bar {x}, \sum_ {j \neq i} (x ^ {(j)} - \bar {x}) \right\rangle = - \left\| x ^ {(i)} - \bar {x} \right\| ^ {2},
$$
$$
\begin{array}{l} \sum_ {j \neq i} \sum_ {k \neq i, k \neq j} \left\langle x ^ {(j)} - \bar {x}, x ^ {(k)} - \bar {x} \right\rangle = \sum_ {j \neq i} \left\langle x ^ {(j)} - \bar {x}, \sum_ {k \neq i, k \neq j} \left(x ^ {(k)} - \bar {x}\right) \right\rangle \\ = - \sum_ {j \neq i} \left\langle x ^ {(j)} - \bar {x}, \left(x ^ {(i)} - \bar {x}\right) + \left(x ^ {(j)} - \bar {x}\right) \right\rangle \\ = \left\| x ^ {(i)} - \bar {x} \right\| ^ {2} - \sum_ {j \neq i} \left\| x ^ {(j)} - \bar {x} \right\| ^ {2}. \\ \end{array}
$$
Combining all yields
$$
\begin{array}{l} \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| y ^ {(i)} - \bar {x} \right\| ^ {2} \right] = \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \frac {1}{(A ^ {(i)} + 1) ^ {2}} \left(\left\| x ^ {(i)} - \bar {x} \right\| ^ {2} + \frac {A ^ {(i)}}{n - 1} \sum_ {j \neq i} \left\| x ^ {(j)} - \bar {x} \right\| ^ {2}\right) \right] \\ + \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \frac {1}{(A ^ {(i)} + 1) ^ {2}} \left(- \frac {2 A ^ {(i)}}{n - 1} \left\| x ^ {(i)} - \bar {x} \right\| ^ {2}\right) \right] \\ + \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \frac {1}{\left(A ^ {(i)} + 1\right) ^ {2}} \cdot \frac {A ^ {(i)} \left(A ^ {(i)} - 1\right)}{(n - 1) (n - 2)} \left(\left\| x ^ {(i)} - \bar {x} \right\| ^ {2} - \sum_ {j \neq i} \left\| x ^ {(j)} - \bar {x} \right\| ^ {2}\right) \right] \\ = \frac {1}{n} \sum_ {i \in [ n ]} \left\| x ^ {(i)} - \bar {x} \right\| ^ {2} \mathbb {E} \left[ \frac {1}{(A ^ {(i)} + 1) ^ {2}} \left(1 - \frac {2 A ^ {(i)}}{n - 1} + \frac {A ^ {(i)} (A ^ {(i)} - 1)}{(n - 1) (n - 2)}\right) \right] \\ + \frac {1}{n} \sum_ {i \in [ n ]} \left(\mathbb {E} \left[ \frac {1}{\left(A ^ {(i)} + 1\right) ^ {2}} \left(\frac {A ^ {(i)}}{n - 1} - \frac {A ^ {(i)} \left(A ^ {(i)} - 1\right)}{(n - 1) (n - 2)}\right) \right] \sum_ {j \neq i} \left\| x ^ {(j)} - \bar {x} \right\| ^ {2}\right) \\ \end{array}
$$
Now note that by symmetry, the distribution of  $A^{(i)}$  is the same as  $A^{(j)}$  for any  $i,j\in [n]$ . Therefore,
$$
\begin{array}{l} \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| y ^ {(i)} - \bar {x} \right\| ^ {2} \right] \\ = \frac {1}{n} \sum_ {i \in [ n ]} \left\| x ^ {(i)} - \bar {x} \right\| ^ {2} \mathbb {E} \left[ \frac {1}{(A ^ {(1)} + 1) ^ {2}} \left(1 - \frac {2 A ^ {(1)}}{n - 1} + \frac {A ^ {(1)} (A ^ {(1)} - 1)}{(n - 1) (n - 2)} + A ^ {(1)} - \frac {A ^ {(1)} (A ^ {(1)} - 1)}{n - 2}\right) \right] \\ \end{array}
$$
Now note that
$$
1 - \frac {2 A ^ {(1)}}{n - 1} + \frac {A ^ {(1)} (A ^ {(1)} - 1)}{(n - 1) (n - 2)} + A ^ {(1)} - \frac {A ^ {(1)} (A ^ {(1)} - 1)}{n - 2} = 1 + A ^ {(1)} - \frac {A ^ {(1) ^ {2}} + A ^ {(1)}}{n - 1} = (1 + A ^ {(1)}) (1 - \frac {A ^ {(1)}}{n - 1}),
$$
$$
\frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| y ^ {(i)} - \bar {x} \right\| ^ {2} \right] = \frac {1}{n} \sum_ {i \in [ n ]} \left\| x ^ {(i)} - \bar {x} \right\| ^ {2} \left(\mathbb {E} \left[ \frac {1}{A ^ {(1)} + 1} \right] - \frac {1}{n - 1} \cdot \mathbb {E} \left[ \frac {A ^ {(1)}}{A ^ {(1)} + 1} \right]\right)
$$
Now note that as each node  $j \neq 1$ , independently and uniformly selects a set of  $s$  nodes,  $A^{(1)}$  has a binomial distribution with parameters  $n - 1$ , and  $\frac{s}{n - 1}$ . Therefore, for  $s > 0$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ \frac {1}{A ^ {(1)} + 1} \right] = \sum_ {k = 0} ^ {n - 1} \frac {1}{k + 1} \binom {n - 1} {k} \left(\frac {s}{n - 1}\right) ^ {k} \left(1 - \frac {s}{n - 1}\right) ^ {n - 1 - k} \\ = \frac {n - 1}{s n} \sum_ {k = 0} ^ {n - 1} \binom {n} {k + 1} \left(\frac {s}{n - 1}\right) ^ {k + 1} \left(1 - \frac {s}{n - 1}\right) ^ {n - 1 - k} \\ = \frac {n - 1}{s n} \left(1 - \left(1 - \frac {s}{n - 1}\right) ^ {n}\right). \\ \end{array}
$$
Also noting that
$$
\mathbb {E} \left[ \frac {A ^ {(1)}}{A ^ {(1)} + 1} \right] = 1 - \mathbb {E} \left[ \frac {1}{A ^ {(1)} + 1} \right],
$$
$$
\frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| y ^ {(i)} - \bar {x} \right\| ^ {2} \right] = \left[ \frac {1}{s} \left(1 - \left(1 - \frac {s}{n - 1}\right) ^ {n}\right) - \frac {1}{n - 1} \right] \frac {1}{n} \sum_ {i \in [ n ]} \left\| x ^ {(i)} - \bar {x} \right\| ^ {2}. \tag {25}
$$
Noting that  $\frac{1}{n}\sum_{i\in [n]}\mathbb{E}\left[\left\| y^{(i)} - \bar{y}\right\| ^2\right]\leq \frac{1}{n}\sum_{i\in [n]}\mathbb{E}\left[\left\| y^{(i)} - \bar{x}\right\| ^2\right]$ , from (24), and using Lemma 5, we obtain that
$$
\frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| y ^ {(i)} - y ^ {(j)} \right\| ^ {2} \right] \leq \left[ \frac {1}{s} \left(1 - \left(1 - \frac {s}{n - 1}\right) ^ {n}\right) - \frac {1}{n - 1} \right] \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x ^ {(i)} - x ^ {(j)} \right\| ^ {2} \right],
$$
Property (c):
$$
\begin{array}{l} \mathbb {E} \left[ \| \bar {y} - \bar {x} \| ^ {2} \right] = \mathbb {E} \left[ \left\| \frac {1}{n} \sum_ {i \in [ n ]} y ^ {(i)} - \bar {x} \right\| ^ {2} \right] \\ = \frac {1}{n ^ {2}} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| y ^ {(i)} - \bar {x} \right\| ^ {2} \right] + \frac {1}{n ^ {2}} \sum_ {i \neq j} \mathbb {E} \left[ \left\langle y ^ {(i)} - \bar {x}, y ^ {(j)} - \bar {x} \right\rangle \right]. \tag {26} \\ \end{array}
$$
Now recall that
$$
y ^ {(i)} - \bar {x} = \frac {1}{A ^ {(i)} + 1} \left((x ^ {(i)} - \bar {x}) + \sum_ {k \in [ n ] \backslash \{i \}} \mathcal {I} _ {k} ^ {(i)} (x ^ {(k)} - \bar {x})\right).
$$
$$
\begin{array}{l} \frac {1}{n ^ {2}} \sum_ {i \neq j} \mathbb {E} \left[ \left\langle y ^ {(i)} - \bar {x}, y ^ {(j)} - \bar {x} \right\rangle \right] = \frac {1}{n ^ {2}} \sum_ {i \in [ n ]} \sum_ {j \neq i} \mathbb {E} \left[ \frac {1}{(A ^ {(i)} + 1) (A ^ {(j)} + 1)} \right] \left\langle x ^ {(i)} - \bar {x}, x ^ {(j)} - \bar {x} \right\rangle \\ + \frac {2}{n ^ {2}} \sum_ {i \in [ n ]} \sum_ {j \neq i} \sum_ {\substack {k \neq i, k \neq j}} \mathbb {E} \left[ \frac {\mathcal {I} _ {k} ^ {(i)}}{(A ^ {(i)} + 1) (A ^ {(j)} + 1)} \right] \left\langle x ^ {(k)} - \bar {x}, x ^ {(j)} - \bar {x} \right\rangle \\ + \frac {2}{n ^ {2}} \sum_ {i \in [ n ]} \sum_ {j \neq i} \sum_ {k \neq i, k \neq j} \sum_ {l \neq i, l \neq j, l \neq k} \mathbb {E} \left[ \frac {\mathcal {I} _ {k} ^ {(i)} \mathcal {I} _ {l} ^ {(j)}}{\left(A ^ {(i)} + 1\right) \left(A ^ {(j)} + 1\right)} \right] \left\langle x ^ {(k)} - \bar {x}, x ^ {(l)} - \bar {x} \right\rangle \tag {27} \\ \end{array}
$$
Now note that by symmetry, for any  $i,j\in [n]$ , we have
$$
\mathbb {E} \left[ \frac {1}{(A ^ {(i)} + 1) (A ^ {(j)} + 1)} \right] = \mathbb {E} \left[ \frac {1}{(A ^ {(1)} + 1) (A ^ {(2)} + 1)} \right].
$$
$$
\mathbb {E} \left[ \frac {\mathcal {I} _ {k} ^ {(i)}}{(A ^ {(i)} + 1) (A ^ {(j)} + 1)} \right] = \mathbb {E} \left[ \frac {\mathcal {I} _ {3} ^ {(1)}}{(A ^ {(1)} + 1) (A ^ {(2)} + 1)} \right],
$$
$$
\mathbb {E} \left[ \frac {\mathcal {I} _ {k} ^ {(i)} \mathcal {I} _ {l} ^ {(j)}}{(A ^ {(i)} + 1) (A ^ {(j)} + 1)} \right] = \mathbb {E} \left[ \frac {\mathcal {I} _ {3} ^ {(1)} \mathcal {I} _ {4} ^ {(2)}}{(A ^ {(1)} + 1) (A ^ {(2)} + 1)} \right].
$$
This implies that all three terms in (27) can be written as
$$
c \sum_ {i \in [ n ]} \sum_ {j \neq i} \left\langle x ^ {(i)} - \bar {x}, x ^ {(j)} - \bar {x} \right\rangle ,
$$
where  $c$  is a positive constant. We also have
$$
\begin{array}{l} \sum_ {i \in [ n ]} \sum_ {j \neq i} \left\langle x ^ {(i)} - \bar {x}, x ^ {(j)} - \bar {x} \right\rangle = \sum_ {i \in [ n ]} \left\langle x ^ {(i)} - \bar {x}, \sum_ {j \neq i} (x ^ {(j)} - \bar {x}) \right\rangle \\ = - \sum_ {i \in [ n ]} \left\| x ^ {(i)} - \bar {x} \right\| ^ {2}. \\ \end{array}
$$
Therefore all the terms in (27) are non-positive. Combining this with (26), we obtain that
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \bar {y} - \bar {x} \right\| ^ {2} \right] \leq \frac {1}{n ^ {2}} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| y ^ {(i)} - \bar {x} \right\| ^ {2} \right] \\ \leq \frac {\beta_ {s}}{n} \cdot \frac {1}{n} \sum_ {i \in [ n ]} \left\| x ^ {(i)} - \bar {x} \right\| ^ {2}, \\ \end{array}
$$
where the second inequality uses (25). Combining this with Lemma 5 then concludes the proof.
Remark 2. Note that  $\beta_{s}$  computed in Lemma 1 is decreasing in  $s$  and increasing in  $n$ , therefore, for any  $s \geq 1$ , and  $n \geq 2$ , we have
$$
\begin{array}{l} \beta_ {s} \Big | _ {n <   \infty} \leq \lim  _ {n \rightarrow \infty} \beta_ {1} \\ = \lim  _ {n \rightarrow \infty} \left(1 - \left(1 - \frac {1}{n - 1}\right) ^ {n} - \frac {1}{n - 1}\right) \\ = 1 - \frac {1}{e}, \\ \end{array}
$$
where  $e$  is Euler's Number and we used the fact that  $\lim_{n\to \infty}\left(1 - \frac{1}{n}\right)^n = \frac{1}{e}$ . Similarly, for any  $s\geq 1$ , and  $n\geq 2$ , we have
$$
\left. \alpha_ {s} \right| _ {n <   \infty} \leq \lim  _ {n \rightarrow \infty} \alpha_ {1} = \frac {1}{2} <   1 - \frac {1}{e}.
$$
Proof. First note that for any  $i \in [n]$ , we have
$$
\begin{array}{l} g _ {t} ^ {(i)} - g _ {t} ^ {(j)} = g _ {t} ^ {(i)} - \nabla f ^ {(i)} \left(x _ {t} ^ {(i)}\right) + \nabla f ^ {(i)} \left(x _ {t} ^ {(i)}\right) - \nabla f ^ {(i)} (\bar {x} _ {t}) + \nabla f ^ {(i)} (\bar {x} _ {t}) \\ - \nabla f ^ {(j)} (\bar {x} _ {t}) + \nabla f ^ {(j)} (\bar {x} _ {t}) - \nabla f ^ {(j)} (x _ {t} ^ {(j)}) + \nabla f ^ {(j)} (x _ {t} ^ {(j)}) - g _ {t} ^ {(j)} \\ \end{array}
$$
Thus, using Lemma 4, we have
$$
\begin{array}{l} \left\| g _ {t} ^ {(i)} - g _ {t} ^ {(j)} \right\| ^ {2} \leq 5 \left\| g _ {t} ^ {(i)} - \nabla f ^ {(i)} \left(x _ {t} ^ {(i)}\right) \right\| ^ {2} + 5 \left\| \nabla f ^ {(i)} \left(x _ {t} ^ {(i)}\right) - \nabla f ^ {(i)} (\bar {x} _ {t}) \right\| ^ {2} \\ + 5 \left\| \nabla f ^ {(j)} \left(x _ {t} ^ {(j)}\right) - \nabla f ^ {(j)} (\bar {x} _ {t}) \right\| ^ {2} + 5 \left\| g _ {t} ^ {(j)} - \nabla f ^ {(j)} \left(x _ {t} ^ {(j)}\right) \right\| ^ {2} \\ + 5 \left\| \nabla f ^ {(i)} (\bar {x} _ {t}) - \nabla f ^ {(j)} (\bar {x} _ {t}) \right\| ^ {2}. \\ \end{array}
$$
Taking the conditional expectation, we have
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left\| g _ {t} ^ {(i)} - g _ {t} ^ {(j)} \right\| ^ {2} \right] \leq 5 \mathbb {E} _ {t} \left[ \left\| g _ {t} ^ {(i)} - \nabla f ^ {(i)} \left(x _ {t} ^ {(i)}\right) \right\| ^ {2} \right] + 5 \mathbb {E} _ {t} \left[ \left\| \nabla f ^ {(i)} \left(x _ {t} ^ {(i)}\right) - \nabla f ^ {(i)} (\bar {x} _ {t}) \right\| ^ {2} \right] \\ + 5 \mathbb {E} _ {t} \left[ \left\| \nabla f ^ {(j)} \left(x _ {t} ^ {(j)}\right) - \nabla f ^ {(j)} (\bar {x} _ {t}) \right\| ^ {2} \right] + 5 \mathbb {E} _ {t} \left[ \left\| g _ {t} ^ {(j)} - \nabla f ^ {(j)} \left(x _ {t} ^ {(j)}\right) \right\| ^ {2} \right] \\ + 5 \mathbb {E} _ {t} \left[ \left\| \nabla f ^ {(i)} (\bar {x} _ {t}) - \nabla f ^ {(j)} (\bar {x} _ {t}) \right\| ^ {2} \right] \tag {28} \\ \end{array}
$$
Now by Assumption 2, we have
$$
\mathbb {E} _ {t} \left[ \left\| g _ {t} ^ {(i)} - \nabla f ^ {(i)} \left(x _ {t} ^ {(i)}\right) \right\| ^ {2} \right] \leq \sigma^ {2}. \tag {29}
$$
By Assumption 1, we have
$$
\mathbb {E} _ {t} \left[ \left\| \nabla f ^ {(i)} \left(x _ {t} ^ {(i)}\right) - \nabla f ^ {(i)} (\bar {x} _ {t}) \right\| ^ {2} \right] \leq L ^ {2} \mathbb {E} _ {t} \left[ \left\| x _ {t} ^ {(i)} - \bar {x} _ {t} \right\| ^ {2} \right]. \tag {30}
$$
Thus, by Assumption 3, and Lemma 5, we obtain that
$$
\frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} _ {t} \left[ \left\| \nabla f ^ {(i)} (\bar {x} _ {t}) - \nabla f ^ {(j)} (\bar {x} _ {t}) \right\| ^ {2} \right] \leq 2 \mathcal {H} ^ {2}. \tag {31}
$$
Combining (28), (29), (30), and (31), and taking total expectation from both sides, we obtain that
$$
\frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| g _ {t} ^ {(i)} - g _ {t} ^ {(j)} \right\| ^ {2} \right] \leq \frac {1 0 L ^ {2}}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - \bar {x} _ {t} \right\| ^ {2} \right] + 1 0 \sigma^ {2} + 1 0 \mathcal {H} ^ {2}. \tag {32}
$$
Now Lemma 5 yields
$$
\frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| g _ {t} ^ {(i)} - g _ {t} ^ {(j)} \right\| ^ {2} \right] \leq \frac {5 L ^ {2}}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] + 1 0 \sigma^ {2} + 1 0 \mathcal {H} ^ {2}. \tag {33}
$$
We now analyze  $\frac{1}{n^2}\sum_{i,j\in [n]}\mathbb{E}\left[\left\| x_t^{(i)} - x_t^{(j)}\right\|^2\right]$ . From Algorithm 1, recall that for all  $i\in [n]$ , we have  $x_{t + 1 / 2}^{(i)} = x_t^{(i)} - \gamma g_t^{(i)}$ . We obtain for all  $i$ ,  $j\in [n]$ , that
$$
\begin{array}{l} \mathbb {E} \left[ \left\| x _ {t + 1 / 2} ^ {(i)} - x _ {t + 1 / 2} ^ {(j)} \right\| ^ {2} \right] \leq \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} - \gamma \left(g _ {t} ^ {(i)} - g _ {t} ^ {(j)}\right) \right\| ^ {2} \right] \tag {34} \\ \leq (1 + c) \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] + \left(1 + \frac {1}{c}\right) \gamma^ {2} \mathbb {E} \left[ \left\| g _ {t} ^ {(i)} - g _ {t} ^ {(j)} \right\| ^ {2} \right], \\ \end{array}
$$
where we used the fact that  $(x + y)^2 \leq (1 + c)x^2 + (1 + 1 / c)y^2$  for any  $c > 0$ . Now recall that  $\eta_s = \alpha_s$  for EL-Oracle, and  $\eta_s = \beta_s$  for EL-Local. Therefore, by Property (b) of Lemma 1, for both variant of the algorithm, we have
$$
\frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| x _ {t + 1} ^ {(i)} - \bar {x} _ {t + 1} \right\| ^ {2} \right] \leq \frac {\eta_ {s}}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| x _ {t + 1 / 2} ^ {(i)} - \bar {x} _ {t + 1 / 2} \right\| ^ {2} \right]
$$
Then, applying Lemma 5, we have
$$
\frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t + 1} ^ {(i)} - x _ {t + 1} ^ {(j)} \right\| ^ {2} \right] \leq \frac {\eta_ {s}}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t + 1 / 2} ^ {(i)} - x _ {t + 1 / 2} ^ {(j)} \right\| ^ {2} \right].
$$
Combining this with (34), we obtain that
$$
\begin{array}{l} \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t + 1} ^ {(i)} - x _ {t + 1} ^ {(j)} \right\| ^ {2} \right] \leq (1 + c) \eta_ {s} \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] \\ + \left(1 + \frac {1}{c}\right) \eta_ {s} \gamma^ {2} \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| g _ {t} ^ {(i)} - g _ {t} ^ {(j)} \right\| ^ {2} \right]. \\ \end{array}
$$
For  $c = \frac{1 - \eta_s}{4\eta_s}$ , we obtain that
$$
\begin{array}{l} \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t + 1} ^ {(i)} - x _ {t + 1} ^ {(j)} \right\| ^ {2} \right] \leq \frac {1 + 3 \eta_ {s}}{4} \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] \\ + \frac {1 + 3 \eta_ {s}}{1 - \eta_ {s}} \eta_ {s} \gamma^ {2} \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| g _ {t} ^ {(i)} - g _ {t} ^ {(j)} \right\| ^ {2} \right]. \\ \end{array}
$$
Combining this with (33), we obtain that
$$
\begin{array}{l} \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t + 1} ^ {(i)} - x _ {t + 1} ^ {(j)} \right\| ^ {2} \right] \leq \frac {1 + 3 \eta_ {s}}{4} \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] \\ + \frac {1 + 3 \eta_ {s}}{1 - \eta_ {s}} \eta_ {s} \gamma^ {2} \left(\frac {5 L ^ {2}}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] + 1 0 \sigma^ {2} + 1 0 \mathcal {H} ^ {2}\right) \\ = \left(\frac {1 + 3 \eta_ {s}}{4} + 5 \frac {1 + 3 \eta_ {s}}{1 - \eta_ {s}} \eta_ {s} \gamma^ {2} L ^ {2}\right) \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] \\ + \frac {1 + 3 \eta_ {s}}{1 - \eta_ {s}} \eta_ {s} \gamma^ {2} \left(1 0 \sigma^ {2} + 1 0 \mathcal {H} ^ {2}\right). \\ \end{array}
$$
Now note that for by Remark 2, for both variants of the algorithm, we have  $\eta_s \leq 1 - \frac{1}{e}$ , which implies that
$$
\gamma^ {2} \leq \frac {1}{(2 0 L) ^ {2}} \leq \frac {(1 - \eta_ {s}) ^ {2}}{2 0 \eta_ {s} (1 + 3 \eta_ {s}) L ^ {2}}.
$$
$$
\frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t + 1} ^ {(i)} - x _ {t + 1} ^ {(j)} \right\| ^ {2} \right] \leq \frac {1 + \eta_ {s}}{2} \frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] + \frac {1 + 3 \eta_ {s}}{1 - \eta_ {s}} \eta_ {s} \gamma^ {2} \left(1 0 \sigma^ {2} + 1 0 \mathcal {H} ^ {2}\right).
$$
Unrolling the recursion, we obtain that
$$
\frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] \leq 2 0 \frac {1 + 3 \eta_ {s}}{(1 - \eta_ {s}) ^ {2}} \eta_ {s} \gamma^ {2} \left(\sigma^ {2} + \mathcal {H} ^ {2}\right).
$$
Combining this with (32), we obtain that
$$
\frac {1}{n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| g _ {t} ^ {(i)} - g _ {t} ^ {(j)} \right\| ^ {2} \right] \leq 1 5 \left(\sigma^ {2} + \mathcal {H} ^ {2}\right).
$$
This is the desired result.
B.3 Proof of Lemma 3
$$
\begin{array}{l} \mathbb {E} \left[ \| \nabla F (\bar {x} _ {t}) \| ^ {2} \right] \leq \frac {2}{\gamma} \mathbb {E} [ F (\bar {x} _ {t}) - F (\bar {x} _ {t + 1}) ] + \frac {L ^ {2}}{2 n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] \\ + 2 L \gamma \frac {\sigma^ {2}}{n} + \frac {2 L}{\gamma} \mathbb {E} \left[ \left\| \bar {x} _ {t + 1} - \bar {x} _ {t + 1 / 2} \right\| ^ {2} \right]. \\ \end{array}
$$
Proof. Consider an arbitrary  $t \in [T]$ . Note that, we do not necessarily have  $\bar{x}_{t+1} = \bar{x}_{t+1/2}$ . But by Lemma 1, we have  $\mathbb{E}\left[\bar{x}_{t+1}\right] = \mathbb{E}\left[\bar{x}_{t+1/2}\right]$ . Now by the smoothness of the loss function (Assumption 1), we have
$$
\begin{array}{l} F (\bar {x} _ {t + 1}) - F (\bar {x} _ {t}) \leq \left\langle \bar {x} _ {t + 1} - \bar {x} _ {t}, \nabla F (\bar {x} _ {t}) \right\rangle + \frac {L}{2} \left\| \bar {x} _ {t + 1} - \bar {x} _ {t} \right\| ^ {2} \\ = \left\langle \bar {x} _ {t + 1} - \bar {x} _ {t + 1 / 2} + \bar {x} _ {t + 1 / 2} - \bar {x} _ {t}, \nabla F (\bar {x} _ {t}) \right\rangle + \frac {L}{2} \left\| \bar {x} _ {t + 1} - \bar {x} _ {t + 1 / 2} + \bar {x} _ {t + 1 / 2} - \bar {x} _ {t} \right\| ^ {2}. \\ \end{array}
$$
Now denoting  $\overline{\nabla F}_t = \frac{1}{n}\sum_{i\in [n]}\nabla f^{(i)}\left(x_t^{(i)}\right)$ ,  $\bar{g}_t = \frac{1}{n}\sum_{i\in [n]}g_t^{(i)}$  and taking the conditional expectation, we have
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ F (\bar {x} _ {t + 1}) \right] - F (\bar {x} _ {t}) \leq \left\langle \mathbb {E} _ {t} \left[ \bar {x} _ {t + 1 / 2} - \bar {x} _ {t} \right], \nabla F (\bar {x} _ {t}) \right\rangle + \frac {L}{2} \mathbb {E} _ {t} \left[ \left\| \bar {x} _ {t + 1} - \bar {x} _ {t + 1 / 2} + \bar {x} _ {t + 1 / 2} - \bar {x} _ {t} \right\| ^ {2} \right] \\ \stackrel {(a)} {\leq} - \gamma \left\langle \overline {{\nabla F}} _ {t}, \nabla F \left(\bar {x} _ {t}\right) \right\rangle + L \gamma^ {2} \mathbb {E} _ {t} \left[ \left\| \bar {g} _ {t} \right\| ^ {2} \right] + L \mathbb {E} _ {t} \left[ \left\| \bar {x} _ {t + 1} - \bar {x} _ {t + 1 / 2} \right\| ^ {2} \right] \\ \stackrel {(b)} {\leq} - \gamma \left\langle \overline {{\nabla F}} _ {t}, \nabla F \left(\bar {x} _ {t}\right) \right\rangle + L \gamma^ {2} \mathbb {E} _ {t} \left[ \left\| \overline {{\nabla F}} _ {t} \right\| ^ {2} \right] + L \gamma^ {2} \frac {\sigma^ {2}}{n} + L \mathbb {E} _ {t} \left[ \left\| \bar {x} _ {t + 1} - \bar {x} _ {t + 1 / 2} \right\| ^ {2} \right]. \tag {35} \\ \end{array}
$$
where (a) uses Young's inequality and (b) is based on the facts that by Assumption 2, we have  $\mathbb{E}_t[\overline{g}_t] = \overline{\nabla F}_t$ , and  $\mathbb{E}_t\left[\left\| \overline{g}_t - \overline{\nabla F}_t\right\|^2\right] \leq \frac{\sigma^2}{n}$ . Now note that using the fact that  $\gamma \leq \frac{1}{2L}$ , we have
$$
\begin{array}{l} - \gamma \left\langle \overline {{\nabla F}} _ {t}, \nabla F (\bar {x} _ {t}) \right\rangle + L \gamma^ {2} \left\| \overline {{\nabla F}} _ {t} \right\| ^ {2} \leq \frac {\gamma}{2} \left(- 2 \left\langle \overline {{\nabla F}} _ {t}, \nabla F (\bar {x} _ {t}) \right\rangle + \left\| \overline {{\nabla F}} _ {t} \right\| ^ {2}\right) \\ = \frac {\gamma}{2} \left(- \| \nabla F (\bar {x} _ {t}) \| ^ {2} + \left\| \nabla F (\bar {x} _ {t}) - \overline {{\nabla F}} _ {t} \right\| ^ {2}\right). \tag {36} \\ \end{array}
$$
Combining (35), and (36), we obtain that
$$
\mathbb {E} _ {t} \left[ F (\bar {x} _ {t + 1}) \right] - F (\bar {x} _ {t}) \leq - \frac {\gamma}{2} \left\| \nabla F (\bar {x} _ {t}) \right\| ^ {2} + \frac {\gamma}{2} \left\| \nabla F (\bar {x} _ {t}) - \overline {{\nabla F}} _ {t} \right\| ^ {2} + L \gamma^ {2} \frac {\sigma^ {2}}{n} + L \mathbb {E} _ {t} \left[ \left\| \bar {x} _ {t + 1} - \bar {x} _ {t + 1 / 2} \right\| ^ {2} \right].
$$
Taking total expectation, we obtain that
$$
\begin{array}{l} \mathbb {E} \left[ F (\bar {x} _ {t + 1}) - F (\bar {x} _ {t}) \right] \leq - \frac {\gamma}{2} \mathbb {E} \left[ \| \nabla F (\bar {x} _ {t}) \| ^ {2} \right] + \frac {\gamma}{2} \mathbb {E} \left[ \| \nabla F (\bar {x} _ {t}) - \overline {{\nabla F}} _ {t} \| ^ {2} \right] \\ + L \gamma^ {2} \frac {\sigma^ {2}}{n} + L \mathbb {E} \left[ \left\| \bar {x} _ {t + 1} - \bar {x} _ {t + 1 / 2} \right\| ^ {2} \right]. \tag {37} \\ \end{array}
$$
Now, note that
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \overline {{\nabla F}} _ {t} - \nabla F (\bar {x} _ {t}) \right\| ^ {2} \right] = \mathbb {E} \left[ \left\| \frac {1}{n} \sum_ {i \in [ n ]} \nabla f ^ {(i)} (x _ {t} ^ {(i)}) - \frac {1}{n} \sum_ {i \in [ n ]} \nabla f ^ {(i)} (\bar {x} _ {t}) \right\| ^ {2} \right] \\ = \mathbb {E} \left[ \left\| \frac {1}{n} \sum_ {i \in [ n ]} \left(\nabla f ^ {(i)} \left(x _ {t} ^ {(i)}\right) - \nabla f ^ {(i)} (\bar {x} _ {t})\right) \right\| ^ {2} \right] \\ \leq \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| \nabla f ^ {(i)} \left(x _ {t} ^ {(i)}\right) - \nabla f ^ {(i)} (\bar {x} _ {t}) \right\| ^ {2} \right] \\ \stackrel {(a)} {\leq} \frac {L ^ {2}}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - \bar {x} _ {t} \right\| ^ {2} \right]. \\ \stackrel {(b)} {\leq} \frac {L ^ {2}}{2 n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right], \\ \end{array}
$$
where (a) uses Assumption 1, in (b), we used Lemma 5. Combining this with (37), we obtain that
$$
\begin{array}{l} \mathbb {E} \left[ F (\bar {x} _ {t + 1}) - F (\bar {x} _ {t}) \right] \leq - \frac {\gamma}{2} \mathbb {E} \left[ \| \nabla F (\bar {x} _ {t}) \| ^ {2} \right] + \frac {\gamma}{2} \frac {L ^ {2}}{2 n ^ {2}} \sum_ {i, j \in [ n ]} \mathbb {E} \left[ \left\| x _ {t} ^ {(i)} - x _ {t} ^ {(j)} \right\| ^ {2} \right] \\ + L \gamma^ {2} \frac {\sigma^ {2}}{n} + L \mathbb {E} \left[ \left\| \bar {x} _ {t + 1} - \bar {x} _ {t + 1 / 2} \right\| ^ {2} \right]. \\ \end{array}
$$
Rearranging the terms we have
Table 3: Summary of learning parameters used for evaluation. For each dataset, batch size (b), and the number of training steps per communication round (r) are presented.
TASK	DATASET	MODEL	b	r	TRAINING
SAMPLES	TOTAL
PARAMETERS
Image Classification	CIFAR-10	GN-LeNet	8	3	50000	89834
Image Classification	FEMNIST	CNN	16	7	734463	1690046
Figure 6: The test accuracy for EL-Oracle, 7-Regular static topologies, and 8-U-EquiStatic topologies with different learning rates  $\gamma$ , with the CIFAR-10 dataset.
which is the lemma.
C Experimental Details and Further Evaluation
C.1 Experiment setup and learning rate tuning