SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$^{7}$ Code: https://github.com/mysteryresearcher/dasha-partial-participation
A.1 Experiments in Partial Participation Setting
In this experiments, we compare our new algorithm DASHA-PP with previous baselines MARINA and FRECON in the partial participation setting. We consider MARINA and FRECON because they are the previous SOTA methods in the partial participation setting with compression. We investigate the same optimization problem and setup as in Section A of the paper. All methods use the RandK compressor in these experiments.
1. Finite-Sum Setting. We now consider the function from (11). In Figures 2 and 3, we compare all three methods in the finite-sum setting on two different datasets: real-sim and MNIST. The parameter  $s$  is the number of clients participating in each round that are selected randomly using the  $s$ -nice sampling (server chooses uniformly  $s$  nodes without replacement). We can see that DASHA-PP converges faster than MARINA. Since FRECON does not support variance reduction of stochastic gradients, it converges to less accurate solutions.
(a)  $1\%$  of nodes participating
(b)  $10\%$  of nodes participating
(c)  $90\%$  of nodes participating
(a)  $10\%$  of nodes participating
Figure 2: Classification task on real-sim
(b)  $50\%$  of nodes participating
(c)  $90\%$  of nodes participating
Figure 3: Classification task on MNIST
2. Stochastic Setting. In Figures 4 and 5, we consider the stochastic setting with the function from (11). We can see that DASHA-PP converges to high accuracy solutions, unlike FRECON. Moreover, DASHA-PP improves the convergence rates of MARINA.
(b)  $50\%$  of nodes participating
(c)  $100\%$  of nodes participating
Figure 4: Classification task on real-sim
(a)  $10\%$  of nodes participating
Figure 5: Classification task on MNIST
B Original DASHA and DASHA-MVR Methods
To simplify the discussion and explanation from the main part, we present the algorithms from (Tyurin and Richtárik, 2023)
Algorithm 6 DASHA
1: Input: starting point  $x^0 \in \mathbb{R}^d$ , stepsize  $\gamma > 0$ , momentum  $a \in (0,1]$ , number of iterations  $T \geq 1$
2: Initialize  $g_i^0 \in \mathbb{R}^d$  on the nodes and  $g^0 = \frac{1}{n}\sum_{i=1}^{n} g_i^0$  on the server
3: for  $t = 0,1,\ldots ,T - 1$  do
4:  $x^{t + 1} = x^t -\gamma g^t$
5: Broadcast  $x^{t + 1}$  and  $x^t$
6: for  $i = 1,\dots ,n$  in parallel do
7:  $m_{i}^{t + 1} = \mathcal{C}_{i}\left(\nabla f_{i}(x^{t + 1}) - \nabla f_{i}(x^{t}) - a(g_{i}^{t} - \nabla f_{i}(x^{t}))\right)$
8:  $g_{i}^{t + 1} = g_{i}^{t} + m_{i}^{t + 1}$
9: Send  $m_i^{t+1}$  to the server
0: end for
1:  $g^{t + 1} = g^t +\frac{1}{n}\sum_{i = 1}^n m_i^{t + 1}$
2: end for
3: Output:  $\hat{x}^T$  chosen uniformly at random from  $\{x^t\}_{k=0}^{T-1}$
Algorithm 7 DASHA-MVR (with batch size  $B = 1$ )
1: Input: starting point  $x^0 \in \mathbb{R}^d$ , stepsize  $\gamma > 0$ , momentum  $a, b \in (0,1]$ , number of iterations  $T \geq 1$
2: Initialize  $g_i^0 \in \mathbb{R}^d$  on the nodes and  $g^0 = \frac{1}{n}\sum_{i=1}^{n} g_i^0$  on the server
3: for  $t = 0,1,\ldots ,T - 1$  do
4:  $x^{t + 1} = x^t -\gamma g^t$
5: Broadcast  $x^{t + 1}$  and  $x^t$
6: for  $i = 1,\dots ,n$  in parallel do
7:  $h_i^{t + 1} = \nabla f_i(x^{t + 1};\xi_i^{t + 1}) + (1 - b)(h_i^t -\nabla f_i(x^t;\xi_i^{t + 1}))$ $\xi_{i}^{t + 1}\sim \mathcal{D}_{i}$
8:  $m_{i}^{t + 1} = \mathcal{C}_{i}\left(h_{i}^{t + 1} - h_{i}^{t} - a\left(g_{i}^{t} - h_{i}^{t}\right)\right)$
9:  $g_{i}^{t + 1} = g_{i}^{t} + m_{i}^{t + 1}$
0: Send  $m_i^{t + 1}$  to the server
1: end for
2:  $g^{t + 1} = g^t +\frac{1}{n}\sum_{i = 1}^n m_i^{t + 1}$
3: end for
14: Output:  $\hat{x}^T$  chosen uniformly at random from  $\{x^t\}_{k=0}^{T-1}$
C Problem of Estimating the Mean in the Partial Participation Setting
We now provide the example to explain why the only choice of  $B = \mathcal{O}\left(\min \left\{\frac{1}{p_{\mathrm{a}}}\sqrt{\frac{m}{n}},\frac{L_{\max}^{2}}{\mathbb{1}_{p_{\mathrm{a}}}^{2}\hat{L}^{2}}\right\}\right)$  and  $B = \mathcal{O}\left(\min \left\{\frac{\sigma}{p_{\mathrm{a}}\sqrt{\varepsilon n}},\frac{L_{\sigma}^{2}}{\mathbb{1}_{p_{\mathrm{a}}}^{2}\hat{L}^{2}}\right\}\right)$  in DASHA-PP-PAGE and DASHA-PP-MVR, accordingly, guarantees the degeneration up to  $1 / p_{\mathrm{a}}$ . This is surprising, because in methods with the variance reduction of stochastic gradients (Li et al., 2021a; Tyurin and Richtárik, 2023) we can take the size of batch size  $B = \mathcal{O}\left(\sqrt{\frac{m}{n}}\right)$  and  $B = \mathcal{O}\left(\frac{\sigma}{\sqrt{\varepsilon n}}\right)$  and guarantee the optimality. Note that the smaller the batch size  $B$ , the more the server and the nodes have to communicate to get  $\varepsilon$ -solution.
Let us consider the task of estimating the mean of vectors in the distributed setting. Suppose that we have  $n$  nodes, and each of them contains  $m$  vectors  $\{x_{ij}\}_{j=1}^{m}$ , where  $x_{ij} \in \mathbb{R}^d$  for all  $i \in [n], j \in [m]$ . First, let us consider that each node samples a mini-batch  $I^i$  of size  $B$  with replacement and sends it to the server. Then the server calculates the mean of the mini-batches from nodes. One can easily show that the variance of the estimator is
$$
\begin{array}{l} \left. \operatorname {E} \left[ \left\| \frac {1}{n B} \sum_ {i = 1} ^ {n} \sum_ {j \in I ^ {i}} x _ {i j} - \frac {1}{n m} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {m} x _ {i j} \right\| ^ {2} \right] \right. \tag {13} \\ = \frac {1}{n B} \frac {1}{n m} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {m} \left\| x _ {i j} - \frac {1}{m} \sum_ {j = 1} ^ {m} x _ {i j} \right\| ^ {2}. \\ \end{array}
$$
Next, we consider the same task in the partial participation setting with  $s$ -nice sampling, i.e., we sample a random set  $S \subset [n]$  of  $s \in [n]$  nodes without replacement and receive the mini-batches only from the sampled nodes. Such sampling of nodes satisfy Assumption 8 with  $p_{\mathrm{a}} = s / n$  and  $p_{\mathrm{a}} = s(s - 1) / n(n - 1)$ . In this case, the variance of the estimator (See Lemma 1 with  $r_i = 0$  and  $s_i = \sum_{j \in I^i} x_{ij}$ ) is
$$
\begin{array}{l} \left. \operatorname {E} \left[ \left\| \frac {1}{s B} \sum_ {i \in S} \sum_ {j \in I ^ {i}} x _ {i j} - \frac {1}{n m} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {m} x _ {i j} \right\| ^ {2} \right] \right. \tag {14} \\ = \frac {1}{s B} \underbrace {\frac {1}{n m} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {m} \left\| x _ {i j} - \frac {1}{m} \sum_ {j = 1} ^ {m} x _ {i j} \right\| ^ {2}} _ {\mathcal {L} _ {\max} ^ {2}} \\ + \frac {n - s}{s (n - 1)} \underbrace {\frac {1}{n} \sum_ {i = 1} ^ {n} \left\| \frac {1}{m} \sum_ {j = 1} ^ {m} x _ {i j} - \frac {1}{n m} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {m} x _ {i j} \right\| ^ {2}} _ {\hat {\mathcal {L}} ^ {2}}. \\ \end{array}
$$
Let us assume that  $s \leq n/2$ . Note that (13) scales with any  $B \geq 1$ , while (14) only scales when  $B = \mathcal{O}\left(\mathcal{L}_{\max}^2/\widehat{\mathcal{L}}^2\right)$ . In other words, for large enough  $B$ , the variance in (14) does not significantly improve with the growth of  $B$  due to the term  $\widehat{\mathcal{L}}^2$ . In our proof, due to partial participation, the variance from (14) naturally appears, and we get the same effect. As was mentioned in Sections 6.2 and 6.3, it can be seen in our convergence rate bounds.
D Auxiliary facts
We list auxiliary facts that we use in our proofs:
1. For all  $x,y\in \mathbb{R}^d$  , we have
$$
\left\| x + y \right\| ^ {2} \leq 2 \left\| x \right\| ^ {2} + 2 \left\| y \right\| ^ {2} \tag {15}
$$
2. Let us take a random vector  $\xi \in \mathbb{R}^d$ , then
$$
\operatorname {E} \left[ \| \xi \| ^ {2} \right] = \operatorname {E} \left[ \| \xi - \operatorname {E} [ \xi ] \| ^ {2} \right] + \| \operatorname {E} [ \xi ] \| ^ {2}. \tag {16}
$$
D.1 Sampling Lemma
This section provides a lemma that we regularly use in our proofs, and it is useful for samplings that satisfy Assumption 8.
Lemma 1. Suppose that a set  $S$  is a random subset of a set  $[n]$  such that
1.  $\mathbf{Prob}(i\in S) = p_{\mathrm{a}},\quad \forall i\in [n],$
2.  $\mathbf{Prob}(i\in S,j\in S) = p_{\mathrm{aa}},\quad \forall i\neq j\in [n],$
3.  $p_{\mathrm{aa}} \leq p_{\mathrm{a}}^{2}$ ,
where  $p_{\mathrm{a}} \in (0,1]$  and  $p_{\mathrm{aa}} \in [0,1]$ . Let us take random independent vectors  $s_i \in \mathbb{R}^d$  for all  $i \in [n]$ , nonrandom vector  $r_i \in \mathbb{R}^d$  for all  $i \in [n]$ , and random vectors
$$
v _ {i} = \left\{ \begin{array}{l} r _ {i} + \frac {1}{p _ {\mathrm {a}}} s _ {i}, i \in S, \\ r _ {i}, i \notin S, \end{array} \right.
$$
$$
\begin{array}{l} \operatorname {E} \left[ \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i} - \operatorname {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i} \right] \right\| ^ {2} \right] \\ = \frac {1}{n ^ {2} p _ {\mathrm {a}}} \sum_ {i = 1} ^ {n} \operatorname {E} \left[ \left\| s _ {i} - \operatorname {E} [ s _ {i} ] \right\| ^ {2} \right] + \frac {p _ {\mathrm {a}} - p _ {\mathrm {a a}}}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| \operatorname {E} [ s _ {i} ] \right\| ^ {2} + \frac {p _ {\mathrm {a a}} - p _ {\mathrm {a}} ^ {2}}{p _ {\mathrm {a}} ^ {2}} \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \operatorname {E} [ s _ {i} ] \right\| \\ \leq \frac {1}{n ^ {2} p _ {\mathrm {a}}} \sum_ {i = 1} ^ {n} \operatorname {E} \left[ \| s _ {i} - \operatorname {E} [ s _ {i} ] \| ^ {2} \right] + \frac {p _ {\mathrm {a}} - p _ {\mathrm {a a}}}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \| \operatorname {E} [ s _ {i} ] \| ^ {2}. \\ \end{array}
$$
Proof. Let us define additional constants  $p_{\mathrm{an}}$  and  $p_{\mathrm{nn}}$ , such that
1.  $\mathbf{Prob}(i\in S,j\notin S) = p_{\mathrm{an}},\quad \forall i\neq j\in [n],$
2.  $\mathbf{Prob}(i\notin S,j\notin S) = p_{\mathrm{nn}},\quad \forall i\neq j\in [n].$
Note, that
$$
p _ {\mathrm {a}} = p _ {\mathrm {a a}} + p _ {\mathrm {a n}} \tag {17}
$$
$$
p _ {\mathrm {n n}} = 1 - p _ {\mathrm {a a}} - 2 p _ {\mathrm {a n}}. \tag {18}
$$
Using the law of total expectation and
$$
\operatorname {E} \left[ v _ {i} \right] = p _ {\mathrm {a}} \left(r _ {i} + \operatorname {E} \left[ \frac {1}{p _ {\mathrm {a}}} s _ {i} \right]\right) + (1 - p _ {\mathrm {a}}) r _ {i} = r _ {i} + \operatorname {E} \left[ s _ {i} \right],
$$
$$
\begin{array}{l} \mathrm {E} \left[ \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i} - \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i} \right] \right\| ^ {2} \right] \\ = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathrm {E} \left[ \| v _ {i} - \left(r _ {i} + \mathrm {E} [ s _ {i} ]\right) \| ^ {2} \right] \\ + \frac {1}{n ^ {2}} \sum_ {i \neq j} ^ {n} \operatorname {E} \left[ \langle v _ {i} - (r _ {i} + \operatorname {E} [ s _ {i} ]), v _ {j} - (r _ {j} + \operatorname {E} [ s _ {j} ]) \rangle \right] \\ = \frac {p _ {\mathrm {a}}}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathrm {E} \left[ \left\| r _ {i} + \frac {1}{p _ {\mathrm {a}}} s _ {i} - \left(r _ {i} + \mathrm {E} [ s _ {i} ]\right) \right\| ^ {2} \right] \\ + \frac {1 - p _ {\mathrm {a}}}{n ^ {2}} \sum_ {i = 1} ^ {n} \| r _ {i} - \left(r _ {i} + \operatorname {E} [ s _ {i} ]\right) \| ^ {2} \\ + \frac {p _ {\mathrm {a a}}}{n ^ {2}} \sum_ {i \neq j} ^ {n} \mathrm {E} \left[ \left\langle r _ {i} + \frac {1}{p _ {\mathrm {a}}} s _ {i} - (r _ {i} + \mathrm {E} [ s _ {i} ]), r _ {j} + \frac {1}{p _ {\mathrm {a}}} s _ {j} - (r _ {j} + \mathrm {E} [ s _ {j} ]) \right\rangle \right] \\ + \frac {2 p _ {\mathrm {a n}}}{n ^ {2}} \sum_ {i \neq j} ^ {n} \mathrm {E} \left[ \left\langle r _ {i} + \frac {1}{p _ {\mathrm {a}}} s _ {i} - (r _ {i} + \mathrm {E} [ s _ {i} ]), r _ {j} - (r _ {j} + \mathrm {E} [ s _ {j} ]) \right\rangle \right] \\ + \frac {p _ {\mathrm {n n}}}{n ^ {2}} \sum_ {i \neq j} ^ {n} \left\langle r _ {i} - \left(r _ {i} + \operatorname {E} \left[ s _ {i} \right]\right), r _ {j} - \left(r _ {j} + \operatorname {E} \left[ s _ {j} \right]\right) \right\rangle . \\ \end{array}
$$
From the independence of random vectors  $s_i$ , we obtain
$$
\begin{array}{l} \mathrm {E} \left[ \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i} - \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i} \right] \right\| ^ {2} \right] \\ = \frac {p _ {\mathrm {a}}}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathrm {E} \left[ \left\| \frac {1}{p _ {\mathrm {a}}} s _ {i} - \mathrm {E} [ s _ {i} ] \right\| ^ {2} \right] \\ + \frac {1 - p _ {\mathrm {a}}}{n ^ {2}} \sum_ {i = 1} ^ {n} \| \operatorname {E} [ s _ {i} ] \| ^ {2} \\ + \frac {p _ {\mathrm {a a}} (1 - p _ {\mathrm {a}}) ^ {2}}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i \neq j} ^ {n} \langle \mathrm {E} [ s _ {i} ], \mathrm {E} [ s _ {j} ] \rangle \\ + \frac {2 p _ {\mathrm {a n}} (p _ {\mathrm {a}} - 1)}{n ^ {2} p _ {\mathrm {a}}} \sum_ {i \neq j} ^ {n} \langle \operatorname {E} [ s _ {i} ], \operatorname {E} [ s _ {j} ] \rangle \\ + \frac {p _ {\mathrm {n n}}}{n ^ {2}} \sum_ {i \neq j} ^ {n} \left\langle \operatorname {E} \left[ s _ {i} \right], \operatorname {E} \left[ s _ {j} \right] \right\rangle . \\ \end{array}
$$
Using (17) and (18), we have
$$
\begin{array}{l} \mathrm {E} \left[ \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i} - \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i} \right] \right\| ^ {2} \right] \\ = \frac {p _ {\mathrm {a}}}{n ^ {2}} \sum_ {i = 1} ^ {n} \operatorname {E} \left[ \left\| \frac {1}{p _ {\mathrm {a}}} s _ {i} - \operatorname {E} [ s _ {i} ] \right\| ^ {2} \right] \\ + \frac {1 - p _ {\mathrm {a}}}{n ^ {2}} \sum_ {i = 1} ^ {n} \| \operatorname {E} [ s _ {i} ] \| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} + \frac {p _ {\mathrm {a a}} - p _ {\mathrm {a}} ^ {2}}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i \neq j} ^ {n} \langle \mathrm {E} [ s _ {i} ], \mathrm {E} [ s _ {j} ] \rangle \\ \stackrel {(1 6)} {=} \frac {1}{n ^ {2} p _ {\mathrm {a}}} \sum_ {i = 1} ^ {n} \operatorname {E} \left[ \| s _ {i} - \operatorname {E} [ s _ {i} ] \| ^ {2} \right] \\ + \frac {1 - p _ {\mathrm {a}}}{n ^ {2} p _ {\mathrm {a}}} \sum_ {i = 1} ^ {n} \| \operatorname {E} [ s _ {i} ] \| ^ {2} \\ + \frac {p _ {\mathrm {a a}} - p _ {\mathrm {a}} ^ {2}}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i \neq j} ^ {n} \langle \operatorname {E} [ s _ {i} ], \operatorname {E} [ s _ {j} ] \rangle \\ = \frac {1}{n ^ {2} p _ {\mathrm {a}}} \sum_ {i = 1} ^ {n} \mathrm {E} \left[ \| s _ {i} - \mathrm {E} [ s _ {i} ] \| ^ {2} \right] \\ + \frac {p _ {\mathrm {a}} - p _ {\mathrm {a a}}}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \| \mathrm {E} [ s _ {i} ] \| ^ {2} \\ + \frac {p _ {\mathrm {a a}} - p _ {\mathrm {a}} ^ {2}}{p _ {\mathrm {a}} ^ {2}} \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \operatorname {E} [ s _ {i} ] \right\|. \\ \end{array}
$$
Finally, using that  $p_{\mathrm{aa}} \leq p_{\mathrm{a}}^2$ , we have
$$
\begin{array}{l} \mathrm {E} \left[ \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i} - \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i} \right] \right\| ^ {2} \right] \\ \leq \frac {1}{n ^ {2} p _ {\mathrm {a}}} \sum_ {i = 1} ^ {n} \operatorname {E} \left[ \| s _ {i} - \operatorname {E} [ s _ {i} ] \| ^ {2} \right] + \frac {p _ {\mathrm {a}} - p _ {\mathrm {a a}}}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \| \operatorname {E} [ s _ {i} ] \| ^ {2}. \\ \end{array}
$$
D.2 Compressors Facts
We define the Rand  $K$  compressor that chooses without replacement  $K$  coordinates, scales them by a constant factor to preserve unbiasedness and zero-out other coordinates.
Definition 5. Let us take a random subset  $S$  from  $[d]$ ,  $|S| = K$ ,  $K \in [d]$ . We say that a stochastic mapping  $\mathcal{C} : \mathbb{R}^d \to \mathbb{R}^d$  is Rand $K$  if
$$
\mathcal {C} (x) = \frac {d}{K} \sum_ {j \in S} x _ {j} e _ {j},
$$
where  $\{e_i\}_{i = 1}^d$  is the standard unit basis.
Theorem 6. If  $\mathcal{C}$  is RandK, then  $\mathcal{C} \in \mathbb{U}\left(\frac{d}{k} - 1\right)$ .
See the proof in (Beznosikov et al., 2020).
There are three different sources of randomness in Algorithm 1: the first one from vectors  $\{k_i^{t + 1}\}_{i = 1}^n$ , the second one from compressors  $\{\mathcal{C}_i\}_{i = 1}^n$ , and the third one from availability of nodes. We define  $\mathrm{E}_k[\cdot ]$ ,  $\mathrm{E}_{\mathcal{C}}[\cdot ]$  and  $\mathrm{E}_{p_{\mathrm{a}}}[\cdot ]$  to be conditional expectations w.r.t.  $\{k_i^{t + 1}\}_{i = 1}^n$ ,  $\{\mathcal{C}_i\}_{i = 1}^n$ , and availability, accordingly, conditioned on all previous randomness. Moreover, we define  $\mathrm{E}_{t + 1}[\cdot ]$  to be a conditional expectation w.r.t. all randomness in iteration  $t + 1$  conditioned on all previous randomness. Note, that  $\mathrm{E}_{t + 1}[\cdot ] = \mathrm{E}_k[\mathrm{E}_{\mathcal{C}}[\mathrm{E}_{p_{\mathrm{a}}}[\cdot ]]]$ .
In the case of DASHA-PP-PAGE, there are two different sources of randomness from  $\{k_i^{t + 1}\}_{i = 1}^n$ . We define  $\mathrm{E}_{p_{\mathrm{page}}}$  and  $\mathrm{E}_B$  to be conditional expectations w.r.t. the probabilistic switching and mini-batch indices  $I_{i}^{t}$ , accordingly, conditioned on all previous randomness. Note, that  $\mathrm{E}_{t + 1}\left[\cdot \right] = \mathrm{E}_B\left[\mathrm{E}_{\mathcal{C}}\left[\mathrm{E}_{p_{\mathrm{a}}}\left[\mathrm{E}_{p_{\mathrm{page}}}\left[\cdot \right]\right]\right]\right]$  and  $\mathrm{E}_{t + 1}\left[\cdot \right] = \mathrm{E}_B\left[\mathrm{E}_{p_{\mathrm{page}}}\left[\mathrm{E}_{\mathcal{C}}\left[\mathrm{E}_{p_{\mathrm{a}}}\left[\cdot \right]\right]\right]\right]$ .
E.1 Standard Lemmas in the Nonconvex Setting
We start the proof of theorems by providing standard lemmas from the nonconvex optimization.
Lemma 2. Suppose that Assumption 2 holds and let  $x^{t + 1} = x^t - \gamma g^t$ . Then for any  $g^t \in \mathbb{R}^d$  and  $\gamma > 0$ , we have
$$
f \left(x ^ {t + 1}\right) \leq f \left(x ^ {t}\right) - \frac {\gamma}{2} \left\| \nabla f \left(x ^ {t}\right) \right\| ^ {2} - \left(\frac {1}{2 \gamma} - \frac {L}{2}\right) \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \frac {\gamma}{2} \left\| g ^ {t} - \nabla f \left(x ^ {t}\right) \right\| ^ {2}. \tag {19}
$$
Proof. Using  $L$ -smoothness, we have
$$
\begin{array}{l} f (x ^ {t + 1}) \leq f (x ^ {t}) + \left\langle \nabla f (x ^ {t}), x ^ {t + 1} - x ^ {t} \right\rangle + \frac {L}{2} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} \\ = f (x ^ {t}) - \gamma \left\langle \nabla f (x ^ {t}), g ^ {t} \right\rangle + \frac {L}{2} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2}. \\ \end{array}
$$
Next, due to  $-\langle x,y\rangle = \frac{1}{2}\| x - y\|^2 -\frac{1}{2}\| x\|^2 -\frac{1}{2}\| y\|^2$  , we obtain
$$
f (x ^ {t + 1}) \leq f (x ^ {t}) - \frac {\gamma}{2} \left\| \nabla f (x ^ {t}) \right\| ^ {2} - \left(\frac {1}{2 \gamma} - \frac {L}{2}\right) \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \frac {\gamma}{2} \left\| g ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2}.
$$
Lemma 3. Suppose that Assumption 1 holds and
$$
\operatorname {E} \left[ f \left(x ^ {t + 1}\right) \right] + \gamma \Psi^ {t + 1} \leq \operatorname {E} \left[ f \left(x ^ {t}\right) \right] - \frac {\gamma}{2} \operatorname {E} \left[ \left\| \nabla f \left(x ^ {t}\right) \right\| ^ {2} \right] + \gamma \Psi^ {t} + \gamma C, \tag {20}
$$
where  $\Psi^t$  is a sequence of numbers,  $\Psi^t \geq 0$  for all  $t \in [T]$ , constant  $C \geq 0$ , and constant  $\gamma > 0$ . Then
$$
\mathrm {E} \left[ \left\| \nabla f \left(\widehat {x} ^ {T}\right) \right\| ^ {2} \right] \leq \frac {2 \Delta_ {0}}{\gamma T} + \frac {2 \Psi^ {0}}{T} + 2 C, \tag {21}
$$
where a point  $\widehat{x}^T$  is chosen uniformly from a set of points  $\{x^t\}_{t=0}^{T-1}$ .
Proof. By unrolling (20) for  $t$  from 0 to  $T - 1$ , we obtain
$$
\frac {\gamma}{2} \sum_ {t = 0} ^ {T - 1} \mathrm {E} \left[ \left\| \nabla f (x ^ {t}) \right\| ^ {2} \right] + \mathrm {E} \left[ f (x ^ {T}) \right] + \gamma \Psi^ {T} \leq f (x ^ {0}) + \gamma \Psi^ {0} + \gamma T C.
$$
We subtract  $f^{*}$ , divide inequality by  $\frac{\gamma T}{2}$ , and take into account that  $f(x) \geq f^{*}$  for all  $x \in \mathbb{R}$ , and  $\Psi^t \geq 0$  for all  $t \in [T]$ , to get the following inequality:
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathrm {E} \left[ \left\| \nabla f (x ^ {t}) \right\| ^ {2} \right] \leq \frac {2 \Delta_ {0}}{\gamma T} + \frac {2 \Psi^ {0}}{T} + 2 C.
$$
It is left to consider the choice of a point  $\widehat{x}^T$  to complete the proof of the lemma.
Lemma 4. If  $0 < \gamma \leq (L + \sqrt{A})^{-1}$ ,  $L > 0$ , and  $A \geq 0$ , then
$$
\frac {1}{2 \gamma} - \frac {L}{2} - \frac {\gamma A}{2} \geq 0.
$$
The lemma can be easily checked with the direct calculation.
E.2 Generic Lemmas
Lemma 5. Suppose that Assumptions 7 and 8 hold and let us consider sequences  $g_{i}^{t + 1}, h_{i}^{t + 1}$ , and  $k_{i}^{t + 1}$  from Algorithm 1, then
$$
\begin{array}{l} \mathrm {E} _ {\mathcal {C}} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] \right] \\ \leq \frac {2 \omega}{n ^ {2} p _ {\mathrm {a}}} \sum_ {i = 1} ^ {n} \left\| k _ {i} ^ {t + 1} \right\| ^ {2} + \frac {a ^ {2} \left(\left(2 \omega + 1\right) p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} + (1 - a) ^ {2} \left\| g ^ {t} - h ^ {t} \right\| ^ {2}, \tag {22} \\ \end{array}
$$
$$
\begin{array}{l} \mathrm {E} _ {\mathcal {C}} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \right] \\ \leq \frac {2 \omega}{p _ {\mathrm {a}}} \left\| k _ {i} ^ {t + 1} \right\| ^ {2} + \left(\frac {a ^ {2} (2 \omega + 1 - p _ {\mathrm {a}})}{p _ {\mathrm {a}}} + (1 - a) ^ {2}\right) \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \quad \forall i \in [ n ]. \tag {23} \\ \end{array}
$$
Proof. First, we estimate  $\operatorname{E}_{\mathcal{C}}\left[\operatorname{E}_{p_{\mathrm{a}}}\left[\left\|g^{t+1}-h^{t+1}\right\|^2\right]\right]$ :
$$
\begin{array}{l} \mathrm {E} _ {\mathcal {C}} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] \right] \\ = \operatorname {E} _ {\mathcal {C}} \left[ \operatorname {E} _ {p _ {\mathrm {a}}} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} - \operatorname {E} _ {\mathcal {C}} \left[ \operatorname {E} _ {p _ {\mathrm {a}}} \left[ g ^ {t + 1} - h ^ {t + 1} \right] \right] \right\| ^ {2} \right] \right] + \left\| \operatorname {E} _ {\mathcal {C}} \left[ \operatorname {E} _ {p _ {\mathrm {a}}} \left[ g ^ {t + 1} - h ^ {t + 1} \right] \right] \right\| ^ {2}, \\ \end{array}
$$
where we used (16). Due to Assumption 8, we have
$$
\begin{array}{l} \mathbf {E} _ {\mathcal {C}} \left[ \begin{array}{l l} \mathrm {E} _ {\mathrm {p} _ {\mathrm {a}}} & \left[ \begin{array}{l} \mathrm {g} _ {i} ^ {t + 1} \end{array} \right] \end{array} \right] \\ = p _ {\mathrm {a}} \mathrm {E} _ {\mathcal {C}} \left[ g _ {i} ^ {t} + \mathcal {C} _ {i} \left(\frac {1}{p _ {\mathrm {a}}} k _ {i} ^ {t + 1} - \frac {a}{p _ {\mathrm {a}}} \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right)\right) \right] + (1 - p _ {\mathrm {a}}) g _ {i} ^ {t} \\ = g _ {i} ^ {t} + p _ {\mathrm {a}} \mathrm {E} _ {\mathcal {C}} \left[ \mathcal {C} _ {i} \left(\frac {1}{p _ {\mathrm {a}}} k _ {i} ^ {t + 1} - \frac {a}{p _ {\mathrm {a}}} \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right)\right) \right] \\ = g _ {i} ^ {t} + k _ {i} ^ {t + 1} - a \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right), \\ \end{array}
$$
$$
\mathrm {E} _ {\mathcal {C}} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ h _ {i} ^ {t + 1} \right] \right] = p _ {\mathrm {a}} \mathrm {E} _ {\mathcal {C}} \left[ h _ {i} ^ {t} + \frac {1}{p _ {\mathrm {a}}} k _ {i} ^ {t + 1} \right] + (1 - p _ {\mathrm {a}}) h _ {i} ^ {t} = h _ {i} ^ {t} + k _ {i} ^ {t + 1}.
$$
Thus, we can get
$$
\begin{array}{l} \mathrm {E} _ {\mathcal {C}} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] \right] \\ = \operatorname {E} _ {\mathcal {C}} \left[ \operatorname {E} _ {p _ {\mathrm {a}}} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} - \operatorname {E} _ {\mathcal {C}} \left[ \operatorname {E} _ {p _ {\mathrm {a}}} \left[ g ^ {t + 1} - h ^ {t + 1} \right] \right] \right\| ^ {2} \right] \right] + (1 - a) ^ {2} \left\| g ^ {t} - h ^ {t} \right\| ^ {2}. \\ \end{array}
$$