SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \left| \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right.\right) \\ \leq p (1 - p) \sum_ {k _ {1} <   \dots <   k _ {l _ {j + 1} - 2} = 1} ^ {\infty} \mathbb {P} \left(\bigcap_ {i = 1} ^ {l _ {j + 1} - 2} \{\xi^ {k _ {i}} = 0 \}, \bigcap_ {i = 1} ^ {l _ {j + 1} - 2} \left(\{k _ {j, i} ^ {\xi} = k _ {i} \} \bigcap A _ {k _ {i}}\right), \right. \\ \bigcup_ {i = k l _ {j + 1} - 2 + 1} ^ {\infty} \left(\left\{k _ {j, l _ {j + 1} - 1} ^ {\xi} = i \right\} \bigcap A _ {i}\right) \Bigg | \bigcap_ {i = 1} ^ {j} \left\{\eta_ {i} = l _ {i} \right\} \Bigg) \\ \leq p (1 - p) \sum_ {k _ {1} <   \dots <   k _ {l _ {j + 1} - 2} = 1} ^ {\infty} \mathbb {P} \left(\bigcap_ {i = 1} ^ {l _ {j + 1} - 2} \left\{\xi^ {k _ {i}} = 0 \right\}, \bigcap_ {i = 1} ^ {l _ {j + 1} - 2} \left(\left\{k _ {j, i} ^ {\xi} = k _ {i} \right\} \bigcap A _ {k _ {i}}\right) \Bigg | \bigcap_ {i = 1} ^ {j} \left\{\eta_ {i} = l _ {i} \right\}\right), \\ \end{array}
$$
where we use  $\mathbb{P}(A,B)\leq \mathbb{P}(A)$  for any events  $A$  and  $B$ . Using mathematical induction, we can continue and get that
$$
\mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \Bigg | \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right) \leq p (1 - p) ^ {l _ {j + 1} - 1}.
$$
D.3.3 Lemma D.7
This is a technical lemma that we use in the proof of Lemma D.2.
Lemma D.7. Let us consider a sorted sequence  $0 < \tau_{1} \leq \dots \leq \tau_{n} \leq \tau_{n+1} = \infty$  and a constant  $S \geq \frac{1}{4}$ . We define
$$
t _ {1} := S \left(\sum_ {i = 1} ^ {j _ {1} ^ {*}} \frac {1}{\tau_ {i}}\right) ^ {- 1},
$$
$$
j _ {1} ^ {*} = \inf \left\{m \in [ n ] \left| S \left(\sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} <   \tau_ {m + 1} \right. \right\},
$$
$$
t _ {2} := \min  _ {j \in [ n ]} \left[ \left(\sum_ {i = 1} ^ {j} \frac {1}{\tau_ {i}}\right) ^ {- 1} (S + j) \right].
$$
$$
t _ {1} \leq t _ {2} \leq 6 t _ {1}.
$$
Proof. Additionally, let us define
$$
j _ {2} ^ {*} = \underset {j \in [ n ]} {\arg \min } \left[ \left(\sum_ {i = 1} ^ {j} \frac {1}{\tau_ {i}}\right) ^ {- 1} (S + j) \right],
$$
where  $j_2^*$  is the smallest index. For  $j_2^* = 1$ , we have
$$
t _ {2} = \tau_ {1} (S + 1) > \tau_ {1}.
$$
For  $j_2^* > 1$ , we have
$$
\left(\sum_ {i = 1} ^ {j _ {2} ^ {*}} \frac {1}{\tau_ {i}}\right) ^ {- 1} (S + j _ {2} ^ {*}) <   \left(\sum_ {i = 1} ^ {j _ {2} ^ {*} - 1} \frac {1}{\tau_ {i}}\right) ^ {- 1} (S + j _ {2} ^ {*} - 1).
$$
From this inequality, we get
$$
\left(\sum_ {i = 1} ^ {j _ {2} ^ {*} - 1} \frac {1}{\tau_ {i}}\right) (S + j _ {2} ^ {*}) <   \left(\sum_ {i = 1} ^ {j _ {2} ^ {*}} \frac {1}{\tau_ {i}}\right) (S + j _ {2} ^ {*} - 1)
$$
$$
\left(\sum_ {i = 1} ^ {j _ {2} ^ {*}} \frac {1}{\tau_ {i}}\right) <   \frac {1}{\tau_ {j _ {2} ^ {*}}} \left(S + j _ {2} ^ {*}\right).
$$
Thus  $\tau_{j_2^*} < t_2$  for all  $j_2^* \geq 1$ .
Then either  $j_2^* \leq j_1^*$  and
$$
t _ {2} = \left(\sum_ {i = 1} ^ {j _ {2} ^ {*}} \frac {1}{\tau_ {i}}\right) ^ {- 1} (S + j _ {2} ^ {*}) \geq S \left(\sum_ {i = 1} ^ {j _ {2} ^ {*}} \frac {1}{\tau_ {i}}\right) ^ {- 1} \geq S \left(\sum_ {i = 1} ^ {j _ {1} ^ {*}} \frac {1}{\tau_ {i}}\right) ^ {- 1} = t _ {1},
$$
or  $j_2^* > j_1^*$  and
$$
t _ {2} > \tau_ {j _ {2} ^ {*}} \geq \tau_ {j _ {1} ^ {*} + 1} > S \left(\sum_ {i = 1} ^ {j _ {1} ^ {*}} \frac {1}{\tau_ {i}}\right) ^ {- 1} = t _ {1},
$$
where we used the definition of  $j_1^*$ . It concludes that  $t_2 \geq t_1$ .
Assume that  $j_1^* > S + 1$ . Since the harmonic mean of a sequence less or equal to the maximum, we have
$$
S \left(\sum_ {i = 1} ^ {j _ {1} ^ {*} - 1} \frac {1}{\tau_ {i}}\right) ^ {- 1} <   (j _ {1} ^ {*} - 1) \left(\sum_ {i = 1} ^ {j _ {1} ^ {*} - 1} \frac {1}{\tau_ {i}}\right) ^ {- 1} \leq \tau_ {j _ {1} ^ {*} - 1} \leq \tau_ {j _ {1} ^ {*}}.
$$
This inequality contradicts the definition of  $j_1^*$ . It means that  $j_1^* \leq S + 1$  and
$$
t _ {2} \leq \left(\sum_ {i = 1} ^ {j _ {1} ^ {*}} \frac {1}{\tau_ {i}}\right) ^ {- 1} (S + j _ {1} ^ {*}) \leq \left(\sum_ {i = 1} ^ {j _ {1} ^ {*}} \frac {1}{\tau_ {i}}\right) ^ {- 1} (2 S + 1) \leq \left(\sum_ {i = 1} ^ {j _ {1} ^ {*}} \frac {1}{\tau_ {i}}\right) ^ {- 1} (6 S) \leq 6 t _ {1}.
$$
D.4 Proof of Theorems 7.4 and 7.5
D.4.1 Proof of Theorems 7.4
Theorem 7.4. Assume that Assumptions 7.1, 7.2 and 7.3 hold. Let us take the batch size  $S = \max \left\{\left[\sigma^2 /\varepsilon \right],1\right\}$ , and  $\gamma = \min \left\{\frac{1}{L},\frac{\varepsilon S}{2L\sigma^2}\right\} = \Theta (1 / L)$  in Method 4. Then after
Proof. Note that Method 4 is just the stochastic gradient method with the batch size  $S$ . Method 4 can be rewritten as  $x^{k + 1} = x^k -\gamma \frac{1}{S}\sum_{i = 1}^{S}\hat{\nabla} f(x^k;\xi_i)$ , where the  $\xi_{i}$  are independent random samples. It means that we can use the classical SGD result (see Theorem D.8). For a stepsize
$$
\gamma = \min  \left\{\frac {1}{L}, \frac {\varepsilon S}{2 L \sigma^ {2}} \right\},
$$
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\| \nabla f (x ^ {k}) \right\| ^ {2} \right] \leq \varepsilon ,
$$
$$
K \geq \frac {1 2 \Delta L}{\varepsilon} + \frac {1 2 \Delta L \sigma^ {2}}{\varepsilon^ {2} S}.
$$
Using the choice of  $S$ , we showed that Method 4 converges after
steps with
$$
\gamma = \min  \left\{\frac {1}{L}, \frac {\varepsilon S}{2 L \sigma^ {2}} \right\} \geq \frac {1}{2 L}.
$$
D.4.2 The classical SGD theorem
We reprove the classical SGD result (Ghadimi and Lan, 2013; Khaled and Richtárik, 2020).
Theorem D.8. Assume that Assumptions 7.1 and 7.2 hold. We consider the SGD method:
$$
x ^ {k + 1} = x ^ {k} - \gamma g (x ^ {k}),
$$
$$
\gamma = \min  \left\{\frac {1}{L}, \frac {\varepsilon}{2 L \sigma^ {2}} \right\}
$$
For a fixed  $x\in \mathbb{R}^d$ $g(x)$  is a random vector such that  $\mathbb{E}\left[g(x)\right] = \nabla f(x)$
$$
\mathbb {E} \left[ \| g (x) - \nabla f (x) \| ^ {2} \right] \leq \sigma^ {2}, \tag {26}
$$
and  $g(x^{k})$  are independent vectors for all  $k \geq 0$ . Then
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\| \nabla f (x ^ {k}) \right\| ^ {2} \right] \leq \varepsilon
$$
$$
K \geq \frac {4 \Delta L}{\varepsilon} + \frac {8 \Delta L \sigma^ {2}}{\varepsilon^ {2}}.
$$
Proof. From Assumption 7.1, we have
$$
\begin{array}{l} f \left(x ^ {k + 1}\right) \leq f \left(x ^ {k}\right) + \left\langle \nabla f \left(x ^ {k}\right), x ^ {k + 1} - x ^ {k} \right\rangle + \frac {L}{2} \| x ^ {k + 1} - x ^ {k} \| ^ {2} \\ = f (x ^ {k}) - \gamma \left\langle \nabla f (x ^ {k}), g (x ^ {k}) \right\rangle + \frac {L \gamma^ {2}}{2} \left\| g (x ^ {k}) \right\| ^ {2}. \\ \end{array}
$$
We denote  $\mathcal{G}^k$  as a sigma-algebra generated by  $g(x^0),\ldots ,g(x^{k - 1})$ . Using unbiasedness and (26), we obtain
$$
\begin{array}{l} \mathbb {E} \left[ f (x ^ {k + 1}) \mid \mathcal {G} ^ {k} \right] \leq f (x ^ {k}) - \gamma \left(1 - \frac {L \gamma}{2}\right) \left\| \nabla f (x ^ {k}) \right\| ^ {2} + \frac {L \gamma^ {2}}{2} \mathbb {E} \left[ \left\| g ^ {k} - \nabla f (x ^ {k}) \right\| ^ {2} \mid \mathcal {G} ^ {k} \right] \\ \leq f (x ^ {k}) - \gamma \left(1 - \frac {L \gamma}{2}\right) \left\| \nabla f (x ^ {k}) \right\| ^ {2} + \frac {L \gamma^ {2} \sigma^ {2}}{2}. \\ \end{array}
$$
Since  $\gamma \leq 1 / L$ , we get
$$
\mathbb {E} \left[ f (x ^ {k + 1}) \mid \mathcal {G} ^ {k} \right] \leq f (x ^ {k}) - \frac {\gamma}{2} \left\| \nabla f (x ^ {k}) \right\| ^ {2} + \frac {L \gamma^ {2} \sigma^ {2}}{2}.
$$
We subtract  $f^{*}$  and take the full expectation to obtain
$$
\mathbb {E} \left[ f (x ^ {k + 1}) - f ^ {*} \right] \leq \mathbb {E} \left[ f (x ^ {k}) - f ^ {*} \right] - \frac {\gamma}{2} \mathbb {E} \left[ \left\| \nabla f (x ^ {k}) \right\| ^ {2} \right] + \frac {L \gamma^ {2} \sigma^ {2}}{2}.
$$
Next, we sum the inequality for  $k \in \{0, \dots, K - 1\}$ :
$$
\begin{array}{l} \mathbb {E} \left[ f (x ^ {K}) - f ^ {*} \right] \leq f (x ^ {0}) - f ^ {*} - \sum_ {k = 0} ^ {K - 1} \frac {\gamma}{2} \mathbb {E} \left[ \left\| \nabla f (x ^ {k}) \right\| ^ {2} \right] + \frac {K L \gamma^ {2} \sigma^ {2}}{2} \\ = \Delta - \sum_ {k = 0} ^ {K - 1} \frac {\gamma}{2} \mathbb {E} \left[ \left\| \nabla f (x ^ {k}) \right\| ^ {2} \right] + \frac {K L \gamma^ {2} \sigma^ {2}}{2}. \\ \end{array}
$$
Finally, we rearrange the terms and use that  $\mathbb{E}\left[f(x^K) - f^*\right] \geq 0$ :
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\| \nabla f (x ^ {k}) \right\| ^ {2} \right] \leq \frac {2 \Delta}{\gamma K} + L \gamma \sigma^ {2}.
$$
The choice of  $\gamma$  and  $K$  ensures that
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\| \nabla f (x ^ {k}) \right\| ^ {2} \right] \leq \varepsilon .
$$
D.4.3 Proof of Theorems 7.5
Proof. In this setup, the method converges after  $K \times \{ \text{time required to collect a batch of the size } S \}$ . Without loss of generality, we assume that  $\tau_1 \leq \dots \leq \tau_n$ .
Let us define time that is enough to collect a batch of the size  $S$  as  $t'$ . Obviously, one can always take  $t' = 3\tau_n\left\lceil \frac{S}{n} \right\rceil$  and guarantees that every worker calculates at least  $\left\lceil \frac{S}{n} \right\rceil$  stochastic gradients, but we will provide a tighter  $t'$ .
We define  $B_{i}$  as the number of received gradients with an iteration index equals to  $k$ . For each worker, there are two options: either the  $i^{\text{th}}$  worker does not send a gradient with an iteration index  $k$  and  $B_{i} = 0$ , or it sends at least once and  $B_{i} > 0$ .
In the worst case, for  $i^{\mathrm{th}}$  worker, the time required to calculate  $B_{i}$  gradients equals
$$
t _ {i} := \left\{ \begin{array}{l l} \tau_ {i} \left(1 + B _ {i}\right) & B _ {i} > 0 \\ 0 & B _ {i} = 0 \end{array} \right.
$$
because either  $B_{i} > 0$  and, in the worst case, a worker finishes the calculation of a gradient from the previous iteration (that we ignore) and only then starts the calculation of a gradient of the current iteration  $k$ , or  $B_{i} = 0$  and the server does not receive any gradients from a worker.
Note that all workers work in parallel, so our goal is to find feasible points  $t' \in \mathbb{R}$  and  $B_1, \dots, B_n \in \mathbb{N}_0$  such that
$$
t ^ {\prime} \geq \max  _ {i \in [ n ]} t _ {i}
$$
$$
B _ {1}, \dots , B _ {n} \in \mathbb {N} _ {0} \tag {27}
$$
$$
\sum_ {i = 1} ^ {n} B _ {i} \geq S
$$
First, we relax an assumption that  $B_{i}\in \mathbb{N}_{0}$  and assume that  $B_{i}\in \mathbb{R}$  for all  $i\in [n]$ :
$$
t^{\prime}\geq \max_{i\in [n]}t_{i}
$$
$$
B _ {1}, \dots , B _ {n} \in \mathbb {R} \tag {28}
$$
$$
B _ {1}, \dots , B _ {n} \geq 0
$$
If  $B_{i}\in \mathbb{R}$  are feasible points of (28), then
$$
\max  _ {i \in [ n ]} t _ {i} = \max  _ {B _ {i} > 0} \tau_ {i} \left(1 + B _ {i}\right) \leq \max  _ {B _ {i} > 0} \tau_ {i} \left(1 + \lceil B _ {i} \rceil\right)
$$
$$
\leq \max  _ {B _ {i} > 0} \tau_ {i} \left(2 + B _ {i}\right) \leq 2 \max  _ {B _ {i} > 0} \tau_ {i} \left(1 + B _ {i}\right) = 2 \max  _ {i \in [ n ]} t _ {i}.
$$
It means that if  $t' \in \mathbb{R}$  and  $B_1, \dots, B_n$  are feasible points of (28), then  $2t'$  and  $\lceil B_1 \rceil, \dots, \lceil B_n \rceil$  are feasible points of (27).
$$
t ^ {\prime} (j) := \left(\sum_ {i = 1} ^ {j} \frac {1}{\tau_ {i}}\right) ^ {- 1} (S + j) \quad \forall j \in [ n ],
$$
Note that a worker may send a gradient from the previous iterations that we ignore in the method.
and take  $j^{*} = \arg \min_{j\in [n]}t^{\prime}(j)$ ,  $j^{*}$  is the smallest index from all minimizers of  $t^\prime (j)$ . Let us show that  $t^\prime (j^*)$  and
$$
B _ {i} = \left\{ \begin{array}{l l} \frac {t ^ {\prime} (j ^ {*})}{\tau_ {i}} - 1, & i \leq j ^ {*} \\ 0, & i > j ^ {*} \end{array} \right.
$$
are feasible points of (28). First, we have
$$
\sum_ {i = 1} ^ {n} B _ {i} = \sum_ {i = 1} ^ {j ^ {*}} \left(\frac {t ^ {\prime} (j ^ {*})}{\tau_ {i}} - 1\right) = \left(\sum_ {i = 1} ^ {j ^ {*}} \frac {1}{\tau_ {i}}\right) ^ {- 1} (S + j ^ {*}) \left(\sum_ {i = 1} ^ {j ^ {*}} \frac {1}{\tau_ {i}}\right) - j ^ {*} = S.
$$
Next, we show that  $B_{i} > 0$  for all  $i \leq j^{*}$ . If  $j^{*} = 1$ , then  $t^{\prime}(1) = \tau_{1}(S + 1)$ , thus  $B_{1} = S > 0$ . If  $j^{*} > 1$ , then, by its definition, we have  $t^{\prime}(j^{*}) < t^{\prime}(j^{*} - 1)$ , thus
$$
\left(\sum_ {i = 1} ^ {j ^ {*}} \frac {1}{\tau_ {i}}\right) ^ {- 1} (S + j ^ {*}) <   \left(\sum_ {i = 1} ^ {j ^ {*} - 1} \frac {1}{\tau_ {i}}\right) ^ {- 1} (S + j ^ {*} - 1).
$$
$$
\left(\sum_ {i = 1} ^ {j ^ {*} - 1} \frac {1}{\tau_ {i}}\right) (S + j ^ {*}) <   \left(\sum_ {i = 1} ^ {j ^ {*}} \frac {1}{\tau_ {i}}\right) (S + j ^ {*} - 1)
$$
$$
\left(\sum_ {i = 1} ^ {j ^ {*}} \frac {1}{\tau_ {i}}\right) <   \frac {1}{\tau_ {j ^ {*}}} (S + j ^ {*}).
$$
From the last inequality, we get that  $\tau_{j^*} < t'(j^*)$ , thus  $B_i \geq B_{j^*} > 0$  for all  $i \leq j^*$ . It is left to show that
$$
\max  _ {i \in [ n ]} t _ {i} = \max  _ {i \leq j ^ {*}} \tau_ {i} \left(B _ {i} + 1\right) = t ^ {\prime} (j ^ {*}).
$$
Finally, we can conclude that Method 4 returns a solution after
$$
K \times 2 t ^ {\prime} (j ^ {*}) = \frac {4 8 \Delta L}{\varepsilon} \min _ {j \in [ n ]} \left[ \left(\sum_ {i = 1} ^ {j} \frac {1}{\tau_ {i}}\right) ^ {- 1} (S + j) \right]
$$
seconds.
E Proofs for Heterogeneous Regime
E.1 Proof of Theorem A.2
Theorem A.2. Let us consider the oracle class  $\mathcal{O}_{\tau_1,\dots,\tau_n}^{\sigma^2,heterog}$  for some  $\sigma^2 > 0$  and  $0 < \tau_1 \leq \dots \leq \tau_n$ . We fix any  $L, \Delta > 0$  and  $0 < \varepsilon \leq c'L\Delta$ . In the view Protocol 3, for any algorithm  $A \in \mathcal{A}_{\mathrm{zx}}$ , there exists a function  $f = \frac{1}{n}\sum_{i=1}^{n}f_i \in \mathcal{F}_{\Delta,L}$  and oracles and distributions  $(O_1,\ldots,O_n), (\mathcal{D}_1,\ldots,\mathcal{D}_n)) \in \mathcal{O}_{\tau_1,\ldots,\tau_n}^{\sigma^2,heterog}(f_1,\ldots,f_n)$  such that  $\mathbb{E}\left[\inf_{k \in S_t}\left\|\nabla f(x^k)\right\|^2\right] > \varepsilon$ , where  $S_t := \{k \in \mathbb{N}_0|t^k \leq t\}$ , and
The structure of the following proof is similar to the proof of Theorem 6.4. In the heterogeneous regime, the main difference is that we have more freedom to choose the functions  $f_{i}$ .
Proof. In (14), we have the sum of two terms. We split the proof in two parts for each of the terms.
(Part 1)
(Step 1:  $f\in \mathcal{F}_{\Delta ,L}$ )
Let us fix  $\lambda > 0$ . We consider the following functions  $f_{i}$ :
$$
f _ {i} (x) := \left\{ \begin{array}{l l} 0, & i <   n, \\ \frac {n L \lambda^ {2}}{l _ {1}} F _ {T} \left(\frac {x}{\lambda}\right), & i = n. \end{array} \right.
$$
Let us show that the function  $f$  is  $L$ -smooth:
$$
\| \nabla f (x) - \nabla f (y) \| = \frac {1}{n} \left\| \sum_ {i = 1} ^ {n} \left(\nabla f _ {i} (x) - \nabla f _ {i} (y)\right) \right\| = \frac {L \lambda}{l _ {1}} \left\| \nabla F _ {T} \left(\frac {x}{\lambda}\right) - \nabla F _ {T} \left(\frac {y}{\lambda}\right) \right\| \leq L \| x - y \|.
$$
$$
f (0) - \inf  _ {x \in \mathbb {R} ^ {T}} f (x) = \frac {1}{n} \frac {n L \lambda^ {2}}{l _ {1}} \left(F _ {T} (0) - \inf  _ {x \in \mathbb {R} ^ {T}} F _ {T} (x)\right) \leq \frac {L \lambda^ {2} \Delta^ {0} T}{l _ {1}} \leq \Delta .
$$
In the oracles  $O_{i}$ , we have the freedom to choose a mapping  $\widehat{\nabla} f_{i}(\cdot ;\cdot)$  (see (7)). In this part of the proof, we simply take non-stochastic mappings  $\widehat{\nabla} f_{i}(x;\xi)\coloneqq \nabla f_{i}(x)$  that are, obviously, unbiased and  $\sigma^2$  variance-bounded. We can take an arbitrary distribution, for instance, let us take  $\mathcal{D}_i = \mathrm{Bernoulli}(1)$  for all  $i\in [n]$ .
We take
$$
\lambda = \frac {l _ {1} \sqrt {\varepsilon}}{L}
$$
to ensure that
$$
\left\| \nabla f (x) \right\| ^ {2} = \frac {1}{n ^ {2}} \left\| \nabla f _ {n} (x) \right\| ^ {2} = \frac {L ^ {2} \lambda^ {2}}{l _ {1} ^ {2}} \left\| \nabla F _ {T} \left(\frac {x}{\lambda}\right) \right\| ^ {2} > \frac {L ^ {2} \lambda^ {2}}{l _ {1} ^ {2}} = \varepsilon
$$
for all  $x\in \mathbb{R}^T$  such that  $\mathrm{prog}(x) <   T$  . Thus
$$
T = \left\lfloor \frac {\Delta L}{l _ {1} \varepsilon \Delta^ {0}} \right\rfloor .
$$
Only the  $n^{\mathrm{th}}$  worker contains a nonzero function and can provide a gradient every  $\tau_{n}$  seconds. Since  $A$  is a zero-respecting algorithm and the function  $f_{n}$  is a zero-chain function, for all  $k \geq 0$  such that
$$
t ^ {k} <   \tau_ {n} T,
$$
$$
\left\| \nabla f (x ^ {k}) \right\| > \varepsilon
$$
because we need at least  $T$  oracle calls to obtain  $\mathrm{prog}(x^k) \geq T$ . It means that
$$
\inf _ {k \in S _ {t}} \left\| \nabla f (x ^ {k}) \right\| ^ {2} > \varepsilon
$$
$$
t = \tau_ {n} \left(\frac {\Delta L}{l _ {1} \varepsilon \Delta^ {0}} - 1\right).
$$
We now prove the second part of the lower bound.
Let us fix  $\lambda > 0$ . We assume that  $x = [x_1, \ldots, x_n] \in \mathbb{R}^{nT}$ . We define  $x_i \in \mathbb{R}^T$  as the  $i^{\text{th}}$  block of a vector  $x = [x_1, \ldots, x_n] \in \mathbb{R}^{nT}$ . We consider the following functions  $f_i$ :
$$
f _ {i} (x) := \frac {n L \lambda_ {i} ^ {2}}{l _ {1}} F _ {T} \left(\frac {x _ {i}}{\lambda_ {i}}\right).
$$
The function  $f_{i}$  depends only on a subset of variables  $x_{i}$  from  $x$ . Let us show that the function  $f$  is  $L$ -smooth. Indeed, we have
$$
\| \nabla f _ {i} (x) - \nabla f _ {i} (y) \| = \frac {n L \lambda_ {i}}{l _ {1}} \left\| \nabla F _ {T} \left(\frac {x _ {i}}{\lambda_ {i}}\right) - \nabla F _ {T} \left(\frac {y _ {i}}{\lambda_ {i}}\right) \right\| \leq n L \| x _ {i} - y _ {i} \| \quad \forall i \in [ n ],
$$
$$
\begin{array}{l} \left\| \nabla f (x) - \nabla f (y) \right\| ^ {2} = \frac {1}{n ^ {2}} \left\| \sum_ {i = 1} ^ {n} \left(\nabla f _ {i} (x) - \nabla f _ {i} (y)\right) \right\| ^ {2} = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} (x) - \nabla f _ {i} (y) \| ^ {2} \\ \leq \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} n ^ {2} L ^ {2} \left\| x _ {i} - y _ {i} \right\| ^ {2} = L ^ {2} \left\| x - y \right\| ^ {2}. \\ \end{array}
$$
$$
T = \left\lfloor \frac {\Delta l _ {1}}{L \sum_ {j = 1} ^ {n} \lambda_ {j} ^ {2} \Delta^ {0}} \right\rfloor ,
$$
$$
f (0) - \inf _ {x \in \mathbb {R} ^ {T}} f (x) = \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {n L \lambda_ {i} ^ {2}}{l _ {1}} \left(F _ {T} (0) - \inf _ {x \in \mathbb {R} ^ {T}} F _ {T} (x)\right) \leq \sum_ {i = 1} ^ {n} \frac {L \lambda_ {i} ^ {2} \Delta^ {0} T}{l _ {1}} \leq \Delta .
$$
We showed that the function  $f\in \mathcal{F}_{\Delta ,L}$
In the oracles  $O_{i}$ , we have the freedom to choose a mapping  $\widehat{\nabla} f_{i}(\cdot ;\cdot)$  (see (7)). Let us take
$$
[ \widehat {\nabla} f _ {i} (x; \xi) ] _ {j} := \nabla_ {j} f _ {i} (x) \left(1 + \mathbb {1} [ j > (i - 1) T + \operatorname {p r o g} (x _ {i}) ] \left(\frac {\xi}{p _ {i}} - 1\right)\right) \quad \forall x \in \mathbb {R} ^ {n T},
$$
$\mathcal{D}_i = \mathrm{Bernoulli}(p_i)$ , and  $p_i \in (0,1]$  for all  $i \in [n]$ . Let us show it is unbiased and  $\sigma^2$ -variance-bounded:
$$
\mathbb {E} \left[ [ \widehat {\nabla} f _ {i} (x; \xi) ] _ {j} \right] = \nabla_ {j} f _ {i} (x) \left(1 + \mathbb {1} [ j > (i - 1) T + \operatorname {p r o g} (x _ {i}) ] \left(\frac {\mathbb {E} [ \xi ]}{p _ {i}} - 1\right)\right) = \nabla_ {j} f _ {i} (x)
$$
for all  $j\in nT$  , and
$$
\mathbb {E} \left[ \left\| \widehat {\nabla} f _ {i} (x; \xi) - \nabla f _ {i} (x) \right\| ^ {2} \right] \leq \| \nabla f _ {i} (x) \| _ {\infty} ^ {2} \mathbb {E} \left[ \left(\frac {\mathbb {E} [ \xi ]}{p _ {i}} - 1\right) ^ {2} \right]
$$
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \widehat {\nabla} f _ {i} (x; \xi) - \nabla f _ {i} (x) \right\| ^ {2} \right] \leq \frac {\| \nabla f _ {i} (x) \| _ {\infty} ^ {2} (1 - p _ {i})}{p _ {i}} = \frac {n ^ {2} L ^ {2} \lambda_ {i} ^ {2} \left\| F _ {T} \left(\frac {x _ {i}}{\lambda_ {i}}\right) \right\| _ {\infty} ^ {2} (1 - p _ {i})}{l _ {1} ^ {2} p _ {i}} \\ \leq \frac {n ^ {2} L ^ {2} \lambda_ {i} ^ {2} \gamma_ {\infty} ^ {2} (1 - p _ {i})}{l _ {1} ^ {2} p _ {i}} \leq \sigma^ {2}, \\ \end{array}
$$
$$
p _ {i} = \min \left\{\frac {n ^ {2} L ^ {2} \lambda_ {i} ^ {2} \gamma_ {\infty} ^ {2}}{\sigma^ {2} l _ {1} ^ {2}}, 1 \right\}.
$$
We fix  $\eta > 0$  and choose
$$
\lambda_ {i} = \frac {l _ {1} \sqrt {\eta \varepsilon \tau_ {i}}}{L \sqrt {\sum_ {i = 1} ^ {n} \tau_ {i}}}
$$
$$
\begin{array}{l} \left\| \nabla f (x) \right\| ^ {2} = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} (x) \right\| ^ {2} = \sum_ {i = 1} ^ {n} \frac {L ^ {2} \lambda_ {i} ^ {2}}{l _ {1} ^ {2}} \left\| \nabla F _ {T _ {i}} \left(\frac {x _ {i}}{\lambda_ {i}}\right) \right\| ^ {2} \\ = \sum_ {i = 1} ^ {n} \frac {\eta \varepsilon \tau_ {i}}{\sum_ {i = 1} ^ {n} \tau_ {i}} \left\| \nabla F _ {T _ {i}} \left(\frac {x _ {i}}{\lambda_ {i}}\right) \right\| ^ {2} > \sum_ {i = 1} ^ {n} \frac {\eta \varepsilon \tau_ {i}}{\sum_ {i = 1} ^ {n} \tau_ {i}} \mathbb {1} [ \operatorname {p r o g} (x _ {i}) <   T ] \tag {29} \\ \end{array}
$$
for all  $x = [x_1,\ldots ,x_n]\in \mathbb{R}^T$ . Thus
$$
T = \left\lfloor \frac {\Delta L}{\eta \varepsilon l _ {1} \Delta^ {0}} \right\rfloor
$$
$$
p _ {i} = \min  \left\{\frac {n ^ {2} \gamma_ {\infty} ^ {2} \eta \varepsilon \tau_ {i}}{\sigma^ {2} \sum_ {i = 1} ^ {n} \tau_ {i}}, 1 \right\} \quad \forall i \in [ n ]. \tag {30}
$$
Protocol 3 generates the sequence  $\{x^k\}_{k=0}^\infty \equiv \{[x_1^k, \ldots, x_n^k]\}_{k=0}^\infty$ . From (29), we have
$$
\inf  _ {k \in S _ {t}} \left\| \nabla f \left(x ^ {k}\right) \right\| ^ {2} > \inf  _ {k \in S _ {t}} \sum_ {i = 1} ^ {n} \frac {\eta \varepsilon \tau_ {i}}{\sum_ {i = 1} ^ {n} \tau_ {i}} \mathbb {1} [ \operatorname {p r o g} \left(x _ {i} ^ {k}\right) <   T ] \geq \sum_ {i = 1} ^ {n} \frac {\eta \varepsilon \tau_ {i}}{\sum_ {i = 1} ^ {n} \tau_ {i}} \inf  _ {k \in S _ {t}} \mathbb {1} [ \operatorname {p r o g} \left(x _ {i} ^ {k}\right) <   T ]. \tag {31}
$$
Further, we require the following auxiliary lemma. See the proof in Section E.2.
Lemma E.1. For  $\eta = 4$ , with probability not less than  $1/2$ ,
$$
\sum_ {i = 1} ^ {n} \frac {\eta \varepsilon \tau_ {i}}{\sum_ {i = 1} ^ {n} \tau_ {i}} \inf _ {k \in S _ {t}} \mathbb {1} [ \mathrm {p r o g} (x _ {i} ^ {k}) <   T ] > 2 \varepsilon
$$
$$
t \leq \frac {1}{2 4} \left(\frac {\sigma^ {2} \sum_ {i = 1} ^ {n} \tau_ {i}}{n ^ {2} \gamma_ {\infty} ^ {2} \eta \varepsilon}\right) \left(\frac {T}{2} - 1\right).
$$
Using Lemma E.1 and (31), we have
$$
\mathbb {E} \left[ \inf  _ {k \in S _ {t}} \left\| \nabla f (x ^ {k}) \right\| ^ {2} \right] > \varepsilon
$$
$$
t = \frac {1}{2 4} \left(\frac {\sigma^ {2} \sum_ {i = 1} ^ {n} \tau_ {i}}{4 n ^ {2} \gamma_ {\infty} ^ {2} \varepsilon}\right) \left(\frac {\Delta L}{8 \varepsilon l _ {1} \Delta^ {0}} - 2\right).
$$
This finishes the proof of Part 2.
In the following lemma, we use notations from the proof of Theorem A.2.
$$
\sum_ {i = 1} ^ {n} \frac {\eta \varepsilon \tau_ {i}}{\sum_ {i = 1} ^ {n} \tau_ {i}} \inf  _ {k \in S _ {t}} \mathbb {1} \left[ \operatorname {p r o g} \left(x _ {i} ^ {k}\right) <   T \right] > 2 \varepsilon
$$
Proof. Let us fix  $t \geq 0$ . Our goal is to show that the probability of an inequality
$$
\sum_ {i = 1} ^ {n} \frac {\eta \varepsilon \tau_ {i}}{\sum_ {i = 1} ^ {n} \tau_ {i}} \inf  _ {k \in S _ {t}} \mathbb {1} \left[ \operatorname {p r o g} \left(x _ {i} ^ {k}\right) <   T \right] \leq 2 \varepsilon \tag {32}
$$
is small.
We now use the same reasoning as in Lemma D.2. We use a notation  $\{x\}_i$  is  $i^{\mathrm{th}}$  block of the vector  $x$ . Let us fix a worker's index  $i \in [n]$ .
Definition E.2 (Sequence  $k_{i,j,l}^{\xi}$ ). Let us consider a set
$$
\left\{k \in \mathbb {N} \mid s _ {i ^ {k}, q} ^ {k - 1} = 1, t ^ {k} \geq s _ {i ^ {k}, t} ^ {k - 1} + \tau_ {i ^ {k}}, \operatorname {p r o g} \left(\left\{s _ {i ^ {k}, x} ^ {k - 1} \right\} _ {i}\right) = j, i ^ {k} = i \right\}, \quad s _ {i ^ {k}} ^ {k - 1} \equiv (s _ {i ^ {k}, t} ^ {k - 1}, s _ {i ^ {k}, q} ^ {k - 1}, s _ {i ^ {k}, x} ^ {k - 1}).
$$