SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We order this set and define the result sequence as  $\{k_{i,j,l}^{\xi}\}_{i=1}^{m_{i,j+1}}$ , where  $m_{i,j+1} \in [0,\infty]$  is the size of the sequence. The sequence  $k_{i,j,l}^{\xi}$  is a subsequence of iterations where the  $i^{\text{th}}$  oracle use the generated Bernoulli random variables in (7) when  $\operatorname{prog}(\{s_x\}_i) = j$ . The sequence  $s_{i^k}^{k-1}$  is defined in Protocol 3.
$$
\eta_ {i, j + 1} := \inf  \left\{l \mid \xi^ {k _ {i, j, l} ^ {\xi}} = 1 \text {a n d} l \in [ 1, m _ {i, j + 1} ] \right\} \in \mathbb {N} \cup \{\infty \}.
$$
The quantity  $\eta_{i,j+1}$  is the index of the first successful trial, when  $\mathrm{prog}(\cdot) = j$  in the  $i^{\text{th}}$  block of the sequence  $x^{k}$ . Since the algorithm  $A$  is a zero-respecting algorithm, for all  $k < k_{i,j,\eta_{i,j+1}}^{\xi}$ , the progress  $\mathrm{prog}(x_{i}^{k}) < j + 1$ .
As in Lemma D.2 (we skip the proof since the idea is the same. It is only required to use the different notations:  $x^{k} \to x_{i}^{k}, \widehat{t}_{\eta_{j}} \to \widehat{t}_{i,\eta_{i,j}}$ ), for all  $i \in [n]$ , one can show that if  $\inf_{k \in S_t} \mathbb{1}[\mathrm{prog}(x_i^k) < T] < 1$  holds, then  $\sum_{j=1}^{T} \widehat{t}_{i,\eta_{i,j}} \leq t$ , where  $\widehat{t}_{i,k} \coloneqq k\tau_i$  for all  $k \geq 1$ . The time  $\widehat{t}_{i,k}$  is the smallest possible time when the  $i^{\text{th}}$  oracle can return the  $k^{\text{th}}$  stochastic gradient. Thus
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {n} \frac {\eta \varepsilon \tau_ {i}}{\sum_ {i = 1} ^ {n} \tau_ {i}} \inf  _ {k \in S _ {t}} \mathbb {1} [ \operatorname {p r o g} (x _ {i} ^ {k}) <   T ] \leq 2 \varepsilon\right) \leq \mathbb {P} \left(\sum_ {i = 1} ^ {n} \frac {\eta \varepsilon \tau_ {i}}{\sum_ {i = 1} ^ {n} \tau_ {i}} \mathbb {1} \left[ \sum_ {j = 1} ^ {T} \widehat {t} _ {i, \eta_ {i, j}} \leq t \right] \leq 2 \varepsilon\right).
$$
Using (25) with  $n = 1$  (and the different notations  $p \to p_i, \tau_1 \to \tau_i$ , and  $\widehat{t}_{\eta_j} \to \widehat{t}_{i,\eta_{i,j}}$ ), we have
$$
\mathbb {P} \left(\sum_ {j = 1} ^ {T} \widehat {t} _ {i, \eta_ {i, j}} \leq t\right) \leq \delta \quad \forall t \leq \frac {\tau_ {i}}{2 4 p _ {i}} \left(\frac {T}{2} + \log \delta\right). \tag {33}
$$
We now rearrange the terms and use Markov's inequality to obtain
$$
\begin{array}{l} \mathbb {P} \left(\sum_ {i = 1} ^ {n} \frac {\eta \varepsilon \tau_ {i}}{\sum_ {i = 1} ^ {n} \tau_ {i}} \mathbb {1} \left[ \sum_ {j = 1} ^ {T} \widehat {t} _ {i, \eta_ {i, j}} > t \right] \leq 2 \varepsilon\right) \\ = \mathbb {P} \left(\sum_ {i = 1} ^ {n} \tau_ {i} \mathbb {1} \left[ \sum_ {j = 1} ^ {T} \widehat {t} _ {i, \eta_ {i, j}} \leq t \right] \geq \left(1 - \frac {2}{\eta}\right) \sum_ {i = 1} ^ {n} \tau_ {i}\right) \\ \leq \left(1 - \frac {2}{\eta}\right) ^ {- 1} \left(\sum_ {i = 1} ^ {n} \tau_ {i}\right) ^ {- 1} \mathbb {E} \left[ \sum_ {i = 1} ^ {n} \tau_ {i} \mathbb {1} \left[ \sum_ {j = 1} ^ {T} \widehat {t} _ {i, \eta_ {i, j}} \leq t \right] \right] \\ = \left(1 - \frac {2}{\eta}\right) ^ {- 1} \left(\sum_ {i = 1} ^ {n} \tau_ {i}\right) ^ {- 1} \sum_ {i = 1} ^ {n} \tau_ {i} \mathbb {P} \left(\sum_ {j = 1} ^ {T} \widehat {t} _ {i, \eta_ {i, j}} \leq t\right), \\ \end{array}
$$
for  $\eta > 2$ . Using the choice of  $p_i$  in (30), we have
$$
\frac {\tau_ {i}}{p _ {i}} \geq \frac {\sigma^ {2} \sum_ {i = 1} ^ {n} \tau_ {i}}{n ^ {2} \gamma_ {\infty} ^ {2} \eta \varepsilon}
$$
for all  $i \in [n]$ . The last term does not depend on  $i$ . Therefore, we can use (33) with
$$
t \leq \frac {1}{2 4} \left(\frac {\sigma^ {2} \sum_ {i = 1} ^ {n} \tau_ {i}}{n ^ {2} \gamma_ {\infty} ^ {2} \eta \varepsilon}\right) \left(\frac {T}{2} + \log \delta\right)
$$
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {n} \frac {\eta \varepsilon \tau_ {i}}{\sum_ {i = 1} ^ {n} \tau_ {i}} \mathbb {1} \left[ \sum_ {j = 1} ^ {T} \widehat {t} _ {i, \eta_ {i, j}} > t \right] \leq 2 \varepsilon\right) \leq \left(1 - \frac {2}{\eta}\right) ^ {- 1} \left(\sum_ {i = 1} ^ {n} \tau_ {i}\right) ^ {- 1} \left(\sum_ {i = 1} ^ {n} \tau_ {i}\right) \delta = \left(1 - \frac {2}{\eta}\right) ^ {- 1} \delta .
$$
Finally, for  $\eta = 4$  and  $\delta = 1 / 4$ , we have
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {n} \frac {\eta \varepsilon \tau_ {i}}{\sum_ {i = 1} ^ {n} \tau_ {i}} \inf _ {k \in S _ {t}} \mathbb {1} [ \mathrm {p r o g} (x _ {i} ^ {k}) <   T ] \leq 2 \varepsilon\right) \leq \frac {1}{2}.
$$
E.3 Proof of Theorem A.3
Theorem A.3. Assume that Assumptions 7.1 and 7.2 hold for the function  $f$ . Assumption 7.3 holds for the function  $f_{i}$  for all  $i \in [n]$ . Let us take the parameter  $S = \max \left\{\left[\sigma^{2} / \varepsilon\right], n\right\}$ , and  $\gamma = \min \left\{\frac{1}{L}, \frac{\varepsilon S}{2L\sigma^{2}}\right\} = \Theta(1 / L)$  in Method 6, then after  $K \geq 24\Delta L / \varepsilon$  iterations the method guarantees that  $\frac{1}{K}\sum_{k=0}^{K-1}\mathbb{E}\left[\left\|\nabla f(x^{k})\right\|^{2}\right] \leq \varepsilon$ .
Proof. Note that Method 6 can be rewritten as  $x^{k + 1} = x^k - \gamma \frac{1}{n}\sum_{i = 1}^{n}\frac{1}{B_i}\sum_{j = 1}^{B_i}\widehat{\nabla} f_i(x^k;\xi_{i,j})$ , where the  $\xi_{i,j}$  are independent random samples. The variance of the gradient estimator equals
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{B _ {i}} \sum_ {j = 1} ^ {B _ {i}} \widehat {\nabla} f _ {i} (x ^ {k}; \xi_ {i, j}) - \nabla f (x ^ {k}) \right\| ^ {2} \right] \\ = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathbb {E} \left[ \left\| \frac {1}{B _ {i}} \sum_ {j = 1} ^ {B _ {i}} \widehat {\nabla} f _ {i} (x ^ {k}; \xi_ {i, j}) - \nabla f _ {i} (x ^ {k}) \right\| ^ {2} \right] \\ = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \frac {1}{B _ {i} ^ {2}} \sum_ {j = 1} ^ {B _ {i}} \mathbb {E} \left[ \left\| \widehat {\nabla} f _ {i} (x ^ {k}; \xi_ {i, j}) - \nabla f _ {i} (x ^ {k}) \right\| ^ {2} \right] \leq \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \frac {\sigma^ {2}}{B _ {i}} = \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{B _ {i}}\right) \frac {\sigma^ {2}}{n} \leq \frac {\sigma^ {2}}{S}, \\ \end{array}
$$
where we use the inequality  $\left(\frac{1}{n}\sum_{i = 1}^{n}\frac{1}{B_i}\right)^{-1}\geq \frac{S}{n}$ , We can use the classical SGD result (see Theorem D.8). For a stepsize
$$
\gamma = \min \left\{\frac {1}{L}, \frac {\varepsilon S}{2 L \sigma^ {2}} \right\},
$$
Using the choice of  $S$ , we obtain that Method 6 converges after
E.4 Proof of Theorem A.4
Proof. The method converges after  $K \times \{ \text{time required to collect batches with the sizes } B_i \text{ such that } \left( \frac{1}{n} \sum_{i=1}^{n} 1 / B_i \right)^{-1} \geq \frac{S}{n} \text{ holds} \}$ .
$$
t _ {i} := \tau_ {i} \left(1 + B _ {i}\right)
$$
because it is possible that a worker finishes the calculation of a gradient from the previous iteration (that we ignore) and only then starts the calculation of a gradient of the current iteration.
Our goal is to find feasible points  $t' \in \mathbb{R}$  and  $B_1, \dots, B_n \in \mathbb{N}$  such that
$$
t^{\prime}\geq \max_{i\in [n]}t_{i},
$$
$$
B _ {1}, \dots , B _ {n} \in \mathbb {N}, \tag {34}
$$
$$
\left(\frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{B _ {i}}\right) ^ {- 1} \geq \frac {S}{n}.
$$
Using the same reasoning as in Theorem 7.5, we relax the assumption that  $B_{i}\in \mathbb{N}$  and assume that  $B_{i}\in \mathbb{R}$  for all  $i\in [n]$ :
$$
B _ {1}, \dots , B _ {n} \in \mathbb {R} \tag {35}
$$
$$
B _ {1}, \dots , B _ {n} > 0
$$
$$
\left(\frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{B _ {i}}\right) ^ {- 1} \geq \frac {S}{n},
$$
If  $t' \in \mathbb{R}$  and  $B_1, \dots, B_n$  are feasible points of (35), then  $2t'$  and  $\lceil B_1 \rceil, \dots, \lceil B_n \rceil$  are feasible points of (34). Let us show that  $t' = 2\left(\tau_n + \left(\frac{1}{n} \sum_{i=1}^{n} \tau_i\right) \frac{S}{n}\right)$  and  $B_i = \frac{t'}{\tau_i} - 1$  are feasible points. Indeed, for all  $i \in [n]$ ,
$$
B _ {i} = \frac {t ^ {\prime}}{\tau_ {i}} - 1 \geq \frac {2 \tau_ {n}}{\tau_ {i}} - 1 \geq 1.
$$
$$
\max _ {i \in [ n ]} t _ {i} = \max _ {i \in [ n ]} \tau_ {i} \left(B _ {i} + 1\right) = t ^ {\prime}
$$
$$
\left(\frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{B _ {i}}\right) ^ {- 1} = \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \frac {\tau_ {i}}{t ^ {\prime} - \tau_ {i}}\right) ^ {- 1} \geq \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \frac {2 \tau_ {i}}{t ^ {\prime}}\right) ^ {- 1} = \frac {t ^ {\prime}}{2} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \tau_ {i}\right) ^ {- 1} \geq \frac {S}{n},
$$
where we use  $t' - \tau_i \geq t'/2 + \tau_i - \tau_i = t'/2$  for all  $i \in [n]$ . Finally, it means that Method 6 returns a solution after
$$
K \times 2 t ^ {\prime} = \frac {9 6 \Delta L}{\varepsilon} \left(\tau_ {n} + \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \tau_ {i}\right) \frac {S}{n}\right)
$$
F Interrupt Oracle Calculations
Let us define a protocol and an oracle where an algorithm can stop the oracle anytime. If an algorithm stops the oracle, its current calculations are canceled and discarded.
Protocol 8 Time Multiple Oracles Protocol With Control
1: Input: functions  $f \in \mathcal{F}$ , oracles and distributions  $((O_1, \ldots, O_n), (\mathcal{D}_1, \ldots, \mathcal{D}_n)) \in \mathcal{O}(f)$ , algorithm  $A \in \mathcal{A}$
2:  $s_i^0 = 0$  for all  $i \in [n]$
3: for  $k = 0, \dots, \infty$  do
4:  $(t^{k + 1},i^{k + 1},c^k,x^k) = A^k (g^1,\dots ,g^k),$
5:  $(s_{i^{k + 1}}^{k + 1},g^{k + 1}) = O_{i^{k + 1}}(t^{k + 1},x^k,c^k,s_{i^{k + 1}}^k,\xi^{k + 1}),\quad \xi^{k + 1}\sim \mathcal{D}$ $\triangleright s_j^{k + 1} = s_j^k\quad \forall j\neq i^{k + 1}$
6: end for
In Protocol 8, we allow algorithms to output the control variables  $c^k$  that can be used in the following oracle.
We take an oracle
$$
O_{\tau}^{\overline{\nabla} f}  :  \mathbb{R}_{\geq 0}\times \mathbb{R}^{d}\times \underbrace{(\mathbb{R}_{\geq 0}\times\mathbb{R}^{d}\times\{0,1\})}_{\text{input state}}\times \underbrace{\{0,1\}}_{\text{control}}\times \mathbb{S}_{\xi}\to \underbrace{(\mathbb{R}_{\geq 0}\times\mathbb{R}^{d}\times\{0,1\})}_{\text{output state}}\times \mathbb{R}^{d}
$$
$$
O _ {\tau} ^ {\widehat {\nabla} f} (t, x, \left(s _ {t}, s _ {x}, s _ {q}\right), c, \xi) = \left\{ \begin{array}{l l} ((t, x, 1), & 0), \\ \left(\left(s _ {t}, s _ {x}, 1\right), & 0\right), & c = 0 \text {a n d} s _ {q} = 0, \\ \left((0, 0, 0), & \widehat {\nabla} f \left(s _ {x}; \xi\right)\right), & c = 0 \text {a n d} s _ {q} = 1 \text {a n d} t <   s _ {t} + \tau , \\ \left((0, 0, 0), & 0\right), & c = 1, \end{array} \right. \tag {36}
$$
and  $\widehat{\nabla} f$  is a mapping such that
$$
\widehat {\nabla} f: \mathbb {R} ^ {d} \times \mathbb {S} _ {\xi} \rightarrow \mathbb {R} ^ {d}.
$$
The oracle (36) generalizes the oracle (7) since an algorithm can send a signal  $c$  to the oracle (36) and interrupt the calculations. Note that if  $c = 1$ , then (36) has the same behavior as (7). But, if  $c = 0$ , then the oracle (36) discards all previous information in the state, and changes  $s_q$  to 0.
Let us define an oracle class:
Definition F.1 (Oracle Class  $\mathcal{O}_{\tau_1,\dots ,\tau_n}^{\sigma^2,\mathrm{stop}}$
Let us consider an oracle class such that, for any  $f \in \mathcal{F}_{\Delta,L}$ , it returns oracles  $O_{i} = O_{\tau_{i}}^{\widehat{\nabla}f}$  and distributions  $\mathcal{D}_i$  for all  $i \in [n]$ , where  $\widehat{\nabla}f$  is an unbiased  $\sigma^2$ -variance-bounded mapping (see Assumption 7.3). The oracles  $O_{\tau_i}^{\widehat{\nabla}f}$  are defined in (36). We define such oracle class as  $\mathcal{O}_{\tau_1,\dots,\tau_n}^{\sigma^2,\mathrm{stop}}$ . Without loss of generality, we assume that  $0 < \tau_{1} \leq \dots \leq \tau_{n}$ .
For the oracle class  $\mathcal{O}_{\tau_1,\dots ,\tau_n}^{\sigma^2,\mathrm{stop}}$  we state that
$$
\mathfrak {m} _ {\mathrm {t i m e}} \left(\mathcal {A} _ {\mathrm {z r}}, \mathcal {F} _ {\Delta , L}\right) = \Omega \left(\min _ {m \in [ n ]} \left[ \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {L \Delta}{\varepsilon} + \frac {\sigma^ {2} L \Delta}{m \varepsilon^ {2}}\right) \right]\right).
$$
The lower bound is the same as for the oracle class from Definition 6.3. We do not provide a formal proof, but a close investigation can reveal that the proof is the same as in Theorem 6.4.
Indeed, in Part 1 of the proof of Lemma D.2, we reduce the inequality  $\inf_{k\in S_t}\mathbb{1}\left[\mathrm{prog}(x^k) < T\right] < 1$  to the inequality  $\sum_{i = 1}^{T}\widehat{t}_{\eta_i}\leq t$ , where  $\widehat{t}_{\eta_i}$  is the shortest time when the oracles can draw a successful Bernoulli random variable. The fact that an algorithm can interrupt the oracles can not change the quantity  $\widehat{t}_{\eta_i}$ .
G Time Complexity with Synchronized Start
In this section, we continue and fill up the discussion in Section 8.
Let us design an oracle for the synchronized start setting. We take an oracle
$$
O_{\tau_{1},\ldots ,\tau_{n}}^{\widehat{\nabla} f}:\mathbb{R}_{\geq 0}\times \mathbb{R}^{d}\times \underbrace{(\mathbb{R}_{\geq 0}\times \mathbb{R}^{d}\times\{0,1\})}_{\text{input state}}\times \underbrace{(\mathbb{S}_{\xi}\times\cdots\times\mathbb{S}_{\xi})}_{n\text{times}}\to \underbrace{(\mathbb{R}_{\geq 0}\times \mathbb{R}^{d}\times\{0,1\})}_{\text{output state}}\times \mathbb{R}^{d}
$$
$$
\begin{array}{l} O _ {\tau_ {1}, \dots , \tau_ {n}} ^ {\widehat {\nabla} f} (t, x, \left(s _ {t}, s _ {x}, s _ {q}\right), \left(\xi_ {1}, \dots , \xi_ {n}\right)) = \\ \left\{ \begin{array}{l l} ((t, x, 1), & 0), \quad s _ {q} = 0, \\ ((0, 0, 0), & 0), \quad s _ {q} = 1 \text {a n d} t \in [ 0, s _ {t} + \tau_ {1}), \\ ((0, 0, 0), & \widehat {\nabla} f \left(s _ {x}; \xi_ {1}\right)), \quad s _ {q} = 1 \text {a n d} t \in \left[ s _ {t} + \tau_ {1}, s _ {t} + \tau_ {2}\right), \\ ((0, 0, 0), & \sum_ {i = 1} ^ {2} \widehat {\nabla} f \left(s _ {x}; \xi_ {i}\right)), \quad s _ {q} = 1 \text {a n d} t \in \left[ s _ {t} + \tau_ {2}, s _ {t} + \tau_ {3}\right), \\ \dots \\ ((0, 0, 0), & \sum_ {i = 1} ^ {n} \widehat {\nabla} f \left(s _ {x}; \xi_ {i}\right)), \quad s _ {q} = 1 \text {a n d} t \in \left[ s _ {t} + \tau_ {n}, \infty\right), \end{array} \right. \tag {37} \\ \end{array}
$$
We assume that  $\xi^{k + 1}$  is a tuple in Protocol 2:  $\xi^{k + 1} \equiv (\xi_1^{k + 1},\dots ,\xi_n^{k + 1}) \sim \mathcal{D}$ .
The oracle (37) with Protocol 2 emulates the behavior of a setting where we broadcast an iterate  $x$  to all workers, and they start calculations simultaneously. The workers have different time delays, hence some finish earlier than others. An algorithm can stop the procedure earlier and get calculated stochastic gradients, but other non-calculated ones will be discarded.
For this oracle, we define an oracle class:
Definition G.1 (Oracle Class  $\mathcal{O}_{\tau_1,\dots ,\tau_n}^{\sigma^2,\mathrm{sync}}$
Let us consider an oracle class such that, for any  $f \in \mathcal{F}_{\Delta,L}$ , it returns an oracle  $O = \mathcal{O}_{\tau_1,\dots,\tau_n}^{\overline{\nabla}f}$  and a distribution  $\mathcal{D}$ , where  $\widehat{\nabla}f$  is an unbiased  $\sigma^2$ -variance-bounded mapping (see Assumption 7.3). The oracle  $O_{\tau_1,\dots,\tau_n}^{\overline{\nabla}f}$  is defined in (37). We define such oracle class as  $\mathcal{O}_{\tau_1,\dots,\tau_n}^{\sigma^2,\mathrm{sync}}$ . Without loss of generality, we assume that  $0 < \tau_1 \leq \dots \leq \tau_n$ .
We now provide the lower bound for Protocol 2 and the oracle class  $\mathcal{O}_{\tau_1,\dots ,\tau_n}^{\sigma^2,\mathrm{sync}}$
Theorem G.2. Let us consider the oracle class  $\mathcal{O}_{\tau_1,\dots,\tau_n}^{\sigma^2,\mathrm{sync}}$  for some  $\sigma^2 > 0$  and  $0 < \tau_1 \leq \dots \leq \tau_n$ . We fix any  $L, \Delta > 0$  and  $0 < \varepsilon \leq c' L \Delta$ . In the view Protocol 2, for any algorithm  $A \in \mathcal{A}_{\mathrm{zr}}$ , there exists a function  $f \in \mathcal{F}_{\Delta,L}$  and an oracle and a distribution  $(O, \mathcal{D}) \in \mathcal{O}_{\tau_1,\dots,\tau_n}^{\sigma^2,\mathrm{sync}}(f)$  such that  $\mathbb{E}\left[\inf_{k \in S_t} \left\| \nabla f(x^k) \right\|^2\right] > \varepsilon$ , where  $S_t := \{k \in \mathbb{N}_0 | t^k \leq t\}$ , and
$$
t = c \times \min _ {m \in [ n ]} \left[ \tau_ {m} \left(\frac {L \Delta}{\varepsilon} + \frac {\sigma^ {2} L \Delta}{m \varepsilon^ {2}}\right) \right].
$$
G.1 Minimax optimal method
In this section, we analyze the  $m$ -Minibatch SGD method (see Method 9). This method generalizes the Minibatch SGD method from Section 1.2. Unlike Minibatch SGD, the  $m$ -Minibatch SGD method only asks for stochastic gradients from the first  $m \in [n]$  (fastest) workers. Later, we show that optimal
Method 9 m-Minibatch SGD
1: Input: starting point  $x^0$ , stepsize  $\gamma$ , number of workers  $m \in [n]$
2: for  $k = 0,1,\dots ,K - 1$  do
3: Send the point  $x^k$  to the first  $m$  workers
4:Receive i.i.d. stochastic gradients  $\widehat{\nabla} f(x^k,\xi_1),\ldots ,\widehat{\nabla} f(x^k,\xi_m)$  from the workers
5:  $x^{k + 1} = x^k -\gamma \frac{1}{m}\sum_{i = 1}^{m}\hat{\nabla} f(x^k,\xi_i)$
6: end for
$m$  is determined by (38). And with this parameter,  $m$ -Minibatch SGD method is minimax optimal under the setting from Sections 8 and G. Note that  $m$ -Minibatch SGD is Minibatch SGD if  $m = n$ .
We now provide the convergence rate and the time complexity.
Theorem G.3. Assume that Assumptions 7.1, 7.2 and 7.3 hold. Let us take the step size
$$
\gamma = \min \left\{\frac {1}{L}, \frac {\varepsilon m}{2 L \sigma^ {2}} \right\}
$$
in Method 9, then after
$$
K \geq \frac {1 2 \Delta L}{\varepsilon} + \frac {1 2 \Delta L \sigma^ {2}}{\varepsilon^ {2} m}
$$
iterations the method guarantees that  $\frac{1}{K}\sum_{k = 0}^{K - 1}\mathbb{E}\left[\| \nabla f(x^k)\|^2\right]\leq \varepsilon$
Theorem G.4. Let us consider Theorem G.3. We assume that  $i^{th}$  worker returns a stochastic gradient every  $\tau_{i}$  seconds for all  $i \in [n]$ . Without loss of generality, we assume that  $0 < \tau_{1} \leq \dots \leq \tau_{n}$ . Let us take
$$
m = \underset {m ^ {\prime} \in [ n ]} {\arg \min } \tau_ {m ^ {\prime}} \left(1 + \frac {\sigma^ {2}}{m ^ {\prime} \varepsilon}\right). \tag {38}
$$
Then after
$$
1 2 \min  _ {m \in [ n ]} \tau_ {m} \left(\frac {L \Delta}{\varepsilon} + \frac {\sigma^ {2} L \Delta}{m \varepsilon^ {2}}\right) \tag {39}
$$
seconds Method 9 guarantees to find an  $\varepsilon$ -stationary point.
Despite the triviality of the  $m$ -Minibatch SGD and its analysis, we provide it to show that the lower bound in Theorem G.2 is tight.
G.2 Proof of Theorem G.2
Theorem G.2. Let us consider the oracle class  $\mathcal{O}_{\tau_1,\dots,\tau_n}^{\sigma^2,\mathrm{sync}}$  for some  $\sigma^2 > 0$  and  $0 < \tau_1 \leq \dots \leq \tau_n$ . We fix any  $L, \Delta > 0$  and  $0 < \varepsilon \leq c'L\Delta$ . In the view Protocol 2, for any algorithm  $A \in \mathcal{A}_{\mathrm{zr}}$ , there exists a function  $f \in \mathcal{F}_{\Delta,L}$  and an oracle and a distribution  $(O,\mathcal{D}) \in \mathcal{O}_{\tau_1,\dots,\tau_n}^{\sigma^2,\mathrm{sync}}(f)$  such that  $\mathbb{E}\left[\inf_{k \in S_t} \left\| \nabla f(x^k) \right\|^2\right] > \varepsilon$ , where  $S_t := \{k \in \mathbb{N}_0 | t^k \leq t\}$ , and
Step 1 and Step 2 mirrors the corresponding steps from the proof of Theorem G.2.
Let us fix  $\lambda > 0$ . We take the same function  $f \in \mathcal{F}_{\Delta, L}$  as in the proof of Theorem 6.4. We define
$$
f (x) := \frac {L \lambda^ {2}}{l _ {1}} F _ {T} \left(\frac {x}{\lambda}\right)
$$
$$
T = \left\lfloor \frac {\Delta l _ {1}}{L \lambda^ {2} \Delta^ {0}} \right\rfloor .
$$
Following the proof of Theorem 6.4, in the oracle  $O$ , we take the following stochastic estimator
$$
[ \widehat {\nabla} f (x; \xi) ] _ {j} := \nabla_ {j} f (x) \left(1 + \mathbb {1} [ j > \operatorname {p r o g} (x) ] \left(\frac {\xi}{p} - 1\right)\right) \quad \forall x \in \mathbb {R} ^ {T}, \tag {40}
$$
and  $\mathcal{D} = \underbrace{\left(\text{Bernoulli}(p),\ldots,\text{Bernoulli}(p)\right)}_{n\text{times}}$ , where  $p\in (0,1]$ . The stochastic gradient is unbiased
and  $\sigma^2$ -variance-bounded if
to ensure that  $\| \nabla f(x)\| ^2 = \frac{L^2\lambda^2}{l_1^2}\left\| \nabla F_T(\frac{x}{\lambda})\right\|^2 >2\varepsilon \mathbb{1}\left[\operatorname {prog}(x) <   T\right]$  for all  $x\in \mathbb{R}^T$  , where we use Lemma D.1. Thus
Protocol 2 generates a sequence  $\{x^k\}_{k = 0}^{\infty}$ . We have
$$
\inf  _ {k \in S _ {t}} \left\| \nabla f \left(x ^ {k}\right) \right\| ^ {2} > 2 \varepsilon \inf  _ {k \in S _ {t}} \mathbb {1} \left[ \operatorname {p r o g} \left(x ^ {k}\right) <   T \right]. \tag {41}
$$
Further, we require the following auxiliary lemma. See the proof in Section D.3.
Lemma G.5. With probability not less than  $1 - \delta$
$$
t \leq \frac {1}{2} \min  _ {m \in [ n ]} \tau_ {m} \left(1 + \frac {1}{4 p m}\right) \left(\frac {T}{2} + \log \delta\right).
$$
Using Lemma G.5 with  $\delta = 1 / 2$  and (41), we obtain
$$
\mathbb {E} \left[ \inf  _ {k \in S _ {t}} \left\| \nabla f (x ^ {k}) \right\| ^ {2} \right] \geq 2 \varepsilon \mathbb {P} \left(\inf  _ {k \in S _ {t}} \mathbb {1} \left[ \operatorname {p r o g} (x ^ {k}) <   T \right]\right) > \varepsilon
$$
$$
t = \frac {1}{2} \min _ {m \in [ n ]} \tau_ {m} \left(1 + \frac {\sigma^ {2}}{8 \gamma_ {\infty} ^ {2} m \varepsilon}\right) \left(\frac {\Delta L}{2 \varepsilon l _ {1} \Delta^ {0}} - 2\right).
$$
G.2.1 Proof of Lemma G.5
Proof. (Part 1): Comment: in this part, we mirror the proof of Lemma D.2. We also show that if  $\inf_{k\in S_t}\mathbb{1}\left[\mathrm{prog}(x^k) < T\right] < 1$  holds, then we have the inequality  $\sum_{i = 1}^{T}\widehat{t}_{\eta_i}\leq t$ , where  $\widehat{t}_{\eta_i}$  are random variables with some known "good" distributions. However, in this lemma the quantities  $\widehat{t}_{\eta_i}$  are different.
In Protocol 2, the algorithm  $A$  consequently calls the oracle  $O$ . If  $\inf_{k \in S_t} \mathbb{1}\left[\mathrm{prog}(x^k) < T\right] < 1$  holds, then exists  $k \in S_t$  such that  $\mathrm{prog}(x^k) = T$ . Since the algorithm  $A$  is zero-respecting, the mappings  $A^k$  will not output a non-zero vector (a vector with a non-zero first coordinate) unless the oracle returns a non-zero vector.
Let us define the smallest index  $k(i)$  of the sequence when the progress  $\mathrm{prog}(x^{k(i)})$  equals  $i$ :
Definition G.6 (Sequence  $(k_p^\xi, i_p^\xi)$ ). Let us consider a set
$$
\{(k, i) \in \mathbb {N} \times [ n ] \mid s _ {q} ^ {k - 1} = 1 \text {a n d} t ^ {k} \geq s _ {t} ^ {k - 1} + \tau_ {i} \}, \quad s ^ {k - 1} \equiv \left(s _ {t} ^ {k - 1}, s _ {q} ^ {k - 1}, s _ {x} ^ {k - 1}\right).
$$
We order this set lexicographically and define the result sequence as  $\{(k_p^\xi, i_p^\xi)\}_{p=1}^m$ , where  $m \in [0, \infty]$  is the size of the sequence. The sequence  $s^{k-1}$  is defined in Protocol 2.
Note that the algorithm  $A$  is only depends on the random samples  $\left\{\xi_{i_p^{\xi}}^{k_\xi}\right\}_{p = 1}^m$  since the sequence  $\{(k_p^\xi ,i_p^\xi)\}_{p = 1}^m$  are the indices of the random samples that are used in the oracle (37).
Let us denote the index of the first successful trial as  $\eta_{1}$ , i.e.,
$$
\eta_ {1} := \inf  \left\{i \middle |   \xi_ {\iota_ {p} ^ {\xi}} ^ {k _ {p} ^ {\xi}} = 1 \text {a n d} p \in [ 1, m ] \right\} \in \mathbb {N} \cup \{\infty \}.
$$
If  $\inf_{k\in S_t}\mathbb{1}\left[\mathrm{prog}(x^k) < T\right] < 1$  holds, then  $\eta_1 < \infty$ . Using the times  $t^k$ , the algorithm  $A$  consequently requests the gradient estimators from the oracle (37).
In each round of the oracle's calculation, the algorithm can get the vector  $\widehat{\nabla} f(s_x;\xi_1)$  in not less than  $\tau_{1}$  seconds, the vector  $\sum_{i = 1}^{2}\widehat{\nabla} f(s_x;\xi_i)$  in less than  $\tau_{2}$  seconds, and so forth (see (37)). The algorithm can repeat any number of these rounds sequentially.