SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we provide Malenia<sup>5</sup> SGD (see Method 6) that is slightly different from Rennala SGD (Method 4). There are two main differences: the first one is that Method 6 has different gradients estimators  $g_{i}^{k}$  for each worker, and the second one is the constraint  $\left(\frac{1}{n}\sum_{i = 1}^{n}1 / B_{i}\right)^{-1} < S / n$  in the inner loop<sup>6</sup>. The more gradients we get from the workers, the larger the term  $\left(\frac{1}{n}\sum_{i = 1}^{n}1 / B_{i}\right)^{-1}$ .
Method 6 Malenia SGD
1: Input: starting point  $x^0$ , stepsize  $\gamma$ , parameter  $S$   
2: Run Method 7 in all workers  
3: for  $k = 0, 1, \dots, K - 1$  do  
4: Init  $g_i^k = 0$  and  $B_i = 0$   
5: while  $\left(\frac{1}{n} \sum_{i=1}^{n} \frac{1}{B_i}\right)^{-1} < \frac{S}{n}$  do  
6: Wait for the next worker  
7: Receive gradient, iteration index, worker's index  $(g, k', i)$   
8: if  $k' = k$  then  
9:  $g_i^k = g_i^k + g$   
10:  $B_i = B_i + 1$   
11: end if  
12: Send  $(x^k, k)$  to the worker  
13: end while  
14:  $g^k = \frac{1}{n} \sum_{i=1}^{n} \frac{1}{B_i} g_i^k$   
15:  $x^{k+1} = x^k - \gamma g^k$   
16: end for
As in Section 7.2, we can provide the convergence theorems.
Theorem A.3. Assume that Assumptions 7.1 and 7.2 hold for the function  $f$ . Assumption 7.3 holds for the function  $f_{i}$  for all  $i \in [n]$ . Let us take the parameter  $S = \max \left\{\left[\sigma^{2} / \varepsilon \right], n\right\}$ , and
4 In the nonconvex case,  $\widehat{L}$  can be arbitrarily larger than  $L$ .
5 https://eldenring.wik.fextralife.com/Malenia+Blade+of+Miquella: Malenia, Blade of Miquella and Malenia, Goddess of Rot is two-phase a Demigod Boss in Elden Ring. She's the twin of Miquella, the most powerful of the Empyreans, and gained renown for her legendary battle against Starscourge Radahn during the Shattering, in which she unleashed the power of the Scarlet Rot and reduced Caelid to ruins.
<sup>6</sup>We assume that  $\left(\frac{1}{n}\sum_{i = 1}^{n}1 / B_i\right)^{-1} = 0$  if exists  $i\in [n]$  such that  $B_{i} = 0$
Method 7 Worker's Infinite Loop
1: Init  $g = 0$ ,  $k' = -1$ , and worker's index  $i$
2: while True do
3: Send  $(g, k', i)$  to the server
4: Receive  $(x^{k}, k)$  from the server
5:  $k' = k$
6:  $g = \widehat{\nabla} f_{i}(x^{k};\xi),\quad \xi \sim \mathcal{D}$
7: end while
$\gamma = \min \left\{\frac{1}{L},\frac{\varepsilon S}{2L\sigma^2}\right\} = \Theta (1 / L)$  in Method 6, then after  $K\geq 24\Delta L / \varepsilon$  iterations the method guarantees that  $\frac{1}{K}\sum_{k = 0}^{K - 1}\mathbb{E}\left[\| \nabla f(x^k)\| ^2\right]\leq \varepsilon$
Theorem A.4. Let us consider Theorem A.3. We assume that  $i^{th}$  worker returns a stochastic gradient every  $\tau_{i}$  seconds for all  $i \in [n]$ . Without loss of generality, we assume that  $0 < \tau_{1} \leq \dots \leq \tau_{n}$ . Then after
$$
9 6 \left(\tau_ {n} \frac {L \Delta}{\varepsilon} + \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \tau_ {i}\right) \frac {\sigma^ {2} L \Delta}{n \varepsilon^ {2}}\right) \tag {15}
$$
seconds, Method 6 guarantees to find an  $\varepsilon$ -stationary point.
Comparing Theorem A.2 and Theorem A.4, one can see that the complexity (15) is optimal. Note that Theorem A.3 holds without assumptions that the delays  $\tau_{i}$  are fixed.
A.4 Discussion
Unlike Asynchronous SGD and PIAG, Malenia SGD ignores all stochastic gradients that were calculated in the previous iterations, which appears to be counterproductive. Nevertheless, we show that Malenia SGD converges, and the time complexity is optimal with respect to all parameters. Note that Malenia SGD does not require the Lipschitz smoothness of the local functions  $f_{i}$ , does not depend on the time delays  $\tau_{i}$ , does not need any similarity assumptions about the functions  $f_{i}$ , and can be applied to problems where the function is not Lipschitz (does not have bounded gradients). The analysis of the method is elementary and does not go far away from the theory of the classical SGD method. When the ratio  $\sigma^2 /\varepsilon$  is large, Malenia SGD is better than Minibatch SGD (see (2)) by  $\Theta \left(\tau_n / \left(\frac{1}{n}\sum_{i = 1}^{n}\tau_i\right)\right)$  times.
B Convex Case
B.1 Lower Bound
Let us consider the optimization problem (1) in the case when the function  $f$  is convex. For the convex case, using Protocol 2, we propose to use another complexity measure instead of (6):
$$
\mathfrak {m} _ {\text {t i m e}} (\mathcal {A}, \mathcal {F}) := \inf  _ {A \in \mathcal {A}} \sup  _ {f \in \mathcal {F}} \sup  _ {(O, \mathcal {D}) \in \mathcal {O} (f)} \inf  \left\{t \geq 0 \left| \mathbb {E} \left[ \inf  _ {k \in S _ {t}} f \left(x ^ {k}\right) \right] - \inf  _ {x \in Q} f (x) \leq \varepsilon \right. \right\}, \tag {16}
$$
$$
S _ {t} := \left\{k \in \mathbb {N} _ {0} \mid t ^ {k} \leq t \right\},
$$
where the sequences  $t^k$  and  $x^k$  are generated by Protocol 2. Let us consider the following class of convex functions:
Definition B.1 (Function Class  $\mathcal{F}_{R,M,L}^{\mathrm{conv}}$ )
Let us define  $B_{2}(0,R)\coloneqq \left\{x\in \mathbb{R}^{d}\mid \| x\| \leq R\right\}$ . We assume that a function  $f:\mathbb{R}^d\to \mathbb{R}$  is convex, differentiable,  $L$ -smooth on the set  $B_{2}(0,R)$ , i.e.,
$$
\| \nabla f (x) - \nabla f (y) \| \leq L \| x - y \| \quad \forall x, y \in B _ {2} (0, R),
$$
and  $M$ -Lipschitz on the set  $B_{2}(0,R)$ , i.e.,
$$
\left| f (x) - f (y) \right| \leq M \| x - y \| \quad \forall x, y \in B _ {2} (0, R).
$$
A set of all functions with such properties we define as  $\mathcal{F}_{R,M,L}^{\mathrm{conv}}$ .
Table 3: Convex Homogeneous Case. The required time to get an  $\varepsilon$ -solution in the convex setting, where  $i^{\mathrm{th}}$  worker requires  $\tau_{i}$  seconds to calculate a stochastic gradient. We assume that  $0 < \tau_{1} \leq \dots \leq \tau_{n}$ .
Method	Time Complexity
Minibatch SGD	τn(min{√LR/√ε, M2R2/ε2} + σ2R2/nε2)
Asynchronous SGD(Mishchenko et al., 2022)	(1/n ∑i=1n 1/τi)-1(LR/ε + σ2R2/nε2)
(Accelerated) Rennala SGD(Theorems B.9 and B.11)	minm∈[n] [ (1/m ∑i=1m 1/τi)-1(min{√LR/√ε, M2R2/ε2} + σ2R2/mε2)
Lower Bound (Theorem B.4)	minm∈[n] [ (1/m ∑i=1m 1/τi)-1(min{√LR/√ε, M2R2/ε2} + σ2R2/mε2)
Lower Bound (Section M)(Woodworth et al., 2018)	τ1min{√LR/√ε, M2R2/ε2} + (1/n ∑i=1n 1/τi)-1σ2R2/nε2
For the convex case, we analyze the following class of algorithms:
Definition B.2 (Algorithm Class  $\mathcal{A}_{\mathrm{tr}}^R$ ).
Let us consider Protocol 3. We say that an algorithm  $A$  from Definition 4.1 belongs to a class  $\mathcal{A}_{\mathrm{zr}}^R$  iff  $A \in \mathcal{A}_{\mathrm{zr}}$  and  $x^{k} \in B_{2}(0,R)$  for all  $k \geq 0$ .
We also define an oracle class:
Definition B.3 (Oracle Class  $\mathcal{O}_{\tau_1,\dots ,\tau_n}^{\mathrm{conv},\sigma^2}$
Let us consider an oracle class such that, for any  $f \in \mathcal{F}_{R,M,L}^{\mathrm{conv}}$ , it returns oracles  $O_{i} = O_{\tau_{i}}^{\widehat{\nabla}f}$  and distributions  $\mathcal{D}_i$  for all  $i \in [n]$ , where  $\widehat{\nabla}f$  is an unbiased  $\sigma^2$ -variance-bounded mapping on the set  $B_2(0,R)$ . The oracles  $O_{\tau_i}^{\widehat{\nabla}f}$  are defined in (7). We define such oracle class as  $\mathcal{O}_{\tau_1,\dots,\tau_n}^{\mathrm{conv},\sigma^2}$ . Without loss of generality, we assume that  $0 < \tau_1 \leq \dots \leq \tau_n$ .
For this setup, we provide the lower bound for the class of convex functions in the next theorem.
Theorem B.4. Let us consider the oracle class  $\mathcal{O}_{\tau_1,\dots ,\tau_n}^{\mathrm{conv},\sigma^2}$  for some  $\sigma^2 >0$  and  $0 < \tau_{1}\leq \dots \leq \tau_{n}$ . We fix any  $R,L,M,\varepsilon >0$  such that  $\sqrt{L} R > c_1\sqrt{\varepsilon} >0$  and  $M^{2}R^{2} > c_{2}\varepsilon^{2}$ . In the view Protocol 3, for any algorithm  $A\in \mathcal{A}_{\mathrm{zr}}^R$ , there exists a function  $f\in \mathcal{F}_{R,M,L}^{\mathrm{conv}}$  and oracles and distributions  $((O_{1},\ldots ,O_{n}),(\mathcal{D}_{1},\ldots ,\mathcal{D}_{n}))\in \mathcal{O}_{\tau_{1},\dots ,\tau_{n}}^{\mathrm{conv},\sigma^{2}}(f)$  such that
$$
\mathbb {E} \left[ \inf  _ {k \in S _ {t}} f (x ^ {k}) \right] - \inf  _ {x \in B _ {2} (0, R)} f (x) > \varepsilon ,
$$
where  $S_{t}\coloneqq \left\{k\in \mathbb{N}_{0}|t^{k}\leq t\right\}$  , and
$$
t = c \times \min _ {m \in [ n ]} \left[ \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\min \left\{\frac {\sqrt {L} R}{\sqrt {\varepsilon}}, \frac {M ^ {2} R ^ {2}}{\varepsilon^ {2}} \right\} + \frac {\sigma^ {2} R ^ {2}}{m \varepsilon^ {2}}\right) \right].
$$
The quantities  $c_{1}, c_{2}$  and  $c$  are universal constants.
B.1.1 Discussion
We improve the lower bound obtained by (Woodworth et al., 2018) (see Table 3). Woodworth et al. (2018) try to reduce any optimization problem to an oracle graph. Then, they get a lower bound using the depth and the number of nodes in a graph. Our approach is different, as we directly estimate the required time and avoid the reduction to an oracle graph. One can think that our "oracle graph" is always linear in Protocol 3, but every node in an "oracle graph" is associated with a timestamp and an index. Unlike the oracle in (Woodworth et al., 2018), which always returns a stochastic gradient, our oracle (7) returns a stochastic gradient only if the conditions are satisfied. Also, Woodworth et al. (2018) construct different "worst case" functions and oracles for the "optimization" and "statistical" terms. While our construction consists only of one function and one oracle.
B.2 Minimax optimal method
B.2.1 Assumptions
Additionally to some assumptions from Section 7.1, we use the following assumptions in the convex case.
Assumption B.5. The function  $f$  is convex and attains the minimum at some point  $x^{*} \in \mathbb{R}^{d}$ .
Assumption B.6. The function  $f$  is  $M$ -Lipschitz, i.e.,
$$
\left| f (x) - f (y) \right| \leq M \| x - y \|, \quad \forall x, y \in \mathbb {R} ^ {d}.
$$
Assumption B.7. For all  $x \in \mathbb{R}^d$ , stochastic gradients  $\widehat{\nabla} f(x;\xi)$  are unbiased and have  $\sigma^2$ -variance-bounded, i.e.,  $\mathbb{E}_{\xi \sim \mathcal{D}}\left[\widehat{\nabla} f(x;\xi)\right] \in \partial f(x)$  and  $\mathbb{E}_{\xi \sim \mathcal{D}}\left[\left\| \widehat{\nabla} f(x;\xi) - \mathbb{E}\left[\widehat{\nabla} f(x;\xi)\right]\right\|^2\right] \leq \sigma^2$ , where  $\sigma^2 \geq 0$ .
B.2.2 Analysis of Rennala SGD and Accelerated Rennala SGD in convex case
Theorem B.8. Assume that Assumptions B.5, B.6 and B.7 hold. Let us take the batch size  $S = \max \left\{\left\lceil \sigma^2 / M^2 \right\rceil, 1 \right\}$ , and  $\gamma = \frac{\varepsilon}{M^2 + \sigma^2 / S} = \Theta(\varepsilon / M^2)$  in Method 4, then after  $K \geq 2M^2 R^2 / \varepsilon^2$  iterations the method guarantees that  $\mathbb{E}\left[f(\widehat{x}^K)\right] - f(x^*) \leq \varepsilon$ , where  $\widehat{x}^K = \frac{1}{K} \sum_{k=0}^{K-1} x^k$  and  $R = \|x^* - x^0\|$ .
Theorem B.9. Let us consider Theorem B.8. We assume that  $i^{th}$  worker returns a stochastic gradient every  $\tau_{i}$  seconds for all  $i \in [n]$ . Without loss of generality, we assume that  $0 < \tau_{1} \leq \dots \leq \tau_{n}$ . Then after
$$
8 \min  _ {m \in [ n ]} \left[ \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {M ^ {2} R ^ {2}}{\varepsilon^ {2}} + \frac {\sigma^ {2} R ^ {2}}{m \varepsilon^ {2}}\right) \right] \tag {17}
$$
seconds Method 4 guarantees to find an  $\varepsilon$ -solution.
Let us provide the theorems in the smooth convex case. We consider the accelerated version of Rennala SGD. In particular, we assume that instead of Line 13 in Method 4, we have
$$
\gamma_ {k + 1} = \gamma (k + 1), \quad \alpha_ {k + 1} = 2 / (k + 2)
$$
$$
y _ {k + 1} ^ {k + 1} = \left(1 - \alpha_ {k + 1}\right) x ^ {k} + \alpha_ {k + 1} u ^ {k}, \quad \left(u ^ {0} = x ^ {0}\right) \tag {18}
$$
$$
u ^ {k + 1} = u ^ {k} - \gamma_ {k + 1} g _ {k}, \tag {18}
$$
$$
x ^ {k + 1} = \left(1 - \alpha_ {k + 1}\right) x ^ {k} + \alpha_ {k + 1} u ^ {k + 1}.
$$
We refer to such method as Accelerated Method 4 or Accelerated Rennala SGD. The acceleration technique is based on (Lan, 2020).
Theorem B.10. Assume that Assumptions B.5, 7.1 and 7.3 hold. Let us take the batch size  $S = \max \left\{\left[\left(\sigma^{2}R\right) / \left(\varepsilon^{3 / 2}\sqrt{L}\right)\right],1\right\}$ , and  $\gamma = \min \left\{\frac{1}{4L},\left[\frac{3R^2S}{4\sigma^2(K + 1)(K + 2)^2}\right]^{1 / 2}\right\}$  in Accelerated Method 4, then after  $K\geq \frac{8\sqrt{L}R}{\sqrt{\varepsilon}}$  iterations the method guarantees that  $\mathbb{E}\left[f(x^K)\right] - f(x^*)\leq \varepsilon$  where  $R\geq \| x^{*} - x^{0}\|$ .
Theorem B.11. Let us consider Theorem B.10. We assume that  $i^{th}$  worker returns a stochastic gradient every  $\tau_{i}$  seconds for all  $i \in [n]$ . Without loss of generality, we assume that  $0 < \tau_{1} \leq \dots \leq \tau_{n}$ . Then after
$$
32 \min  _ {m \in [ n ]} \left[ \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {\sqrt {L} R}{\sqrt {\varepsilon}} + \frac {\sigma^ {2} R ^ {2}}{m \varepsilon^ {2}}\right) \right]
$$
seconds Accelerated Method 4 guarantees to find an  $\varepsilon$ -solution.
Notation	Meaning
g = O(f)	Exist C > 0 such that g(z) ≤ C × f(z) for all z ∈ Z
g = Ω(f)	Exist C' > 0 such that g(z) ≥ C' × f(z) for all z ∈ Z
g = Θ(f)	g = O(f) and g = Ω(f)
{a, ..., b}	Set {i ∈ Z | a ≤ i ≤ b}
[n]	{1, ..., n}
D Proofs for Homogeneous Regime
D.1 The "worst case" function
In this section, we recall the "worst case" function that we use to prove our lower bounds. This is the standard function that is used in nonconvex optimization. Let us define
$$
\operatorname {p r o g} (x) := \max  \{i \geq 0 \mid x _ {i} \neq 0 \} \quad (x _ {0} \equiv 1).
$$
In our proofs, we use the construction from (Carmon et al., 2020; Arjevani et al., 2022). For any  $T \in \mathbb{N}$ , the authors define
$$
F _ {T} (x) := - \Psi (1) \Phi \left(x _ {1}\right) + \sum_ {i = 2} ^ {T} \left[ \Psi \left(- x _ {i - 1}\right) \Phi \left(- x _ {i}\right) - \Psi \left(x _ {i - 1}\right) \Phi \left(x _ {i}\right) \right], \tag {19}
$$
$$
\Psi (x) = \left\{ \begin{array}{l l} 0, & x \leq 1 / 2, \\ \exp \left(1 - \frac {1}{(2 x - 1) ^ {2}}\right), & x \geq 1 / 2, \end{array} \right. \text {a n d} \Phi (x) = \sqrt {e} \int_ {- \infty} ^ {x} e ^ {- \frac {1}{2} t ^ {2}} d t.
$$
The main property of the function  $F_{T}(x)$  is that its gradients are large unless  $\mathrm{prog}(x) \geq T$ .
Lemma D.1 (Carmon et al. (2020); Arjevani et al. (2022)). The function  $F_{T}$  satisfies:
1.  $F_{T}(0) - \inf_{x\in \mathbb{R}^{T}}F_{T}(x)\leq \Delta^{0}T$  , where  $\Delta^0 = 12$
2. The function  $F_{T}$  is  $l_{1}$ -smooth, where  $l_{1} = 152$ .
3. For all  $x\in \mathbb{R}^T$ $\| \nabla F_T(x)\|_{\infty}\leq \gamma_{\infty}$  , where  $\gamma_{\infty} = 23$
4. For all  $x\in \mathbb{R}^T$ $\operatorname {prog}(\nabla F_T(x))\leq \operatorname {prog}(x) + 1.$
5. For all  $x\in \mathbb{R}^T$  , if  $\mathrm{prog}(x) <   T$  , then  $\| \nabla F_T(x)\| >1$
We use these properties in the proofs.
Theorem 6.4. Let us consider the oracle class  $\mathcal{O}_{\tau_1,\dots,\tau_n}^{\sigma^2}$  for some  $\sigma^2 > 0$  and  $0 < \tau_1 \leq \dots \leq \tau_n$ . We fix any  $L, \Delta > 0$  and  $0 < \varepsilon \leq c' L\Delta$ . In view Protocol 3, for any algorithm  $A \in \mathcal{A}_{\mathrm{ZF}}$ , there exists a function  $f \in \mathcal{F}_{\Delta,L}$  and oracles and distributions  $((O_1, \ldots, O_n), (\mathcal{D}_1, \ldots, \mathcal{D}_n)) \in \mathcal{O}_{\tau_1,\dots,\tau_n}^{\sigma^2}(f)$  such that  $\mathbb{E}\left[\inf_{k \in S_t} \left\| \nabla f(x^k) \right\|^2\right] > \varepsilon$ , where  $S_t := \{k \in \mathbb{N}_0 | t^k \leq t\}$ , and
Before we prove the theorem, let us briefly explain the idea. In Steps 1 and 2 of the proof, we construct the appropriate scaled function and stochastic oracles using the function (19). These steps are almost the same as in (Carmon et al., 2020; Arjevani et al., 2022).
In Step 3, we use the zero-chain property of the function (19) and the zero-respecting property of algorithms that would guarantee us that unless oracles send us a non-zero coordinate, an algorithm would not be able to progress to a new coordinate. The oracles send a non-zero coordinate with some probability  $p$ . We have  $n$  parallel oracles that flip random coins in parallel. With a large probability, we show that will not get a new coordinate earlier than
$$
\approx \min  _ {m \in [ n ]} \left[ \left(\sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {1}{p} + m\right) \right]
$$
seconds, where  $\tau_{i}$  are the delays of the oracles. So, with a large probability, we will not be able to solve the optimization earlier than
$$
\approx T \times \min  _ {m \in [ n ]} \left[ \left(\sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {1}{p} + m\right) \right],
$$
where  $T$  is the dimension of the problem.
Proof. (Step 1:  $f \in \mathcal{F}_{\Delta, L}$ )
Let us fix  $\lambda > 0$  and take a function  $f(x) \coloneqq L\lambda^2 / l_1F_T\left(\frac{x}{\lambda}\right)$ , where the function  $F_T$  is defined in Section D.1. Note that the function  $f$  is  $L$ -smooth:
$$
\| \nabla f (x) - \nabla f (y) \| = L \lambda / l _ {1} \left\| F _ {T} \left(\frac {x}{\lambda}\right) - F _ {T} \left(\frac {y}{\lambda}\right) \right\| \leq L \lambda \left\| \frac {x}{\lambda} - \frac {y}{\lambda} \right\| = L \| x - y \| \quad \forall x, y \in \mathbb {R} ^ {d}.
$$
$$
T = \left\lfloor \frac {\Delta l _ {1}}{L \lambda^ {2} \Delta^ {0}} \right\rfloor ,
$$
$$
f (0) - \inf  _ {x \in \mathbb {R} ^ {T}} f (x) = \frac {L \lambda^ {2}}{l _ {1}} \left(F _ {T} (0) - \inf  _ {x \in \mathbb {R} ^ {T}} F _ {T} (x)\right) \leq \frac {L \lambda^ {2} \Delta^ {0} T}{l _ {1}} \leq \Delta .
$$
We showed that the function  $f \in \mathcal{F}_{\Delta, L}$ .
(Step 2: Oracle Class)
In the oracles  $O_{i}$ , we have the freedom to choose a mapping  $\widehat{\nabla} f(\cdot ;\cdot)$  (see (7)). Let us take
$$
[ \widehat {\nabla} f (x; \xi) ] _ {j} := \nabla_ {j} f (x) \left(1 + \mathbb {1} [ j > \operatorname {p r o g} (x) ] \left(\frac {\xi}{p} - 1\right)\right) \quad \forall x \in \mathbb {R} ^ {T},
$$
and  $\mathcal{D}_i = \operatorname{Bernoulli}(p)$  for all  $i \in [n]$ , where  $p \in (0,1]$ . We denote  $[x]_j$  as the  $j^{\text{th}}$  index of a vector  $x \in \mathbb{R}^T$ . It is left to show this mapping is unbiased and  $\sigma^2$ -variance-bounded. Indeed,
$$
\mathbb {E} \left[ [ \widehat {\nabla} f (x, \xi) ] _ {i} \right] = \nabla_ {i} f (x) \left(1 + \mathbb {1} [ i > \operatorname {p r o g} (x) ] \left(\frac {\mathbb {E} [ \xi ]}{p} - 1\right)\right) = \nabla_ {i} f (x)
$$
for all  $i\in [T]$  , and
$$
\mathbb {E} \left[ \left\| \widehat {\nabla} f (x; \xi) - \nabla f (x) \right\| ^ {2} \right] \leq \max _ {j \in [ T ]} | \nabla_ {j} f (x) | ^ {2} \mathbb {E} \left[ \left(\frac {\xi}{p} - 1\right) ^ {2} \right]
$$
because the difference is non-zero only in one coordinate. Thus
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \widehat {\nabla} f (x, \xi) - \nabla f (x) \right\| ^ {2} \right] \leq \frac {\| \nabla f (x) \| _ {\infty} ^ {2} (1 - p)}{p} = \frac {L ^ {2} \lambda^ {2} \left\| F _ {T} \left(\frac {x}{\lambda}\right) \right\| _ {\infty} ^ {2} (1 - p)}{l _ {1} ^ {2} p} \\ \leq \frac {L ^ {2} \lambda^ {2} \gamma_ {\infty} ^ {2} (1 - p)}{l _ {1} ^ {2} p} \leq \sigma^ {2}, \\ \end{array}
$$
where we take
$$
p = \min \left\{\frac {L ^ {2} \lambda^ {2} \gamma_ {\infty} ^ {2}}{\sigma^ {2} l _ {1} ^ {2}}, 1 \right\}.
$$
(Step 3: Analysis of Protocol)
We choose
$$
\lambda = \frac {\sqrt {2 \varepsilon} l _ {1}}{L}
$$
to ensure that  $\| \nabla f(x)\| ^2 = \frac{L^2\lambda^2}{l_1^2}\left\| \nabla F_T(\frac{x}{\lambda})\right\|^2 >2\varepsilon \mathbb{1}$  [prog(x) < T] for all  $x\in \mathbb{R}^T$  , where we use Lemma D.1. Thus
$$
T = \left\lfloor \frac {\Delta L}{2 \varepsilon l _ {1} \Delta^ {0}} \right\rfloor
$$
$$
p = \min \left\{\frac {2 \varepsilon \gamma_ {\infty} ^ {2}}{\sigma^ {2}}, 1 \right\}.
$$
Protocol 3 generates a sequence  $\{x^k\}_{k=0}^{\infty}$ . We have
$$
\inf  _ {k \in S _ {t}} \left\| \nabla f \left(x ^ {k}\right) \right\| ^ {2} > 2 \varepsilon \inf  _ {k \in S _ {t}} \mathbb {1} \left[ \operatorname {p r o g} \left(x ^ {k}\right) <   T \right]. \tag {20}
$$
Using Lemma D.2 with  $\delta = 1 / 2$  and (20), we obtain
$$
\mathbb {E} \left[ \inf  _ {k \in S _ {t}} \left\| \nabla f (x ^ {k}) \right\| ^ {2} \right] \geq 2 \varepsilon \mathbb {P} \left(\inf  _ {k \in S _ {t}} \mathbb {1} \left[ \operatorname {p r o g} (x ^ {k}) <   T \right] \geq 1\right) > \varepsilon
$$
$$
t = \frac {1}{2 4} \min _ {m \in [ n ]} \left[ \left(\sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {\sigma^ {2}}{2 \varepsilon \gamma_ {\infty} ^ {2}} + m\right) \right] \left(\frac {\Delta L}{2 \varepsilon l _ {1} \Delta^ {0}} - 2\right).
$$
D.3 Auxiliary lemmas
D.3.1 Proof of Lemma D.2
Lemma D.2. Let us fix  $T, T' \in \mathbb{N}$  such that  $T \leq T'$ , consider Protocol 3 with a differentiable function  $f: \mathbb{R}^{T'} \to \mathbb{R}$  such that  $\operatorname{prog}(\nabla f(x)) \leq \operatorname{prog}(x) + 1$  for all  $x \in \operatorname{domain}(f)$ , delays  $0 < \tau_1 \leq \dots \leq \tau_n$ , distributions  $\mathcal{D}_i = \text{Bernoulli}(p)$  and oracles  $O_i = O_{\tau_i}^{\widehat{\nabla} f}$  for all  $i \in [n]$ , mappings
$$
[ \widehat {\nabla} f (x; \xi) ] _ {j} = \nabla_ {j} f (x) \left(1 + \mathbb {1} [ j > \operatorname {p r o g} (x) ] \left(\frac {\xi}{p} - 1\right)\right) \quad \forall x \in \mathbb {R} ^ {T ^ {\prime}}, \forall \xi \in \{0, 1 \}, \forall j \in [ T ], \tag {21}
$$
and an algorithm  $A \in \mathcal{A}_{\mathrm{zr}}$ . With probability not less than  $1 - \delta$
$$
\inf  _ {k \in S _ {t}} \mathbb {1} \left[ \operatorname {p r o g} \left(x ^ {k}\right) <   T \right] \geq 1
$$
$$
t \leq \frac {1}{2 4} \min  _ {m \in [ n ]} \left[ \left(\sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {1}{p} + m\right) \right] \left(\frac {T}{2} + \log \delta\right),
$$
where the iterates  $x^{k}$  are defined in Protocol 3.
Proof. (Part 1): Comment: in this part, we formally show that if  $\inf_{k\in S_t}\mathbb{1}\left[\mathrm{prog}(x^k) < T\right] < 1$  holds, then we have the inequality  $\sum_{i = 1}^{T}\widehat{t}_{\eta_i}\leq t$ , where  $\widehat{t}_{\eta_i}$  are random variables with some known "good" distributions. If  $\inf_{k\in S_t}\mathbb{1}\left[\mathrm{prog}(x^k) < T\right] < 1$ , then it means that exists  $k$  such that  $\mathrm{prog}(x^k) = T$ . Note that the algorithm is zero-respecting, so it can not progress to  $T^{th}$  coordinate unless the oracles generate stochastic gradients with non-zero  $1^{st},2^{nd},\ldots ,T^{th}$  coordinates. The oracles flip coins in parallel, so the algorithm should wait for the moment when the oracles flip a success. At the same time, it takes time to generate a coin (calculate a stochastic gradient), and the oracles can not flip more than  $k$  coins before some time  $\widehat{t}_k$ . So if the  $\eta_{i}$  is an index of the first success to generate a non-zero  $i^{th}$  coordinate, then the algorithm should wait at least  $\widehat{t}_{\eta_i}$  seconds. Next, we give a formal proof.
Let us fix  $t \geq 0$  and define the smallest index  $k(i)$  of the sequence when the progress  $\mathrm{prog}(x^{k(i)})$  equals  $i$ :
$$
k (i) := \inf  \left\{k \in \mathbb {N} _ {0} \mid i = \operatorname {p r o g} \left(x ^ {k}\right) \right\} \in \mathbb {N} _ {0} \cup \{\infty \}.
$$
If  $\inf_{k\in S_t}\mathbb{1}\left[\mathrm{prog}(x^k) < T\right] < 1$  holds, then exists  $k\in S_t$  such that  $\mathrm{prog}(x^k) = T$ , thus, by the definition of  $k(T)$ ,  $t^{k(T)}\leq t^k\leq t$ , and  $k(T) < \infty$ . Note that  $t^{k(T)}$  is the smallest time when we make progress to the  $T^{\mathrm{th}}$  coordinate.