SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The algorithm  $A$  one by one requests  $m_{1},\ldots ,m_{i},\dots \in \{0,\ldots ,n\}$  gradient estimators from the oracle (37). It takes at least  $\tau_{m_i}$  seconds  $(\tau_0\equiv 0)$  to get a vector  $\sum_{i = 1}^{m_i}\widehat{\nabla} f(s_x;\xi_i)$ . Let us consider that  $k\in \mathbb{N}$  is the first index of gradient estimators such that it depends on some  $\xi_{i} = 1$ . Necessarily, we have  $\sum_{j = 1}^{k}m_{j}\geq \eta_{1}$ . Also, since the  $A$  is zero-respecting, we have  $\sum_{j = 1}^{k}\tau_{m_j}\leq t^{k(1)}$ . Note that  $k$ , and  $m_{1},\ldots ,m_{k}$  depend on the algorithm's strategy. Let us find "the best possible" quantities  $k$  and  $m_{1},\ldots ,m_{k}$  that are independent of an algorithm.
Let us assume that  $k^*$ , and  $m_1^*, \ldots, m_k^*$  minimize the quantity
$$
\min  _ {k, m _ {1}, \dots , m _ {k}} \sum_ {j = 1} ^ {k} \tau_ {m _ {j}},
$$
s.t.  $k\in \mathbb{N},$ $m_{1},\ldots m_{k}\in \{0,\ldots ,n\} ,$  (42)
$$
\sum_ {j = 1} ^ {k} m _ {j} \geq \eta_ {1}.
$$
$$
t ^ {k (1)} \geq \sum_ {j = 1} ^ {k ^ {*}} \tau_ {m _ {j} ^ {*}}.
$$
Note that if exists  $j \in [k^{*}]$  such that  $m_j^* > \eta_1$ , then  $k^{*}$ , and  $m_{j-1}^{*}, \ldots, m_{j+1}^{*}, \eta_{1}, m_{j+1}^{*}, \ldots, m_{k}^{*}$  are also minimizers of (42) since the sequence  $\tau_{k}$  is not decreasing. Therefore, (42) is equivalent to
s.t.  $k\in \mathbb{N},$ $m_{1},\ldots m_{k}\in \{0,\ldots ,\eta_{1}\} ,$  (43)
Then, using the simple algebra, we have
$$
\begin{array}{l} t ^ {k (1)} \geq \sum_ {j = 1} ^ {k ^ {*}} \tau_ {m _ {j} ^ {*}} = \sum_ {j: m _ {j} ^ {*} \neq 0} \tau_ {m _ {j} ^ {*}} = \sum_ {j: m _ {j} ^ {*} \neq 0} m _ {j} ^ {*} \frac {\tau_ {m _ {j} ^ {*}}}{m _ {j} ^ {*}} \geq \sum_ {j: m _ {j} ^ {*} \neq 0} m _ {j} ^ {*} \min  _ {m \in [ \eta_ {1} ]} \frac {\tau_ {m}}{m} \\ = \sum_ {j = 1} ^ {k ^ {*}} m _ {j} ^ {*} \min  _ {m \in [ \eta_ {1} ]} \frac {\tau_ {m}}{m} \geq \eta_ {1} \min  _ {m \in [ \eta_ {1} ]} \frac {\tau_ {m}}{m}. \\ \end{array}
$$
In the first inequality, we use that  $m_j^* \in \{0, \dots, \eta_1\}$  for all  $j \in [k^*]$ . Next, using Lemma G.9, we get
$$
t ^ {k (1)} \geq \frac {1}{2} \min _ {m \in [ n ]} \tau_ {m} \left(1 + \frac {\eta_ {1}}{m}\right).
$$
Using the same reasoning, for  $j \in \{0, \dots, T - 1\}$ ,
$$
t ^ {k (j + 1)} \geq t ^ {k (j)} + \frac {1}{2} \min _ {m \in [ n ]} \tau_ {m} \left(1 + \frac {\eta_ {j + 1}}{m}\right),
$$
where  $\eta_{j + 1}$  is the index of the first successful trial of Bernoulli random variables when  $\mathrm{prog}(\cdot) = j$ . More formally, for all  $j\in \{0,\dots ,T - 1\}$ :
Definition G.7 (Sequence  $(k_{j,p}^{\xi},i_{j,p}^{\xi})$ ). Let us consider a set
$\{(k,i)\in \mathbb{N}\times [n]\mid s_q^{k - 1} = 1$  and  $t^k\geq s_t^{k - 1} + \tau_i$  and  $\mathrm{prog}(s_x^{k - 1}) = j\}$ $s^{k - 1}\equiv (s_t^{k - 1},s_q^{k - 1},s_x^{k - 1}).$
We order this set lexicographically and define the result sequence as  $\{(k_{j,p}^{\xi},i_{j,p}^{\xi})\}_{p = 1}^{m_{j + 1}}$ , where  $m_{j + 1}\in [0,\infty ]$  is the size of the sequence. The sequence  $s^{k - 1}$  is defined in Protocol 2.
$$
\eta_ {j + 1} := \inf  \left\{i \middle | \xi_ {i _ {j, p} ^ {\xi}} ^ {k _ {j, p} ^ {\xi}} = 1 \text {a n d} p \in [ 1, m _ {j + 1} ] \right\} \in \mathbb {N} \cup \{\infty \}.
$$
By the definition of  $k(j)$ ,  $x^{k(j)}$  is the first iterate such that  $\mathrm{prog}(\cdot) = j$ . Therefore, the oracle can potentially start returning gradient estimators with the non-zero  $j + 1^{\text{th}}$  coordinate from the  $k(j)^{\text{th}}$  iteration.
Thus, if  $\inf_{k\in S_t}\mathbb{1}\left[\mathrm{prog}(x^k) < T\right] < 1$  holds, then
$$
\frac {1}{2} \sum_ {i = 1} ^ {T} \min  _ {m \in [ n ]} \tau_ {m} \left(1 + \frac {\eta_ {i}}{m}\right) \leq t ^ {k (T)} \leq t.
$$
Finally, we can conclude that
$$
\mathbb {P} \left(\inf  _ {k \in S _ {t}} \mathbb {1} [ \operatorname {p r o g} \left(x ^ {k}\right) <   T ] <   1\right) \leq \mathbb {P} \left(\frac {1}{2} \sum_ {i = 1} ^ {T} \min  _ {m \in [ n ]} \tau_ {m} \left(1 + \frac {\eta_ {i}}{m}\right) \leq t\right) \quad \forall t \geq 0. \tag {44}
$$
As in Lemma D.6, we show that
Lemma G.8. Let us take  $l_{j + 1}\in \mathbb{N}$  . Then
$$
\mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \mid \eta_ {j}, \dots , \eta_ {1}\right) \leq (1 - p) ^ {l _ {j + 1} - 1} p \tag {45}
$$
for all  $j\in \{0,\dots ,T - 1\}$
See the proof in Section G.2.3. Intuitively, the algorithm  $A$  can not increase the probability of getting a successful Bernoulli random variable earlier with its decisions.
Let us temporally define
$$
\widehat {t} _ {\eta_ {i}} := \frac {1}{2} \min  _ {m \in [ n ]} \tau_ {m} \left(1 + \frac {\eta_ {i}}{m}\right)
$$
for all  $i\in [T]$
Note that a function  $g: \mathbb{R} \to \mathbb{R}$  such that  $g(x) := \frac{1}{2} \min_{m \in [n]} \tau_m \left(1 + \frac{x}{m}\right)$  is continuous, strongly-monotone and invertible. For  $t' \geq 0$ , we have
$$
\mathbb {P} \left(\widehat {t} _ {\eta_ {j + 1}} \leq t ^ {\prime} \mid \eta_ {j}, \dots , \eta_ {1}\right) = \mathbb {P} \left(g \left(\eta_ {j + 1}\right) \leq t ^ {\prime} \mid \eta_ {j}, \dots , \eta_ {1}\right) = \mathbb {P} \left(\eta_ {j + 1} \leq g ^ {- 1} \left(t ^ {\prime}\right) \mid \eta_ {j}, \dots , \eta_ {1}\right).
$$
Using (45), we obtain
$$
\mathbb {P} \left(\widehat {t} _ {\eta_ {j + 1}} \leq t ^ {\prime} \mid \eta_ {j}, \dots , \eta_ {1}\right) \leq \sum_ {j = 1} ^ {\left\lfloor g ^ {- 1} \left(t ^ {\prime}\right) \right\rfloor} (1 - p) ^ {j - 1} p \leq p \left\lfloor g ^ {- 1} \left(t ^ {\prime}\right) \right\rfloor .
$$
$$
p ^ {\prime} := p \left\lfloor g ^ {- 1} \left(t ^ {\prime}\right) \right\rfloor ,
$$
$$
\mathbb {P} \left(\widehat {t} _ {\eta_ {j + 1}} \leq t ^ {\prime} \mid \eta_ {j}, \dots , \eta_ {1}\right) \leq p ^ {\prime}.
$$
Using the Chernoff method, as in Lemma D.2, one can get
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {T} \widehat {t} _ {\eta_ {i}} \leq \widehat {t}\right) \leq e ^ {\widehat {t} / t ^ {\prime} - T + 2 p ^ {\prime} T}.
$$
$$
t ^ {\prime} = g \left(\frac {1}{4 p}\right) = \frac {1}{2} \min _ {m \in [ n ]} \tau_ {m} \left(1 + \frac {1}{4 p m}\right),
$$
$$
p ^ {\prime} = p \left\lfloor g ^ {- 1} \left(g \left(\frac {1}{4 p}\right)\right) \right\rfloor = p \left\lfloor \frac {1}{4 p} \right\rfloor \leq \frac {1}{4}.
$$
Using (44), for
$$
t \leq \frac {1}{2} \min  _ {m \in [ n ]} \tau_ {m} \left(1 + \frac {1}{4 p m}\right) \left(\frac {T}{2} + \log \delta\right),
$$
$$
\mathbb {P} \left(\inf  _ {k \in S _ {t}} \mathbb {1} [ \operatorname {p r o g} (x ^ {k}) <   T ] <   1\right) \leq \mathbb {P} \left(\sum_ {i = 1} ^ {T} \widehat {t} _ {\eta_ {i}} \leq t\right) \leq \delta .
$$
The last inequality concludes the proof.
G.2.2 Lemma G.9
Lemma G.9. Let us consider a sorted sequence  $0 < \tau_{1} \leq \dots \leq \tau_{n}$  and a constant  $\eta \in \mathbb{N}$ . We define
$$
t_{1}:= \eta \min_{m\in [\eta ]}\frac{\tau_{m}}{m},
$$
$$
t _ {2} := \min  _ {m \in [ n ]} \tau_ {m} \left(1 + \frac {\eta}{m}\right).
$$
$$
t _ {1} \leq t _ {2} \leq 2 t _ {1}.
$$
$$
m _ {1} := \underset {m \in [ \eta ]} {\arg \min} \frac {\tau_ {m}}{m}
$$
$$
m_{2}:= \operatorname *{arg  min}_{m\in [n]}\tau_{m}\left(1 + \frac{\eta}{m}\right).
$$
Then, using  $m_{1} \leq \eta$ , we have
$$
t _ {2} = \min  _ {m \in [ n ]} \tau_ {m} \left(1 + \frac {\eta}{m}\right) \leq \tau_ {m _ {1}} \left(1 + \frac {\eta}{m _ {1}}\right) \leq 2 \tau_ {m _ {1}} \frac {\eta}{m _ {1}} = 2 t _ {1}.
$$
If  $m_2 \leq \eta$ , then
$$
t _ {1} = \eta \min _ {m \in [ \eta ]} \frac {\tau_ {m}}{m} \leq \eta \frac {\tau_ {m _ {2}}}{m _ {2}} \leq \tau_ {m _ {2}} \left(1 + \frac {\eta}{m _ {2}}\right) = t _ {2}.
$$
Otherwise, if  $m_2 > \eta$
$$
t _ {1} = \eta \min  _ {m \in [ \eta ]} \frac {\tau_ {m}}{m} \leq \eta \frac {\tau_ {\eta}}{\eta} = \tau_ {\eta} \leq \tau_ {m _ {2}} \leq \tau_ {m _ {2}} \left(1 + \frac {\eta}{m _ {2}}\right) = t _ {2}.
$$
G.2.3 Proof of Lemma G.8
In the following lemma, we use notations from Part 1 of the proof of Lemma G.5.
The idea of the following proof repeats the proof of Lemma G.8. But Protocol 2 with the oracle (37) differ, so we present the proof for completeness.
Proof. We prove that
$$
\mathbb {P} \left( \right.\eta_ {j + 1} = l _ {j + 1} \left| \right. \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\left. \right) \leq (1 - p) ^ {l _ {j + 1} - 1} p.
$$
for all  $l_{1},\ldots ,l_{j}\in \mathbb{N}\cup \{\infty \}$  such that  $\mathbb{P}\left(\bigcap_{i = 1}^{j}\{\eta_{i} = l_{i}\}\right) > 0$
If exists  $i\in [j]$  such that  $l_{i} = \infty$  , then
for all  $l_{j + 1}\in \mathbb{N}$  (see details in the proof of Lemma D.6).
Assume that  $l_{i} < \infty$  for all  $i \in [j]$ . By the definition of  $\eta_{j+1}$ , we have  $m_{j+1} \geq l_{j+1}$  and  $\xi_{i_{j,1}}^{k_{j,1}^{\xi}} =$
$$
\dots = \xi_ {i _ {j, l _ {j + 1} - 1} ^ {\xi}} ^ {k _ {j, l _ {j + 1} - 1} ^ {\xi}} = 0 \text {a n d} \xi_ {i _ {j, l _ {j + 1}} ^ {\xi}} ^ {k _ {j, l _ {j + 1}} ^ {\xi}} = 1. \text {T h u s}
$$
$$
\mathbb{P}\left(\eta_{j + 1} = l_{j + 1}\bigg|\bigcap_{i = 1}^{j}\{\eta_{i} = l_{i}\}\right)\leq \mathbb{P}\left(\bigcap_{p = 1}^{l_{j + 1} - 1}\{\xi_{i_{j,p}^{\xi}}^{k_{j,p}^{\xi}} = 0\} ,\xi_{i_{j,p}^{\xi}}^{k_{j,l_{j + 1}^{\xi}}} = 1,m_{j + 1}\geq l_{j + 1}\bigg|\bigcap_{i = 1}^{j}\{\eta_{i} = l_{i}\}\right).
$$
Let us define a set
$$
S _ {l _ {j + 1}} := \left\{\left(\left(k _ {1}, i _ {1}\right), \ldots , \left(k _ {l _ {j + 1}}, i _ {l _ {j + 1}}\right)\right) \in (\mathbb {N} \times [ n ]) ^ {l _ {j + 1}}   \mid \forall p <   j \in [ l _ {j + 1} ]: \left(k _ {p}, i _ {p}\right) <   \left(k _ {j}, i _ {j}\right) \right\} \quad \forall l _ {j + 1} \geq 1.
$$
Using the law of total probability, we have
$$
\begin{array}{l} \mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \Bigg | \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right) \\ \leq \sum_ {S _ {l _ {j + 1}}} \mathbb {P} \left(\bigcap_ {p = 1} ^ {l _ {j + 1} - 1} \{\xi_ {i _ {j, p} ^ {\xi}} ^ {k _ {j, p} ^ {\xi}} = 0 \}, \xi_ {i _ {j, l _ {j + 1}} ^ {\xi}} ^ {k _ {j, l _ {j + 1}} ^ {\xi}} = 1, m _ {j + 1} \geq l _ {j + 1}, \bigcap_ {p = 1} ^ {l _ {j + 1}} \{(k _ {j, p} ^ {\xi}, i _ {j, p} ^ {\xi}) = (k _ {p}, i _ {p}) \} \Bigg | \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right) \\ = \sum_ {S _ {l _ {j + 1}}} \mathbb {P} \left(\bigcap_ {p = 1} ^ {l _ {j + 1} - 1} \{\xi_ {i _ {p}} ^ {k _ {p}} = 0 \}, \xi_ {i _ {l _ {j + 1}}} ^ {k _ {l _ {j + 1}}} = 1, m _ {j + 1} \geq l _ {j + 1}, \bigcap_ {p = 1} ^ {l _ {j + 1}} \{(k _ {j, p} ^ {\xi}, i _ {j, p} ^ {\xi}) = (k _ {p}, i _ {p}) \} \Bigg | \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right). \\ \end{array}
$$
where we take the sum over all  $((k_{1},i_{1}),\ldots ,(k_{l_{j + 1}},i_{l_{j + 1}}))\in S_{l_{j + 1}}$  . If the event
$$
\bigcap_ {p = 1} ^ {l _ {j + 1}} \left\{\left(k _ {j, p} ^ {\xi}, i _ {j, p} ^ {\xi}\right) = \left(k _ {p}, i _ {p}\right) \right\} \bigcap \left\{m _ {j + 1} \geq l _ {j + 1} \right\}
$$
holds, then an event  $\bigcap_{p = 1}^{l_{j + 1}}A_{(k_p,i_p)}$  holds, where
$$
A _ {\left(k _ {p}, i _ {p}\right)} := \left\{s _ {q} ^ {k _ {p} - 1} = 1 \text {a n d} t ^ {k _ {p}} \geq s _ {t} ^ {k _ {p} - 1} + \tau_ {i _ {p}} \text {a n d} \operatorname {p r o g} \left(s _ {x} ^ {k _ {p} - 1}\right) = j \right\}.
$$
At the same time, if  $\bigcap_{p=1}^{l_{j+1}} A_{(k_p, i_p)}$  holds, then  $\{m_{j+1} \geq l_{j+1}\}$  holds. Therefore,
$$
\bigcap_ {p = 1} ^ {l _ {j + 1}} \{(k _ {j, p} ^ {\xi}, i _ {j, p} ^ {\xi}) = (k _ {p}, i _ {p}) \} \bigcap \{m _ {j + 1} \geq l _ {j + 1} \} = \bigcap_ {p = 1} ^ {l _ {j + 1}} \left(\{(k _ {j, p} ^ {\xi}, i _ {j, p} ^ {\xi}) = (k _ {p}, i _ {p}) \} \bigcap A _ {(k _ {p}, i _ {p})}\right)
$$
$$
\begin{array}{l} \mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \left| \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right.\right) \\ \leq \sum_ {S _ {l _ {j + 1}}} \mathbb {P} \left(\bigcap_ {p = 1} ^ {l _ {j + 1} - 1} \{\xi_ {i _ {p}} ^ {k _ {p}} = 0 \}, \xi_ {i _ {l _ {j + 1}}} ^ {k _ {l _ {j + 1}}} = 1, \bigcap_ {p = 1} ^ {l _ {j + 1}} \left(\{(k _ {j, p} ^ {\xi}, i _ {j, p} ^ {\xi}) = (k _ {p}, i _ {p}) \} \bigcap A _ {(k _ {p}, i _ {p})}\right) \Bigg | \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right). \\ \end{array}
$$
Let us define  $\sigma_{j}^{i}$  as a sigma-algebra generated by  $(\xi_1^1,\dots ,\xi_n^1),(\xi_1^2,\dots ,\xi_n^2),\dots ,(\xi_1^i,\dots ,\xi_j^i)$  for all  $i\geq 0$  and  $j\in \{0,\ldots ,n\}$ . Then, we have
$$
\bigcap_ {p = 1} ^ {l _ {j + 1} - 1} \{\xi_ {i _ {p}} ^ {k _ {p}} = 0 \} \in \sigma_ {i _ {l _ {j + 1} - 1}} ^ {k _ {l _ {j + 1} - 1}},
$$
$$
\bigcap_ {p = 1} ^ {l _ {j + 1}} \left(\left\{\left(k _ {j, p} ^ {\xi}, i _ {j, p} ^ {\xi}\right) = \left(k _ {p}, i _ {p}\right) \right\} \bigcap A _ {\left(k _ {p}, i _ {p}\right)}\right) \in \sigma_ {n} ^ {k _ {l _ {j + 1}} - 1}
$$
since, for all  $p \in [l_{j+1}]$ , the event  $A_{(k_p,i_p)}$  is only determined by  $s^{k_p-1}$  and  $t^{k_p}$  that do not depend on  $\{\xi^j\}_{j=k_{l_{j+1}}}^\infty$ . And, for all  $p \in [l_{j+1}]$ , the fact that  $(k_{j,p}^\xi,i_{j,p}^\xi)=(k_p,i_p)$  does not depend on  $\{\xi^j\}_{j=k_{l_{j+1}}}^\infty$ . Note that  $k_{i-1,l_i}^\xi < k_{j,l_{j+1}}^\xi$  (a.s.) for all  $i \in [j]$ . Thus
$$
A _ {\left(k _ {l _ {j + 1}}, i _ {l _ {j + 1}}\right)} \bigcap \left\{\left(k _ {j, l _ {j + 1}} ^ {\xi}, i _ {j, l _ {j + 1}} ^ {\xi}\right) = \left(k _ {l _ {j + 1}}, i _ {l _ {j + 1}}\right) \right\} \bigcap_ {i = 1} ^ {j} \left\{\eta_ {i} = l _ {i} \right\} \in \sigma_ {n} ^ {k _ {l _ {j + 1}} - 1}
$$
since, for all  $i \in [j]$ , this event implies that  $k_{i - 1,l_i}^\xi < k_{l_{j + 1}}$ . All in all, we have that
$$
\bigcap_ {p = 1} ^ {l _ {j + 1} - 1} \{\xi_ {i _ {p}} ^ {k _ {p}} = 0 \} \bigcap_ {p = 1} ^ {l _ {j + 1}} \left(\{(k _ {j, p} ^ {\xi}, i _ {j, p} ^ {\xi}) = (k _ {p}, i _ {p}) \} \bigcap A _ {(k _ {p}, i _ {p})}\right) \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \} \in \sigma_ {n} ^ {k _ {l _ {j + 1}} - 1} \bigcup \sigma_ {i _ {l _ {j + 1} - 1}} ^ {k _ {l _ {j + 1} - 1}}.
$$
Since  $\{\xi_{i_{l_{j + 1}}}^{k_{l_{j + 1}}} = 1\}$  is independent of  $\sigma_n^{k_{l_j + 1} - 1}\bigcup \sigma_{i_{l_j + 1} - 1}^{k_{l_j + 1} - 1}$ , we get
$$
\begin{array}{l} \mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \left| \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right.\right) \\ \leq p \sum_ {S _ {l _ {j + 1}}} \mathbb {P} \left(\bigcap_ {p = 1} ^ {l _ {j + 1} - 1} \left\{\xi_ {i _ {p}} ^ {k _ {p}} = 0 \right\}, \bigcap_ {p = 1} ^ {l _ {j + 1}} \left(\left\{\left(k _ {j, p} ^ {\xi}, i _ {j, p} ^ {\xi}\right) = \left(k _ {p}, i _ {p}\right) \right\} \bigcap A _ {\left(k _ {p}, i _ {p}\right)}\right) \Bigg | \bigcap_ {i = 1} ^ {j} \left\{\eta_ {i} = l _ {i} \right\}\right). \\ \end{array}
$$
If  $l_{j+1} = 1$ , observe that the events  $\{(k_{j,l_{j+1}}^{\xi}, i_{j,l_{j+1}}^{\xi}) = (k_{l_{j+1}}, i_{l_{j+1}})\} \cap A_{(k_{l_{j+1}}, i_{l_{j+1}})}$  do not intersect for all  $(k_{l_{j+1}}, i_{l_{j+1}}) \in \mathbb{N} \times [n]$ . Thus, we can use the additivity of the probability, and obtain
$$
\begin{array}{l} \mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \left| \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right.\right) \\ \leq p \mathbb {P} \left(\bigcup_ {(k, i) \in \mathbb {N} \times [ n ]} \left(\left\{\left(k _ {j, l _ {j + 1}} ^ {\xi}, i _ {j, l _ {j + 1}} ^ {\xi}\right) = (k, i) \right\} \bigcap A _ {(k, i)}\right) \Bigg | \bigcap_ {i = 1} ^ {j} \left\{\eta_ {i} = l _ {i} \right\}\right) \leq p. \\ \end{array}
$$
Otherwise, if  $l_{j+1} > 1$ , we also use the fact that the events do not intersect and get
$$
\begin{array}{l} \mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \left| \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right.\right) \\ \leq p \sum_ {S _ {l _ {j + 1} - 1}} \mathbb {P} \left(\bigcap_ {p = 1} ^ {l _ {j + 1} - 1} \left\{\xi_ {i _ {p}} ^ {k _ {p}} = 0 \right\}, \bigcap_ {p = 1} ^ {l _ {j + 1} - 1} \left(\left\{\left(k _ {j, p} ^ {\xi}, i _ {j, p} ^ {\xi}\right) = \left(k _ {p}, i _ {p}\right) \right\} \bigcap A _ {\left(k _ {p}, i _ {p}\right)}\right), \right. \\ \bigcup_ {(k, i) > \left(k _ {l _ {j + 1} - 1}, i _ {l _ {j + 1} - 1}\right)} \left(\left\{\left(k _ {j, l _ {j + 1}} ^ {\xi}, i _ {j, l _ {j + 1}} ^ {\xi}\right) = (k, i) \right\} \bigcap A _ {(k, i)}\right) \Bigg | \bigcap_ {i = 1} ^ {j} \left\{\eta_ {i} = l _ {i} \right\} \Bigg) \\ \leq p \sum_ {S _ {l _ {j + 1} - 1}} \mathbb {P} \left(\bigcap_ {p = 1} ^ {l _ {j + 1} - 1} \left\{\xi_ {i _ {p}} ^ {k _ {p}} = 0 \right\}, \bigcap_ {p = 1} ^ {l _ {j + 1} - 1} \left(\left\{(k _ {j, p} ^ {\xi}, i _ {j, p} ^ {\xi}) = (k _ {p}, i _ {p}) \right\} \bigcap A _ {(k _ {p}, i _ {p})}\right) \Bigg | \bigcap_ {i = 1} ^ {j} \left\{\eta_ {i} = l _ {i} \right\}\right), \\ \end{array}
$$
where we used an inequality  $\mathbb{P}(A,B)\leq \mathbb{P}(A)$  for any events  $A$  and  $B$ . We take the sum over all  $((k_{1},i_{1}),\ldots ,(k_{l_{j + 1} - 1},i_{l_{j + 1} - 1}))\in S_{l_{j + 1} - 1}$ , where
$$
S _ {l _ {j + 1} - 1} = \left\{\left(\left(k _ {1}, i _ {1}\right), \dots , \left(k _ {l _ {j + 1} - 1}, i _ {l _ {j + 1} - 1}\right)\right) \in \left(\mathbb {N} \times [ n ]\right) ^ {l _ {j + 1} - 1} \mid \forall p <   j \in \left[ l _ {j + 1} - 1 \right]: \left(k _ {p}, i _ {p}\right) <   \left(k _ {j}, i _ {j}\right) \right\}.
$$
Using the same reasoning, one can continue and get that
$$
\mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \left| \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right) \leq p (1 - p) ^ {l _ {j + 1} - 1}. \right.
$$
For all  $i,j\geq 0$  and  $l,p\in \{0,\dots ,n\}$  , the union of  $\sigma_l^i$  and  $\sigma_p^j$  is a sigma-algebra since either  $\sigma_l^i\subseteq \sigma_p^j$  , or  $\sigma_p^j\subseteq \sigma_l^i$
G.3 Proof of Theorem G.3
$$
\gamma = \min  \left\{\frac {1}{L}, \frac {\varepsilon m}{2 L \sigma^ {2}} \right\}
$$
$$
x ^ {k + 1} = x ^ {k} - \gamma \frac {1}{m} \sum_ {i = 1} ^ {m} \widehat {\nabla} f (x ^ {k}, \xi_ {i}),
$$
where the stochastic gradients are i.i.d. Therefore, we can use the classical SGD result (see Theorem D.8). For a stepsize
$$
\gamma = \min  \left\{\frac {1}{L}, \frac {\varepsilon m}{2 L \sigma^ {2}} \right\},
$$
$$
K \geq \frac {1 2 \Delta L}{\varepsilon} + \frac {1 2 \Delta L \sigma^ {2}}{\varepsilon^ {2} m}.
$$
G.4 Proof of Theorem G.4
Proof. In this setup, the method converges after  $K \times \tau_{m}$  seconds because the delay of each iteration is determined by the slowest worker. Thus, the time complexity equals
$$
\tau_ {m} \left(\frac {1 2 \Delta L}{\varepsilon} + \frac {1 2 \Delta L \sigma^ {2}}{\varepsilon^ {2} m}\right) = \frac {1 2 \Delta L}{\varepsilon} \tau_ {m} \left(1 + \frac {\sigma^ {2}}{\varepsilon m}\right) = \frac {1 2 \Delta L}{\varepsilon} \min _ {m ^ {\prime} \in [ n ]} \tau_ {m} \left(1 + \frac {\sigma^ {2}}{m ^ {\prime} \varepsilon}\right),
$$
where we use the choice of the number of workers  $m$ .
H Proofs for Convex Case
H.1 The "worst case" function in convex case
In this proof, we use the construction from (Woodworth et al., 2018). Let us define  $B_{2}(0,R) \coloneqq \{x \in \mathbb{R}^{T + 1} : \| x \| \leq R\}$ .
Let us take functions  $f_{l,\eta}:\mathbb{R}^{T + 1}\to \mathbb{R}$  and  $\widetilde{f}_l,\eta :\mathbb{R}^{T + 1}\to \mathbb{R}$  such that
$$
f _ {l, \eta} (x) := \min  _ {y \in \mathbb {R} ^ {T + 1}} \left\{\widetilde {f} _ {l, \eta} (y) + \frac {\eta}{2} \| y - x \| ^ {2} \right\}
$$
$$
\widetilde {f} _ {l, \eta} (x) := \max _ {1 \leq r \leq T + 1} \left(l x _ {r} - \frac {5 l ^ {2} (r - 1)}{\eta}\right),
$$
where  $l, \eta > 0$  are free parameters. Let us define
$$
y (x) := \underset {y \in \mathbb {R} ^ {T + 1}} {\arg \min } \left\{\widetilde {f} _ {l, \eta} (y) + \frac {\eta}{2} \| y - x \| ^ {2} \right\}.
$$
For this function  $f_{l,\eta}$ , we have the following properties:
Lemma H.1 (Woodworth et al. (2018)). The function  $f_{l,\eta}$  satisfies:
1. (Lemma 4) The function  $f_{l,\eta}$  is convex,  $l$ -Lipschitz, and  $\eta$ -smooth.
2. (eq. 75)
$$
\min_{x\in B_{2}(0,1)}f_{l,\eta}(x)\leq -\frac{l}{\sqrt{T + 1}}.
$$
3. (Lemma 6) For all  $x \in B_2(0,1)$ ,  $\operatorname{prog}(\nabla f_{l,\eta}(x)) \leq \operatorname{prog}(x) + 1$  and  $\operatorname{prog}(y(x)) \leq \operatorname{prog}(x) + 1$ .
H.2 Proof of Theorem B.4
Theorem B.4. Let us consider the oracle class  $\mathcal{O}_{\tau_1,\dots,\tau_n}^{\mathrm{conv},\sigma^2}$  for some  $\sigma^2 > 0$  and  $0 < \tau_1 \leq \dots \leq \tau_n$ . We fix any  $R, L, M, \varepsilon > 0$  such that  $\sqrt{L} R > c_1\sqrt{\varepsilon} > 0$  and  $M^2 R^2 > c_2\varepsilon^2$ . In the view Protocol 3, for any algorithm  $A \in \mathcal{A}_{\mathrm{zr}}^R$ , there exists a function  $f \in \mathcal{F}_{R,M,L}^{\mathrm{conv}}$  and oracles and distributions  $((O_1, \ldots, O_n), (\mathcal{D}_1, \ldots, \mathcal{D}_n)) \in \mathcal{O}_{\tau_1,\dots,\tau_n}^{\mathrm{conv},\sigma^2}(f)$  such that
In Step 1, we almost repeat the proof from Woodworth et al. (2018). Steps 2 and 3 are very close to Steps 2 and 3 of the proofs for the nonconvex case.
Proof. (Step 1:  $f \in \mathcal{F}_{R,M,L}^{\mathrm{conv}}$ ) Following Woodworth et al. (2018), we assume that  $R = 1$ . Otherwise, one can rescale the parameters of the construction. Let us take the function  $f_{l,\eta}$  from Section H.1 with parameters
$$
l = \min  \left\{M, \frac {L}{1 0 (T + 1) ^ {3 / 2}} \right\} \text {a n d} \eta = 1 0 (T + 1) ^ {3 / 2} l.
$$
Using Lemma H.1, one can see that  $f_{l,\eta}$  is convex,  $M$ -Lipschitz, and  $L$ -smooth. Therefore,  $f_{l,\eta} \in \mathcal{F}_{R,M,L}^{\mathrm{conv}}$ . Further, we use the notation  $f \coloneqq f_{l,\eta}$ .
and  $\mathcal{D}_i = \operatorname{Bernoulli}(p)$  for all  $i \in [n]$ , where  $p \in (0,1]$ . We denote  $[x]_j$  as the  $j^{\mathrm{th}}$  index of a vector  $x \in \mathbb{R}^{T+1}$ . It is left to show this mapping is unbiased and  $\sigma^2$ -variance-bounded. Indeed,
for all  $i\in [T + 1]$  , and
$$
\mathbb {E} \left[ \left\| \widehat {\nabla} f (x; \xi) - \nabla f (x) \right\| ^ {2} \right] \leq \max _ {j \in [ T + 1 ]} | \nabla_ {j} f (x) | ^ {2} \mathbb {E} \left[ \left(\frac {\xi}{p} - 1\right) ^ {2} \right]
$$
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \widehat {\nabla} f (x, \xi) - \nabla f (x) \right\| ^ {2} \right] \leq \frac {\| \nabla f (x) \| _ {\infty} ^ {2} (1 - p)}{p} \leq \frac {\| \nabla f (x) \| ^ {2} (1 - p)}{p} \\ \leq \frac {l ^ {2} (1 - p)}{p} \leq \sigma^ {2} \quad \forall x \in B _ {2} (0, 1). \\ \end{array}
$$
$$
p = \min  \left\{\frac {l ^ {2}}{\sigma^ {2}}, 1 \right\}.
$$
Protocol 3 generates the sequence  $\{x^k\}_{k=0}^{\infty}$ . Then, we have
$$
f (x ^ {k}) = \max _ {1 \leq r \leq T + 1} \left(l [ y (x ^ {k}) ] _ {r} - \frac {5 l ^ {2} (r - 1)}{\eta}\right) + \frac {\eta}{2} \left\| y (x ^ {k}) - x ^ {k} \right\| ^ {2} \geq l [ y (x ^ {k}) ] _ {T + 1} - \frac {5 l ^ {2} T}{\eta}.
$$
Assume that  $\operatorname{prog}(x^k) < T$ . Using Lemma H.1, if  $\operatorname{prog}(x^k) < T$ , then  $\operatorname{prog}(y(x^k)) \leq T$  and  $[y(x^k)]_{T+1} = 0$ . Therefore,  $f(x^k) \geq -\frac{5l^2T}{\eta} \geq -\frac{5l^2(T+1)}{\eta}$  and
$$
f (x ^ {k}) - \min _ {x \in B _ {2} (0, 1)} f (x) \geq \frac {l}{\sqrt {T + 1}} - \frac {5 l ^ {2} (T + 1)}{\eta} = \frac {l}{2 \sqrt {T + 1}} = \min \left\{\frac {M}{2 \sqrt {T + 1}}, \frac {L}{2 0 (T + 1) ^ {2}} \right\}
$$
if  $\operatorname{prog}(x^k) < T$ . Let us take
$$
T = \min \left\{\left\lfloor \frac {M ^ {2}}{6 4 \varepsilon^ {2}} - 1 \right\rfloor , \left\lfloor \frac {\sqrt {L}}{\sqrt {8 0} \sqrt {\varepsilon}} - 1 \right\rfloor \right\}
$$
$$
f \left(x ^ {k}\right) - \min  _ {x \in B _ {2} (0, 1)} f (x) \geq \min  \left\{\frac {M}{2 \sqrt {T + 1}}, \frac {L}{2 0 (T + 1) ^ {2}} \right\} \geq 4 \varepsilon > 2 \varepsilon
$$
if  $\mathrm{prog}(x^k) < T$ . It is left to use Lemma D.2 with  $\delta = 1/2$ :
$$
\mathbb {E} \left[ \inf  _ {k \in S _ {t}} f \left(x ^ {k}\right) \right] - \min  _ {x \in B _ {2} (0, 1)} f (x) > 2 \varepsilon \mathbb {P} \left(\inf  _ {k \in S _ {t}} \mathbb {1} \left[ \operatorname {p r o g} \left(x ^ {k}\right) <   T \right] \geq 1\right) > \varepsilon \tag {46}
$$
$$
t \leq \frac {1}{2 4} \min  _ {m \in [ n ]} \left[ \left(\sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {\sigma^ {2}}{l ^ {2}} + m\right) \right] \left(\frac {T}{2} - 1\right)
$$