SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since  $x^0 = 0$  and  $A$  is a zero-respecting algorithm, the algorithm can return a vector  $x^k$  with the non-zero first coordinate only if some of returned by the oracles stochastic gradients have the first coordinate not equal to zero. The oracles  $O_i$  are constructed in such a way (see (21) and (7)) that they zero out a coordinate based on i.i.d. Bernoulli trials.
Definition D.3 (Sequence  $k_j^\xi$ ). Let us consider a set
$$
\{k \in \mathbb {N} | s _ {i ^ {k}, q} ^ {k - 1} = 1 \text {a n d} t ^ {k} \geq s _ {i ^ {k}, t} ^ {k - 1} + \tau_ {i ^ {k}} \}, \quad s _ {i ^ {k}} ^ {k - 1} \equiv (s _ {i ^ {k}, t} ^ {k - 1}, s _ {i ^ {k}, q} ^ {k - 1}, s _ {i ^ {k}, x} ^ {k - 1}).
$$
We order this set and define the result sequence as  $\{k_j^\xi\}_{j=1}^m$ , where  $m \in [0, \infty]$  is the size of the sequence. The sequence  $k_i^\xi$  is a subsequence of iterations where the oracles use the generated Bernoulli random variables in the third output of (7). The sequence  $s_{i^k}^{k-1}$  is defined in Protocol 3.
Let  $k_{\mathrm{success}}$  be the first iteration index when the oracles use a draw  $\xi = 1$ , i.e.,
$$
k _ {\text {s u c c e s s}} := \inf  \{k \mid \xi^ {k} = 1 \text {a n d} k \in \left\{k _ {j} ^ {\xi} \right\} _ {j = 1} ^ {m} \} \in \mathbb {N} \cup \{\infty \}.
$$
Since the algorithm  $A$  is a zero-respecting and the function  $f$  is a zero-chain function, i.e.,  $\operatorname{prog}(\nabla f(x)) \leq \operatorname{prog}(x) + 1$  for all  $x \in \operatorname{domain}(f)$ , then  $\operatorname{prog}(g^k) = \operatorname{prog}(x^k) = 0$  for all  $k < k_{\mathrm{success}}$ . If  $\inf_{k \in S_t} \mathbb{1}\left[\operatorname{prog}(x^k) < T\right] < 1$  holds, then  $k_{\mathrm{success}} < \infty$ , and  $t^{k_{\mathrm{success}}} \leq t^{k(1)}$ .
The oracles use the generated Bernoulli random variables  $\{\xi^k\mid k\in \{k_j^\xi \}_{j = 1}^m\}$ . Let us denote the index of the first successful trial as  $\eta_{1}$ , i.e.,
$$
\eta_ {1} := \inf  \left\{i \mid \xi^ {k _ {i} ^ {\xi}} = 1 \text {a n d} i \in [ 1, m ] \right\} \in \mathbb {N} \cup \{\infty \}.
$$
The  $i^{\mathrm{th}}$  worker can generate the first Bernoulli random variable not earlier than after  $\tau_{i}$  seconds, the second Bernoulli random variable not earlier than after  $2\tau_{i}$  seconds, and so forth.
Definition D.4 (Sequence  $\widehat{t}_k$ ). Let us consider a multi-set of times
$$
\{j \tau_ {i} \mid j \geq 1, i \in [ n ] \} \equiv \left\{\tau_ {1}, 2 \tau_ {1}, \dots \right\} \uplus \dots \uplus \left\{\tau_ {n}, 2 \tau_ {n}, \dots \right\}.
$$
We order this multi-set and define the result sequence as  $\{\widehat{t_k}\}_{k = 1}^{\infty}$ , and  $\widehat{t_{\infty}} \coloneqq \lim_{k\to \infty}\widehat{t}_k = \infty$ .
Then  $\eta_1^{\mathrm{th}}$  Bernoulli random variable can not be generated earlier than  $\widehat{t}_{\eta_1}$  because  $\widehat{t}_{\eta_1}$  is the earliest time when the oracles can generate  $\eta_1$  random variables. Therefore, if  $\inf_{k\in S_t}\mathbb{1}\left[\mathrm{prog}(x^k) < T\right] < 1$  holds, then  $\widehat{t}_{\eta_1} \leq t^{k_{\mathrm{success}}} \leq t^{k(1)}$ .
Using the same reasoning,  $t^{k(j + 1)}\geq t^{k(j)} + \widehat{t}_{\eta_{j + 1}}$  , where  $\eta_{j + 1}$  is the index of the first successful trial of Bernoulli random variables when  $\mathrm{prog}(\cdot) = j$  in the sequence  $x^k$  . More formally:
Definition D.5 (Sequence  $k_{j,i}^{\xi}$ ). Let us consider a set
$$
\{k \in \mathbb {N} \mid s _ {i ^ {k}, q} ^ {k - 1} = 1 \text {a n d} t ^ {k} \geq s _ {i ^ {k}, t} ^ {k - 1} + \tau_ {i ^ {k}} \text {a n d} \operatorname {p r o g} (s _ {i ^ {k}, x} ^ {k - 1}) = j \}.
$$
We order this set and define the result sequence as  $\{k_{j,i}^{\xi}\}_{i = 1}^{m_{j + 1}}$ , where  $m_{j + 1}\in [0,\infty ]$  is the size of the sequence. The sequence  $k_{j,i}^{\xi}$  is a subsequence of iterations where the oracles use the generated Bernoulli random variables in (7) when  $\mathrm{prog}(s_x) = j$ .
$$
\eta_ {j + 1} := \inf  \left\{i \mid \xi^ {k _ {j, i} ^ {\xi}} = 1 \text {a n d} i \in [ 1, m _ {j + 1} ] \right\} \in \mathbb {N} \cup \{\infty \} \quad \forall j \in \{0, \dots , T - 1 \}. \tag {22}
$$
By the definition of  $k(j)$ ,  $x^{k(j)}$  is the first vector of the sequence, that contains a non-zero  $j$ th coordinate. Thus the oracles will start returning stochastic gradients that potentially have a non-zero  $j + 1^{\text{th}}$  coordinate starting only from the iteration  $k(j)$ . Therefore,
$$
t ^ {k (T)} \geq t ^ {k (T - 1)} + \widehat {t} _ {\eta_ {T}} \geq \sum_ {i = 1} ^ {T} \widehat {t} _ {\eta_ {i}}.
$$
Combining the observations, if  $\inf_{k\in S_t}\mathbb{1}\left[\mathrm{prog}(x^k) < T\right] < 1$  holds, then  $\sum_{i = 1}^{T}\widehat{t}_{\eta_i}\leq t^{k(T)}\leq t.$  Thus
$$
\mathbb {P} \left(\inf  _ {k \in S _ {t}} \mathbb {1} [ \operatorname {p r o g} (x ^ {k}) <   T ] <   1\right) \leq \mathbb {P} \left(\sum_ {i = 1} ^ {T} \widehat {t} _ {\eta_ {i}} \leq t\right) \quad \forall t \geq 0.
$$
In Section D.3.2, we prove the following inequality that we use in Part 2 of the proof.
Lemma D.6. Let us take  $l_{j + 1}\in \mathbb{N}$  . Then
$$
\mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \mid \eta_ {j}, \dots , \eta_ {1}\right) \leq (1 - p) ^ {l _ {j + 1} - 1} p
$$
for all  $j\in \{0,\ldots ,T - 1\}$
(Part 2): Comment: in this part, we use the standard technique to bound the large deviations of the sum  $\sum_{i=1}^{T} \widehat{t}_{\eta_i}$ .
Let us fix  $t' \geq 0$ . Recall Definition D.4 of  $\{\widehat{t}_k\}_{k=1}^{\infty}$ . If the number of workers  $n = 1$ , then  $\widehat{t}_k = k\tau_1$  for all  $k \geq 1$ . For  $n > 1$ , the sequence  $\{\widehat{t}_k\}_{k=1}^{\infty}$  has more complicated structure and depends on the delays  $\tau_1, \ldots, \tau_n$ .
For any  $k \geq 1$ , if  $\widehat{t_k} \leq t'$ , then  $k \leq \sum_{i=1}^{n} \left\lfloor \frac{t'}{\tau_i} \right\rfloor$ . Indeed, let us assume that  $k > \sum_{i=1}^{n} \left\lfloor \frac{t'}{\tau_i} \right\rfloor$ . The sequence  $\widehat{t_k}$  is constructed by the ordering the multi-set  $\{j\tau_i \mid j \geq 1, i \in [n]\}$ . The number of elements, which are less or equal to  $t'$ , equals  $\sum_{i=1}^{n} \left\lfloor \frac{t'}{\tau_i} \right\rfloor$ . Thus, we get a contradiction.
$$
\mathbb {P} \left(\widehat {t} _ {\eta_ {j + 1}} \leq t ^ {\prime} \mid \eta_ {j}, \dots , \eta_ {1}\right) \leq \mathbb {P} \left(\eta_ {j + 1} \leq \sum_ {i = 1} ^ {n} \left\lfloor \frac {t ^ {\prime}}{\tau_ {i}} \right\rfloor \mid \eta_ {j}, \dots , \eta_ {1}\right).
$$
Using Lemma D.6, we have
$$
\mathbb {P} \left(\widehat {t} _ {\eta_ {j + 1}} \leq t ^ {\prime} \mid \eta_ {j}, \dots , \eta_ {1}\right) \leq \sum_ {j = 1} ^ {\sum_ {i = 1} ^ {n} \left\lfloor \frac {t ^ {\prime}}{\tau_ {i}} \right\rfloor} (1 - p) ^ {j - 1} p.
$$
If  $0 \leq t' < \tau_{1}$ , then  $\sum_{i=1}^{n} \left\lfloor t'/\tau_{i} \right\rfloor = 0$ , and
$$
\mathbb {P} \left(\widehat {t} _ {\eta_ {j + 1}} \leq t ^ {\prime} \mid \eta_ {j}, \dots , \eta_ {1}\right) = 0.
$$
Otherwise, if  $t' > \tau_1$ , then  $\sum_{i=1}^{n} \left\lfloor t'/\tau_i \right\rfloor \geq 1$ , and
$$
\mathbb {P} \left(\widehat {t} _ {\eta_ {j + 1}} \leq t ^ {\prime} \mid \eta_ {j}, \dots , \eta_ {1}\right) \leq 1 - (1 - p) ^ {\sum_ {i = 1} ^ {n} \left\lfloor \frac {t ^ {\prime}}{\tau_ {i}} \right\rfloor} \leq p \sum_ {i = 1} ^ {n} \left\lfloor \frac {t ^ {\prime}}{\tau_ {i}} \right\rfloor ,
$$
where we use the fact that  $1 - (1 - p)^{m} \leq pm$  for all  $p \in [0, 1]$  and  $m \in \mathbb{N}$ . For all  $t' \geq 0$ , we have
$$
\mathbb {P} \left(\widehat {t} _ {\eta_ {j + 1}} \leq t ^ {\prime} \mid \eta_ {j}, \dots , \eta_ {1}\right) \leq p \sum_ {i = 1} ^ {n} \left\lfloor \frac {t ^ {\prime}}{\tau_ {i}} \right\rfloor .
$$
$$
p ^ {\prime} := p \sum_ {i = 1} ^ {n} \left\lfloor \frac {t ^ {\prime}}{\tau_ {i}} \right\rfloor ,
$$
$$
\mathbb {P} \left(\widehat {t} _ {\eta_ {j + 1}} \leq t ^ {\prime} \mid \eta_ {j}, \dots , \eta_ {1}\right) \leq p ^ {\prime}. \tag {23}
$$
Let us fix  $s \geq 0$  and  $\widehat{t} \geq 0$ . Using the Chernoff method, we have
$$
\begin{array}{l} \mathbb {P} \left(\sum_ {i = 1} ^ {T} \widehat {t} _ {\eta_ {i}} \leq \widehat {t}\right) = \mathbb {P} \left(- s \left(\sum_ {i = 1} ^ {T} \widehat {t} _ {\eta_ {i}}\right) \geq - s \widehat {t}\right) = \mathbb {P} \left(\exp \left(- s \sum_ {i = 1} ^ {T} \widehat {t} _ {\eta_ {i}}\right) \geq \exp (- s \widehat {t})\right) \\ \leq e ^ {s \widehat {t}} \mathbb {E} \left[ \exp \left(- s \sum_ {i = 1} ^ {T} \widehat {t} _ {\eta_ {i}}\right) \right]. \\ \end{array}
$$
Let us bound the expected value separately:
$$
\mathbb {E} \left[ \exp \left(- s \sum_ {i = 1} ^ {T} \widehat {t} _ {\eta_ {i}}\right) \right] = \mathbb {E} \left[ \prod_ {i = 1} ^ {T} \mathbb {E} \left[ e ^ {- s \widehat {t} _ {\eta_ {i}}} \mid \eta_ {i - 1}, \dots , \eta_ {1} \right] \right].
$$
Since  $\widehat{t}_{\eta_i} \geq 0$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ e ^ {- s \widehat {t} _ {\eta_ {i}}} \mid \eta_ {i - 1}, \dots , \eta_ {1} \right] = \mathbb {E} \left[ e ^ {- s \widehat {t} _ {\eta_ {i}}} \mid \widehat {t} _ {\eta_ {i}} \leq t ^ {\prime}, \eta_ {i - 1}, \dots , \eta_ {1} \right] \mathbb {P} \left(\widehat {t} _ {\eta_ {i}} \leq t ^ {\prime} \mid \eta_ {i - 1}, \dots , \eta_ {1}\right) \\ + \mathbb {E} \left[ e ^ {- s \widehat {t} _ {\eta_ {i}}} \Big | \widehat {t} _ {\eta_ {i}} > t ^ {\prime}, \eta_ {i - 1}, \dots , \eta_ {1} \right] (1 - \mathbb {P} (\widehat {t} _ {\eta_ {i}} \leq t ^ {\prime} | \eta_ {i - 1}, \dots , \eta_ {1})) \\ \leq \mathbb {P} \left(\widehat {t} _ {\eta_ {i}} \leq t ^ {\prime} \mid \eta_ {i - 1}, \dots , \eta_ {1}\right) + e ^ {- s t ^ {\prime}} \left(1 - \mathbb {P} \left(\widehat {t} _ {\eta_ {i}} \leq t ^ {\prime} \mid \eta_ {i - 1}, \dots , \eta_ {1}\right)\right) \\ \stackrel {(2 3)} {\leq} p ^ {\prime} + e ^ {- s t ^ {\prime}} \left(1 - p ^ {\prime}\right). \\ \end{array}
$$
$$
\mathbb {E} \left[ \exp \left(- s \sum_ {i = 1} ^ {T} \widehat {t} _ {\eta_ {i}}\right) \right] \leq \left(p ^ {\prime} + e ^ {- s t ^ {\prime}} \left(1 - p ^ {\prime}\right)\right) ^ {T}
$$
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {T} \widehat {t} _ {\eta_ {i}} \leq \widehat {t}\right) \leq e ^ {s \widehat {t}} \left(p ^ {\prime} + e ^ {- s t ^ {\prime}} \left(1 - p ^ {\prime}\right)\right) ^ {T} = e ^ {s \widehat {t} - s t ^ {\prime} T} \left(1 + \left(e ^ {s t ^ {\prime}} - 1\right) p ^ {\prime}\right) ^ {T}.
$$
Let us take  $s = \frac{1}{t'}$ , and get
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {T} \widehat {t} _ {\eta_ {i}} \leq \widehat {t}\right) \leq e ^ {\widehat {t} / t ^ {\prime} - T} \left(1 + (e - 1) p ^ {\prime}\right) ^ {T} \leq e ^ {\widehat {t} / t ^ {\prime} - T + 2 p ^ {\prime} T}. \tag {24}
$$
Let us recall the definition of  $p'$ :
$$
p ^ {\prime} = p \left(\sum_ {i = 1} ^ {n} \left\lfloor \frac {t ^ {\prime}}{\tau_ {i}} \right\rfloor\right)
$$
Now, we have to take the right  $t'$ . We will take it using a nonconstructive definition. Assume that  $t' = \frac{1}{4p} \left( \sum_{i=1}^{j^*} \frac{1}{\tau_i} \right)^{-1}$ , where
$$
j ^ {*} = \inf  \left\{m \in [ n ] \left| \frac {1}{4 p} \left(\sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} <   \tau_ {m + 1} \right. \right\} (\tau_ {n + 1} \equiv \infty).
$$
This set is not empty because  $n$  belongs to it. Using the definition of  $j^*$ , we have
$$
p ^ {\prime} = p \left(\sum_ {i = 1} ^ {n} \left\lfloor \frac {t ^ {\prime}}{\tau_ {i}} \right\rfloor\right) = p \left(\sum_ {i = 1} ^ {j ^ {*}} \left\lfloor \frac {t ^ {\prime}}{\tau_ {i}} \right\rfloor\right) \leq p t ^ {\prime} \left(\sum_ {i = 1} ^ {j ^ {*}} \frac {1}{\tau_ {i}}\right) = \frac {1}{4}.
$$
Substituting this inequality to (24), we obtain
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {T} \widehat {t} _ {\eta_ {i}} \leq \widehat {t}\right) \leq e ^ {\widehat {t} / t ^ {\prime} - \frac {T}{2}}.
$$
For  $\widehat{t} \leq t' \left( \frac{T}{2} + \log \delta \right)$ , we have
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {T} \widehat {t} _ {\eta_ {i}} \leq \widehat {t}\right) \leq \delta .
$$
Recall that  $t' = \frac{1}{4p}\left(\sum_{i=1}^{j^*}\frac{1}{\tau_i}\right)^{-1}$ . Using Lemma D.7, we have
$$
t ^ {\prime} \geq \frac {1}{2 4} \min  _ {m \in [ n ]} \left[ \left(\sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {1}{p} + m\right) \right].
$$
Finally, we obtain
$$
\mathbb {P} \left(\inf  _ {k \in S _ {t}} \mathbb {1} [ \operatorname {p r o g} (x ^ {k}) <   T ] <   1\right) \leq \mathbb {P} \left(\sum_ {i = 1} ^ {T} \widehat {t} _ {\eta_ {i}} \leq t\right) \leq \delta \tag {25}
$$
$$
t \leq \frac {1}{2 4} \min  _ {m \in [ n ]} \left[ \left(\sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {1}{p} + m\right) \right] \left(\frac {T}{2} + \log \delta\right).
$$
D.3.2 Proof of Lemma D.6
In the following lemma, we use notations from Part 1 of the proof of Lemma D.2.
In this lemma, we want to bound the probability for the random variable  $\eta_{j+1}$  from (22).  $\eta_{j+1}$  is the index of the first successful trial of the sequence of Bernoulli random variables. At first sight, this is a trivial task since  $\eta_{j+1}$  has a distribution similar to the geometric distribution. But the main problem here is that the sequence  $k_{j,i}^{\xi}$  and the quantity  $m_{j+1}$  are also random variables. Therefore, we must be careful with this.
Proof. Since the image of the random variables  $\eta_j, \ldots, \eta_1$  is in  $\mathbb{N} \cap \{\infty\}$ . Let us take  $l_1, \ldots, l_j \in \mathbb{N} \cup \{\infty\}$ , and prove the theorem for a probability conditioned on an event  $\bigcap_{i=1}^{j} \{\eta_i = l_i\}$  such that  $\mathbb{P}\left(\bigcap_{i=1}^{j} \{\eta_i = l_i\}\right) > 0$ . Therefore, it is enough to prove that
$$
\mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \Bigg | \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right) \leq (1 - p) ^ {l _ {j + 1} - 1} p.
$$
First, assume that exists  $i \in [j]$  such that  $l_i = \infty$ . It means that, for all  $k \geq 0$ ,  $\mathrm{prog}(x^k) < j$ , thus
$$
\mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \left| \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right.\right) = 0
$$
for all  $l_{j+1} \in \mathbb{N}$ . Let us explain this step. If exists  $i \in [j]$  such that  $l_i = \infty$ , then an algorithm never gets a progress to  $j^{\text{th}}$  coordinate, thus  $\eta_{j+1} = \inf \{i | \xi^{k_{j,i}^\xi} = 1 \text{ and } i \in [1, m_{j+1}]\} = \infty$  a.s. because  $m_{j+1} = 0$  and  $k_{j,i}^\xi$  is an empty sequence.
Assume that  $l_{i} < \infty$  for all  $i \in [j]$ . By the definition of  $\eta_{j+1}$ , we have  $m_{j+1} \geq l_{j+1}$  and  $\xi^{k_{j,1}^{\xi}} = \dots = \xi^{k_{j,l_{j+1}-1}^{\xi}} = 0$  and  $\xi^{k_{j,l_{j+1}}^{\xi}} = 1$ . Thus
$$
\mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \left| \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right.\right) \leq \mathbb {P} \left(\bigcap_ {i = 1} ^ {l _ {j + 1} - 1} \{\xi^ {k _ {j, i} ^ {\xi}} = 0 \}, \xi^ {k _ {j, l _ {j + 1}} ^ {\xi}} = 1, m _ {j + 1} \geq l _ {j + 1} \left| \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right.\right).
$$
Since  $k_{j,s}^{\xi} < k_{j,i}^{\xi}$  a.s. for all  $s < i \in [m_{j+1}]$ , using the law of total probability, we have
$$
\begin{array}{l} \mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \left| \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right.\right) \\ \leq \sum_ {k _ {1} <   \dots <   k _ {l _ {j + 1}} = 1} ^ {\infty} \mathbb {P} \left(\bigcap_ {i = 1} ^ {l _ {j + 1} - 1} \{\xi^ {k _ {j, i} ^ {\xi}} = 0 \}, \xi^ {k _ {j, l _ {j + 1}} ^ {\xi}} = 1, m _ {j + 1} \geq l _ {j + 1}, \bigcap_ {i = 1} ^ {l _ {j + 1}} \{k _ {j, i} ^ {\xi} = k _ {i} \} \Bigg | \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right) \\ = \sum_ {k _ {1} <   \dots <   k _ {l _ {j + 1}} = 1} ^ {\infty} \mathbb {P} \left(\bigcap_ {i = 1} ^ {l _ {j + 1} - 1} \{\xi^ {k _ {i}} = 0 \}, \xi^ {k _ {l _ {j + 1}}} = 1, m _ {j + 1} \geq l _ {j + 1}, \bigcap_ {i = 1} ^ {l _ {j + 1}} \{k _ {j, i} ^ {\xi} = k _ {i} \} \Bigg | \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right), \\ \end{array}
$$
where  $\sum_{k_1 < \dots < k_{l_{j+1}} = 1}^{\infty}$  is a sum over a set  $\{(k_1, \ldots, k_{l_{j+1}}) \in \mathbb{N}^{l_{j+1}} \mid \forall i < p \in [l_{j+1} : k_i < k_p\}$ . Next, if the event
$$
\bigcap_ {i = 1} ^ {l _ {j + 1}} \left\{k _ {j, i} ^ {\xi} = k _ {i} \right\} \bigcap \left\{m _ {j + 1} \geq l _ {j + 1} \right\}
$$
holds, then an event  $\bigcap_{i = 1}^{l_j + 1}A_{k_i}$  holds, where
$$
A _ {k _ {i}} := \left\{s _ {i ^ {k _ {i}}, q} ^ {k _ {i} - 1} = 1 \text {a n d} t ^ {k _ {i}} \geq s _ {i ^ {k _ {i}}, t} ^ {k _ {i} - 1} + \tau_ {i ^ {k _ {i}}} \text {a n d} \operatorname {p r o g} \left(s _ {i ^ {k _ {i}}, x} ^ {k _ {i} - 1}\right) = j \right\}.
$$
At the same time, if  $\bigcap_{i=1}^{l_{j+1}} A_{k_i}$  holds, then  $\{m_{j+1} \geq l_{j+1}\}$  holds. Therefore,
$$
\bigcap_ {i = 1} ^ {l _ {j + 1}} \left\{k _ {j, i} ^ {\xi} = k _ {i} \right\} \bigcap \left\{m _ {j + 1} \geq l _ {j + 1} \right\} = \bigcap_ {i = 1} ^ {l _ {j + 1}} \left(\left\{k _ {j, i} ^ {\xi} = k _ {i} \right\} \bigcap A _ {k _ {i}}\right)
$$
$$
\mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \left| \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right.\right)
$$
$$
\leq \sum_ {k _ {1} <   \dots <   k _ {l _ {j + 1}} = 1} ^ {\infty} \mathbb {P} \left(\bigcap_ {i = 1} ^ {l _ {j + 1} - 1} \{\xi^ {k _ {i}} = 0 \}, \xi^ {k _ {l _ {j + 1}}} = 1, \bigcap_ {i = 1} ^ {l _ {j + 1}} \left(\{k _ {j, i} ^ {\xi} = k _ {i} \} \bigcap A _ {k _ {i}}\right) \Bigg | \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right).
$$
Let us define  $\sigma(\xi^1, \ldots, \xi^{k_{l_{j+1}-1}})$  as a sigma-algebra generated by  $\xi^1, \ldots, \xi^{k_{l_{j+1}-1}}$ . Note that, for all  $i \in [l_{j+1} - 1]$ , the event  $\{\xi^{k_i} = 0\} \in \sigma(\xi^1, \ldots, \xi^{k_{l_{j+1}-1}})$ . Also, for all  $i \in [l_{j+1}]$ , the event  $\{k_{j,i}^\xi = k_i\} \cap A_{k_i} \in \sigma(\xi^1, \ldots, \xi^{k_{l_{j+1}-1}})$ . Finally, since  $k_{i-1,l_i}^\xi < k_{j,l_{j+1}}^\xi$ , the event
$$
A _ {k _ {l _ {j + 1}}} \bigcap \left\{k _ {j, l _ {j + 1}} ^ {\xi} = k _ {l _ {j + 1}} \right\} \bigcap \left\{\eta_ {i} = l _ {i} \right\} \subseteq \sigma \left(\xi^ {1}, \dots , \xi^ {k _ {l _ {j + 1}} - 1}\right)
$$
for all  $i\in [j]$ . Therefore, the event  $\{\xi^{k_{l_j + 1}} = 1\}$  is independent of the event
$$
\bigcap_ {i = 1} ^ {l _ {j + 1} - 1} \left\{\xi^ {k _ {i}} = 0 \right\} \bigcap_ {i = 1} ^ {l _ {j + 1}} \left(\left\{k _ {j, i} ^ {\xi} = k _ {i} \right\} \bigcap A _ {k _ {i}}\right) \bigcap_ {i = 1} ^ {j} \left\{\eta_ {i} = l _ {i} \right\} \in \sigma \left(\xi^ {1}, \dots , \xi^ {k _ {l _ {j + 1}} - 1}\right)
$$
because  $\xi^k$  are i.i.d. random variables. Using the independence and the equality  $\mathbb{P}\left(\xi^{k_{l_j + 1}} = 1\right) = p$ , we have
$$
\begin{array}{l} \mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \left| \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right.\right) \\ \leq p \sum_ {k _ {1} <   \dots <   k _ {l _ {j + 1}} = 1} ^ {\infty} \mathbb {P} \left(\bigcap_ {i = 1} ^ {l _ {j + 1} - 1} \left\{\xi^ {k _ {i}} = 0 \right\}, \bigcap_ {i = 1} ^ {l _ {j + 1}} \left(\left\{k _ {j, i} ^ {\xi} = k _ {i} \right\} \bigcap A _ {k _ {i}}\right) \Bigg | \bigcap_ {i = 1} ^ {j} \left\{\eta_ {i} = l _ {i} \right\}\right). \\ \end{array}
$$
Since the events  $\{k_{j,l_{j + 1}}^{\xi} = k_{l_{j + 1}}\} \bigcap A_{k_{l_{j + 1}}}$  do not intersect, we can use the additivity of the probability. If  $l_{j + 1} = 1$ , we get
$$
\mathbb {P} \left( \right.\eta_ {j + 1} = l _ {j + 1} \left| \right. \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\left. \right) \leq p \mathbb {P} \left( \right.\bigcup_ {i = 1} ^ {\infty} \left(\left\{k _ {j, l _ {j + 1}} ^ {\xi} = i \right\} \bigcap A _ {i}\right)\left| \right. \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\left. \right)\left. \right) \leq p,
$$
and prove the lemma for  $l_{j + 1} = 1$ . Otherwise, if  $l_{j + 1} > 1$ , we obtain
$$
\begin{array}{l} \mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \left| \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right.\right) \\ \leq p \sum_ {k _ {1} <   \dots <   k _ {l _ {j + 1} - 1} = 1} ^ {\infty} \mathbb {P} \left(\bigcap_ {i = 1} ^ {l _ {j + 1} - 1} \{\xi^ {k _ {i}} = 0 \}, \bigcap_ {i = 1} ^ {l _ {j + 1} - 1} \left(\{k _ {j, i} ^ {\xi} = k _ {i} \} \bigcap A _ {k _ {i}}\right), \right. \\ \bigcup_ {i = k _ {l _ {j + 1} - 1} + 1} ^ {\infty} \left(\left\{k _ {j, l _ {j + 1}} ^ {\xi} = i \right\} \bigcap A _ {i}\right) \left| \bigcap_ {i = 1} ^ {j} \left\{\eta_ {i} = l _ {i} \right\}\right). \\ \end{array}
$$
For any events  $A$  and  $B$ , we have  $\mathbb{P}(A,B)\leq \mathbb{P}(A)$ , thus
$$
\begin{array}{l} \mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \left| \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right.\right) \\ \leq p \sum_ {k _ {1} <   \dots <   k _ {l _ {j + 1} - 1} = 1} ^ {\infty} \mathbb {P} \left(\bigcap_ {i = 1} ^ {l _ {j + 1} - 1} \{\xi^ {k _ {i}} = 0 \}, \bigcap_ {i = 1} ^ {l _ {j + 1} - 1} \left(\{k _ {j, i} ^ {\xi} = k _ {i} \} \bigcap A _ {k _ {i}}\right) \Bigg | \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right). \\ \end{array}
$$
Let us continue for  $l_{j+1} > 1$  and rewrite the last inequality:
$$
\leq p \sum_ {k _ {1} <   \dots <   k _ {l _ {j + 1} - 1} = 1} ^ {\infty} \mathbb {P} \left(\bigcap_ {i = 1} ^ {l _ {j + 1} - 2} \{\xi^ {k _ {i}} = 0 \}, \xi^ {k _ {l _ {j + 1} - 1}} = 0, \bigcap_ {i = 1} ^ {l _ {j + 1} - 1} \left(\left\{k _ {j, i} ^ {\xi} = k _ {i} \right\} \bigcap A _ {k _ {i}}\right) \Bigg | \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right).
$$
Note that, for all  $i \in [l_{j+1} - 2]$ , the event  $\{\xi^{k_i} = 0\} \in \sigma(\xi^1, \ldots, \xi^{k_{l_{j+1}-2}})$ . Also, for all  $i \in [l_{j+1} - 1]$ , the event  $\{k_{j,i}^\xi = k_i\} \cap A_{k_i} \in \sigma(\xi^1, \ldots, \xi^{k_{l_{j+1}-2}})$ . Finally, since  $k_{i-1,l_i}^\xi < k_{j,l_{j+1}-1}^\xi$ , the event
$$
A _ {k _ {l _ {j + 1} - 1}} \bigcap \left\{k _ {j, l _ {j + 1} - 1} ^ {\xi} = k _ {l _ {j + 1} - 1} \right\} \bigcap \left\{\eta_ {i} = l _ {i} \right\} \subseteq \sigma \left(\xi^ {1}, \dots , \xi^ {k _ {l _ {j + 1} - 2}}\right)
$$
for all  $i\in [j]$ . Therefore, the event  $\{\xi^{k_{l_j + 1} - 1} = 0\}$  is independent of the event
$$
\bigcap_ {i = 1} ^ {l _ {j + 1} - 2} \left\{\xi^ {k _ {i}} = 0 \right\} \bigcap_ {i = 1} ^ {l _ {j + 1} - 1} \left(\left\{k _ {j, i} ^ {\xi} = k _ {i} \right\} \bigcap A _ {k _ {i}}\right) \bigcap_ {i = 1} ^ {j} \left\{\eta_ {i} = l _ {i} \right\}.
$$
$$
\begin{array}{l} \mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \left| \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right.\right) \\ \leq p (1 - p) \sum_ {k _ {1} <   \dots <   k _ {l _ {j + 1} - 1} = 1} ^ {\infty} \mathbb {P} \left(\bigcap_ {i = 1} ^ {l _ {j + 1} - 2} \left\{\xi^ {k _ {i}} = 0 \right\}, \bigcap_ {i = 1} ^ {l _ {j + 1} - 1} \left(\left\{k _ {j, i} ^ {\xi} = k _ {i} \right\} \bigcap A _ {k _ {i}}\right) \Bigg | \bigcap_ {i = 1} ^ {j} \left\{\eta_ {i} = l _ {i} \right\}\right). \\ \end{array}
$$
Since the events  $\{k_{j,l_{j+1}-1}^{\xi} = k_{l_{j+1}-1}\} \cap A_{k_{l_{j+1}-1}}$  do not intersect, we use the additivity of the probability. If  $l_{j+1} = 2$ , we get
$$
\begin{array}{l} \mathbb {P} \left(\eta_ {j + 1} = l _ {j + 1} \left| \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right) \leq p (1 - p) \mathbb {P} \left(\left. \bigcup_ {i = 1} ^ {\infty} \left(\left\{k _ {j, l _ {j + 1} - 1} ^ {\xi} = i \right\} \bigcap A _ {i}\right) \right| \bigcap_ {i = 1} ^ {j} \{\eta_ {i} = l _ {i} \}\right)\right) \\ \leq p (1 - p), \\ \end{array}
$$
and prove the lemma for  $l_{j+1} = 2$ . Otherwise, if  $l_{j+1} > 2$ , we obtain