SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
= \frac {1}{2 4} \min  _ {m \in [ n ]} \left[ \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {\sigma^ {2}}{m l ^ {2}} + 1\right) \right] \left(\frac {T}{2} - 1\right).
$$
We can take universal constants  $c_1$  and  $c_2$  equal to 8-64 to ensure that  $T \geq 6$ . Therefore,  $16\varepsilon \geq \frac{l}{2\sqrt{T + 1}}$  and (46) holds for
$$
t \leq \frac {1}{9 6} \min _ {m \in [ n ]} \left[ \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {\sigma^ {2}}{m l ^ {2}} (T + 1) + (T + 2)\right) \right],
$$
$$
t \leq \frac {1}{9 6} \min _ {m \in [ n ]} \left[ \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {\sigma^ {2}}{m l ^ {2}} \frac {l ^ {2}}{3 2 ^ {2} \varepsilon^ {2}} + (T + 2)\right) \right],
$$
$$
t \leq \frac {1}{9 6} \min  _ {m \in [ n ]} \left[ \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {\sigma^ {2}}{3 2 ^ {2} m \varepsilon^ {2}} + \min  \left\{\frac {M ^ {2}}{6 4 \varepsilon^ {2}}, \frac {\sqrt {L}}{\sqrt {8 0} \sqrt {\varepsilon}} \right\}\right) \right].
$$
H.3 Proof of Theorem B.8
Proof. Using the same reasoning as in the proof of Theorem 7.4, one can see that Method 4 is just the stochastic gradient method with the batch size  $S$ . Method 4 can be rewritten as  $x^{k + 1} = x^k -\gamma \frac{1}{S}\sum_{i = 1}^{S}\widehat{\nabla} f(x^k;\xi_i)$ , where the  $\xi_{i}$  are independent random samples. It means that we can use the classical SGD result (Theorem H.2). For a stepsize
$$
\gamma = \frac {\varepsilon}{M ^ {2} + \frac {\sigma^ {2}}{S}},
$$
$$
\mathbb {E} \left[ f (\widehat {x} ^ {K}) \right] - f (x ^ {*}) \leq \varepsilon
$$
$$
K \geq \frac {2 M ^ {2} \left\| x ^ {*} - x ^ {0} \right\| ^ {2}}{\varepsilon^ {2}} \geq \frac {\left(M ^ {2} + \frac {\sigma^ {2}}{S}\right) \left\| x ^ {*} - x ^ {0} \right\| ^ {2}}{\varepsilon^ {2}}.
$$
H.3.1 The classical SGD theorem in convex optimization
We reprove the classical SGD result (see, for instance, (Lan, 2020)) for convex functions.
Theorem H.2. Assume that Assumptions B.5 and B.6 hold. We consider the SGD method:
$$
\gamma = \frac {\varepsilon}{M ^ {2} + \sigma^ {2}}
$$
For a fixed  $x \in \mathbb{R}^d$ ,  $g(x)$  is a random vector such that  $\mathbb{E}[g(x)] \in \partial f(x)$  ( $\partial f(x)$  is the subdifferential of the function  $f$  at the point  $x$ ),
$$
\mathbb {E} \left[ \| g (x) - \mathbb {E} [ g (x) ] \| ^ {2} \right] \leq \sigma^ {2},
$$
$$
\mathbb {E} \left[ f \left(\frac {1}{K} \sum_ {k = 0} ^ {K - 1} x ^ {k}\right) \right] - f \left(x ^ {*}\right) \leq \varepsilon \tag {47}
$$
$$
K \geq \frac {\left(M ^ {2} + \sigma^ {2}\right) \left\| x ^ {*} - x ^ {0} \right\| ^ {2}}{\varepsilon^ {2}}.
$$
Proof. We denote  $\mathcal{G}^k$  as a sigma-algebra generated by  $g(x^0),\ldots ,g(x^{k - 1})$ . Using the convexity, for all  $x\in \mathbb{R}^d$ , we have
$$
f (x) \geq f (x ^ {k}) + \left\langle \mathbb {E} \left[ g (x ^ {k}) \mid \mathcal {G} ^ {k} \right], x - x ^ {k} \right\rangle = f (x ^ {k}) + \mathbb {E} \left[ \left\langle g (x ^ {k}), x - x ^ {k} \right\rangle \mid \mathcal {G} ^ {k} \right].
$$
$$
\begin{array}{l} \left\langle g \left(x ^ {k}\right), x - x ^ {k} \right\rangle = \left\langle g \left(x ^ {k}\right), x ^ {k + 1} - x ^ {k} \right\rangle + \left\langle g \left(x ^ {k}\right), x - x ^ {k + 1} \right\rangle \\ = - \gamma \left\| g \left(x ^ {k}\right) \right\| ^ {2} + \frac {1}{\gamma} \left\langle x ^ {k} - x ^ {k + 1}, x - x ^ {k + 1} \right\rangle \\ = - \gamma \| g (x ^ {k}) \| ^ {2} + \frac {1}{2 \gamma} \| x ^ {k} - x ^ {k + 1} \| ^ {2} + \frac {1}{2 \gamma} \| x - x ^ {k + 1} \| ^ {2} - \frac {1}{2 \gamma} \| x - x ^ {k} \| ^ {2} \\ = - \frac {\gamma}{2} \left\| g (x ^ {k}) \right\| ^ {2} + \frac {1}{2 \gamma} \left\| x - x ^ {k + 1} \right\| ^ {2} - \frac {1}{2 \gamma} \left\| x - x ^ {k} \right\| ^ {2} \\ \end{array}
$$
$$
\mathbb {E} \left[ \left| \left| g (x ^ {k}) \right| \right| ^ {2} \right\rvert   \mathcal {G} ^ {k} \biggr ] = \mathbb {E} \left[ \left| \left| g (x ^ {k}) - \mathbb {E} \left[ g (x ^ {k}) \right| \mathcal {G} ^ {k} \right] \right| ^ {2} \biggr |   \mathcal {G} ^ {k} \right] + \left| \left| \mathbb {E} \left[ g (x ^ {k}) \right| \mathcal {G} ^ {k} \right] \right| ^ {2} \leq \sigma^ {2} + M ^ {2}.
$$
$$
\begin{array}{l} f \left(x ^ {k}\right) \leq f (x) + \mathbb {E} \left[ \left\langle g \left(x ^ {k}\right), x ^ {k} - x \right\rangle \mid \mathcal {G} ^ {k} \right] \\ = f (x) + \frac {\gamma}{2} \mathbb {E} \left[ \left\| g (x ^ {k}) \right\| ^ {2} \Big | \mathcal {G} ^ {k} \right] + \frac {1}{2 \gamma} \left\| x - x ^ {k} \right\| ^ {2} - \frac {1}{2 \gamma} \mathbb {E} \left[ \left\| x - x ^ {k + 1} \right\| ^ {2} \Big | \mathcal {G} ^ {k} \right] \\ \leq f (x) + \frac {\gamma}{2} \left(M ^ {2} + \sigma^ {2}\right) + \frac {1}{2 \gamma} \left\| x - x ^ {k} \right\| ^ {2} - \frac {1}{2 \gamma} \mathbb {E} \left[ \left\| x - x ^ {k + 1} \right\| ^ {2} \right\rvert \mathcal {G} ^ {k} ]. \\ \end{array}
$$
By taking the full expectation and summing the last inequality for  $t$  from 0 to  $K - 1$ , we obtain
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {k = 0} ^ {K - 1} f (x ^ {k}) \right] \leq K f (x) + \frac {K \gamma}{2} \left(M ^ {2} + \sigma^ {2}\right) + \frac {1}{2 \gamma} \left\| x - x ^ {0} \right\| ^ {2} - \frac {1}{2 \gamma} \mathbb {E} \left[ \| x - x ^ {K} \| ^ {2} \right] \\ \leq K f (x) + \frac {K \gamma}{2} \left(M ^ {2} + \sigma^ {2}\right) + \frac {1}{2 \gamma} \left\| x - x ^ {0} \right\| ^ {2}. \\ \end{array}
$$
Let divide the last inequality by  $K$ , take  $x = x^{*}$ , and use the convexity:
$$
\mathbb {E} \left[ f \left(\frac {1}{K} \sum_ {k = 0} ^ {K - 1} x ^ {k}\right) \right] - f (x ^ {*}) \leq \frac {\gamma}{2} \left(M ^ {2} + \sigma^ {2}\right) + \frac {1}{2 \gamma K} \left\| x ^ {*} - x ^ {0} \right\| ^ {2}.
$$
The choices of  $\gamma$  and  $K$  ensure that (47) holds.
H.4 Proof of Theorem B.9
Proof. The proof is the same as in Theorem 7.5. It is only required to estimate the time that is required to collect a batch of size  $S$ . Method 4 returns a solution after
$$
K \times 2 t ^ {\prime} (j ^ {*}) = \frac {4 M ^ {2} \left\| x ^ {*} - x ^ {0} \right\| ^ {2}}{\varepsilon^ {2}} \min _ {j \in [ n ]} \left[ \left(\sum_ {i = 1} ^ {j} \frac {1}{\tau_ {i}}\right) ^ {- 1} (S + j) \right]
$$
H.5 Proof of Theorem B.10
Proof. One can see that Accelerated Method 4 is just the accelerated stochastic gradient method with the batch size  $S$ . It means that we can use the classical SGD result (Proposition 4.4 in Lan (2020)). For a stepsize
$$
\gamma = \min  \left\{\frac {1}{4 L}, \left[ \frac {3 R ^ {2} S}{4 \sigma^ {2} (K + 1) (K + 2) ^ {2}} \right] ^ {1 / 2} \right\},
$$
$$
\mathbb {E} \left[ f (x ^ {K}) \right] - f (x ^ {*}) \leq \frac {4 L R ^ {2}}{K ^ {2}} + \frac {4 \sqrt {\sigma^ {2} R ^ {2}}}{\sqrt {S K}}.
$$
$$
\mathbb {E} \left[ f (x ^ {K}) \right] - f (x ^ {*}) \leq \varepsilon
$$
$$
K \geq \frac {8 \sqrt {L} R}{\sqrt {\varepsilon}} \geq 8 \max \left\{\frac {\sqrt {L} R}{\sqrt {\varepsilon}}, \frac {\sigma^ {2} R ^ {2}}{\varepsilon^ {2} S} \right\},
$$
where we use the choice of  $S$ .
H.6 Proof of Theorem B.11
Theorem B.11. Let us consider Theorem B.10. We assume that  $i^{th}$  worker returns a stochastic gradient every  $\tau_{i}$  seconds for all  $i\in [n]$ . Without loss of generality, we assume that  $0 < \tau_{1}\leq \dots \leq \tau_{n}$ . Then after
$$
3 2 \min _ {m \in [ n ]} \left[ \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {\sqrt {L} R}{\sqrt {\varepsilon}} + \frac {\sigma^ {2} R ^ {2}}{m \varepsilon^ {2}}\right) \right]
$$
Proof. The proof is the same as in Theorem 7.5. It is only required to estimate the time that is required to collect a batch of size  $S$ . Accelerated Method 4 returns a solution after
$$
K \times 2 t ^ {\prime} (j ^ {*}) = \frac {1 6 \sqrt {L} R}{\sqrt {\varepsilon}} \min  _ {j \in [ n ]} \left[ \left(\sum_ {i = 1} ^ {j} \frac {1}{\tau_ {i}}\right) ^ {- 1} (S + j) \right]
$$
I Construction of Algorithm for Rennala SGD
In this section, we provide the formal construction of the algorithm from Definition 4.1 for Rennala SGD. We consider the fixed computation model, where  $i^{\mathrm{th}}$  worker requires  $\tau_{i}$  seconds to calculate stochastic gradients. We now define the corresponding sequence  $\{A^k\}_{k = 0}^{\infty}$ . Let us fix a starting point  $x^0\in \mathbb{R}^d$ , a stepsize  $\gamma \geq 0$ , and a batch size  $S\in \mathbb{N}$ .
First, let us consider the sequence  $\{\widehat{t_k}\}_{k = 1}^{\infty}$  from Definition D.4 that represents the times when the workers would be ready to provide stochastic gradients. Additionally, let us define  $\widehat{t_0} \coloneqq 0$  and a sequence of the workers' indices  $\{\widehat{i}_k\}_{k = 1}^{\infty}$  that are corresponding to the times  $\{\widehat{t_k}\}_{k = 1}^{\infty}$ . We can define
$$
A ^ {k} \left(g ^ {1}, \dots , g ^ {k}\right) = \left\{ \begin{array}{l l} \left(\widehat {t} _ {\left(\lfloor k / 2 \rfloor\right)}, \widehat {i} _ {\left(\lfloor k / 2 \rfloor + 1\right)}, x ^ {0} - \frac {\gamma}{S} \sum_ {j = 1} ^ {2 S \times \lfloor k / (2 S) \rfloor} g ^ {j}\right), & k (\bmod 2) = 0, \\ \left(\widehat {t} _ {\left(\lfloor k / 2 \rfloor + 1\right)}, \widehat {i} _ {\left(\lfloor k / 2 \rfloor + 1\right)}, 0\right), & k (\bmod 2) = 1, \end{array} \right. \tag {48}
$$
for all  $k \geq 1$ , and  $A^0 = (\widehat{t}_0, \widehat{i}_1, x^0)$ . For the fixed computation model, one can use this algorithm in Protocol 3 with the oracle (7) to get an equivalent procedure to Method 4.
J Experiments
In this section, we compare Rennala SGD with Asynchronous SGD and Minibatch SGD on quadratic optimization tasks with stochastic gradients. The experiments were implemented in Python 3.7.9. The distributed environment was emulated on machines with Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz.
J.1 Setup
We consider the homogeneous optimization problem (1) with the convex quadratic function
$$
f (x) = \frac {1}{2} x ^ {\top} \mathbf {A} x - b ^ {\top} x \quad \forall x \in \mathbb {R} ^ {d}.
$$
We take  $d = 1000$
$$
\mathbf {A} = \frac {1}{4} \left( \begin{array}{c c c c} 2 & - 1 & & 0 \\ - 1 & \ddots & \ddots & \\ & \ddots & \ddots & - 1 \\ 0 & & - 1 & 2 \end{array} \right) \in \mathbb {R} ^ {d \times d} \quad \text {a n d} \quad b = \frac {1}{4} \left[ \begin{array}{c} - 1 \\ 0 \\ \vdots \\ 0 \end{array} \right] \in \mathbb {R} ^ {d}.
$$
Assume that all  $n$  workers has access to the following unbiased stochastic gradients:
$$
[ \widehat {\nabla} f (x; \xi) ] _ {j} := \nabla_ {j} f (x) \left(1 + \mathbb {1} [ j > \operatorname {p r o g} (x) ] \left(\frac {\xi}{p} - 1\right)\right) \quad \forall x \in \mathbb {R} ^ {d},
$$
where  $\xi \sim \mathcal{D}_i = \mathrm{Bernoulli}(p)$  for all  $i\in [n]$ , where  $p\in (0,1]$ . We denote  $[x]_j$  as the  $j^{\mathrm{th}}$  index of a vector  $x\in \mathbb{R}^d$ . In our experiments, we take  $p = 0.01$  and the starting point  $x^0 = [\sqrt{d},0,\dots ,0]^\top$ . We emulate our setup by considering that the  $i^{\mathrm{th}}$  worker requires  $\sqrt{i}$  seconds to calculate a stochastic gradient. In all methods, we fine-tune step sizes from a set  $\{2^i\mid i\in [-20,20]\}$ . In Rennala SGD, we fine-tune the batch size  $S\in \{1,5,10,20,40,80,100,200,500,1000\}$ .
J.2 Results
In Figures 1, 2, and 3, we present experiments with different number of workers  $n \in \{100, 1000, 10000\}$ . When the number of workers  $n = 100$ , Rennala SGD with Asynchronous SGD converge to the minimum at almost the same rate. However, when we start increasing the number of workers  $n$ , one can see that Asynchronous SGD starts converging slower. This is an expected behavior since the maximum theoretical step size in Asynchronous SGD decreases as the number of workers  $n$  increases (Koloskova et al., 2022; Mishchenko et al., 2022).
K Experiment with Small-Scale Machine Learning Task
We also consider the methods in a more practical scenario. We solve a logistic regression problem with the MNIST dataset (LeCun et al., 2010). We take  $n = 1000$  workers that hold the same subset of MNIST of the size 3000. Each worker samples stochastic gradients of size 4. In Figures 4 and 5, we provide convergence rates and a histogram of the time delays from the experiment. As in (Mishchenko et al., 2018), we can observe that asynchronous methods converge faster than Minibatch SGD. Unlike Section J where Rennala SGD converges faster than Asynchronous SGD, these methods have almost the same performance in this particular experiment.
<sup>9</sup>We implemented Asynchronous SGD with delay-adaptive stepsizes from (Koloskova et al., 2022)
Figure 1: # of workers  $n = 100$ .
Figure 2: # of workers  $n = 1000$ .
Figure 3: # of workers  $n = 10000$ .
Figure 4: Logistic regression experiment
Figure 5: Histogram of time delays
L Time Complexity of Asynchronous SGD
The works (Mishchenko et al., 2022; Koloskova et al., 2022) state that Asynchronous SGD converges after
$$
\mathrm {O} \left(\frac {n L \Delta}{\varepsilon} + \frac {\sigma^ {2} L \Delta}{\varepsilon^ {2}}\right)
$$
iterations. This result directly does not reveal the time complexity of Asynchronous SGD. Let us provide the time complexity for the case when the workers require exactly  $\tau_{i}$  seconds to compute stochastic gradients. Let us fix a time  $t\geq 0$ . Then the workers will calculate at most
$$
\sum_ {i = 1} ^ {n} \left\lfloor \frac {t}{\tau_ {i}} \right\rfloor
$$
stochastic gradients. To get  $\varepsilon$ -stationary point, we have to find the minimal  $t$  such that
$$
c \times \left(\frac {n L \Delta}{\varepsilon} + \frac {\sigma^ {2} L \Delta}{\varepsilon^ {2}}\right) \leq \sum_ {i = 1} ^ {n} \left\lfloor \frac {t}{\tau_ {i}} \right\rfloor , \tag {49}
$$
where the quantity  $c$  is a numerical constant, since the number of iterations can not be larger than the number of calculated gradients. Note that
$$
\sum_ {i = 1} ^ {n} \left\lfloor \frac {t}{\tau_ {i}} \right\rfloor \leq \sum_ {i = 1} ^ {n} \frac {t}{\tau_ {i}}
$$
The time  $t'$  that satisfies
$$
c \left(\frac {n L \Delta}{\varepsilon} + \frac {\sigma^ {2} L \Delta}{\varepsilon^ {2}}\right) = \sum_ {i = 1} ^ {n} \frac {t ^ {\prime}}{\tau_ {i}}
$$
$$
t ^ {\prime} = c \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {L \Delta}{\varepsilon} + \frac {\sigma^ {2} L \Delta}{n \varepsilon^ {2}}\right). \tag {50}
$$
Therefore, the minimal time  $t$  that satisfies (49) is greater or equal to (50).
M Analysis of Fixed-Computation Model Using Graph Oracle Models
In this section, we analyze the fixed computation model, where  $i^{\mathrm{th}}$  worker requires  $\tau_{i}$  seconds to calculate stochastic gradients. Without loss of generality, let us assume that  $0 < \tau_{1} \leq \dots \leq \tau_{n}$ . We use the graph oracle framework by Woodworth et al. (2018). Let us fix some  $t > \tau_{n}$ . Then, in the fixed computation model, the depth of the graph oracle  $D$  is at least  $\Omega\left(\frac{t}{\tau_1}\right)$ . This is the number of gradients that the first node can calculate after  $t$  seconds. The size of the graph  $N$  equals  $\Theta\left(\sum_{i=1}^{n} \frac{t}{\tau_i}\right)$  since all workers calculate in parallel. Applying these estimates to Theorem 1 of (Woodworth et al., 2018), one can see that, for convex,  $L$ -smooth,  $M$ -Lipschitz problems with unbiased and  $\sigma^2$ -variance-bounded stochastic gradients on the ball  $B_{2}(0,R)$ , the lower bound is at most equals
$$
\mathrm {O} \left(\min  \left\{\frac {M R}{\sqrt {t / \tau_ {1}}}, \frac {L R ^ {2}}{(t / \tau_ {1}) ^ {2}} \right\} + \frac {\sigma R}{\sqrt {\sum_ {i = 1} ^ {n} \frac {t}{\tau_ {i}}}}\right).
$$
From this estimate, we can conclude that in order to get an  $\varepsilon$ -solution, it is required
$$
t = \mathrm {O} \left(\tau_ {1} \min  \left\{\frac {M ^ {2} R ^ {2}}{\varepsilon^ {2}}, \frac {\sqrt {L} R}{\sqrt {\varepsilon}} \right\} + \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{\tau_ {i}}\right) ^ {- 1} \frac {\sigma^ {2} R ^ {2}}{n \varepsilon^ {2}}\right)
$$
M.1 Example when the lower bound from (Woodworth et al., 2018) is not tight
Let us provide an example when the lower bound from Theorem B.4 is strictly higher. Let us take  $\tau_{i} = \sqrt{i}$  for all  $i\in [n]$ . Then, it is sufficient to compare
$$
\min  _ {m \in [ n ]} \left[ \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\min  \left\{\frac {\sqrt {L} R}{\sqrt {\varepsilon}}, \frac {M ^ {2} R ^ {2}}{\varepsilon^ {2}} \right\} + \frac {\sigma^ {2} R ^ {2}}{m \varepsilon^ {2}}\right) \right]
$$
$$
\tau_ {1} \min \left\{\frac {\sqrt {L} R}{\sqrt {\varepsilon}}, \frac {M ^ {2} R ^ {2}}{\varepsilon^ {2}} \right\} + \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{\tau_ {i}}\right) ^ {- 1} \frac {\sigma^ {2} R ^ {2}}{n \varepsilon^ {2}}.
$$
Let us divide both formulas by the term with minimum and obtain
$$
\min  _ {m \in [ n ]} \left[ \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(1 + \frac {a}{m}\right) \right]
$$
$$
\tau_ {1} + \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{\tau_ {i}}\right) ^ {- 1} \frac {a}{n}
$$
for some  $a \geq 0$ . Since  $\sum_{i=1}^{m} \frac{1}{\tau_i} = \sum_{i=1}^{m} \frac{1}{\sqrt{i}} = \Theta(\sqrt{m})$ , then we get
$$
\min _ {m \in [ n ]} \left[ \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(1 + \frac {a}{m}\right) \right] = \Theta \left(\min _ {m \in [ n ]} \left[ \sqrt {m} \left(1 + \frac {a}{m}\right) \right]\right) = \Theta \left(\sqrt {a}\right)
$$
$$
\tau_ {1} + \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{\tau_ {i}}\right) ^ {- 1} \frac {a}{n} = \Theta \left(1 + \frac {a}{\sqrt {n}}\right),
$$
if  $a \leq n$ . For instance, let us additionally assume that  $a = \sqrt{n}$ , then the first term equals  $\Theta\left(n^{1/4}\right)$ , while the second equals  $\Theta(1)$ . Therefore, the lower bound from Theorem B.4 is tighter.