SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Parallelization is a popular strategy for improving the performance of iterative algorithms. Optimization methods are no exception: design of efficient parallel optimization methods and tight analysis of their theoretical properties are important research endeavors. While the minimax complexities are well known for sequential optimization methods, the theory of parallel optimization methods is less explored. In this paper, we propose a new protocol that generalizes the classical oracle framework approach. Using this protocol, we establish minimax complexities for parallel optimization methods that have access to an unbiased stochastic gradient oracle with bounded variance. We consider a fixed computation model characterized by each worker requiring a fixed but worker-dependent time to calculate stochastic gradient. We prove lower bounds and develop optimal algorithms that attain them. Our results have surprising consequences for the literature of asynchronous optimization methods.
We consider the nonconvex optimization problem
$$
\min  _ {x \in Q} \left\{f (x) := \mathbb {E} _ {\xi \sim \mathcal {D}} [ f (x; \xi) ] \right\}, \tag {1}
$$
where  $f: \mathbb{R}^d \times \mathbb{S}_\xi \to \mathbb{R}$ ,  $Q \subseteq \mathbb{R}^d$ , and  $\xi$  is a random variable with some distribution  $\mathcal{D}$  on  $\mathbb{S}_\xi$ . In machine learning,  $\mathbb{S}_\xi$  could be the space of all possible data,  $\mathcal{D}$  is the distribution of the training dataset, and  $f(\cdot, \xi)$  is the loss of a data sample  $\xi$ . In this paper we address the following natural setup:
(i)  $n$  workers are available to work in parallel,
(ii) the  $i^{\mathrm{th}}$  worker requires  $\tau_{i}$  seconds to calculate a stochastic gradient of  $f$ .
The function  $f$  is  $L$ -smooth and lower-bounded (see Assumptions 7.1-7.2), and stochastic gradients are unbiased and  $\sigma^2$ -variance-bounded (see Assumption 7.3).
1.1 Classical theory
In the nonconvex setting, gradient descent (GD) is an optimal method with respect to the number of gradient  $(\nabla f)$  calls (Lan, 2020; Nesterov, 2018; Carmon et al., 2020) for finding an approximately stationary point of  $f$ . Obviously, a key issue with GD is that it requires access to the exact gradients
1Or any other unit of time.
$\nabla f$  of the function  $f$ . However, in many practical applications, it can be infeasible to calculate the gradient of  $\mathbb{E}[f(\cdot ;\xi)]$  analytically. Moreover, even if this is possible, e.g., if the distribution  $\mathcal{D}$  is described by  $m$  possible samples, so that  $\mathbb{E}_{\xi \sim \mathcal{D}}[f(\cdot ;\xi)] = (1 / m)\sum_{i = 1}^{m}f(\cdot ;\xi_i)$ ,  $m$  can be huge (Krizhevsky et al., 2017), and gradient evaluation can be arbitrarily expensive.
Stochastic Gradient Descent. Due to the above-mentioned problem, machine learning literature is preoccupied with the study of algorithms that can work with stochastic gradients instead (Lan, 2020; Ghadimi and Lan, 2013). For all  $x \in \mathbb{R}^d$ , we assume that the  $n$  workers have access to independent, unbiased, and  $\sigma^2$ -variance-bounded stochastic gradients  $\widehat{\nabla} f(x,\xi)$  (see Assumption 7.3), where  $\xi$  is a random sample from  $\mathcal{D}$ . Under such assumptions, with one worker, stochastic gradient descent (SGD), i.e., the method  $x^{k+1} = x^k - \gamma \widehat{\nabla} f(x^k;\xi^k)$ , where  $\xi^k$  are i.i.d. random samples from  $\mathcal{D}$ , is known to be optimal with respect to the number of stochastic gradient calls (Ghadimi and Lan, 2013; Arjevani et al., 2022). SGD guarantees convergence to an  $\varepsilon$ -stationary point in expectation after O  $(L\Delta/\varepsilon + \sigma^2 L\Delta/\varepsilon^2)$  stochastic gradient evaluations, where  $\Delta := f(x^0) - f^*$  and  $x^0 \in \mathbb{R}^d$  is a starting point.
1.2 Parallel optimization methods
Using the bounds from Section 1.1, one can easily estimate the performance of these algorithms in real systems. For instance, if it takes  $\tau_{1}$  seconds to calculate a stochastic gradient with one worker, then SGD guarantees to return a solution after
$$
\mathrm {O} \left(\tau_ {1} \left(\frac {L \Delta}{\varepsilon} + \frac {\sigma^ {2} L \Delta}{\varepsilon^ {2}}\right)\right)
$$
seconds. If instead of a single worker we can access  $n$  workers that can calculate stochastic gradients in parallel, we can consider the following classical parallel methods:
Minibatch SGD. The minibatch SGD method (Minibatch SGD), i.e., the iterative process
$$
x ^ {k + 1} = x ^ {k} - \gamma \frac {1}{n} \sum_ {i = 1} ^ {n} \widehat {\nabla} f (x ^ {k}; \xi_ {i} ^ {k}),
$$
where  $\gamma$  is a stepsize,  $\xi_i^k$  are i.i.d. samples from  $\mathcal{D}$ , and the gradients  $\widehat{\nabla} f(x^k;\xi_i^k)$  are calculated in parallel. This method converges after O  $(L\Delta /\varepsilon +\sigma^{2}L\Delta /n\varepsilon^{2})$  iterations (Cotter et al., 2011; Goyal et al., 2017; Gower et al., 2019) and after
$$
O \left(\tau_ {\max } \left(\frac {L \Delta}{\varepsilon} + \frac {\sigma^ {2} L \Delta}{n \varepsilon^ {2}}\right)\right) \tag {2}
$$
seconds, where  $\tau_{\mathrm{max}}\coloneqq \max_{i\in [n]}\tau_i$  is the processing time associated with the slowest machine2.
Although the time complexity (2) of Minibatch SGD improves with the number of workers  $n$ , in general, this does not guarantee better performance due to the delay  $\tau_{\mathrm{max}}$ . In real systems, parallel computations can be very chaotic, e.g., they can be slow due to inconsistent network communications, or GPU computation delays (Dutta et al., 2018; Chen et al., 2016).
Asynchronous SGD. We now consider the asynchronous SGD method (Asynchronous SGD) (Recht et al., 2011; Nguyen et al., 2018; Arjevani et al., 2020; Feyzmahdavian et al., 2016) described by
1. Receive  $\widehat{\nabla} f(x^{k - \delta_k};\xi^{k - \delta_k})$  from a worker,
2.  $x^{k + 1} = x^k -\gamma^k\widehat{\nabla} f(x^{\delta_k};\xi^{\delta_k})$
3. Ask the worker to calculate  $\widehat{\nabla} f(x^{k + 1};\xi^{k + 1})$
where  $\xi^k$  are i.i.d. samples from  $\mathcal{D}$ , and  $\delta_k$  are gradient iteration delays. This is an asynchronous method: the workers work independently, finish calculations of stochastic gradients with potentially large and chaotic delays  $\delta_k$ , and the result of their computation is applied as soon as it is ready, without having to wait for other workers. Asynchronous SGD was also considered in the heterogeneous setting (see details in Section A.2).
2Further, we assume that the last  $n^{\mathrm{th}}$  worker is the slowest one:  $\tau_{n} = \tau_{\mathrm{max}}$
Table 1: Homogeneous and Heterogeneous Case. The required time to get an  $\varepsilon$ -stationary point  $(\mathbb{E}[\| \nabla f(\widehat{x})\|^2] \leq \varepsilon)$  in the nonconvex setting, where  $i^{\text{th}}$  worker requires  $\tau_i$  seconds to calculate a stochastic gradient. We assume that  $0 < \tau_1 \leq \dots \leq \tau_n$ .
Homogeneous Case	Heterogeneous Case
Method	Time Complexity	Method	Time Complexity
Minibatch SGD	τn(LΔ/ε + σ2LΔ/nε2)	Minibatch SGD	τn(LΔ/ε + σ2LΔ/nε2)
Asynchronous SGD(Cohen et al., 2021)(Koloskova et al., 2022)(Mishchenko et al., 2022)	(1/n∑i=1n1/τi)-1(LΔ/ε + σ2LΔ/nε2)	Maleria SGD(Theorem A.4)	τnLΔ/ε + (1/n∑i=1nτi) σ2LΔ/nε2
Lower Bound(Theorem A.2)	τnLΔ/ε + (1/n∑i=1nτi) σ2LΔ/nε2
Rennala SGD(Theorem 7.5)	minm∈[n] [ (1/m∑i=1m1/τi)-1(LΔ/ε + σ2LΔ/mε2)]
Lower Bound(Theorem 6.4)	minm∈[n] [ (1/m∑i=1m1/τi)-1(LΔ/ε + σ2LΔ/mε2)]
Cohen et al. (2021); Mishchenko et al. (2022); Koloskova et al. (2022) provide the current state-of-the-art analysis of Asynchronous SGD. In particular, they prove that Asynchronous SGD converges after  $\mathrm{O}\left( {{nL\Delta }/\varepsilon  + {\sigma }^{2}{L\Delta }/{\varepsilon }^{2}}\right)$  iterations. To show the superiority of Asynchronous SGD,Mishchenko et al. (2022) consider the following fixed computation model: the  ${i}^{\text{th }}$  worker requires  ${\tau }_{i}$  seconds to calculate stochastic gradients. In this setting,Asynchronous SGD converges after
$$
\mathrm {O} \left(\left(\frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {L \Delta}{\varepsilon} + \frac {\sigma^ {2} L \Delta}{n \varepsilon^ {2}}\right)\right) \tag {3}
$$
seconds (we reprove this fact in Section L). Thus, Asynchronous SGD can be  $\left(1 / n\right)\sum_{i = 1}^{n}\tau_{\max} / \tau_{i}$  times faster than Minibatch SGD.
Besides Asynchronous SGD, many other strategies utilize parallelization (Dutta et al., 2018; Woodworth et al., 2020; Wu et al., 2022), and can potentially improve over minibatch SGD.
2 Problem and Contribution
In this paper, we seek to find the optimal time complexity in the setting from Section 1: our goal is to provide a lower bound and a method that attains it. Our main contributions are:
(i) Lower bound. In Sections 4 & 5, we define new Protocols 2 & 3, and a new complexity measure  $\mathfrak{m}_{\mathrm{time}}$  (see (6)), which we believe are more appropriate for the analysis of parallel optimization algorithms. In Section 6, we prove the time complexity lower bound for (nonconvex) functions and algorithms that work with parallel asynchronous oracles.
(i) Optimal method. In Section 7, we develop a minimax optimal method—Rennala<sup>3</sup> SGD—that attains this lower bound.
In addition, we investigate several other related questions. As an independent result, in Section 8 we prove that all methods which synchronize workers in each iteration (e.g., Minibatch SGD) have provably worse time complexity than asynchronous methods (e.g., Rennala SGD (see Method 4), Asynchronous SGD). In Section A, we extend our theory to the heterogeneous case, in which the workers have access to different distributions (datasets), and provide a lower bound and a new method that attains it. In Section B, we provide the optimal time complexities in the convex setting.
3 https://eldenring.wiki.fextralife.com/Rennala+Queen+of+the+Full+Moon: Rennala, Queen of the Full Moon is a Legend Boss in Elden Ring. Though not a demigod, Rennala is one of the shardbearers who resides in the Academy of Raya Lucaria. Rennala is a powerful sorceress, head of the Carian Royal family, and erstwhile leader of the Academy.
3 Classical Oracle Protocol
Let us recall the classical approach to obtaining lower bounds for optimization algorithms. We need to define a function class  $\mathcal{F}$ , an oracle class  $\mathcal{O}$ , and an algorithm class  $\mathcal{A}$ . We then analyze the complexity of an algorithm  $A = \{A^k\}_{k=0}^\infty \in \mathcal{A}$ , using the following protocol:
Protocol 1 Classical Oracle Protocol
1: Input: function  $f \in \mathcal{F}$ , oracle and distribution  $(O, \mathcal{D}) \in \mathcal{O}(f)$ , algorithm  $A \in \mathcal{A}$
2: for  $k = 0, \dots, \infty$  do
3:  $x^{k} = A^{k}(g^{1},\ldots ,g^{k})$
4:  $g^{k + 1} = O(x^k,\xi^{k + 1}),\quad \xi^{k + 1}\sim \mathcal{D}$
5: end for
$$
\triangleright x ^ {0} = A ^ {0} \text {f o r} k = 0.
$$
More formally, in first-order stochastic optimization, the oracle class  $\mathcal{O}$  returns a random mapping  $O: \mathbb{R}^d \times \mathbb{S}_\xi \to \mathbb{R}^d$  based on a function  $f \in \mathcal{F}$  and a distribution  $\mathcal{D}$ ; we use the notation  $(O, \mathcal{D}) \in \mathcal{O}(f)$ . An algorithm  $A = \{A^k\}_{k=0}^\infty \in \mathcal{A}$  is a sequence such that
$$
A ^ {k}: \underbrace {\mathbb {R} ^ {d} \times \cdots \times \mathbb {R} ^ {d}} _ {k \text {t i m e s}} \rightarrow \mathbb {R} ^ {d} \forall k \geq 1, \text {a n d} A ^ {0} \in \mathbb {R} ^ {d}. \tag {4}
$$
Typically, an oracle  $O$  returns an unbiased stochastic gradient that satisfies Assumption 7.3:  $O(x,\xi) = \widehat{\nabla} f(x;\xi)$  for all  $x\in \mathbb{R}^d$  and  $\xi \in \mathbb{S}_{\xi}$ . Let us fix an oracle class  $\mathcal{O}$ . Then, in the nonconvex first-order stochastic setting, we analyze the complexity measure
$$
\mathfrak {m} _ {\text {o r a c l e}} (\mathcal {A}, \mathcal {F}) := \inf  _ {A \in \mathcal {A}} \sup  _ {f \in \mathcal {F}} \sup  _ {(O, \mathcal {D}) \in \mathcal {O} (f)} \inf  \left\{k \in \mathbb {N} \mid \mathbb {E} \left[ \| \nabla f (x ^ {k}) \| ^ {2} \right] \leq \varepsilon \right\}, \tag {5}
$$
where the sequence  $\{x^k\}_k$  is generated by Protocol 1. Virtually all previous works are concerned with lower bounds of optimization problems using Protocol 1 and the complexity measure (5) (Nemirovskij and Yudin, 1983; Carmon et al., 2020; Arjevani et al., 2022; Nesterov, 2018).
4 Time Oracle Protocol
In the previous sections, we discuss the classical approach to estimating the complexities of algorithms. Briefly, these approaches seek to quantify the worst-case number of iterations or oracle calls that are required to find a solution (see (5)), which is very natural for sequential methods. However, and this is a key observation of our work, this approach is not convenient if we want to analyze parallel methods. We now propose an alternative protocol that can be more helpful in this situation:
Protocol 2 Time Oracle Protocol
1: Input: functions  $f \in \mathcal{F}$ , oracle and distribution  $(O, \mathcal{D}) \in \mathcal{O}(f)$ , algorithm  $A \in \mathcal{A}$
2:  $s^0 = 0$
3: for  $k = 0, \dots, \infty$  do
4:  $(t^{k + 1},x^k) = A^k (g^1,\ldots ,g^k),$
5:  $(s^{k + 1},g^{k + 1}) = O(t^{k + 1},x^k,s^k,\xi^{k + 1}),\quad \xi^{k + 1}\sim \mathcal{D}$
6: end for
$$
\triangleright t ^ {k + 1} \geq t ^ {k}
$$
Protocol 2 is almost identical to Protocol 1 except for one key detail: Protocol 2 requires the algorithms to return a sequence  $\{t^{k + 1}\}_{k = 1}^{\infty}$  such that  $t^{k + 1}\geq t^k\geq 0$  for all  $k\geq 0$ . We assume that  $t^0 = 0$ . We also assume that the oracles take to the input the states  $s^k$  and output them (the role of these states will be made clear later). In this case, we provide the following definition of an algorithm.
Definition 4.1. An algorithm  $A = \{A^k\}_{k=0}^{\infty}$  is a sequence such that
$$
A ^ {k}: \underbrace {\mathbb {R} ^ {d} \times \cdots \times \mathbb {R} ^ {d}} _ {k \text {t i m e s}} \to \mathbb {R} _ {\geq 0} \times \mathbb {R} ^ {d} \quad \forall k \geq 1, A ^ {0} \in \mathbb {R} _ {\geq 0} \times \mathbb {R} ^ {d},
$$
and, for all  $k \geq 1$  and  $g^1, \ldots, g^k \in \mathbb{R}^d$ ,  $t^{k+1} \geq t^k$ , where  $t^{k+1}$  and  $t^k$  are defined as  $(t^{k+1}, \cdot) = A^k(g^1, \ldots, g^k)$  and  $(t^k, \cdot) = A^{k-1}(g^1, \ldots, g^{k-1})$ .
Let us explain the role of the sequence  $\{t^k\}_k$ . In Protocol 1, an algorithm outputs a point  $x^k$  and then asks the oracle: Provide me a gradient at the point  $x^k$ . In contrast, in Protocol 2 an algorithm outputs a point  $x^k$  and a time  $t^{k + 1}$ , and asks the oracle: Start calculating a gradient at the point  $x^k$  at a time  $t^{k + 1}$ . We have a constraint that  $t^{k + 1} \geq t^k$  for all  $k \geq 0$ , which means that the algorithm is not allowed to travel into the past.
Using Protocol 2, we propose to use another complexity measure instead of (5):
$$
\mathfrak {m} _ {\text {t i m e}} (\mathcal {A}, \mathcal {F}) := \inf  _ {A \in \mathcal {A}} \sup  _ {f \in \mathcal {F} (O, \mathcal {D}) \in \mathcal {O} (f)} \sup  _ {t \geq 0} \left\{t \geq 0 \mid \mathbb {E} \left[ \inf  _ {k \in S _ {t}} \| \nabla f (x ^ {k}) \| ^ {2} \right] \leq \varepsilon \right\}, \tag {6}
$$
$$
S _ {t} := \left\{k \in \mathbb {N} _ {0} \big | t ^ {k} \leq t \right\},
$$
where the sequences  $t^k$  and  $x^k$  are generated by Protocol 2. In (5), we seek to find the worst-case number of iterations  $k$  required to get  $\mathbb{E}\left[\left\|\nabla f(x^k)\right\|^2\right] \leq \varepsilon$  for any  $A \in \mathcal{A}$ . In (6), we seek to find the worst-case case time  $t$  required to find an  $\varepsilon$ -stationary point for any  $A \in \mathcal{A}$ .
We now provide an example, considering an oracle that calculates a stochastic gradient in  $\tau$  seconds. Let us define the appropriate oracle for this problem:
$$
O_{\tau}^{\widehat{\nabla}^{f}}: \underbrace{\mathbb{R}_{\geq 0}}_{\text{time}}\times \underbrace{\mathbb{R}^{d}}_{\text{point}}\times \underbrace{(\mathbb{R}_{\geq 0}\times\mathbb{R}^{d}\times\{0,1\})}_{\text{input state}}\times \mathbb{S}_{\xi}\to \underbrace{(\mathbb{R}_{\geq 0}\times\mathbb{R}^{d}\times\{0,1\})}_{\text{output state}}\times \mathbb{R}^{d}
$$
such that  $O_{\tau}^{\widehat{\nabla} f}(t,x,(s_t,s_x,s_q),\xi) = \left\{ \begin{array}{ccc}((t,x,1), & 0), & s_q = 0,\\ ((s_t,s_x,1), & 0), & s_q = 1\text{and} t <   s_t + \tau ,\\ ((0,0,0), & \widehat{\nabla} f(s_x;\xi)), & s_q = 1\text{and} t\geq s_t + \tau , \end{array} \right.$
and  $\widehat{\nabla} f$  is a mapping such that  $\widehat{\nabla} f: \mathbb{R}^d \times \mathbb{S}_\xi \to \mathbb{R}^d$ . Further, we additionally assume that  $\widehat{\nabla} f$  is an unbiased  $\sigma^2$ -variance-bounded stochastic gradient (see Assumption 7.3).
Note that the oracle  $O_{\tau}^{\overline{\nabla} f}$  emulates the behavior of a real worker. Indeed, the oracle can return three different outputs. If  $s_q = 0$ , it means that the oracle has been idle, then "starts the calculation" of the gradient at the point  $x$ , and changes the state  $s_q$  to 1. Also, using the state, it remembers the time moment  $t$  when the calculation began and the point  $x$ . Next, if  $s_q = 1$  and  $t < s_t + \tau$ , it means the oracle is still calculating the gradient, so if an algorithm sends time  $t$  such that  $t < s_t + \tau$ , then it receives the zero vector. Finally, if  $s_q = 1$ , as soon as an algorithm sends time  $t$  such that  $t \geq s_t + \tau$ , then the oracle will be ready to provide the gradient. Note that the oracle provides the gradient calculated at the point  $x$  that was requested when the oracle was idle. Thus, the time between the request of an algorithm to get the gradient and the time when the algorithm gets the gradient is at least  $\tau$  seconds.
In Protocol 2, we have a game between an algorithm  $A \in \mathcal{A}$  and an oracle class  $\mathcal{O}$ , where algorithms can decide the sequence of times  $t^k$ . Thus, an algorithm wants to find enough information from an oracle as soon as possible to obtain  $\varepsilon$ -stationary point.
Let us consider an example. For the oracle class  $\mathcal{O}$  that generates the oracle from (7), we can define the SGD method in the following way. We take any starting point  $x^0\in \mathbb{R}^d$ , a step size  $\gamma = \min \left\{1 / L,\varepsilon /2L\sigma^2\right\}$  (see Theorem D.8) and define  $A^{k}:\underbrace{(\mathbb{R}^{d}\times\cdots\times\mathbb{R}^{d})}_{k\text{ times}}\to \mathbb{R}_{\geq 0}\times \mathbb{R}^{d}$  such that
$$
A ^ {k} \left(g ^ {1}, \dots , g ^ {k}\right) = \left\{ \begin{array}{l l} \left(\tau \lfloor k / 2 \rfloor , x ^ {0} - \gamma \sum_ {j = 1} ^ {k} g ^ {k}\right), & k (\bmod 2) = 0, \\ \left(\tau (\lfloor k / 2 \rfloor + 1), 0\right), & k (\bmod 2) = 1, \end{array} \right. \tag {8}
$$
for all  $k \geq 1$ , and  $A^0 = (0, x^0)$ . Let us explain the behavior of the algorithm. In the first step of Protocol 2, when  $k = 0$ , the algorithm requests the gradient at the point  $x^0$  at the time  $t^1 = 0$  since
$A^0 = (0,x^0)$ . The oracle  $O$  changes the state from  $s_q^0 = 0$  to  $s_q^1 = 1$  and remembers the point  $x^0$  in the state  $s_x^1$ . In the second step of the protocol, when  $k = 1$ , the algorithm calls the oracle at the time  $\tau \left( \lfloor k/2 \rfloor + 1 \right) = \tau$ . In the oracle, the condition  $t^2 \geq s_t^1 + \tau \Leftrightarrow \tau \geq 0 + \tau$  is satisfied, and it returns the gradient at the point  $x^0$ . Note that this can only happen if an algorithm does the second call at a time that is greater or equal to  $\tau$ .
One can see that after  $\tau K$  seconds, the algorithm returns the point  $x^{2K} = x^0 - \gamma \sum_{j=0}^{K-1} \widehat{\nabla} f(x^{2j}; \xi^{2j+1})$ , where  $\xi^j \sim \mathcal{D}$  are i.i.d. random variables. The algorithm is equivalent to the SGD method that converges after  $K = \mathrm{O}\left(L\Delta/\varepsilon + \sigma^2 L\Delta/\varepsilon^2\right)$  steps for the function class  $\mathcal{F}_{\Delta,L}$  (see Definition 6.1) for  $x^0 = 0$ . Thus, the complexity  $\mathfrak{m}_{\mathrm{time}}(\{A\}, \mathcal{F}_{\Delta,L})$  equals  $\mathrm{O}\left(\tau \times \left(L\Delta/\varepsilon + \sigma^2 L\Delta/\varepsilon^2\right)\right)$ .
Actually, any algorithm that was designed for Protocol 1 can be used in Protocol 2 with the oracle (7). Assuming that we have mappings  $A^k: \mathbb{R}^d \times \dots \times \mathbb{R}^d \to \mathbb{R}^d$  for all  $k \geq 1$ , we can define mappings  $\widehat{A}^k: \mathbb{R}^d \times \dots \times \mathbb{R}^d \to \mathbb{R}_{\geq 0} \times \mathbb{R}^d$  via
$$
\widehat {A} ^ {k} (g ^ {1}, \ldots , g ^ {k}) = \left\{ \begin{array}{l l} \Big (\tau \lfloor k / 2 \rfloor , A ^ {\lfloor k / 2 \rfloor} (g ^ {2}, g ^ {4}, \ldots , g ^ {2 k}) \Big), & k (\mathrm {m o d} 2) = 0, \\ \Big (\tau \left(\lfloor k / 2 \rfloor + 1\right), 0 \Big), & k (\mathrm {m o d} 2) = 1. \end{array} \right.
$$
For  $k = 0$ , we define  $\widehat{A}^0 = (0, A^0)$ .
5 Time Multiple Oracles Protocol
The protocol framework from the previous section does not seem to be very powerful because one can easily find the time complexity (6) by knowing (5) and the amount of time that oracle needs to calculate a gradient. In fact, we provide Protocol 2 for simplicity only. We now consider a protocol that works with multiple oracles:
Protocol 3 Time Multiple Oracles Protocol
1: Input: function(s)  $f \in \mathcal{F}$ , oracles and distributions  $((O_1, \dots, O_n), (\mathcal{D}_1, \dots, \mathcal{D}_n)) \in \mathcal{O}(f)$ , algorithm  $A \in \mathcal{A}$
2:  $s_i^0 = 0$  for all  $i \in [n]$
3: for  $k = 0, \dots, \infty$  do
4:  $(t^{k + 1},i^{k + 1},x^k) = A^k (g^1,\ldots ,g^k),$
5:  $(s_{i^{k + 1}}^{k + 1},g^{k + 1}) = O_{i^{k + 1}}(t^{k + 1},x^k,s_{i^{k + 1}}^k,\xi^{k + 1}),\quad \xi^{k + 1}\sim \mathcal{D}_{i^{k + 1}}\quad \triangleright s_j^{k + 1} = s_j^k\quad \forall j\neq i^{k + 1}$
6: end for
Compared to Protocol 2, Protocol 3 works with multiple oracles, and algorithms return the indices  $i^{k+1}$  of the oracle they want to call. This minor add-on to the protocol enables the possibility of analyzing parallel optimization methods. Also, each oracle  $O_i$  can have its own distribution  $\mathcal{D}_i$ .
Let us consider an example with two oracles  $O_{1} = O_{\tau_{1}}^{\overline{\nabla} f}$  and  $O_{2} = O_{\tau_{2}}^{\overline{\nabla} f}$  from (7). One can see that a "wise" algorithm will first call the oracle  $O_{1}$  with the time  $t^{0} = 0$ , and then, in the second step, it will call the oracle  $O_{2}$  also with the time  $t^{1} = 0$ . Note that it is impossible to do the following steps: in the first step an algorithm calls the oracle  $O_{1}$  with the time  $t^{0} = 0$ , in the second step, the algorithm calls the oracle  $O_{1}$  with the time  $t^{1} = \tau_{1}$  and receives the gradient, in the third step, the algorithm calls the oracle  $O_{2}$  with the time  $t^{2} = 0$ . Indeed, this can't happen because  $t^{2} < t^{1}$ .
An example of a "non-wise" algorithm is an algorithm that, in the first step, calls the oracle  $O_{1}$  with the time  $t^{0} = 0$ . In the second step, the algorithm calls the oracle  $O_{1}$  with the time  $t^{1} = \tau_{1}$  and receives the gradient. In the third step, the algorithm calls the oracle  $O_{2}$  with the time  $t^{2} = \tau_{1}$ . It would mean that the "non-wise" algorithm did not use the oracle  $O_{2}$  for  $\tau_{1}$  seconds. Consequently, the "wise" algorithm can receive two gradients after  $\max \{\tau_{1},\tau_{2}\}$  seconds, while the "non-wise" algorithm can only receive two gradients after  $\tau_{1} + \tau_{2}$  seconds.
We believe that Protocol 3 and the complexity (6) is a better choice for analyzing the complexities of parallel methods than the classical Protocol 1. In the next section, we will use Protocol 3 to obtain lower bounds for parallel optimization methods.
6 Lower Bound for Parallel Optimization Methods