SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Considering Protocol 3, we define a special function class  $\mathcal{F}$ , oracle class  $\mathcal{O}$ , and algorithm class  $\mathcal{A}$ . We consider the same function class as Nesterov (2018); Arjevani et al. (2022); Carmon et al. (2020):
Definition 6.1 (Function Class  $\mathcal{F}_{\Delta,L}$ ). We assume that function  $f: \mathbb{R}^d \to \mathbb{R}$  is differentiable,  $L$ -smooth, i.e.,  $\| \nabla f(x) - \nabla f(y) \| \leq L \| x - y \| \quad \forall x, y \in \mathbb{R}^d$ , and  $\Delta$ -bounded, i.e.,  $f(0) - \inf_{x \in \mathbb{R}^d} f(x) \leq \Delta$ . A set of all functions with such properties we denote by  $\mathcal{F}_{\Delta,L}$ .
In this paper, we analyze the class of "zero-respecting" algorithms, defined next.
Definition 6.2 (Algorithm Class  $\mathcal{A}_{\mathrm{zr}}$ ). Let us consider Protocol 3. We say that an algorithm  $A$  from Definition 4.1 is a zero-respecting algorithm, if  $\operatorname{supp}\left(x^{k}\right) \subseteq \bigcup_{j=1}^{k} \operatorname{supp}\left(g^{j}\right)$  for all  $k \in \mathbb{N}_0$ , where  $\operatorname{supp}(x) := \{i \in [d] \mid x_{i} \neq 0\}$ . A set of all algorithms with this property we define as  $\mathcal{A}_{\mathrm{zr}}$ .
A zero-respecting algorithm does not try to change the coordinates for which no information was received from oracles. This family is considered by Arjevani et al. (2022); Carmon et al. (2020), and includes SGD, Minibatch and Asynchronous SGD, and Adam (Kingma and Ba, 2014).
Definition 6.3 (Oracle Class  $\mathcal{O}_{\tau_1,\dots ,\tau_n}^{\sigma^2}$ ). Let us consider an oracle class such that, for any  $f\in \mathcal{F}_{\Delta ,L}$  it returns oracles  $O_{i} = O_{\tau_{i}}^{\widehat{\nabla} f}$  and distributions  $\mathcal{D}_i$  for all  $i\in [n]$ , where  $\widehat{\nabla} f$  is an unbiased  $\sigma^2$ -variance-bounded mapping (see Assumption 7.3). The oracles  $O_{\tau_i}^{\widehat{\nabla} f}$  are defined in (7). We define such oracle class as  $\mathcal{O}_{\tau_1,\dots ,\tau_n}^{\sigma^2}$ . Without loss of generality, we assume that  $0 < \tau_{1}\leq \dots \leq \tau_{n}$ .
We take  $\mathcal{O}_{\tau_1,\dots ,\tau_n}^{\sigma^2}$  because it emulates the behavior of workers in real systems, where workers can have different processing times (delays)  $\tau_{i}$ . Note that  $\mathcal{O}_{\tau_1,\dots ,\tau_n}^{\sigma^2}$  has the freedom to choose a mapping  $\widehat{\nabla} f$ . We only assume that the mapping is unbiased and  $\sigma^2$ -variance-bounded. We are now ready to present our first result; a lower bound:
Theorem 6.4. Let us consider the oracle class  $\mathcal{O}_{\tau_1,\dots,\tau_n}^{\sigma^2}$  for some  $\sigma^2 > 0$  and  $0 < \tau_1 \leq \dots \leq \tau_n$ . We fix any  $L, \Delta > 0$  and  $0 < \varepsilon \leq c' L \Delta$ . In view Protocol 3, for any algorithm  $A \in \mathcal{A}_{\mathrm{zr}}$ , there exists a function  $f \in \mathcal{F}_{\Delta,L}$  and oracles and distributions  $((O_1, \ldots, O_n), (\mathcal{D}_1, \ldots, \mathcal{D}_n)) \in \mathcal{O}_{\tau_1,\dots,\tau_n}^{\sigma^2}(f)$  such that  $\mathbb{E}\left[\inf_{k \in S_t} \left\| \nabla f(x^k) \right\|^2\right] > \varepsilon$ , where  $S_t := \{k \in \mathbb{N}_0 | t^k \leq t\}$ , and
$$
t = c \times \min _ {m \in [ n ]} \left[ \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {L \Delta}{\varepsilon} + \frac {\sigma^ {2} L \Delta}{m \varepsilon^ {2}}\right) \right].
$$
The quantities  $c'$  and  $c$  are universal constants.
Theorem 6.4 states that
$$
\mathfrak {m} _ {\text {t i m e}} \left(\mathcal {A} _ {\mathrm {z r}}, \mathcal {F} _ {\Delta , L}\right) = \Omega \left(\min  _ {m \in [ n ]} \left[ \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {L \Delta}{\varepsilon} + \frac {\sigma^ {2} L \Delta}{m \varepsilon^ {2}}\right) \right]\right). \tag {9}
$$
The interpretation behind this complexity will be discussed later in Section 7.3. No algorithms known to us attain (9). For instance, Asynchronous SGD has the time complexity (3). Let us assume that  $\sigma^2 /\varepsilon \leq p$  and  $p\in [n]$ . Then (lower bound from (9)) = O  $\left(\left(\frac{1}{p}\sum_{i = 1}^{p}\frac{1}{\tau_i}\right)^{-1}\left(\frac{L\Delta}{\varepsilon}\right)\right)$ . In this case, the lower bound in (9) will be at least  $\left(\frac{1}{n}\sum_{i = 1}^{n}\frac{1}{\tau_i}\right)^{-1} / \left(\frac{1}{p}\sum_{i = 1}^{p}\frac{1}{\tau_i}\right)^{-1}$  times smaller. It means that either the obtained lower bound is not tight, or Asynchronous SGD is a suboptimal method. In the following section we provide a method that attains the lower bound. The obtained lower bound is valid even if an algorithm has the freedom to interrupt oracles. See details in Section F.
For convex problems, Woodworth et al. (2018) proposed the graph oracle, which generalizes the classical gradient oracle (Nemirovskij and Yudin, 1983; Nesterov, 2018), and provided lower bounds
for a rather general family of parallel methods. Arjevani et al. (2020) analyzed the delayed gradient descent method, which is Asynchronous SGD when all iteration delays  $\delta_{k} = \delta$  are a constant.
As far as we know, Woodworth et al. (2018) provide the most suitable and tightest prior framework for analyzing lower bound complexities for problem (1). However, as we shall see, our framework us more powerful. Moreover, they only consider the convex case. In Section M, we use the framework of Woodworth et al. (2018) and analyze the fixed computation model, where  $i^{\text{th}}$  worker requires  $\tau_{i}$  seconds to calculate stochastic gradients. In Section B, we consider the convex setting and show that the lower bound obtained by their framework is not tight and can be improved. While the graph oracle framework by Woodworth et al. (2018) is related to the classical oracle protocol (Section 3) and also calculates the number of oracle calls in order to get lower bounds, our approach directly estimates the required time. For more details, see Section B and the discussion in Section B.1.1.
Method 4 Rennala SGD
1: Input: starting point  $x^0$ , stepsize  $\gamma$ , batch size  $S$
2: Run Method 5 in all workers
3: for  $k = 0,1,\dots ,K - 1$  do
4: Init  $g^{k} = 0$  and  $s = 1$
5: while  $s \leq S$  do
6: Wait for the next worker
7: Receive gradient and iteration index  $(g, k')$
8: if  $k' = k$  then
9:  $g^{k} = g^{k} + \frac{1}{s} g; s = s + 1$
10: end if
11: Send  $(x^{k}, k)$  to the worker
12: end while
13:  $x^{k + 1} = x^k -\gamma g^k$
14: end for
Method 5 Worker's Infinite Loop
1: Init  $g = 0$  and  $k' = -1$
2: while True do
3: Send  $(g, k')$  to the server
4: Receive  $(x^{k}, k)$  from the server
5:  $k' = k$
6:  $g = \widehat{\nabla} f(x^k;\xi),\quad \xi \sim \mathcal{D}$
7: end while
7 Minimax Optimal Method
We now propose and analyze a new method: Rennala SGD (see Method 4). Methods with a similar structure were proposed previously (e.g., (Dutta et al., 2018)), but we are not aware of any method with precisely the same parameters and structure. For us, in this paper, the theoretical bounds are more important than the method itself.
Let us briefly describe the structure of the method. At the start, Method 4 asks all workers to run Method 5. Method 5 is a standard routine: the workers receive points  $x^{k}$  from the server, calculate stochastic gradients, and send them back to the server. Besides that, the workers receive and send the iteration counter  $k$  of the received points  $x^{k}$ . At the server's side, in each iteration  $k$ , Method 4 calculates  $g^{k}$  and performs the standard gradient-type step  $x^{k+1} = x^{k} - \gamma g^{k}$ . The calculation of  $g^{k}$  is done in a loop. The server waits for the workers to receive a stochastic gradient and an iteration index. The most important part of the method is that the server ignores a stochastic gradient if its iteration index is not equal to the current iteration index. In fact, this means that  $g^{k} = (1 / s) \sum_{i=1}^{S} \widehat{\nabla} f(x^{k}; \xi_{i})$ , where  $\xi_{i}$  are i.i.d. samples. In other words, the server ignores all stochastic gradients that were calculated at the points  $x^{0}, \dots, x^{k-1}$ .
It may seem that Method 4 does not fully use the information due to ignoring some stochastic gradients. That contradicts the philosophy of Asynchronous SGD, which tries to use all stochastic gradients calculated in the previous points. Nevertheless, we show that Rennala SGD has better time complexity than Asynchronous SGD, and this complexity matches the lower bound from Theorem 6.4. The fact that Rennala SGD ignores the previous iterates is motivated by the proof of the lower bound in Section 6. In the proof, any algorithm, on the constructed "worst case" function, does not progress to a stationary point if it calculates a stochastic gradient at a non-relevant point. This suggested to us to construct a method that would focus all workers on the last iterate.
7.1 Assumptions
Let us consider the following assumptions.
Assumption 7.1.  $f$  is differentiable &  $L$ -smooth, i.e.,  $\| \nabla f(x) - \nabla f(y)\| \leq L\| x - y\|$ ,  $\forall x,y\in \mathbb{R}^d$ .
Assumption 7.2. There exist  $f^{*} \in \mathbb{R}$  such that  $f(x) \geq f^{*}$  for all  $x \in \mathbb{R}^d$ .
Assumption 7.3. For all  $x \in \mathbb{R}^d$ , stochastic gradients  $\widehat{\nabla} f(x;\xi)$  are unbiased and  $\sigma^2$ -variance-bounded, i.e.,  $\mathbb{E}_{\xi}\left[\widehat{\nabla} f(x;\xi)\right] = \nabla f(x)$  and  $\mathbb{E}_{\xi}\left[\left\|\widehat{\nabla}f(x;\xi)-\nabla f(x)\right\|^2\right] \leq \sigma^2$ , where  $\sigma^2 \geq 0$ .
7.2 Analysis of Rennala SGD
Theorem 7.4. Assume that Assumptions 7.1, 7.2 and 7.3 hold. Let us take the batch size  $S = \max \left\{\left\lceil \sigma^2 / \varepsilon \right\rceil, 1\right\}$ , and  $\gamma = \min \left\{\frac{1}{L}, \frac{\varepsilon S}{2L\sigma^2}\right\} = \Theta(1/L)$  in Method 4. Then after
$$
K \geq \frac {2 4 \Delta L}{\varepsilon}
$$
iterations, the method guarantees that  $\frac{1}{K}\sum_{k = 0}^{K - 1}\mathbb{E}\left[\left\| \nabla f(x^k)\right\| ^2\right]\leq \varepsilon$
In the following theorem, we provide the time complexity of Method 4.
Theorem 7.5. Consider Theorem 7.4. We assume that  $i^{th}$  worker returns a stochastic gradient every  $\tau_{i}$  seconds for all  $i \in [n]$ . Without loss of generality, we assume that  $0 < \tau_{1} \leq \dots \leq \tau_{n}$ . Then after
$$
9 6 \times \min  _ {m \in [ n ]} \left[ \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {L \Delta}{\varepsilon} + \frac {\sigma^ {2} L \Delta}{m \varepsilon^ {2}}\right) \right] \tag {10}
$$
seconds, Method 4 guarantees to find an  $\varepsilon$ -stationary point.
This result with Theorem 6.4 state that
$$
\mathfrak {m} _ {\text {t i m e}} \left(\mathcal {A} _ {\mathrm {z r}}, \mathcal {F} _ {\Delta , L}\right) = \Theta \left(\min  _ {m \in [ n ]} \left[ \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{\tau_ {i}}\right) ^ {- 1} \left(\frac {L \Delta}{\varepsilon} + \frac {\sigma^ {2} L \Delta}{m \varepsilon^ {2}}\right) \right]\right) \tag {11}
$$
for Protocol 3 and and the oracle class  $\mathcal{O}_{\tau_1,\dots ,\tau_n}^{\sigma^2}$  from Definition 6.3.
7.3 Discussion
Theorem 7.5 and Theorem 6.4 state that Method 4 is minimax optimal under the assumption that the delays of the workers are fixed and equal to  $\tau_{i}$ . Note that this assumption is required only in Theorem 7.5, and Theorem 7.4 holds without it.
In the same setup, the previous works (Cohen et al., 2021; Mishchenko et al., 2022; Koloskova et al., 2022) obtained the weaker time complexity (3). We do not rule out that it might be possible for the analysis, the parameters or the structure of Asynchronous SGD to be improved and obtain the optimal time complexity (10). We leave this to future work. However, instead, we developed Method 4 that has not only the optimal time complexity, but also a very simple structure and analysis (see Section D.4.1). Our claims are supported by experiments in Section J.
The reader can see that we provide the complexity in a nonconstructive way, as the minimization over the parameter  $m \in [n]$ . Note that Method 4 automatically finds the optimal  $m$  in (7.5), and it does not require the knowledge of the delays  $\tau_{i}$  to do so! Let us explain the intuition behind the complexity (10). Let  $m^{*}$  be the optimal parameter of (10) with the smallest index. In Section D.4.3, we show that all workers with the delays  $\tau_{i}$  for all  $i > m^{*}$  can be simply ignored since their delays are too large, and their inclusion would only harm the convergence time of the method. So, the method automatically ignores them! However, in Asynchronous SGD, these harmful workers can contribute to the optimization process, which can be the reason for the suboptimality of Asynchronous SGD.
In general, there are two important regimes:  $\sigma^2 /\varepsilon \ll n$  ("low noise/large # of workers") and  $\sigma^2 /\varepsilon \gg n$  ("high noise/small # of workers"). Intuitively, in the "high noise/small # of workers" regime, (11) is minimized when  $m$  is close to  $n$ . However, in the "low noise/large # of workers", the optimal  $m$  can be much smaller than  $n$ .
8 Synchronized Start of Workers
In the previous sections, we obtain the time complexities for the case when the workers asynchronously compute stochastic gradients. It is important that the complexities are obtained assuming that the workers can start their calculations asynchronously. However, in practice, it is common to train machine learning models with multiple workers/GPUs, so that all workers are synchronized after each stochastic gradient calculation (Goyal et al., 2017; Sergeev and Balso, 2018). The simplest example of such a strategy is Minibatch SGD (see Section 1.2). We want to find an answer to the question: what is the best time complexity we can get if we assume that the workers start simultaneously? In Section G, we formalize this setting, and show that the time complexity equals to
$$
\mathfrak {m} _ {\text {t i m e}} \left(\mathcal {A} _ {\mathrm {z r}}, \mathcal {F} _ {\Delta , L}\right) = \Theta \left(\min  _ {m \in [ n ]} \left[ \tau_ {m} \left(\frac {L \Delta}{\varepsilon} + \frac {\sigma^ {2} L \Delta}{m \varepsilon^ {2}}\right) \right]\right) \tag {12}
$$
for Protocol 2 and the oracle class  $\mathcal{O}_{\tau_1,\dots,\tau_n}^{\sigma^2,\mathrm{sync}}$  from Definition G.1. Comparing (11) and (12), one can see that methods that start the calculations of workers simultaneously are provably worse than methods that allow workers to start the calculations asynchronously.
9 Future Work
In this work, we consider the setup where the times  $\tau_{i}$  are fixed. In future work, one can consider natural, important, and more general scenarios where they can be random, follow some distribution, and/or depend on the random variables  $\xi$  from Assumption 7.3 (be correlated with stochastic gradients).
Arjevani, Y., Carmon, Y., Duchi, J. C., Foster, D. J., Srebro, N., and Woodworth, B. (2022). Lower bounds for non-convex stochastic optimization. Mathematical Programming, pages 1-50.
Arjevani, Y., Shamir, O., and Srebro, N. (2020). A tight convergence analysis for stochastic gradient descent with delayed updates. In Algorithmic Learning Theory, pages 111-132. PMLR.
Aytekin, A., Feyzmahdavian, H. R., and Johansson, M. (2016). Analysis and implementation of an asynchronous optimization algorithm for the parameter server. arXiv preprint arXiv:1610.05507.
Carmon, Y., Duchi, J. C., Hinder, O., and Sidford, A. (2020). Lower bounds for finding stationary points i. Mathematical Programming, 184(1):71-120.
Chen, J., Pan, X., Monga, R., Bengio, S., and Jozefowicz, R. (2016). Revisiting distributed synchronous sgd. arXiv preprint arXiv:1604.00981.
Cohen, A., Daniely, A., Drori, Y., Koren, T., and Schain, M. (2021). Asynchronous stochastic optimization robust to arbitrary delays. Advances in Neural Information Processing Systems, 34:9024-9035.
Cotter, A., Shamir, O., Srebro, N., and Sridharan, K. (2011). Better mini-batch algorithms via accelerated gradient methods. Advances in Neural Information Processing Systems, 24.
Dutta, S., Joshi, G., Ghosh, S., Dube, P., and Nagpurkar, P. (2018). Slow and stale gradients can win the race: Error-routine trade-offs in distributed SGD. In International Conference on Artificial Intelligence and Statistics, pages 803-812. PMLR.
Woodworth, B. E., Wang, J., Smith, A., McMahan, B., and Srebro, N. (2018). Graph oracle models, lower bounds, and gaps for parallel stochastic optimization. Advances in Neural Information Processing Systems, 31.
Wu, X., Magnusson, S., Feyzmahdavian, H. R., and Johansson, M. (2022). Delay-adaptive step-sizes for asynchronous learning. arXiv preprint arXiv:2202.08550.
1.1 Classical theory 1
1.2 Parallel optimization methods 2
2 Problem and Contribution 3
3 Classical Oracle Protocol 4
4 Time Oracle Protocol 4
5 Time Multiple Oracles Protocol 6
6 Lower Bound for Parallel Optimization Methods 7
6.1 Related work 7
7 Minimax Optimal Method 8
7.1Assumptions 8
7.2 Analysis of Rennala SGD 9
7.3 Discussion 9
8 Synchronized Start of Workers 10
9 Future Work 10
A Heterogeneous Regime 16
A.1 Lower bound 16
A.2 Related work and discussion 17
A.3 Minimax optimal method 17
A.4 Discussion 18
B Convex Case 18
B.1 Lower Bound 18
B.1.1 Discussion 19
B.2 Minimax optimal method 20
B.2.1 Assumptions 20
B.2.2 Analysis of Rennala SGD and Accelerated Rennala SGD in convex case 20
C Table of Notations 21
D Proofs for Homogeneous Regime 21
D.1 The "worst case" function 21
D.2 Proof of Theorem 6.4 21
D.3 Auxiliary lemmas 23
D.3.1 Proof of Lemma D.2 23
D.3.2 Proof of Lemma D.6 27
D.3.3 Lemma D.7 30
D.4 Proof of Theorems 7.4 and 7.5 32
D.4.1 Proof of Theorems 7.4 32
D.4.2 The classical SGD theorem 32
D.4.3 Proof of Theorems 7.5 34
E Proofs for Heterogeneous Regime 35
E.1 Proof of Theorem A.2 35
E.2 Proof of Lemma E.1 38
E.3 Proof of Theorem A.3 40
E.4 Proof of Theorem A.4 41
F Interrupt Oracle Calculations 42
G Time Complexity with Synchronized Start 43
G.1 Minimax optimal method 43
G.2 Proof of Theorem G.2 44
G.2.1 Proof of Lemma G.5 46
G.2.2 Lemma G.9 49
G.2.3 Proof of Lemma G.8 49
G.3 Proof of Theorem G.3 52
G.4 Proof of Theorem G.4 52
H Proofs for Convex Case 53
H.1 The "worst case" function in convex case 53
H.2 Proof of Theorem B.4 53
H.3 Proof of Theorem B.8 55
H.3.1 The classical SGD theorem in convex optimization 55
H.4 Proof of Theorem B.9 56
H.5 Proof of Theorem B.10 57
H.6 Proof of Theorem B.11 57
I Construction of Algorithm for Rennala SGD 58
J Experiments 59
J.1 Setup 59
J.2 Results 59
K Experiment with Small-Scale Machine Learning Task 59
L Time Complexity of Asynchronous SGD 61
M Analysis of Fixed-Computation Model Using Graph Oracle Models 62
M.1 Example when the lower bound from (Woodworth et al., 2018) is not tight 62
Table 2: Heterogeneous Case. The required time to get an  $\varepsilon$ -stationary point in the nonconvex setting, where  $i^{\text{th}}$  worker requires  $\tau_{i}$  seconds to calculate a stochastic gradient. We assume that  $0 < \tau_{1} \leq \dots \leq \tau_{n}$ .
Heterogeneous Case
Method	Time Complexity
Minibatch SGD	τn(LΔ/ε + σ2LΔ/nε2)
Maleria SGD (Theorem A.4)	τnLΔ/ε + (1/n ∑i=1n τi) σ2LΔ/nε2
Lower Bound (Theorem A.2)	τnLΔ/ε + (1/n ∑i=1n τi) σ2LΔ/nε2
A Heterogeneous Regime
Up to this point, we discussed the regime when all workers calculate i.i.d. stochastic gradients. In distributed optimization and federated learning (Konečný et al., 2016), it can be possible that the workers hold different datasets. Let us consider the following optimization problem:
$$
\min  _ {x \in \mathbb {R} ^ {d}} \left\{f (x) := \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\xi_ {i} \sim \mathcal {D} _ {i}} \left[ f _ {i} \left(x; \xi_ {i}\right) \right] \right\}, \tag {13}
$$
where  $f_{i}:\mathbb{R}^{d}\times \mathbb{S}_{\xi_{i}}\to \mathbb{R}^{d}$  and  $\xi_{i}$  are random variables with some distributions  $\mathcal{D}_i$  on  $\mathbb{S}_{\xi_i}$ . Problem (13) generalizes problem (1). Here we have the same goals as in the previous sections. We want to obtain the minimax complexities for the case when the workers contain different datasets.
A.1 Lower bound
For the heterogeneous case, we modify Definition 6.3:
Definition A.1 (Oracle Class  $\mathcal{O}_{\tau_1,\dots ,\tau_n}^{\sigma^2,\text{heterog}}$
Let us consider an oracle class such that, for any  $f = \frac{1}{n}\sum_{i=1}^{n}f_i \in \mathcal{F}_{\Delta,L}$ , it returns oracles  $O_i = O_{\tau_i}^{\overline{\widehat{\nu}}f_i}$  and distributions  $\mathcal{D}_i$  for all  $i \in [n]$ , where  $\widehat{\nabla}f_i$  is an unbiased  $\sigma^2$ -variance-bounded mapping for all  $i \in [n]$  (see Assumption 7.3). The oracles  $O_{\tau_i}^{\overline{\widehat{\nu}}f_i}$  are defined in (7). We define such oracle class as  $\mathcal{O}_{\tau_1,\dots,\tau_n}^{\sigma^2,heterog}$ . Without loss of generality, we assume that  $0 < \tau_1 \leq \dots \leq \tau_n$ .
Theorem A.2. Let us consider the oracle class  $\mathcal{O}_{\tau_1,\dots,\tau_n}^{\sigma^2,heterog}$  for some  $\sigma^2 > 0$  and  $0 < \tau_1 \leq \dots \leq \tau_n$ . We fix any  $L, \Delta > 0$  and  $0 < \varepsilon \leq c' L \Delta$ . In the view Protocol 3, for any algorithm  $A \in \mathcal{A}_{\mathrm{zi}}$ , there exists a function  $f = \frac{1}{n} \sum_{i=1}^{n} f_i \in \mathcal{F}_{\Delta,L}$  and oracles and distributions  $(O_1, \ldots, O_n), (\mathcal{D}_1, \ldots, \mathcal{D}_n)) \in \mathcal{O}_{\tau_1,\ldots,\tau_n}^{\sigma^2,heterog}(f_1, \ldots, f_n)$  such that  $\mathbb{E}\left[\inf_{k \in S_t} \| \nabla f(x^k) \|^2\right] > \varepsilon$ , where  $S_t := \{k \in \mathbb{N}_0 | t^k \leq t\}$ , and
$$
t = c \times \left(\tau_ {n} \frac {L \Delta}{\varepsilon} + \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \tau_ {i}\right) \frac {\sigma^ {2} L \Delta}{n \varepsilon^ {2}}\right). \tag {14}
$$
The quantity  $c'$  and  $c$  are universal constants.
Theorem A.2 states that
$$
\mathfrak {m} _ {\text {t i m e}} \left(\mathcal {A} _ {\mathrm {z r}}, \mathcal {F} _ {\Delta , L}\right) = \Omega \left(\tau_ {n} \frac {L \Delta}{\varepsilon} + \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \tau_ {i}\right) \frac {\sigma^ {2} L \Delta}{n \varepsilon^ {2}}\right).
$$
One can see that the lower bound for the heterogeneous case is larger than (9). In Section A.3, we provide a method that attains the lower bound.
A.2 Related work and discussion
The optimization problem (13) is well-investigated by many papers, including (Aytekin et al., 2016; Mishchenko et al., 2018; Nguyen et al., 2022; Wu et al., 2022; Koloskova et al., 2022; Mishchenko et al., 2022). There were attempts to analyze Asynchronous SGD in the heterogeneous regime. For instance, Mishchenko et al. (2022) proved the convergence to a neighborhood of a solution only. In general, it is quite challenging to get good rates for Asynchronous SGD without additional assumptions about the similarity of the functions  $f_{i}$  (Koloskova et al., 2022; Mishchenko et al., 2022).
In the non-stochastic case, when  $\sigma^2 = 0$ , Wu et al. (2022) analyzed the PIAG method in the non-stochastic heterogeneous regime and showed convergence. Although the performance of PIAG can be good in practice, in the worst case PIAG requires  $\mathrm{O}\left(\tau_n\widehat{L}\Delta /\varepsilon\right)$  seconds to converge, where  $\tau_{n}$  is the time delay of the slowest worker,  $\widehat{L} := \sqrt{\sum_{i = 1}^{n}L_i^2}$ , and  $L_{i}$  is a Lipschitz constant of  $\nabla f_{i}$ . Note that the synchronous Minibatch SGD (see Section 1.2) method has the complexity  $\mathrm{O}\left(\tau_nL\Delta /\varepsilon\right)$ , which is always better.
Our lower bound in Theorem A.2 does not leave hope of breaking the dependence on the worst straggler in the heterogeneous case. In the stochastic case, the lower bound is slightly more optimistic in the regimes when the statistical term (the second term in (14)) is large. If the stragglers do not have too large delays, then their contributions to the arithmetic mean can be small. Note that in Theorem 6.4 in the homogeneous case, we have the harmonic mean of the delays instead.
A.3 Minimax optimal method