Title: Freya PAGE: First Optimal Time Complexity for Large-Scale Nonconvex Finite-Sum Optimization with Heterogeneous Asynchronous Computations
Venue: NeurIPS 2024
Accepted: yes
Keywords: Nonconvex Optimization

Asynchronous Methods

Distributed Optimization

Lower Bounds

Variance Reduction
Abstract: In practical distributed systems, workers are typically not homogeneous, and due to differences in hardware configurations and network conditions, can have highly varying processing times. We consider smooth nonconvex finite-sum (empirical risk minimization) problems in this setup and introduce a new parallel method, Freya PAGE, designed to handle arbitrarily heterogeneous and asynchronous computations. By being robust to "stragglers" and adaptively ignoring slow computations, Freya PAGE offers significantly improved time complexity guarantees compared to all previous methods, including Asynchronous SGD, Rennala SGD, SPIDER, and PAGE, while requiring weaker assumptions. The algorithm relies on novel generic stochastic gradient collection strategies with theoretical guarantees that can be of interest on their own, and may be used in the design of future optimization methods. Furthermore, we establish a lower bound for smooth nonconvex finite-sum problems in the asynchronous setup, providing a fundamental time complexity limit. This lower bound is tight and demonstrates the optimality of Freya PAGE in the large-scale regime, i.e., when $\sqrt{m} \geq n,$ where $n$ is \# of workers, and $m$ is \# of data samples.

Freya PAGE: First Optimal Time Complexity for Large-Scale Nonconvex Finite-Sum Optimization with Heterogeneous Asynchronous Computations

Abstract

In practical distributed systems, workers are typically not homogeneous, and due to differences in hardware configurations and network conditions, can have highly varying processing times. We consider smooth nonconvex finite-sum (empirical risk minimization) problems in this setup and introduce a new parallel method, Freya PAGE , designed to handle arbitrarily heterogeneous and asynchronous computations. By being robust to 'stragglers' and adaptively ignoring slow computations, Freya PAGE offers significantly improved time complexity guarantees compared to all previous methods, including Asynchronous SGD , Rennala SGD , SPIDER , and PAGE , while requiring weaker assumptions. The algorithm relies on novel generic stochastic gradient collection strategies with theoretical guarantees that can be of interest on their own, and may be used in the design of future optimization methods. Furthermore, we establish a lower bound for smooth nonconvex finite-sum problems in the asynchronous setup, providing a fundamental time complexity limit. This lower bound is tight and demonstrates the optimality of Freya PAGE in the large-scale regime, i.e., when √ m ≥ n, where n is # of workers, and m is # of data samples.

1 Introduction

In real-world distributed systems used for large-scale machine learning tasks, it is common to encounter device heterogeneity and variations in processing times among different computational units. These can stem from GPU computation delays, disparities in hardware configurations, network conditions, and other factors, resulting in different computational capabilities and speeds across devices [Chen et al., 2016, Tyurin and Richtárik, 2023]. As a result, some clients may execute computations faster, while others experience delays or even fail to participate in the training altogether.

Due to the above reasons, we aim to address the challenges posed by device heterogeneity in the context of solving finite-sum nonconvex optimization problems of the form

where f i : R d → R can be viewed as the loss of a machine learning model x on the i th example in a training dataset with m samples. Our goal is to find an ε -stationary point, i.e., a (possibly random) point ˆ x such that E [ ∥∇ f (ˆ x ) ∥ 2 ] ≤ ε . We focus on the homogeneous distributed setup:

there are n workers / clients / devices able to work in parallel,

each worker has access to stochastic gradients ∇ f j , j ∈ [ m ] ,

worker i calculates ∇ f j ( · ) in less or equal to τ i ∈ [0 , ∞ ] seconds for all i ∈ [ n ] , j ∈ [ m ] .

Without loss of generality, we assume that τ 1 ≤ . . . ≤ τ n . One can think of τ i ∈ [0 , ∞ ] as an upper bound on the computation time rather than a fixed deterministic time. Looking ahead, iteration complexity can be established even if τ i = ∞ for all i ∈ [ n ] (Theorem 4). We also provide results where the bounds { τ k i } are dynamic and change with every iteration k (Section 4.4). For simplicity of presentation, however, we assume that τ k i = τ i for i ∈ [ n ] , k ≥ 0 , unless explicitly stated otherwise.

1.1 Assumptions

We adopt two weak assumptions, which are standard for the problem (1) [Fang et al., 2018].

Assumption 1. The function f is L -smooth and lower-bounded by f ∈ R

-∗ .

We also consider Assumption 3. Note that this assumption does not restrict the class of considered functions { f i } . Indeed, if Assumption 2 holds with L + , then Assumption 3 holds with some L ± ≤ L + . If one only wants to rely on Assumptions 1 and 2, it is sufficient to take L ± = L + . However, Assumption 3 enables us to derive sharper rates, since L ± can be small or even 0 , even if L -and L + are large [Szlendak et al., 2021, Tyurin et al., 2023, Kovalev et al., 2022].

Assumption 3 (Hessian variance [Szlendak et al., 2021]) . There exists L ± ≥ 0 such that

1.2 Gradient oracle complexities

Iterative algorithms are traditionally evaluated based on their gradient complexity . Let us present a brief overview of existing theory. The classical result of Gradient Descent ( GD ) says that in the smooth nonconvex regime, the number of oracle calls needed to solve problem (1) is O ( mε -1 ) because GD converges in O ( ε -1 ) iterations, and calculates the full gradient ∇ f = 1 / m ∑ m i =1 ∇ f i in each iteration. This was improved to O ( m + m 2 / 3 ε -1 ) by several variance-reduced methods, including SVRG and SCSG [Allen-Zhu and Hazan, 2016, Reddi et al., 2016, Lei et al., 2017, Horváth and Richtárik, 2019]. Since then, various other algorithms, such as SNVRG , SARAH , SPIDER , SpiderBoost , PAGE and their variants, have been developed [Fang et al., 2018, Wang et al., 2019, Nguyen et al., 2017, Li et al., 2021, Zhou et al., 2020, Horváth et al., 2022]. These methods achieve a gradient complexity of O ( m + √ mε -1 ) , matching the lower bounds [Fang et al., 2018, Li et al., 2021].

That said, in practical scenarios, what often truly matters is the time complexity rather than the gradient complexity [Tyurin and Richtárik, 2023]. Although the latter metric serves as a natural benchmark for sequential methods, it seems ill-suited in the context of parallel methods.

1.3 Some previous time complexities

Let us consider some examples to provide intuition about time complexities for problem (1).

GD with 1 worker ( Hero GD ). In principle, each worker can solve the problem on their own. Hence, one approach would be to select the fastest client (assuming it is known) and delegate the task to them exclusively. A well-known result says that for L --smooth objective function f (Assumption 1), GD converges in δ 0 L -ε -1 iterations, where δ 0 := f ( x 0 ) -f ∗ , and x 0 is the starting point. Since at each iteration the method computes m gradients ∇ f i ( · ) , i ∈ [ m ] , the time required to find an ε -stationary point is δ 0 L -ε -1 × mτ 1 seconds.

GD with n workers and equal data allocation ( Soviet GD ). The above strategy leaves the remaining n -1 workers idle, and thus potentially useful computing resources are wasted. A common approach is to instead divide the data into n equal parts and assign one such part to each worker, so that each has to compute m / n gradients (assuming for simplicity that m is divisible by n ). Since at each iteration the strategy needs to wait for the slowest worker, the total time is δ 0 L -ε -1 × mτ n / n . Depending

Freya PAGE has universally better guarantees than all previous methods: the dependence on ε is O ( 1 / ε ) (unlike Rennala SGD and Asynchronous SGD ), the dependence on { τ i } is harmonic-like and robust to slow workers (robust to τ n →∞ ) (unlike Soviet PAGE and SYNTHESIS ), the assumptions are weak, and the time complexity of Freya PAGE is optimal when √ m ≥ n .

on the relationship between τ 1 and τ n / n , this could be more efficient or less efficient compared to Hero GD . This shows that the presence of stragglers can eliminate the potential speedup expected from parallelizing the training [Dutta et al., 2018].

SPIDER / PAGE with 1 worker or n workers and equal data allocation ( Hero PAGE and Soviet PAGE ). As mentioned in Section 1.2, SPIDER / PAGE can have better gradient complexity guarantees than GD . Using the result of Li et al. [2021], the equal data allocation strategy with n workers leads to the time complexity of

seconds. We refer to this method as Soviet PAGE . In practical regimes, when ε is small and L -≈ L ± , this complexity can be √ m better than that of GD . Running PAGE on the fastest worker (which we will call Hero PAGE ), we instead get the time complexity T Hero PAGE := Θ ( τ 1 m + τ 1 δ 0 / ε √ m ) .

Given these examples, the following question remains unanswered: what is the best possible time complexity in our setting? This paper aims to answer this question.

2 Contributions

We consider the finite-sum optimization problem (1) under weak assumptions and develop a new method, Freya PAGE . The method works with arbitrarily heterogeneous and asynchronous computations on the clients without making any assumptions about the bounds on the processing times τ i . We show that the time complexity of Freya PAGE is provably better than that of all previously proposed synchronous/asynchronous methods (Table 1). Moreover, we prove a lower bound that guarantees optimality of Freya PAGE in the large-scale regime ( √ m ≥ n ). The algorithm leverages new computation strategies, ComputeGradient (Alg. 2) and ComputeBatchDifference (Alg. 3), which are generic

Algorithm 1 Freya PAGE

and can be used in any other asynchronous method. These strategies enable the development of our new SGD method ( Freya SGD ); see Sections 6 and H. Experiments from Section A on synthetic optimization problems and practical logistic regression tasks support our theoretical results.

3 The Design of the New Algorithm

It is clear that to address the challenges arising in the setup under consideration and achieve optimality, a distributed algorithm has to adapt to and effectively utilize the heterogeneous nature of the underlying computational infrastructure. With this in mind, we now present a new algorithm, Freya PAGE , that can efficiently coordinate and synchronize computations across the n devices, accommodating arbitrarily varying processing speeds, while mitigating the impact of slow devices or processing delays on the overall performance of the system.

Freya PAGE is formalized in Algorithm 1. The update rule is just the regular PAGE [Li et al., 2021] update: at each iteration, with some (typically small) probability p , the algorithm computes the full gradient ∇ f ( x k +1 ) , and otherwise, it samples a minibatch S k of size S and reuses the gradient estimator g k from the previous iteration, updated by the cheaper-to-compute adjustment 1 S ∑ i ∈S k ( ∇ f i ( x k +1 ) -∇ f i ( x k ) ) .

Within Algorithm 1, at each iteration we call one of two subroutines: ComputeGradient (Alg. 2, performing the low-probability step), and ComputeBatchDifference (Alg. 3, performing the highprobability step). Let us focus on ComputeGradient , designed to collect the full gradient: it takes a point x as input and returns ∇ f ( x ) = 1 m ∑ m i =1 ∇ f i ( x ) . There exist many strategies for implementing this calculation, some of which were outlined in Section 1.3. The most naive one is to assign the task of calculating the whole gradient ∇ f to a single worker i , resulting in a worst-case running time of mτ i seconds for ComputeGradient . Another possible strategy is to distribute the functions { f i } evenly among the workers; in this case, calculating ∇ f takes τ n max { m / n , 1 } seconds in the worst case.

Clearly, we could do better if we knew { τ i } in advance . Indeed, let us allocate to each worker j a number of functions { f i } inversely proportional to τ j . This strategy is reasonable - the faster the worker, the more gradients it can compute. We can show that such a strategy finds ∇ f in

seconds in the worst case (see the proof of Theorem 2). This complexity is better than mτ 1 and τ n max { m / n , 1 } (Theorem 31). However, this approach comes with two major limitations: i) it requires knowledge of the upper bounds { τ i } , ii) even if we have access to { τ i } , the computation environment can be adversarial: theoretically and practically, it is possible that at the beginning the first worker is the fastest and the last worker is the slowest, but after some time, their performances

swap. Consequently, the first worker might end up being assigned the largest batch, despite now having the lowest performance. Thus, this strategy is not robust to time-varying speeds.

New gradient computation strategy. The key innovation of this work lies in the introduction of new computation strategies: Algorithms 2 and 3. We start by examining Algorithm 2. It first broadcasts the input x ∈ R d to all workers. Then, for each worker, it samples j uniformly from [ m ] and asks it to calculate ∇ f j ( x ) (with a non-zero probability, two workers can be assigned the same computation). Then, the algorithm enters the loop and waits for any worker to finish their calculations. Once this happens, it asks this worker to compute a stochastic gradient with a new index sampled uniformly from the set [ m ] \M of indices that have not yet been processed (again, it is possible to resample an index previously assigned to another worker). This continues until all indices i ∈ [ m ] have been processed and the full gradient 1 m ∑ m i =1 ∇ f i has been collected. Unlike the previous strategies, our Algorithm 2 does not use { τ i } , thus being robust and adaptive to the changing compute times . Furthermore, we can prove that its time complexity (almost) equals (3):

Theorem 1. The expected time needed by Algorithm 2 to calculate g = 1 m m ∑ i =1 ∇ f i is at most

seconds.

The result (4) (the proof of which can be found in Section C) is slightly worse than (3) due to the extra min { m,n } log (min { m,n } ) term. This term arises because a worker may be assigned a gradient ∇ f j ( x ) that was previously assigned to another worker (in Line 11 of Algorithm 2). Hence, with a small (but non-zero) probability, two workers can perform the same calculations. However, typically the number of samples m is much larger than the number of workers n. If we assume that m ≥ n log n, which is satisfied in many practical scenarios, then the time complexity (4) equals

and the term min { m,n } log (min { m,n } ) never dominates. Since this complexity is not worse than (3), our strategy behaves as if it knew { τ i } in advance! To simplify formulas and avoid the logarithmic term, we use the following assumption throughout the main part of this paper:

Assumption 4. m ≥ n log n, where m is the # of data samples and n is the # of workers.

We now proceed to discuss ComputeBatchDifference (Algorithm 3), designed to compute a minibatch of stochastic gradient differences. Both Algorithms 2 and 3 calculate sums. However the latter only

waits until there are S samples in the sum, where some indices in the batch may not be unique. On the other hand, Algorithm 2 must ensure the collection of a full batch of m unique stochastic gradients. As a result, Algorithm 3 offers better complexity results and, unlike ComputeGradient , does not suffer from suboptimal guarantees and logarithmic terms, as demonstrated in the theorem below.

Theorem 2. The time needed by Algorithm 3 to calculate g is at most

seconds.

Algorithm 1 uses uniform sampling with replacement , implemented in Algorithm 3. However, one can modify the two algorithms slightly to support virtually any unbiased sampling (see Section E).

Note on asynchronicity. It is clear that to eliminate the need of waiting for very slow machines, some level of asynchronicity has to be injected into an algorithm for it to be efficient. Asynchronous SGD [Recht et al., 2011, Nguyen et al., 2018, Koloskova et al., 2022, Mishchenko et al., 2022] takes this concept to the extreme by never synchronizing and continually overwriting the updates. Consequently, the algorithm's time complexity is suboptimal. Conversely, imposing limitations on asynchronicity leads to optimal methods, both in our context (in the large-scale regime) and in the scenario examined by Tyurin and Richtárik [2023]. Freya PAGE seamlessly combines synchrony and asynchrony, getting the best out of the two worlds.

4 Time Complexities and Convergence Rates

Formulas (3) and (4) will be used frequently throughout the paper. To lighten up the heavy notation, let us define the following mapping.

Definition 3 (Equilibrium time) . A mapping t ∗ : R ≥ 0 × R n ≥ 0 → R ≥ 0 defined by

is called the equilibrium time . We let t ∗ ( S ) ≡ t ∗ ( S, [ τ i ] n i =1 ) when considering { τ i } from Section 1.

Returning to the algorithm, we guarantee the following iteration complexity.

Theorem 4 (Iteration complexity) . Let Assumptions 1, 2 and 3 hold. Consider any minibatch size S ∈ N := { 1 , 2 , . . . } , any probability p ∈ (0 , 1] , and let the stepsize be γ = ( L -+ L ± √ 1 -p pS ) -1 . Then, after

iterations of Algorithm 1, we have E [ ∥ ∥ ∇ f (ˆ x K ) ∥ ∥ 2 ] ≤ ε , where ˆ x K is sampled uniformly at random from the iterates { x 0 , . . . , x K -1 } .

Theorem 4 states that the iteration complexity is the same as in the optimal PAGE method [Li et al., 2021]. Note that we can guarantee convergence even if the upper bounds { τ i } are unknown or infinite (as long as there exists some worker that can complete computations within a finite time).

We now derive time complexity guarantees. With probability p , the workers need to supply to the algorithm m stochastic gradients at each of the m data samples, which by Theorem 1 can be done in at most t ∗ ( m ) seconds (up to a log factor). Otherwise, they compute S differences of stochastic gradients, which by Theorem 2 takes at most t ∗ ( S ) seconds (up to a constant factor). The resulting time complexity is given in the theorem below.

Theorem 5 (Time complexity with free parameters p and S ) . Consider the assumptions and the parameters from Theorem 4, plus Assumption 4. The expected time complexity of Algorithm 1 is at most

The first term comes from the preprocessing step, where the full gradient is calculated to obtain g 0 = ∇ f ( x 0 ) . Here, we use Assumption 4 that m ≥ n log n. The result (8) is valid even without this assumption, but at the cost of extra logarithmic factors.

4.1 Optimal parameters S and p

The time complexity (8) depends on two free parameters, S ∈ N and p ∈ (0 , 1] . The result below (following from Theorems 13 and 14) determines their optimal choice.

Theorem 6 (Main result) . Consider the assumptions and parameters from Theorem 4, plus Assumption 4. Up to a constant factor, the time complexity (8) is at least T ([ τ i ] n i =1 ) := t ∗ ( m, [ τ i ] n i =1 )

and this lower bound is achieved with S ∗ = arg min S ∈ N F ( S ) and p ∗ = p ∗ ( S ∗ ) , where

Result (9) is the best possible time complexity that can be achieved with the Freya PAGE method. Unfortunately, the final time complexity has non-trivial structure, and the optimal parameters depend on { τ i } in the general case. If we have access to all parameters and times { τ i } , then (9), S ∗ , and p ∗ can be computed efficiently. Indeed, the main problem is to find min S ∈ N F ( S ) , which can be solved, for instance, by using the bisection method, because L -t ∗ ( S, [ τ i ] n i =1 ) is non-decreasing and L ± √ t ∗ ( m, [ τ i ] n i =1 ) t ∗ ( S, [ τ i ] n i =1 ) / √ S is non-increasing in S .

4.2 Optimal parameters S and p in the large-scale regime

Surprisingly, we can significantly simplify the choice of the optimal parameters S and p in the large-scale regime, when √ m ≥ n . This is a weak assumption, since typically the number of data samples is much larger than the number of workers.

Theorem 7 (Main result in the large-scale regime) . Consider the assumptions and parameters from Theorem 4, plus Assumption 4. Up to a constant factor and smoothness constants, if √ m ≥ n, then the optimal choice of parameters in (8) is S ∗ = ⌈ √ m ⌉ and p ∗ = 1 / √ m . For this choice, the expected time complexity of Algorithm 1 is at most

seconds. The iteration complexity with S = ⌈ √ m ⌉ and p = 1 / √ m is K PAGE ≤ 4 δ 0 max { L -,L ± } / ε .

We cannot guarantee that S = ⌈ √ m ⌉ and p = 1 / √ m is the optimal pair when √ m < n, but it is a valid choice for all m ≥ 1 . Note that (10) is true if m ≥ n log n , and it is true up to a log factor if m<n log n . In light of Theorem 15, we can further refine Theorem 7 if the ratio L ± / L is known:

Theorem 8 (Main result in the large-scale regime using the ratio L ± / L ) . Consider the assumptions and parameters from Theorem 4, plus Assumption 4. The expected time complexity of Algorithm 1 is at most Θ( t ∗ ( m, [ τ i ] n i =1 ) + δ 0 L -/ ε × t ∗ (min { max {⌈ L ± √ m / L -⌉ , 1 } , m } , [ τ i ] n i =1 )) seconds, where S = min { max {⌈ L ± √ m / L -⌉ , 1 } , m } and p = S / m .

For brevity reasons, we will continue working with the result from Theorem 7 in the main part of this paper. Note that the optimal parameters do not depend on { τ i } , and can be easily calculated since the number of functions m is known in advance. Hence, our method is fully adaptive to changing and heterogeneous compute times of the workers .

Even if the bounds are unknown and τ i = ∞ for all i ∈ [ n ] , our method converges after 4 δ 0 max { L -, L ± } /ε iterations, and calculates the optimal number of stochastic gradients equal to O ( m + √ mδ 0 max { L -, L ± } /ε ) .

4.3 Discussion of the time complexity

Let us use Definition 3 and unpack the second term in the complexity (10), namely,

A somewhat similar expression involving min j ∈ [ n ] and harmonic means was obtained by Tyurin and Richtárik [2023], Tyurin et al. [2024] for minimizing expectation under the bounded variance assumption. The term δ 0 max { L -, L ± } /ε is standard in optimization [Nesterov, 2018, Lan, 2020] and describes the difficulty of the problem (1). The term min j ∈ [ n ] (( ∑ j i =1 1 / τ i ) -1 ( √ m + j )) represents the average time of one iteration and has some nice properties. For instance, if the last worker is slow and τ n ≈ ∞ , then min j ∈ [ n ] ( · · · ) = min j ∈ [ n -1] ( · · · ) , so the complexity effectively ignores it. Moreover, if j ∗ is an index that minimizes min j ∈ [ n ] ( · · · ) , then min j ∈ [ n ] (( ∑ j i =1 1 / τ i ) -1 ( √ m + j )) = (( ∑ j ∗ i =1 1 / τ i ) -1 ( √ m + j ∗ )) . The last formula, again, does not depend on the slowest workers { j ∗ +1 , . . . , n } , which are automatically excluded from the time complexity expression. The same reasoning applies to the term t ∗ ( m, [ τ i ] n i =1 ) . Let us now consider some extreme examples which are meant to shed some light on our time complexity result (10):

Example 1. [Equally Fast Workers] Suppose that the upper bounds on the processing times are equal, i.e., τ j = τ for all j ∈ [ n ] . Then

The complexity in Example 1 matches that in (2), which makes sense since Soviet PAGE is a reasonable method when { τ i } are equal. Note that the reduction happens without prior knowledge of { τ i } .

Example 2. [Infinitely Fast Worker] If τ 1 = 0 , then T ( 1 / √ m , √ m, [ τ i ] n i =1 ) = 0 .

Example 3. [Infinitely Slow Workers] If τ j = ∞∀ j ∈ [ n ] , then T ( 1 / √ m , √ m, [ τ i ] n i =1 ) = ∞ .

Example 4. [Extremely Slow Workers] Suppose that the times τ j < ∞ are fixed ∀ j ≤ j B and τ j ≥ B ∀ j > j B for some B large enough. Then T ( 1 / √ m , √ m, [ τ i ] n i =1 ) = T ( 1 / √ m , √ m, [ τ i ] j B i =1 ) .

Example 4 says that the workers whose processing time is too large are ignored , which supports the discussion preceding the examples.

4.4 Dynamic bounds

It turns out that the guarantees from Theorem 7 can be generalized to the situation where the compute times { τ i } are allowed to dynamically change throughout the iterations. Consider the k th iteration of Algorithm 1 and assume that worker i calculates one ∇ f j in at most τ k i ∈ [0 , ∞ ] seconds ∀ i ∈ [ n ] , j ∈ [ m ] . Clearly, max k ≥-1 τ k i ≤ τ i ∀ i ∈ [ n ] (where { τ -1 i } are upper bounds from the preprocessing step), but τ k i can be arbitrarily smaller than τ i (possibly τ k i = 0 and τ i = ∞ ).

Theorem 9. Consider the assumptions and parameters from Theorem 4, plus Assumption 4. Up to a constant factor, the expected time complexity of Algorithm 1 with S = ⌈ √ m ⌉ and p = 1 / √ m is at most

where π k, · is a permutation such that τ k π k, 1 ≤ · · · ≤ τ k π k,n for all k ≥ -1 . (This theorem follows from Theorem 22 with the chosen parameters).

Hence, our algorithm is adaptive to the dynamic compute times { τ k i } . Let us consider an example with 2 workers. Assume that the first worker is stable: τ k 1 = τ for all k ≥ 0 , and the second worker is unstable: τ 0 2 = τ is small in the first iteration, and τ 1 2 ≈ ∞ in the second iteration. For explanation purposes, we ignore the preprocessing term t ∗ ( m, · ) , which is not a factor if ε is small. Then,

because t ∗ ( √ m, [ τ 1 π 1 , 1 , τ 1 π 1 , 2 ]) = t ∗ ( √ m, [ τ ]) when τ 1 2 ≈ ∞ . The time complexity in the second iteration depends on the first (stable) worker only, which is reasonable since τ 1 2 ≈ ∞ , and this happens automatically . At the same time, the first term t ∗ ( √ m, [ τ, τ ]) depends on both workers, and this iteration will be faster because t ∗ ( √ m, [ τ, τ ]) ≤ t ∗ ( √ m, [ τ ]) .

4.5 Comparison with previous strategies from Section 1.3

Our time complexities (9) and (10) are better than all known previous guarantees if m ≥ n log n. In particular, T ( 1 / √ m , √ m, [ τ i ] n i =1 ) ≤ T Soviet PAGE (from (2)), because t ∗ ( √ m, [ τ i ] n i =1 ) ≲ √ mτ n / n (Theorem 31). In fact, since lim τ n →∞ t ∗ ( √ m, [ τ i ] n i =1 ) = t ∗ ( √ m, [ τ i ] n -1 i =1 ) < ∞ and lim τ n →∞ = √ mτ n / n = ∞ , T Soviet PAGE can be arbitrarily larger. We also improve on Hero PAGE (see Remark 32).

4.6 Comparison with existing asynchronous variance reduced methods

Several studies have explored asynchronous variance reduced algorithms. Essentially all of them are variants of the existing synchronous methods discussed in Section 1.2 and depend on the slowest worker in every iteration. There have been several attempts to combine variance reduction techniques with asynchronous computations. Perhaps the most relevant baseline is SYNTHESIS , an asynchronous variant of SPIDER [Fang et al., 2018] introduced by Liu et al. [2022]. The obtained gradient complexity matches that of SPIDER in terms of dependence on m , but scales linearly with the bound on the time performance of the slowest worker, making it non-adaptive to slow computations. Moreover, in Line 3 of their Algorithm 3 , the full gradient is calculated, assuming that it can be done for free. Lastly, the analysis assumes the gradients to be bounded.

5 Lower Bound

In previous sections, we showed that Freya PAGE converges in at most (9) or (10) seconds, providing better time complexity guarantees compared to all previous methods. The natural question is: how good are these time complexities, and can they be improved? In Section J, we formalize our setup and prove Theorems 29 and 30, which collectively yield the following lower bound.

Theorem 10 (Less formal version of Theorems 29 and 30) . Assume that 0 < τ 1 ≤ · · · ≤ τ n and take any L + , δ 0 , ε > 0 and m ∈ N such that ε < c 1 L + δ 0 . Then, for any (zero-respecting) algorithm, there exists a function f that satisfies f (0) -f ∗ ≤ δ 0 and Assumption 2, such that it is impossible to find an ε -stationary point faster than in

seconds using uniform sampling with replacement.

Comparing (10) and (12), we see that Freya PAGE is optimal under Assumptions 1 and 2 in the large-scale regime ( √ m ≥ n ). Indeed, without Assumption 3, we have max { L -, L ± } = max { L -, L + } = L + . Up to constant factor, (10) is less or equal to (12) since

This is the first optimal method for the problem we consider , and Theorem 10 gives the first lower bound .

6 Using the Developed Strategies in Other Methods

ComputeBatchDifference (Algorithm 3) is a generic subroutine and can be used in other methods. In Section H, we introduce Freya SGD , a simple algorithm with update rule x k +1 = x k -γ × ComputeBatch ( S, x k ) , where S is a batch size and ComputeBatch (Algorithm 4) is a

minor modification of ComputeBatchDifference . Theorem 25 establishes that Freya SGD converges in O ( 1 / ε × t ∗ ( 1 / ε , [ τ i ] n i =1 )) seconds (where we only keep the dependence on ε and { τ i } ). For small ε , this complexity is worse than (10), but it can be better, for instance, in the interpolation regime [Schmidt and Roux, 2013, Ma et al., 2018]. Freya SGD resembles Rennala SGD [Tyurin and Richtárik, 2023], but unlike the latter, it is specialized to work with finite-sum problems (1) and does not require the σ 2 -bounded variance assumption on stochastic gradients (which is not satisfied in our setting).

Acknowledgments and Disclosure of Funding

The research reported in this publication was supported by funding from King Abdullah University of Science and Technology (KAUST): i) KAUST Baseline Research Scheme, ii) Center of Excellence for Generative AI, under award number 5940, iii) SDAIA-KAUST Center of Excellence in Artificial Intelligence and Data Science. The work of A.T. was partially supported by the Analytical center under the RF Government (subsidy agreement 000000D730321P5Q0002, Grant No. 70-2021-00145 02.11.2021).

References

Allen-Zhu and E. Hazan. Variance reduction for faster non-convex optimization. In International Conference on Machine Learning , pages 699-707. PMLR, 2016.

Arjevani, Y. Carmon, J. C. Duchi, D. J. Foster, N. Srebro, and B. Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming , pages 1-50, 2022.

Carmon, J. C. Duchi, O. Hinder, and A. Sidford. Lower bounds for finding stationary points I. Mathematical Programming , 184(1):71-120, 2020.

Chen, X. Pan, R. Monga, S. Bengio, and R. Jozefowicz. Revisiting distributed synchronous SGD. arXiv preprint arXiv:1604.00981 , 2016.

Dutta, G. Joshi, S. Ghosh, P. Dube, and P. Nagpurkar. Slow and stale gradients can win the race: Error-runtime trade-offs in distributed SGD. In International Conference on Artificial Intelligence and Statistics , pages 803-812. PMLR, 2018.

Fang, C. J. Li, Z. Lin, and T. Zhang. SPIDER: Near-optimal non-convex optimization via stochastic path-integrated differential estimator. Advances in Neural Information Processing Systems , 31, 2018.

Horváth and P. Richtárik. Nonconvex variance reduced optimization with arbitrary sampling. In International Conference on Machine Learning , pages 2781-2789. PMLR, 2019.

Horváth, L. Lei, P. Richtárik, and M. I. Jordan. Adaptivity of stochastic gradient methods for nonconvex optimization. SIAM Journal on Mathematics of Data Science , 4(2):634-648, 2022.

Khaled and P. Richtárik. Better theory for SGD in the nonconvex world. Transactions on Machine Learning Research , 2022.

P. Kingma and J. Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations , 2015.

Koloskova, S. U. Stich, and M. Jaggi. Sharper convergence guarantees for asynchronous SGD for distributed and federated learning. Advances in Neural Information Processing Systems , 2022.

Kovalev, A. Beznosikov, E. Borodich, A. Gasnikov, and G. Scutari. Optimal gradient sliding and its application to optimal distributed optimization under similarity. Advances in Neural Information Processing Systems , 35:33494-33507, 2022.

Lan. First-order and stochastic optimization methods for machine learning . Springer, 2020.

LeCun, C. Cortes, and C. Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , 2, 2010.

Lei, C. Ju, J. Chen, and M. I. Jordan. Non-convex finite-sum optimization via SCSG methods. Advances in Neural Information Processing Systems , 30, 2017.

Li, H. Bao, X. Zhang, and P. Richtárik. PAGE: A simple and optimal probabilistic gradient estimator for nonconvex optimization. In International Conference on Machine Learning , pages 6286-6295. PMLR, 2021.

Liu, X. Zhang, and J. Liu. Synthesis: A semi-asynchronous path-integrated stochastic gradient method for distributed learning in computing clusters. In Proceedings of the Twenty-Third International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing , pages 151-160, 2022.

Ma, R. Bassily, and M. Belkin. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning , pages 3325-3334. PMLR, 2018.

Mishchenko, F. Bach, M. Even, and B. Woodworth. Asynchronous SGD beats minibatch SGD under arbitrary delays. Advances in Neural Information Processing Systems , 2022.

S. Nemirovskij and D. B. Yudin. Problem complexity and method efficiency in optimization . Wiley-Interscience, 1983.

Nesterov. Lectures on convex optimization , volume 137. Springer, 2018.

Nguyen, P. H. Nguyen, M. Dijk, P. Richtárik, K. Scheinberg, and M. Takác. SGD and hogwild! convergence without the bounded gradients assumption. In International Conference on Machine Learning , pages 3750-3758. PMLR, 2018.

M. Nguyen, J. Liu, K. Scheinberg, and M. Takáˇ c. SARAH: A novel method for machine learning problems using stochastic recursive gradient. In International Conference on Machine Learning , pages 2613-2621. PMLR, 2017.

Recht, C. Re, S. Wright, and F. Niu. Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. Advances in Neural Information Processing Systems , 24, 2011.

J. Reddi, A. Hefny, S. Sra, B. Poczos, and A. Smola. Stochastic variance reduction for nonconvex optimization. In International Conference on Machine Learning , pages 314-323. PMLR, 2016.

Schmidt and N. L. Roux. Fast convergence of stochastic gradient descent under a strong growth condition. arXiv preprint arXiv:1308.6370 , 2013.

Szlendak, A. Tyurin, and P. Richtárik. Permutation compressors for provably faster distributed nonconvex optimization. In International Conference on Learning Representations , 2021.

Tyurin and P. Richtárik. Optimal time complexities of parallel stochastic optimization methods under a fixed computation model. Advances in Neural Information Processing Systems , 2023.

Tyurin, L. Sun, K. Burlachenko, and P. Richtárik. Sharper rates and flexible framework for nonconvex sgd with client and data sampling. Transactions on Machine Learning Research , 2023.

Tyurin, M. Pozzi, I. Ilin, and P. Richtárik. Shadowheart SGD: Distributed asynchronous SGD with optimal time complexity under arbitrary computation and communication heterogeneity. arXiv preprint arXiv:2402.04785 , 2024.

Wang, K. Ji, Y. Zhou, Y. Liang, and V. Tarokh. SpiderBoost and momentum: Faster variance reduction algorithms. Advances in Neural Information Processing Systems , 32, 2019.

Zhou, P. Xu, and Q. Gu. Stochastic nested variance reduction for nonconvex optimization. Journal of Machine Learning Research , 21(103):1-63, 2020.

APPENDIX

Contents

A Experiments

We compare Freya PAGE with Rennala SGD , Asynchronous SGD , and Soviet PAGE on nonconvex quadratic optimization tasks and practical machine learning problems. The experiments were conducted in Python 3.8 with Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz. We developed a library that emulates the working behavior of thousands of nodes.

A.1 Experiments with nonconvex quadratic optimization

0

100000

200000

300000

400000

500000

times (seconds)

10

2

10

1

f

(

x t

)

f

(

x

*

)

Soviet PAGE: Step size: 1.0

Soviet PAGE: Step size: 0.5

(a) n = 1000

0

100000

200000

300000

400000

500000

times (seconds)

10

2

10

1

f

(

x t

)

f

(

x

*

)

Soviet PAGE: Step size: 1.0

Soviet PAGE: Step size: 0.5

(b) n = 10000

In the first set of experiments, we compare the algorithms on a synthetic quadratic optimization task generated using the procedure from Section I. To ensure robust and fair comparison, we fix the performance of each worker and emulate our setup by assuming that the i th worker requires √ i seconds to calculate a stochastic gradient. For each algorithm, we fine-tune the step size from the set { 2 i | i ∈ [ -20 , 20] } . Uniform sampling with replacement is used across all methods. In Freya PAGE , we set S = ⌈ √ m ⌉ according to Theorem 7. We consider n ∈ { 1000 , 10000 } and in each case plot the best run of each method.

The results are presented in Figure 1. It is evident that our new method, Freya PAGE , has the best convergence performance among all algorithms considered. The convergence behavior of Rennala SGD and Asynchronous SGD is very noisy, and both achieve lower accuracy than Freya PAGE . Furthermore, the gap between Freya PAGE and Soviet PAGE widens with increasing n because Soviet PAGE is not robust to the presence of slow workers.

A.2 Experiments with logistic regression

0.0

0.2

0.4

0.6

0.8

1.0

times (seconds)

1e6

2.2 × 10

1

2.4 × 10

1

2.6 × 10

1

2.8 × 10

1

3×10 1

f

(

x t

)

f

(

x

*

)

Soviet PAGE: Step size: 0.5

Soviet PAGE: Step size: 0.25

(a)

n

= 100

0

100000 200000 300000 400000 500000 600000

times (seconds)

2.2 × 10

1

2.4 × 10

1

2.6 × 10

1

2.8 × 10

1

3×10 1

f

(

x t

)

f

(

x

*

)

Soviet PAGE: Step size: 0.5

Soviet PAGE: Step size: 0.25

(b)

n

= 10000

We now consider the logistic regression problem on the MNIST dataset [LeCun et al., 2010], where each algorithm samples one data point at a time. The results of the experiments are presented in Figure 2. The difference between Freya PAGE and Rennala SGD / Asynchronous SGD is not as pronounced as in Section A.1: the methods have almost the same performance for this particular problem. However, our method still outperforms its competitors in the low accuracy regime, and is significantly better than Soviet PAGE .

A critical disadvantage of Rennala SGD and Asynchronous SGD is their noisy behavior, evident in both the plots and reflected in higher variance of the accuracy (see Table 2). In contrast, the iterations of Freya PAGE in Figure 2 are smooth, and its accuracy exhibits the lowest variance, as shown in Table 2. This stability can be attributed to the variance-reduction nature of Freya PAGE .

B The Time Complexity Guarantees of Algorithms 3 and 4

In addition to ComputeBatchDifference (Algorithm 3) introduced in the main part, we also analyze ComputeBatch (Algorithm 4) that is similar to ComputeBatchDifference , but calculates a minibatch of stochastic gradients ∇ f i ( x ) instead of stochastic gradient differences ∇ f i ( x ) -∇ f i ( y ) .

Algorithm 4 ComputeBatch ( S , x )

1: Input: batch size S ∈ N , point x ∈ R d

10:

As soon as some worker finishes calculating the stochastic gradient difference, it immediately starts computing the difference of the next pair. Hence, by the time t, all workers will have processed at least

Using Lemma 4, we have t ≥ 4 τ i for all i ≤ j ∗ . Thus, we get t 2 τ i ≥ 1 for all i ≤ j ∗ and

We can conclude that by the time (5), the algorithm will have calculated S pairs of stochastic gradients and exited the loop.

Theorem 11. The time needed by Algorithm 4 to calculate g is at most

seconds.

Proof. Let pairs. Assume that

seconds.

Proof. The proof of this theorem is essentially the same as the proof of Theorem 2. The only difference is that Algorithm 4 calculates ∇ f i ( x ) instead of ∇ f i ( x ) -∇ f i ( y ) .

C The Time Complexity Guarantees of Algorithms 2 and 5

Instead of Algorithm 2, we analyze a more general Algorithm 5 that reduces to Algorithm 2 when S = [ m ] .

Algorithm 5 ComputeBatchAnySampling ( S , x ) 1: Input: multiset S , point x ∈ R d 2: Init g = 0 ∈ R d , multiset M = ∅ 3: Broadcast x to all workers 4: For each worker, sample j from S (uniformly) and ask it to calculate ∇ f j ( x ) 5: while M̸ = S do 6: Wait for ∇ f p ( x ) from a worker 7: if p ∈ S\M then 8: g ← g + 1 |S| ∇ f p ( x ) 9: Update M←M∪{ p } 10: end if 11: Sample j from S\M (uniformly) and ask this worker to calculate ∇ f j ( x ) 12: end while 13: Return g = 1 |S| ∑ i ∈S ∇ f i ( x )

Theorem 12. The expected time needed by Algorithm 5 to calculate g = 1 |S| ∑ i ∈S ∇ f i is at most

seconds.

Proof Sketch: While the following proof is technical, the intuition and idea behind it and the algorithm are relatively simple. For simplicity, assume that n ≥ |S| . The set S\M includes all indices that have not yet been calculated. Each worker is assigned a new random index from S\M and starts the calculation of the gradient. At the beginning of the algorithm, when the set S\M is large, the probability that two workers are assigned the same index is very small. Hence, using the same idea as in the proof of Theorem 2, the workers will calculate ≈ |S| n stochastic gradients after

seconds. However, once the size of S\M becomes roughly equal to n , the probability that two workers sample the same index increases. In the final steps of the algorithm, we encounter the same issue as in the famous coupon collector's problem , resulting in an additional factor of n log n because some stochastic gradients will be calculated multiple times.

Proof. Let us define S = |S| and take any k ∈ [ n ] . We refer to the workers with the upper bounds τ i such that τ i ≤ τ k as 'fast', and the others will be termed 'slow'.

Consider the moment when the algorithm samples j from the set S\M to allocate it to one of the 'fast' workers (Line 4 or 11). The probability of sampling j such that ∇ f j ( x ) is currently being calculated by another 'fast' worker is

|{ indices from S\M taken by 'fast' workers }| + |{ indices from S\M not taken by 'fast' workers }|

because there are at most k 'fast' workers and at most S distinct stochastic gradients. Let us define the set

A 'fast' worker can be 'unlucky' and start calculating a stochastic gradient that is being computed by another 'fast' worker. However, with probability at least

it will take a new index j that was not previously taken by another 'fast' worker.

Thus, the while loop of the algorithm defines a Markov process that begins with some U ⊆ S . The size of U decreases by one with probability at least |U| min { k,S } + |U| in iterations where the algorithm samples j from U and asks a 'fast' worker to calculate the stochastic gradient. Additionally, the size of U can decrease by one when a 'slow' worker finishes calculating a stochastic gradient from U .

Let ¯ t be the time required for the Markov process to reach the state U = ∅ . Then, the while loop in Algorithm 2 will finish after at most

seconds because once U = ∅ , all non-processed indices from S\M are assigned to the 'fast' workers, so calculating the remaining stochastic gradients will take at most τ k seconds.

It remains to estimate ¯ t. Let η p be the number of iterations of the while loop where the algorithm samples j from S\M and asks a 'fast' worker to calculate the stochastic gradient when |U| = p. By the definition of the Markov chain, we have

because with probability at least p min { k,S } + p , one of the ('lucky') 'fast' workers receives j from U and decreases the size of U by 1 ( η p has a geometric-like distribution).

Since |U| ≤ S at the beginning of the while loop, it is sufficient for the 'fast' workers to calculate at most

stochastic gradients to ensure that U = ∅ (it is possible that some stochastic gradients will be calculated many times). Indeed, if |U| = p for the first moment, then after η p +1 calculations of stochastic gradients by the 'fast' workers, the size of set will be at most p -1 . The last 'plus one' calculation can only happen when |U| = p -1 .

The time required for the 'fast' workers to process this number of stochastic gradients is at most

because for this choice of t ′ , we have

where ⌊ t ′ τ i ⌋ is the number of stochastic gradients that worker i can calculate in t ′ seconds. Taking expectation gives

where we use the standard bound on the harmonic series. Thus, the expectation of the total time (15) can be bounded by

where in the last line we add k ≥ 0 . Recall that k is a parameter we can choose. Let us take

Using Lemma 4, we have

and hence

D Proofs for Algorithm 1 ( Freya PAGE )

The proofs use the simplified notation t ∗ ( S ) := t ∗ ( S, [ τ i ] n i =1 ) from Definition 3.

Since the update rule of PAGE coincides with that of Freya PAGE , one can directly apply the iteration complexity results established in Tyurin et al. [2023].

Theorem 4 (Iteration complexity) . Let Assumptions 1, 2 and 3 hold. Consider any minibatch size S ∈ N := { 1 , 2 , . . . } , any probability p ∈ (0 , 1] , and let the stepsize be γ = ( L -+ L ± √ 1 -p pS ) -1 . Then, after

iterations of Algorithm 1, we have E [ ∥ ∥ ∇ f (ˆ x K ) ∥ ∥ 2 ] ≤ ε , where ˆ x K is sampled uniformly at random from the iterates { x 0 , . . . , x K -1 } .

Proof. The result follows from Theorem 6 of Tyurin et al. [2023], using the parameters from the 'Uniform With Replacement' line in Table 1 of the same work.

Theorem 5 (Time complexity with free parameters p and S ) . Consider the assumptions and the parameters from Theorem 4, plus Assumption 4. The expected time complexity of Algorithm 1 is at most

Proof. The result established in Theorem 4 says that the iteration complexity of the algorithm is

At each iteration, with probability 1 -p , the workers compute S differences of stochastic gradients, which by Theorem 2 takes

seconds. Otherwise, they collect the full gradient, which can be done (Theorem 1) in

seconds, where the inequality uses Assumption 4. Hence, recalling the notation

the (expected) time complexity of the method is

where the term t ∗ ( m ) corresponds to the preprocessing step, when the algorithm needs to calculate g 0 = ∇ f ( x 0 ) = 1 / m ∑ m i =1 ∇ f i ( x 0 ) .

Theorem 13. Up to a constant factor, the time complexity T ( p, S, [ τ i ] n i =1 ) from (8) is at least

and attains this value with

Proof. Up to a constant factor, by Theorem 5, the time complexity of Freya PAGE is

Let us denote the second term in the above equation as T p,S . Then for all p ≥ 1 2 , we have

and for all S ≥ 2

Otherwise, when p < 1 2 and S < m 2 , we have

Hence, up to a constant factor,

It can be easily verified that this bound can be attained (up to a constant factor) using the parameter p as defined in (18).

Theorem 14. Up to a constant factor, the minimum of the time complexities (8) and (17) is

and is achieved for

and p ∗ = p ∗ ( S ∗ ) , where p ∗ ( S ) is defined in (18) .

Proof. This is a simple corollary of Theorem 13: we take S

that minimizes (17).

In certain scenarios, we can derive the optimal parameter values explicitly.

Theorem 15.

If n ≤ L ± √ m L -≤ m , then (up to constants) S ∗ = L ± √ m L -and p ∗ = L ± L -√ m are optimal parameters and

2. If L ± √ m L -≤ 1 , then (up to constants) S ∗ = 1 and p ∗ = 1 / m are optimal parameters and

3. If L ± √ m L -≥ m , then (up to constants) p ∗ = 1 is an optimal parameter and

for any S ∈ [ m ] .

Proof. We fist consider the case when n ≤ L ± √ m L -≤ m . Since j ≤ n ≤ L ± √ m L -, we have

and from the assumption that L -≥ L ± √ m it follows that

for all j ∈ [ n ] . Thus,

and

It follows that

Since L ± √ m L -≤ m , we have

and thus, according to the result from Theorem 14, it is sufficient to minimize

First, let us note that

If S ≥ L ± m L -, then

Otherwise, if S < L ± m L -, then

Therefore, the optimal choice is S ∗ = L ± m L -, and by Theorem 13 and inequality (23), p should chosen to be

Using (22), we can conclude that p ∗ = L ± L -√ m is optimal, proving the first part of the Theorem.

Next, consider the case when L ± √ m L -≤ 1 . By the reasoning above, it is sufficient to minimize

First, let us note that

where the last inequality follows from the fact that for any S ∈ [ m ] , t ∗ ( S ) / S ≤ t ∗ (1) . On the other hand, if S ≥ 1 , then

Therefore, the optimal choice is S ∗ = 1 . Then

while for any p

Hence p ∗ = 1 / m .

It remains to prove the third result. Suppose that L -< L ± √ m . Then, the last part of the theorem follows from the fact that

In practice, the values of smoothness constants are often unknown. However, the algorithm can still be run with close to optimal parameters.

Theorem 7 (Main result in the large-scale regime) . Consider the assumptions and parameters from Theorem 4, plus Assumption 4. Up to a constant factor and smoothness constants, if √ m ≥ n, then the optimal choice of parameters in (8) is S ∗ = ⌈ √ m ⌉ and p ∗ = 1 / √ m . For this choice, the expected time complexity of Algorithm 1 is at most

seconds. The iteration complexity with S = ⌈ √ m ⌉ and p = 1 / √ m is K PAGE ≤ 4 δ 0 max { L -,L ± } / ε .

Proof. The proof is the same as in Theorem 15. Indeed, up to a constant factor, the time complexity (8) can be bounded as

Therefore, by setting L ± = L -in Theorem 15, one can easily derive the parameters p and S that are optimal up to the smoothness constants. The time complexity (10) can be obtained by applying S = ⌈ √ m ⌉ and p = 1 / √ m to (8).

E Freya PAGE with Other Samplings

Algorithm 1 can be adapted to accommodate other sampling methods. This brings us to the introduction of Algorithm 6, which supports virtually any sampling strategy, formalized by the following mapping:

Definition 16 (Sampling) . A sampling is a random mapping S S , which takes as an input a set of indices I := { a 1 , . . . , a m } and returns a (multi)set { a i 1 , . . . , a i S } , where a i j ∈ I for all j ∈ [ S ] .

Algorithm 6 Freya PAGE (with virtually any sampling)

Algorithm 7 ComputeBatchDifferenceAnySampling ( S , x , y )

1: Input: multiset S , points x, y ∈ R d

2: Init g = 0 ∈ R d , multiset M = ∅ 3: Broadcast x to all workers 4: For each worker, sample j from S (uniformly) and ask it to calculate ∇ f j ( x ) -∇ f j ( y ) 5: while M̸ = S do 6: Wait for ∇ f p ( x ) -∇ f p ( y ) from a worker 7: if p ∈ S\M then 8: g ← g + 1 |S| ( ∇ f p ( x ) -∇ f p ( y )) 9: Update M←M∪{ p } 10: end if 11: Sample j from S\M (uniformly) and ask this worker to calculate ∇ f j ( x ) -∇ f j ( y ) 12: end while 13: Return g = 1 |S| ∑ i ∈S ( ∇ f i ( x ) -∇ f i ( y ))

The only difference is that instead of ComputeBatchDifference (Algorithm 5), Algorithm 6 uses a new subroutine, called ComputeBatchDifferenceAnySampling (Algorithm 3).

For this algorithm, we can prove the following time complexity guarantees.

Theorem 17. The expected time needed by Algorithm 7 to calculate g = 1 |S| ∑ i ∈S ( ∇ f i ( x ) -∇ f i ( y )) is at most

seconds.

Proof. The proof of this theorem is the same as the proof of Theorem 12. We only have to multiply (14) by 2 because Algorithm 3 calculates ∇ f i ( x ) -∇ f i ( y ) instead of ∇ f i ( x ) .

While changing the sampling strategy might affect the iteration complexity of the method, for a fixed minibatch size S , the time complexity of a single iteration remains unchanged. Thus, having established the expected time needed by the algorithm to perform a single iteration (i.e., to collect a minibatch of stochastic gradients of the required size), one can simply multiply it by the iteration complexity of the method determined for any supported sampling technique to obtain the resulting time complexity.

With this in mind, we now analyse the time complexity of Algorithm 6 with 2 different sampling techniques: nice sampling and importance sampling. However, it can be analyzed with virtually any other unbiased sampling [Tyurin et al., 2023].

E.1 Nice sampling

Nice sampling returns a random subset of fixed cardinality S chosen uniformly from [ m ] . Unlike uniform sampling with replacement used in Algorithm 5, which returns a random multiset (that can include repetitions), the samples obtained by nice sampling are distinct. The iteration complexity of Algorithm 6 with nice sampling is given by the following theorem.

Theorem 18 (Tyurin et al. [2023], Section 3 ) . Let Assumptions 1, 2 and 3 hold. Choose a minibatch size S ∈ [ m ] , a probability p ∈ (0 , 1] and the stepsize

Then, the number of iteration needed by Algorithm 6 with nice sampling to reach an ε -stationary point is

Proof. The result follows from Theorem 6 and Table 1 from [Tyurin et al., 2023].

Theorem 19. Consider the assumptions and parameters from Theorem 18 and Assumption 4. Up to a constant factor, the time complexity of Algorithm 6 is

where t ∗ is defined from Definition 3.

Remark 20. Compared to Theorem 5, which uses uniform sampling with replacement , the guarantees for nice sampling are slightly worse: the term t ∗ ( S, [ τ i ] n i =1 ) from Theorem 5 here is replaced with t ∗ ( S + min { S, n } log (min { S, n } ) , [ τ i ] n i =1 ) . Ignoring the logarithmic term log (min { S, n } ) ( ≤ log (min { m,n } ) ), the result from Theorem 19 is equivalent to that in Theorem 5. Thus, Theorems 6 and 7 hold also for the nice sampling (up to logarithmic factors).

Proof. We use the same reasoning as in the proof of Theorem 5. With probability p, the algorithm calculates the full gradients, which by Theorem 1 requires

seconds, where we use Assumption 4. With probability 1 -p, the algorithm calls ComputeBatchDifferenceAnySampling , which by Theorem 17 requires

seconds. One can obtain the result by multiplying the iteration complexity (25) by the expected time needed to collect the required number of stochastic gradients per iteration and adding the preprocessing time.

E.2 Importance sampling

Here we additionally assume L i -smoothness of the local objective functions f i . Assumption 5. The functions f i are L i -smooth. We denote ¯ L := 1 m ∑ m j =1 L j and L max max i ∈ [ n ] L i .

:=

Importance sampling is a sampling technique that returns a multiset of indices with repetitions . Index j is included in the multiset with probability L j ∑ n i =1 L j .

Theorem 21 (Tyurin et al. [2023], Section 3 ) . Let Assumptions 1 and 5 hold. Choose a minibatch size S ∈ [ m ] , probability p ∈ (0 , 1] and the stepsize

Then, the number of iteration needed by Algorithm 1 with importance sampling to reach an ε -stationary point is

The complexity (27) is nearly identical to (7) and (25), with the only difference being the dependence on ¯ L rather than L ± . Thus, all the results up to constant and logarithmic factors can be derived using the same methodology as that outlined in Section 4, with the simple substitution of L ± with ¯ L .

F Dynamic Bounds

As noted in Section 4.4, the results from Section D can be easily generalized to iteration-dependent processing times.

Theorem 22. Consider the assumptions and the parameters from Theorem 4 and Assumption 4. Up to a constant factor, the time complexity of Freya PAGE (Algorithm 1) with iteration-dependent processing times { τ k i } , which are defined in Section 4.4, is at most

to find an ε -stationary point, where p ∈ (0 , 1] and S ∈ N are free parameters, and π k, · is a permutation such that τ k π k, 1 ≤ · · · ≤ τ k π k,n for all k ≥ -1 .

Remark 23. The theorem can be trivially extended to other samplings by changing K PAGE to the iteration complexities from Theorems 18 and 21.

Proof. The reasoning behind this result is exactly the same as in the proof of Theorem 5. The only difference is that in this more general setting, the expected time per iteration varies across iterations. Therefore, instead of simply multiplying, one needs to sum over the iterations to obtain the total time complexity.

We introduce the permutations to ensure that { τ k π k,i } n i =1 are sorted. When τ k i = τ i , there is no need to introduce them because, throughout the paper, it is assumed τ 1 ≤ . . . ≤ τ n (see Section 1).

G Examples

Here we provide the proofs for the examples from Section 4.3. We will use the notation

for a fixed S ∈ [ m ] and all j ∈ [ n ] .

Example 1. [Equally Fast Workers] Suppose that the upper bounds on the processing times are equal, i.e., τ j = τ for all j ∈ [ n ] . Then

Proof. First, when τ j = τ for all j ∈ [ n ] , then for any S ∈ [ m ] , t ( S, j ) is minimized by taking j = n :

It remains to substitute this equality in (10).

Example 2. [Infinitely Fast Worker] If τ 1 = 0 , then T ( 1 / √ m , √ m, [ τ i ] n i =1 ) = 0 .

Proof. The statement follows easily from the fact that for any S ∈ [ m ] we have

Example 3. [Infinitely Slow Workers] If τ j = ∞∀ j ∈ [ n ] , then T ( 1 / √ m , √ m, [ τ i ] n i =1 ) = ∞ .

Proof. This follows from the fact that for any S ∈ [ m ] and any j ∈ [ n ] we have

Example 4. [Extremely Slow Workers] Suppose that the times τ j < ∞ are fixed ∀ j ≤ j B and τ j ≥ B ∀ j > j B for some B large enough. Then T ( 1 / √ m , √ m, [ τ i ] n i =1 ) = T ( 1 / √ m , √ m, [ τ i ] j B i =1 ) .

Proof. Suppose that B ≥ m + j B ∑ j B i =1 1 τ i and fix any k > j B . Then, since τ j ≥ B for all j > j B , we have

for any S ∈ [ m ] . Rearranging and adding ( S + j B ) ∑ j B i =1 1 τ i to both sides of the inequality, we obtain

meaning that

for any k > j B and any S ∈ [ m ] . Therefore,

for any S ∈ [ m ] , which proves the claim.

H A New Stochastic Gradient Method: Freya SGD

In this section, we introduce a new a non-variance reduced SGD method that we call Freya SGD . Freya SGD is closely aligned with Rennala SGD [Tyurin and Richtárik, 2023], but Freya SGD does not require the σ 2 -bounded variance assumption on stochastic gradients.

Algorithm 8 Freya SGD

Parameters: starting point x ∈ R , learning rate γ > 0 , minibatch size S

1: 0 d 2: for k = 0 , 1 , . . . , K -1 do 3: 1 S ∑ i ∈S k ∇ f i ( x k ) = ComputeBatch ( S, x k ) 4: x k +1 = x k -γ 1 S ∑ i ∈S k ∇ f i ( x k )

5: end for

(note): S k is a set of size ∣ ∣ S k ∣ ∣ = S of i.i.d. indices sampled from [ m ] uniformly with replacement

Assumption 6. For all i ∈ [ n ] , there exists f ∗ i such that f i ( x ) ≥ f ∗ i for all x ∈ R d .

We define

Theorem 24. Let Assumptions 1, 5 and 6 hold. Choose minibatch size S ∈ [ m ] and stepsize

where

Then, the number of iterations needed by Algorithm 8 to reach an ε -stationary point is ⌈ K SGD ⌉ .

Proof. The iteration complexity can be proved using Corollary 1 and Proposition 3 (i) of Khaled and Richtárik [2022] (with q i = 1 /n ).

Theorem 25. Consider the assumptions and the parameters from Theorem 24. Up to a constant factor, the time complexity of Freya SGD (Algorithm 8) is at most

and is minimized by choosing

Up to a constant factor, we get

Proof. At each iteration, the algorithm needs to collect a minibatch of stochastic gradients of size S . Multiplying the iteration complexity of Theorem 24 by the time needed to gather such a minibatch (Theorem 11), the resulting time complexity is

(Alg. 4)

We now find the optimal S . Assume first that L max ε ( δ 0 +∆ ∗ ) ≤ 1 2 . Assumption 5 ensures that the function f is L max -smooth. Thus, we have ∥ ∥ ∇ f ( x 0 ) ∥ ∥ 2 ≤ 2 L max δ 0 ≤ ε, and x 0 is an ε -solution. Therefore, we can take any S ≥ 1 .

Now, suppose that L max ε ( δ 0 +∆ ∗ ) > 1 2 . Then we have

For all S > max { L max ε ( δ 0 +∆ ∗ ) , 1 } , we get S ≥ 2 and hence

Let us now consider the case 1 < S ≤ max { L max ε ( δ 0 +∆ ∗ ) , 1 } . We can additionally assume that L max ε ( δ 0 +∆ ∗ ) > 1 and S ≤ L max ε ( δ 0 +∆ ∗ ) (otherwise, the set S that satisfies the condition 1 < S ≤ max { L max ε ( δ 0 +∆ ∗ ) , 1 } is empty). We get

Finally, for S = 1 , we have

because we assume L max ε ( δ 0 +∆ ∗ ) > 1 2 . Therefore, an optimal choice is S ∗ = L max ε ( δ 0 +∆ ∗ ) .

I Setup of the Experiments from Section A.1

We consider the optimization problem (1) with nonconvex quadratic functions. The matrices and vectors defining the objective functions f i are generated using Algorithm 9 with m = 10000 , d = 1000 , λ = 1e -6 , and s = 10 . The output is used to construct

Algorithm 9 Quadratic optimization task generation

1: Parameters: number of functions m , dimension d , regularizer λ , noise scale s

2: for i = 1 , . . . , m do

3: Generate random noises ν s i = 1 + sξ s i and ν b i = sξ b i , i.i.d. ξ s i , ξ b i ∼ N (0 , 1)

5: Take the initial tridiagonal matrix

4: Let b i = ν s i 4 ( -1 + ν b i , 0 , · · · , 0) ∈ R d

7: Take the mean of matrices A = 1 m ∑ m i =1 A i

6: end for

8: Find the minimum eigenvalue λ min ( A )

9: for i = 1 , . . . , m do

10: Update matrix A i = A i +( λ -λ min ( A )) I

11: end for

12: Take starting point x 0 = ( √ d, 0 , · · · , 0)

13: Output: matrices A 1 , · · · , A m , vectors b 1 , · · · , b m , starting point x 0

J Lower bound

J.1 Time multiple oracles protocol

The classical lower bound frameworks [Nemirovskij and Yudin, 1983, Carmon et al., 2020, Arjevani et al., 2022, Nesterov, 2018] are not convenient in the analysis of parallel algorithms since they are designed to estimate lower bounds on iteration complexities . In order to obtain time complexity lower bounds, we use the framework by Tyurin and Richtárik [2023]. Let us briefly explain the main idea. A more detailed explanation can be found in [Tyurin and Richtárik, 2023][Sections 3-6].

We start by introducing an appropriate oracle for our setup:

where j ∼ Uniform ([ m ]) , i.e., j is a random index sampled uniformly from the set [ m ] . We assume that all draws from Uniform ([ m ]) are i.i.d..

Next, we define the time multiple oracles protocol , first introduced in [Tyurin and Richtárik, 2023].

Protocol 10 Time Multiple Oracles Protocol

1: Input: function f = 1 m ∑ m i =1 f i , oracles and distributions ( O 1 , ..., O n ) ∈ O ( f ) , algorithm A ∈ A 0

2: s i = 0 for all i ∈ [ n ]

3: for k = 0 , . . . , ∞ do

4:

(

t

k

+1

, i

k

+1

, x

k

) =

5:

(

s

k

+1

k

i

+1

6: end for

Let us explain the behavior of the protocol. At each iteration, the algorithm A returns three outputs, based on the available information/gradients: time t k +1 , the index of a worker i k +1 , and a new point x k . Depending on the current time t k +1 and the state of the worker, three options are possible (see (29)). If s q = 0 , then the worker is idle. It then starts calculations at the point x k , changes the state s q from 0 to 1 , stores the point x k in s x (at which a new stochastic gradient should be calculated), and returns a zero vector. If s q = 0 and t k +1 < s t + τ, then the worker is still calculating a stochastic gradient. It does not change the state and returns a zero vector because the computation has not finished yet. If s q = 0 and t k +1 ≥ s t + τ, the worker can finally return a stochastic gradient at s x because sufficient time has passed since the worker was idle ( s q = 0 ). Note that with this oracle, the algorithm will never receive the first stochastic gradient before time τ (assuming that all oracles have the same processing time τ ; in general, we will assume that the processing times are different).

In the setting considered in this work, there are n oracles that can do calculations in parallel, and an algorithm orchestrates their work. Let the processing times of the oracles be equal to τ 1 , . . . , τ n . A reasonable strategy would be to call each oracle with t k = 0 , then to call the fastest worker with t k = min i ∈ [ n ] τ i to get the first stochastic gradients as soon as possible, then to call this worker again with t k = min i ∈ [ n ] τ i to request calculation of the next stochastic gradient, and so on. One unusual thing about this protocol is that the algorithm controls the time. The oracle is designed to force the algorithm to increase the time; otherwise, the algorithm would not receive new information about the function.

Our goal will be to bound the complexity measure m time ( A , F ) , defined as

, g

k

+1

) =

O

i

A

k

k

+1

(

g

(

1

t

k

, . . . , g

+1

k

)

k

, x

,

k

i

k

, s

+1

)

̸

where the sequences t k and x k are generated by Protocol 10. Hence, unlike the classical approach, where the lower bounds are obtained for the minimum number of iterations required to find an ε -stationary point, we seek to find the minimum time needed to get an ε -stationary point.

We consider a standard for our setup class of functions [Fang et al., 2018]:

Definition 26 (Function Class F m δ 0 ,L + ) . We say that f ∈ F m δ 0 ,L + if it is δ 0 -bounded, i.e., f (0) -inf x ∈ R d f ( x ) ≤ δ 0 , and

where the functions f i : R d → R are differentiable and satisfy

Next, we define the class of algorithms we will analyze.

Definition 27 (Algorithm Class A zr ) . Let us consider Protocol 10. We say that a sequence of mappings A = { A k } ∞ k =0 is a zero-respecting algorithm , if

̸

We denote the set of all algorithms with these properties as A zr .

In the above definition, property 1 defines the domain of the mappings A k , and property 2 ensures that our algorithm does not 'cheat' and does not 'travel into the past': the time can only go forward (see [Tyurin and Richtárik, 2023][Section 4, Definition 4.1]). Property 3 is a standard assumption for zero-respecting algorithms [Arjevani et al., 2022] that is satisfied by virtually all algorithms, including Adam [Kingma and Ba, 2015], SGD , PAGE [Li et al., 2021] and Asynchronous SGD .

It remains to define an oracle class for our problem that employs oracles from (29). We design oracles that emulate the real behavior of the workers.

Definition 28 (Computation Oracle Class O τ 1 ,...,τ n ) . For any f ∈ F m δ 0 ,L + , the oracle class O τ 1 ,...,τ n returns oracles O i = O τ i , i ∈ [ n ] , where the mappings O τ i are defined in (29).

J.2 The 'worst case' function in the nonconvex world

The analysis uses a standard function, commonly employed to derive lower bounds in the nonconvex regime. First, let us define

̸

Our choice of the underlying function F follows the construction introduced in Carmon et al. [2020], Arjevani et al. [2022]: for any T ∈ N , define

where

Throughout the proof, we only rely on the following properties of the function:

Lemma 1 (Carmon et al. [2020], Arjevani et al. [2022]) . The function F T satisfies:

F T (0) -inf x ∈ R T F T ( x ) ≤ ∆ 0 T, where ∆ 0 = 12 .

The function F T is l 1 -smooth, where l 1 = 152 .

For all x ∈ R T , ∥∇ F T ( x ) ∥ ∞ ≤ γ ∞ , where γ ∞ = 23 .

For all x ∈ R T , prog ( ∇ F T ( x )) ≤ prog ( x ) + 1 .

For all x ∈ R T , if prog ( x ) < T, then ∥∇ F T ( x ) ∥ > 1 .

J.3 The first lower bound

We are ready to present the main results of this section.

Theorem 29. Let us consider Protocol 10. Without loss of generality, assume that 0 < τ 1 ≤ · · · ≤ τ n and take any L + , δ 0 , ε > 0 , and m ∈ N such that ε < c 1 L + δ 0 and δ 0 L + ε > c 2 √ m. Then, for any algorithm A ∈ A zr , there exists a function f ∈ F m δ 0 ,L + and computation oracles ( O 1 , . . . , O n ) ∈

The quantities c 1 ,c 2 , and c 3 are universal constants. The sequences x k and t k are defined in Protocol 10.

Proof. (Step 1: Construction of a hard problem)

We start our proof by constructing an appropriate function f ∈ F m δ 0 ,L + . Let us fix any T ≥ N and define f i : R T → R such that

for all x ∈ R T , and f i ( x ) = 0 for all i ∈ { 2 , . . . , m } and x ∈ R T . Essentially, all information about the function f = 1 m ∑ m i =1 f i is in the first function. Note that

Then, using Lemma 1, we have

Taking we ensure that

where in the inequalities we use Lemma 1 and the choice of T. Now, inequalities (32) and (33) imply that f = 1 m ∑ m i =1 f i ∈ F m δ 0 ,L + , and hence, using Lemma 1 again, we get

Let us take

Then and

The last inequality means that while prog ( x k ) < T for all k ∈ S t , all gradients are large. The function F T is a zero-chain : due to Lemma 1, we know that prog ( ∇ F T ( x )) ≤ prog ( x ) + 1 for all x ∈ R T . This implies that we can discover at most one new non-zero coordinate by calculating the gradient of the function ∇ f 1 . Since the algorithm A ∈ A zr is zero-respecting, by definition it cannot return a point x k with progress greater than that of the vectors returned by the oracles. In view of this, it is necessary to calculate the gradient of f 1 at least T times to get prog ( x k ) ≥ T.

The gradient of f 1 can be calculated if and only if j = 1 , where j ∼ Uniform ([ m ]) (see (29)). Consider worker i and define η 1 i to be the number of draws from Uniform ([ m ]) until the index j = 1 is sampled. Clearly, η 1 i is a Geometric random variable with parameter P ( j = 1) = 1 m . Recall that the workers can do the computations in parallel, and by the design of the oracles, worker i needs at least τ i η 1 i seconds to calculate η 1 i stochastic gradients. Hence, it is impossible to calculate ∇ f 1 before the time

Once the algorithm calculates ∇ f 1 for the first time, it needs to do so at least T -1 times more to achieve prog ( x k ) ≥ T. Thus, one should wait at least

seconds, where η k i i.i.d. ∼ Geometric ( 1 / m ) . We can conclude that

(Step 2: The Chernoff Method)

The theorem's proof is now reduced to the analysis of the concentration of ∑ T k =1 min i ∈ [ n ] τ i η k i . Using the Chernoff method, for all s > 0 , we have

Independence gives

Let us consider the term in the last bracket separately. For a fixed t ′ > 0 , we have

We now consider the last term. Due to the independence, we have

where we use the cumulative distribution function of a geometric random variable and temporarily define p := 1 m . Using Lemma 3, we get

Let us take

where g ( j ) := ( ∑ j i =1 1 τ i ) -1 ( 1 p + j ) for all j ∈ [ n ] and assume that j ∗ is the largest index such that min j ∈ [ n ] g ( j ) = g ( j ∗ ) . Then, Lemma 4 gives

where we let τ n +1 ≡ ∞ . Therefore, t ′ < τ j ∗ +1 and (39) gives

Using (40), we get t ′ τ i = min j ∈ [ n ] g ( j ) 8 τ i ≥ 1 8 for all i ≤ j ∗ . Since ⌊ x ⌋ ≤ 2 x -1 4 for all x ≥ 1 8 , we obtain

Substituting the last inequality to (38) gives

We now take s = 1 /t ′ to obtain

Substituting this inequality in (35) and (36) gives

Therefore,

for all

and ρ > 0 . Using the bound on the probability with ρ = 1 2 and (34), we finally conclude for all

It remains to use the conditions ε < c 1 Lδ 0 and δ 0 L + ε > c 2 √ m from the theorem to finish the proof.

J.4 The second lower bound

Theorem 30. Let us consider Protocol 10. Without loss of generality, assume that 0 < τ 1 ≤ · · · ≤ τ n and take any L + , δ 0 , ε > 0 , and m ∈ N such that ε < c 1 L + δ 0 . Then, for any algorithm A ∈ A zr , there exists a function f ∈ F m δ 0 ,L + and computation oracles ( O 1 , . . . , O n ) ∈ O τ 1 ,...,τ n ( f ) such that

The quantities c 1 and c 2 are universal constants. The sequences x k and t k are defined in Protocol 10.

Proof. We use the same construction as in the proof of Theorem 2 of Li et al. [2021]. Let us consider

for all x ∈ R T , where c ∈ R and v i ∈ R T , i ∈ [ n ] are parameters that we define later. Then

for all x, y ∈ R d and

where we take

Thus, we have f = 1 m ∑ m i =1 f i ∈ F m δ 0 ,L + . Now, let

and choose any T ∈ { sm | s ∈ N } (one can always take T = m ). Then

Let us fix some time t > 0 to be determined later and recall that the workers can calculate the stochastic gradients ∇ f i ( x ) = cv i + L + x in parallel. Suppose that up to time t , fewer than m 2 stochastic gradients have been computed. Then, since the algorithm is a zero-respecting algorithm, it cannot have discovered more than m 2 × T m = T 2 coordinates. Thus, at least T 2 coordinates are equal to 0 and

for all k ∈ { k ∈ N 0 | t k ≤ t } . Therefore, substituting our choice of c and using the assumptions from the theorem, we get

It remains to find the time t > 0 . The workers work in parallel, so in t seconds they calculate at most

stochastic gradients. Now, let

and define g ( j ) := ( ∑ j i =1 1 τ i ) -1 ( m + j ) for all j ∈ [ n ] . Assume that j ∗ is the largest index such that min j ∈ [ n ] g ( j ) = g ( j ∗ ) . Using Lemma 4, we have

Since t τ i ≥ 1 8 for all i ≤ j ∗ and ⌊ x ⌋ ≤ 2 x -1 4 for all x ≥ 1 8 , we obtain

Therefore, it is possible to calculate at most m 4 stochastic gradient in time (43) and we can finally conclude that inequality (42) holds for any time that is less than or equal (43).

K Useful Identities and Inequalities

Lemma 2 ([Szlendak et al., 2021]) . It holds that L -≤ L + , L -≤ 1 m ∑ m i =1 L i and L 2 + -L 2 -≤ L 2 ± ≤ L 2 + ≤ 1 m ∑ m i =1 L 2 i .

Lemma 3. Consider a sequence q 1 , . . . , q n ∈ [0 , 1] . Then

Proof. We prove the result by induction. It is clearly true for n = 1 : 1 -∑ 1 m =1 q m = ∏ 1 m =1 (1 -q m ) . Now, assume that that it holds for n -1 , meaning that

Multiplying both sides of the inequality by 1 -q n ∈ [0 , 1] gives

since q m ∈ [0 , 1] for all m ∈ [ n ]

Theorem 31. Let us consider the equilibrium time mapping from Definition 3. Then

for all S ≥ 0 , τ i ∈ [0 , ∞ ] for all i ∈ [ n ] , and τ 1 ≤ · · · ≤ τ n .

Remark 32. Assume that τ 1 = · · · = τ n -1 = τ and τ n →∞ , then

Thus, t ∗ ( S, [ τ i ] n i =1 ) can be arbitrarily smaller than 2 τ n max { S n , 1 } and 2 τ 1 max { S, 1 } . This implies that our new complexities (9) and (10) can be arbitrarily better than T Soviet PAGE and T Hero PAGE .

Lemma 4. Consider a sequence 0 < τ 1 ≤ . . . ≤ τ n and fix some S > 0 . For all j ∈ [ n ] , define

Let j ∗ max be the largest index such that min j ∈ [ n ] g ( j ) = g ( j ∗ max ) . For j ∗ max < n, we have

Let j ∗ be any index such that min j ∈ [ n ] g ( j ) = g ( j ∗ ) . For j ∗ < n, we have

and

Let j ∗ min be the smallest index such that min j ∈ [ n ] g ( j ) = g ( j ∗ min ) . Then

Let j ∗ be any index such that min j ∈ [ n ] g ( j ) = g ( j ∗ ) . Then

Proof. We first prove the first inequality. Suppose that j ∗ max < n . Then g ( j ∗ max ) < g ( j ∗ max +1) , meaning that

This implies the following series of inequalities:

The proof of the second inequality is the same, but with non-strict inequalities. To prove the third inequality, first suppose that j ∗ min = 1 . Then g ( j ∗ min ) = τ j ∗ min ( S +1) > τ j ∗ min as needed. On the other hand, j ∗ min > 1 implies

The proof of the fourth inequality is the same, but with non-strict inequalities.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Sections 4 and 5, Table 1

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Sections 3, 4.1, and 5

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Sections 1.1, 4, and the appendix

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section A

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: In the supplementary materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section A

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide two plots for each algorithm to ensure the soundness of our experiments. We also calculate the variance of accuracies in Table 2.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Section A

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have thoroughly reviewed the code of ethics and we can confidently state that our paper is fully compliant with it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our paper considers a mathematical topic.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Section A

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: In the supplementary materials.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this study, the authors developed a new distributed optimization algorithm, Freya Page method, for the non-convex finite-sum optimization problem. The authors provided the corresponding iteration and time complexity bounds. In addition, a lower bound on the time complexity is also provided.\n\nSTRENGTHS:\nThe paper is well-written and easy to follow. The developed Algorithms 2-3 are novel and lead to useful time complexity bounds. The results should be interesting to audiences in distributed optimization and machine learning fields. Due to the time limit, I did not check the proofs in the appendix. But I feel that the proofs should be correct, except that of Theorem 10, which I have a concern about.\n\nWEAKNESSES:\nI did not see major weaknesses. But I am a little confused by the lower bound in Theorem 10 and the claims after the theorem. I have also included a few comments for the authors' consideration.\n\nQUESTIONS:\n- I wonder if the results can be extended to the case when the computation time for each gradient $\\nabla f_j$ is different for different $j$. In other words, can we replace $\\tau_{i}$ in Line 31 with $\\tau_{i, j}$, which may be different for different $j$?\n\n- Line 118: It would be better if the authors can provide more details on the choice of inverse proportion to $\\tau_i$. I feel that this choice can guarantee that the worst-case running time is the same for all workers?\n\n- In Section 1, the authors claimed that Theorem 4 allows $\\tau_i = \\infty$ for all $i$. But I feel that the authors are referring to Theorem 5? Also, I think Theorem 5 (as well as Theorems 1-2) requires $\\tau_i$ to be finite for some $i$ so that $t^*$ is finite? It seems that this is supported by Example 3, where the current analysis is not able to provide an upper bound on the time complexity.\n\n- For the time complexity in Theorem 4, I think the authors are computing the *expected* time complexity. This information is not mentioned in the paper and I feel it might be better to add a formal definition of the expected time complexity in the paper.\n\n- It seems that the authors defined the notion of $t^*(S)$ in Definition 3 but did not used it in the following theorems.\n\n- Line 200: I am not sure why the monotony of the two terms enable the application of the bi-section method.\n\n- Line 294: I am a little confused when the authors mentioned that (10) is less than or equal to (12). It seems that (12) should be a lower bound on the complexity (up to a constant) and the achieved time complexity (10) should not be smaller than it?\n\nLIMITATIONS:\nSee my comments in the Questions section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Freya PAGE, a novel parallel optimization method designed for distributed systems where computational resources vary in capability and speed. This method addresses the challenge of stragglers in asynchronous environments by adopting a strategy that adaptively ignores slower computations, thereby significantly improving the time complexity over previous methods like Asynchronous SGD and PAGE. The paper claims that Freya PAGE achieves optimal time complexity under weaker assumptions and also presents a theoretical analysis that establishes a lower bound for the time complexity in such settings. The approach is particularly effective in large-scale scenarios where the number of data samples is greater than the square root of the number of workers, thus demonstrating the algorithm's suitability for practical large-scale machine learning tasks.\n\nSTRENGTHS:\n1. The problem tackled is relevant to current large-scale machine learning applications, making the findings applicable and valuable to both academics and practitioners working with distributed computing environments.\n2. Freya PAGE introduces a novel approach to handling heterogeneous and asynchronous computational environments in distributed systems. Its ability to adaptively ignore slower computations is a significant advancement over traditional methods.\n3. The paper demonstrates that Freya PAGE achieves an optimal time complexity, which is a significant improvement compared to existing methods like asynchronous SGD.\n\nWEAKNESSES:\n1. Determining the optimal parameters $S$ and $p$ as outlined in Theorem 7 requires knowledge of unknown $\\tau_i$ and solving an optimization problem, which can be challenging or even impractical.\n2. The paper lacks sufficient experimental results on more complex machine learning tasks, including comparision with cutting-edge distributed learning algorithms.\n3. The algorithm does not address the non-iid data senarios, workers may access data from varying distributions.\n\nQUESTIONS:\nHow do the workers operate asynchronously? And why do the authors assert that Freya PAGE combines synchrony and asynchrony?\n\nLIMITATIONS:\nThe authors have partially addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Freya PAGE, a new parallel method for large-scale nonconvex finite-sum optimization in heterogeneous and asynchronous computational environments. Freya PAGE, specifically addresses the variability in processing times across different workers due to hardware and network differences. By being robust to \"stragglers\" and adaptively ignoring slow computations, Freya PAGE  improves time complexity guarantees compared to existing methods. Additionally, the paper proves a lower bound for smooth nonconvex finite-sum problems in asynchronous setups.\n\nSTRENGTHS:\nFreya PAGE is a novel parallel optimization method specifically designed to handle heterogeneous and asynchronous computations. This appears to be a significant advancement and addresses real-world challenges in distributed systems. By addressing the variability in processing times across different workers due to hardware and network differences, the method is highly relevant to practical distributed systems used in large-scale machine learning tasks. This method shows improved time complexity guarantees compared to existing methods. The paper also provides a strong theoretical framework in general. Specifically, proving lower bound for smooth nonconvex finite-sum problems in asynchronous setups shows the optimality of Freya PAGE in large-scale regimes. The stochastic gradient collection strategies ComputeGradient and ComputeBatchDifference introduced in this paper is innovative. Overall, this is a very well written paper and easy to follow and understand.\n\nWEAKNESSES:\nThe empirical results of this paper are primarily focused on synthetic quadratic optimization tasks and logistic regression problems. More diverse and extensive experimentation on a wider range of real-world datasets and applications would strengthen the practical validation. \n\nThe algorithm's performance might vary significantly for different configurations of worker speeds and hardware setups. The variance in performance could be an issue in highly dynamic environments.\n\nQUESTIONS:\nIf you have tested Freya PAGE in real-world scenarios, can you share the results and insights?\nHow does Freya PAGE specifically determine which computations to ignore or adaptively manage in the presence of stragglers? Can this mechanism be fine-tuned for different distributed system configurations?\n\nLIMITATIONS:\nThe performance of Freya PAGE might vary significantly with different configurations of worker speeds and hardware setups. This variance could be problematic in environments with highly variable computational resources.\nThe impact of communication overhead in distributed systems with significant network latency, is not extensively discussed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of minimizing a finite sum of non-convex objective terms with multiple computational units that calculate gradient oracles. It introduces two efficient computational subroutines for the main steps of the PAGE algorithm, and provides novel bounds on the real time of computation assuming a heterogeneous computational scenario. For this, the paper gives bounds on the execution time of these subroutines and combines them with the previously known iteration complexity of PAGE. Next, the paper considers multiple variations: it optimizes the hyperparameters of the PAGE algorithm in different regimes, and also extends the results to a dynamic scenario. Finally, it provides a lower bound on the overall execution time and in this way establishes that their strategy is optimal in large-scale regimes.\n\nSTRENGTHS:\nThe paper is well written and has an excellent presentation of the subject. It provides an extensive theoretical study of a novel gradient management strategy in the PAGE algorithms, and achieves tight bounds on the convergence rate.\n\nWEAKNESSES:\nI do not have any major concern about this work. The only limitation for me is that from a practical standpoint, this algorithm might not be optimal as it may introduce a significantly higher communication overhead. This is because each gradient is now required to be sent in an individual message.\n\nQUESTIONS:\nCompared to the results in the appendix, the stated theorems are generally simplified for better comprehension, but some confusion is also introduced. In particular, I am not sure which results are deterministic and which results are in terms of statistical expectation or high-probability. Could you clarify? \n\nIn the statement of Theorem 4, you assume two bounds on m, where one of them seem to be stronger than the other. Why do you keep both?\n\nLIMITATIONS:\nNo limitations or potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}