Title: A Computation and Communication Efficient Method for Distributed Nonconvex Problems in the Partial Participation Setting
Venue: NeurIPS 2023
Accepted: yes
Keywords: Nonconvex Optimization

Partial Participation

Variance Reduction

Compressed Communication

Distributed Optimization
Abstract: We present a new method that includes three key components of distributed optimization and federated learning: variance reduction of stochastic gradients, partial participation, and compressed communication. We prove that the new method has optimal oracle complexity and state-of-the-art communication complexity in the partial participation setting. Regardless of the communication compression feature, our method successfully combines variance reduction and partial participation: we get the optimal oracle complexity, never need the participation of all nodes, and do not require the bounded gradients (dissimilarity) assumption.

A Computation and Communication Efficient Method for Distributed Nonconvex Problems in the Partial Participation Setting

Abstract

We present a new method that includes three key components of distributed optimization and federated learning: variance reduction of stochastic gradients, partial participation, and compressed communication. We prove that the new method has optimal oracle complexity and state-of-the-art communication complexity in the partial participation setting. Regardless of the communication compression feature, our method successfully combines variance reduction and partial participation: we get the optimal oracle complexity, never need the participation of all nodes, and do not require the bounded gradients (dissimilarity) assumption.

1 Introduction

Federated and distributed learning have become very popular in recent years (Koneˇ cn´ y et al., 2016; McMahan et al., 2017). The current optimization tasks require much computational resources and machines. Such requirements emerge in machine learning, where massive datasets and computations are distributed between cluster nodes (Lin et al., 2017; Ramesh et al., 2021). In federated learning, nodes, represented by mobile phones, laptops, and desktops, do not send their data to a server due to privacy and their huge number (Ramaswamy et al., 2019), and the server remotely orchestrates the nodes and communicates with them to solve an optimization problem.

As in classical optimization tasks, one of the main current challenges is to find computationally efficient optimization algorithms. However, the nature of distributed problems induces many other (Kairouz et al., 2021), including i) partial participation of nodes in algorithm steps: due to stragglers (Li et al., 2020) or communication delays (Vogels et al., 2021), ii) communication bottleneck : even if a node participates, it can be costly to transmit information to a server or other nodes (Alistarh et al., 2017; Ramesh et al., 2021; Kairouz et al., 2021; Sapio et al., 2019; Narayanan et al., 2019). It is necessary to develop a method that considers these problems.

2 Optimization Problem

Let us consider the nonconvex distributed optimization problem

where f i : R d → R is a smooth nonconvex function for all i ∈ [ n ] := { 1 , . . . , n } . The full information about function f i is stored on i th node. The communication between nodes is maintained in the parameters server fashion (Kairouz et al., 2021): we have a server that receives compressed

information from nodes, updates a state, and broadcasts an updated model. 1 Since we work in the nonconvex world, our goal is to find an ε -solution ( ε -stationary point) of (1): a (possibly random) point ̂ x ∈ R d , such that E [ ∥∇ f ( ̂ x ) ∥ 2 ] ≤ ε.

We consider three settings:

1. Gradient Setting. The i th node has only access to the gradient ∇ f i : R d → R d of function f i . Moreover, the following assumptions for the functions f i hold.

Assumption 1. There exists f ∗ ∈ R such that f ( x ) ≥ f ∗ for all x ∈ R .

Assumption 2. The function f is L -smooth, i.e., ∥∇ f ( x ) -∇ f ( y ) ∥ ≤ L ∥ x -y ∥ for all x, y ∈ R d .

Assumption 3. The functions f i are L i -smooth for all i ∈ [ n ] . Let us define ̂ L 2 := 1 n ∑ n i =1 L 2 i . 2

Finite-Sum Setting. The functions { f i } n i =1 have the finite-sum form

where f ij : R d → R is a smooth nonconvex function for all j ∈ [ m ] .

We assume that Assumptions 1, 2 and 3 hold and the following assumption.

Assumption 4. The function f ij is L ij -smooth for all i ∈ [ n ] , j ∈ [ m ] . Let L max := max i ∈ [ n ] ,j ∈ [ m ] L ij .

3. Stochastic Setting. The function f i is an expectation of a stochastic function,

where f i : R d × Ω ξ → R . For a fixed x ∈ R , f i ( x ; ξ ) is a random variable over some distribution D i , and, for a fixed ξ ∈ Ω ξ , f i ( x ; ξ ) is a smooth nonconvex function. The i th node has only access to a stochastic gradients ∇ f i ( · ; ξ ij ) of the function f i through the distribution D i , where ξ ij is a sample from D i . We assume that Assumptions 1, 2 and 3 hold and the following assumptions.

Assumption 5. For all i ∈ [ n ] and for all x ∈ R d , the stochastic gradient ∇ f i ( x ; ξ ) is unbiased and has bounded variance, i.e., E ξ [ ∇ f i ( x ; ξ )] = ∇ f i ( x ) , and E ξ [ ∥∇ f i ( x ; ξ ) -∇ f i ( x ) ∥ 2 ] ≤ σ 2 , where σ 2 ≥ 0 .

Assumption 6. For all i ∈ [ n ] and for all x, y ∈ R , the stochastic gradient ∇ f i ( x ; ξ ) satisfies the mean-squared smoothness property, i.e., E ξ [ ∥∇ f i ( x ; ξ ) -∇ f i ( y ; ξ ) ∥ 2 ] ≤ L 2 σ ∥ x -y ∥ 2 .

We compare algorithms using the oracle complexity , i.e., the number of (stochastic) gradients that each node has to calculate to get ε -solution, and the communication complexity , i.e., the number of bits that each node has to send to the server to get ε -solution.

2.1 Unbiased Compressors

We use the concept of unbiased compressors to alleviate the communication bottleneck. The unbiased compressors quantize and/or sparsify vectors that the nodes send to the server.

Definition 1. A stochastic mapping C : R d → R d is an unbiased compressor if there exists ω ∈ R

We denote a set of stochastic mappings that satisfy Definition 1 as U ( ω ) . In our methods, the nodes make use of unbiased compressors {C i } n i =1 . The community developed a large number of unbiassed compressors, including Rand K (see Definition 5) (Beznosikov et al., 2020; Stich et al., 2018), Adaptive sparsification (Wangni et al., 2018) and Natural compression and dithering (Horv´ ath et al., 2019a). We are aware of correlated compressors by Szlendak et al. (2021) and quantizers by Suresh et al. (2022) that help in the homogeneous regimes, but in this work, we are mainly concentrated on generic heterogeneous regimes, though, for simplicity, assume the independence of the compressors.

Assumption 7. C i ∈ U ( ω ) for all i ∈ [ n ] , and the compressors are statistically independent.

2.2 Nodes Partial Participation Assumptions

We now try to formalize the notion of partial participation. Let us assume that we have n events { i th node is participating } with the following properties.

Assumption 8. The partial participation of nodes has the following distribution: exists constants p a ∈ (0 , 1] and p aa ∈ [0 , 1] , such that

̸

Prob ( i th and j th nodes are participating ) = p aa ∀ i = j ∈ [ n ] .

and these events from different communication rounds are independent.

We are not fighting for the full generality and believe that more complex sampling strategies can be considered in the analysis. For simplicity, we settle upon Assumption 8. Standard partial participation strategies, including s -nice sampling, where the server chooses uniformly s nodes without replacement ( p a = s / n and p aa = s ( s -1) / n ( n -1) ), and independent participation, where each

: see Table 1.

(a), (b)

node independently participates with probability p a (due to independence, we have p aa = p 2 a ), satisfy Assumption 8. In the literature, s -nice sampling is one of the most popular strategies (Zhao et al., 2021a; Richt´ arik et al., 2021; Reddi et al., 2020; Koneˇ cn´ y et al., 2016).

3 Motivation and Related Work

The main goal of our paper is to develop a method for the nonconvex distributed optimization that will include three key features: variance reduction of stochastic gradients, compressed communication, and partial participation. We now provide an overview of the literature (see also Table 1 and Table 2).

1. Variance reduction of stochastic gradients

It is important to consider finite-sum (2) and stochastic (3) settings because, in machine learning tasks, either the number of local functions m is huge or the functions f i is an expectation of a stochastic function due to the batch normalization (Ioffe and Szegedy, 2015) or random augmentation (Goodfellow et al., 2016), and it is infeasible to calculate the full gradients analytically. Let us recall the results from the nondistributed optimization. In the gradient setting, the optimal oracle complexity is O ( 1 / ε ) , achieved by the vanilla gradient descent ( GD ) (Carmon et al., 2020; Nesterov, 2018). In the finite-sum setting and stochastic settings, the optimal oracle complexities are O ( m + √ m ε ) and O ( σ 2 ε + σ ε 3 / 2 ) (Fang et al., 2018; Li et al., 2021a; Arjevani et al., 2019), accordingly, achieved by methods SPIDER , SARAH , PAGE , and STORM from (Fang et al., 2018; Nguyen et al., 2017; Li et al., 2021a; Cutkosky and Orabona, 2019).

2. Compressed communication

In distributed optimization (Ramesh et al., 2021; Xu et al., 2021), lossy communication compression can be a powerful tool to increase the communication speed between the nodes and the server. Different types of compressors are considered in the literature, including unbiased compressors (Alistarh et al., 2017; Beznosikov et al., 2020; Szlendak et al., 2021), contractive (biased) compressors (Richt´ arik et al., 2021), 3PC compressors (Richt´ arik et al., 2022). We will focus on unbiased compressors because methods DASHA and MARINA (Tyurin and Richt´ arik, 2023; Szlendak et al., 2021; Gorbunov et al., 2021) that employ unbiased compressors provide the current theoretical state-of-the-art (SOTA) communication complexities.

Many methods analyzed optimization methods with the unbiased compressors (Alistarh et al., 2017; Mishchenko et al., 2019; Horv´ ath et al., 2019b; Gorbunov et al., 2021; Tyurin and Richt´ arik, 2023). In the gradient setting, the methods MARINA and DASHA by Gorbunov et al. (2021) and Tyurin and Richt´ arik (2023) establish the current SOTA communication complexity, each method needs 1+ ω / √ n ε communication rounds to get an ε -solution. In the finite-sum and stochastic settings, the current

SOTA communication complexity is attained by the DASHA method, while maintaining the optimal oracle complexities O ( m + √ m ε √ n ) and O ( σ 2 εn + σ ε 3 / 2 n ) per node.

3. Partial participation

From the beginning of federated learning era, the partial participation has been considered to be the essential feature of distributed optimization methods (McMahan et al., 2017; Koneˇ cn´ y et al., 2016; Kairouz et al., 2021). However, previously proposed methods have limitations: i) methods MARINA and FedPAGE from (Gorbunov et al., 2021; Zhao et al., 2021b) still require synchronization of all nodes with a small probability. ii) in the stochastic settings, methods FedAvg , SCAFFOLD , and FRECON with the partial participation mechanism (McMahan et al., 2017; Karimireddy et al., 2020b; Zhao et al., 2021a) provide results without variance reduction techniques from (Fang et al., 2018; Li et al., 2021a; Cutkosky and Orabona, 2019) and, therefore, get suboptimal oracle complexities. Note that FRECON and DASHA reduce the variance only from compressors (in the partial participation and stochastic setting). iii) in the finite-sum setting, the ZeroSARAH method by Li et al. (2021b) focuses on the homogeneous regime only (the functions f i are equal). iv) The MIME method by Karimireddy et al. (2020a) and the CE-LSGD method (for Partial Participation) by the concurrent paper (Patel et al., 2022) consider the online version of the problem (1). Therefore, MIME and CE-LSGD (for Partial Participation) require stricter assumptions, including the bounded inter-client gradient variance assumption. In the finite-sum setting (2), MIME and CE-LSGD obtain a suboptimal oracle complexity O ( 1 / ε 3 / 2 ) while, in the full participation setting, it is possible to get the complexity O ( 1 / ε ) .

4 Contributions

We propose a new method DASHA-PP for the nonconvex distributed optimization.

As far as we know, this is the first method that includes three key ingredients of federated learning methods: variance reduction of stochastic gradients, compressed communication, and partial participation.

Moreover, this is the first method that combines variance reduction of stochastic gradients and partial participation flawlessly: i) it gets the optimal oracle complexity ii) does not require the participation of all nodes iii) does not require the bounded gradients assumption of the functions f i .

We prove convergence rates and show that this method has the optimal oracle complexity and the state-of-the-art communication complexity in the partial participation setting. Moreover, in our work, we observe a nontrivial side-effect from mixing the variance reduction of stochastic gradients and partial participation. It is a general problem not related to our methods or analysis that we discuss in Section C.

In Section A, we present experiments where we validate our theory and compare our new methods to previous ones.

5 Algorithm Description and Main Challenges Towards Partial Participation

We now present DASHA-PP (see Algorithm 1), a family of methods to solve the optimization problem (1). When we started investigating the problem, we took DASHA as a baseline method for two reasons: the family of algorithms DASHA provides the current state-of-the-art communication complexities in the non-partial participation setting, and, unlike MARINA , it does not send non-compressed gradients and does not synchronize all nodes. Let us briefly discuss the main idea of DASHA , its problem in the partial participation setting, and why the refinement of DASHA is not an exercise.

In fact, the original DASHA method supports the partial participation of nodes in the gradient setting . Since the nodes only do the following steps (see full algorithm in Algorithm 6):

Algorithm 1 DASHA-PP

Algorithm 2 Calculate k t +1 i for DASHA-PP in the gradient setting. See line 9 in Alg. 1

Algorithm 3 Calculate k t +1 i for DASHA-PP-PAGE in the finite-sum setting. See line 9 in Alg. 1

1: Generate a random set I t i of size B from [ m ] with replacement

Algorithm 4 Calc. k t +1 i for DASHA-PP-FINITE-MVR in the finite-sum setting. See line 9 in Alg. 1

1: Generate a random set I t i of size B from [ m ] without replacement

4: k t +1 i = 1 m ∑ m j =1 k t +1 ij

Algorithm 5 Calculate k t +1 i for DASHA-PP-MVR in the stochastic setting. See line 9 in Alg. 1

1: Generate i.i.d. samples { ξ t +1 ij } B j =1 of size B from D i .

The partial participation mechanism (independent participation from Section 2.2) can be easily implemented here if we temporally redefine the compressor and use another one 3 instead:

With probability 1 -p a , a node does not update g t i and does not send anything to the server. The main observation is that we can do this trick since g t +1 i depends only on the vectors x t +1 , x t , and g t i . The points x t +1 and x t are only available in a node only during its participation.

However, we focus our attention on partial participation in the finite-sum and stochastic settings . Consider the nodes' steps in DASHA-MVR (Tyurin and Richt´ arik, 2023) (see Algorithm 7) that is designed for the stochastic setting:

Now we have two sequences h t i and g t i . Even if we use the same trick for (8), we still have to update (7) in every iteration of the algorithm since g t +1 i additionally depends on h t +1 i and h t i . In other words, if a node does not update g t i and does not send anything to the server, it still has to update h t i , what is impossible without the points x t +1 and x t . One of the main challenges was to 'guess' how to generalize (7) and (8) to the partial participation setting. We now provide a solution ( DASHA-PP-MVR with the batch size B = 1 ):

Now both control variables g t i and h t i do not change with the probability 1 -p a . When the i th node participates, the update rules of g t +1 i and h t +1 i in (9) were adapted to make the proof work. When p a = 1 (no partial participation), the update rules from (9) reduce to (7) and (8).

The theoretical analysis of the new algorithm became more complicated: unlike (7) and (8), the control variables h t +1 i and g t +1 i in (9) (see also main Algorithm 1) are coupled by the randomness from the partial participation. Going deeper into details, for instance, one can compare Lemma I.2 from (Tyurin and Richt´ arik, 2023) and Lemma 5, which both bound ∥ ∥ g t +1 i -h t +1 i ∥ ∥ 2 . The former lemma does not use the knowledge about the update rules of h t +1 i , works with one expectation E C [ · ] , uses only (4), (15), and (16). The latter lemma additionally requires and uses the structure of the update rule of h t +1 i (the structure is very important in the lemma since the control variables h t +1 i and g t +1 i are coupled), surgically copes with the expectations E C [ · ] and E p a [ · ] (for instance, it is not trivial in each order one should apply the expectations), and uses the sampling lemma (Lemma 1). The same reasoning applies to other parts of the analysis and the finite-sum setting: the generalization of the previous algorithm and the additional randomness from the partial participation required us to rethink the previous proofs.

At the first reading of the proofs, we suggest the reader follow the proof of Theorem 2 in the gradient setting ( DASHA-PP ), which takes a small part of the paper. Although the appendix seems to be dense and large, the size is justified by the fact that we consider four different sub-algorithms, DASHA-PP , DASHA-PP-PAGE , DASHA-PP-FINITE-MVR , and DASHA-PP-MVR , and also PŁ-condition (The theory is designed so that the proofs do not repeat steps of each other and use one framework).

6 Theorems

We now present the convergence rates theorems of DASHA-PP in different settings. We will compare the theorems with the results of the current state-of-the-art methods, MARINA and DASHA , that work

in the full participation setting. Suppose that MARINA or DASHA converges to ε -solution after T communication rounds. Then, ideally, we would expect the convergence of the new algorithms to ε -solution after up to T / p a communication rounds due to the partial participation constraints 4 . The detailed analysis of the algorithms under Polyak-Łojasiewicz condition we provide in Section F. Let us define ∆ 0 := f ( x 0 ) -f ∗ .

6.1 Gradient Setting

Theorem 2. Suppose that Assumptions 1, 2, 3, 7 and 8 hold. Let us take a = p a 2 ω +1 , b = p a 2 -p a ,

and g 0 i = h 0 i = ∇ f i ( x 0 ) for all i ∈ [ n ] in Algorithm 1 (DASHA-PP) , then E [ ∥ ∥ ∇ f ( ̂ x T ) ∥ ∥ 2 ] ≤ 2∆ 0 γT .

Let us recall the convergence rate of MARINA or DASHA , the number of communication rounds to get ε -solution equals O ( ∆ 0 ε [ L + ω √ n ̂ L ]) , while the rate of DASHA-PP equals O ( ∆ 0 ε [ L + ω +1 p a √ n ̂ L ]) . Up to Lipschitz constants factors, we get the degeneration up to 1 / p a factor due to the partial participation. This is the expected result since each worker sends useful information only with the probability p a .

6.2 Finite-Sum Setting

Theorem 3. Suppose that Assumptions 1, 2, 3, 4, 7, and 8 hold. Let us take a = p a 2 ω +1 , b = p page p a 2 -p a , probability p page ∈ (0 , 1] ,

and g 0 i = h 0 i = ∇ f i ( x 0 ) for all i ∈ [ n ] in Algorithm 1 (DASHA-PP-PAGE) then E [ ∥ ∥ ∇ f ( ̂ x T ) ∥ ∥ 2 ] ≤ 2∆ 0 γT .

We now choose p page to balance heavy full gradient and light mini-batch calculations. Let us define p a := √ 1 -p aa p a ∈ [0 , 1] . Note that if p a = 1 then p aa = 1 and p a = 0 .

✶ ✶ Corollary 1. Let the assumptions from Theorem 3 hold and p page = B / ( m + B ) . Then DASHA-PP-PAGE needs

communication rounds to get an ε -solution and the expected number of gradient calculations per node equals O ( m + BT ) .

The convergence rate the rate of the current state-of-the-art method DASHA-PAGE without partial participation equals O ( ∆ 0 ε [ L + ω √ n ( ̂ L + L max √ B ) + √ m n L max B ]) . Let us closer compare it with (10). As expected, we see that the second term w.r.t. ω degenerates up to 1 / p a . Surprisingly, the third term w.r.t. √ m / n can degenerate up to √ B / p a when ̂ L ≈ L max . Hence, in order to keep degeneration up to 1 / p a , one should take the batch size B = O ( L 2 max / ̂ L 2 ) . This interesting effect we analyze separately in Section C. The fact that the degeneration is up to 1 / p a we check numerically in Section A.

In the following corollary, we consider Rand K compressors 5 (see Definition 5) and show that with the particular choice of parameters, up to the Lipschitz constants factors, DASHA-PP-PAGE gets the

optimal oracle complexity and SOTA communication complexity. Indeed, comparing the following result with (Tyurin and Richt´ arik, 2023, Corollary 6.6), one can see that we get the degeneration up to 1 / p a factor, which is expected in the partial participation setting. Note that the complexities improve with the number of workers n.

Corollary 2. Suppose that assumptions of Corollary 1 hold, B ≤ min { 1 p a √ m n , L 2 max ✶ 2 p a ̂ L 2 } 6 , and we use the unbiased compressor Rand K with K = Θ( Bd / √ m ) . Then the communication complexity of Algorithm 1 is O ( d + L max ∆ 0 d p a ε √ n ) , and the expected number of gradient calculations per node equals O ( m + L max ∆ 0 √ m p a ε √ n ) .

The convergence rate of DASHA-PP-FINITE-MVR is provided in Section E.5.

6.3 Stochastic Setting

We define h t := 1 n ∑ n i =1 h t i .

Theorem 4. Suppose that Assumptions 1, 2, 3, 5, 6, 7 and 8 hold. Let us take a = p a 2 ω +1 , b ∈ ( 0 , p a 2 -p a ] , γ ≤ ( L + [ 48 ω (2 ω +1) np 2 a ( ̂ L 2 + (1 -b ) 2 L 2 σ B ) + 12 np a b ( ( 1 -p aa p a ) ̂ L 2 + (1 -b ) 2 L 2 σ B )] 1 / 2 ) -1 , and g 0 i = h 0 i for all i ∈ [ n ] in Algorithm 1 (DASHA-PP-MVR) . Then

0

In the next corollary, we choose momentum b and initialize vectors h i to get ε -solution. Corollary 3. Suppose that assumptions from Theorem 4 hold, momentum b = Θ ( min { p a ω √ nεB σ 2 , p a nεB σ 2 }) , σ 2 nεB ≥ 1 , and h 0 i = 1 B init ∑ B init k =1 ∇ f i ( x 0 ; ξ 0 ik ) for all i ∈ [ n ] , and batch size B init = Θ ( √ p a B b ) , then Algorithm 1 (DASHA-PP-MVR) needs

communication rounds to get an ε -solution and the number of stochastic gradient calculations per node equals O ( B init + BT ) .

The convergence rate of the DASHA-SYNC-MVR , the state-of-the-art method without partial participation, equals O ( ∆ 0 ε [ L + ω √ n ( ̂ L + L σ √ B ) + σ √ εn L σ B ] + σ 2 nεB ) . Similar to Section 6.2, we see that in the regimes when ̂ L ≈ L σ the third term w.r.t. 1 / ε 3 / 2 can degenerate up to √ B / p a . However, if we take B = O ( L 2 σ / ̂ L 2 ) , then the degeneration of the third term will be up to 1 / p a . This effect we analyze in Section C. The fact that the degeneration is up to 1 / p a we check numerically in Section A.

In the following corollary, we consider Rand K compressors (see Definition 5) and show that with the particular choice of parameters, up to the Lipschitz constants factors, DASHA-PP-MVR gets the optimal oracle complexity and SOTA communication complexity of DASHA-SYNC-MVR method. Indeed, comparing the following result with (Tyurin and Richt´ arik, 2023, Corollary 6.9), one can see that we get the degeneration up to 1 / p a factor, which is expected in the partial participation setting. Note that the complexities improve with the number of workers n.

Corollary 4. Suppose that assumptions of Corollary 3 hold, batch size B ≤ min { σ p a √ εn , L 2 σ 2 p a ̂ L 2 } ,

✶ we take Rand K compressors with K = Θ ( Bd √ εn σ ) . Then the communication complexity equals

O ( dσ √ p a √ nε + L σ ∆ 0 d p a √ nε ) , and the expected number of stochastic gradient calculations per node equals O ( σ 2 √ p a nε + L σ ∆ 0 σ p a ε 3 / 2 n ) .

We are aware that the initial batch size B init can be suboptimal w.r.t. ω in DASHA-PP-MVR in some regimes (see also (Tyurin and Richt´ arik, 2023)). This is a side effect of mixing the variance reduction of stochastic gradients and compression. However, Corollary 4 reveals that we can escape these regimes by choosing the parameter K of Rand K compressors in a particular way. To get the complete picture, we analyze the same phenomenon under PŁ condition (see Section F) and provide a new method DASHA-PP-SYNC-MVR (see Section G).

References

Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic, M. (2017). QSGD: Communicationefficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems (NIPS) , pages 1709-1720.

Arjevani, Y., Carmon, Y., Duchi, J. C., Foster, D. J., Srebro, N., and Woodworth, B. (2019). Lower bounds for non-convex stochastic optimization. arXiv preprint arXiv:1912.02365 .

Beznosikov, A., Horv´ ath, S., Richt´ arik, P., and Safaryan, M. (2020). On biased compression for distributed learning. arXiv preprint arXiv:2002.12410 .

Carmon, Y., Duchi, J. C., Hinder, O., and Sidford, A. (2020). Lower bounds for finding stationary points i. Mathematical Programming , 184(1):71-120.

Chang, C.-C. and Lin, C.-J. (2011). LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) , 2(3):1-27.

Cutkosky, A. and Orabona, F. (2019). Momentum-based variance reduction in non-convex SGD. arXiv preprint arXiv:1905.10018 .

Fang, C., Li, C. J., Lin, Z., and Zhang, T. (2018). SPIDER: Near-optimal non-convex optimization via stochastic path integrated differential estimator. In NeurIPS Information Processing Systems .

Goodfellow, I., Bengio, Y., Courville, A., and Bengio, Y. (2016). Deep learning , volume 1. MIT Press.

Gorbunov, E., Burlachenko, K., Li, Z., and Richt´ arik, P. (2021). MARINA: Faster non-convex distributed learning with compression. In 38th International Conference on Machine Learning .

Horv´ ath, S., Ho, C.-Y., Horvath, L., Sahu, A. N., Canini, M., and Richt´ arik, P. (2019a). Natural compression for distributed deep learning. arXiv preprint arXiv:1905.10988 .

Horv´ ath, S., Kovalev, D., Mishchenko, K., Stich, S., and Richt´ arik, P. (2019b). Stochastic distributed learning with gradient quantization and variance reduction. arXiv preprint arXiv:1904.05115 .

Ioffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning , pages 448456. PMLR.

Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. (2021). Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210.

Karimireddy, S. P., Jaggi, M., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., and Suresh, A. T. (2020a). Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint arXiv:2008.03606 .

Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. (2020b). Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning , pages 5132-5143. PMLR.

Koneˇ cn´ y, J., McMahan, H. B., Yu, F. X., Richt´ arik, P., Suresh, A. T., and Bacon, D. (2016). Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 .

Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. (2020). Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems , 2:429450.

Li, Z., Bao, H., Zhang, X., and Richt´ arik, P. (2021a). PAGE: A simple and optimal probabilistic gradient estimator for nonconvex optimization. In International Conference on Machine Learning , pages 6286-6295. PMLR.

Li, Z., Hanzely, S., and Richt´ arik, P. (2021b). ZeroSARAH: Efficient nonconvex finite-sum optimization with zero full gradient computation. arXiv preprint arXiv:2103.01447 .

Lin, Y., Han, S., Mao, H., Wang, Y., and Dally, W. J. (2017). Deep gradient compression: Reducing the communication bandwidth for distributed training. arXiv preprint arXiv:1712.01887 .

McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communicationefficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR.

Mishchenko, K., Gorbunov, E., Tak´ aˇ c, M., and Richt´ arik, P. (2019). Distributed learning with compressed gradient differences. arXiv preprint arXiv:1901.09269 .

Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G. R., Gibbons, P. B., and Zaharia, M. (2019). PipeDream: generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles , pages 1-15.

Nesterov, Y. (2018). Lectures on convex optimization , volume 137. Springer.

Nguyen, L., Liu, J., Scheinberg, K., and Tak´ aˇ c, M. (2017). SARAH: A novel method for machine learning problems using stochastic recursive gradient. In The 34th International Conference on Machine Learning .

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019). Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS) .

Patel, K. K., Wang, L., Woodworth, B., Bullins, B., and Srebro, N. (2022). Towards optimal communication complexity in distributed non-convex optimization. In Advances in Neural Information Processing Systems .

Ramaswamy, S., Mathews, R., Rao, K., and Beaufays, F. (2019). Federated learning for emoji prediction in a mobile keyboard. arXiv preprint arXiv:1906.04329 .

Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021). Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092 .

Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Koneˇ cn` y, J., Kumar, S., and McMahan, H. B. (2020). Adaptive federated optimization. arXiv preprint arXiv:2003.00295 .

Richt´ arik, P., Sokolov, I., and Fatkhullin, I. (2021). EF21: A new, simpler, theoretically better, and practically faster error feedback. In Neural Information Processing Systems, 2021.

Richt´ arik, P., Sokolov, I., Fatkhullin, I., Gasanov, E., Li, Z., and Gorbunov, E. (2022). 3PC: Three point compressors for communication-efficient distributed training and a better theory for lazy aggregation. arXiv preprint arXiv:2202.00998 .

Sapio, A., Canini, M., Ho, C.-Y., Nelson, J., Kalnis, P., Kim, C., Krishnamurthy, A., Moshref, M., Ports, D. R., and Richt´ arik, P. (2019). Scaling distributed machine learning with in-network aggregation. arXiv preprint arXiv:1903.06701 .

Stich, S. U., Cordonnier, J.-B., and Jaggi, M. (2018). Sparsified SGD with memory. Advances in Neural Information Processing Systems , 31.

Suresh, A. T., Sun, Z., Ro, J. H., and Yu, F. (2022). Correlated quantization for distributed mean estimation and optimization. arXiv preprint arXiv:2203.04925 .

Szlendak, R., Tyurin, A., and Richt´ arik, P. (2021). Permutation compressors for provably faster distributed nonconvex optimization. arXiv preprint arXiv:2110.03300 .

Tyurin, A. and Richt´ arik, P. (2023). DASHA: Distributed nonconvex optimization with communication compression and optimal oracle complexity. International Conference on Learning Representations (ICLR) .

Vogels, T., He, L., Koloskova, A., Karimireddy, S. P., Lin, T., Stich, S. U., and Jaggi, M. (2021). RelaySum for decentralized deep learning on heterogeneous data. Advances in Neural Information Processing Systems , 34.

Wangni, J., Wang, J., Liu, J., and Zhang, T. (2018). Gradient sparsification for communicationefficient distributed optimization. Advances in Neural Information Processing Systems , 31.

Xu, H., Ho, C.-Y., Abdelmoniem, A. M., Dutta, A., Bergou, E. H., Karatsenidis, K., Canini, M., and Kalnis, P. (2021). Grace: A compressed communication framework for distributed machine learning. In 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS) , pages 561-572. IEEE.

Zhao, H., Burlachenko, K., Li, Z., and Richt´ arik, P. (2021a). Faster rates for compressed federated learning with client-variance reduction. arXiv preprint arXiv:2112.13097 .

Zhao, H., Li, Z., and Richt´ arik, P. (2021b). FedPAGE: A fast local stochastic gradient method for communication-efficient federated learning. arXiv preprint arXiv:2108.04755 .

Contents

A Numerical Verification of Theoretical Dependencies

10 2

10 3

10 4

10 5

10

9

10

8

10

7

10

6

10

5

10

4

10

3

10

2

||

f

(

x k

)||

2

K

= 500

Number of nodes: 100

DASHA-PP-PAGE (

s

=1): Step size: 0.0625

DASHA-PP-PAGE (

s

=10): Step size: 0.5

DASHA-PP-PAGE (

s

=50): Step size: 2.0

DASHA-PP-PAGE (

s

=90): Step size: 8.0

DASHA-PP-PAGE (

s

=100): Step size: 8.0

DASHA-PAGE: Step size: 8.0

10 3

10 4

10 5

10 6

# communication rounds

10

6

10

5

10

4

10

3

10

2

||

f

(

x k

)||

2

K

= 200

2

n

= 10000

DASHA-PP-MVR (

s

= 1): Step size: 0.00390625

DASHA-PP-MVR (

s

=10): Step size: 0.03125

DASHA-PP-MVR (

s

=50): Step size: 0.25

DASHA-PP-MVR (

s

=100): Step size: 0.25

DASHA-SYNC-MVR: Step size: 0.5

# communication rounds

(a) Finite-sum setting, K = 500 in Rand K .

(b) Stochastic setting, σ 2 / nεB = 10000 , and K = 200 in Rand K .

Our main goal is to verify the dependeces from the theory. We compare DASHA-PP with DASHA . Clearly, DASHA-PP can not generally perform better than DASHA . In different settings, we verify that the bigger p a, the closer DASHA-PP is to DASHA , i.e., DASHA-PP converges no slower than 1 / p a times.

In all experiments, we take the real-sim dataset with dimension d = 20 , 958 and the number of samples equals 72 , 309 from LIBSVM datasets (Chang and Lin, 2011) (under the 3-clause BSD license), and randomly split the dataset between n = 100 nodes equally, ignoring residual samples. In the finite-sum setting, we solve a classification problem with functions

where a ij ∈ R d is the feature vector of a sample on the i th node, y ij ∈ {-1 , 1 } is the corresponding label, and m is the number of samples on the i th node for all i ∈ [ n ] . In the stochastic setting, we consider functions

where x 1 , x 2 ∈ R d , {·} k is an indexing operation, a ij ∈ R d is a feature of a sample on the i th node, y ij ∈ { 1 , 2 } is a corresponding label, m is the number of samples located on the i th node, constant λ = 0 . 001 for all i ∈ [ n ] .

The code was written in Python 3.6.8 using PyTorch 1.9 (Paszke et al., 2019). A distributed environment was emulated on a machine with Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz and 64 cores.

We use the standard setting in experiments 7 where all parameters except step sizes are taken as suggested in theory. Step sizes are finetuned from a set { 2 i | i ∈ [ -10 , 10] } . We emulate the partial participation setting using s -nice sampling with the number of nodes n = 100 . We consider the Rand K compressor and take the batch size B = 1 . We plot the relation between communication rounds and values of the norm of gradients at each communication round.

In the finite-sum (Figure 1a) and in the stochastic setting (Figure 1b), we see that the bigger probability p a = s / n to 1 , the closer DASHA-PP to DASHA . Moreover, DASHA-PP with s = 10 and s = 1 converges approximately × 10 ( = 1 / p a ) and × 100 ( = 1 / p a ) times slower, accordingly. Our theory predicts such behavior.

A.1 Experiments in Partial Participation Setting

In this experiments, we compare our new algorithm DASHA-PP with previous baselines MARINA and FRECON in the partial participation setting. We consider MARINA and FRECON because they are the previous SOTA methods in the partial participation setting with compression . We investigate the same optimization problem and setup as in Section A of the paper. All methods use the Rand K compressor in these experiments.

Finite-Sum Setting. We now consider the function from (11). In Figures 2 and 3, we compare all three methods in the finite-sum setting on two different datasets: real-sim and MNIST . The parameter s is the number of clients participating in each round that are selected randomly using the s -nice sampling (server chooses uniformly s nodes without replacement). We can see that DASHA-PP converges faster than MARINA . Since FRECON does not support variance reduction of stochastic gradients, it converges to less accurate solutions.

0

10000

20000

30000

40000

50000

10

6

10

5

10

4

10

3

10

2

||

f

(

x k

)||

2

K

= 500

Number of nodes: 100

DASHA-PP-PAGE (

s

=1): Step size: 0.0625

VR-MARINA (

s

=1): Step size: 0.03125

FRECON (

s

=1): Step size: 0.125

0

10000

20000

30000

40000

50000

10

7

10

6

10

5

10

4

10

3

10

2

||

f

(

x k

)||

2

K

= 500

Number of nodes: 100

DASHA-PP-PAGE (

s

=10): Step size: 0.5

VR-MARINA (

s

=10): Step size: 0.25

FRECON (

s

=10): Step size: 1.0

0

10000

20000

30000

40000

50000

10

8

10

7

10

6

10

5

10

4

10

3

10

2

||

f

(

x k

)||

2

K

= 500

Number of nodes: 100

DASHA-PP-PAGE (

s

=90): Step size: 8.0

VR-MARINA (

s

=90): Step size: 4.0

FRECON (

s

=90): Step size: 8.0

# communication rounds

# communication rounds

# communication rounds

(a) 1 % of nodes participating (b) 10 % of nodes participating (c) 90 % of nodes participating

0

20000

40000

60000

80000

100000

120000

140000

10

6

10

5

10

4

10

3

10

2

||

f

(

x k

)||

2

K

= 1000

Number of nodes: 10

DASHA-PP-PAGE (

s

=1): Step size: 0.125

VR-MARINA (

s

=1): Step size: 0.0625

FRECON (

s

=1): Step size: 0.0625

0

20000

40000

60000

80000

100000

120000

140000

10

7

10

6

10

5

10

4

10

3

10

2

||

f

(

x k

)||

2

K

= 1000

Number of nodes: 10

DASHA-PP-PAGE (

s

=5): Step size: 0.5

VR-MARINA (

s

=5): Step size: 0.25

FRECON (

s

=5): Step size: 0.25

0

20000

40000

60000

80000

100000

120000

140000

10

8

10

7

10

6

10

5

10

4

10

3

10

2

||

f

(

x k

)||

2

K

= 1000

Number of nodes: 10

DASHA-PP-PAGE (

s

=9): Step size: 2.0

VR-MARINA (

s

=9): Step size: 0.5

FRECON (

s

=9): Step size: 0.25

# communication rounds

# communication rounds

(a) 10 % of nodes participating

(b) 50 % of nodes participating

# communication rounds

(c) 90 % of nodes participating

Stochastic Setting. In Figures 4 and 5, we consider the stochastic setting with the function from (11). We can see that DASHA-PP convergences to high accuracy solutions, unlike FRECON . Moreover, DASHA-PP improves the convergence rates of MARINA .

10 3

10 4

10 5

# communication rounds

10

5

10

4

10

3

10

2

||

f

(

x k

)||

2

K

= 200

2

n

= 100000

FRECON (

s

= 1): Step size: 0.0009765625

VR-MARINA (

s

= 1): Step size: 0.0009765625

DASHA-PP-MVR (

s

= 1): Step size: 0.0078125

10 3

10 4

10 5

# communication rounds

10

6

10

5

10

4

10

3

10

2

||

f

(

x k

)||

2

K

= 200

2

n

= 100000

FRECON (

s

= 5): Step size: 0.0078125

VR-MARINA (

s

= 5): Step size: 0.00390625

DASHA-PP-MVR (

s

= 5): Step size: 0.03125

10 3

10 4

10 5

# communication rounds

10

6

10

5

10

4

10

3

10

2

||

f

(

x k

)||

2

K

= 200

2

n

= 100000

FRECON (

s

=10): Step size: 0.015625

VR-MARINA (

s

=10): Step size: 0.0078125

DASHA-PP-MVR (

s

=10): Step size: 0.0625

(a) 10 % of nodes participating

(b) 50 % of nodes participating

(c) 100 % of nodes participating

10 3

10 4

10 5

10 6

10

5

10

4

10

3

10

2

||

f

(

x k

)||

2

K

= 100

2

n

= 100000

FRECON (

s

= 1): Step size: 0.0009765625

VR-MARINA (

s

= 1): Step size: 0.0009765625

DASHA-PP-MVR (

s

= 1): Step size: 0.0009765625

10 3

10 4

10 5

10 6

10

6

10

5

10

4

10

3

10

2

||

f

(

x k

)||

2

K

= 100

2

n

= 100000

FRECON (

s

= 5): Step size: 0.0009765625

VR-MARINA (

s

= 5): Step size: 0.0009765625

DASHA-PP-MVR (

s

= 5): Step size: 0.0078125

10 3

10 4

10 5

10 6

10

6

10

5

10

4

10

3

10

2

||

f

(

x k

)||

2

K

= 100

2

n

= 100000

FRECON (

s

=10): Step size: 0.0009765625

VR-MARINA (

s

=10): Step size: 0.0009765625

DASHA-PP-MVR (

s

=10): Step size: 0.03125

# communication rounds

# communication rounds

# communication rounds

(a) 10 % of nodes participating (b) 50 % of nodes participating (c) 100 % of nodes participating

B Original DASHA and DASHA-MVR Methods

To simplify the discussion and explanation from the main part, we present the algorithms from (Tyurin and Richt´ arik, 2023)

Algorithm 6 DASHA

1: Input: starting point x 0 ∈ R d , stepsize γ > 0 , momentum a ∈ (0 , 1] , number of iterations T ≥ 1

Algorithm 7 DASHA-MVR (with batch size B = 1 )

1: Input: starting point x 0 ∈ R d , stepsize γ > 0 , momentums a, b ∈ (0 , 1] , number of iterations T ≥ 1

3: for t = 0 , 1 , . . . , T -1 do

2: Initialize g 0 i ∈ R d on the nodes and g 0 = 1 n ∑ n i =1 g 0 i on the server

4: x t +1 = x t -γg t

5: Broadcast x t +1 and x t

6: for i = 1 , . . . , n in parallel do

9:

g

t

+1

i

t

t

+1

i

11: end for

=

g

+

m

10: Send m t +1 i to the server

i

14: Output: ˆ x T chosen uniformly at random from { x t } T -1 k =0

13: end for

C Problem of Estimating the Mean in the Partial Participation Setting

Wenow provide the example to explain why the only choice of B = O ( min { 1 p a √ m n , L 2 max ✶ 2 p a ̂ L 2 }) and B = O ( min { σ p a √ εn , L 2 σ ✶ 2 p a ̂ L 2 }) in DASHA-PP-PAGE and DASHA-PP-MVR , accordingly, guarantees the degeneration up to 1 / p a . This is surprising, because in methods with the variance reduction of stochastic gradients (Li et al., 2021a; Tyurin and Richt´ arik, 2023) we can take the size of batch size B = O (√ m n ) and B = O ( σ √ εn ) and guarantee the optimality. Note that the smaller the batch size B , the more the server and the nodes have to communicate to get ε -solution.

Let us consider the task of estimating the mean of vectors in the distributed setting. Suppose that we have n nodes, and each of them contains m vectors { x ij } j m =1 , where x ij ∈ R d for all i ∈ [ n ] , j ∈ [ m ] . First, let us consider that each node samples a mini-batch I i of size B with replacement and sends it to the server. Then the server calculates the mean of the mini-batches from nodes. One can easily show that the variance of the estimator is

Next, we consider the same task in the partial participation setting with s -nice sampling, i.e., we sample a random set S ⊂ [ n ] of s ∈ [ n ] nodes without replacement and receive the mini-batches only from the sampled nodes. Such sampling of nodes satisfy Assumption 8 with p a = s / n and p a = s ( s -1) / n ( n -1) . In this case, the variance of the estimator (See Lemma 1 with r i = 0 and s i = ∑ j ∈ I i x ij ) is

Let us assume that s ≤ n / 2 . Note that (13) scales with any B ≥ 1 , while (14) only scales when B = O ( L 2 max / ̂ L 2 ) . In other words, for large enough B, the variance in (14) does not significantly improves with the growth of B due to the term ̂ L 2 . In our proof, due to partial participation, the variance from (14) naturally appears, and we get the same effect. As was mentioned in Sections 6.2 and 6.3, it can be seen in our convergence rate bounds.

D Auxiliary facts

We list auxiliary facts that we use in our proofs:

For all x, y ∈ R d , we have

Let us take a random vector ξ ∈ R d , then

D.1 Sampling Lemma

This section provides a lemma that we regularly use in our proofs, and it is useful for samplings that satisfy Assumption 8.

Lemma 1. Suppose that a set S is a random subset of a set [ n ] such that

̸

where p a ∈ (0 , 1] and p aa ∈ [0 , 1] . Let us take random independent vectors s i ∈ R d for all i ∈ [ n ] , nonrandom vector r i ∈ R d for all i ∈ [ n ] , and random vectors

then

Proof. Let us define additional constants p an and p nn, such that

̸

̸

Note, that and

Using the law of total expectation and

we have

̸

From the independence of random vectors s i , we obtain

̸

̸

̸

̸

̸

Using (17) and (18), we have

̸

̸

̸

Finally, using that p aa ≤ p 2 a , we have

D.2 Compressors Facts

We define the Rand K compressor that chooses without replacement K coordinates, scales them by a constant factor to preserve unbiasedness and zero-out other coordinates.

Definition 5. Let us take a random subset S from [ d ] , | S | = K, K ∈ [ d ] . We say that a stochastic mapping C : R d → R d is Rand K if

where { e i } d i =1 is the standard unit basis.

Theorem 6. If C is Rand K , then C ∈ U ( d k -1 ) .

E Proofs of Theorems

There are three different sources of randomness in Algorithm 1: the first one from vectors { k t +1 i } n i =1 , the second one from compressors {C i } n i =1 , and the third one from availability of nodes. We define E k [ · ] , E C [ · ] and E p a [ · ] to be conditional expectations w.r.t. { k t +1 i } n i =1 , {C i } n i =1 , and availability, accordingly, conditioned on all previous randomness. Moreover, we define E t +1 [ · ] to be a conditional expectation w.r.t. all randomness in iteration t +1 conditioned on all previous randomness. Note, that E t +1 [ · ] = E k [E C [E p a [ · ]]] .

In the case of DASHA-PP-PAGE , there are two different sources of randomness from { k t +1 i } n i =1 . We define E p page [ · ] and E B [ · ] to be conditional expectations w.r.t. the probabilistic switching and mini-batch indices I t i , accordingly, conditioned on all previous randomness. Note, that E t +1 [ · ] = E B [ E C [ E p a [ E p page [ · ] ]]] and E t +1 [ · ] = E B [ E p page [E C [E p a [ · ]]] ] .

E.1 Standard Lemmas in the Nonconvex Setting

We start the proof of theorems by providing standard lemmas from the nonconvex optimization. Lemma 2. Suppose that Assumption 2 holds and let x t +1 = x t -γg t . Then for any g t ∈ R d and γ > 0 , we have

Proof. Using L -smoothness, we have

Next, due to -⟨ x, y ⟩ = 1 2 ∥ x -y ∥ 2 -1 2 ∥ x ∥ 2 -1 2 ∥ y ∥ 2 , we obtain

Lemma 3. Suppose that Assumption 1 holds and

where Ψ t is a sequence of numbers, Ψ t ≥ 0 for all t ∈ [ T ] , constant C ≥ 0 , and constant γ > 0 . Then

where a point ̂ x T is chosen uniformly from a set of points { x t } T -1 t =0 .

Proof. By unrolling (20) for t from 0 to T -1 , we obtain

We subtract f ∗ , divide inequality by γT 2 , and take into account that f ( x ) ≥ f ∗ for all x ∈ R , and Ψ t ≥ 0 for all t ∈ [ T ] , to get the following inequality:

It is left to consider the choice of a point ̂ x T to complete the proof of the lemma.

Lemma 4. If 0 < γ ≤ ( L + √ A ) -1 , L > 0 , and A ≥ 0 , then

The lemma can be easily checked with the direct calculation.

E.2 Generic Lemmas

Lemma 5. Suppose that Assumptions 7 and 8 hold and let us consider sequences g t +1 i , h t +1 i , and k t +1 i from Algorithm 1, then

and

Proof. First, we estimate E C [ E p a [ ∥ ∥ g t +1 -h t +1 ∥ ∥ 2 ]] :

where we used (16). Due to Assumption 8, we have

and

Thus, we can get

t t

From Assumption 7, we have

The second inequality can be proved almost in the same way:

Lemma 6. Suppose that Assumptions 2, 7, and 8 hold and let us take a = p a 2 ω +1 , then

Proof. Due to Lemma 2 and the update step from Line 5 in Algorithm 1, we have

E t +1 [ f ( x t +1 ) ]

Let us fix some constants κ, η ∈ [0 , ∞ ) that we will define later. Combining the last inequality, bounds (22), (23) and using the law of total expectation, we get

[

t

+1

]

Now, by taking κ = γ a , we can see that γ + κ (1 -a ) 2 ≤ κ, and thus

Next, by taking η = γ ((2 ω +1) p a -p aa ) np 2 a and considering the choice of a , one can show that ( γa ((2 ω +1) p a -p aa ) np 2 a + η ( a 2 (2 ω +1 -p a ) p a +(1 -a ) 2 )) ≤ η. Thus

E.3 Proof for DASHA-PP

Lemma7. Suppose that Assumptions 3 and 8 hold. For h t +1 i and k t +1 i from Algorithm 1 ( DASHA-PP ) we have

Using and (16), we have

We can use Lemma 1 with r i = h t i and s i = k t +1 i to obtain

In the last in inequality, we used Assumption 3. Now, we prove the second inequality:

Finally, the third inequality of the theorem follows from (15) and Assumption 3.

Theorem 2. Suppose that Assumptions 1, 2, 3, 7 and 8 hold. Let us take a = p a 2 ω +1 , b = p a 2 -p a ,

and g 0 i = h 0 i = ∇ f i ( x 0 ) for all i ∈ [ n ] in Algorithm 1 (DASHA-PP) , then E [ ∥ ∥ ∇ f ( ̂ x T ) ∥ ∥ 2 ] ≤ 2∆ 0 γT .

Proof. Let us fix constants ν, ρ ∈ [0 , ∞ ) that we will define later. Considering Lemma 6, Lemma 7, and the law of total expectation, we obtain

After rearranging the terms, we get

By taking ν = γ b , one can show that ( γ + ν (1 -b ) 2 ) ≤ ν, and

Note that b = p a 2 -p a , thus

And if we take ρ = 8 bγω (2 ω +1) np 2 a + 2 γ ( p a -p aa ) np 2 a , then

and

Let us simplify the last inequality. First, note that

due to b ≥ p a 2 . All in all, we have

due to b ≤ p a . Second,

Using Lemma 4 and the assumption about γ, we get

It is left to apply Lemma 3 with

to conclude the proof.

E.4 Proof for DASHA-PP-PAGE

Let us denote

Lemma 8. Suppose that Assumptions 3, 4, and 8 hold. For h t +1 i and k t +1 i from Algorithm 1 ( DASHA-PP-PAGE ) we have

2.

3.

Proof. First, we prove the first inequality of the theorem:

Using and

we obtain

Next, we consider E p a [ ∥ ∥ h t +1 1 -E p a [ h t +1 1 ]∥ ∥ 2 ] . We can use Lemma 1 with r i = h t i and s i = k t +1 i, 1 to obtain

From Assumption 3, we have

Now, we prove the bound for E B [ E p a [ ∥ ∥ h t +1 2 -E B [ E p a [ h t +1 2 ]]∥ ∥ 2 ]] . Considering that minibatches in the algorithm are independent, we can use Lemma 1 with r i = h t i and s i = k t +1 i, 2 to obtain

Next, we use Assumptions 3 and 4 to get

Applying (25) and (26) into (24), we get

The proof of the second inequality almost repeats the previous one:

Let us consider [ [ ∥ t +1 [ [ t +1 ]]∥ 2 ]]

Considering (15) and Assumption 3, we obtain

Next, we obtain the bound for E B [ E p a [ ∥ ∥ h t +1 i, 2 -E B [ E p a [ h t +1 i, 2 ]]∥ ∥ 2 ]] :

where we used Assumption 3. By plugging (28) and (29) into (27), we get

From the independence of elements in the mini-batch, we obtain

where we used Assumption 4. Finally, we prove the last inequality:

Using the independence of elements in the mini-batch, we have

.

It it left to consider Assumptions 3 and 4 to get

Theorem 3. Suppose that Assumptions 1, 2, 3, 4, 7, and 8 hold. Let us take a = p a 2 ω +1 , b = p page p a 2 -p a , probability p page ∈ (0 , 1] ,

and g 0 i = h 0 i = ∇ f i ( x 0 ) for all i ∈ [ n ] in Algorithm 1 (DASHA-PP-PAGE) then E [ ∥ ∥ ∇ f ( ̂ x T ) ∥ ∥ 2 ] ≤ 2∆ 0 γT .

Proof. Let us fix constants ν, ρ ∈ [0 , ∞ ) that we will define later. Considering Lemma 6, Lemma 8, and the law of total expectation, we obtain

After rearranging the terms, we get

Let us simplify the inequality. First, due to b ≥ p page p a 2 , we have

Second, due to b ≤ p a p page and p aa ≤ p 2 a , we get

Combining all bounds together, we obtain the following simplified inequality:

Using Lemma 4 and the assumption about γ, we get

It is left to apply Lemma 3 with

to conclude the proof.

Corollary 1. Let the assumptions from Theorem 3 hold and p page = B / ( m + B ) . Then DASHA-PP-PAGE needs

communication rounds to get an ε -solution and the expected number of gradient calculations per node equals O ( m + BT ) .

Proof. In the view of Theorem 3, it is enough to do

steps to get ε -solution. Using the choice of p mega and the definition of ✶ p a , we can get (10). Note that the expected number of gradients calculations at each communication round equals p mega m + (1 -p mega ) B = 2 mB m + B ≤ 2 B.

Corollary 2. Suppose that assumptions of Corollary 1 hold, B ≤ min { 1 p a √ m n , L 2 max ✶ 2 p a ̂ L 2 } 8 , and we use the unbiased compressor Rand K with K = Θ( Bd / √ m ) . Then the communication complexity of

Algorithm 1 is O ( d + L max ∆ 0 d p a ε √ n ) , and the expected number of gradient calculations per node equals O ( m + L max ∆ 0 √ m p a ε √ n ) .

Proof. The communication complexity equals

O ( d + KT ) = O ( d + ∆ 0 ε [ KL + K ω p a √ n ( ̂ L + L max √ B ) + K 1 p a √ m n ( ✶ p a ̂ L √ B + L max B )]) . Since B ≤ L 2 max ✶ 2 p a ̂ L 2 , we have ✶ p a ̂ L √ B + L max B ≤ 2 L max B and O ( d + KT ) = O ( d + ∆ 0 ε [ KL + K ω p a √ n ( ̂ L + L max √ B ) + K 1 p a √ m n L max B ]) . Note that K = Θ ( Bd √ m ) = O ( d p a √ n ) and ω +1 = d K due to Theorem 6, thus ])

Using the same reasoning, the expected number of gradient calculations per node equals

E.5 Proof for DASHA-PP-FINITE-MVR

Lemma 9. Suppose that Assumptions 3, 4, and 8 hold. For h t +1 i , h t +1 ij and k t +1 i from Algorithm 1 ( DASHA-PP-FINITE-MVR ) we have

3.

4.

Proof. We start by proving the first inequality. Note that

thus

We can use Lemma 1 with r i = h t i and s i = k t +1 i to obtain

Next, we again use Lemma 1 with r i = 0 , s i = ∇ f ij ( x t +1 ) - ∇ f ij ( x t ) -b ( h t ij -∇ f ij ( x t ) ) , p a = B m , and p aa = B ( B -1) m ( m -1) :

Due to Assumptions 3 and 4, we have

Let us get the bound for the second inequality:

Let us use Lemma 1 with r i = 0 , s i = ∇ f ij ( x t +1 ) -∇ f ij ( x t ) -b ( h t ij -∇ f ij ( x t ) ) , p a = B m , and p aa = B ( B -1) m ( m -1) :

E B [ E p a [ ∥ ∥ h t +1 i -∇ f i ( x t +1 ) ∥ ∥ 2 ]]

It is left to consider Assumption 4:

Finally, we obtain the bound for the last inequality of the lemma:

Using Lemma 1, we get

where we used Assumptions 3 and 4.

g 0 i = h 0 i = ∇ f i ( x 0 ) for all i ∈ [ n ] and h 0 ij = ∇ f ij ( x 0 ) for all i ∈ [ n ] , j ∈ [ m ] (DASHA-PP-FINITE-MVR) then E [ ∥ ∥ ∇ f ( ̂ x T ) ∥ ∥ 2 ] ≤ 2∆ 0 γT .

Proof. Let us fix constants ν, ρ, δ ∈ [0 , ∞ ) that we will define later. Considering Lemma 6, Lemma 9, and the law of total expectation, we obtain

Due to b = p a B m 2 -p a B m ≤ p a 2 -p a , we have

Moreover, we consider that 1 -p a B m ≤ 1 , therefore

and

After rearranging the terms, we get

Thus, if we take ν = γ b , then γ + ν (1 -b ) 2 ≤ ν and

Next, if we take ρ = 8 bγω (2 ω +1) np 2 a + 2 γ ( p a -p aa ) np 2 a , then

therefore

Due to b ≤ p a and p a -p aa p a ≤ 1 , we have

Let us take δ = 24 bγω (2 ω +1) np 2 a B + 6 γ np a B . Thus

and

Let us simplify the term near E [ ∥ ∥ x t +1 -x t ∥ ∥ 2 ] . Due to b ≤ p a , p a -p aa p a ≤ 1 , and 1 -p a ≤ 1 , we have

Considering that b ≤ p a B m and b ≥ p a B 2 m , we obtain

All in all, we have

Using Lemma 4 and the assumption about γ, we get

It is left to apply Lemma 3 with

to conclude the proof.

E.6 Proof for DASHA-PP-MVR

Let us denote ∇ f i ( x t +1 ; ξ t +1 i ) := 1 B ∑ B j =1 ∇ f i ( x t +1 ; ξ t +1 ij ) .

Lemma 10. Suppose that Assumptions 3, 5, 6 and 8 hold. For h t +1 i and k t +1 i from Algorithm 1 ( DASHA-PP-MVR ) we have

3.

Proof. First, let us proof the bound for E k [ E p a [ ∥ ∥ h t +1 -∇ f ( x t +1 ) ∥ ∥ 2 ]] :

Using

E k [ E p a [ h t +1 i ]] = h t i +E k [ k t +1 i ] = h t i + ∇ f i ( x t +1 ) -∇ f i ( x t ) -b ( h t i -∇ f i ( x t )) and (16), we have

We can use Lemma 1 with r i = h t i and s i = k t +1 i to obtain

In the last equality, we use the independence of elements in the mini-batches. Due to Assumption 5, we get

where we use the independence of elements in the mini-batches. Using Assumptions 3 and 6, we obtain

Now, we prove the second inequality:

Considering the independence of elements in the mini-batch, we obtain

Next, we use Assumptions 3, 6, 5, to get

It is left to prove the bound for E k [ ∥ ∥ k t +1 i ∥ ∥ 2 ] :

Using Assumptions 3, 6, 5 and the independence of elements in the mini-batch, we get

Theorem 4. Suppose that Assumptions 1, 2, 3, 5, 6, 7 and 8 hold. Let us take a = p a 2 ω +1 , b ∈ ( 0 , p a 2 -p a ] , γ ≤ ( L + [ 48 ω (2 ω +1) np 2 a ( ̂ L 2 + (1 -b ) 2 L 2 σ B ) + 12 np a b ( ( 1 -p aa p a ) ̂ L 2 + (1 -b ) 2 L 2 σ B )] 1 / 2 ) -1 , and g 0 i = h 0 i for all i ∈ [ n ] in Algorithm 1 (DASHA-PP-MVR) . Then

Proof. Let us fix constants ν, ρ ∈ [0 , ∞ ) that we will define later. Considering Lemma 6, Lemma 10, and the law of total expectation, we obtain

After rearranging the terms, we get

Note that b ≤ p a 2 -p a , thus

And if we take ρ = 8 bγω (2 ω +1) np 2 a + 2 γ ( p a -p aa ) np 2 a , then

and

( )

Let us simplify the inequality. First, due to b ≤ p a and (1 -p a ) ≤ 1 -p aa p a , we have

therefore

E [ f ( x t +1 ) ] + γ (2 ω +1) p a E [ ∥ ∥ g t +1 -h t +1 ∥ ∥ 2 ] + γ ((2 ω +1) p a -p aa ) np 2 a E [ 1 n n ∑ i =1 ∥ ∥ g t +1 i -h t +1 i ∥ ∥ 2 ] + γ b E [ ∥ ∥ h t +1 -∇ f ( x t +1 ) ∥ ∥ 2 ] + ( 8 bγω (2 ω +1) np 2 a + 2 γ ( p a -p aa ) np 2 a ) E [ 1 n n ∑ i =1 ∥ ∥ h t +1 i -∇ f i ( x t +1 ) ∥ ∥ 2 ] ≤ E [ f ( x t ) ] -γ 2 E [ ∥ ∥ ∇ f ( x t ) ∥ ∥ 2 ] + γ (2 ω +1) p a E [ ∥ ∥ g t -h t ∥ ∥ 2 ] + γ ((2 ω +1) p a -p aa ) np 2 a E [ 1 n n ∑ i =1 ∥ ∥ g t i -h t i ∥ ∥ 2 ] -( 1 2 γ -L 2 -12 γω (2 ω +1) np 2 a ( 2(1 -b ) 2 L 2 σ B +2 ̂ L 2 ) -3 γ np a b ( 2(1 -b ) 2 L 2 σ B +2 ( 1 -p aa p a ) ̂ L 2 ) ) E [ ∥ ∥ x t +1 -x t ∥ ∥ 2 ] + γ b E [ ∥ ∥ h t -∇ f ( x t ) ∥ ∥ 2 ] + ( 8 bγω (2 ω +1) np 2 a + 2 γ ( p a -p aa ) np 2 a ) E [ 1 n n ∑ i =1 ∥ ∥ h t i -∇ f i ( x t ) ∥ ∥ 2 ] + ( 8 b 2 γω (2 ω +1) np 2 a + 2 γb np a + ( 8 bγω (2 ω +1) np 2 a + 2 γ ( p a -p aa ) np 2 a ) 2 b 2 p a ) σ 2 B = E [ f ( x t ) ] -γ 2 E [ ∥ ∥ ∇ f ( x t ) ∥ ∥ 2 ] + γ (2 ω +1) p a E [ ∥ ∥ g t -h t ∥ ∥ 2 ] + γ ((2 ω +1) p a -p aa ) np 2 a E [ 1 n n ∑ i =1 ∥ ∥ g t i -h t i ∥ ∥ 2 ] -( 1 2 γ -L 2 -24 γω (2 ω +1) np 2 a ( (1 -b ) 2 L 2 σ B + ̂ L 2 ) -6 γ np a b ( (1 -b ) 2 L 2 σ B + ( 1 -p aa p a ) ̂ L 2 ) ) E [ ∥ ∥ x t +1 -x t ∥ ∥ 2 ] + γ b E [ ∥ ∥ h t -∇ f ( x t ) ∥ ∥ 2 ] + ( 8 bγω (2 ω +1) np 2 a + 2 γ ( p a -p aa ) np 2 a ) E [ 1 n n ∑ i =1 ∥ ∥ h t i -∇ f i ( x t ) ∥ ∥ 2 ] + ( 8 b 2 γω (2 ω +1) np 2 a + 2 γb np a + ( 8 bγω (2 ω +1) np 2 a + 2 γ ( p a -p aa ) np 2 a ) 2 b 2 p a ) σ 2 B . Also, we can simplify the last term: ( 8 bγω (2 ω +1) np 2 a + 2 γ ( p a -p aa ) np 2 a ) 2 b 2 p a = 16 b 3 γω (2 ω +1) np 3 a + 4 b 2 γ ( 1 -p aa p a ) np 2 a ≤ 16 b 2 γω (2 ω +1) np 2 a + 4 bγ np a , thus

Using Lemma 4 and the assumption about γ, we get

It is left to apply Lemma 3 with

Corollary 3. Suppose that assumptions from Theorem 4 hold, momentum b = Θ ( min { p a ω √ nεB σ 2 , p a nεB σ 2 }) , σ 2 nεB ≥ 1 , and h 0 i = 1 B init ∑ B init k =1 ∇ f i ( x 0 ; ξ 0 ik ) for all i ∈ [ n ] , and batch size B init = Θ ( √ p a B b ) , then Algorithm 1 (DASHA-PP-MVR) needs

communication rounds to get an ε -solution and the number of stochastic gradient calculations per node equals O ( B init + BT ) .

Proof. Using the result from Theorem 4, we have

where ✶ p a = √ 1 -p aa p a . It enough to take the following T to get ε -solution.

Let us bound the norms:

Using the same reasoning, one cat get 1 n ∑ n i =1 E [ ∥ ∥ h 0 i -∇ f i ( x 0 ) ∥ ∥ 2 ] ≤ σ 2 B init . Combining all inequalities, we have

Using the choice of B init and b, we obtain

Using σ 2 nεB ≥ 1 , we can conclude the proof of the inequality. The number of stochastic gradients that each node calculates equals B init +2 BT = O ( B init + BT ) .

Corollary 4. Suppose that assumptions of Corollary 3 hold, batch size B ≤ min { σ p a √ εn , L 2 σ ✶ 2 p a ̂ L 2 } , we take Rand K compressors with K = Θ ( Bd √ εn σ ) . Then the communication complexity equals O ( dσ √ p a √ nε + L σ ∆ 0 d p a √ nε ) , and the expected number of stochastic gradient calculations per node equals O ( σ 2 √ p a nε + L σ ∆ 0 σ p a ε 3 / 2 n ) .

Proof. The communication complexity equals

Due to B ≤ L 2 σ 2 p a ̂ L 2 , we have ✶ p a ̂ L + L σ √ B ≤ 2 L σ √ B and

From Theorem 6, we have ω +1 = d K . Since K = Θ ( Bd √ εn σ ) = O ( d p a √ n ) , the communication complexity equals

And the expected number of stochastic gradient calculations per node equals

O

(

B

init

+

BT

)

F Analysis of DASHA-PP under Polyak-Łojasiewicz Condition

In this section, we provide the theoretical convergence rates of DASHA-PP under Polyak-Łojasiewiczc Condition.

Assumption 9. The function f satisfy (Polyak-Łojasiewicz) PŁ-condition:

Under Polyak-Łojasiewicz condition, a (random) point ̂ x is ε -solution, if E[ f ( ̂ x )] -f ∗ ≤ ε.

We now provide the convergence rates of DASHA-PP under PŁ-condition.

F.1 Gradient Setting

Theorem 8. Suppose that Assumption 1, 2, 3, 7, 8 and 9 hold. Let us take a = p a 2 ω +1 , b = p a 2 -p a ,

and h 0 i = g 0 i = ∇ f i ( x 0 ) for all i ∈ [ n ] in Algorithm 1 (DASHA-PP) , then E [ f ( x T ) ] -f ∗ ≤ (1 -γµ ) T ∆ 0 .

Let us provide bounds up to logarithmic factors and use ˜ O ( · ) notation. The provided theorem states that to get ε -solution DASHA-PP have to run

communication rounds. The method DASHA from (Tyurin and Richt´ arik, 2023), have to run

communication rounds to get ε -solution. The difference is the same as in the general nonconvex case (see Section 6.1). Up to Lipschitz constants factors, we get the degeneration up to 1 / p a factor due to the partial participation.

F.2 Finite-Sum Setting

Theorem 9. Suppose that Assumption 1, 2, 3, 7, 4, 8, and 9 hold. Let us take a = p a , probability

2 ω +1 p page = B m + B , b = p page p a 2 -p a ,

and h 0 i = g 0 i = ∇ f i ( x 0 ) for all i ∈ [ n ] in Algorithm 1 (DASHA-PP-PAGE) , then E [ f ( x T ) ] -f ∗ ≤ (1 -γµ ) T ∆ 0 .

The provided theorem states that to get ε -solution DASHA-PP have to run

communication rounds. The method DASHA-PAGE from (Tyurin and Richt´ arik, 2023), have to run

communication rounds to get ε -solution. We can guarantee the degeneration up to 1 / p a factor due to the partial participation only if B = O ( L 2 max ̂ L 2 ) . The same conclusion we have in Section 6.2.

F.3 Stochastic Setting

Theorem 10. Suppose that Assumption 1, 2, 3, 7, 5, 6, 8 and 9 hold. Let us take a = p a 2 ω +1 ,

and h 0 i = g 0 i for all i ∈ [ n ] in Algorithm 1 (DASHA-PP-MVR) , then E [ f ( x T ) -f ∗ ]

The provided theorems states that to get ε -solution DASHA-PP have to run

(

{

p

a

ω

√

µnεB

σ

2

p

µnεB

σ

2

})

communication rounds.

The method DASHA-SYNC-MVR from (Tyurin and Richt´ arik, 2023), have to run

communication rounds to get ε -solution 9 .

In the stochastic setting, the comparison is a little bit more complicated. As in the finite-sum setting, we have to take B = O ( L 2 σ ̂ L 2 ) to guarantee the degeneration up to 1 / p a of the term P 1 from (31). However, DASHA-PP-MVR has also suboptimal term P 2 . This suboptimality is tightly connected with the suboptimality of B init in the general nonconvex case, which we discuss in Section 6.3, and it also appears in the analysis of DASHA-MVR (Tyurin and Richt´ arik, 2023). Let us provide the counterpart of Corollary 4. The corollary reveals that we can escape regimes when P 2 is the bottleneck by choosing the parameters of the compressors.

,

Corollary 5. Suppose that assumptions of Theorem 10 hold, batch size B ≤ min { σ p a √ µεn , L 2 σ ̂ L 2 } we take Rand K compressors with K = Θ ( Bd √ µεn σ ) . Then the communication complexity equals

and the expected number of stochastic gradient calculations per node equals

Up to Lipschitz constants, DASHA-PP-MVR has the state-of-the-art oracle complexity under PŁcondition (see (Li et al., 2021a)). Moreover, DASHA-PP-MVR has the state-of-the-art communication complexity of DASHA for a small enough µ .

We take

b

=

Θ

min

,

a

F.4 Proofs of Theorems

The following proofs almost repeat the proofs from Section E. And one of the main changes is that instead of Lemma 3, we use the following lemma.

F.4.1 Standard Lemma under Polyak-Łojasiewicz Condition

Lemma 11. Suppose that Assumptions 1 and 9 hold and

where Ψ t is a sequence of numbers, Ψ t ≥ 0 for all t ∈ [ T ] , constant C ≥ 0 , constant µ > 0 , and constant γ ∈ (0 , 1 /µ ) . Then

Proof. We subtract f ∗ and use PŁ-condition (30) to get

Unrolling the inequality, we have

It is left to note that Ψ t ≥ 0 for all t ∈ [ T ] .

F.4.2 Generic Lemma

We now provide the counterpart of Lemma 6.

Lemma 12. Suppose that Assumptions 2, 7, 8 and 9 hold and let us take a = p a , then

Proof. Let us fix some constants κ, η ∈ [0 , ∞ ) that we will define later. Using the same reasoning as in Lemma 6, we can get

Let us take κ = 2 γ a . One can show that γ + κ (1 -a ) 2 ≤ ( 1 -a 2 ) κ, and thus

Considering the choice of a , one can show that ( a 2 (2 ω +1 -p a ) p a +(1 -a ) 2 ) ≤ 1 -a. If we take η = 4 γ ((2 ω +1) p a -p aa ) np 2 a , then ( 2 γa ((2 ω +1) p a -p aa ) np 2 a + η ( a 2 (2 ω +1 -p a ) p a +(1 -a ) 2 )) ≤ ( 1 -a 2 ) η and

It it left to consider that γ ≤ , and therefore 1 -≤ 1 -γµ.

a 2 µ a 2

F.4.3 Proof for DASHA-PP under PŁ-condition

Theorem 8. Suppose that Assumption 1, 2, 3, 7, 8 and 9 hold. Let us take a = p a 2 ω +1 , b = p a 2 -p a ,

and h 0 i = g 0 i = ∇ f i ( x 0 ) for all i ∈ [ n ] in Algorithm 1 (DASHA-PP) , then E [ f ( x T ) ] -f ∗ ≤ (1 -γµ ) T ∆ 0 .

Proof. Let us fix constants ν, ρ ∈ [0 , ∞ ) that we will define later. Considering Lemma 12, Lemma 7, and the law of total expectation, we obtain

After rearranging the terms, we get

+ 2 γ b E [ ∥ ∥ h t +1 -∇ f ( x t +1 ) ∥ ∥ 2 ] + ρ E [ 1 n n ∑ i =1 ∥ ∥ h t +1 i -∇ f i ( x t +1 ) ∥ ∥ 2 ] ≤ E [ f ( x t ) ] -γ 2 E [ ∥ ∥ ∇ f ( x t ) ∥ ∥ 2 ] +(1 -γµ ) 2 γ (2 ω +1) p a E [ ∥ ∥ g t -h t ∥ ∥ 2 ] +(1 -γµ ) 4 γ ((2 ω +1) p a -p aa ) np 2 a E [ 1 n n ∑ i =1 ∥ ∥ g t i -h t i ∥ ∥ 2 ] -( 1 2 γ -L 2 -20 γω (2 ω +1) ̂ L 2 np 2 a -4 γ ( p a -p aa ) ̂ L 2 bnp 2 a -ρ 2(1 -p a ) ̂ L 2 p a ) E [ ∥ ∥ x t +1 -x t ∥ ∥ 2 ] + ( 1 -b 2 ) 2 γ b E [ ∥ ∥ h t -∇ f ( x t ) ∥ ∥ 2 ] + ( 20 b 2 γω (2 ω +1) np 2 a + 4 γb ( p a -p aa ) np 2 a + ρ ( 2 b 2 (1 -p a ) p a +(1 -b ) 2 )) E [ 1 n n ∑ i =1 ∥ ∥ h t i -∇ f i ( x t ) ∥ ∥ 2 ] . Note that b = p a 2 -p a , thus ( 20 b 2 γω (2 ω +1) np 2 a + 4 γb ( p a -p aa ) np 2 a + ρ ( 2 b 2 (1 -p a ) p a +(1 -b ) 2 )) ≤ ( 20 b 2 γω (2 ω +1) np 2 a + 4 γb ( p a -p aa ) np 2 a + ρ (1 -b ) ) . And if we take ρ = 40 bγω (2 ω +1) np 2 a + 8 γ ( p a -p aa ) np 2 a , then ( 20 b 2 γω (2 ω +1) np 2 a + 4 γb ( p a -p aa ) np 2 a + ρ (1 -b ) ) ≤ ( 1 -b 2 ) ρ, and E [ f ( x t +1 ) ] + 2 γ (2 ω +1) p a E [ ∥ ∥ g t +1 -h t +1 ∥ ∥ 2 ] + 4 γ ((2 ω +1) p a -p aa ) np 2 a E [ 1 n n ∑ i =1 ∥ ∥ g t +1 i -h t +1 i ∥ ∥ 2 ] + 2 γ b E [ ∥ ∥ h t +1 -∇ f ( x t +1 ) ∥ ∥ 2 ] + ( 40 bγω (2 ω +1) np 2 a + 8 γ ( p a -p aa ) np 2 a ) E [ 1 n n ∑ i =1 ∥ ∥ h t +1 i -∇ f i ( x t +1 ) ∥ ∥ 2 ] ≤ E [ f ( x t ) ] -γ 2 E [ ∥ ∥ ∇ f ( x t ) ∥ ∥ 2 ] +(1 -γµ ) 2 γ (2 ω +1) p a E [ ∥ ∥ g t -h t ∥ ∥ 2 ] +(1 -γµ ) 4 γ ((2 ω +1) p a -p aa ) np 2 a E [ 1 n n ∑ i =1 ∥ ∥ g t i -h t i ∥ ∥ 2 ] -( 1 2 γ -L 2 -20 γω (2 ω +1) ̂ L 2 np 2 a -4 γ ( p a -p aa ) ̂ L 2 bnp 2 a -80 bγω (2 ω +1)(1 -p a ) ̂ L 2 np 3 a -16 γ ( p a -p aa ) (1 -p a ) ̂ L 2 np 3 a ) E [ ∥ ∥ x t +1 -x t ∥ ∥ 2 ] + ( 1 -b 2 ) 2 γ b E [ ∥ ∥ h t -∇ f ( x t ) ∥ ∥ 2 ] + ( 1 -b 2 )( 40 bγω (2 ω +1) np 2 a + 8 γ ( p a -p aa ) np 2 a ) E [ 1 n n ∑ i =1 ∥ ∥ h t i -∇ f i ( x t ) ∥ ∥ 2 ] . Due to p a 2 ≤ b ≤ p a , we have E [ f ( x t +1 ) ] + 2 γ (2 ω +1) p a E [ ∥ ∥ g t +1 -h t +1 ∥ ∥ 2 ] + 4 γ ((2 ω +1) p a -p aa ) np 2 a E [ 1 n n ∑ i =1 ∥ ∥ g t +1 i -h t +1 i ∥ ∥ 2 ] + 2 γ b E [ ∥ ∥ h t +1 -∇ f ( x t +1 ) ∥ ∥ 2 ] + ( 40 bγω (2 ω +1) np 2 a + 8 γ ( p a -p aa ) np 2 a ) E [ 1 n n ∑ i =1 ∥ ∥ h t +1 i -∇ f i ( x t +1 ) ∥ ∥ 2 ]

. Using Lemma 4 and the assumption about γ, we get E [ f ( x t +1 ) ] + 2 γ (2 ω +1) p a E [ ∥ ∥ g t +1 -h t +1 ∥ ∥ 2 ] + 4 γ ((2 ω +1) p a -p aa ) np 2 a E [ 1 n n ∑ i =1 ∥ ∥ g t +1 i -h t +1 i ∥ ∥ 2 ] + 2 γ b E [ ∥ ∥ h t +1 -∇ f ( x t +1 ) ∥ ∥ 2 ] + ( 40 bγω (2 ω +1) np 2 a + 8 γ ( p a -p aa ) np 2 a ) E [ 1 n n ∑ i =1 ∥ ∥ h t +1 i -∇ f i ( x t +1 ) ∥ ∥ 2 ] ≤ E [ f ( x t ) ] -γ 2 E [ ∥ ∥ ∇ f ( x t ) ∥ ∥ 2 ] +(1 -γµ ) 2 γ (2 ω +1) p a E [ ∥ ∥ g t -h t ∥ ∥ 2 ] +(1 -γµ ) 4 γ ((2 ω +1) p a -p aa ) np 2 a E [ 1 n n ∑ i =1 ∥ ∥ g t i -h t i ∥ ∥ 2 ] + ( 1 -b 2 ) 2 γ b E [ ∥ ∥ h t -∇ f ( x t ) ∥ ∥ 2 ] + ( 1 -b 2 )( 40 bγω (2 ω +1) np 2 a + 8 γ ( p a -p aa ) np 2 a ) E [ 1 n n ∑ i =1 ∥ ∥ h t i -∇ f i ( x t ) ∥ ∥ 2 ] . Note that γ ≤ a 4 µ ≤ p a 4 µ ≤ b 2 µ , thus 1 -b 2 ≤ 1 -γµ and

In the view of Lemma 11 with

we can conclude the proof of the theorem.

F.4.4 Proof for DASHA-PP-PAGE under PŁ-condition

Theorem 9. Suppose that Assumption 1, 2, 3, 7, 4, 8, and 9 hold. Let us take a = a 2 ω +1 , probability p page = B , b = p page p a ,

p

and h 0 i = g 0 i = ∇ f i ( x 0 ) for all i ∈ [ n ] in Algorithm 1 (DASHA-PP-PAGE) , then E [ f ( x T ) ] -f ∗ ≤ (1 -γµ ) T ∆ 0 .

Proof. Let us fix constants ν, ρ ∈ [0 , ∞ ) that we will define later. Considering Lemma 12, Lemma 8, and the law of total expectation, we obtain

After rearranging the terms, we get

therefore

Let us simplify the inequality. First, due to b ≥ page a 2 , we have

Second, due to b ≤ p p page and p ≤ p 2 , we get

Combining all bounds together, we obtain the following inequality:

Using Lemma 4 and the assumption about γ, we get

Note that γ ≤ b 2 µ , thus 1 -b 2 ≤ 1 -γµ and

It is left to apply Lemma 11 with

to conclude the proof.

F.4.5 Proof for DASHA-PP-MVR under PŁ-condition

Theorem 10. Suppose that Assumption 1, 2, 3, 7, 5, 6, 8 and 9 hold. Let us take a = p a 2 ω +1 , b ∈ ( 0 , p a 2 -p a ] ,

and h 0 i = g 0 i for all i ∈ [ n ] in Algorithm 1 (DASHA-PP-MVR) , then

Proof. Let us fix constants ν, ρ ∈ [0 , ∞ ) that we will define later. Considering Lemma 12, Lemma 10, and the law of total expectation, we obtain

After rearranging the terms, we get

By taking ν = 2 γ b , one can show that ( γ + ν (1 -b ) 2 ) ≤ ( 1 -b 2 ) ν, and

Note that 2 -p a

And if we take ρ = 40 bγω (2 ω +1) np 2 a + 8 γ ( p a -p aa ) np 2 a , then

and

therefore

Also, we can simplify the last term:

thus

Using Lemma 4 and the assumption about γ, we get

Note that γ ≤ b 2 µ , thus 1 -b 2 ≤ 1 -γµ and

It is left to apply Lemma 11 with

Corollary 5. Suppose that assumptions of Theorem 10 hold, batch size B ≤ min { σ p a √ µεn , L 2 σ ̂ L 2 } , we take Rand K compressors with K = Θ ( Bd √ µεn σ ) . Then the communication complexity equals

and the expected number of stochastic gradient calculations per node equals

Proof. In the view of Theorem 10, DASHA-PP have to run

communication rounds in the stochastic settings to get ε -solution. Note that K = O ( d p a √ n ) . Moreover, we can skip the initialization procedure and initialize h 0 i and g 0 i , for instance, with zeros because the initialization error is under a logarithm. Considering Theorem 6, the communication complexity equals

The expected number of stochastic gradient calculations per node equals

G Description of DASHA-PP-SYNC-MVR

By analogy to (Tyurin and Richt´ arik, 2023), we provide a 'synchronized' version of the algorithm. With a small probability, participating nodes calculate and send a mega batch without compression. This helps us to resolve the suboptimality of DASHA-PP-MVR w.r.t. ω. Note that this suboptimality is not a problem. We show in Corollary 4 that DASHA-PP-MVR can have the optimal oracle complexity and SOTA communication complexity with the particular choices of parameters of the compressors.

Algorithm 8 DASHA-PP-SYNC-MVR

In the following theorem, we provide the convergence rate of DASHA-PP-SYNC-MVR .

Theorem 11. Suppose that Assumptions 1, 2, 3, 5, 6, 7 and 8 hold. Let us take a = p a 2 ω +1 , b = p mega p a 2 -p a , probability p mega ∈ (0 , 1] , batch size B ′ ≥ B ≥ 1

and h 0 i = g 0 i for all i ∈ [ n ] in Algorithm 8. Then

First, we introduce the expected density of compressors (Gorbunov et al., 2021; Tyurin and Richt´ arik, 2023).

Definition 12. The expected density of the compressor C i is ζ C i := sup x ∈ R d E[ ∥C i ( x ) ∥ 0 ] , where ∥ x ∥ 0 is the number of nonzero components of x ∈ R d . Let ζ C = max i ∈ [ n ] ζ C i .

Note that ζ C is finite and ζ C ≤ d.

In the next corollary, we choose particular algorithm parameters to reveal the communication and oracle complexity.

Corollary 6. Suppose that assumptions from Theorem 11 hold, probability p mega = min { ζ C d , nεB σ 2 }

,

batch size B ′ = Θ ( σ 2 nε ) , and h 0 i = g 0 i = 1 B init ∑ B init k =1 ∇ f i ( x 0 ; ξ 0 ik ) for all i ∈ [ n ] , initial batch size B init = Θ ( B p mega √ p a ) = Θ ( max { Bd √ p a ζ C , σ 2 √ p a nε }) , then DASHA-PP-SYNC-MVR needs

communication rounds to get an ε -solution, the expected communication complexity is equal to O ( d + ζ C T ) , and the expected number of stochastic gradient calculations per node equals O ( B init + BT ) , where ζ C is the expected density from Definition 12.

The main improvement of Corollary 6 over Corollary 3 is the size of the initial batch size B init . However, Corollary 4 reveals that we can avoid regimes when DASHA-PP-MVR is suboptimal.

We also provide a theorem under PŁ-condition (see Assumption 9).

Theorem 13. Suppose that Assumptions 1, 2, 3, 5, 6, 7, 8 and 9 hold. Let us take a = p a 2 ω +1 , b = p mega p a 2 -p a , probability p mega ∈ (0 , 1] , batch size B ′ ≥ B ≥ 1 ,

and h 0 i = g 0 i for all i ∈ [ n ] in Algorithm 8. Then

Let us provide bounds up to logarithmic factors and use ˜ O ( · ) notation. Corollary 7. Suppose that assumptions from Theorem 13 hold, probability p mega = min { ζ C d , µnεB σ 2 } , batch size B ′ = Θ ( σ 2 µnε ) then DASHA-PP-SYNC-MVR needs

communication rounds to get an ε -solution, the expected communication complexity is equal to ˜ O ( ζ C T ) , and the expected number of stochastic gradient calculations per node equals ˜ O ( BT ) , where ζ C is the expected density from Definition 12.

The proof of this corollary almost repeats the proof of Corollary 6. Note that we can skip the initialization procedure and initialize h 0 i and g 0 i , for instance, with zeros because the initialization error is under a logarithm.

Let us assume that d ζ C = Θ( ω ) (holds for the Rand K compressor), then the convergence rate of DASHA-PP-SYNC-MVR is

Comparing (34) with the rate of DASHA-PP-MVR (31), one can see that DASHA-PP-SYNC-MVR improves the suboptimal term P 2 from (31). However, Corollary 5 reveals that we can escape these suboptimal regimes by choosing the parameter K of Rand K compressors in a particular way.

G.1 Proof for DASHA-PP-SYNC-MVR

In this section, we provide the proof of the convergence rate for DASHA-PP-SYNC-MVR . There are four different sources of randomness in Algorithm 8: the first one from random samples ξ t +1 · , the second one from compressors {C i } n i =1 , the third one from availability of nodes, and the fourth one from c t +1 . We define E k [ · ] , E C [ · ] , E p a [ · ] and E p mega [ · ] to be conditional expectations w.r.t. ξ t +1 · , {C i } n i =1 , availability, and c t +1 , accordingly, conditioned on all previous randomness. Moreover, we define E t +1 [ · ] to be a conditional expectation w.r.t. all randomness in iteration t +1 conditioned on all previous randomness.

Let us denote

h t +1 1 := 1 n ∑ n i =1 h t +1 i, 1 , h t +1 2 := 1 n ∑ n i =1 h t +1 i, 2 , g t +1 1 := 1 n ∑ n i =1 g t +1 i, 1 , and g t +1 2 := 1 n ∑ n i =1 g t +1 i, 2

First, we will prove two lemmas.

Lemma 13. Suppose that Assumptions 3, 5, 7 and 8 hold and let us consider sequences { g t +1 i } n i =1 and { h t +1 i } n i =1 from Algorithm 8, then

and

Proof. First, we get the bound for E t +1 [ ∥ ∥ g t +1 -h t +1 ∥ ∥ 2 ] :

Using

and

we have

and

In the last inequality, we use Assumption 7. Next, using (15), we have

The second inequality can be proved almost in the same way:

Lemma 14. Suppose that Assumptions 3, 5, 6 and 8 hold and let us consider sequence { h t +1 i } n i =1 from Algorithm 8, then

and

Proof. First, we prove the bound for E k [ E p a [ E p mega [ ∥ ∥ h t +1 -∇ f ( x t +1 ) ∥ ∥ 2 ]]] . Using E k [ E p a [ h t +1 i, 1 ]]

and we have

We can use Lemma 1 two times with i) r i = h t i and s i = k t +1 i, 1 and ii) r i = h t i and s i = k t +1 i, 2 , to obtain

where we used independence of the mini-batch samples. Using (15), we get

Due to Assumptions 5 and 6, we have

Next, we estimate the bound for E k [ ∥ ∥ k t +1 i, 2 -E k [ k t +1 i, 2 ]∥ ∥ 2 ] .

Due to Assumptions 6, we have

Plugging (36) and (37) into (35), we obtain

Using Assumption 3, we get

Using almost the same derivations, we can prove the second inequality:

Using (36) and (37), we get

Next, due to Assumption 3, we obtain

The third inequality can be proved with the help of (37) and Assumption 3.

Theorem 11. Suppose that Assumptions 1, 2, 3, 5, 6, 7 and 8 hold. Let us take a = p a 2 ω +1 , b = p mega p a 2 -p a , probability p mega ∈ (0 , 1] , batch size B ′ ≥ B ≥ 1

, and h 0 i = g 0 i for all i ∈ [ n ] in Algorithm 8. Then E [ ∥ ∥ ∇ f ( ̂ x T ) ∥ ∥ 2 ] ≤ 1 T     2∆ 0 γ + 4 p mega p a ∥ ∥ h 0 -∇ f ( x 0 ) ∥ ∥ 2 + 4 ( 1 -p aa p a ) np mega p a 1 n n ∑ i =1 ∥ ∥ h 0 i -∇ f i ( x 0 ) ∥ ∥ 2    

Proof. Due to Lemma 2 and the update step from Line 5 in Algorithm 8, we have E t +1 [ f ( x t +1 ) ]

Let us fix constants κ, η, ν, ρ ∈ [0 , ∞ ) that we will define later. Considering Lemma 13, Lemma 14, and the law of total expectation, we obtain

Let us simplify the last inequality. Since B ′ ≥ B and b = p mega p a 2 -p a ≤ p mega , we have 1 -p mega ≤ 1 , and

Thus

After rearranging the terms, we get

Let us take

where simplified the term using p aa ≥ 0 . Let us take ν = γ b to obtain

Next, we take ρ = 2 γ ( p a -p aa ) np 2 a p mega , thus

Since p mega p a 2 ≤ b ≤ p mega p a and 1 -p a ≤ 1 -p aa p a ≤ 1 , we get

Using Lemma 4 and the assumption about γ, we get

It is left to apply Lemma 3 with

and C = 6 σ 2 nB ′ to conclude the proof.

Corollary 6. Suppose that assumptions from Theorem 11 hold, probability p mega = min { ζ C d , nεB σ 2 } , batch size B ′ = Θ ( σ 2 nε ) , and h 0 i = g 0 i = 1 B init ∑ B init k =1 ∇ f i ( x 0 ; ξ 0 ik ) for all i ∈ [ n ] , initial batch size B init = Θ ( B p mega √ p a ) = Θ ( max { Bd √ p a ζ C , σ 2 √ p a nε }) , then DASHA-PP-SYNC-MVR needs

communication rounds to get an ε -solution, the expected communication complexity is equal to O ( d + ζ C T ) , and the expected number of stochastic gradient calculations per node equals O ( B init + BT ) , where ζ C is the expected density from Definition 12.

Proof. Due to the choice of B ′ , we have

Using and

we have

Therefore, we can take the following T to get ε -solution.

Considering the choice of p mega and B init , we obtain

The expected communication complexity equals O ( d + p mega d +(1 -p mega ) ζ C ) = O ( d + ζ C ) and the expected number of stochastic gradient calculations per node equals O ( B init + p mega B ′ +(1 -p mega ) B ) = O ( B init + B ) .

Theorem 13. Suppose that Assumptions 1, 2, 3, 5, 6, 7, 8 and 9 hold. Let us take a = p a 2 ω +1 , b = p mega p a 2 -p a , probability p mega ∈ (0 , 1] , batch size B ′ ≥ B ≥ 1 ,

and h 0 i = g 0 i for all i ∈ [ n ] in Algorithm 8. Then

Proof. Let us fix constants κ, η, ν, ρ ∈ [0 , ∞ ) that we will define later. As in the proof of Theorem 11, we can get

Let us take κ = 2 γ a , thus γ + κ (1 -a ) 2 ≤ ( 1 -a 2 ) κ and

Next, since a = p a 2 ω +1 , we have ( (2 ω +1 -p a ) a 2 p a +(1 -a ) 2 ) ≤ 1 -a. We the choice η = 2 γ ((2 ω +1) p a -p aa ) np 2 a , we guarantee γ ((2 ω +1) p a -p aa ) a np 2 a + η ( (2 ω +1 -p a ) a 2 p a +(1 -a ) 2 ) ≤ ( 1 -a 2 ) η and

where simplified the term using p aa ≥ 0 . Let us take ν = 2 γ b to obtain

Using Lemma 4 and the assumption about γ, we get

Due to γ ≤ a 2 µ and γ ≤ b 2 µ , we have

It is left to apply Lemma 11 with

and C = 20 σ 2 nB ′ to conclude the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis research focuses on distributed learning problems and introduces a novel approach that incorporates three key elements: variance reduction, compression, and partial participation. It extends the DASHA method to address the partial participation scenario in both finite-sum and stochastic scenarios. In the context of partial participation, the proposed method achieves state-of-the-art communication complexity. Furthermore, in the scenario involving variance reduction and partial participation (without compression), it achieves optimal oracle complexity without relying on strict bounded gradient assumptions.\n\nSTRENGTHS:\nThe extension of DASHA to accommodate the partial participation setting is nontrivial addition. The analysis involving partial participation introduces additional complexity, but this work successfully establishes state-of-the-art bounds (although not that surprising).\n\nWEAKNESSES:\nThe presentation can be further improved in several ways to enhance clarity and readability. Firstly, instead of listing the assumptions and results for all three settings (full gradient, finite sum, stochastic) consecutively, it would be beneficial to present each setting individually with a particular focus on the stochastic setting, which is the main focus of this work. This approach would make it easier for the reader to understand and follow the discussion. Additionally, by adopting this approach, it would be possible to cover the results for both the nonconvex and PL condition cases in the main text.\n\nFurthermore, there are a few minor typos that need to be addressed. The set $\\mathbb{R}^d$ is written as $\\mathbb{R}$ in various places, specifically lines 35, 44, 51, and 59. Additionally, in line 164, \"what\" should be replaced with \"which.\" Finally, in line 405, \"dependeces\" should be corrected to \"dependencies.\"\n\nAnother aspect that requires attention is the presentation of the algorithm. Currently, it lacks motivation and clarity. For instance, it is unclear how step (9) is derived to address the issue in steps (7) to (8), where it is mentioned that these updates are adapted to ensure the validity of the proof. Providing a clearer explanation or justification for step (9) would greatly improve the understanding of the algorithm and its purpose.\n\n\nAlthough the specific setup presented in this study has not been previously explored, and its analysis poses challenges, it can be viewed as a minor blend of previous setups, and the results can be considered as a slight extension of the work done in DASHA.\n\nFinally, the scope of the experiments conducted in this research is somewhat limited, as they do not encompass the more intriguing nonconvex neural network scenarios. Furthermore, there is a notable absence of numerical comparisons with other methods, such as FRECON or any other method within the full gradient setup. While I understand that space constraints may be a factor, with careful organization, it may be possible to allocate some room for such comparisons. One possible approach could involve dedicating the main body solely to the stochastic setting, while relegating the finite-sum case to the appendix. This would create additional space for the inclusion of numerical comparisons and provide a more comprehensive evaluation of the proposed approach.\n\nQUESTIONS:\nAfter examining equations (7)-(8), it is mentioned that step (7) requires further updating if the same idea as in the full gradient step is utilized. The question arises as to why this is necessary. Would the method still function if only the participating nodes updated $h$ while the others retained the previous values? Essentially, is this a limitation of the analysis or would the method diverge under such circumstances?\n\nAnother query pertains to practical applications that satisfy Assumption 6. Is there a specific real-world scenario that meets this criterion? Additionally, it is worth exploring whether achieving state-of-the-art bounds in the stochastic setting is possible without relying on Assumption 6.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel method for distributed optimization and federated learning that integrates three critical components: variance reduction of stochastic gradients, partial participation, and compressed communication. The proposed method is shown to have optimal oracle complexity and leading-edge communication complexity in a setting of partial participation. A significant advantage of this method is that it effectively blends variance reduction and partial participation, thus providing optimal oracle complexity without the necessity of all nodes' participation or the bounded gradients assumption.\n\nSTRENGTHS:\nOne of the primary strengths of this paper is the integration of three crucial elements in distributed optimization and federated learning. By encompassing variance reduction of stochastic gradients, partial participation, and compressed communication into a single method, the authors have created a holistic approach that takes into account the multifaceted nature of distributed optimization.\n\nThe authors demonstrate that their method achieves optimal oracle complexity, which is an important measure of the efficiency of an algorithm. This is a significant contribution, suggesting that the method has a high degree of efficiency in processing the data.\n\nWEAKNESSES:\nDespite the significant theoretical contributions, this paper falls short in one crucial aspect, which restricts its overall impact and practical applicability.\n\nLack of Empirical Validation: The major limitation of this paper is the absence of empirical experiments to substantiate the theoretical findings. While the theoretical results are compelling, they need to be complemented with practical validation to establish the practicality and effectiveness of the proposed method. The lack of experimental results makes it difficult to ascertain the method's performance in real-world scenarios, to gauge its scalability with varying data sizes and distributions, and to compare it with existing methods.\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript considers distributed non-convex optimization in the federated learning setting. It proposes an algorithm DASHA-PP that brings three important features of federated learning together: i) variance reduction, ii) compressed communication, and iii) partial participation. The authors derive the oracle and communication complexities of finding an $\\epsilon$-stationary point of the smooth non-convex objective. It is the first method that includes those three important features.\n\nThe algorithm presented in the paper is built upon DASHA, presented in [1]. In particular, in its form in [1], DASHA could work in the partial participation case only when the exact gradient oracle was available. In this manuscript, the authors extended the partial participation property of DASHA to the stochastic and finite-sum settings.\n\n[1] Tyurin, Alexander and Peter Richt'arik. “DASHA: Distributed Nonconvex Optimization with Communication Compression, Optimal Oracle Complexity, and No Client Synchronization.” ArXiv abs/2202.01268 (2022): n. pag.\n\nSTRENGTHS:\n* I think the paper is very well-written.\n\n* The problem the manuscript considers is an important problem for the large-scale federated learning setting, where including all three features, i.e., partial participation/compressed communication/stochastic gradient, is inevitable.\n\nWEAKNESSES:\n* The main technical contribution of the manuscript over [1] is the update rule given in Eq. (9). Similar update rule adjustments are frequently used in bandit optimization literature, cf [Section 2, 2] and [Section 6, 2].\n\n* Besides the new update rule, it seems to me that the mathematical derivations in the manuscript are very similar to those in [1]. In that sense, I find the contribution of the manuscript is rather incremental.\n\n[1] Tyurin, Alexander and Peter Richt'arik. “DASHA: Distributed Nonconvex Optimization with Communication Compression, Optimal Oracle Complexity, and No Client Synchronization.” ArXiv abs/2202.01268 (2022): n. pag.\n\n[2] Cesa-Bianchi, Nicolò and Gábor Lugosi. “Prediction, learning, and games.” (2006).\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper injects existing algorithm for distributed optimisation in the compressed and variance reduced setting with a mechanism for partial participation of nodes. Theoretical guarantees are given in the general non-convex setting, but also under PL-condition. The theoretical results are verified empirically in a classification problem.\n\nSTRENGTHS:\nThe paper considers the setting of partial participation, which is of importance in distributed optimisation. The setting of partial participation  considered, is general enough to include at least two specific partial participation strategies. The theoretical results match the state of the art guarantees for variance reduced algorithms with limited communication up to scaling of $1/p_a$, where $p_a$ governs the probability of some node to participate in the gradient estimation during some iteration of the algorithm. I find this result elegant and I like that this scaling is pretty much observed in the experiments. The authors give some intuition on why passing from the full to the partial participation setting is non-trivial.\n\nWEAKNESSES:\nThe paper is very technical (if one wishes to follow proofs of the claims) and builds to an extensive amount of previous work, thus it is difficult to be appreciated by readers that do not belong to this specific research community. The appendix is about 90 (!) pages and a significant portion of them is essential to the proper understanding of this work, thus it's quite hard to review it in the context of neurips (the results are believable though). Since the topic is distributed computing with limited communication, I would expect a wider experimental analysis of the proposed algorithms in various tasks (also would be good some of the experiments to be in the main text).\n\nQUESTIONS:\nI struggle a bit to understand the intuition behind Algorithms 3 and 4. For algorithm 3, I don't see immediately the connection with the PAGE algorithm and, for Algorithm 4, I don't see which algorithm tries to generalise. Could you elaborate?\n\nIs it possible to show that the chosen batch sizes of $B=O((L_{max}/\\hat L)^2)$ and $O((L_{\\sigma}/\\hat L)^2)$ are at most $n$? Otherwise, there is a chance that such a batch size is not available.\n\nMinor: there are some issues with the writing, e.g.:\n\"it is not trivial in each order one should apply the expectations\"\n\"We are not fighting for the full generality\"\n\" The main goal of our paper is to develop a method for the nonconvex distributed optimization\"\n\"From the beginning of federated learning era, the partial participation...\"\n\nI would recommend to the authors to go through the text once more and improve the quality of writing.\n\nLIMITATIONS:\nThe authors do not discuss limitations of their work, but I cannot see any profound ones. Perhaps the issue of batch sizes specified in my questions deserves a bit more attention.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-23"}