Title: Don't Compress Gradients in Random Reshuffling: Compress Gradient Differences
Venue: NeurIPS 2024
Accepted: yes
Keywords: Random reshuffling

communication compression

distributed optimization

Federated Learning
Abstract: Gradient compression is a popular technique for improving communication complexity of stochastic first-order methods in distributed training of machine learning models. However, the existing works consider only with-replacement sampling of stochastic gradients. In contrast, it is well-known in practice and recently confirmed in theory that stochastic methods based on without-replacement sampling, e.g., Random Reshuffling (RR) method, perform better than ones that sample the gradients with-replacement. In this work, we close this gap in the literature and provide the first analysis of methods with gradient compression and without-replacement sampling. We first develop a distributed variant of random reshuffling with gradient compression (Q-RR), and show how to reduce the variance coming from gradient quantization through the use of control iterates. Next, to have a better fit to Federated Learning applications, we incorporate local computation and propose a variant of Q-RR called Q-NASTYA. Q-NASTYA uses local gradient steps and different local and global stepsizes. Next, we show how to reduce compression variance in this setting as well. Finally, we prove the convergence results for the proposed methods and outline several settings in which they improve upon existing algorithms.

Don't Compress Gradients in Random Reshuffling: Compress Gradient Differences

Abstract

Gradient compression is a popular technique for improving communication complexity of stochastic first-order methods in distributed training of machine learning models. However, the existing works consider only with-replacement sampling of stochastic gradients. In contrast, it is well-known in practice and recently confirmed in theory that stochastic methods based on without-replacement sampling, e.g., Random Reshuffling ( RR ) method, perform better than ones that sample the gradients with-replacement. In this work, we close this gap in the literature and provide the first analysis of methods with gradient compression and without-replacement sampling. We first develop a distributed variant of random reshuffling with gradient compression ( Q-RR ), and show how to reduce the variance coming from gradient quantization through the use of control iterates. Next, to have a better fit to Federated Learning applications, we incorporate local computation and propose a variant of Q-RR called Q-NASTYA . Q-NASTYA uses local gradient steps and different local and global stepsizes. Next, we show how to reduce compression variance in this setting as well. Finally, we prove the convergence results for the proposed methods and outline several settings in which they improve upon existing algorithms.

1 Introduction

Distributed learning plays a crucial role in the training of modern Deep Learning (DL) models since distributed approaches are able to significantly reduce training time [Goyal et al., 2017, You et al., 2019]. Moreover, distributed methods are mandatory for such applications as Federated learning (FL) [Koneˇ cný et al., 2016, McMahan et al., 2017], where multiple nodes connected over a network collaborate on a learning task. Each node possesses its own dataset and cannot share this data with other nodes or a central server. As a result, algorithms for federated learning often rely on local computation and lack access to the entire dataset of training examples. Federated learning finds applications in diverse fields, including language modeling for mobile keyboards [Liu et al., 2021],

healthcare [Antunes et al., 2022], and wireless communications [Yang et al., 2022]. Its applications extend to various other domains [Kairouz et al., 2019].

Distributed learning tasks are often solved through empirical-risk minimization (ERM), where the m -th device contributes an empirical loss function f m ( x ) representing the average loss of model x on its local dataset, and our goal is to then minimize the average loss over all the nodes:

where the function f represents the average loss. Every f m is an average of sample loss functions f i m each representing the loss of model x on the i -th datapoint on the m -th clients' dataset: that is for each m ∈ { 1 , 2 , . . . , M } we have

For simplicity we shall assume that the datasets on all clients are of equal size: n 1 = n 2 = . . . = n M , though this assumption is only for convenience and our results easily extend to the case when clients have datasets of unequal sizes. Thus our optimization problem is

Because d is often very large in practice, the dominant paradigm for solving (2) relies on first-order (gradient) information. Federated learning algorithms have access to two key primitives: (a) local computation, where for a given model x ∈ R d we can compute stochastic gradients ∇ f i m ( x ) locally on client m , and (b) communication, where the different clients can exchange their gradients or models with a central server.

1.1 Communication compression

In practice, communication is more expensive than local computation [Kairouz et al., 2019], and as such one of the chief concerns of algorithms for distributed learning is communication efficiency. Algorithms for distributed/federated learning have thus focused on achieving communication efficiency, with one common ingredient being the use of gradient compression , where each client sends a compressed or quantized version of their update instead of the full update vector, potentially saving communication bandwidth by sending fewer bits over the network. There are many operators that can be used for compressing the update vectors: stochastic quantization [Alistarh et al., 2017], random sparsification [Wangni et al., 2018, Stich et al., 2018], and others [Tang et al., 2020]. In this work we consider compression operators satisfying the following assumption:

Assumption 1. A compression operator is an operator Q : R d → R d such that for some ω > 0 , the relations

Unbiased compressors can reduce the number of bits clients communicate per round, but also increases the variance of the stochastic gradients used slowing down overall convergence, see e.g. [Khirirat et al., 2018, Theorem 5.2] and [Stich, 2020, Theorem 1]. By using control iterates, Mishchenko et al. [2019b] developed DIANA -an algorithm that can reduce the variance due to gradient compression with unbiased compression operators, and thus ensure fast convergence. DIANA has been extended and analyzed in many settings [Horváth et al., 2019, Stich, 2020, Safaryan et al., 2021] and forms an important tool in our arsenal for using gradient compression.

1.2 Random Reshuffling

Despite the importance of addressing the communication bottleneck, local computations also significantly affect the training. For simplicity, consider the 1 -node scenario. In this case, the update rule of the standard work-horse method in stochastic optimization - stochastic gradient descent ( SGD )

2

⋆

1

M

M

∑

m

=1

[Robbins and Monro, 1951] - can be written as follows: x t +1 = x t -γ ∇ f j ( x t ) , where j is sampled from { 1 , . . . , n } uniformly at random. This procedure thus uses with-replacement sampling in order to select the stochastic gradient used at each step from the dataset. However, in the training of DL models, without-replacement sampling is used much more often: that is, at the beginning of each epoch we choose a permutation π 1 , π 2 , . . . , π n of { 1 , 2 , . . . , n } and do the i -th update using the π i -ith gradient: x i +1 t = x i t -γ ∇ f π i ( x i t ) . Without-replacement sampling SGD , also known as Random Reshuffling ( RR ) [Bottou, 2009], typically achieves better asymptotic convergence rates compared to with-replacement SGD and can improve upon it in many settings as shown by recent theoretical progress [Mishchenko et al., 2020, Ahn et al., 2020, Rajput et al., 2020, Safran and Shamir, 2021]. While with-replacement SGD achieves an error proportional to O ( 1 T ) after T steps [Stich, 2019], Random Reshuffling achieves an error of O ( n T 2 ) after T steps, faster than SGD when the number of steps T is large.

1.3 Can Communication Compression and Random Reshuffling be Friends?

As we described earlier, Random Reshuffling and communication compression are two important tools for training modern DL models, and both techniques are relatively well understood. However, there are no papers that study Random Reshuffling and communication compression in combination. This leads us to the natural question: how these techniques should be combined to improve the convergence speed of existing distributed methods?

1.4 Contributions

In this paper, we aim to develop methods for Distributed and Federated Learning that combine gradient compression and random reshuffling. While each of these techniques can aid in reducing the communication complexity of distributed optimization, their combination is under-explored. Thus our goal is to design methods that improve upon existing algorithms in convergence rates and in practice. We summarize our contributions as follows.

2

ζ

=

∥∇

f

m

(

x

⋆

)

∥

. The results of this paper are highlighted in blue.

⋄ The issue: naïve combination has no improvements. As a natural step towards our goal, we propose and study a new algorithm, Q-RR (Algorithm 1), that combines random reshuffling with gradient compression at every communication round. However, for Q-RR our theoretical results do not show any improvement upon QSGD when the compression level is reasonable (see Table 1). Moreover, we observe similar performance of Q-RR and QSGD in various numerical experiments. Therefore, we conclude that this phenomenon is not an artifact of our analysis but rather an issue of Q-RR : communication compression adds an additional noise that dominates the one coming from the stochastic gradients sampling.

⋄ The remedy: reduction of compression variance. To remove the additional variance added by the compression and unleash the potential of Random Reshuffling in distributed learning with compression, we propose DIANA-RR (Algorithm 2), a combination of Q-RR and the DIANA algorithm. We derive the convergence rates of the new method and show that it improves upon the convergence rates of Q-RR , QSGD , and DIANA (see Table 1). We point out that to achieve such results we use n shift-vectors per worker in DIANA-RR unlike DIANA that uses only 1 shift-vector.

⋄ Extensions to the local steps. Inspired by the NASTYA algorithm of Malinovsky et al. [2022], we propose a variant of NASTYA , Q-NASTYA (Algorithm 3), that naïvely mixes quantization, local steps with random reshuffling, and uses different local and server stepsizes. Although it improves in per-round communication cost over NASTYA but, similar to Q-RR , we show that Q-NASTYA suffers from added variance due to gradient quantization. To overcome this issue, we propose another algorithm, DIANA-NASTYA (Algorithm 4), that adds DIANA -style variance reduction to Q-NASTYA and removes the additional variance.

Finally, to illustrate our theoretical findings we conduct experiments on federated logistic regression tasks and on distributed training of neural networks.

2 Algorithms and convergence theory

We will primarily consider the setting of strongly-convex and smooth optimization. We assume that the average function f is strongly convex:

Assumption 2. Function f : R d → R is µ -strongly convex, i.e., for all x, y ∈ R d ,

and functions f i 1 , f i 2 , . . . , f i M : R d → R are convex for all i = 1 , . . . , n .

Examples of objectives satisfying Assumption 2 include ℓ 2 -regularized linear and logistic regression. Throughout the paper, we assume that f has the unique minimizer x ⋆ ∈ R d . We also use the assumption that each individual loss f i m is smooth, i.e. has Lipschitz-continuous first-order derivatives:

Assumption 3. Function f i m : R d → R is L i,m -smooth for every i ∈ [ n ] and m ∈ [ M ] , i.e., for all x, y ∈ R d and for all m ∈ [ M ] and i ∈ [ n ] ,

We denote the maximal smoothness constant as L max def = max i,m L i,m .

For some methods, we shall additionally impose the assumption that each function is strongly convex:

Assumption 4. Each function f i m : R d → R is ˜ µ -strongly convex.

The Bregman divergence associated with a convex function h is defined for all x, y ∈ R d as

Note that the inequality (3) defining strong convexity can be written as D f ( x, y ) ≥ µ 2 ∥ x -y ∥ 2 .

2.1 Algorithm Q-RR

The first method we introduce is Q-RR (Algorithm 1). Q-RR is a straightforward combination of distributed random reshuffling and gradient quantization. This method can be seen as the stochastic without-replacement analogue of the distributed quantized gradient method of Khirirat et al. [2018].

Algorithm 1 Q-RR: Distributed Random Reshuffling with Quantization

Input: x - starting point, γ > 0 - stepsize

0 1: for t = 0 , 1 , . . . , T -1 do 2: Receive x t from the server and set x 0 t,m = x t 3: Sample random permutation of [ n ] : π m = ( π 0 m , . . . , π n -1 m ) 4: for i = 0 , 1 , . . . , n -1 do 5: for m = 1 , . . . , M in parallel do 6: Receive x i t from the server, compute and send Q ( ∇ f π i m m ( x i t ) ) back 7: end for 8: Compute and send x i +1 t = x i t -γ 1 M ∑ M m =1 Q ( ∇ f π i m m ( x i t ) ) to the workers 9: end for 10: x t +1 = x n t 11: end for Output: x T

We shall use the notion of shuffling radius defined by Mishchenko et al. [2021] for the analysis of distributed methods with random reshuffling:

Definition 2.1. Define the iterate sequence x i +1 ⋆ = x i ⋆ -γ M ∑ M m =1 ∇ f π i m m ( x ⋆ ) . Then the shuffling radius is the quantity

We provide clarifications regarding this term in Appendix C.1. To compare our subsequent results with known ones, we introduce bounds on the shuffling radius. The following lemma demonstrates that these bounds are independent of the stepsize γ , even though γ is used in Definition 2.1.

Lemma 2.1 ([Mishchenko et al., 2020]) . Let Assumptions 3, 4 hold. Then the shuffling radius σ 2 rad satisfies the following inequlity

We now state the main convergence theorem for Algorithm 1:

Theorem 2.1. Let Assumptions 1, 3, 4 hold and let the stepsize satisfy 0 < γ ≤ 1 ( 1+2 ω M ) L max . Then, for all T ≥ 0 the iterates produced by Q-RR (Algorithm 1) satisfy

All proofs are relegated to the appendix. By choosing the stepsize γ properly, we can obtain the communication complexity (number of communication rounds) needed to find an ε -approximate solution as follows:

Corollary 1. Under the same conditions as Theorem 2.1 and for Algorithm 1, there exists a stepsize γ > 0 such that the number of communication rounds nT to find a solution with accuracy ε > 0 (i.e. E ∥ x T -x ⋆ ∥ 2 ≤ ϵ ) is equal to ˜ O ( ( 1 + ω M ) L max ˜ µ + ω ( ζ 2 ⋆ + σ 2 ⋆ ) M ˜ µ 2 ε + σ rad √ ˜ µ 3 ε ) , where ˜ O ( · ) hides constants and logarithmic factors.

The complexity of Quantized SGD ( QSGD ) is [Gorbunov et al., 2020]: ˜ O ( ( 1 + ω M ) L max µ + ( ωζ 2 ⋆ +(1+ ω ) σ 2 ⋆ ) Mµ 2 ε ) . For simplicity, let us neglect the differences between

Algorithm 2 DIANA-RR

Input: x 0 - starting point, { h i 0 ,m } M,n m,i =1 , 1 - initial shift-vectors, γ > 0 - stepsize, α > 0 - stepsize for learning the shifts

1: for t = 0 , 1 , . . . , T -1 do

2: Receive x t from the server and set x 0 t,m = x t

3: Sample random permutation of [ n ] : π m = ( π 0 m , . . . , π n -1 m )

4: for i = 0 , 1 , . . . , n -1 do

5: for m = 1 , 2 , . . . , M in parallel do

6: Receive x i t from the server, compute and send Q ( ∇ f π i m m ( x i t ) -h π i m t,m ) back

9: end for

11: end for

10: Compute x i +1 t = x i t -γ 1 M ∑ M m =1 ˆ g π i m t,m and send x i +1 t to the workers

12: x t +1 = x n t

13: end for

Output:

x

T

µ and ˜ µ . First, when ω = 0 we recover the complexity of FedRR [Mishchenko et al., 2021] which is known to be better than the one of SGD as long as ε is sufficiently small as we have nµσ 2 ⋆,n / 8 ≤ σ 2 rad ≤ nLσ 2 ⋆,n / 4 from Lemma 2.1. Next, when M = 1 and ω = 0 (single node, no compression) our results recovers the rate of RR [Mishchenko et al., 2020].

However, it is more interesting to compare Q-RR and QSGD when M > 1 and ω > 1 , which is typically the case. In these settings, Q-RR and QSGD have the same complexity since the O ( 1 / ε ) term dominates the O ( 1 / √ ε ) one if ε is sufficiently small. That is, the derived result for Q-RR has no advantages over the known one for QSGD unless ω is very small, which means that there is almost no compression at all. We also observe this phenomenon in the experiments.

The main reason for that is the variance appearing due to compression. Indeed, even if the current point is the solution of the problem ( x i t = x ∗ ), the update direction -γ 1 M ∑ M m =1 Q ( ∇ f π i m m ( x i t ) ) has the compression variance

This upper bound is tight and non-zero in general. Moreover, it is proportional to γ 2 that creates the term proportional to γ in (5) like in the convergence results for QSGD / SGD , while the RR -variance is proportional to γ 2 in the same bound. Therefore, during the later stages of the convergence Q-RR behaves similarly to QSGD when we decrease the stepsize.

2.2 Algorithm DIANA-RR

To reduce the additional variance caused by compression, we apply DIANA -style shift sequences [Mishchenko et al., 2019b, Horváth et al., 2019]. Thus we obtain DIANA-RR (Algorithm 2), which applies compression to the differences between the gradients and learnable shifts. Since the shifts are updated using the past gradients information, one can see DIANA-RR as a method with compression of gradient differences. We notice that unlike DIANA , DIANA-RR has n shift-vectors on each node.

Theorem 2.2. Let Assumptions 1, 3, 4 hold and suppose that the stepsizes satisfy γ ≤ min { α 2 n ˜ µ , 1 ( 1+ 6 ω M ) L max } , and α ≤ 1 1+ ω . Define the following Lyapunov function for every t ≥ 0

Algorithm 3 Q-NASTYA

Input: x 0 - starting point, γ > 0 - local stepsize, η > 0 - global stepsize

1: for t = 0 , 1 , . . . , T -1 do

2: for m ∈ [ M ] in parallel do

3: Receive x t from the server and set x 0 t,m = x t

4: Sample random permutation of [ n ] : π m = ( π 0 m , . . . , π n -1 m )

5: for i = 0 , 1 , . . . , n -1 do

6: Set x i +1 t,m = x i t,m -γ ∇ f π i m m ( x i t,m )

7: end for

8: Compute g t,m = 1 γn ( x t -x n t,m ) and send Q t ( g t,m ) to the server

9: end for

11: Compute x t +1 = x t -ηg t and send x t +1 to the workers

10: Compute g t = 1 M ∑ M m =1 Q t ( g t,m )

12: end for

Output: x T

where ∆ j t +1 ,m = h π j m t +1 ,m - ∇ f π j m m ( x ⋆ ) Then, for all T ≥ 0 the iterates produced by DIANA-RR (Algorithm 2) satisfy

Corollary 2. Under the same conditions as Theorem 2.2 and for Algorithm 2, there exists stepsizes γ, α > 0 such that the number of communication rounds nT to find a solution with accuracy ε > 0 is ( ) .

Unlike Q-RR / QSGD / DIANA , DIANA-RR does not have a ˜ O ( 1 / ε ) -term, which makes it superior to Q-RR / QSGD / DIANA for small enough ε . However, the complexity of DIANA-RR has an additive ˜ O ( n (1 + ω )) term arising due to learning the shifts { h i t,m } m ∈ [ M ] ,i ∈ [ n ] . Nevertheless, this additional term is not the dominating one when ε is small enough. Next, we elaborate a bit more on the comparison between DIANA and DIANA-RR . That is, DIANA has ˜ O ( ( 1 + ω M ) L max µ + (1+ ω ) σ 2 ⋆ Mµ 2 ε ) complexity [Gorbunov et al., 2020]. Neglecting the differences between µ and ˜ µ , we observe a similar relation between DIANA-RR and DIANA as between RR and SGD : instead of the term O ( (1+ ω ) σ 2 ⋆ / ( Mµ 2 ε ) ) appearing in the complexity of DIANA , DIANA-RR has O ( σ rad / √ ε ˜ µ 3 ) term much better depending on ε . To the best of our knowledge, our result is the only known one establishing the theoretical superiority of RR to regular SGD in the context of distributed learning with gradient compression. Moreover, when ω = 0 (no compression) we recover the rate of FedRR and when additionally M = 1 (single worker) we recover the rate of RR .

2.3 Algorithms with Local Steps

In this subsection, we study a new variant of NASTYA , Q-NASTYA (Algorithm 3), that unifies quantization, local steps with random reshuffling, and uses different local and server stepsizes. Although it improves in per-round communication cost over NASTYA but, similar to Q-RR , we show that Q-NASTYA suffers from added variance due to gradient quantization. To overcome this issue, we propose another algorithm, DIANA-NASTYA (Algorithm 4), that adds DIANA -style variance reduction to Q-NASTYA and removes the additional variance.

Theorem 2.3. Let Assumptions 1, 2, 3 hold. Let the stepsizes γ , η satisfy 0 < η ≤ 1 16 L max ( 1+ ω M ) , 0 < γ ≤ 1 5 nL max . Then, for all T ≥ 0 the iterates produced by Q-NASTYA (Algorithm 3) satisfy

Corollary 3. Under the same conditions as Theorem E.1 and for Algorithm 3, there exist stepsizes γ = η / n and η > 0 such that the number of communication rounds T to find a solution with accuracy

Algorithm 4 DIANA-NASTYA

Input: x 0 - starting point, { h 0 ,m } M m =1 - initial shift-vectors, γ > 0 - local stepsize, η > 0 - global stepsize, α > 0 - stepsize for learning the shifts

1: for t = 0 , 1 , . . . , T -1 do

2: for m = 1 , . . . , M in parallel do

3: Receive x t from the server and set x 0 t,m = x t

4: Sample random permutation of [ n ] : π m = ( π 0 m , . . . , π n -1 m )

5: for i = 0 , 1 , . . . , n -1 do

6:

Set

x

=

x

7: end for

11: end for

13: ˆ g t = 1 M ∑ m =1 ˆ g t,m = h t + 1 M ∑ m =1 Q t ( g t,m -h t,m )

14: x t +1 = x t -η ˆ g t

15: end for

Output: x T

We emphasize several differences with the known theoretical results. First, the FedCOM method of Haddadpour et al. [2021] was analyzed in the homogeneous setting only, i.e., f m ( x ) = f ( x ) for all m ∈ [ M ] , which is an unrealistic assumption for FL applications. In contrast, our result holds in the fully heterogeneous case. Next, the analysis of FedPAQ of Reisizadeh et al. [2020] uses a bounded variance assumption, which is also known to be restrictive. Nevertheless, let us compare to their result. Reisizadeh et al. [2020] derive the following complexity for their method: ˜ O ( L max µ ( 1 + ω M ) + ω M σ 2 µ 2 ε + σ 2 Mµ 2 ε ) . This result is inferior to the one we show for Q-NASTYA :

when ω is small, the main term in the complexity bound of FedPAQ is ˜ O ( 1 / ε ) , while for Q-NASTYA the dominating term is of the order ˜ O ( 1 / √ ε ) (when ω and ε are sufficiently small). We also highlight that FedCRR [Malinovsky and Richtárik, 2022] does not converge if ω > M 2 γµε / (2 ∥ x n ∗ ,m ∥ 2 ) , while Q-NASTYA does for any ω ≥ 0 . Finally, when ω = 0 (no compression) we recover NASTYA as a special case, and using γ = η / n , we recover the rate of FedRR [Mishchenko et al., 2021].

Theorem 2.4. Let Assumptions 1, 2, 3 hold. Suppose the stepsizes γ , η, α satisfy 0 < γ ≤ 1

16 L max n ,

Then, for all T ≥ 0 the iterates produced by DIANA-NASTYA (Algorithm 4) satisfy

Corollary 4. Under the same conditions as Theorem E.2 and for Algorithm 4, there exist stepsizes γ = η / n , η > 0 , α > 0 such that the number of communication rounds T to find a solution with accuracy ε > 0 is ˜ O ( ω + L max µ ( 1 + ω M ) + √ L max εµ 3 √ ζ 2 ⋆ + σ 2 ⋆ n ) . If γ → 0 , one can choose η > 0 such that the above complexity bound improves to ˜ O ( ω + L max µ ( 1 + ω M ) ) .

i

+1

t,m t,m

i

-

γ

∇

f

π

m

i m

(

x

i

t,m

)

1000

3000

5000

Data passes

10

7

10

5

10

3

10

1

f

(

xt

)

f

(

x

)

mushrooms; Rand-2

QSGD

Q-RR

DIANA

DIANA-RR

DIANA-RR-1S

1000

3000

5000

Data passes

10

7

10

5

10

3

10

1

f

(

xt

)

f

(

x

)

w8a; Rand-6

QSGD

Q-RR

DIANA

DIANA-RR

DIANA-RR-1S

1000

3000

5000

Data passes

10

7

10

5

10

3

10

1

f

(

xt

)

f

(

x

)

a9a; Rand-2

QSGD

Q-RR

DIANA

DIANA-RR

DIANA-RR-1S

Figure 1: Non-local methods

1000

3000

5000

Data passes

10

3

10

1

f

(

xt

)

f

(

x

)

mushrooms; Rand-2

Q-NASTYA

DIANA-NASTYA

FedCOM

FedPAQ

1000

3000

5000

Data passes

10

3

10

1

f

(

xt

)

f

(

x

)

w8a; Rand-6

Q-NASTYA

DIANA-NASTYA

FedCOM

FedPAQ

1000

3000

5000

Data passes

10

3

10

1

f

(

xt

)

f

(

x

)

a9a; Rand-2

Q-NASTYA

DIANA-NASTYA

FedCOM

FedPAQ

In contrast to Q-NASTYA , DIANA-NASTYA does not suffer from the ˜ O ( 1 / ε ) term in the complexity bound. This shows the superiority of DIANA-NASTYA to Q-NASTYA . Next, FedCRR-VR [Malinovsky and Richtárik, 2022] has the rate ˜ O ( ( ω +1) ( 1 -1 κ ) n ( 1 -( 1 -1 κ ) n ) 2 + √ κ ( ζ ⋆ + σ ⋆ ) µ √ ε ) , which depends on ˜ O ( 1 / √ ε )

However, the first term is close to ˜ O ( ( ω +1) κ 2 ) for a large condition number. FedCRR-VR-2 utilizes variance reduction technique from Malinovsky et al. [2021] and it allows to get rid of

additional assumption on number of functions n and thus not directly comparable with our result. Note that if we have no compression ( ω = 0) , DIANA-NASTYA recovers rate of NASTYA .

In Appendix J, we provide versions of Q-NASTYA and DIANA-NASTYA with partial participation of clients, which is another important aspect of FL, and derive the convergence results for them.

3 Experiments

We evaluated our methods for solving logistic regression problems and training neural networks in three parts: (i) Comparison of the proposed non-local methods with existing baselines; (ii) Comparison of the proposed local methods with existing baselines; (iii) Comparison of the proposed non-local methods in training ResNet-18 on CIFAR10 .

Logistic Regression. To confirm our theoretical results we conducted several numerical experiments on binary classification problem with L2 regularized logistic regression of the form

where f m,i def = log ( 1 + exp( -y mi a ⊤ mi x ) ) + λ ∥ x ∥ 2 2 ( a mi , y mi ) ∈ R d × ∈ {-1 , 1 } , i = 1 , . . . , n m are the training data samples stored on machines m = 1 , . . . , M , and λ > 0 is a regularization parameter. In all experiments, for each method, we used the largest stepsize allowed by its theory

multiplied by some individually tuned constant multiplier. For better parallelism, each worker m uses mini-batches of size ≈ 0 . 1 n m . In all algorithms, as a compression operator Q , we use Randk [Beznosikov et al., 2020] with fixed compression ratio k / d ≈ 0 . 02 , where d is the number of features in the dataset.

In our first experiment (see Figure 1), we compare Q-RR , DIANA-RR , and DIANA-RR-1S with classical baselines ( QSGD [Alistarh et al., 2017], DIANA [Mishchenko et al., 2019b]) that use a withreplacement mini-batch SGD estimator. DIANA-RR-1S is a memory-friendly version of DIANA-RR

that stores and uses a single shift h t,m on the worker side rather than n individual shifts h π i m t,m . Figure 1 illustrates that Q-RR exhibits similar behavior to QSGD , with both methods being slower than DIANA methods across all considered datasets. DIANA-RR-1S and DIANA show comparable convergence rates, indicating that random reshuffling alone, without introducing additional shifts, does not make a significant difference. Finally, DIANA-RR achieves the best rate among all considered non-local methods, efficiently reducing the variance and reaching the lowest functional sub-optimality tolerance. These experimental results align perfectly with our theoretical analysis.

The second experiment shows that DIANA -based method can significantly outperform in practice when one applies it to local methods as well. In particular, whereas Q-NASTYA shows comparative behavior as existing methods FedCOM [Haddadpour et al., 2021], FedPAQ [Reisizadeh et al., 2020] in all considered datasets, DIANA-NASTYA noticeably outperforms other methods.

0

2

4

6

8

#bits/n

1e7

20

40

60

Top1 acc (validation)

QSGD

= 0.001

QSGD-RR

= 0.001

0

1

2

3

#bits/n

1e11

25

50

75

Top1 acc (validation)

DIANA

=1.0

DIANA-RR

=1.0

Training Deep Neural Network model: ResNet-18 on CIFAR-10. Since random reshuffling is a very popular technique in training neural networks, it is natural to test the proposed methods on such problems. Therefore, in the second set of experiments, we consider training ResNet-18 [He et al., 2016] model on the CIFAR10 dataset Krizhevsky and Hinton [2009]. To conduct these experiments we use FL_PyTorch simulator [Burlachenko et al., 2021].

The main goal of this experiment is to verify the phenomenon observed in Experiment 1 on the training of a deep neural network. That is, we tested Q-RR , QSGD , DIANA , and DIANA-RR in the distributed training of ResNet-18 on CIFAR10 , see Figure 4. As in the logistic regression experiments, we observe that (i) Q-RR and QSGD behave similarly and (ii) DIANA-RR outperforms DIANA . For further experimental results and details, we refer to Appendix B.

4 Conclusion

In this work, we provide the first study of distributed random reshuffling with communication compression. Our theoretical and empirical findings illustrate the inefficiency of naïve combination of random reshuffling and communication compression. We also show how this issue can be resolved via the usage of shifts for communication compression. Finally, we develop and analyze methods with random reshuffling, communication compression, and local steps. It is worth mentioning that although our theoretical results are obtained for strongly convex problems, the considered methods perform well in the experiments on non-convex tasks like training neural networks.

References

Kwangjun Ahn, Chulhee Yun, and Suvrit Sra. SGD with shuffling: optimal rates without component convexity and large epoch requirements. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/ hash/cb8acb1dc9821bf74e6ca9068032d623-[]Abstract.html .

Dan Alistarh, Demjan Grubic, Jerry Z. Li, Ryota Tomioka, and Milan Vojnovic. QSGD: Communication-efficient SGD via gradient quantization and encoding. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 1707-1718, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.

Rodolfo Stoffel Antunes, Cristiano André da Costa, Arne Küderle, Imrana Abdullahi Yari, and Björn Eskofier. Federated learning for healthcare: Systematic review and architecture proposal. ACM Trans. Intell. Syst. Technol. , 13(4), 2022. ISSN 2157-6904. doi: 10.1145/3501813. URL https://doi.org/10.1145/3501813 .

Debraj Basu, Deepesh Data, Can Karakus, and Suhas Diggavi. Qsparse-local-SGD: Distributed SGD with quantization, sparsification and local computations. volume 32, 2019.

Aleksandr Beznosikov, Samuel Horváth, Peter Richtárik, and Mher Safaryan. On biased compression for distributed learning. arXiv preprint arXiv:2002.12410 , abs/2002.12410, 2020. URL https: //arXiv.org/abs/2002.12410 .

Léon Bottou. Curiously fast convergence of some stochastic gradient descent algorithms. In Proceedings of the symposium on learning and data science, Paris , volume 8, pages 2624-2633. Citeseer, 2009.

Konstantin Burlachenko, Samuel Horváth, and Peter Richtárik. Fl_pytorch: optimization research simulator for federated learning. In Proceedings of the 2nd ACM International Workshop on Distributed Machine Learning , pages 1-7, 2021.

Chih-Chung Chang and Chih-Jen Lin. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) , 2(3):1-27, 2011.

Ilyas Fatkhullin, Igor Sokolov, Eduard Gorbunov, Zhize Li, and Peter Richtárik. Ef21 with bells & whistles: Practical algorithmic extensions of modern error feedback. arXiv preprint arXiv:2110.03294 , 2021.

Margalit R. Glasgow, Honglin Yuan, and Tengyu Ma. Sharp bounds for federated averaging (local SGD) and continuous perspective. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , volume 151 of Proceedings of Machine Learning Research , pages 9050-9090. PMLR, 2022. URL https://proceedings.mlr.press/v151/glasgow22a.html .

Eduard Gorbunov, Filip Hanzely, and Peter Richtárik. A unified theory of SGD: variance reduction, sampling, quantization and coordinate descent. In International Conference on Artificial Intelligence and Statistics , pages 680-690. PMLR, 2020.

Eduard Gorbunov, Filip Hanzely, and Peter Richtárik. Local SGD: Unified theory and new efficient methods. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130 of Proceedings of Machine Learning Research , pages 3556-3564. PMLR, 13-15 Apr 2021. URL https://proceedings. mlr.press/v130/gorbunov21a.html .

Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richtárik. Sgd: General analysis and improved rates. In International conference on machine learning , pages 5200-5209. PMLR, 2019.

Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 , 2017.

Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, and Mehrdad Mahdavi. Federated learning with compression: Unified analysis and sharp guarantees. In Arindam Banerjee and Kenji Fukumizu, editors, The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event , volume 130 of Proceedings of Machine Learning Research , pages 2350-2358. PMLR, 2021. URL http://proceedings.mlr.press/v130/ haddadpour21a.html .

He et al. Deep residual learning for image recognition. In CVPR , pages 770-778, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

Samuel Horváth, Dmitry Kovalev, Konstantin Mishchenko, Sebastian Stich, and Peter Richtárik. Stochastic distributed learning with gradient quantization and variance reduction. arXiv preprint arXiv:1904.05115 , abs/1904.05115, 2019. URL https://arXiv.org/abs/1904.05115 .

Samuel Horváth, Maziar Sanjabi, Lin Xiao, Peter Richtárik, and Michael Rabbat. FedShuffle: Recipes for better use of local work in federated learning. arXiv preprint arXiv:2204.13169 , abs/2204.13169, 2022. URL https://arXiv.org/abs/2204.13169 .

Kun Huang, Xiao Li, Andre Milzarek, Shi Pu, and Junwen Qiu. Distributed random reshuffling over networks. arXiv preprint arXiv:2112.15287 , abs/2112.15287, 2021. URL https://arXiv.org/ abs/2112.15287 .

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. PMLR, 2015.

Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Hubert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Koneˇ cný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Mariana Raykova, Hang Qi, Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977 , abs/1912.04977, 2019. URL https://arXiv.org/abs/1912.04977 .

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. SCAFFOLD: stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 5132-5143. PMLR, 2020. URL http://proceedings.mlr.press/v119/karimireddy20a.html .

Sarit Khirirat, Hamid Reza Feyzmahdavian, and Mikael Johansson. Distributed learning with compressed gradients. arXiv preprint arXiv:1806.06573 , abs/1806.06573, 2018. URL https: //arXiv.org/abs/1806.06573 .

Jakub Koneˇ cný, H. Brendan McMahan, Felix Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: strategies for improving communication efficiency. In NIPS Private Multi-Party Machine Learning Workshop , 2016.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.

Ming Liu, Stella Ho, Mengqi Wang, Longxiang Gao, Yuan Jin, and He Zhang. Federated learning meets natural language processing: a survey. arXiv preprint arXiv:2107.12603 , abs/2107.12603, 2021. URL https://arXiv.org/abs/2107.12603 .

Grigory Malinovsky and Peter Richtárik. Federated random reshuffling with compression and variance reduction. arXiv preprint arXiv:2205.03914 , abs/2205.03914, 2022. URL https: //arXiv.org/abs/2205.03914 .

Grigory Malinovsky, Alibek Sailanbayev, and Peter Richtárik. Random reshuffling with variance reduction: New analysis and better rates. arXiv preprint arXiv:2104.09342 , 2021.

Grigory Malinovsky, Konstantin Mishchenko, and Peter Richtárik. Server-side stepsizes and sampling without replacement provably help in federated optimization. arXiv preprint arXiv:2201.11066 , abs/2201.11066, 2022. URL https://arXiv.org/abs/2201.11066 .

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Konstantin Mishchenko, Eduard Gorbunov, Martin Takáˇ c, and Peter Richtárik. Distributed learning with compressed gradient differences. arXiv preprint arXiv:1901.09269 , 2019a.

Konstantin Mishchenko, Eduard Gorbunov, Martin Takác, and Peter Richtárik. Distributed learning with compressed gradient differences. arXiv preprint arXiv:1901.09269 , abs/1901.09269, 2019b. URL https://arXiv.org/abs/1901.09269 .

Konstantin Mishchenko, Ahmed Khaled, and Peter Richtárik. Random reshuffling: Simple analysis with vast improvements. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ c8cc6e90ccbff44c9cee23611711cdc4-[]Abstract.html .

Konstantin Mishchenko, Ahmed Khaled, and Peter Richtárik. Proximal and federated random reshuffling. arXiv preprint arXiv:2102.06704 , abs/2102.06704, 2021. URL https://arXiv. org/abs/2102.06704 .

Aritra Mitra, Rayana Jaafar, George J. Pappas, and Hamed Hassani. Linear convergence in federated learning: Tackling client heterogeneity and sparse gradients. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 14606-14619. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/file/ 7a6bda9ad6ffdac035c752743b7e9d0e-[]Paper.pdf .

Tomoya Murata and Taiji Suzuki. Bias-variance reduced local SGD for less heterogeneous federated learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 7872-7881. PMLR, 2021. URL http: //proceedings.mlr.press/v139/murata21a.html .

Jose Javier Gonzalez Ortiz, Jonathan Frankle, Mike Rabbat, Ari Morcos, and Nicolas Ballas. Tradeoffs of local SGD at scale: an empirical study. arXiv preprint arXiv:2110.08133 , abs/2110.08133, 2021. URL https://arXiv.org/abs/2110.08133 .

Shashank Rajput, Anant Gupta, and Dimitris S. Papailiopoulos. Closing the convergence gap of SGD without replacement. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 7964-7973. PMLR, 2020. URL http://proceedings.mlr.press/ v119/rajput20a.html .

Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization. In Silvia Chiappa and Roberto Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy] , volume 108 of Proceedings of Machine Learning Research , pages 2021-2031. PMLR, 2020. URL http://proceedings.mlr.press/v108/reisizadeh20a.html .

Peter Richtárik, Elnur Gasanov, and Konstantin Burlachenko. Error feedback reloaded: From quadratic to arithmetic mean of smoothness constants. arXiv preprint arXiv:2402.10774 , 2024.

Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics , pages 400-407, 1951.

Mher Safaryan, Filip Hanzely, and Peter Richtárik. Smoothness matrices beat smoothness constants: Better communication compression techniques for distributed optimization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 25688-25702. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/file/ d79c6256b9bdac53a55801a066b70da3-[]Paper.pdf .

Itay Safran and Ohad Shamir. Random shuffling beats SGD only after many epochs on ill-conditioned problems. arXiv preprint arXiv:2106.06880 , abs/2106.06880, 2021. URL https://arXiv.org/ abs/2106.06880 .

Sebastian U. Stich. Unified optimal analysis of the (stochastic) gradient method. arXiv preprint arXiv:1907.04232 , abs/1907.04232, 2019. URL https://arXiv.org/abs/1907.04232 .

Sebastian U. Stich. On communication compression for distributed optimization on heterogeneous data. arXiv preprint arXiv:2009.02388 , abs/2009.02388, 2020. URL https://arXiv.org/abs/ 2009.02388 .

Sebastian U. Stich, Jean-Baptiste Cordonnier, and Martin Jaggi. Sparsified SGD with memory. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 4452-4463, 2018. URL https://proceedings.neurips.cc/ paper/2018/hash/b440509a0106086a67bc2ea9df0a1dab-[]Abstract.html .

Zhenheng Tang, Shaohuai Shi, Xiaowen Chu, Wei Wang, and Bo Li. Communication-efficient distributed deep learning: a comprehensive survey. arXiv preprint arXiv:2003.06307 , abs/2003.06307, 2020. URL https://arXiv.org/abs/2003.06307 .

Jianqiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang. Gradient sparsification for communicationefficient distributed optimization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. CesaBianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper/2018/file/ 3328bdf9a4b9504b9398284244fe97c2-[]Paper.pdf .

Blake Woodworth, Brian Bullins, Ohad Shamir, and Nathan Srebro. The min-max complexity of distributed stochastic convex optimization with intermittent communication. arXiv preprint arXiv:2102.01583 , abs/2102.01583, 2021. URL https://arXiv.org/abs/2102.01583 .

Blake E. Woodworth, Kumar Kshitij Patel, and Nati Srebro. Minibatch vs local SGD for heterogeneous distributed learning. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020a. URL https://proceedings.neurips.cc/paper/2020/hash/ 45713f6ff2041d3fdfae927b82488db8-[]Abstract.html .

Blake E. Woodworth, Kumar Kshitij Patel, Sebastian U. Stich, Zhen Dai, Brian Bullins, H. Brendan McMahan, Ohad Shamir, and Nathan Srebro. Is local SGD better than minibatch SGD? In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 1033410343. PMLR, 2020b. URL http://proceedings.mlr.press/v119/woodworth20a.html .

Zhaohui Yang, Mingzhe Chen, Kai-Kit Wong, H. Vincent Poor, and Shuguang Cui. Federated learning for 6g: Applications, challenges, and opportunities. Engineering , 8:33-41, 2022. ISSN 20958099. doi: https://doi.org/10.1016/j.eng.2021.12.002. URL https://www.sciencedirect. com/science/article/pii/S2095809921005245 .

Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962 , 2019.

Chulhee Yun, Shashank Rajput, and Suvrit Sra. Minibatch vs local SGD with shuffling: Tight convergence bounds and beyond. arXiv preprint arXiv:2110.10342 , abs/2110.10342, 2021. URL https://arXiv.org/abs/2110.10342 .

Contents

A Extra Related Works

Federated optimization has been the subject of intense study, with many open questions even in the setting when all clients have identical data [Woodworth et al., 2020b,a, 2021]. The FedAvg algorithm (also known as Local SGD ) has also been a subject of intense study, with tight bounds obtained only very recently by Glasgow et al. [2022]. It is now understood that using many local steps adds bias to distributed SGD , and hence several methods have been developed to mitigate it, e.g. [Karimireddy et al., 2020, Murata and Suzuki, 2021], see the work of Gorbunov et al. [2021] for a unifying lens on many variants of Local SGD . Note that despite the bias, even vanilla FedAvg / Local SGD still reduces the overall communication overhead in practice [Ortiz et al., 2021].

The success of RR in the single-machine setting has inspired several recent methods that use it as a local update method as part of distributed training: Mishchenko et al. [2021] developed a distributed variant of random reshuffling, FedRR . FedRR uses RR as a local client update method in lieu of SGD . They show that FedRR can improve upon the convergence of Local SGD when the number of local steps is fixed as the local dataset size, i.e. when H = n . Yun et al. [2021] study the same method under the name Local RR under a more restrictive assumption of bounded inter-machine gradient deviation and show that by varying H to be smaller than n better rates can be obtained in this setting than the rates of Mishchenko et al. [2021]. Other work has explored more such combinations between RR and distributed training algorithms [Huang et al., 2021, Malinovsky et al., 2022, Horváth et al., 2022].

There are several methods that combine compression or quantization and local steps: both Basu et al. [2019] and Reisizadeh et al. [2020] combined Local SGD with quantization and sparsification, and Haddadpour et al. [2021] later improved their results using a gradient tracking method, achieving linear convergence under strong convexity. In parallel, Mitra et al. [2021] also developed a variancereduced method, FedLin , that achieves linear convergence under strong convexity despite using local steps and compression. The paper most related to our work is [Malinovsky and Richtárik, 2022] in which the authors combine iterate compression, random reshuffling, and local steps. We study gradient compression instead, which is a more common form of compression in both theory and practice [Kairouz et al., 2019]. We compare our results against [Malinovsky and Richtárik, 2022] and show we obtain better rates compared to their work.

B Experimental details

In this section, we provide missing details on the experimental setting from Section 3. The codes are provided in the following anonymous repository: https://anonymous.4open.science/r/ diana_rr-[]B0A5 .

B.1 Logistic Regression

To confirm our theoretical results we conducted several numerical experiments on binary classification problem with L2 regularized logistic regression of the form

where f m,i def = log ( 1 + exp( -y mi a ⊤ mi x ) ) + λ ∥ x ∥ 2 2 ( a mi , y mi ) ∈ R d × ∈ {-1 , 1 } , i = 1 , . . . , n m are the training data samples stored on machines m = 1 , . . . , M , and λ > 0 is a regularization parameter. In all experiments, for each method, we used the largest stepsize allowed by its theory multiplied by some individually tuned constant multiplier. For better parallelism, each worker m uses mini-batches of size ≈ 0 . 1 n m . In all algorithms, as a compression operator Q , we use Randk [Beznosikov et al., 2020] with fixed compression ratio k / d ≈ 0 . 02 , where d is the number of features in the dataset.

Hardware and Software. All algorithms were written in Python 3.8. We used three different CPU cluster node types:

1. AMDEPYC 7702 64-Core;

Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz;

Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz.

Datasets. The datasets were taken from open LibSVM library Chang and Lin [2011], sorted in ascending order of labels, and equally split among 20 machines \ clients \ workers. The remaining part of size N -20 · ⌊ N / 20 ⌋ was assigned to the last worker, where N = ∑ M m =1 n m is the total size of the dataset. A summary of the splitting and the data samples distribution between clients can be found in Tables 2, 3, 4, 5.

Hyperparameters. Regularization parameter λ was chosen individually for each dataset to guarantee the condition number L / µ to be approximately 10 4 , where L and µ are the smoothness and strong-convexity constants of function f . For the chosen logistic regression problem of the form (10), smoothness and strong convexity constants L , L m , L i,m , µ , ˜ µ of functions f , f m and f i m were computed explicitly as

where A m is the dataset associated with client m , and a mi is the i -th row of data matrix A m . In general, the fact that f is L -smooth with

follows from the L i,m -smoothness of f i m (see Assumption 3).

In all algorithms, as a compression operator Q , we use Randk as a canonical example of unbiased compressor with relatively bounded variance, and fix the compression parameter k = ⌊ 0 . 02 d ⌋ , where d is the number of features in the dataset.

In addition, in all algorithms, for all clients m = 1 , . . . , M , we set the batch size for the SGD estimator to be b m = ⌊ 0 . 1 n m ⌋ , where n m is the size of the local dataset.

The summary of the values L , L m , L i,m L max , µ , b m and k for each dataset can be found in Table 6.

In all experiments, we follow constant stepsize strategy within the whole iteration procedure. For each method, we set the largest possible stepsize predicted by its theory multiplied by some individually tuned constant multiplier. For a more detailed explanation of the tuning routine, see Sections B.1.1 and B.1.2.

SGD implementation. We considered two approaches to minibatching: random reshuffling and with-replacement sampling. In the first, all clients m = 1 , . . . , M independently permute their local datasets and pass through them within the next subsequent ⌊ n m b m ⌋ steps. In our implementations of Q-RR , Q-NASTYA and DIANA-NASTYA , all clients permuted their datasets in the beginning of every new epoch, whereas for the DIANA-RR method they do so only once in the beginning of the iteration procedure. Second approach of minibatching is called with-replacement sampling, and it requires every client to draw b m data samples from the local dataset uniformly at random. We used this strategy in the baseline algorithms ( QSGD , DIANA , FedCOM and FedPAQ ) we compared our proposed methods to.

Experimental setup. To compare the performance of methods within the whole optimization process, we track the functional suboptimality metric f ( x t ) -f ( x ⋆ ) that was recomputed after each epoch. For each dataset, the value f ( x ⋆ ) was computed once at the preprocessing stage with 10 -16 tolerance via conjugate gradient method. We terminate our algorithms after performing 5000 epochs.

B.1.1 Experiment 1: Comparison of the Proposed Non-Local Methods with Existing Baselines

In our first experiment (see Figure 1), we compare Q-RR , DIANA-RR , and DIANA-RR-1S with classical baselines ( QSGD [Alistarh et al., 2017], DIANA [Mishchenko et al., 2019b]) that use

1000

3000

5000

Data passes

10

7

10

5

10

3

10

1

f

(

xt

)

f

(

x

)

mushrooms; Rand-2

QSGD

Q-RR

DIANA

DIANA-RR

DIANA-RR-1S

1000

3000

5000

Data passes

10

7

10

5

10

3

10

1

f

(

xt

)

f

(

x

)

w8a; Rand-6

QSGD

Q-RR

DIANA

DIANA-RR

DIANA-RR-1S

1000

3000

5000

Data passes

10

7

10

5

10

3

10

1

f

(

xt

)

f

(

x

)

a9a; Rand-2

QSGD

Q-RR

DIANA

DIANA-RR

DIANA-RR-1S

Figure 5: Non-local methods

1000

3000

5000

Data passes

10

3

10

1

f

(

xt

)

f

(

x

)

mushrooms; Rand-2

Q-NASTYA

DIANA-NASTYA

FedCOM

FedPAQ

1000

3000

5000

Data passes

10

3

10

1

f

(

xt

)

f

(

x

)

w8a; Rand-6

Q-NASTYA

DIANA-NASTYA

FedCOM

FedPAQ

1000

3000

5000

Data passes

10

3

10

1

f

(

xt

)

f

(

x

)

a9a; Rand-2

Q-NASTYA

DIANA-NASTYA

FedCOM

FedPAQ

a with-replacement mini-batch SGD estimator. DIANA-RR-1S is a memory-friendly version of DIANA-RR that stores and uses a single shift h t,m on the worker side rather than n individual shifts h π i m t,m . Figure 1 illustrates that Q-RR exhibits similar behavior to QSGD , with both methods being slower than DIANA methods across all considered datasets. DIANA-RR-1S and DIANA show comparable convergence rates, indicating that random reshuffling alone, without introducing additional shifts, does not make a significant difference. Finally, DIANA-RR achieves the best rate among all considered non-local methods, efficiently reducing the variance and reaching the lowest functional sub-optimality tolerance. These experimental results align perfectly with our theoretical analysis. For each of the considered non-local methods, we take the stepsize as the largest one predicted by the theory premultiplied by the individually tuned constant factor from the set { 0 . 000975 , 0 . 00195 , 0 . 0039 , 0 . 0078 , 0 . 0156 , 0 . 0312 , 0 . 0625 , 0 . 125 , 0 . 25 , 0 . 5 , 1 , 2 , 4 , 8 , 16 , 32 , 64 , 128 , 256 , 512 , 1024 , 2048 , 4096 } .

Therefore, for each local method on every dataset, we performed 20 launches to find the stepsize multiplier showing the best convergence behavior (the fastest reaching the lowest possible level of functional suboptimality f ( x t ) -f ( x ⋆ ) ).

Theoretical stepsizes for methods Q-RR and DIANA-RR are provided by the Theorems 2.1 and 2.2, whereas stepsizes for QSGD and DIANA were taken from the paper Gorbunov et al. [2020].

B.1.2 Experiment 2: Comparison of the Proposed Local Methods with Existing Baselines

The second experiment shows that DIANA -based method can significantly outperform in practice when one applies it to local methods as well. In particular, whereas Q-NASTYA shows comparative behavior as existing methods FedCOM [Haddadpour et al., 2021], FedPAQ [Reisizadeh et al., 2020] in all considered datasets, DIANA-NASTYA noticeably outperforms other methods.

In this set of experiments, we tuned stepsizes similarly to the non-local methods. However, for algorithms Q-NASTYA , DIANA-NASTYA , and FedCOM we needed to independently adjust the client and server stepsizes, leading to a more extensive tunning routine.

As before, for each local method on every dataset, tuned client and server stepsizes are defined by the theoretical one and adjusted constant multiplier. Theoretical stepsizes for methods Q-NASTYA and DIANA-NASTYA are given by the Theorems E.1 and E.2, whereas FedCOM and FedPAQ stepsizes were taken from the papers by Haddadpour et al. [2021] and Reisizadeh et al. [2020] respectively.

We now list all the considered multipliers of client and server stepsizes for every method (i.e. γ and η respectively):

· Q-NASTYA :

-Multipliers for γ : { 0.000975, 0.00195, 0.0039, 0.0078, 0.0156, 0.0312, 0.0625, 0.125, 0.25, 0.5, 1, 2, 4, 8, 16, 32, 64,128 } ;

-Multipliers for η : { 0.0039, 0.0078, 0.0156, 0.0312, 0.0625, 0.125, 0.25, 0.5, 1, 2, 4, 8, 16, 32, 64, 128 } .

DIANA-NASTYA :

-Multipliers for γ and η : { 0.000975, 0.00195, 0.0039, 0.0078, 0.0156, 0.0312, 0.0625, 0.125, 0.25, 0.5, 1, 2, 4, 8, 16, 32, 64,128 } ;

FedCOM :

-Multipliers for γ : { 0.0312, 0.0625, 0.125, 0.25, 0.5, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512,1024, 2048, 4096, 8192, 16384, 32768 } ;

-Multipliers for η : { 0.000975, 0.00195, 0.0039, 0.0078, 0.0156, 0.0312, 0.0625, 0.125, 0.25, 0.5, 1, 2, 4, 8, 16, 32, 64, 128 } .

FedPAQ :

-Multipliers for γ : { 0.00195, 0.0039, 0.0078, 0.0156, 0.0312, 0.0625, 0.125, 0.25, 0.5, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536, 131072, 262144, 524288, 1048576 } .

For example, to find the best pair ( γ, η ) for FedCOM method on each dataset, we performed 378 launches. A similar subroutine was executed for all algorithms on all datasets independently.

B.1.3 Experiment 3: Comparison of DIANA-RR with EF21 and DIANA

In our third experiment (see Figure 8), we compared DIANA-RR with DIANA and EF21-SGD [Fatkhullin et al., 2021]. The EF21-SGD is the state-of-the-art algorithm for contractive compressors in distributed non-convex settings. All compared algorithms used a with-replacement mini-batch SGD estimator, consistent with the setup in Section B.1.1. However, in this experiment contrast with Section B.1.1, we employed reliable theoretical step sizes that ensure guaranteed convergence.

The EF21 algorithm family is designed for usage with contraction compressors in non-convex optimization for L -smooth objective functions in for of Equation 1. For scenarios involving unbiased compressors such as Randk , the EF21-SGD can be adapted through scaling [Fatkhullin et al., 2021]. More specifically, an unbiased compressor C ( x ) : R d → R d which satisfied Assumption 1 via applying the transformation C ′ ( x ) def = ( ω +1) -1 · C ( x ) yields a contraction compressor E [ ∥ C ′ ( x ) -x ∥ 2 ] ≤ (1 -α ) ∥ x ∥ 2 , ∀ x ∈ R d with α = 1 / ω +1 . In particular, this procedure makes Randk compatible with the EF21 algorithm family. In this experiment, we implemented EF21-SGD following the refined analysis from [Richtárik et al., 2024], which offers a stricter better convergence guarantee through improved bounds on the theoretical step size compared to [Fatkhullin et al., 2021].

As shown in Figure 8 EF21-SGD does not perform fast enough in scenarios involving using unbiased compressors for strongly-convex optimization problems compared to DIANA-RR and DIANA .

B.2 Training Deep Neural Network model: ResNet-18 on CIFAR-10

Since random reshuffling is a very popular technique in training neural networks, it is natural to test the proposed methods on such problems. Therefore, in the second set of experiments, we consider training ResNet-18 [He et al., 2016] model on the CIFAR10 dataset Krizhevsky and Hinton [2009]. To conduct these experiments we use FL_PyTorch simulator [Burlachenko et al., 2021].

The main goal of this experiment is to verify the phenomenon observed in Experiment 1 on the training of a deep neural network. That is, we tested Q-RR , QSGD , DIANA , and DIANA-RR in the distributed training of ResNet-18 on CIFAR10 , see Figure 9. As in the logistic regression experiments, we observe that (i) Q-RR and QSGD behave similarly and (ii) DIANA-RR outperforms DIANA .

0

1000

2000

3000

4000

5000

Data passes

0.01

0.1

1

f

(

x t

)

-

f

∗

mushrooms; RandK[K=2]

DIANA

DIANA-RR

EF21-SGD

0

1000

2000

3000

4000

5000

Data passes

0.01

0.1

1

f

(

x t

)

-

f

∗

a9a; RandK[K=2]

DIANA

DIANA-RR

EF21-SGD

0

2

4

6

8

#bits/n

1e7

20

40

60

Top1 acc (validation)

QSGD

= 0.001

QSGD-RR

= 0.001

0

1

2

3

#bits/n

1e11

25

50

75

Top1 acc (validation)

DIANA

=1.0

DIANA-RR

=1.0

To illustrate the behavior of the proposed methods in training Deep Neural Networks (DNN), we consider the ResNet-18 [He et al., 2016] model. This model is used for image classification, feature extraction for image segmentation, object detection, image embedding, and image captioning. We train all layers of ResNet-18 model meaning that the dimension of the optimization problem equals d = 11 , 173 , 962 . During the training, the ResNet-18 model normalizes layer inputs via exploiting 20 Batch Normalization [Ioffe and Szegedy, 2015] layers that are applied directly before nonlinearity in the computation graph of this model. Batch normalization (BN) layers add 9600 trainable parameters to the model. Besides trainable parameters, a BN layer has its internal state that is used for computing the running mean and variance of inputs due to its own specific regime of working. We use He initialization [He et al., 2015].

B.2.1 Computing Environment

We performed numerical experiments on a server-grade machine running Ubuntu 18.04 and Linux Kernel v5.4.0, equipped with 16-cores (2 sockets by 16 cores per socket) 3.3 GHz Intel Xeon, and four NVIDIA A100 GPU with 40GB of GPU memory. The distributed environment is simulated in Python 3.9 via using the software suite FL_PyTorch [Burlachenko et al., 2021] that serves for carrying complex Federate Learning experiments. FL_PyTorch allowed us to simulate the distributed environment in the local machine. Besides storing trainable parameters per client, this simulator stores all not trainable parameters including BN statistics per client.

B.2.2 Loss Function

Training of ResNet-18 can be formalized as problem (1) with the following choice of f i m

where CE ( p, q ) def = -∑ #classes k =1 p i · log( q i ) with agreement 0 · log(0) = 0 is a standard cross-entropy loss, function g : R 28 × 28 × R d → [0 , 1] #classes is a neural network taking image a ( j ) and vector of parameters x as an input and returning a vector in probability simplex, and n m is the size of the dataset on worker m .

B.2.3 Dataset and Metric

In our experiments, we used CIFAR10 dataset Krizhevsky and Hinton [2009]. The dataset consists of input variables a i ∈ R 28 × 28 × 3 , and response variables b i ∈ { 0 , 1 } 10 and is used for training 10-way classification. The sizes of training and validation set are 5 × 10 4 and 10 4 respectively. The training set is partitioned heterogeneously across 10 clients. To measure the performance, we evaluate the loss function value f ( x ) , norm of the gradient ∥∇ f ( x ) ∥ 2 and the Top-1 accuracy of the obtained model as a function of passed epochs and the normalized number of bits sent from clients to the server.

B.2.4 Tuning Process

In this set of experiments, we tested QSGD [Alistarh et al., 2017], Q-RR (Algorithm 1), DIANA [Mishchenko et al., 2019a] and DIANA-RR (Algorithm 2) algorithms. For all algorithms, we tuned the strategy ∈ { A,B,C } of decaying stepsize model via selecting the best in terms of the norm of the full gradient on the train set in the final iterate produced after 20000 rounds. The stepsize policies are described below.

Stepsizes decaying as inverse square root of the number epochs

where γ e denotes the stepsize used during epoch e +1 , s is a fixed shift.

Stepsizes decaying as inverse of number epochs

We say that the algorithm passed e epochs if the total number of computed gradient oracles lies between e ∑ M m =1 n m and ( e +1) ∑ M m =1 n m . For each algorithm the used stepsize γ init and shift parameter s were tuned via selecting from the following sets:

In all tested methods, clients independently apply Randk compression with carnality k = ⌊ 0 . 05 d ⌋ . Computation for all gradient oracles is carried out in single precision float (FP64) arithmetic.

C. Fixed stepsize

0

200

400

600

800

1000

Data passes

20

40

60

80

Top1 acc (validation)

QSGD

=3.0

QSGD-RR

=3.0

0

200

400

600

800

1000

Data passes

10 0

10 2

||

f

(

xt

)|| (train)

QSGD

=3.0

QSGD-RR

=3.0

0

200

400

600

800

1000

Data passes

10 0

10 1

10 2

f

(

xt

) (train)

QSGD

=3.0

QSGD-RR

=3.0

0.0

0.5

1.0

1.5

2.0

#bits/n

1e11

20

40

60

80

Top1 acc (validation)

QSGD

=3.0

QSGD-RR

=3.0

0.0

0.5

1.0

1.5

2.0

#bits/n

1e11

10

1

10 0

10 1

||

f

(

xt

)|| (train)

QSGD

=3.0

QSGD-RR

=3.0

0.0

0.5

1.0

1.5

2.0

#bits/n

1e11

10 0

10 1

10 2

f

(

xt

) (train)

QSGD

=3.0

QSGD-RR

=3.0

Figure 10

0

100

200

300

400

Data passes

25

50

75

Top1 acc (validation)

DIANA

=1.0

DIANA-RR

=1.0

0

100

200

300

400

500

600

Data passes

10 0

10 1

||

f

(

xt

)|| (train)

DIANA

=1.0

DIANA-RR

=1.0

0

200

400

600

Data passes

10

1

10 0

10 1

f

(

xt

) (train)

DIANA

=1.0

DIANA-RR

=1.0

0

1

2

3

#bits/n

1e11

25

50

75

Top1 acc (validation)

DIANA

=1.0

DIANA-RR

=1.0

0

1

2

3

#bits/n

1e11

10 0

10 1

||

f

(

xt

)|| (train)

DIANA

=1.0

DIANA-RR

=1.0

0

1

2

3

#bits/n

1e11

10

1

10 0

10 1

f

(

xt

) (train)

DIANA

=1.0

DIANA-RR

=1.0

B.2.5 Optimization-Based Fine-Tuning for Pretrained ResNet-18 .

In this setting, we trained ResNet-18 image classification in a distributed way across n = 10 clients. In this experiment, we have trained only the last linear layer.

Next, we have turned off batch normalization. Turning off batch normalization implies that the computation graph of NN g ( a, x ) with weights of NN denoted as x is a deterministic function and does not include any internal state.

The loss function is a standard cross-entropy loss augmented with extra ℓ 2 -regularization α ∥ x ∥ 2 / 2 with α = 0 . 0001 . Initially used weights of NN are pretrained parameters after training the model on ImageNet.

The dataset distribution across clients has been set in a heterogeneous manner via presorting dataset D by label class and after this, it was split across 10 clients.

The comparison of stepsizes policies used in QSGD and Q-RR is presented in Figure 14. The behavior of the algorithms with best tuned step sizes is presented in Figure 13. These results demonstrate that in this setting there is no real benefit of using Q-RR in comparison to QSGD .

B.2.6 Experiments

The comparison of QSGD and Q-RR is presented in Figure 11. In particular, Figure 9 shows that in terms of the convergence to stationary points both algorithms exhibit similar behavior. However, QRR has better generalization and in fact, converges to the better loss function value. This experiment demonstrates that Q-RR with manually tuned stepsize can be better compared to QSGD in terms of the final quality of obtained Deep Learning model. For QSGD the tuned meta parameters are:

γ init = 3 . 0 , s = 200 , strategy = B . For QSGD-RR tuned meta parameters are: γ init = 3 . 0 , s = 1000 , strategy = B .

The results of comparison of DIANA and DIANA-RR are presented in Figure 12. For DIANA the tuned meta parameters are: γ init = 1 . 0 , s = 0 , strategy = C and for DIANA-RR tuned meta parameters are: γ init = 1 . 0 , s = 0 , strategy = C . These results show that DIANA-RR outperforms DIANA in terms of all reported metrics.

B.3 Discussion

More about used arithmetics. We used FP64 (IEEE 754) due to its superior numerical stability compared to FP32 , FP16 , and BFloat16 . While FP32 and FP16 are commonly used for inference tasks, the choice of precision for training depends on the specific requirements of the task. In certain cases, FP32 may be sufficient, but for others, FP64 is necessary to ensure stability.

The performance gain from switching from FP64 to FP32 can indeed vary based on the GPU model. For instance, the NVIDIA A100 40GB GPU used in our experiments offers approximately a two-fold increase in computational throughput with FP3 2 compared to FP64 . The specific architecture of the GPU influences the choice of precision, and these characteristics can differ across various GPU models and updates.

The computational burden. The primary focus of the paper is to highlight the fundamental complexities and limits of algorithmic behavior. The experiments presented in our paper are intended for illustrative purposes.

The computational demands of our work are significant. Performing experiments beyond ResNet-18/CIFAR-10/FP64 with 10 clients is near the limit of what is feasible with our computational resources. In our simulation involving 10 clients sharing a common dataset, we ran 2000 rounds/epochs for fine-tuning. Based on an estimate of 2 minutes per epoch, the total computation time would be approximately 66 hours per run (2 minutes/epoch × 2000 epochs = 66 hours). Taking into account the grid search with 18 preset learning rates, 5 sets of decay parameters, and 4 algorithms, the total estimated computation time would be around 23760 hours (66 hours × 18 × 5 × 4). This represents a substantial amount of computation time. Therefore, conducting a comprehensive comparison involving four algorithms with an extensive grid of hyperparameters is already challenging for models larger than ResNet-18 on CIFAR10. To cover 23760 hours of training would indeed require approximately 40 GPUs running continuously for about 25 days. Nonetheless, we have conducted numerous experiments to ensure a thorough and fair comparison.

Training in overparameterized regime. During training image classification Convolution Neural Networks, we got two results for QSGD-RR as an improvement of QSGD . During training only the last layer (see Fig. 13) there are no benefits QSGD-RR , but QSGD does not behave worse.

When training the whole network (Fig. 11), the results suggest that Q-RR is much better than Q-SGD . Although we do not have formal proof explaining this phenomenon, we conjecture that this can be related to the significant overparameterization occurring during the training of a large model on a relatively small dataset. That is, the model can almost perfectly fit the training data on all clients, leading to a decrease in the heterogeneity parameter. In this case, there is no need for shifts since the variance coming from compression naturally goes to zero, and the complexities of QSGD and DIANA match (see Table 1). In this situation, Q-RR performs better than QSGD since the compression does not spoil the convergence of RR . Therefore, DIANA -type shifts are not always necessary to get improvements. Nevertheless, we conjecture that they are necessary when the datasets are larger and more complex.

0

2

4

6

8

#bits/n

1e7

20

40

60

Top1 acc (validation)

QSGD

= 0.001

QSGD-RR

= 0.001

0

2

4

6

8

#bits/n

1e7

1

2

||

f

(

xt

)|| (train)

QSGD

= 0.001

QSGD-RR

= 0.001

0

2

4

6

8

#bits/n

1e7

2.0

2.5

f

(

xt

) (train)

QSGD

= 0.001

QSGD-RR

= 0.001

0

1

2

3

4

5

6

7

8

#bits/n

1e7

10

20

30

40

50

Top1 acc (validation)

QSGD (

=0.0006)

QSGD (

=0.003)

QSGD (

=0.001)

QSGD (

=0.01)

QSGD (

=0.03)

QSGD (

=0.06)

QSGD (

=0.5)

QSGD (

=0.1)

QSGD (

=0.2)

QSGD (

=0.25)

QSGD (

=0.75)

QSGD (

=1.0)

QSGD (

=1.25)

QSGD (

=2.00)

QSGD (

=2.5)

QSGD (

=3.00)

QSGD (

=3.75)

QSGD (

=4.00)

0

1

2

3

4

5

6

7

8

#bits/n

1e7

10

20

30

40

50

Top1 acc (validation)

QSGD-RR (

=0.0006)

= 0.0006

QSGD-RR (

=0.003)

= 0.003

QSGD-RR (

=0.001)

= 0.001

QSGD-RR (

=0.01)

= 0.01

QSGD-RR (

=0.03)

= 0.03

QSGD-RR (

=0.06)

= 0.06

QSGD-RR (

=0.1)

= 0.1

QSGD-RR (

=0.2)

= 0.2

QSGD-RR (

=0.25)

= 0.25

QSGD-RR (

=0.75)

= 0.75

QSGD-RR (

=1.0)

= 1.0

QSGD-RR (

=1.25)

= 1.25

QSGD-RR (

=2.00)

= 2.0

QSGD-RR (

=2.5)

= 2.5

QSGD-RR (

=3.00)

= 3.0

QSGD-RR (

=3.75)

= 3.75

QSGD-RR (

=4.00)

= 4.0

0

1

2

3

4

5

6

7

8

#bits/n

1e7

10 1

10 2

f

(

xt

) (train)

QSGD (

=0.0006)

QSGD (

=0.003)

QSGD (

=0.001)

QSGD (

=0.01)

QSGD (

=0.03)

QSGD (

=0.06)

QSGD (

=0.5)

QSGD (

=0.1)

QSGD (

=0.2)

QSGD (

=0.25)

QSGD (

=0.75)

QSGD (

=1.0)

QSGD (

=1.25)

QSGD (

=2.00)

QSGD (

=2.5)

QSGD (

=3.00)

QSGD (

=3.75)

QSGD (

=4.00)

0

1

2

3

4

5

6

7

8

#bits/n

1e7

10 1

10 2

f

(

xt

) (train)

QSGD-RR (

=0.0006)

= 0.0006

QSGD-RR (

=0.003)

= 0.003

QSGD-RR (

=0.001)

= 0.001

QSGD-RR (

=0.01)

= 0.01

QSGD-RR (

=0.03)

= 0.03

QSGD-RR (

=0.06)

= 0.06

QSGD-RR (

=0.1)

= 0.1

QSGD-RR (

=0.2)

= 0.2

QSGD-RR (

=0.25)

= 0.25

QSGD-RR (

=0.75)

= 0.75

QSGD-RR (

=1.0)

= 1.0

QSGD-RR (

=1.25)

= 1.25

QSGD-RR (

=2.00)

= 2.0

QSGD-RR (

=2.5)

= 2.5

QSGD-RR (

=3.00)

= 3.0

QSGD-RR (

=3.75)

= 3.75

QSGD-RR (

=4.00)

= 4.0

0

1

2

3

4

5

6

7

8

1e7

10 0

||

f

(

xt

)|| (train)

QSGD (

=0.0006)

QSGD (

=0.003)

QSGD (

=0.001)

QSGD (

=0.01)

QSGD (

=0.03)

QSGD (

=0.06)

QSGD (

=0.5)

QSGD (

=0.1)

QSGD (

=0.2)

QSGD (

=0.25)

QSGD (

=0.75)

QSGD (

=1.0)

QSGD (

=1.25)

QSGD (

=2.00)

QSGD (

=2.5)

QSGD (

=3.00)

QSGD (

=3.75)

QSGD (

=4.00)

0

1

2

3

4

5

6

7

8

#bits/n

1e7

10 0

||

f

(

xt

)|| (train)

QSGD-RR (

=0.0006)

= 0.0006

QSGD-RR (

=0.003)

= 0.003

QSGD-RR (

=0.001)

= 0.001

QSGD-RR (

=0.01)

= 0.01

QSGD-RR (

=0.03)

= 0.03

QSGD-RR (

=0.06)

= 0.06

QSGD-RR (

=0.1)

= 0.1

QSGD-RR (

=0.2)

= 0.2

QSGD-RR (

=0.25)

= 0.25

QSGD-RR (

=0.75)

= 0.75

QSGD-RR (

=1.0)

= 1.0

QSGD-RR (

=1.25)

= 1.25

QSGD-RR (

=2.00)

= 2.0

QSGD-RR (

=2.5)

= 2.5

QSGD-RR (

=3.00)

= 3.0

QSGD-RR (

=3.75)

= 3.75

QSGD-RR (

=4.00)

= 4.0

#bits/n

C Missing Proofs for Q-RR

In the main part of the paper, we introduce Assumptions 3 and 4 for the analysis of Q-RR and DIANA-RR . These assumptions can be refined as follows.

Assumption 5. Function f π i = 1 M ∑ M i =1 f π i m m : R d → R is ˜ L -smooth for all sets of permutations π = ( π 1 , . . . , π m ) from [ n ] and all i ∈ [ n ] , i.e.,

Assumption 6. Function f π i = 1 M ∑ M i =1 f π i m m : R d → R is ˜ µ -strongly convex for all sets of permutations π = ( π 1 , . . . , π m ) from [ n ] and all i ∈ [ n ] , i.e.,

Moreover, functions f i 1 , f i 2 , . . . , f i M : R d → R are convex for all i = 1 , . . . , n .

We notice that Assumptions 3 and 4 imply Assumptions 5 and 6. In the proofs of the results for Q-RR and DIANA-RR , we use Assumptions 5 in addition to Assumptions 3 and we use Assumption 6 instead of Assumption 4.

C.1 Shuffle Radius Clarification

Our results depend on the so-called shuffling radius proposed by Mishchenko et al. [2021]:

where x i +1 ⋆ = x i ⋆ -γ M ∑ M m =1 ∇ f π i m m ( x ⋆ ) .

One can think of the shuffling radius as a counterpart to the variance term in SGD . Both concepts measure how much the algorithm's performance can fluctuate near the optimal solution, but the cause of these fluctuations is different: in SGD , it is due to random sampling, and in RR , it is due to reshuffling. Additionally, Lemma 2.1 provides bounds for the shuffling radius - showing the maximum and minimum possible values - based on the variance at the optimum, reinforcing the shuffling radius as a useful way to understand how RR behaves. This relationship helps clarify how the reshuffling process influences the algorithm's path and its efficiency in reaching an optimal point.

C.2 Proof of Theorem 2.1

For convenience, we restate the theorem below.

Theorem C.1 (Theorem 2.1) . Let Assumptions 1, 3, 5, 6 hold and 0 < γ ≤ 1 ˜ L +2 ω M L max . Then, for all T ≥ 0 the iterates produced by Q-RR satisfy

Proof. Using x i +1 ⋆ = x i ⋆ -γ M ∑ M m =1 ∇ f π i m m ( x ⋆ ) and line 7 of Algorithm 1, we get

Taking the expectation w.r.t. Q , we obtain

In view of Assumption 1 and E ξ ∥ ξ -c ∥ 2 = E ξ ∥ ξ -E ξ ξ ∥ 2 + ∥ E ξ ξ -c ∥ 2 , we have

where in the last step we apply independence of Q ( ∇ f π i m m ( x i t ) ) for m ∈ [ M ] . Next, we use three-point identity 4 and obtain

Applying ˜ L -smoothness and convexity of 1 M ∑ m m =1 f π i m m , ˜ µ -strong convexity of 1 M ∑ m m =1 f π i m m , and L max -smoothness and convexity of f i m , we get

So, we get

Taking the full expectation and using a definition of shuffle radius, 0 < γ ≤ 1 ( ˜ L +2 ω M L max ) , and D f π i m m ( x i t , x ⋆ ) ≥ 0 , we obtain

Unrolling the recurrence in i , we derive

Unrolling the recurrence in t , we derive

Since ∑ nT -1 j =0 (1 -γ ˜ µ ) j ≤ 1 γ ˜ µ , we get the result.

Corollary 5. Let the assumptions of Theorem C.1 hold and

Then, Q-RR finds a solution with accuracy ε > 0 after the following number of communication rounds:

Proof. Theorem C.1 implies

To estimate the number of communication rounds required to find a solution with accuracy ε > 0 , we need to upper-bound each term from the right-hand side by ε / 3 . Thus, we get additional conditions on γ :

and also the upper bound on the number of communication rounds nT

Substituting (12), we get a final result.

C.3 Non-Strongly Convex Summands

In this section, we provide the analysis of Q-RR without using Assumptions 4, 6. Before we move one to the proofs, we would like to emphasize that

Then we have

where τ = γn . For convenience, we denote

allowing to write the update rule as x t +1 = x t -τg t .

Lemma C.1 (Lemma 1 from [Malinovsky et al., 2022]) . For any k ∈ [ n ] , let ξ π 1 , . . . , ξ π k be sampled uniformly without replacement from a set of vectors { ξ 1 , . . . , ξ n } and ¯ ξ π be their average. Then, it holds

where ¯ ξ = 1 n ∑ n i =1 ξ i , ¯ ξ π = 1 k ∑ k i =1 ξ π i , σ 2 = 1 n ∑ n i =1 ∥ ξ i -¯ ξ ∥ 2

Lemma C.2. Under Assumptions 1, 2, 3, 5, the following inequality holds

Proof. Using that E Q [ g t ] = 1 Mn ∑ n -1 i =0 ∑ M m =1 ∇ f π i m m ( x i t ) and definition of h ⋆ , we get

Using three-point identity, we obtain

where in the last inequality we apply ˜ L -smoothness and convexity of each function f π i . Finally, using µ -strong convexity of f , we finish the proof of the lemma.

Lemma C.3. Under Assumptions 1, 2, 3, 5, the following inequality holds

Proof. Taking the expectation w.r.t. Q and using variance decomposition E [ ∥ ξ ∥ 2 ] = E [ ∥ ξ -E [ ξ ] ∥ 2 ] + ∥ E ξ ∥ 2 , we get

Next, Assumption 1 and conditional independence of Q ( ∇ f π i m m ( x i t ) ) for m = 1 , . . . , M, i = 0 , . . . , n -1 imply

Using L max -smoothness and convexity of f i m and ˜ L -smoothness and convexity of f π i = 1 M ∑ M m =1 f π i m m , we derive

Taking the full expectation, we obtain

Lemma C.4. Let Assumptions 1, 2, 3, 5 hold and τ ≤ 1 2 √ ˜ L ( ˜ L + ω Mn L max ) . Then, the following inequality holds

where σ 2 ⋆,n = 1 n ∑ n i =1 ∥∇ f i ( x ⋆ ) ∥ 2 , f i ( x ) = 1 M ∑ M m =1 f i m ( x ) , i ∈ [ n ] .

Proof. Since x i t = x t -τ Mn ∑ M m =1 ∑ i -1 j =0 Q ( ∇ f π j m m ( x j t ) ) , we have

Using Assumption 1, ˜ L -smoothness and convexity of f π i = 1 M ∑ M m =1 f π i m m and L max -smoothness and convexity of f i m , we obtain

Next, we need to estimate the second term from the previous inequality. Taking the full expectation and using Lemma C.1 and using new notation σ 2 t = 1 n ∑ n j =1 E [ ∥∇ f j ( x t ) -∇ f ( x t ) ∥ 2 ] , we get

Taking the full expectation from (16) and using (17), we obtain

Using ˜ L -smoothness of f π j , we get

Since τ ≤ 1 2 √ ˜ L ( ˜ L + ω Mn L max ) , we have

Summing from i = 0 to n -1 and using ˜ L -smoothness of f i and L max -smoothness of f i m , we obtain

Theorem C.2. Let Assumptions 1, 2, 3, 5 hold and stepsize γ satisfy

Then, for all T ≥ 0 the iterates produced by Q-RR satisfy

where

Proof. Taking expectation w.r.t. Q and using Lemma C.3, we get

Using Lemma C.2, we obtain

Next, we take the full expectation and apply Lemma C.4:

Using (18), we derive

Recursively unrolling the inequality, substituting τ = nγ and using + ∞ ∑ t =0 ( 1 -τµ 2 ) t ≤ 2 µτ , we get the result.

Corollary 6. Let the assumptions of Theorem C.2 hold and

where ∆ 2 ⋆ = ζ 2 ⋆ + σ 2 ⋆ . Then, Q-RR finds a solution with accuracy ε > 0 after the following number of communication rounds:

Proof. Theorem C.2 implies

To estimate the number of communication rounds required to find a solution with accuracy ε > 0 , we need to upper bound each term from the right-hand side by ε / 3 . Thus, we get additional conditions on γ :

and also the upper bound on the number of communication rounds nT

Substituting (20) in the previous equation, we get the result.

D Missing Proofs for DIANA-RR

D.1 Proof of Theorem 2.2

Lemma D.1. Let Assumptions 1, 3, 5, 6 hold and α ≤ 1 1+ ω . Then, the iterates of DIANA-RR satisfy

Proof. Taking expectation w.r.t. Q , we obtain

Assumption 1, L max -smoothness and convexity of f i m and α ≤ 1 / (1+ ω ) imply

Summing up the above inequality for m = 1 , . . . , M , we get the result.

Theorem D.1. Let Assumptions 1, 3, 5, 6 hold and 0 < γ ≤ min { α 2 n ˜ µ , 1 ˜ L + 6 ω M L max } , α ≤ 1 1+ ω . Then, for all T ≥ 0 the iterates produced by DIANA-RR satisfy

where Ψ t is defined in (6) .

Proof. Using x i +1 ⋆ = x i ⋆ -γ M ∑ M m =1 ∇ f π i m m ( x ⋆ ) and line 9 of Algorithm 2, we derive

Taking expectation w.r.t. Q and using E ∥ ξ -c ∥ 2 = E ∥ ξ -E ξ ∥ 2 + ∥ E ξ -c ∥ 2 , we obtain

Independence of Q ( ∇ f π i m m ( x i t ) -h π i m t,m ) , m ∈ [ M ] , assumption 1, and three-point identity imply

Using L max -smoothness and µ -strong convexity of functions f i m and ˜ L -smoothness and ˜ µ -strong convexity of f π i = 1 M ∑ M i =1 f π i m m , we obtain

Taking the full expectation and using Defenition 2, we derive

Recursively unrolling the inequality, we get

Next, we apply (6) and Lemma D.1:

Recursively rewriting the inequality, we obtain

Using that + ∞ ∑ k =0 ( 1 -γ ˜ µ 2 ) k ≤ 2 ˜ µγ , we finish proof.

Corollary 7. Let the assumptions of Theorem D.1 hold, α = 1 1+ ω and

Then DIANA-RR finds a solution with accuracy ε > 0 after the following number of communication rounds:

Proof. Theorem D.1 implies

To estimate the number of communication rounds required to find a solution with accuracy ε > 0 , we need to upper bound each term from the right-hand side by ε 2 . Thus, we get an additional condition on γ :

and also the upper bound on the number of communication rounds nT

Substituting (22) in the previous equation, we get the result.

D.2 Non-Strongly Convex Summands

In this section, we provide the analysis of DIANA-RR without using Assumptions 4, 6. We emphasize that x i +1 t = x i t -γ 1 M ∑ M m =1 ˆ g π i m t,m . Then we have

We denote ˆ g t = 1 Mn ∑ n -1 i =0 ∑ M m =1 ˆ g π i m t,m .

Lemma D.2. Let Assumptions 1, 2, 3, 5 hold. Then, the following inequality holds

Proof. Since h ⋆ = ∇ f ( x ⋆ ) = 0 , the proof of Lemma D.2 is identical to the proof of Lemma C.2.

Lemma D.3. Let Assumptions 1, 2, 3, 5 hold. Then, the following inequality holds

Proof. Taking expectation w.r.t. Q , we get

Independence of Q ( ∇ f π i m m ( x i t ) -h π i m t,m ) , m ∈ [ M ] and Assumption 1 imply

Using L max -smoothness and convexity of f i m and ˜ L -smoothness and convexity of f π i , we obtain

LemmaD.4. Let α ≤ 1 1+ ω and Assumptions 1, 2, 3, 5 hold. Then, the iterates produced by DIANA-RR satisfy

Proof. Fist of all, we introduce new notation: H t = 1 Mn ∑ n -1 i =0 ∑ M m =1 E Q [ ∥ h π i m t,m -∇ f π i m m ( x ⋆ ) ∥ 2 ] . Using (21) and summing it up for i = 0 , . . . , n -1 , we obtain

Next, we apply L max -smoothness and convexity of f i m and ˜ L -smoothness and convexity of f π i :

Lemma D.5. Let Assumptions 1, 2, 3, 5 and τ ≤ 1 2 √ ˜ L ( ˜ L + ω Mn L max ) . Then, the following inequality holds

where σ 2 ⋆,n = 1 n ∑ n i =1 ∥∇ f i ( x ⋆ ) ∥ 2 .

Independence of Q ( ∇ f π i m m ( x j t ) -h π j m t,m ) , m ∈ [ M ] and Assumption 1 imply

Using L max -smoothness and convexity of f i m and ˜ L -smoothness and convexity of f π j , we obtain

Taking the full expectation and using (17), we derive

Using L max -smoothness and convexity of f i m and ˜ L -smoothness and convexity of f π j , we obtain

Now we need to estimate 2 τ 2 n E [ σ 2 t ] . Due to E [ σ 2 t ] ≤ 1 n ∑ n i =1 E [ ∥∇ f i ( x t ) ∥ 2 ] , we get

Combining two previous inequalities, we get

Summing from i = 0 to n -1 and using τ ≤ 1 2 √ ˜ L ( ˜ L + ω Mn L max ) , we obtain

We consider the following Lyapunov function:

Theorem D.2. Let Assumptions 1, 2, 3, 5 hold and

Then, for all T ≥ 0 the iterates produced by DIANA-RR satisfy

Proof. Taking expectation w.r.t. Q and using Lemma D.2, we get

Next, due to Lemma D.3 we have

Using (23), we obtain

To estimate the last term in the above inequality, we apply Lemma D.4:

Let H t = cτ 2 Mn ∑ n -1 i =0 ∑ M m =1 E [ ∥ h π i m t,m -∇ f π i m m ( x ⋆ ) ∥ 2 ] . Taking the full expectation and using Lemma D.5, we get

Selecting c = Aω αMn , where A is a positive number to be specified later, we have

Then, we have

Taking τ = 1 B ( ˜ L + ( A +1) ω Mn L max ) , where B is some positive constant, we obtain

Choosing A = 10 , B = 12 , τ ≤ α µ , we have

Recursively unrolling the inequality, substituting τ = nγ and using + ∞ ∑ t =0 ( 1 -τµ 2 ) t ≤ 2 µτ , we finish proof.

Corollary 8. Let the assumptions of Theorem D.2 hold, α = 1 1+ ω , and

Then, DIANA-RR finds a solution with accuracy ε > 0 after the following number of communication rounds:

To estimate the number of communication rounds required to find a solution with accuracy ε > 0 , we need to upper bound each term from the right-hand side by ε 2 . Thus, we get an additional condition on γ :

and also the upper bound on the number of communication rounds nT

Substituting (24) in the previous equation, we obtain the result.

E Theoretical Results for Q-NASTYA and DIANA-NASTYA

Theorem E.1. Let Assumptions 1, 2, 3 hold. Let the stepsizes γ , η satisfy 0 < η ≤ 1 16 L max ( 1+ ω M ) , 0 < γ ≤ 1 5 nL max . Then, for all T ≥ 0 the iterates produced by Q-NASTYA (Algorithm 3) satisfy

Corollary 9. Under the same conditions as Theorem E.1 and for Algorithm 3, there exist stepsizes γ = η / n and η > 0 such that the number of communication rounds T to find a solution with accuracy ε > 0 is ˜ O ( L max µ ( 1 + ω M ) + ω M ζ 2 ⋆ εµ 3 + √ L max εµ 3 √ ζ 2 ⋆ + σ 2 ⋆ n ) . If γ → 0 , one can choose η > 0 such that the above complexity bound improves to ˜ O ( L max µ ( 1 + ω M ) + ω M ζ 2 ⋆ εµ 3 ) .

We emphasize several differences with the known theoretical results. First, the FedCOM method of Haddadpour et al. [2021] was analyzed in the homogeneous setting only, i.e., f m ( x ) = f ( x ) for all m ∈ [ M ] , which is an unrealistic assumption for FL applications. In contrast, our result holds in the fully heterogeneous case. Next, the analysis of FedPAQ of Reisizadeh et al. [2020] uses a bounded variance assumption, which is also known to be restrictive. Nevertheless, let us compare to their result. Reisizadeh et al. [2020] derive the following complexity for their method: ˜ O ( L max µ ( 1 + ω M ) + ω M σ 2 µ 2 ε + σ 2 Mµ 2 ε ) . This result is inferior to the one we show for Q-NASTYA : when ω is small, the main term in the complexity bound of FedPAQ is ˜ O ( 1 / ε ) , while for Q-NASTYA the dominating term is of the order ˜ O ( 1 / √ ε ) (when ω and ε are sufficiently small). We also highlight that FedCRR [Malinovsky and Richtárik, 2022] does not converge if ω > M 2 γµε / (2 ∥ x n ∗ ,m ∥ 2 ) , while Q-NASTYA does for any ω ≥ 0 . Finally, when ω = 0 (no compression) we recover NASTYA as a special case, and using γ = η / n , we recover the rate of FedRR [Mishchenko et al., 2021]. Theorem E.2. Let Assumptions 1, 2, 3 hold. Suppose the stepsizes γ , η, α satisfy 0 < γ ≤ 1 16 L max n , 0 < η ≤ min { α 2 µ , 1 16 L max ( 1+ 9 ω M ) } , and α ≤ 1 1+ ω . Define the following Lyapunov function:

Then, for all T ≥ 0 the iterates produced by DIANA-NASTYA (Algorithm 4) satisfy

Corollary 10. Under the same conditions as Theorem E.2 and for Algorithm 4, there exist stepsizes γ = η / n , η > 0 , α > 0 such that the number of communication rounds T to find a solution with accuracy ε > 0 is ˜ O ( ω + L max µ ( 1 + ω M ) + √ L max εµ 3 √ ζ 2 ⋆ + σ 2 ⋆ n ) . If γ → 0 , one can choose η > 0 such that the above complexity bound improves to ˜ O ( ω + L max µ ( 1 + ω M ) ) .

In contrast to Q-NASTYA , DIANA-NASTYA does not suffer from the ˜ O ( 1 / ε ) term in the complexity bound. This shows the superiority of DIANA-NASTYA to Q-NASTYA . Next, FedCRR-VR [Malinovsky and Richtárik, 2022] has the rate ˜ O ( ( ω +1) ( 1 -1 κ ) n ( 1 -( 1 -1 κ ) n ) 2 + √ κ ( ζ ⋆ + σ ⋆ ) µ √ ε ) , which depends on ˜ O ( 1 / √ ε ) . However, the first term is close to ˜ O ( ( ω +1) κ 2 ) for a large condition number. FedCRR-VR-2 utilizes variance reduction technique from Malinovsky et al. [2021] and it allows to get rid of permutation variance. This method has ˜ O   ( ω +1) ( 1 -1 κ √ κn ) n 2 ( 1 -( 1 -1 κ √ κn ) n 2 ) 2 + √ κζ ⋆ µ √ ε   complexity, but it requires additional assumption on number of functions n and thus not directly comparable with our result. Note that if we have no compression ( ω = 0) , DIANA-NASTYA recovers rate of NASTYA .

F Missing Proofs for Q-NASTYA

We start with deriving a technical lemma along with stating several useful results from [Malinovsky et al., 2022]. For convenience, we also introduce the following notation:

Lemma F.1. Let Assumptions 1, 2, 3 hold. Then, for all t ≥ 0 the iterates produced by Q-NASTYA satisfy

where E Q is expectation w.r.t. Q , and ζ 2 ⋆ = 1 M ∑ M m =1 ∥∇ f m ( x ⋆ ) ∥ 2 .

Proof. Using the variance decomposition E [ ∥ ξ ∥ 2 ] = E [ ∥ ξ -E [ ξ ] ∥ 2 ] + ∥ E ξ ∥ 2 , we obtain

Next, we use ∇ f m ( x t ) = 1 n ∑ n -1 i =0 ∇ f π i m m ( x t ) and ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 +2 ∥ b ∥ 2 :

Using L i,m -smoothness of f i m and f and also convexity of f m , we obtain

Lemma F.2 (see [Malinovsky et al., 2022]) . Under Assumptions 1, 2, 3, it holds

Lemma F.3 (see [Malinovsky et al., 2022]) . Under Assumptions 1, 2, 3 and γ ≤ 1 2 L max n , it holds

Theorem F.1. Let Assumptions 1, 2, 3 hold and stepsizes γ , η satisfy

Then, for all T ≥ 0 the iterates produced by Q-NASTYA satisfy

Proof. Taking expectation w.r.t. Q and using Lemma F.1, we get

In view of (27), we have

Recursively unrolling the inequality and using + ∞ ∑ t =0 ( 1 -ηµ 2 ) t ≤ 2 µη , we get the result.

Corollary 11. Let the assumptions of Theorem E.1 hold, γ = η / n , and

Then, Q-NASTYA finds a solution with accuracy ε > 0 after the following number of communication rounds:

If γ → 0 , one can choose η = min { 1 16 L max ( 1+ ω M ) , εµM 24 ωζ 2 ⋆ } such that the above complexity bound improves to

To estimate the number of communication rounds required to find a solution with accuracy ε > 0 , we need to upper bound each term from the right-hand side by ε / 3 . Thus, we get additional conditions on η :

and also the upper bound on the number of communication rounds T

Substituting (31) in the previous equation, we get the first part of the result. When γ → 0 , the proof follows similar steps.

G Missing Proofs for DIANA-NASTYA

Lemma G.1. Under Assumptions 1, 2, 3, the iterates produced by DIANA-NASTYA satisfy

where h ⋆ = ∇ f ( x ⋆ ) .

Proof. Using that E Q [ˆ g t,m ] = g t,m and definition of h ⋆ , we get

Next, three-point identity and L max -smoothness of each function f i m imply

Finally, using µ -strong convexity of f , we finish the proof of lemma.

Lemma G.2. Under Assumptions 1, 2, 3, the iterates produced by DIANA-NASTYA satisfy

Proof. Since g t = 1 M ∑ M m =1 g t,m and E ∥ ξ -c ∥ 2 = E ∥ ξ -E ξ ∥ 2 + E ∥ E ξ -c ∥ 2 , we have

Next, independence of Q ( g t,m -h t,m ) , m ∈ M , Assumption 1, and L max -smoothness and convexity of each function f i m imply

Using L max -smoothness and convexity of f m , we get

Lemma G.3. Under Assumptions 1, 2, 3, and α ≤ 1 1+ ω , the iterates produced by DIANA-NASTYA satisfy

Proof. Taking expectation w.r.t. Q and using Assumption 1, we obtain

Finally, L max -smoothness and convexity of f m imply

Theorem G.1. Let Assumptions 1, 2, 3 hold and stepsizes γ , η, α satisfy

Then, for all T ≥ 0 the iterates produced by DIANA-NASTYA satisfy

Proof. We have

Taking expectation w.r.t. Q and using Lemma G.1, we obtain

Using (6) and Lemma G.3, we get

Taking the full expectation, we derive

In view of (29), we have

Using definition of Lyapunov function and using + ∞ ∑ t =0 ( 1 -ηµ 2 ) t ≤ 2 µη , we get the result.

Corollary 12. Let the assumptions of Theorem E.2 hold, γ = η / n , α = 1 1+ ω , and

Then, DIANA-NASTYA finds a solution with accuracy ε > 0 after the following number of communication rounds:

If γ → 0 , one can choose η = min { α 2 µ , 1 16 L max ( 1+ 9 ω M ) } such that the number of communication rounds T to find solution with accuracy ε > 0 is

To estimate the number of communication rounds required to find a solution with accuracy ε > 0 , we need to upper bound each term from the right-hand side by ε 2 . Thus, we get an additional restriction on η :

and also the upper bound on the number of communication rounds T

Substituting (31) in the previous equation, we get the first part of the result. When γ → 0 , the proof follows similar steps.

H Alternative Analysis of Q-NASTYA

In this analysis, we will use additional sequence:

Theorem H.1. Let Assumptions 1, 3, 4 hold. Moreover, we assume that (1 -γµ ) n ≤ 9 / 10 -1 / C 1+ 1 / C = ̂ C < 1 for some numerical constant C > 1 . Also let β = η γn ≤ 1 3 C ω M +1 and γ ≤ 1 L max . Then, for all T ≥ 0 the iterates produced by Q-NASTYA satisfy

where ˆ ∆ ⋆ = 1 M ∑ M m =1 ∥ x n ⋆,m -x ⋆ ∥ 2 and ˆ σ 2 rad ≤ L max ( ζ 2 ⋆ + nσ 2 ⋆ / 4 ) .

Proof. The update rule for one epoch can be rewritten as

Using this, we derive

Taking conditional expectation w.r.t. the randomness comming from compression, we get

Next, we use the definition of quantization operator and independence of Q ( x t -x n t,m γn ) , m ∈ [ M ] :

Since β = η γn , we obtain

Using the condition that x ∗ = 1 M ∑ M m =1 x n ∗ ,m we have:

Convexity of squared norm and Jensen's inequality imply

Next, from Young's inequality we get

Theorem 4 from [Mishchenko et al., 2021] gives

It leads to

Next, applying β ≤ 1 1+3 C ω M , we derive

Finally, we have

I Alternative Analysis of DIANA-NASTYA

Theorem I.1. Let Assumptions 1, 3, 4 hold. Moreover, we assume that (1 -γµ ) n ≤ 9 / 10 -1 / B 1+ 1 / B = ̂ B < 1 for some numerical constant B > 1 . Also let β = η γn ≤ 1 12 B ω M +1 and γ ≤ 1 L max and also α ≤ 1 ω +1 . Then, for all T ≥ 0 the iterates produced by DIANA-NASTYA satisfy

Proof. We start with expanding the square:

Taking the expectation w.r.t. Q , we get

Next, using definition of g t,m , we obtain

Let us consider recursion for control variable:

Taking the expectation w.r.t. Q , we have

E Q ∥ h t +1 ,m -h ∗ ,m ∥ 2 ≤ ∥ h t,m -h ∗ ,m ∥ 2 +2 α ⟨ g t,m -h t,m , h t,m -h ∗ ,m ⟩ + α 2 ( ω +1) ∥ g t,m -h t,m ∥ 2 . Using α ≤ 1 we have

ω +1

Using this bound we get that

Let us consider Lyapunov function:

Using previous bounds and Theorem 4 from [Mishchenko et al., 2021] we have

Let us consider

Putting all the terms together and using (1 -γµ ) n ≤ 9 / 10 -1 / B 1+ 1 / B = ̂ B < 1 , β ≤ 1 12 B ω M +1 we have

Unrolling this recursion we get the final result.

Algorithm 5 Q-NASTYA-PP

Input: x 0 - starting point, γ > 0 - local stepsize, η > 0 - global stepsize

2: Sample a cohort S t with cardinality C uniformly

3: for m ∈ S t in parallel do

4: Receive x t from the server and set x 0 t,m = x t

5: Sample random permutation of [ n ] : π m = ( π 0 m , . . . , π n -1 m )

8: end for

10: end for

12: Compute x t +1 = x t -ηg t and send x t +1 to the workers

13: end for

Output: x T

J Partial Participation for Method with Local Steps

J.1 Analysis of Q-NASTYA with Partial Participation

Lemma J.1. Let Assumptions 1, 2, 3 hold. Then, for all t ≥ 0 the iterates produced by Q-NASTYA-PP (Algorithm 5) satisfy

where E Q ,S t is expectation w.r.t. Q , S t and σ 2 ⋆ = 1 M ∑ M m =1 ∥∇ f m ( x ⋆ ) ∥ 2 .

̸

Using independence between ξ m and ξ l for different m,l and Using (2), (3), we get

Using L -smoothness of f i m and f and also convexity of f m , we obtain

Taking expectation w.r.t. S t and using uniform sampling, we receive

Theorem J.1. Let step sizes η, γ satisfy the following equations

Under the Assumptions 1, 2, 3 iterates of Q-NASTYA-PP (Algorithm 5) satisfy

where

As we can see, there is an additional error term proportional to M -C C max( M -1 , 1) that arises due to client sampling in the partial participation setting. Note that when C = M (all clients are participating), this error term vanishes, allowing us to recover the previous result for the full participation case. This shows the consistency of our theoretical framework across different participation scenarios.

Proof.

Taking expectation w.r.t. Q , S and using Lemma J.1 updated, we get

Finally, we receive

Recursively rewriting the inequality and using ∑ + ∞ t =0 ( 1 -ηµ 2 ) t ≤ 2 µη , we finish proof.

Algorithm 6 DIANA-NASTYA-PP

Input: x 0 - starting point, { h 0 ,m } M m =1 - initial shift-vectors, γ > 0 - local stepsize, η > 0 - global stepsize, α > 0 - stepsize for learning the shifts

1: for t = 0 , 1 , . . . , T -1 do

2: Sample a cohort S t with cardinality C uniformly

3: for m ∈ S t in parallel do

4: Receive x t from the server and set x 0 t,m = x t

5: Sample random permutation of [ n ] : π m = ( π 0 m , . . . , π n -1 m )

6: for i = 0 , 1 , . . . , n -1 do

7: Set x i +1 t,m = x i t,m -γ ∇ f π i m m ( x i t,m )

8: end for

9: Compute g t,m = 1 γn ( x t -x n t,m ) and send Q t ( g t,m -h t,m ) to the server

11: Set ˆ g t,m = h t,m + Q t ( g t,m -h t,m )

12: end for

13: h t +1 = 1 C ∑ m ∈ S t h t +1 ,m = h t + α C ∑ m ∈ S t Q t ( g t,m -h t,m )

15: x t +1 = x t -η ˆ g t

14: ˆ g t = 1 C ∑ m ∈ S t ˆ g t,m = h t + 1 C ∑ m ∈ S t Q t ( g t,m -h t,m )

16: end for

Output:

x T

J.2 Analysis of DIANA-NASTYA with Partial Participation

Theorem J.2. Let step sizes η, γ satisfy the following equations

Define the Lyapunov function:

where A = λη 2 . Selecting parameters α = 1 1+ ω ; λ = 8 ω αM , γ = 1 5 nL max , also using η ≤ min [ C µ (1+ ω ) M , 1 80 L max ( 1+ ω C ) ] Under the Assumptions 1, 2, 3 iterates of DIANA-NASTYA-PP (Algorithm 6) satisfy

Note that we eliminate the variance term proportional to ω : 8 η µ ω C σ 2 ⋆ . In the Partial Participation regime, we have a variance term proportional to ( M -C ) C max(1 ,M -1) , which equals zero if C = M . This term decreases as O ( 1 C ) , so we achieve the expected linear speedup.

Proof. STEP 1: we need to estimate inner product. By ˆ g t = 1 C ∑ m ∈ S t ˆ g t,m , we have

Taking expectation by subsamling, we have

Thus, we have

STEP 3: Note that

Taking expectation by compression, we have

Taking expectation by subsampling, we have

Thus, we have

STEP 4: Defining Lyapunov function as follows

we have

Setting A = λη 2 and using Lemma F.3, we have

Selecting α = 1 1+ ω ; λ = 8 ω αM ; η ≤ C µ (1+ ω ) M , also using η = 1 80 L max ( 1+ ω C ) , γ = 1 5 nL max and applying previous steps we obtain

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly outline our contributions in the abstract and introduction, and we also include a dedicated Contributions section.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We clearly highlight all assumptions and limitations in the text.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The main contribution of the paper is its theoretical analysis. We support the paper with necessary definitions, assumptions, and lemmas.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper is supported by reproducible experiments, with all stochastic elements from pseudo-random generators fixed in advance. For details, see the supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We aim to make the paper and all source code for experiments open-sourced to accelerate scientific findings in the field of Federated Learning and Machine Learning in general. For details, see the supplementary materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide detailed guidelines for experiments setup in Appendix and in the folder with experiment source code. For details, see the supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide detailed guidelines for experiments setup in Appendix and in the folder with experiment source code. For details, see the supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provides information on the computer resources in Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research focuses on mathematical objects and does not involve human subjects or participants. The data used for our experiments consists of publicly available datasets. Our work does not explicitly address or examine the social implications of applying this research in practice.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work operates on mathematical objects, and the essence of our work provides a new optimization algorithm. Because of theoretical nature of our work the impact discussion is not applied.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper provides an optimization algorithm with the required theory. The data or models are not output assets of our work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We provide references for used datasets and deep learning models used in experiments.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The output assets of our paper is Algorithm and Source code for experiments. Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes compressed Federated Learning algorithms in the setting of Random Reshuffling. It first provides theory for RR with compression (Q-RR), and several variants (based on existing methods for compressed SGD) which have improved convergence guarantees. The paper sets the hypothesis that naive Q-RR does not yield improvements compared to Q-SGD (based on theory and experiments). However, the theoretical results show that the proposed DIANA-RR indeed improves over its SGD-variant when the number of samples is comparatively small to the number of iterations.\n\nSTRENGTHS:\nThe proposed algorithms are the natural extensions of existing methods (DIANA, NASTYA,..) for the RR setting. All methods are supported with theory for the strongly convex, smooth case. The theory further sheds light on possible improvements from RR in the setup of Federated Learning with compressed communication.\nThe provided experiments nicely complement the theoretical results (with some caveats, see below). One of the main takeaways (also represented in the title of the paper) is that the DIANA-variance-reduction technique for compression yields improvements from RR compared to the SGD-variant.\n\nWEAKNESSES:\n1) On the main hypothesis that reducing the compression variance brings improvement, it is yet unclear whether this is also a necessary condition. The paper does provide a convergence result for Q-RR that does not improve over Q-SGD; but what would be preferable is a lower bound that shows how Q-RR can not improve over Q-SGD in a worst-case example. This is currently missing to complete the picture. While the experimental result presented in the main text also support the hypothesis, there are some caveats (see next point).\n\n2) The results for CIFAR10 in the main text suggest that Q-RR is on par with Q-SGD. However, in the appendix it becomes clear that this experiment refers to only training the last layer (see Fig. 12). When training the whole network (Fig. 10), the results suggest that Q-RR is actually much better than Q-SGD, and the gap is roughly equal to DIANA vs. DIANA-RR. The main text does not mention this, so this result is hard to find if it's only presented in the appendix. I would ask the authors to clarify this part in the main text - in particular, because this result weakens one of the hypotheses that the DIANA technique is necessary to get improvements (at least for nonconvex problems, like the ResNet example is). If I missed something important here, please let me know - I am happy to discuss details.\n\n3) The logistic regression examples show a clearer picture, but only three datasets are tested, and all of them are very low-dimensional ($d<500$). I would suggest to add more high-dimensional datasets, or other deep learning experiments to have more insight whether there is a difference between Q-SGD and Q-RR. \n\n4) All theoretical results only apply to strongly convex, smooth functions. While this is a standard framework for optimization analysis, it is very far from most modern machine learning applications, thus restricting the applicability of the results for such nonconvex, nonsmooth problems.\n\nQUESTIONS:\nThis question relates more broader to Random Reshuffling: if you use mini-batching, and do RR with the standard Pytorch framework, then the composition of the mini-batches will be different in every epoch (the DataLoader shuffles, and then partitions into batches). In that case, the theoretical framework of the paper does not fully apply anymore, correct? This is, because in every epoch you would have a different set of $f_m , m=1,..,M$ where each $f_m$ is a mini-batch loss.\n\nIs there a way to overcome this small incosistency between Random Reshuffling analysis and the actual usage? Or is there RR analysis that explicitly takes into account mini-batching?\n\nLIMITATIONS:\nI commented on possible limitations in terms of content above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose communication-efficient distributed learning algorithms with faster convergence rates than previous methods. Their starting point is mini-batch stochastic gradient descent (SGD): Previous works have shown that random reshuffling (RR), i.e. sampling mini-batches without replacement (in other words, having clearly defined training epochs), has better theoretical convergence guarantees than sampling with replacement. \n\nInspired by this, in the first part of the paper, the authors propose the most natural extension of RR to the distributed learning setting: the clients quantise the gradients they computed to communicate the gradients to the server efficiently. The authors derive a bound on the quantised RR (Q-RR) convergence rate (assuming the loss is convex and its gradient is Lipschitz) and show that as the quantisation error goes to 0, their bound recovers the convergence results for non-quantised RR. However, when the quantisation error is nonzero, Q-RR no longer has an advantage over quantised SGD (Q-SGD), as quantisation introduces an additional error term that negates the benefits of RR. The authors use a variance reduction technique called DIANA to deal with this issue and name this improved method DIANA-RR. They show that DIANA-RR removes the troublesome error term and attains the faster convergence rate of RR even when the gradients are quantised. \n\nIn the second part of the paper, the authors observe the same behaviour when the clients perform the model updates and send quantised weight differences instead of gradients. That is, the authors show that quantising the weight differences introduces too much noise, which negates the benefits of RR. However, introducing the same variance reduction technique as in the SGD case restores the benefits of RR.\n\nFinally, the authors perform some basic federated learning experiments and show that their proposed methods outperform the competitors even when the losses involved are non-convex.\n\nSTRENGTHS:\nThe authors provide a rigorous, insightful analysis of simple extensions of previous methods. Concretely, their analysis reveals the flaws of the naively quantising SGD using random reshuffling, which motivates the more involved, control-iterate-based extensions. They then show that the more involved methods retain the benefits of their original, unquantised counterparts. I checked the proofs of theorems in appendices C.1 and D.1, and I believe they are correct.\n\nWEAKNESSES:\nI should note that I am not an expert in federated learning, so I cannot fully evaluate the significance of the authors' contributions or how realistic some of their assumptions are in practice.\n\nFrom a technical perspective, my main concern is that I am unsure how challenging it is to realise random reshuffling in practice. As I understand, using the authors' algorithms does not allow the server to start the next training epoch before every client sends updates for the current epoch. This sounds like a severe limitation, as in the worst case, some of the clients might drop out (e.g. due to connectivity issues) and never send any updates, causing the training loop to get stuck. Could the authors please comment on this? If this is a genuine concern, then the authors should discuss this limitation in the main text.\n\nThe other issue with the paper is the writing needs to be improved significantly. While the introduction section is well-written, it is three pages long, which means there is not enough space to expose the large amount of content that the authors wish to include at the level of formality they want to use. Concretely, the paper goes from a high-level discussion in the introduction section to being immensely technical in section two, with many new quantities and notations introduced without explaining what they represent. For example, what is the interpretation of the shuffling radius, or the quantity in Eq (6)? However, even more importantly, the methods the authors base their work on, namely DIANA and NASTYA, are not explained; the reader only has the pseudocode given in Algorithms 2-4 to work with. The best illustration of this is the disconnect between the paper title and its contents: there is no explicit mention of compressing gradient differences in the main text. I believe the authors' use of DIANA inspires the title, but I am not fully sure. Furthermore, the pseudocodes in Algorithms 1-4 are ambiguous, as the authors mix what happens on the client's side and on the server's side. I suggest the authors break up the code into two parts, one demonstrating the server-side operations and one showing the clients' side.\n\nBesides this, more minor weaknesses include:\n - I find Eq (1) and similar equations combining definitions with statements very confusing; please separate the two.\n - As defined, assumption (1) doesn't make sense; the expectations have no meaning as $Q$ is not stochastic; please make the meaning precise.\n - The symbol $n$ denoting the batch size is technically undefined; it should be included in the chain of equalities on line 34\n - Table 1 is difficult to interpret, and it is unclear to me what value it brings to the paper. I would either move it to the appendix or cut it altogether and use the space to address my main concern, which I outlined above.\n - Over what is the expectation in Def 2.1?\n - Figures 2 and 3 should be renamed to Figures 1a and 1b, respectively.\n - Merge the two panels in Figure 4\n - Algorithm 1, steps 2 and 3, $x_{t, m}^0$: the symbol $m$ is undefined\n\nQUESTIONS:\nn/a\n\nLIMITATIONS:\nThe authors don't discuss any of their methods' limitations (see, e.g., my main technical concern in the weaknesses section); I think such a discussion would significantly strengthen the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGradient compression is a popular technique for improving the communication complexity of stochastic first-order methods in the distributed training of machine learning models. This work points out that existing work only considers with-replacement sampling of stochastic gradients, while in practice stochastic methods based on without-replacement sampling performs better. This paper fills this gap in the literature by analyzing for the first time methods that combine gradient compression and putback-free sampling.\n\nSTRENGTHS:\n(1) This work introduces a unique combination of gradient compression and random reshuffling, which addresses a gap in the existing literature. This innovative approach aims to improve convergence rates and communication efficiency in distributed learning.\n\n(2) This work summarizes previous algorithms while providing rigorous theoretical analysis and convergence proofs for the proposed new methods (Q-RR, DIANA-RR, Q-NASTYA, DIANA-NASTYA).This solid theoretical foundation helps in understanding the behavior and benefits of the new algorithms.\n\n(3) The article is clearly structured, logically coherent and step-by-step. The research is clearly presented.\n\nWEAKNESSES:\nWhile the experiments conducted are thorough, they are limited to certain types of tasks (logistic regression and ResNet-18 on CIFAR-10). The generalizability of the results to other models and datasets is not fully explored, which could limit the broader applicability of the findings.\n\nQ-NASTYA and DIANA-NASTYA algorithms are too dependent on hyperparameters, which requires additional computation and management, affecting the practical application and deployment of the algorithms, especially in large-scale distributed environments.\n\nQUESTIONS:\nWhen tuning parameters (e.g. step size), could you try a more efficient way of tuning the parameters to increase the scalability of the algorithm? Whether there are plans to develop automated methods for tuning parameters to reduce the complexity of manual tuning and increase the feasibility of the algorithm in practical applications?\n\nDue to the increased implementation complexity of the algorithms proposed by DIANA-RR et al. are there any performance evaluations in practical applications, especially how well they perform in large-scale distributed systems?\n\nLIMITATIONS:\nThe algorithm in this work relies on many parameters, but lacks corresponding experimental proofs. For example, DIANA-RR uses multiple shift vectors (n of them), while DIANA uses only 1 shift vector, and there are no experimental results showing the effect of different number of shift vectors on the performance. There are no detailed experimental results showing the effect of different compression levels on the performance of the algorithm at different compression levels. etc.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provide analysis of methods with gradient compression and without-replacement sampling. Based on this analysis, this paper proposes several new algorithms for distributed optimization with communication, including Q-RR and its vairants Q-NASTYA, which compress gradient differences in the scenario of data random shuffling. Empirical results on logistic regression and resnet-18 on cifar10 show that the proposed algorithm outperforms the baselines.\n\nSTRENGTHS:\n1. This paper provide analysis of methods with gradient compression and without-replacement sampling. \n\n2. Based on this analysis, this paper proposes several new algorithms for distributed optimization with communication, including Q-RR and its vairants Q-NASTYA, which compress gradient differences in the scenario of data random shuffling. \n\n3. Empirical results on logistic regression and resnet-18 on cifar10 show that the proposed algorithm outperforms the baselines.\n\nWEAKNESSES:\n1. The idea of compressing gradient differences is actually not very novel. EF21 [1] proposed compressing gradient differences a long time ago. The algorithm of EF21 also looks very similar to DIANA-RR. I think DIANA-RR is basically a combination of DIANA and EF21. Unfortunately, I lost track of the huge number of variants of DIANA, so I'm not exactly sure whether the variant of DIANA + EF21 has already existed somewhere.\n\n2. Since EF21 already uses compression of gradient differences, I think EF21 is a better baseline than QSGD. However, EF21 is not included in the experiments.\n\n\n--------------------\nReferences:\n\n[1] Richtárik, Peter et al. “EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedback.” Neural Information Processing Systems (2021).\n\nQUESTIONS:\n1. Regardless of DIANA itself, what exactly is the difference between the \"compress gradient differences\" proposed in this paper and the algorithm of EF21?\n\n2. Could EF21 also be included in the experiments as a baseline?\n\nLIMITATIONS:\nExcept for the weakness mentioned above, the limitations of this paper is well discussed and addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}