Title: SCAFFLSA: Taming Heterogeneity in Federated Linear Stochastic Approximation and TD Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Stochastic Approximation

Reinforcement learning

Federated Learning

Machine Learning
Abstract: In this paper, we analyze the sample and communication complexity of the federated linear stochastic approximation (FedLSA) algorithm. We explicitly quantify the effects of local training with agent heterogeneity. We show that the communication complexity of FedLSA scales polynomially with the inverse of the desired accuracy ϵ. To overcome this, we propose SCAFFLSA a new variant of FedLSA that uses control variates to correct for client drift, and establish its sample and communication complexities. We show that for statistically heterogeneous agents, its communication complexity scales logarithmically with the desired accuracy, similar to Scaffnew. An important finding is that, compared to the existing results for Scaffnew, the sample complexity scales with the inverse of the number of agents, a property referred to as linear speed-up. Achieving this linear speed-up requires completely new theoretical arguments. We apply the proposed method to federated temporal difference learning with linear function approximation and analyze the corresponding complexity improvements.

SCAFFLSA : Taming Heterogeneity in Federated Linear Stochastic Approximation and TD Learning

Abstract

In this paper, we analyze the sample and communication complexity of the federated linear stochastic approximation ( FedLSA ) algorithm. We explicitly quantify the effects of local training with agent heterogeneity. We show that the communication complexity of FedLSA scales polynomially with the inverse of the desired accuracy ϵ . To overcome this, we propose SCAFFLSA , a new variant of FedLSA that uses control variates to correct for client drift, and establish its sample and communication complexities. We show that for statistically heterogeneous agents, its communication complexity scales logarithmically with the desired accuracy, similar to Scaffnew [37]. An important finding is that, compared to the existing results for Scaffnew , the sample complexity scales with the inverse of the number of agents, a property referred to as linear speed-up . Achieving this linear speed-up requires completely new theoretical arguments. We apply the proposed method to federated temporal difference learning with linear function approximation and analyze the corresponding complexity improvements.

1 Introduction

Heterogeneity has a major impact on communication complexity in federated learning (FL) [28, 36]. In FL, multiple agents use different local oracles to update a global model together. A central server then performs a consensus step to incrementally update the global model. Since communication with the server is costly, reducing the frequency of the consensus steps is a central challenge. At the same time, limiting communications induces client drift when agents are heterogeneous, biasing them towards their local solutions. This issue has mostly been discussed for FL with stochastic gradient methods [23, 51]. In this paper, we investigate the impact of heterogeneity in the field of federated linear stochastic approximation (federated LSA). The goal is to solve a system of linear equations where (i) the system matrix and the corresponding objective are only accessible via stochastic oracles, and (ii) these oracles are distributed over an ensemble of heterogeneous agents. This problem can be solved with the FedLSA method, which performs LSA locally with periodic consensus steps. This approach suffers from two major drawbacks: heterogeneity bias, and high variance of local oracles.

A popular means of overcoming heterogeneity problems is the method of control variables, which goes back to the line of research initiated by [23]. However, existing results on the complexity of these methods tend to neglect the linear decrease of the mean squared error (MSE) of the algorithm with the number of agents N [37], or they require a lot of communication [23]. In this paper, we

MZBUAI

O

O

O

show that it is possible to reduce communication complexity using control variates while preserving the linear speed-up in terms of sample complexity. Our contributions are the following:

We provide the sample and communication complexity of the FedLSA algorithm, inspired by the work of [51]. Our analysis highlights the relationship between the MSE of the FedLSA method and three key factors: the number of local updates, the step size, and the number of agents. We provide an exact analytical formulation of the algorithm's bias, which is confirmed in our numerical study. We also give results under Markovian noise sampling.

We propose SCAFFLSA , a method that provably reduces communication while maintaining linear speed-up in the number of agents. This method uses control variates to allow for extended local training. We establish finite sample and communication complexity for SCAFFLSA . Our study is based on a new analysis technique, that carefully tracks the fluctuations of the parameters and ccommunicationsontrol variates. This allows to prove that SCAFFLSA simultanously maintains linear speedup and reduced communication . To our knowledge, this is the first time that these two phenomenons are proven to occur simultaneously in FL.

We apply both these methods to TD learning with linear function approximation, where heterogeneous agents collaboratively estimate the value function of a common policy.

We provide a synthetic overview of this paper's theoretical results in Table 1 in the general federated LSA setting, and we instantiate these results for federated TD learning in Table 2 (Appendix E). We start by discussing related work in Section 2. We then introduce federated LSA in Section 3, and analyze it in Section 4. In Section 5 we introduce SCAFFLSA , a novel strategy to mitigate the bias. Finally, we illustrate our results numerically in Section 6. Since an important application of LSA is TD learning [47] with linear function approximation, we instantiate the results of Section 3-5 for federated TD learning.

Notations. For matrix A we denote by ∥ A ∥ its operator norm. Setting N for the number of agents, we use the notation E c [ a c ] = N -1 ∑ N c =1 a c for the average over different clients. For the matrix A = A ⊤ ⪰ 0 , A ∈ R d × d and x ∈ R d we define the corresponding norm ∥ x ∥ A = √ x ⊤ Ax . For sequences a n and b n , we write a n ≲ b n if there exists a constant c > 0 such that a n ≤ cb n for n ≥ 0 .

2 Related Work

Federated Learning. With few exceptions (see e.g. [12]), most of the FL literature is devoted to federated stochastic gradient (SG) methods. A strong focus has been placed on the Federated Averaging ( FedAvg ) algorithm [36], which aims to reduce communication through local training, resulting in local drift when agents are heterogeneous [53]. Sample and communication complexity of FedAvg were investigated under a variety of conditions covering both homogeneous [31, 20] and heterogeneous agents [25, 27]. Different ways of measuring heterogeneity for FedAvg have then been proposed [51, 41]. In [44] it was also shown that FedAvg yields linear speedup in the number of agents when gradients are stochastic, a phenomenon that we prove is still present in FedLSA .

In order to correct the client drift of FedAvg , [23] proposed Scaffold , a method that tames heterogeneity using control variates. [17, 38] prove that Scaffold retrieves the rate of convergence of the gradient descent independently of heterogeneity, although without benefit from local training. It has

Algorithm 1 FedLSA

been shown in [37] (with the analysis of ProxSkip , which generalizes Scaffold ) that such methods accelerate training. However, unlike Scaffold , the analysis of [37] loses the linear speedup in the number of agents. Several other methods with accelerated rates have been proposed [35, 5, 6, 18, 21], albeit all of them lose the linear speedup. Contrary to these papers, we show that our approach to FedLSA with control variates preserves both the acceleration and the linear speedup .

Federated TD learning. Temporal difference (TD) learning has a long history in policy evaluation [47, 9], with the asymptotic analysis under linear function approximation (LFA) setting performed in [49, 48]. Several non-asymptotic MSE analyses have been carried out in [4, 8, 42, 32, 45]. Much attention has been paid to federated reinforcement learning [33, 43, 52] and federated TD learning with LFA. [26, 7, 34] provides an analysis under the strong homogeneity assumption. Federated TD was also investigated with heterogeneous agents, first without local training [12], then with local training but without linear acceleration [11, 22]. Recently, [50] proposed an analysis of federated TD with heterogeneous agents, local training, and linear speed-up in number of agents. However, [50] do not mitigate the local drift effects, and their conclusions are valid only in the low-heterogeneity setting. In high heterogeneity settings, their analysis exhibits a large bias. Additionally, their analysis requires the server to project aggregated iterates to a ball of unknown radius. In contrast, our analysis shows that FedLSA converges to the true solution without bias even without such projection.

3 Federated Linear Stochastic Approximation and TD learning

3.1 Federated Linear Stochastic Approximation

In federated linear stochastic approximation, N agents collaboratively solve a system linear equation system with the following finite sum structure

where for c ∈ [ N ] , ¯ A c ∈ R d × d , ¯ b c ∈ R d . We assume the solution θ ⋆ to be unique, and that each local system ¯ A c θ c ⋆ = ¯ b c also has a unique solution θ c ⋆ . The values of ¯ A c 's and ¯ b c 's can be different, representing the different realities of the agents. In federated LSA, neither matrices ¯ A c nor vectors ¯ b c are observed directly. Instead, each agent c ∈ [ N ] has access to its own observation sequence ( Z c k ) k ∈ N , that are independent from one agent to another. Agent c obtains estimates { ( A c ( Z c k ) , b c ( Z c k )) } k ∈ N of ¯ A c and ¯ b c , where A c : Z → R d × d and b c : Z → R d are two measurable functions. Naturally, we define the error of estimation of ¯ A c and ¯ b c as ˜ b c ( z ) = b c ( z ) -¯ b c , ˜ A c ( z ) = A c ( z ) -¯ A c . This allows to measure the noise at local and global solutions as

together with the associated covariances,

that are finite whenever one of the following assumptions on the { Z c t,h } t,h ≥ 0 hold.

A1. For each agent c , ( Z c k ) k ∈ N are i.i.d. random variables with values in ( Z , Z ) and distribution π c satisfying E π c [ A c ( Z c k )] = ¯ A c and E π c [ b ( Z c k )] = ¯ b c , and we define C A = sup c ∥ ¯ A c ∥ .

A2. For each c ∈ [ N ] , ( Z c k ) k ∈ N is a Markov chain with values in ( Z , Z ) , with Markov kernel P c . The kernel P c admits a unique invariant distribution π c , Z c 0 ∼ π c , and P c is uniformly geometrically ergodic, that is, there exist τ mix ( c ) ∈ N , such that for any k ∈ N ,

and for c ∈ [ N ] , we have E π c [ A c ( Z c 1 )] = ¯ A c and E π c [ b ( Z c 1 )] = ¯ b c , and we define

Moreover, each of the matrices -¯ A c is Hurwitz.

In A2, random matrices A c ( z ) and noise variables ε c ( z ) are almost surely bounded. This is necessary for working with the uniformly geometrically ergodic Markov kernels P c . For simplicity, we state most of our results using A1, which is classical in finite-time studies of LSA [46, 14]. Nonetheless, we show that our analysis of FedLSA can be extended to the Markovian setting under A2.

In a federated environment, agents can only communicate via a central server, which is generally costly. Hence, in FedLSA , agents' local updates are only aggregated after a given time. During the round t ≥ 0 , the agents start with a shared value θ t and perform H > 0 local updates, for h = 1 to H , given by the recurrence

with θ c t, 0 = θ t , and where we use the alias Z c t,h = Z Ht + h to simplify notations. Agents then send θ t,H to the server, that aggregates them as θ t = N -1 ∑ N c =1 θ c t -1 ,H and sends it back to all agents. We summarize this procedure in Algorithm 1. Our next assumption, which holds whenever ¯ A c is Hurwitz [19, 39, 15], ensures the stability of the local updates.

A3. There exist a > 0 , η ∞ > 0 , such that η ∞ a ≤ 1 / 2 , and for η ∈ (0; η ∞ ) , c ∈ [ N ] , u ∈ R d , it holds for Z c 0 ∼ π c , that E 1 / 2 [ ∥ (I -η A c ( Z c 0 )) u ∥ 2 ] ≤ (1 -ηa ) ∥ u ∥ .

3.2 Federated Temporal Difference Learning

A major application of FedLSA is federated TD learning with linear function approximation. Consider N Markov Decision Processes { ( S , A , P c MDP , r c , γ ) } c ∈ [ N ] with shared state space S , action space A , and discounting factor γ ∈ (0 , 1) . Each agent c ∈ [ N ] has its own transition kernel P c MDP , where P c MDP ( ·| s, a ) specifies the transition probability from state s upon taking action a for this specific agent, as well as its own reward function r c : S × A → [0 , 1] , that we assume to be deterministic for simplicity. Agents' heterogeneity lies in the different transition kernels and reward functions, that are specific to each agent .

In federated TD learning, all agents use the same shared policy π , and aim to construct a single shared function, that simultaneously approximates all value functions, defined as, for s ∈ S and c ∈ [ N ] ,

In the following, we aim to approximate V c,π ( s ) as a linear combination of features built using a mapping φ : S → R d . Formally, we look for θ ∈ R d such that the function V θ ( s ) = φ ⊤ ( s ) θ properly estimate the true value. For c ∈ [ N ] , we denote µ c the invariant distribution over S induced by the policy π and transition kernel P c MDP of agent c . Our goal is to find a parameter θ c ⋆ which is defined as a unique solution to the projected Bellman equation, see [49], which defines the best linear approximation of V c,π . This problem can be cast as a federated LSA problem [42, 50] by viewing the local optimum parameter θ c ⋆ as the solution of the system ¯ A c θ c ⋆ = ¯ b c , where

The global optimal parameter is then defined as the solution θ ⋆ of the averaged system ( 1 N ∑ N c =1 ¯ A c ) θ ⋆ = 1 N ∑ N c =1 ¯ b c . As it is the case for federated LSA, this parameter may give a better overall estimation of the value function. Indeed, the distribution µ c of some agents may be

strongly biased towards some states, whereas obtaining an estimation that is more balanced across all states may be more relevant.

In practice, when computing value functions, the tuples { ( S c k , A c k , S c k +1 ) } k ∈ N are sampled along one of the two following rules.

The generative model assumption TD 1 is common in TD learning [8, 30, 42, 45]. It is possible to generalize all our results to the more general Assumption TD 2, sampling over a single trajectory and leveraging the Markovian noise dynamics. This would have a similar impact on our results on TD(0) as it has on the ones we will present for general FedLSA in Section 4. In our analysis, we require the following assumption on the feature design matrix Σ c φ = E µ c [ φ ( S c 0 ) φ ( S c 0 ) ⊤ ] ∈ R d × d .

This assumption ensures the uniqueness of the optimal parameter θ c ⋆ . Under TD 1 and TD 3 we check the LSA assumptions A1 and A3, and the following holds.

TD 3 . Matrices Σ c φ are non-degenerate with the minimal eigenvalue ν = min c ∈ [ N ] λ min (Σ c φ ) > 0 . Moreover, the feature mapping φ ( · ) satisfies sup s ∈S ∥ φ ( s ) ∥ ≤ 1 .

Claim 3.1. Assume TD 1 and TD 3. Then the sequence of TD(0) updates satisfies A1 and A3 with

We prove this claim in Appendix E.1, and refer to [45, 42] for more details on the link between TD and linear stochastic approximation.

4 Refined Analysis of the FedLSA Algorithm

4.1 Stochastic expansion for FedLSA

We use the error expansion framework [1, 14] for LSA to analyze the MSE of the estimates θ t generated by Algorithm 1. For this purpose, we rewrite local update (4) as θ c t,h -θ c ⋆ = (I -η A ( Z c t,h ))( θ c t,h -1 -θ c ⋆ ) -ηε c ( Z c t,h ) , where ε c ( z ) is defined in (2). Running this recursion until the start of local training, we obtain

where ε c ( z ) is as in (3), and we recall that θ c t, 0 = θ t -1 , ∀ c ∈ [ N ] . We also introduced the notation with the convention Γ ( c,η ) t,m : n = I for m > n . Note that by A3, Γ ( c,η ) t,m : n is exponentially stable. That is, for any h ∈ N , we have E 1 / 2 [ ∥ Γ ( c,η ) t,m : m + h u ∥ 2 ] ≤ (1 -ηa ) h ∥ u ∥ . Using the fact θ c t, 0 = θ t -1 , and employing (1), we obtain that

where ¯ τ t,H = 1 N ∑ N c =1 { (I -η ¯ A c ) H -Γ ( c,η ) t, 1: H }{ θ c ⋆ -θ ⋆ } , ¯ φ t,H = 1 N ∑ N c =1 ∑ H h =1 Γ ( c,η ) t,h +1: H ε c ( Z c t,h ) are zero-mean fluctuation terms, and

is the deterministic heterogeneity bias accumulated in one round of local training. Note that ¯ ρ H vanishes when either (i) agents are homogeneous, or (ii) number of local updates is H = 1 . To analyze FedLSA , we run the recurrence (6) to obtain the decomposition

Here ˜ θ ( tr ) t = ∏ t s =1 ¯ Γ ( η ) s,H { θ 0 -θ ⋆ } is a transient term that vanishes geometrically, ˜ θ ( fl ) t is a zero-mean fluctuation term, with detailed expression provided in Appendix A, and the term ˜ θ ( bi , bi ) t is

and accounts for the bias of FedLSA due to local training, that vanishes whenever ¯ ρ H = 0 .

4.2 Convergence rate of FedLSA for i.i.d. observation model

First, we analyze the rate at which FedLSA converges to a biased solution θ ⋆ + ˜ θ ( bi , bi ) t . The following two quantities, which stem from the heterogeneity and stochasticity of the local estimators, play a central role in this rate

Here ¯ σ ε and ˜ v heter correspond to the different sources of noise in the error decomposition (7). The term ¯ σ ε is related to the variance of the local LSA iterate on each of the agents, while ˜ v heter controls the bias fluctuation term. In the centralized setting (i.e. if N = 1 ), the ˜ v heter term disappears, but not the ¯ σ ε term. We now proceed to analyze the MSE of the iterates of FedLSA :

Theorem 4.1. Assume A1 and A3. Then for any step size η ∈ (0 , η ∞ ) it holds that

where the bias ˜ θ ( bi , bi ) t converges in expectation to ˜ θ ( bi , bi ) ∞ = (I -¯ Γ ( η ) H ) -1 ¯ ρ H at a geometric rate, and is uniformly bounded by E 1 / 2 [ ∥ ˜ θ ( bi , bi ) t ∥ 2 ] ≲ ηH E c [ ∥ θ c ⋆ -θ ⋆ ∥ ] a .

The proof of Theorem 4.1 relies on bounding each term from (7). We provide a proof with explicit constants in Appendix A. Importantly, the fluctuation terms scale linearly with N . Moreover, in the centralized setting (that is, N = 1 ), the bias terms ¯ ρ H , ˜ θ ( bi , bi ) t and ˜ v heter vanish in Theorem 4.1, yielding the last-iterate bound

which is known to be sharp in its dependence on η for single-agent LSA (see Theorem 5 in [15]). Based on Claim 3.1, Theorem 4.1 translates for federated TD(0) as follows.

Corollary 4.2. Assume TD 1 and TD 3. Then for any step size η ∈ (0 , 1 -γ 4 ) , the iterates of federated TD(0) satisfy, with χ ( θ ⋆ , θ 1 ⋆ , . . . , θ N ⋆ ) = E c [ ∥ θ c ⋆ -θ ⋆ ∥ 2 ] ∨ (1 + E c [ ∥ θ c ⋆ ∥ 2 ]) ,

The right-hand side of Corollary 4.2 scales linearly with N , allowing for linear speed-up. This is in line with recent results on federated TD(0), which shows linear speed-up either without local training [7] or up to a possibly large bias term [50] (see analysis of their Theorem 2). While Corollary 4.2 shows the algorithm's convergence to some fixed, biased value, one can set the parameters of FedLSA such that this bias is small. This allows to rewrite the result of Theorem 4.1 in order to get a sample complexity bound in the following form.

Corollary 4.3. Assume A1 and A3. Let H > 1 , and 0 < ϵ < ( √ ˜ v heter ∨ ¯ σ ε E c [ ∥ θ c ⋆ -θ ⋆ ∥ ] ) 2 / 5 a ∨ E c [ ∥ θ c ⋆ -θ ⋆ ∥ ] a C A . Set the step size η = O ( aNϵ 2 ˜ v heter ∨ ¯ σ ε ∧ η ∞ ) and the number of local steps H = O ( ˜ v heter ∨ ¯ σ ε E c [ ∥ θ c ⋆ -θ ⋆ ∥ ] 1 Nϵ ) . Then, to achieve E [ ∥ θ T -θ ⋆ ∥ 2 ] < ϵ 2 the required number of communications for federated LSA is

In Corollary 4.3, the number of oracle calls scales as TH = O ( ˜ v heter ∨ ¯ σ ε Na 2 ϵ 2 log ∥ θ 0 -θ ⋆ ∥ ϵ ) , which shows that FedLSA has linear speed-up. Importantly, the number of communications T required to achieve precision ϵ 2 scales as ϵ -1 . In the next section, we will show how this dependence on ϵ -1 can be reduced from polynomial to logarithmic. Now we state the communication bound of federated TD(0). Corollary 4.4. Assume TD 1 and TD 3. Then for any 0 < ϵ < g 1 ( θ c ⋆ ,θ ⋆ ) (1 -γ ) ν with g 1 = O ((1 + ∥ θ ⋆ ∥ ) E c [ ∥ θ c ⋆ -θ ⋆ ∥ ]) . Set η = O ( (1 -γ ) νNϵ 2 E c [ ∥ θ c ⋆ ∥ 2 ]+1 ) and H = O ( E c [ ∥ θ c ⋆ ∥ 2 +1] Nϵ E c [ ∥ θ c ⋆ -θ ⋆ ∥ 2 ] ) . Then, to achieve E [ ∥ θ T -θ ⋆ ∥ 2 ] < ϵ 2 , the required number of communications for federated TD(0) is .

Corollary 4.4 is the first result to show that, even with local training and heterogeneous agents, federated TD(0) can converge to θ ⋆ with arbitrary precision. Importantly, this result preserves the linear speed-up effect , showing that federated learning indeed accelerates the training.

4.3 Convergence of FedLSA under Markovian observations model

The analysis of FedLSA can be generalized to the setting where observations { Z c k } k ∈ N form a Markov chain with kernel P c . To handle the Markovian nature of observations, we propose a variant of FedLSA that skips some observations (see the full procedure in Appendix B). This follows classical schemes for Markovian data in optimization [40], as adjusting the number of skipped observations (keeping about 1 observation out of τ mix ( c ) ) allows to control the correlation of successive observations. We may now state the counterpart of Corollary 4.3 for the Markovian setting.

Corollary 4.5 (Corollary 4.3 adjusted to the Markov samples) . Assume A2 and A3 and let 0 < ϵ < ( √ ˜ v heter ∨ ¯ σ ε E c [ ∥ θ c ⋆ -θ ⋆ ∥ ] ) 2 / 5 a ∨ E c [ ∥ θ c ⋆ -θ ⋆ ∥ ] a C A . Set the step size η = O ( aNϵ 2 ˜ v heter ∨ ¯ σ ε ∧ η ∞ ∧ η (M) ∞ ) , where we give the expression of η (M) ∞ is (37) . Then, for the iterates of Algorithm 3, in order to achieve E [ ∥ θ T -θ ⋆ ∥ 2 ] ≤ ϵ 2 , the required number of communication is

where the number of local updates H satisfies

The proof of Corollary 4.3 follows the idea outlined in [40], using Berbee's lemma [10]. We give all the details in Appendix B. This result is very similar to Corollary 4.3. Most crucially, it shows that the communication complexity is the same, regardless of the type of noise. The differences with Corollary 4.3 lie in (i) the number local updates H , that is scaled by τ mix (up to logarithmic factors), and (ii) the additional condition η ≤ η (M) ∞ , that allows verifying the stability of random matrix products with Markovian dependence (see Lemma B.2 in the appendix).

Remark 4.6 . Although, for clarity of exposition, we only state the counterpart of Corollary 4.3 in the Markovian result, all of our results can be extended to Markovian observations using the same ideas.

5 SCAFFLSA: Federated LSA with Bias Correction

5.1 Stochastic Controlled Averaging for Federated LSA

We now introduce the Stochastic Controlled Averaging for Federated LSA algorithm ( SCAFFLSA ), an improved version of FedLSA that mitigates client drift using control variates. This method is inspired by Scaffnew (see 37). In SCAFFLSA , each agent c ∈ [ N ] keeps a local variable ξ c t , that remains constant during each communication round t . Agents perform local updates on the current estimates of the parameters ˆ θ c t, 0 = θ t for c ∈ [ N ] , and for h ∈ [ H ] ,

At the end of the round, (i) the agents communicate the current estimate to the central server, (ii) the central server averages local iterates, and (iii) agents update their local control variates; see Algorithm 2. By defining the ideal control variates at the global solution, given by ξ c ⋆ = ¯ A c θ ⋆ -¯ b c = ¯ A c ( θ ⋆ -θ c ⋆ ) , we can rewrite the local update as

where ω c ( z ) is defined in (2). Under A1, it has finite covariance Σ c ω = ∫ Z ω c ( z ) ω c ( z ) ⊤ d π c ( z ) .

Similarly to the analysis of FedLSA , we use (8) to describe the sequence of aggregated iterates and control variates as, for t ≥ 0 and c ∈ [ N ] ,

where C c t +1 = ∑ H h =1 Γ ( c,η ) t,h +1: H and ¯ ω t +1 = 1 N ∑ N c =1 ∑ H h =1 Γ ( c,η ) t,h +1: H ω c ( Z c t,h ) . We now state the convergence rate, as well as sample and communication complexity of Algorithm 2.

Algorithm 2 SCAFFLSA : Stochastic Controlled FedLSA with deterministic communication

Input: η > 0 , θ 0 , ξ c 0 ∈ R d , T, N, H

for

t

= 1

for

c

to

= 1

T

do to

N

do

Set

θ

c

t,

0

for

ˆ

h

=

= 1

θ

t

to

H

do

Receive Z c t,h and perform local update ˆ θ c t,h = ˆ θ c t,h -1 -η ( A c ( Z c t,h ) ˆ θ c t,h -1 -b c ( Z c t,h ) -ξ c t ) Aggregate local iterates: θ t +1 = 1 N ∑ N c =1 ˆ θ c t,H Update local control variates: ξ c t +1 = ξ c t + 1 ηH ( θ t +1 -ˆ θ c t,H )

Theorem 5.1. Assume A1, A3. Let η, H > 0 such that η ≤ η ∞ , and H ≤ a / 240 η { C 2 A + ∥ Σ c ˜ A ∥ } . Set ξ c 0 = 0 for all c ∈ [ N ] . Then we have

Corollary 5.2. Let ϵ > 0 . Set the step size η = O (min( η ∞ , Naϵ 2 / ¯ σ ω )) and the number local updates to H = O ( max ( a η ∞ (C 2 A + ∥ Σ ˜ A ∥ ) , ∥ Σ ω ∥ Nϵ 2 (C 2 A + ∥ Σ ˜ A ∥ ) )) . Then, to achieve E [ ∥ θ T -θ ⋆ ∥ 2 ] ≤ ϵ 2 , the required number of communication for SCAFFLSA is

We provide detailed proof of these statements in Appendix C. They are based on a novel analysis, where we study virtual parameters ˇ θ c t,h , that follow the same update as (8), without the last term ηω c ( Z c t,h ) . After each round, virtual parameters are aggregated, and virtual control variate updated as

This allows to decompose

where ˇ θ t -θ ⋆ and ˇ ξ c t -ξ c ⋆ are transient terms, and ˜ θ t = θ t -ˇ θ t and ˜ ξ c t = ξ c t -ˇ ξ c t capture the fluctuations of the parameters and control variates.

We stress that our analysis shows that, in comparison with FedLSA , the SCAFFLSA algorithm reduces communication complexity while preserving the linear speed-up in the number of agents . This is in stark contrast with existing analyses of control-variate methods in heterogeneous federated learning, that either have large communication cost, or lose the linear speed-up [24, 37, 21]. To obtain this result, we conduct a very careful analysis of the propagation of variances and covariances of ˜ θ t and ˜ ξ c t between successive communication rounds. We describe this in full detail in Appendix C.2.

In Corollary 5.2, we show that the total number of communications depends only logarithmically on the precision ϵ . This is in stark contrast with Algorithm 1, where the necessity of controlling the bias' magnitude prevents from scaling H with 1 / ϵ 2 . Additionally, this shows that the number of required local updates reduces as the number of agents grows. Thus, in the high precision regime (i.e.small ϵ and η ), using control variates reduces communication complexity compared to FedLSA .

5.2 Application to Federated TD(0)

Applying SCAFFLSA to TD learning, we obtain SCAFFTD (0) (see Algorithm 5 in Appendix E). The analysis of SCAFFLSA directly translates to SCAFFTD (0), resulting in the following communication complexity bound.

Corollary 5.3. Assume TD 1 and TD 3 and let 0 < ϵ ≤ √ 8 E c [1 + ∥ θ c ⋆ ∥ 2 ] / ((1 -γ ) ν ) . Set the step size η = O ( (1 -γ ) νNϵ 2 ∥ θ ⋆ ∥ 2 +1 ) and the number local updates to H = O ( ∥ θ ⋆ ∥ 2 +1 Nϵ 2 ) . Then, to achieve E [ ∥ θ T -θ ⋆ ∥ 2 ] ≤ ϵ 2 , the required number of communication for SCAFFTD (0) is

0

10

20

30

40

50

Communications (

×

1000)

10

-

3

10

-

1

10

1

(a) Heterogeneous,

N

= 10

, H

= 10

0

10

20

30

40

50

Communications (

×

10)

10

-

3

10

-

1

10

1

(b) Heterogeneous,

N

= 10

, H

= 1000

0

10

20

30

40

50

Communications (

×

1000)

10

-

3

10

-

1

10

1

(c) Heterogeneous,

N

= 100

, H

= 10

0

10

20

30

40

50

Communications (

×

10)

10

-

3

10

-

1

10

1

FedLSA

SCAFFLSA

FedLSA's bias

(d) Heterogeneous,

N

= 100

, H

= 1000

0

10

20

30

40

50

Communications (

×

1000)

10

-

3

10

-

1

10

1

(e) Homogeneous,

N

= 10

, H

= 10

0

10

20

30

40

50

Communications (

×

10)

10

-

3

10

-

1

10

1

(f) Homogeneous,

N

= 10

, H

= 1000

0

10

20

30

40

50

Communications (

×

1000)

10

-

3

10

-

1

10

1

(g) Homogeneous,

N

= 100

, H

= 10

0

10

20

30

40

50

Communications (

×

10)

10

-

3

10

-

1

10

1

FedLSA

SCAFFLSA

FedLSA's bias

(h) Homogeneous,

N

= 100

, H

= 1000

Corollary 5.3 confirms that, when applied to TD(0), SCAFFLSA 's communication complexity depends only logarithmically on heterogeneity and on the desired precision. In contrast with existing methods for federated TD(0) [11, 22, 50], it converges even with many local steps, whose number diminishes linearly with the number of agents N , producing the linear speed-up effect.

Remark 5.4 . In Appendix F, we extend the analysis of Scaffnew [37] to the LSA setting. Their analysis does not exploit the fact that agents' estimators are not correlated, and thus lose the linear speed-up. In contrast, our novel analysis technique carefully tracks correlations between parameters and control variates throughout the run of the algorithm.

6 Numerical Experiments

In this section, we demonstrate the performance of FedLSA and SCAFFLSA under varying levels of heterogeneity. We consider the Garnet problem [2, 16], with n = 30 states embedded in d = 8 dimensions, a = 2 actions, and each state is linked to b = 2 others in the transition kernel. We aim to estimate the value function of the policy which chooses actions uniformly at random, in homogeneous and heterogeneous setups. In all experiments, we initialize the algorithms in a neighborhood of the solution, allowing to observe both transient and stationary regimes. We provide all details regarding the experimental setup in Appendix G. Our code is available either as supplementary material or online on GitHub: https://github.com/pmangold/scafflsa .

SCAFFLSA properly handles heterogeneity. This heterogeneous scenario is composed of two different Garnet environments, that are each held by half of the agents, with small perturbations. Such a setting may arise in cases where each agent's environment reflects only a part of the world. For instance, if half of the individuals live in the city, while the other half live in the countryside: both have different observations, but learning a shared value function gives a better representation of the overall reality. In Figures 1(a) to 1(d), we plot the MSE with N ∈ { 10 , 100 } , H ∈ { 10 , 1000 } and η = 0 . 1 , with the same total number of updates TH = 500 , 000 . As predicted by our theory, FedLSA stalls when the number of local updates increases, and its bias (green dashed line in Figures 1(a) to 1(d)) is in line with the value predicted by our theory (see Theorem 4.1). For completeness, we plot the error of FedLSA in estimating θ ⋆ + ˜ θ ( bi , bi ) ∞ in Appendix G. On the opposite, SCAFFLSA 's bias-correction mechanism allows to eliminate all bias, improving the MSE until noise dominates.

η

= 0

.

001

η

= 0

.

01

η

= 0

.

1

η

= 1

FedLSA

SCAFFLSA

1

10

1

10

2

10

3

Number of agents

10

-

6

10

-

2

(a) Homogeneous,

H

= 1

1

10

1

10

2

10

3

Number of agents

10

-

5

10

-

1

(b) Heterogeneous,

H

= 1

1

10

1

10

2

10

3

Number of agents

10

-

6

10

-

2

(c) Homogeneous,

H

= 100

1

10

1

10

2

10

3

Number of agents

10

-

5

10

-

1

(d) Heterogeneous,

H

= 100

Both algorithms behave alike in homogeneous settings. In the homogeneous setting, we create one instance of a Garnet environment. Then, each agent receives a slightly perturbed variant of this environment. This illustrates a situation where all agents solve the same exact problem, but may have small divergences in their measures of states and rewards. We plot the MSE in Figures 1(e) to 1(h) with N ∈ { 10 , 100 } agents, η = 0 . 1 , and H ∈ { 10 , 1000 } , with the same total number of updates TH = 500 , 000 . In this case, as predicted in Corollary 4.3, the number of local steps H has little influence on the final MSE. Since agents are homogeneous, control variates have virtually no effect, and SCAFFLSA is on par with FedLSA . The MSE is dominated by the noise term, which diminishes with the step size (see additional experiments in Appendix G with smaller η = 0 . 01 ).

Both algorithms enjoy linear speed-up! In Figure 2, we plot the MSE obtained once algorithms reach the stationary regime, as a function of the number of agents N = 1 to 1000 , for step sizes η ∈ { 0 . 001 , 0 . 01 , 0 . 1 , 1 } and H ∈ { 1 , 100 } , in both homogeneous and heterogeneous settings. Whenever (i) agents are homogeneous, or (ii) the number of local steps is small, both FedLSA and SCAFFLSA can achieve similar precision with a step size that increases with the number of agents. This allows to use larger step sizes, so as to reach a given precision level faster, resulting in the so-called linear speed-up . However, when agents are heterogeneous and the number of local updates increases, FedLSA loses the speed-up due to large bias. Remarkably, and as explained by our theory (see Corollary 5.2), SCAFFLSA maintains this speed-up even in heterogeneous settings.

7 Conclusion

In this paper, we studied the role of heterogeneity in federated linear stochastic approximation. We proposed a new analysis of FedLSA , where we formally characterize FedLSA 's bias. This allows to show that, with proper hyperparameter setting, FedLSA (i) can converge to arbitrary precision even with local training, and (ii) enjoys linear speed-up in the number of agents. We then proposed a novel algorithm, SCAFFLSA , that uses control variates to allow for extended local training. We analyzed this method using on a novel analysis technique, and formally proved that control variates reduce communication complexity of the algorithm. Importantly, our analysis shows that SCAFFLSA preserves the linear speed-up, which is the first time that a federated algorithm provably accelerates while preserving this linear speed-up. Finally, we instantiated our results for federated TD learning, and conducted an empirical study that demonstrates the soundness of our theory in this setting.

References

Rafik Aguech, Eric Moulines, and Pierre Priouret. On a perturbation approach for the analysis of stochastic tracking algorithms. SIAM Journal on Control and Optimization , 39(3):872-899, 2000.

TW Archibald, KIM McKinnon, and LC Thomas. On the generation of markov decision processes. Journal of the Operational Research Society , 46(3):354-361, 1995.

H.C.P. Berbee. Random Walks with Stationary Increments and Renewal Theory . Mathematical Centre tracts. Centrum Voor Wiskunde en Informatica, 1979.

J. Bhandari, D. Russo, and R. Singal. A finite time analysis of temporal difference learning with linear function approximation. In Conference On Learning Theory , pages 1691-1692, 2018.

Laurent Condat, Ivan Agarsky, and Peter Richtárik. Provably doubly accelerated federated learning: The first theoretically successful combination of local training and compressed communication. arXiv preprint arXiv:2210.13277 , 2022.

Laurent Condat and Peter Richtárik. Randprox: Primal-dual optimization algorithms with randomized proximal updates. arXiv preprint arXiv:2207.12891 , 2022.

Nicolò Dal Fabbro, Aritra Mitra, and George J Pappas. Federated td learning over finite-rate erasure channels: Linear speedup under markovian sampling. IEEE Control Systems Letters , 2023.

G. Dalal, Balázs Szörényi, G. Thoppe, and S. Mannor. Finite sample analyses for TD(0) with function approximation. In Thirty-Second AAAI Conference on Artificial Intelligence , 2018.

Christoph Dann, Gerhard Neumann, Jan Peters, et al. Policy evaluation with temporal differences: A survey and comparison. Journal of Machine Learning Research , 15:809-883, 2014.

Jérôme Dedecker and Sana Louhichi. Maximal inequalities and empirical central limit theorems. In Empirical process techniques for dependent data , pages 137-159. Springer, 2002.

Thinh Doan, Siva Maguluri, and Justin Romberg. Finite-Time Analysis of Distributed TD(0) with Linear Function Approximation on Multi-Agent Reinforcement Learning. In Proceedings of the 36th International Conference on Machine Learning , pages 1626-1635. PMLR, May 2019. ISSN: 2640-3498.

Thinh T Doan. Local stochastic approximation: A unified view of federated learning and distributed multi-task reinforcement learning algorithms. arXiv preprint arXiv:2006.13460 , 2020.

R. Douc, E. Moulines, P. Priouret, and P. Soulier. Markov chains . Springer Series in Operations Research and Financial Engineering. Springer, 2018.

Alain Durmus, Eric Moulines, Alexey Naumov, and Sergey Samsonov. Finite-time highprobability bounds for Polyak-Ruppert averaged iterates of linear stochastic approximation. Mathematics of Operations Research , 2024.

Alain Durmus, Eric Moulines, Alexey Naumov, Sergey Samsonov, Kevin Scaman, and Hoi-To Wai. Tight high probability bounds for linear stochastic approximation with fixed stepsize. In M. Ranzato, A. Beygelzimer, K. Nguyen, P. S. Liang, J. W. Vaughan, and Y. Dauphin, editors, Advances in Neural Information Processing Systems , volume 34, pages 30063-30074. Curran Associates, Inc., 2021.

Matthieu Geist, Bruno Scherrer, et al. Off-policy learning with eligibility traces: a survey. J. Mach. Learn. Res. , 15(1):289-333, 2014.

Eduard Gorbunov, Filip Hanzely, and Peter Richtárik. Local SGD: Unified theory and new efficient methods. In International Conference on Artificial Intelligence and Statistics , pages 3556-3564. PMLR, 2021.

Michał Grudzie´ n, Grigory Malinovsky, and Peter Richtárik. Can 5th generation local training methods support client sampling? Yes! In International Conference on Artificial Intelligence and Statistics , pages 1055-1092. PMLR, 2023.

L. Guo and L. Ljung. Exponential stability of general tracking algorithms. IEEE Transactions on Automatic Control , 40(8):1376-1387, 1995.

Farzin Haddadpour and Mehrdad Mahdavi. On the Convergence of Local Descent Methods in Federated Learning, December 2019. arXiv:1910.14425 [cs, stat].

Zhengmian Hu and Heng Huang. Tighter analysis for proxskip. In International Conference on Machine Learning , pages 13469-13496. PMLR, 2023.

Hao Jin, Yang Peng, Wenhao Yang, Shusen Wang, and Zhihua Zhang. Federated Reinforcement Learning with Environment Heterogeneity. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , pages 18-37. PMLR, May 2022. ISSN: 2640-3498.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning , pages 5132-5143. PMLR, 2020.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning, 2021.

Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. Tighter theory for local sgd on identical and heterogeneous data. In International Conference on Artificial Intelligence and Statistics , pages 4519-4529. PMLR, 2020.

Sajad Khodadadian, Pranay Sharma, Gauri Joshi, and Siva Theja Maguluri. Federated reinforcement learning: Linear speedup under markovian sampling. In International Conference on Machine Learning , pages 10997-11057. PMLR, 2022.

Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning , pages 5381-5393. PMLR, 2020.

Jakub Koneˇ cn` y, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527 , 2016.

PS Kostenetskiy, RA Chulkevich, and VI Kozyrev. Hpc resources of the higher school of economics. In Journal of Physics: Conference Series , volume 1740, page 012050. IOP Publishing, 2021.

Gen Li, Weichen Wu, Yuejie Chi, Cong Ma, Alessandro Rinaldo, and Yuting Wei. Highprobability sample complexities for policy evaluation with linear function approximation. IEEE Transactions on Information Theory , 70(8):5969-5999, 2024.

Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine , 37(3):50-60, 2020.

Tianjiao Li, Guanghui Lan, and Ashwin Pananjady. Accelerated and instance-optimal policy evaluation with linear function approximation. SIAM Journal on Mathematics of Data Science , 5(1):174-200, 2023.

Hyun-Kyo Lim, Ju-Bong Kim, Joo-Seong Heo, and Youn-Hee Han. Federated reinforcement learning for training control policies on multiple iot devices. Sensors , 20(5):1359, 2020.

Rui Liu and Alex Olshevsky. Distributed TD(0) with almost no communication. IEEE Control Systems Letters , 7:2892-2897, 2023.

Grigory Malinovsky, Kai Yi, and Peter Richtárik. Variance reduced proxskip: Algorithm, theory and application to federated learning. Advances in Neural Information Processing Systems , 35:15176-15189, 2022.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich, and Peter Richtárik. Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally! In International Conference on Machine Learning , pages 15750-15769. PMLR, 2022.

Aritra Mitra, Rayana Jaafar, George J Pappas, and Hamed Hassani. Linear convergence in federated learning: Tackling client heterogeneity and sparse gradients. Advances in Neural Information Processing Systems , 34:14606-14619, 2021.

Wenlong Mou, Chris Junchi Li, Martin J Wainwright, Peter L Bartlett, and Michael I Jordan. On linear stochastic approximation: Fine-grained Polyak-Ruppert and non-asymptotic concentration. In Conference on Learning Theory , pages 2947-2997. PMLR, 2020.

Dheeraj Nagaraj, Xian Wu, Guy Bresler, Prateek Jain, and Praneeth Netrapalli. Least squares regression with markovian data: Fundamental limits and algorithms. Advances in neural information processing systems , 33:16666-16676, 2020.

Kumar Kshitij Patel, Margalit Glasgow, Lingxiao Wang, Nirmit Joshi, and Nathan Srebro. On the still unreasonable effectiveness of federated averaging for heterogeneous distributed learning. In Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and Opportunities , 2023.

Gandharv Patil, LA Prashanth, Dheeraj Nagaraj, and Doina Precup. Finite time analysis of temporal difference learning with linear function approximation: Tail averaging and regularisation. In International Conference on Artificial Intelligence and Statistics , pages 5438-5448. PMLR, 2023.

Jiaju Qi, Qihao Zhou, Lei Lei, and Kan Zheng. Federated reinforcement learning: Techniques, applications, and open challenges. arXiv preprint arXiv:2108.11887 , 2021.

Zhaonan Qu, Kaixiang Lin, Zhaojian Li, and Jiayu Zhou. Federated learning's blessing: Fedavg has linear speedup. In ICLR 2021-Workshop on Distributed and Private Machine Learning (DPML) , 2021.

Sergey Samsonov, Daniil Tiapkin, Alexey Naumov, and Eric Moulines. Improved HighProbability Bounds for the Temporal Difference Learning Algorithm via Exponential Stability. In Shipra Agrawal and Aaron Roth, editors, Proceedings of Thirty Seventh Conference on Learning Theory , volume 247 of Proceedings of Machine Learning Research , pages 4511-4547. PMLR, 30 Jun-03 Jul 2024.

Rayadurgam Srikant and Lei Ying. Finite-time error bounds for linear stochastic approximation and TD learning. In Conference on Learning Theory , pages 2803-2830. PMLR, 2019.

Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning , 3:9-44, 1988.

Richard S Sutton, Hamid Reza Maei, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesvári, and Eric Wiewiora. Fast gradient-descent methods for temporal-difference learning with linear function approximation. In Proceedings of the 26th annual international conference on machine learning , pages 993-1000, 2009.

J. N. Tsitsiklis and B. Van Roy. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control , 42(5):674-690, May 1997.

Han Wang, Aritra Mitra, Hamed Hassani, George J Pappas, and James Anderson. Federated temporal difference learning with linear function approximation under environmental heterogeneity. arXiv preprint arXiv:2302.02212 , 2023.

Jianyu Wang, Rudrajit Das, Gauri Joshi, Satyen Kale, Zheng Xu, and Tong Zhang. On the unreasonable effectiveness of federated averaging with heterogeneous data. arXiv preprint arXiv:2206.04723 , 2022.

Zhijie Xie and Shenghui Song. FedKL: Tackling data heterogeneity in federated reinforcement learning by penalizing KL divergence. IEEE Journal on Selected Areas in Communications , 41(4):1227-1242, 2023.

Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582 , 2018.

A Analysis of Federated Linear Stochastic Approximation

For the analysis we need to define two filtration: F + s,h := σ ( Z c t,k , t ≥ s, k ≥ h, 1 ≤ c ≤ N ) , corresponding to the future events, and F -s,h := σ ( Z c t,k , t ≤ s, k ≤ h, 1 ≤ c ≤ N ) , corresponding to the preceding events. Recall that the local LSA updates are written as

Performing H local steps and taking average, we end up with the decomposition

where we have defined

The transient term ¯ Γ ( η ) t,H ( θ t -1 -θ ⋆ ) , responsible for the rate of forgetting the previous iteration error θ t -1 -θ ⋆ , and the fluctuation term η ¯ φ t,H , reflecting the oscillations of the iterates around θ ⋆ , are similar to the ones from the standard LSA error decomposition [14]. The two additional terms in (10) reflect the heterogeneity bias . This bias is composed of two parts: the true bias ¯ ρ H , which is non-random, and its fluctuations ¯ τ t,H . To analyze the complexity and communication complexity of FedLSA , we run the recurrence (6) to obtain

with the notations ¯ Γ ( η ) H = E [ ¯ Γ ( η ) s,H ] = 1 N ∑ N c =1 (I -η ¯ A c ) H and ∆ ( η ) H,s,t = {∏ t i = s +1 ¯ Γ ( η ) i,H } -( ¯ Γ ( η ) H ) t -s . The first term, ˜ θ ( tr ) t gives the rate at which the initial error is forgotten. The terms ˜ θ ( bi , bi ) t and ˜ θ ( fl , bi ) t represent the bias and fluctuation due to statistical heterogeneity across agents. Note that in the special case where agents are homogeneous (i.e. ¯ A c = ¯ A for all c ∈ [ N ] ), these two terms vanish. Finally, the term ˜ θ ( fl ) t depicts the fluctuations of θ t around the solution θ ⋆ . Now we need to upper bound each of the terms in decomposition (12). This is done in a sequence of lemmas below: ˜ θ ( fl ) t is bounded in Lemma A.1, ˜ θ ( fl , bi ) t in Lemma A.2, ˜ θ ( tr ) t in Lemma A.4, and ˜ θ ( bi , bi ) t in Lemma A.5. Then we combine the bounds in order to state a version of Theorem 4.1 with explicit constants in Theorem A.6.

Lemma A.1. Assume A1 and A3. Then, for any step size η ∈ (0 , η ∞ ) it holds

where we have defined

Proof. We start from the decomposition (13). With the definition of ˜ θ ( fl ) t and E F + s +1 , [ {∏ t i = s +1 ¯ Γ ( η ) i,H } ¯ φ s,H ] = 0 , we obtain that

Now, using the assumption A3 and Minkowski's inequality, we obtain that

In (a) applied A3 conditionally on F -t -1 ,H . Hence, by induction we get from the previous formulas that

Now we proceed with bounding E [ ∥ ¯ φ s,H ∥ 2 ] . Indeed, since the clients are independent, we get using (11) that

Therefore, using (3) and the following inequality,

we get

Plugging this inequality in (15), we get

where we used additionally which is valid for x ∈ [0; 1 / 2] . Now it remains to notice that

for any x > 0 .

We proceed with analyzing the fluctuation of the true bias component of the error θ t defined in (13). The first step towards this is to obtain the respective bound for ¯ τ s,H , s ∈ { 1 , . . . , T } , where ¯ τ s,H is defined in (11). Now we provide an upper bound for ˜ θ ( fl , bi ) t :

Lemma A.2. Assume A1 and A3. Then, for any step size η ∈ (0 , η ∞ ) it holds

Proof. Recall that ˜ θ ( fl , bi ) t is given (see (13)) by

where ¯ τ s,H and ¯ ρ H are defined in (11). We begin with bounding T 1 . In order to do it we first need to bound ¯ τ s,H . Since the different agents are independent, we have

Applying Lemma D.1 and the fact that { (I -η ¯ A c ) h -1 ˜ A c ( Z c s,h )Γ ( c,η ) s, ( h +1): H ( θ c ⋆ -θ ⋆ ) } H h =1 is a martingale-difference w.r.t. F -s,h , we get that

Using the tower property conditionally on F + s,h +1 , we get

where Σ c ˜ A is the noise covariance matrix defined in (3). Since for any vector u ∈ R d we have ∥ u ∥ Σ c ˜ A ≤ ∥ Σ c ˜ A ∥ 1 / 2 ∥ u ∥ , we get

Combining the above bounds in (18) yields that

Thus, proceeding as in (14) together with (20), we get

In the bound above we used (16) together with the bound

Now we bound the second part of ˜ θ ( fl , bi ) t in (17), that is, T 2 . To begin with, we start with applying Lemma D.1 and we get for any s ∈ { 1 , . . . , t } and i ∈ { s +1 , . . . , t } , that

Note that,

Proceeding as in (19), we get using independence between agents for any u ∈ R d ,

Hence, using (21), we get

Combining the above estimates in (17), and using Minkowski's inequality, we get

where we used that ηa ≤ 1 / 2 and and the statement follows.

Lemma A.3. Recall that ¯ ρ H = 1 N ∑ N c =1 (I -(I -η ¯ A c ) H ) { θ c ⋆ -θ ⋆ } , it satisfies

Proof. Using the identity,

and the inequality ( H k +2 ) ≤ ( H -2 k ) H 2 , we get that

Using (23) with u = η ¯ A c for all c , we get

by definition of θ c ⋆ and θ ⋆ , we have that ∑ N c =1 ¯ A c ( θ c ⋆ -θ ⋆ ) = ∑ N c =1 ¯ A c θ c ⋆ -( ∑ N c =1 ¯ A c ) θ ⋆ = ∑ N c =1 ¯ b c -∑ N c =1 ¯ b c = 0 . Using this and (24), we finally get (22).

Lemma A.4. Assume A1 and A3. Then for any step size η ∈ (0 , η ∞ ) we have

Proof. Proceeding as in (14) for any u ∈ R d we have

Using this result for u = θ 0 -θ ⋆ we get the statement.

Lemma A.5. Assume A1 and A3. Then for any η ∈ (0 , η ∞ ) we have

Proof. Using A3 and Minkowski's inequalitty, we get

and the statement follows.

Theorem A.6. Assume A1 and A3. Then for any step size η ∈ (0 , η ∞ ) it holds that

where the bias ˜ θ ( bi , bi ) t converges to (I -¯ Γ ( η ) H ) -1 ¯ ρ H at a rate

In the lemma below we provide a simplified sample complexity bound of Corollary 4.3 corresponding to the synchronous setting, that is, with number of local training steps H = 1 . There, the bias term disappears, and above results directly give a simplified sample complexity bound.

Corollary A.7. Assume A1 and A3. Let H = 1 , then for any 0 < ϵ < 1 , in order to achieve E [ ∥ θ T -θ ⋆ ∥ 2 ] ≤ ϵ 2 the required number of communications is

number of communications, setting the step size

Proof. Bounding the first two terms in decomposition (25) we get that the step size should satisfy

From the last term we have

in order to achieve E [ ∥ θ T -θ ⋆ ∥ 2 ] < ϵ 2 the required number of communications is

setting the step size and number of local iterations

Proof. We aim to bound separately all the terms in the r.h.s. of Theorem 4.1. Note that it requires to set η ∈ (0; η 0 ) with η 0 given in (26) in order to fulfill the bounds

Now, we should bound the bias term

Thus, using the Neuman series, we can bound the norm of the term above as

Hence, using the bound of Lemma A.3, we get

where we used the fact that the step size η is chosen in order to satisfy ηH C A ≤ 1 . Thus in order to fulfill E 1 / 2 [ ∥ ˜ θ ( bi , bi ) t ∥ 2 ] ≲ ε we need to choose η and H such that

It remains to bound the term √ E c ∥ Σ c ˜ A ∥∥ ¯ ρ H ∥ aH 1 / 2 N 1 / 2 . Using the bound of Lemma A.3, we get

Hence, it remains to combine the bounds above in order to get the sample complexity result (27).

Corollary A.9. Assume TD 1 and TD 3. Then for any

in order to achieve E [ ∥ θ T -θ ⋆ ∥ 2 ] < ϵ 2 the required number of communications for federated TD(0) algorithm is

B Markovian sampling schemes for FedLSA

Note that under A2 each of the matrices ¯ A c , c ∈ [ N ] is Hurwitz. This guarantees the existence and uniqueness of a positive definite matrix Q c which is a solution of the Lyapunov equation

We further introduce the associated quantities, that will be used throughout the proof.

In our statement of A2 we also required that each of the chains ( Z c k ) k ∈ N starts from its invariant distribution π c . This requirement can be removed, and extension to the setting of arbitrary initial distribution can be done based on the maximal exact coupling argument [13, Lemma 19.3.6 and Theorem 19 3.9]. However, to better highlight the main ingredients of the proof, we prefer to keep stationary assumption.

Proof of Corollary 4.5. Assume that the total number of local iterations, that is, TH , satisfies

where q ∈ N is a parameter that will be determined later. With Lemma B.4 we construct for each c ∈ [ N ] a sequence of random variables { ˜ Z ⋆,c 2 jq } j =1 ,...,m , which are i.i.d. with the same distribution π c . Moreover, Lemma B.4 together with union bound imply

̸

The bound (29) implies that m ≤ TH/ (2 q ) . Thus, for any δ ∈ (0 , 1) , in order to guarantee that

̸

Algorithm 3 FedLSA with Markovian data

Input: η > 0 , θ 0 ∈ R d , T, N, H > 0 , time window q ∈ N .

for

t

= 0

to do

Initialize for

c

= 1

for

h

T

θ

t,

0

to

= 1

-

1

=

N

θ

t

do to

H

do

Receive Z c t,h , then check the condition:

if

h

else

Skip current update

it is enough to ensure that

= qj, j ∈ N then Compute local update

Inequality (30) holds for fixed δ ∈ (0 , 1) , if we choose

Thus, setting the block size q as in (31), we get that for total number iterations TH satisfying (29), with probability at least 1 -δ the results of Algorithm 3 are indistinguishable from the result of its counterpart Algorithm 1 applied with number of local steps H/q . We will denote the iterates of the latter algorithm applied with number of local steps h ∈ N as θ ( ind ) ,h T in order to make explicit the dependence of global parameter upon the number of local iterates. We further denote the event, where θ ( ind ) ,H/q T = θ T , by A δ . Thus, setting

similarly to the way the number of local updates is set in Corollary 4.3, we obtain that

where in the last inequality we relied on the special choice of H/q from (32) together with Holder's inequality. Now it remains to bound E [ ∥ θ T -θ ⋆ ∥ 4 ] and tune the parameter δ appropriately. Note that within this bound we can not rely on the estimates based on independent observations { ˜ Z ⋆,c 2 jq } j =1 ,...,m . At the same time, note that the skeleton Z c 2 jq , j ≥ 0 for any c ∈ [ N ] is a Markov chain with the Markov kernel P q c and mixing time τ mix = 1 . This allows us to write a simple upper bound on E [ ∥ θ T -θ ⋆ ∥ 4 ] based on the stability result for product of random matrices provided in [14]. Indeed, applying the result of Lemma B.1, we get

and the corresponding bound (33) can be rewritten as

Thus, setting

we obtain that the corresponding bound for block size q scales as

where we write ≲ for inequality up to an absolute constant and set

Combination of the above bounds yields that and the proof is completed.

Lemma B.1. Assume A2 and A3. Then, for the iterates θ t of Algorithm 3 run with parameters η, H, q satisfying the relation

it holds for any probability distribution ξ on ( Z , Z ) and any t ∈ N , that

Proof. First we write a counterpart of the error decomposition (12) - (13) for the LSA error of the subsampled iterates of Algorithm 3. Namely, we write that

where we have defined

For notation simplicity we have removed the dependence of κ t,H on the subsampling parameter q ∈ N . Thus, applying the result of [14, Proposition 7] (see also Lemma B.2) together with Minkowski's inequality, we obtain from the previous bound that for any distribution ξ on ( Z , Z ) , provided that the ratio ηH/q satisfies the relation (34). This bound yields that

Hence, we obtain by running the recurrence (35), that

and the statement follows.

Stability results on product of random matrices. The results of this paragraph provides the stability bound for the product of random matrices Γ ( c,η,q ) t,m : n defined in (36). Define the quantities

Then the following result holds:

Lemma B.2 (Proposition 7 from [14], simplified) . Assume A2 and A3. Then, for any c ∈ [ N ] , t ∈ N , step size η ∈ ( 0 , η (M) ∞ ] , any n ∈ N , q ≥ τ mix , and probability distribution ξ on ( Z , Z ) , it holds

Proof. It is enough to note that, since q ≥ τ mix , and we consider q -skeleton of each Markov kernels P c , each of the subsampled kernels P q c will have a mixing time 1 .

Berbee's lemma construction. We outline some preliminaries associated with the Berbee's coupling lemma [3] construction. We recall first a definition of the β -mixing coefficient. Consider a probability space (Ω , F , P ) equipped with σ -fields F and G such that F ⊆ F , G ⊆ F . Then the β -mixing coefficient of F and G is defined as

and the supremum is taken over all pairs of partitions { A i } i ∈ I ∈ F I and { B j } j ∈ J ∈ G J of ˜ Z N with finite I and J .

Now let ( Z , d Z ) be a Polish space endowed with its Borel σ -field, denoted by Z , and let ( Z N , Z ⊗ N ) be the corresponding canonical space. Consider a Markov kernel P on Z ×Z and denote by P ξ and E ξ the corresponding probability distribution and expectation with initial distribution ξ . Without loss of generality, we assume that ( Z k ) k ∈ N is the associated canonical process. By construction, for any A ∈ Z , P ξ ( Z k ∈ A | Z k -1 ) = P( Z k -1 , A ) , P ξ -a.s. In the case ξ = δ z , z ∈ Z , P ξ and E ξ are denoted by P z and E z , respectively. We now make an assumption about the mixing properties of P : UGE 1 . The Markov kernel P admits π as an invariant distribution and is uniformly geometrically ergodic, that is, there exists τ mix ∈ N such that for all k ∈ N ,

For q ∈ N , k ∈ N , and the Markov chain { Z n } n ∈ N satisfying the uniform geometric ergodicity constraint UGE 1, we define the σ -algebras F k = σ ( Z ℓ , ℓ ≤ k ) and F + k + q = σ ( Z ℓ , ℓ ≥ k + q ) . In such a scenario, using [13, Theorem 3.3], the respective β -mixing coefficient of F k and F + k + q is bounded by

We rely on the following useful version of Berbee's coupling lemma [3], which is due to [10, Lemma 4 . 1 ]:

Theorem B.3 (Lemma 4 . 1 in [10]) . Let X and Y be two random variables taking their values in Borel spaces X and Y , respectively, and let U be a random variable with uniform distribution on [0; 1] that is independent of ( X,Y ) . There exists a random variable Y ⋆ = f ( X,Y,U ) where f is a measurable function from X × Y × [0 , 1] to Y , such that:

Y ⋆ is independent of X and has the same distribution as Y ;

P ( Y ⋆ = Y ) = β ( σ ( X ) , σ ( Y )) .

̸

Let us now consider the extended measurable space ˜ Z N = Z N × [0 , 1] , equipped with the σ -field ˜ Z N = Z ⊗ N ⊗ B ([0 , 1]) . For each probability measure ξ on ( Z , Z ) , we consider the probability measure ˜ P ξ = P ξ ⊗ Unif ([0 , 1]) and denote by ˜ E ξ the corresponding expected value. Finally, we denote by ( ˜ Z k ) k ∈ N the canonical process ˜ Z k : (( z i ) i ∈ N , u ) ∈ ˜ Z N ↦→ z k and U : (( z i ) i ∈ N , u ) ∈ ˜ Z N ↦→ u . Under ˜ P ξ , { ˜ Z k } k ∈ N is by construction a Markov chain with initial distribution ξ and Markov kernel P independent of U . Moreover, the distribution of U under ˜ P ξ is uniform over [0 , 1] . Using the above construction, we obtain a useful blocking lemma, which is also stated in [10].

Lemma B.4. Assume UGE 1, let q ∈ N and ξ be a probability measure on ( Z , Z ) . Then, there exists a random process ( ˜ Z ⋆ k ) k ∈ N defined on ( ˜ Z N , ˜ Z N , ˜ P ξ ) such that for any k ∈ N , it holds:

For any i , vector V ⋆ i = ( ˜ Z ⋆ iq +1 , . . . , ˜ Z ⋆ iq + q ) has the same distribution as V i = ( Z iq +1 , . . . , Z iq + q ) under ˜ P ξ ;

The sequences ( V ⋆ 2 i ) i ≥ 0 and ( V ⋆ 2 i +1 ) i ≥ 0 are i.i.d. ;

For any i , ˜ P ξ ( V i = V ⋆ i ) ≤ β ( q ) ;

̸

Proof. The proof follows from Theorem B.3 and the relations between UGE 1 and β -mixing coefficient, see e.g. [13, Theorem 3.3].

C Federated Linear Stochastic Approximation with Control Variates

C.1 Technical Lemmas

Lemma C.1. Assume A1 and A3. Recall C ( t,c ) η,H = ∑ H h =1 Γ ( c,η ) t,h +1: H Then it holds that

Proof. We rewrite I -C ( t,c ) η,H using Lemma D.1 as

which can then be decomposed as

Minkowski's inequality and A3 give E 1 / 2 [ ∥ η H ∑ H h =1 ∑ H ℓ = h +1 ¯ A c Γ ( c,η ) t,ℓ +1: H ∥ 2 ] ≤ ηH 2 ∥ ¯ A c ∥ . The second term has a reverse martingale structure, and we thus have

which is the result of the lemma.

Lemma C.2. Assume A1 and A3. Recall ˜ C c t +1 = ∑ H h =1 { Γ ( c,η ) t,h +1: H -(I -¯ A c ) H -h } . Then we have

Proof. We start by recalling the definition of ˜ C c t +1 , that is

Using Lemma D.1, we have

By Minkowski's inequality and Assumption A3, we obtain

Now, we notice that

and the result of the lemma follows.

Lemma C.3. Assume A1 and A3. Recall C ( t,c ) η,H = ∑ H h =1 Γ ( c,η ) t,h +1: H then

Proof. Denote A c h = A c ( Z c t +1 ,h ) ,

Using Lemma D.1, we can rewrite

Using triangle inequality and the fact that A c h 's are independent from each other, we have

By triangle inequality, and using the definition of C A and ∥ Σ c ˜ A ∥ , we have E [ ∥ A c k -A c ′ k ∥ ] ≤ E [ ∥ A c k -¯ A c ∥ + ∥ ¯ A c -¯ A c ′ ∥ + ∥ A c ′ k -¯ A c ′ ∥ ] ≤ 2 C A +2 ∥ Σ c ˜ A ∥ . Therefore, we obtain

and the result follows.

C.2 Proof

The linear structure of SCAFFLSA 's updates allow to decompose the updates between a transient term, and a fluctuation term. To materialize this, we define the following virtual parameters

These parameters are updated similarly to θ t 's and ξ c t 's, although without the last fluctuation term. For the virtual parameter ˇ θ , the update is similar to (8), as follows

which gives, after H local updates,

where we recall Γ c t +1 = ∏ H h =1 (I -η A ( Z c t,h )) and C c t +1 = ∑ H h =1 Γ ( c,η ) t,h +1: H . The virtual parameters obtained after H local updates are then aggregated as

This is then used to define the virtual control variates, similarly to (9),

These updates can be summarized over one block, which gives

The analysis of SCAFFLSA can then be decomposed into (i) analysis of the "transient" virtual iterates ˇ θ t 's and ˇ ξ c t 's, and (ii) analysis of the fluctuations θ t -ˇ θ t and ξ c t -ˇ ξ c t .

Analysis of the Transient Term. First, we analyze the convergence of the virtual variables ˇ θ t and ˇ ξ c t for t ≥ 0 and c ∈ { 1 , . . . , N } . Consider the Lyapunov function,

which is naturally defined as the error in θ ⋆ estimation using the virtual iterates, on communication rounds, and the average error on the virtual control variates.

Theorem C.4. Assume A1 and A3. Let η, H such that ηaH ≤ 1 , and H ≤ a 2 η { C 2 A + ∥ Σ c ε ∥} , and set ξ c 0 = 0 for all c ∈ [ N ] . Then, the sequence ( ψ t ) t ∈ N satisfies, for all t ≥ 0 ,

where ψ 0 = ∥ θ 0 -θ ⋆ ∥ 2 + η 2 H 2 N ∑ N c =1 ∥ ¯ A c ( θ c ⋆ -θ ⋆ ) ∥ 2 .

Proof. Expression of the Lyapunov function. Since the sum virtual control variates is ∑ N t =1 ˇ ξ c t = ∑ N t =1 ˇ ξ ⋆ c = 0 , we have ˇ θ t +1 = 1 N ∑ N c =1 ˇ θ c t,H = 1 N ∑ N c =1 ˇ θ c t,H -ηH ( ξ c t -ξ c ⋆ ) . Applying Lemma D.3, we obtain

since ˇ ξ c t +1 = ˇ ξ c t + 1 ηH ( ˇ θ t +1 -ˇ θ c t,H ) . Adding η 2 H 2 N ∑ N c =1 ∥ ˇ ξ c t +1 -ξ c ⋆ ∥ 2 on both sides, we obtain

where we defined C c η,H = ∑ H h =1 Γ ( c,η ) t,h +1: H . In the following, we will use the filtration of all events up to step t , F t := σ ( Z c s,h , 0 ≤ s ≤ t, 0 ≤ h ≤ H, 1 ≤ c ≤ N ) .

Using Young's inequality, and Assumption A3, we can bound

We thus obtain, for H such that ηaH ≤ 1 , and after setting α 0 = ηaH 2 and using the facts that (1 -ηaH )(1 + α 0 ) ≤ 1 -ηaH 2 and 1 + α -1 0 ≤ 2 α -1 0 ,

Then, since ηH a { C 2 A + ∥ Σ c ε ∥ } ≤ 1 2 , we obtain

and the result follows by plugging (39) back in (38).

Analysis of the Fluctuations. To study the fluctuations, we define the following quantities,

Our analysis is based on a careful study of the recurrence between variances and covariances of parameters and control variates. We thus start by deriving recurrence properties on these quantities. From the update of θ t , we have,

which can be rewritten as a recursive update of the fluctuations

Similarly, we have, for the fluctuations of the control variates

Remark that, for all t ≥ 0 , ˜ θ t and ˜ ξ c t 's are sums of (random) linear operations computed on zero-mean vectors, that are independent from these linear operations. Thus, for all t ≥ 0 and all c ∈ { 1 , . . . , N } we have

̸

We now aim at recursively finding a sequence of upper bounds { b ( θ,θ ) t , b ( θ,ξ ) , b = , b = } t ≥ 0 such that, for all t ≥ 0 , c, c ′ ∈ { 1 , . . . , N } such that c = c ′ ,

̸

̸

(Initialization.) For t = 0 , nothing is random so the fluctuations are zero, and b ( θ,θ ) 0 = b ( θ,ξ ) 0 = b = = b = = 0 . We also study the first iteration of SCAFFLSA . In the following lemma, we give upper bounds on the variances and covariances of the parameters obtained after one iteration.

Lemma C.5. Assume A1 and A3, then the first iterate of SCAFFLSA satisfy the following inequalities

̸

Proof. (Value of b ( θ,θ ) 1 .) From the definition of ˜ θ 1 , we have ˜ θ 1 = η N ∑ N c =1 ∑ H h =1 Γ ( c,η ) 1 ,h +1: H ω c ( Z c 1 ,h ) . By independence of the agents, and since E [Γ ( c,η ) 1 ,h +1: H ω ( Z c 1 ,h )] = 0 for all c ∈ { 1 , . . . , N } , and for all h ∈ { 0 , . . . , H -1 } ,

where the second equality comes from the fact that, for all h ∈ { 1 , . . . , H -1 } , the matrix Γ ( c,η ) 1 ,h +1: H and the vector ω c ( Z c 1 ,h ) are independent. Triangle inequality, Jensen's inequality, and definition of the operator norm then give

Assumption A3 ensures that E [ ∥ ∥ ∥ Γ ( c,η ) 1 ,h +1: H ∥ ∥ ∥ 2 ] ≤ 1 , and we have

(Value of b = 1 .) Let c ∈ { 1 , . . . , N } . The definition of ˜ ξ c 1 gives the following expression for the fluctuation ˜ ξ c 1 = 1 NH ∑ N ˜ c =1 ∑ H h =1 { Γ (˜ c,η ) 1 ,h +1: H ω ˜ c ( Z ˜ c 1 ,h ) } -1 H ∑ H h =1 Γ ( c,η ) 1 ,h +1: H ω c ( Z c 1 ,h ) . Therefore,

̸

we have

With similar arguments as above, we have

Assuming N ≥ 2 , triangle inequality gives

(Value of b ( θ,ξ ) 1 .) For the covariance of ˜ ξ c 1 and ˜ θ 1 , we have

As a result, we have

̸

(Value of b = .) Similarly to above, for c = c ′ , we have

̸

Which results in the bound

̸

Lemma C.6. Let ν > 0 , and assume that ηH { C A + ∥ Σ c ˜ A ∥ 1 / 2 } ≤ ν and η 2 H 2 { C 2 A + ∥ Σ c ˜ A ∥ } ≤ ν . Then the following inequalities hold for any t ≥ 0 , b ( θ,θ ) t +1 ≤ (1 -ηa ) 2 H b ( θ,θ ) t + νηHb ( θ,ξ ) t +2 ν ηH N b = t + νη 2 H 2 b = t + 3 η 2 H N ∥ Σ ω ∥ , ηHb ( θ,ξ ) t +1 ≤ 2 νb ( θ,θ ) t +3 νηHb ( θ,ξ ) t + 2 ν N η 2 H 2 b = t +2 νη 2 H 2 b = t + 2 η 2 H N ∥ Σ ω ∥ , η 2 H 2 b = t +1 ≤ 2 νb ( θ,θ ) t +3 νηHb ( θ,ξ ) t +4 νη 2 H 2 b = t +3 νη 2 H 2 b = t + η 2 H ∥ Σ ω ∥ , η 2 H 2 b = t +1 ≤ 2 νb ( θ,θ ) t +3 νηHb ( θ,ξ ) t + 3 ν N η 2 H 2 b = t +4 νη 2 H 2 b = t + 3 η 2 H N ∥ Σ ω ∥ ,

̸

Proof. (Value of b ( θ,θ ) t +1 .) Replacing ˜ θ t +1 by its expression from (40), then expanding the expression, we have

From the triangle inequality and Jensen's inequality, we have

̸

Now, we have from (41) that E [ ˜ θ t ] = E [ ˜ ξ c t ] = 0 . Thus, we have, for all c ∈ { 1 , . . . , N } ,

Which results in the following inequality

̸

̸

̸

̸

Using Lemma C.2, we obtain

̸

Assuming

̸

This gives our first inequality that links our upper bounds,

(Value of b ( θ,ξ ) t +1 .) As for b ( θ,θ ) t +1 , we bound, for c ∈ { 1 , . . . , N } ,

Now we proceed as above by taking the expectation, then the norm, and using the triangle inequality. Note that by (41), we have E [¯ ε t +1 ˜ θ ⊤ t ˜ Γ c t +1 ⊤ ] = 0 , E [¯ ε t +1 ˜ ξ c t ⊤ ( I -1 H C c t +1 ) ⊤ ] = 0 , E [¯ ε t +1 ∑ N ˜ c =1 ˜ ξ ˜ c t ⊤ ˜ C ˜ c t +1 ⊤ ] = 0 , E [Γ t +1 ˜ θ t ˜ ε c t +1 ⊤ ] = 0 , and E [ ˜ C ˜ c t +1 ˜ ξ ˜ c t ˜ ε c t +1 ⊤ ] = 0 . After using

̸

Jensen's inequality, we obtain

Using Lemma C.1, Lemma C.2, and Lemma C.3, we obtain

where we used the two following inequalities

This leads to the following inequality

̸

where we used ∥ ∥ ∥ E [ η H ¯ ε t +1 ˜ ε c t +1 ⊤ ] ∥ ∥ ∥ ≤ b ( θ,ξ ) 1 = 2 η N ∥ Σ ω ∥ . Assuming ηH { C A + ∥ Σ c ˜ A ∥ 1 / 2 } ≤ ν and η 2 H 2 { C 2 A + ∥ Σ c ˜ A ∥ } ≤ ν , we obtain the following bound

̸

̸

(Value of b = t +1 and b = t +1 .) As above, we start by expanding the matrix product,

Taking the expectation, then the norm, and using triangle inequality and Jensen's inequality, we obtain

We can now use Lemma C.1, Lemma C.2, and Lemma C.3 to obtain the following upper bound

This bound can be simplified as

which can be simplified as

̸

We now distinguish two cases, when c = c ′ and when c = c ′ . First, let c = c ′ , we obtain

̸

since when c = c ′ , we have à ∥ E [ ˜ ξ c t ˜ ξ c ′ t ⊤ ] ∥ ≤ b = t and ∥ ∥ ∥ 1 H 2 E [ ˜ ε c t +1 ˜ ε c ′ t +1 ⊤ ] ∥ ∥ ∥ ≤ b = 1 = N -1 NH ∥ Σ ω ∥ . Assuming ηH { C A + ∥ Σ c ˜ A ∥ 1 / 2 } ≤ ν and η 2 H 2 { C 2 A + ∥ Σ c ˜ A ∥ } ≤ ν , we obtain

̸

η 2 H 2 b = t +1 ≤ 2 νb ( θ,θ ) t +3 νηHb ( θ,ξ ) t +4 νη 2 H 2 b = t +3 νη 2 H 2 b = t + η 2 H ∥ Σ ω ∥ . ′

We proceed similarly for c = c , which gives

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

Corollary C.7. Assume that ηH { C A + ∥ Σ c ˜ A ∥ 1 / 2 } ≤ a 240(C A + ∥ Σ c ˜ A ∥ 1 / 2 ) and η 2 H 2 { C 2 A + ∥ Σ c ˜ A ∥ } ≤ a 240(C A + ∥ Σ c ˜ A ∥ 1 / 2 ) , set ω = min ( 1 , a 12(C A + ∥ Σ c ˜ A ∥ 1 / 2 ) ) , then it holds that

̸

̸

Assuming ηaH ≤ 1 2 , we have 1 -1 2 ≤ 1 -ηaH 2 . This in turn ensures that

̸

Proof. From Lemma C.6, we have, for any 0 < ω < 1 , ν > 0 , and assuming that ηH { C A + ∥ Σ c ˜ A ∥ 1 / 2 } ≤ ν and η 2 H 2 { C 2 A + ∥ Σ c ˜ A ∥ } ≤ ν , and since ω ≤ 1 ,

̸

Now, we choose ω = min ( 1 , a 12(C A + ∥ Σ c ˜ A ∥ 1 / 2 ) ) and obtain

Additionally, ω ≤ 1 , thus 3 + 6 ω ≤ 9 and we obtain

̸

Choosing ν ≤ 20 ≤ 240(C A + ∥ Σ c ˜ A ∥ 1 / 2 ) gives the result.

Complete analysis of SCAFFLSA . We can now state our main theorem, which gives an upper bound on the expected distance between the iterates of SCAFFLSA and the solution θ ⋆ .

Theorem C.8. Assume A1 and A3. Let η, H such that ηaH ≤ 1 , and H ≤ a 240 η { C 2 A + ∥ Σ c ε ∥} , and set ξ c 0 = 0 for all c ∈ [ N ] . Then, the sequence ( ψ t ) t ∈ N satisfies, for all t ≥ 0 ,

Proof. Recall our decomposition θ t -θ ⋆ = ˇ θ t -θ ⋆ + ˜ θ . By Young's inequality, we have

which gives, for any t ≥ 0 ,

̸

̸

̸

By Theorem C.4, we have E [ ∥ ˇ θ t -θ ⋆ ∥ 2 ] ≤ ( 1 -ηaH 2 ) t ψ 0 , and by Corollary C.7, we have E [ ∥ ˜ θ t ∥ 2 ] ≤ db ( θ,θ ) t ≤ 18 ηd Na ∥ Σ ω ∥ . Combine the two results, we obtain

replacing ψ 0 = ∥ θ 0 -θ ⋆ ∥ 2 + η 2 H 2 N ∑ N c =1 ∥ ¯ A c ( θ c ⋆ -θ ⋆ ) ∥ 2 gives the result of the theorem.

Corollary C.9. Under the Assumptions of Theorem C.8, one may set the parameter of SCAFFLSA to

which guarantees E [ ∥ θ t -θ ⋆ ∥ 2 ] ≤ ϵ 2 after a number of communication rounds

The overall sample complexity of the algorithm is then

Proof. Let ϵ > 0 . Starting from Theorem C.8's upper bound, we have E [ ∥ θ t -θ ⋆ ∥ 2 ] ≤ ϵ 2 whenever

where ψ 0 = ∥ θ 0 -θ ⋆ ∥ 2 + η 2 H 2 N ∑ N c =1 ∥ ¯ A c ( θ c ⋆ -θ ⋆ ) ∥ 2 . This gives a first condition 36 dη Na ∥ Σ ω ∥ ≤ ϵ 2 , which requires

This allows to take any value of H such that H ≤ a 240 η { C 2 A + ∥ Σ c ε ∥} = 72 240 Nϵ 2 { C 2 A + ∥ Σ c ε ∥} . With such setting, it remains to set the number of communication T to

which ensures that ( 1 -ηaH 2 ) t 2 ψ 0 ≤ ϵ 2 2 .

D Technical proofs

Lemma D.1. For any matrix-valued sequences ( U n ) n ∈ N , ( V n ) n ∈ N and for any M ∈ N , it holds that:

Lemma D.2 (Stability of the deterministic product) . Assume A3. Then, for any u ∈ R d and h ∈ N ,

Proof. Since ( Z c t,h ) 1 ≤ h ≤ H are i.i.d, we get

The proof then follows from the elementary inequality: for any square-integrable random vector U , ∥ E [ U ] ∥ ≤ ( E [ ∥ U ∥ 2 ]) 1 / 2 .

Lemma D.3. Let ( x i ) N i =1 , and ( y i ) N i =1 be N vectors of R d . Denote ¯ x N = (1 /N ) ∑ N i =1 x i and ¯ y N = (1 /N ) ∑ N i =1 y i . Then,

Proof. Define x = [ x ⊤ 1 , . . . , x ⊤ N ] ⊤ and y = [ y ⊤ 1 , . . . , y ⊤ N ] ⊤ ∈ R Nd . Define by P the orthogonal projector on

We show that Px = [¯ x ⊤ N , . . . , ¯ x ⊤ N ] ⊤ . Note indeed that for any z = [ z ⊤ , . . . , z ⊤ ] ⊤ ∈ E , we get (with a slight abuse of notations, ⟨· , ·⟩ denotes the scalar product in R Nd and R d )

The proof follows from Pythagoras identity which shows that

.

Lemma D.4. Assume A4. Let Z be a random variable taking values in a state space ( Z , Z ) with distribution π c . Set η ≥ 0 , then for any vector u ∈ R d , we have

Proof. First, remark that

Since we have E [ 1 2 ( A c ( Z ) + A c ( Z ) ⊤ )] ≽ a I and E [ 1 2 ( A c ( Z ) + A c ( Z ) ⊤ )] ≽ 1 L E [ A c ( Z ) ⊤ A c ( Z )] , we obtain

which gives the result.

E TD learning as a federated LSA problem

In this section we specify TD(0) as a particular instance of the LSA algorithm. In the setting of linear functional approximation the problem of estimating V π ( s ) reduces to the problem of estimating θ ⋆ ∈ R d , which can be done via the LSA procedure. For the agent c ∈ [ N ] the k -th step randomness is given by the tuple Z c k = ( S c k , A c k , S c k +1 ) . With slight abuse of notation, we write A c t,h instead of A ( Z c t,h ) , and b c t,h instead of b ( Z c t,h ) . Then the corresponding LSA update equation with constant step size η can be written as

where A c t,h and b c t,h are given by

Respective specialisation of FedLSA and SCAFFLSA algorithms to TD learning are stated in Algorithm 4 and Algorithm 5.

Algorithm 4 Federated TD(0): FedLSA applied to TD(0) with linear functional approximation

Input: η > 0 , θ 0 ∈ R d , T, N, H > 0

Initialize θ t, 0 = θ t for t = 0 to T -1 do

for h = 1 to H do

Receive tuple ( S c t,h , A c t,h , S c t,h +1 ) following TD 1 and perform local update:

where A c t,h and b c t,h are given in (42)

Average: θ t +1 = 1 ∑ N θ c

Algorithm 5 SCAFFTD (0): SCAFFLSA applied to TD(0) with linear functional approximation

Input: η > 0 , θ 0 ∈ R d , T, N, H > 0

Initialize θ t, 0 = θ t for t = 0 to T -1 do

for c = 1 to N do for h = 1 to H do

Receive tuple ( S c t,h , A c t,h , S c t,h +1 ) following TD 1 and perform local update:

where A c t,h and b c t,h are given in (42)

Average: θ t +1 = 1 N ∑ N c =1 θ c t,H

Update local control variates: ξ c t +1 = ξ c t + 1 ηH ( θ t +1 -ˆ θ c t,H ) .

The corresponding local agent's system writes as ¯ A c θ c ⋆ = ¯ b c , where we have, respectively,

The authors of [50] study the corresponding virtual MDP dynamics with ˜ P = N -1 ∑ N c =1 P c MDP , ˜ r = N -1 ∑ N c =1 r c . Next, introducing the invariant distribution of the kernel ˜ µ of the averaged state kernel

we have ˜ θ as an optimal parameter corresponding to the system ˜ A ˜ θ = ˜ b . Here

E.1 Proof of Claim 3.1.

We prove the following inequalities

The proof below closely follows [42] (Lemma 7) and [45] (Lemma 1). Everywhere in this subsection we use a generic notation A c 1 as an alias for the random matrix A c 1 , 1 . Now, using TD 3 and (5), we get almost surely, which implies ∥ ¯ A c ∥ ≤ 1 + γ for any c ∈ [ N ] , giving (44). This implies, using the definition of Σ c ˜ A , that

and the bound (45) follows. Next we observe that

where the latter inequality follows from TD 3, and thus (46) holds. In order to check the last equation (47), we note first that the bound for a and η ∞ readily follows from the ones presented in [42][Lemma 5] and [42][Lemma 7]. To check assumption A4, note first that, with s ∼ µ c , s ′ ∼ P π ( ·| s ) , we have where we additionally used that

for any u, v ∈ R d . Thus, we get that

The rest of the proof follows from the fact that

which holds whenever (48) is satisfied; see e.g. in [30] (Lemma 5) or [45] (Lemma 7).

Based on these results, we instantiate the results summarized in Table 1 to Federated TD learning in Table 2.

F Analysis of Scaffnew for Federated LSA

To mitigate the bias caused by local training, we may use control variates. We assume in this section that at each iteration we choose, with probability p , whether agents should communicate or not. Consider the following algorithm, where for k = 1 , . . . , T / p , we compute

Algorithm 6 "Scaffnew": Stochastic Controlled FedLSA with probabilistic communication

Input:

η >

0

,

θ

0

, ξ

c

0

∈

R

d

,

T, N, H, p >

0

Set: K = T / p

for k = 1 to K do

for c = 1 to N do

Receive Z c k and perform local update:

Average local iterates: θ c = 1 ∑ N ˆ θ c

else

i.e. we update the local parameters with LSA adjusted with a control variate ξ c k -1 . This control variate is initialized to zero, and updated after each communication round. We draw a Bernoulli random variable B k with success probability p and then update the parameter as follows:

We then update the control variate

where we have set ξ c 0 = 0 . We state this algorithm in Algorithm 6.

Note that, for all k ∈ N , ∑ N c =1 ξ c t = 0 . . We now proceed to the proof, which amounts to constructing a common Lyapunov function for the sequences { θ c k } k ∈ N and { ξ c k } k ∈ N . Define the Lyapunov function,

where θ ⋆ is the solution of ¯ Aθ ⋆ = ¯ b , and ξ c ⋆ = ¯ A c ( θ ⋆ -θ c ⋆ ) . A natural measure of heterogeneity is then given by

To analyze this algorithm, we'll study the decrease of the expected value of ψ k , where the expectation is over randomness of the communication and the stochastic oracles. This requires a stronger assumption than the Assumption A3 that we used in Section 4.

A4. There exist constants a, L > 0 , such that for any η ∈ (0 , 1 / L ) , c ∈ [ N ] , it holds for Z c 1 ∼ π c , that

This assumption is slightly more restrictive than A3. Indeed, whenever A4 holds, A3 also holds with the same constant a (see 42, 45). In the case of TD, this assumption holds with L = 1+ γ (1 -γ ) 2 ν .

Lemma F.1 (One step progress) . Assume A1 and A4. Assume that η ≤ 1 2 L . The iterates of the algorithm described above satisfy

Proof. Decomposition of the update. Remark that the update can be reformulated as

where ω c ( z ) = ˜ A c ( z ) θ ⋆ -˜ b c ( z ) . This comes from the fact that, for all z ,

Expression of communication steps. Using that ∑ N c =1 ξ c k -1 = 0 and ∑ N c =1 ξ c ⋆ = 0 , we get

The first term can be upper bounded by using Lemma D.3, which gives

We now expand the first term in the right-hand side of the previous equation. This gives

which yields

On the other hand, note that

By combining (51) and (50), we get

Progress in local updates. We now bound the first term of the sum in (52). For c ∈ [ N ] , (49) gives

Define the σ -algebra G k -1 = σ ( B s , s ≤ k -1 , Z c s , s ≤ k -1 , c ∈ [ N ]) . We now bound the conditional expectation of T 1

G

where we used the fact that ⟨ I , ω c ( Z c k ) ⟩ = 0 . Using Young's inequality for products, and Lemma D.4 with η ≤ 1 2 L and u = θ c k -θ ⋆ , we then obtain

Plugging (54) in (53) and using the assumption η ≤ 1 2 L , we obtain

Bounding the Lyapunov function. Taking the condtional expectation of (52) and using (55) for c = 1 to N , we obtain the following bound on the Lyapunov function

and the result of the Lemma follows from the Tower property.

Theorem F.2 (Convergence rate) . Assume A1 and A3( 2 ). Then, for any η ≤ 1 2 L and T > 0 , it holds

where ζ = min ( ηa, p 2 ) .

Corollary F.3 (Iteration complexity) . Let ϵ > 0 . Set η = min ( 1 2 L , ϵ 2 a 8¯ σ ε ) and p = √ ηa (so that ζ = ηa ). Then, E [ ψ K ] ≤ ϵ 2 as long as the number of iterations is

which corresponds to an expected number of communication rounds

Theorem F.4 (No linear speedup in the probabilistic communication setting with control variates) . The bounds obtained in Theorem F.2 are minimax optimal up to constants that are independent from the problem. Precisely, for every ( p, η ) there exists a FLSA problem such that

where we have defined ζ = min ( 2 ηa, p 2 ) .

Proof. Define for all c ∈ [ N ] ,

where u is a vector whom all coordinates are equal to 1. We also consider the sequence of i.i.d random variables ( Z c k ) such that that for all c ∈ [ N ] and 0 ≤ t ≤ T , Z c k follows a Rademacher distribution. Moreover, we define

We follow the same proof of Lemma F.1 until the chain of equalities breaks. Thereby, we start from

where we used that A ( Z ) = A . Unrolling the recursion gives the desired result.

c c k ¯ c

G Experimental Details and Additional Experiments

G.1 Experimental Details

Here, we give additional details regarding the numerical experiments. The environments used are instances of Garnet, where we use 30 states, embedded via a random projection in a d = 8 -dimensional space. We use two actions, and consider a branching factor of two, meaning that, from each state, one can transition to two different states with some probability. The rewards are then drawn uniformly randomly from the interval [0 , 1] .

In the homogeneous setting, we sample one Garnet environment. Each client then receives a perturbation of this instance, where we perturb all non-zeros probabilities of transition from one state to another and all rewards with a random variable ϵ ∼ U (0 , 0 . 02) .

In the heterogeneous setting, we proceed similarly, except that we sample two different Garnet environments, with the same parameters. Half of the agents receive the first environment, and the second half receive the second environment. As in the homogeneous setting, each agent's environment slightly differs from the base environment by a small perturbation ϵ ∼ U (0 , 0 . 02) .

All the experiments presented in this paper can be run on a single laptop in just a few hours.

In particular this implies

0

10

20

30

40

50

Communications (

×

10000)

10

-

3

10

-

1

10

1

(a) Homogeneous,

H

= 1

0

10

20

30

40

50

Communications (

×

1000)

10

-

3

10

-

1

10

1

(b) Homogeneous,

H

= 10

0

10

20

30

40

50

Communications (

×

100)

10

-

3

10

-

1

10

1

(c) Homogeneous,

H

= 100

0

10

20

30

40

50

Communications (

×

10)

10

-

3

10

-

1

10

1

(d) Homogeneous,

H

= 1000

0

10

20

30

40

50

Communications

10

-

3

10

-

1

10

1

(e) Homogeneous,

H

= 10000

0

10

20

30

40

50

Communications (

×

10000)

10

-

3

10

-

1

10

1

(f) Homogeneous,

H

= 1

0

10

20

30

40

50

Communications (

×

1000)

10

-

3

10

-

1

10

1

(g) Homogeneous,

H

= 10

0

10

20

30

40

50

Communications (

×

100)

10

-

3

10

-

1

10

1

(h) Homogeneous,

H

= 100

0

10

20

30

40

50

Communications (

×

10)

10

-

3

10

-

1

10

1

FedLSA

SCAFFLSA

FedLSA's bias

(i) Homogeneous,

H

= 1000

0

10

20

30

40

50

Communications

10

-

3

10

-

1

10

1

(j) Homogeneous,

H

= 10000

0

10

20

30

40

50

Communications (

×

10000)

10

-

3

10

-

1

10

1

(a) Heterogeneous,

H

= 1

0

10

20

30

40

50

Communications (

×

1000)

10

-

3

10

-

1

10

1

(b) Heterogeneous,

H

= 10

0

10

20

30

40

50

Communications (

×

100)

10

-

3

10

-

1

10

1

(c) Heterogeneous,

H

= 100

0

10

20

30

40

50

Communications (

×

10)

10

-

3

10

-

1

10

1

(d) Heterogeneous,

H

= 1000

0

10

20

30

40

50

Communications

10

-

3

10

-

1

10

1

(e) Heterogeneous,

H

= 10000

0

10

20

30

40

50

Communications (

×

10000)

10

-

3

10

-

1

10

1

(f) Heterogeneous,

H

= 1

0

10

20

30

40

50

Communications (

×

1000)

10

-

3

10

-

1

10

1

(g) Heterogeneous,

H

= 10

0

10

20

30

40

50

Communications (

×

100)

10

-

3

10

-

1

10

1

(h) Heterogeneous,

H

= 100

0

10

20

30

40

50

Communications (

×

10)

10

-

3

10

-

1

10

1

FedLSA

SCAFFLSA

FedLSA's bias

(i) Heterogeneous,

H

= 1000

0

10

20

30

40

50

Communications

10

-

3

10

-

1

10

1

(j) Heterogeneous,

H

= 10000

G.2 Additional Experiments: Number of Local Steps and Smaller Step-Size

In this section, we give more experimental results for FedLSA and SCAFFLSA . We use the same setting as in Section 6, but use more settings of local steps.

In Figure 3 and Figure 4, we give report the counterpart of Figure 1 with a wider ranger of number of local updates H ∈ { 1 , 10 , 100 , 1000 , 10000 } . The results obtained here match with observations from Section 6: in homogeneous settings, FedLSA and SCAFFLSA exhibit very similar behavior. In both methods, increasing the number of local steps speeds-up the training, until the stochastic noise dominates. At this point, both algorithms reach a stationary regime with similar error. In heterogeneous settings, while FedLSA 's bias is smaller than the variance of its iterates, train-

0

10

20

30

40

50

Communications (

×

10000)

10

-

4

10

-

2

10

0

(a) Homogeneous,

H

= 1

0

10

20

30

40

50

Communications (

×

1000)

10

-

4

10

-

2

10

0

(b) Homogeneous,

H

= 10

0

10

20

30

40

50

Communications (

×

100)

10

-

4

10

-

2

10

0

(c) Homogeneous,

H

= 100

0

10

20

30

40

50

Communications (

×

10)

10

-

4

10

-

2

10

0

(d) Homogeneous,

H

= 1000

0

10

20

30

40

50

Communications

10

-

4

10

-

2

10

0

(e) Homogeneous,

H

= 10000

0

10

20

30

40

50

Communications (

×

10000)

10

-

4

10

-

2

10

0

(f) Homogeneous,

H

= 1

0

10

20

30

40

50

Communications (

×

1000)

10

-

4

10

-

2

10

0

(g) Homogeneous,

H

= 10

0

10

20

30

40

50

Communications (

×

100)

10

-

4

10

-

2

10

0

(h) Homogeneous,

H

= 100

0

10

20

30

40

50

Communications (

×

10)

10

-

4

10

-

2

10

0

FedLSA

SCAFFLSA

FedLSA's bias

(i) Homogeneous,

H

= 1000

0

10

20

30

40

50

Communications

10

-

4

10

-

2

10

0

(j) Homogeneous,

H

= 10000

0

10

20

30

40

50

Communications (

×

10000)

10

-

4

10

-

2

10

0

(a) Heterogeneous,

H

= 1

0

10

20

30

40

50

Communications (

×

1000)

10

-

4

10

-

2

10

0

(b) Heterogeneous,

H

= 10

0

10

20

30

40

50

Communications (

×

100)

10

-

4

10

-

2

10

0

(c) Heterogeneous,

H

= 100

0

10

20

30

40

50

Communications (

×

10)

10

-

4

10

-

2

10

0

(d) Heterogeneous,

H

= 1000

0

10

20

30

40

50

Communications

10

-

4

10

-

2

10

0

(e) Heterogeneous,

H

= 10000

0

10

20

30

40

50

Communications (

×

10000)

10

-

4

10

-

2

10

0

(f) Heterogeneous,

H

= 1

0

10

20

30

40

50

Communications (

×

1000)

10

-

4

10

-

2

10

0

(g) Heterogeneous,

H

= 10

0

10

20

30

40

50

Communications (

×

100)

10

-

4

10

-

2

10

0

(h) Heterogeneous,

H

= 100

0

10

20

30

40

50

Communications (

×

10)

10

-

4

10

-

2

10

0

FedLSA

SCAFFLSA

FedLSA's bias

(i) Heterogeneous,

H

= 1000

0

10

20

30

40

50

Communications

10

-

4

10

-

2

10

0

(j) Heterogeneous,

H

= 10000

ing speeds up when the number of local steps increases. After that point, bias dominates, while SCAFFLSA preserves the speed-up by eliminating this bias.

Finally, we report in Figure 5 and Figure 6 the results when running the same experiments using a smaller step size η = 0 . 01 for different number of agents and local updates. In this setting, all algorithms manage to find better estimators, since the amount of variance depends on the step size (as seen in Theorem 4.1 and Theorem 5.1). Additionally, FedLSA 's bias is smaller than in Figure 7, which is also in line with the upper bound E 1 / 2 [ ∥ ˜ θ ( bi , bi ) t ∥ 2 ] ≲ ηH E c [ ∥ θ c ⋆ -θ ⋆ ∥ ] a from Theorem 4.1.

0

10

20

30

40

50

Communications (

×

1000)

10

-

3

10

-

1

10

1

(a) Homogeneous,

N

= 10

, H

= 10

0

10

20

30

40

50

Communications (

×

10)

10

-

3

10

-

1

10

1

(b) Homogeneous,

N

= 10

, H

= 1000

0

10

20

30

40

50

Communications (

×

1000)

10

-

3

10

-

1

10

1

(c) Homogeneous,

N

= 100

, H

= 10

0

10

20

30

40

50

Communications (

×

10)

10

-

3

10

-

1

10

1

FedLSA

SCAFFLSA

FedTD without bias

FedLSA's bias

(d) Homogeneous,

N

= 100

, H

= 1000

0

10

20

30

40

50

Communications (

×

1000)

10

-

3

10

-

1

10

1

(e) Heterogeneous,

N

= 10

, H

= 10

0

10

20

30

40

50

Communications (

×

10)

10

-

3

10

-

1

10

1

(f) Heterogeneous,

N

= 10

, H

= 1000

0

10

20

30

40

50

Communications (

×

1000)

10

-

3

10

-

1

10

1

(g) Heterogeneous,

N

= 100

, H

= 10

0

10

20

30

40

50

Communications (

×

10)

10

-

3

10

-

1

10

1

FedLSA

SCAFFLSA

FedTD without bias

FedLSA's bias

(h) Heterogeneous,

N

= 100

, H

= 1000

0

10

20

30

40

50

Communications (

×

1000)

10

-

4

10

-

2

10

0

(a) Homogeneous,

N

= 10

, H

= 10

0

10

20

30

40

50

Communications (

×

10)

10

-

4

10

-

2

10

0

(b) Homogeneous,

N

= 10

, H

= 1000

0

10

20

30

40

50

Communications (

×

1000)

10

-

4

10

-

2

10

0

(c) Homogeneous,

N

= 100

, H

= 10

0

10

20

30

40

50

Communications (

×

10)

10

-

4

10

-

2

10

0

FedLSA

SCAFFLSA

FedTD without bias

FedLSA's bias

(d) Homogeneous,

N

= 100

, H

= 1000

0

10

20

30

40

50

Communications (

×

1000)

10

-

4

10

-

2

10

0

(e) Heterogeneous,

N

= 10

, H

= 10

0

10

20

30

40

50

Communications (

×

10)

10

-

4

10

-

2

10

0

(f) Heterogeneous,

N

= 10

, H

= 1000

0

10

20

30

40

50

Communications (

×

100)

10

-

4

10

-

2

10

0

(g) Heterogeneous,

N

= 100

, H

= 10

0

10

20

30

40

50

Communications (

×

10)

10

-

4

10

-

2

10

0

FedLSA

SCAFFLSA

FedTD without bias

FedLSA's bias

(h) Heterogeneous,

N

= 100

, H

= 1000

G.3 Additional Experiments: Convergence of FedLSA

In Figure 7 and Figure 8, we give the counterpart of Figure 1, where we additionally plot the MSE of the estimator θ t + ˜ θ ( bi , bi ) ∞ , for different settings of all parameters. We recall that ˜ θ ( bi , bi ) ∞ = (I -¯ Γ ( η ) H ) -1 ¯ ρ H is the bias of FedLSA , as we proved in Theorem 4.1. Therefore, θ t + ˜ θ ( bi , bi ) ∞ is a

proper estimator of θ ⋆ , although, of course, it cannot be computed in practice since the bias ˜ θ ( bi , bi ) ∞ is unknown. and we see in Figure 7 that, in homogeneous settings, we recover the same error as FedLSA and SCAFFLSA . Moreover, in heterogeneous settings, it has an error similar to the one of SCAFFLSA , meaning that FedLSA , once its bias is removed, converges similarly to SCAFFLSA . The latter, however, does not require to remove an unknown bias, and directly estimates the right quantity.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract, we claim (i) to analyze the FedLSA algorithm, with exact caracterization of the bias (see Section 4), (ii) to propose and analyze SCAFFLSA (see Section 5), and to apply it to federated TD (see Corollary 4.4, Corollary 5.3). We also propose results in the markovian sampling scheme in Corollary 4.5, and perform a numerical analysis of our method applied to federated TD in Section 6.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The main limitation of our paper is that the numerical analysis is done on federated TD on a synthetic dataset. Nonetheless, we emphasize that our paper is a theoretical paper, and that these experiments rightfully serve the purpose of illustrating the theoretical results.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All our theoretical results are provided with clear references to assumptions, that we all state in Section 3 to ensure that these assumptions are easy to find. All results are given with proofs, that are correctly references for each theorem and corollary.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Numerical results are stated with a complete description of the environments that are used, as well as the precise sets of hyperparameters that we used. We stress that the code (in Python) is provided as supplementary with the paper, making it easy for one to reproduce our numerical experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All the code is open source, and available at https://github.com/ pmangold/scafflsa .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The algorithm used in the numerical experiments are exactly the algorithms described in the paper. The Garnet environements are given with the parameters used for generation, and with reference to the original problem.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All experimental results are given with error bars that consist either in the standard deviation over multiple independent runs, or the minimal/maximal values obtained over multiple independent runs.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The experiments in this paper are ran on a single laptop, and can easily be reproduced by anyone with very limited computational power.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper is of purely theoretical nature, and the proposed methods do not deal with sensitive attributes that could induce unfairness or privacy issues.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper is of purely theoretical nature, and although the proposed methods could help deploy more federated learning solutions, this does not constitute a risk for societal harm.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: N/A.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: N/A since no existing assets are used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: N/A since the paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: N/A since the paper does not involve crowdsourcing nor research on human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Justification: N/A since the paper does not involve crowdsourcing nor research on human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper analyzes the convergence of scaffnew in the quadratic setup and achieves a linear speedup in the number of clients.  It is not attained by the original paper of Scaffnew. Additionaly, the author find an application of the federated quadratic problem -- Federated
Linear Stochastic Approximation and TD Learning.

STRENGTHS:
This paper tries to analyze the convergence of scaffnew in the quadratic setup and achieves a linear speedup in the number of clients.  It is not attained by the original paper of Scaffnew.

WEAKNESSES:
The achieved speedup holds only for quadratic setup and the application of the quadratic loss function is quite limited. 
The experiments are a little simple.

QUESTIONS:
According to my understanding, this work saves more communication complexity compared with scaffnew, while it is not claimed by the authors. This is observed in Table 1. Is it another advantage of this analysis? 

Can the theoretical analysis be extended to a general case?

The comparison in Figure 1 is hard to catch. Can you plot them in one figure for comparison, at least using the same legend? And considering more value of H and N.

LIMITATIONS:
The achieved speedup holds only for quadratic setup and the application of the quadratic loss function is quite limited.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a non-asymptotic analysis of Federated Linear Stochastic Approximation. The authors provide (biased and unbiased) finite-time MSE bounds for general LSA and TD learning under the assumption that the noise is i.i.d.. For Markovian noise, only an unbiased MSE bound is provided. Most importantly is that these bounds express this error as a function of the step-size used, number of agents and number of local updates. Finally, a new algorithmic variant, namely, SCAFFLSA is introduced, reducing communication between agents while maintaining the linear speed up in the algorithm. The contributions of the paper are illustrated through numerical studies.\n\nSTRENGTHS:\nThe reviewer is not very familiar with the federated learning literature, but as far as they are aware, most of the ideas of the paper are new. \n\nThe reviewer was unable to look at all proofs in detail, but have not identified any issues with respect to correctness.\n\nWEAKNESSES:\nThe organization of the paper is not great. First, it is very hard to follow the main text given the amount of symbols and equations. Secondly, contributions related to i.i.d and Markovian noise and TD learning seemed to be scrambled together. I particularly think that splitting the results into different sections (one for i.i.d. noise, one for Markovian noise and one tailored to TD learning) would improved the readability of the paper a lot.\n\nAlso, there is a missing \"c\" superscript right above equation (1) in Algorithm 1.\n\nThe paper should be revised for clarity and typos.\n\nQUESTIONS:\n- Is it possible for the authors provide a table similar to Table 1 in order to illustrate and compare the different outcomes between the three different scenarios analyzed by the paper (i.r. the i.i.d. , Markov and TD setting)? \n\n-Do the authors believe that their analysis could be extended to federated learning algorithms with vanishing step-size?\n\n- It is possible that I missed this in the main text, but are any drawbacks to using SCAFFLSA instead of Federated LSA?\n\nLIMITATIONS:\nThe authors identify and state which assumptions are necessary to hold for each of the results to be true, so the limitations of this work are objectively identified.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper first analyzed the performance of federated linear stochastic approximation or FedLSA algorithm. Second, it proposed a new algorithm called stochastic controlled averaging for Federated LAS or SCAFFLSA and analyzed its performance. The key idea of SCAFFLSA is to use a control variable to mitigate the client drift. The performance of the proposed algorithm was verified using experiments.

STRENGTHS:
1. The paper proposed a new analytical framework to analyze the sample and communication complexity of FedLSA. Using this approach, the paper analyzed the performance of SCAFFLSA. The paper also extended the analytical framework to Federated TD learning. 

2. The paper proposed a new algorithm, SCAFFLSA, which mitigate the client drift using a control variable. The sample and communication complexity of SCAFFLSA is significantly better than that of FedLSA and Scaffnew. The performance analysis of SCAFFLSA was also extended to Federated TD.

WEAKNESSES:
The experiments are relatively weak. It will be more beneficial if the authors could provide more applications of the proposed algorithm to other problems, especially for federated TD learning.

QUESTIONS:
What are the functions $\boldsymbol{A}^c()$ and $\boldsymbol{b}^c()$ like in practice? Could you please provide some examples?

LIMITATIONS:
There was no discussion on the limitations of the proposed algorithm.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15