Fault-Tolerant Federated Reinforcement Learning with Theoretical Guarantee

1 Introduction . Reinforcement learning ( RL ) has recently been applied to many real-world decision-making problems such as gaming , robotics , healthcare , etc . [ 1–3 ] . However , despite its impressive performances in simulation , RL often suffers from poor sample efficiency , which hinders its success in real-world applications [ 4 , 5 ] . For example , when RL is applied to provide clinical decision support [ 3 , 6 , 7 ] , its performance is limited by the number ( i.e. , sample size ) of admission records possessed by a hospital , which can not be synthetically generated [ 3 ] . As this challenge is usually faced by many agents ( e.g. , different hospitals ) , a natural solution is to encourage multiple RL agents to share their trajectories , to collectively build a better decision-making policy that one single agent can not obtain by itself . However , in many applications , raw RL trajectories contain sensitive information ( e.g. , the medical records contain sensitive information about patients ) and thus sharing them is prohibited . To this end , the recent success of Federated Learning ( FL ) [ 8–11 ] has inspired the setting of Federated Reinforcement Learning ( FRL ) [ 12 ] , which aims to federatively build a better policy from multiple RL agents without requiring them to share their raw trajectories . FRL is practically appealing for addressing the sample inefficiency of RL in real systems , such as autonomous driving [ 13 ] , fast personalization [ 14 ] , optimal control of IoT devices [ 15 ] , robots navigation [ 16 ] , and resource management in networking [ 17 ] . Despite its promising applications , FRL is faced by a number of major challenges , which existing works are unable to tackle . Firstly , existing FRL frameworks are not equipped with theoretical convergence guarantee , and thus lack an assurance for the sample efficiency of practical FRL applications , which is a critical drawback 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . due to the high sampling cost of RL trajectories in real systems [ 4 ] . Unlike FL where training data can be collected offline , FRL requires every agent to sample trajectories by interacting with the environment during learning . However , interacting with real systems can be slow , expensive , or fragile . This makes it critical for FRL to be sample-efficient and hence highlights the requirement for convergence guarantee of FRL , without which no assurance on its sample efficiency is provided for practical applications . To fill this gap , we establish on recent endeavors in stochastic variance-reduced optimization techniques to develop a variance-reduced federated policy gradient framework , the convergence of which is guaranteed . We prove that the proposed framework enjoys a sample complexity of O ( 1/ 5/3 ) to converge to an -stationary point in the single-agent setting , which matches recent results of variance-reduced policy gradient [ 18 , 19 ] . More importantly , the aforementioned sample complexity is guaranteed to improve at a rate of O ( 1/K2/3 ) upon the federation of K agents . This guarantees that an agent achieves a better sample efficiency by joining the federation and benefits from more participating agents , which are highly desirable in FRL . Another challenge inherited from FL is that FRL is vulnerable to random failures or adversarial attacks , which poses threats to many real-world RL systems . For example , robots may behave arbitrarily due to random hardware issues ; clinical data may provide inaccurate records and hence create misleading trajectories [ 3 ] ; autonomous vehicles , on which RL is commonly deployed , are subject to adversarial attacks [ 20 ] . As we will show in experiments , including such random failures or adversary agents in FRL can significantly deteriorate its convergence or even result in unlearnability . Of note , random failures and adversarial attacks in FL systems are being encompassed by the Byzantine failure model [ 21 ] , which is considered as the most stringent fault formalism in distributed computing [ 22 , 23 ] – a small fraction of agents may behave arbitrarily and possibly adversarially , with the goal of breaking or at least slowing down the convergence of the system . As algorithms proven to be correct in this setting are guaranteed to converge under arbitrary system behavior ( e.g. , exercising failures or being attacked ) [ 9 , 24 ] , we study the fault tolerance of our proposed FRL framework using the Byzantine failure model . We design a gradient-based Byzantine filter on top of the variance-reduced federated policy gradient framework . We show that , when a certain percentage ( denoted by α < 0.5 ) of agents are Byzantine agents , the sample complexity of the FRL system is worsened by only an additive term of O ( α4/3/ 5/3 ) ( Section 4 ) . Therefore , when α→ 0 , ( i.e. , an ideal system with zero chance of failure ) , the filter induces no impact on the convergence . Contributions . In this paper , we study the federated reinforcement learning problem with theoretical guarantee in the potential presence of faulty agents . We introduce Federated Policy Gradient with Byzantine Resilience ( FedPG-BR ) , the first FRL framework that is theoretically principled and practically effective for the FRL setting , accounting for random systematic failures and adversarial attacks . In particular , FedPG-BR ( a ) enjoys a guaranteed sample complexity which improves with more participating agents , and ( b ) is tolerant to the Byzantine fault in both theory and practice . We discuss the details of problem setting and the technical challenges ( Section 3 ) and provide theoretical analysis of FedPG-BR ( Section 4 ) . We also demonstrate its empirical efficacy on various RL benchmark tasks ( Section 5 ) . 2 Background . Stochastic Variance-Reduced Gradient aims to solve minθ∈Rd [ J ( θ ) , 1B ∑B i=1 Ji ( θ ) ] . Under the common assumption of all function components Ji being smooth and convex in θ , gradient descent ( GD ) achieves linear convergence in the number of iterations of parameter updates [ 25 , 26 ] . However , every iteration of GD requires B gradient computations , which can be expensive for large B . To overcome this problem , stochastic GD ( SGD ) [ 27 , 28 ] samples a single data point per iteration , which incurs lower per-iteration cost yet results in a sub-linear convergence rate [ 29 ] . For a better trade-off between convergence rate and per-iteration computational cost , the stochastic variancereduced gradient ( SVRG ) method has been proposed , which reuses past gradient computations to reduce the variance of the current gradient estimate [ 30–33 ] . More recently , stochastically controlled stochastic gradient ( SCSG ) has been proposed for convex [ 34 ] or smooth non-convex objective function [ 35 ] , to further reduce the computational cost of SVRG especially when required is small in finding -approximate solution . Refer to Appendix A.1 for more details on SVRG and SCSG . Reinforcement Learning ( RL ) can be modelled as a discrete-time Markov Decision Process ( MDP ) [ 36 ] : M , { S , A , P , R , γ , ρ } . S represents the state space , A is the action space , P ( s′|s , a ) defines the transition probability from state s to s′ after taking action a , R ( s , a ) : S × A 7→ [ 0 , R ] is the reward function for state-action pair ( s , a ) and some constant R > 0 , γ ∈ ( 0 , 1 ) is the discount factor , and ρ is the initial state distribution . An agent ’ s behavior is controlled by a policy π , where π ( a|s ) defines the probability that the agent chooses action a at state s. We consider episodic MDPs with trajectory horizon H . A trajectory τ , { s0 , a0 , s1 , a1 , ... , sH−1 , aH−1 } is a sequence of state-action pairs traversed by an agent following any stationary policy , where s0 ∼ ρ. R ( τ ) , ∑H−1 t=0 γ tR ( st , at ) gives the cumulative discounted reward for a trajectory τ . Policy Gradient ( PG ) methods have achieved impressive successes in model-free RL [ 37 , 38 , , etc. ] . Compared with deterministic value-function based methods such as Q-learning , PG methods are generally more effective in high-dimensional problems and enjoy the flexibility of stochasticity . In PG , we use πθ to denote the policy parameterized by θ ∈ Rd ( e.g. , a neural network ) , and p ( τ |πθ ) to represent the trajectory distribution induced by policy πθ . For brevity , we use θ to denote the corresponding policy πθ . The performance of a policy θ can be measured by J ( θ ) , Eτ∼p ( ·|θ ) [ R ( τ ) |M ] . Taking the gradient of J ( θ ) with respect to θ gives ∇θJ ( θ ) = ∫ τ R ( τ ) ∇θp ( τ | θ ) dτ = Eτ∼p ( ·|θ ) [ ∇θ log p ( τ | θ ) R ( τ ) |M ] ( 1 ) Then , the policy θ can be optimized by gradient ascent . Since computing ( 1 ) is usually prohibitive , stochastic gradient ascent is typically used . In each iteration , we sample a batch of trajectories { τi } Bi=1 using the current policy θ , and update the policy by θ ← θ + η∇̂BJ ( θ ) , where η is the step size and ∇̂BJ ( θ ) is an estimate of ( 1 ) using the sampled trajectories { τi } Bi=1 : ∇̂BJ ( θ ) = 1B ∑B i=1∇θ log p ( τi | θ ) R ( τi ) . The most common policy gradient estimators , such as REINFORCE [ 39 ] and GPOMDP [ 40 ] , can be expressed as ∇̂BJ ( θ ) = 1 B B∑ i=1 g ( τi|θ ) ( 2 ) where τi = { si0 , ai0 , si1 , ai1 , . . . , siH−1 , aiH−1 } and g ( τi|θ ) is an unbiased estimate of ∇θ log p ( τi | θ ) R ( τi ) . We provide formal definition of g ( τi|θ ) in Appendix A.2 . SVRPG . A key issue for PG is the high variance of the estimator based on stochastic gradients ( 2 ) which results in slow convergence . Similar to SGD for finite-sum optimization , PG requires O ( 1/ 2 ) trajectories to find an -stationary point such that E [ ‖∇J ( θ ) ‖2 ] ≤ [ 19 ] . That is , PG typically requires a large number of trajectories to find a well-performing policy . To reduce the variance of the gradient estimator in PG ( 2 ) , SVRG has been applied to policy evaluation [ 41 , 42 ] and policy optimization [ 43 ] . The work of Papini et al . [ 18 ] has adapted the theoretical analysis of SVRG to PG to introduce the stochastic variance-reduced PG ( SVRPG ) algorithm . More recently , Xu et al . [ 19 ] has refined the analysis of SVRPG [ 18 ] and shown that SVRPG enjoys a sample complexity of O ( 1/ 5/3 ) . These works have demonstrated both theoretically and empirically that SVRG is a promising approach to reduce the variance and thus improve the sample efficiency of PG methods . Fault tolerance refers to the property that enables a computing system to continue operating properly without interruption when one or more of its workers fail . Among the many fault formalisms , the Byzantine failure model has a rich history in distributed computing [ 22 , 23 ] and is considered as the most stringent fault formalism in fault-tolerant FL system design [ 9 , 24 ] . Originated from the Byzantine generals problem [ 21 ] , the Byzantine failure model allows an α-fraction ( typical α < 0.5 ) of workers to behave arbitrarily and possibly adversarially , with the goal of breaking or at least slowing down the convergence of the algorithm . As algorithms proven to be resilient to the Byzantine failures are guaranteed to converge under arbitrary system behavior ( hence fault-tolerant ) [ 22 , 23 ] , it has motivated a significant interest in providing distributed supervised learning with Byzantine resilience guarantees [ e.g. , 44–49 ] . However , there is yet no existing work studying the correctness of Byzantine resilience in the context of FRL .

Summary: This paper proposes a federated reinforcement learning (FRL) algorithm that uses a variance-reduced federated policy gradient framework. The proposed framework is proven to have convergence and robustness guatantees (can tolerant the failures of up to half of the agents), with some assumptions on the environment and the agents. Both theoretical results and empirical results justify that the proposed federation can boost the sample efficiency of RL agents.