Fault-Tolerant Federated Reinforcement Learning with Theoretical Guarantee

1 Introduction . Reinforcement learning ( RL ) has recently been applied to many real-world decision-making problems such as gaming , robotics , healthcare , etc . [ 1–3 ] . However , despite its impressive performances in simulation , RL often suffers from poor sample efficiency , which hinders its success in real-world applications [ 4 , 5 ] . For example , when RL is applied to provide clinical decision support [ 3 , 6 , 7 ] , its performance is limited by the number ( i.e. , sample size ) of admission records possessed by a hospital , which can not be synthetically generated [ 3 ] . As this challenge is usually faced by many agents ( e.g. , different hospitals ) , a natural solution is to encourage multiple RL agents to share their trajectories , to collectively build a better decision-making policy that one single agent can not obtain by itself . However , in many applications , raw RL trajectories contain sensitive information ( e.g. , the medical records contain sensitive information about patients ) and thus sharing them is prohibited . To this end , the recent success of Federated Learning ( FL ) [ 8–11 ] has inspired the setting of Federated Reinforcement Learning ( FRL ) [ 12 ] , which aims to federatively build a better policy from multiple RL agents without requiring them to share their raw trajectories . FRL is practically appealing for addressing the sample inefficiency of RL in real systems , such as autonomous driving [ 13 ] , fast personalization [ 14 ] , optimal control of IoT devices [ 15 ] , robots navigation [ 16 ] , and resource management in networking [ 17 ] . Despite its promising applications , FRL is faced by a number of major challenges , which existing works are unable to tackle . Firstly , existing FRL frameworks are not equipped with theoretical convergence guarantee , and thus lack an assurance for the sample efficiency of practical FRL applications , which is a critical drawback 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . due to the high sampling cost of RL trajectories in real systems [ 4 ] . Unlike FL where training data can be collected offline , FRL requires every agent to sample trajectories by interacting with the environment during learning . However , interacting with real systems can be slow , expensive , or fragile . This makes it critical for FRL to be sample-efficient and hence highlights the requirement for convergence guarantee of FRL , without which no assurance on its sample efficiency is provided for practical applications . To fill this gap , we establish on recent endeavors in stochastic variance-reduced optimization techniques to develop a variance-reduced federated policy gradient framework , the convergence of which is guaranteed . We prove that the proposed framework enjoys a sample complexity of O ( 1/ 5/3 ) to converge to an -stationary point in the single-agent setting , which matches recent results of variance-reduced policy gradient [ 18 , 19 ] . More importantly , the aforementioned sample complexity is guaranteed to improve at a rate of O ( 1/K2/3 ) upon the federation of K agents . This guarantees that an agent achieves a better sample efficiency by joining the federation and benefits from more participating agents , which are highly desirable in FRL . Another challenge inherited from FL is that FRL is vulnerable to random failures or adversarial attacks , which poses threats to many real-world RL systems . For example , robots may behave arbitrarily due to random hardware issues ; clinical data may provide inaccurate records and hence create misleading trajectories [ 3 ] ; autonomous vehicles , on which RL is commonly deployed , are subject to adversarial attacks [ 20 ] . As we will show in experiments , including such random failures or adversary agents in FRL can significantly deteriorate its convergence or even result in unlearnability . Of note , random failures and adversarial attacks in FL systems are being encompassed by the Byzantine failure model [ 21 ] , which is considered as the most stringent fault formalism in distributed computing [ 22 , 23 ] – a small fraction of agents may behave arbitrarily and possibly adversarially , with the goal of breaking or at least slowing down the convergence of the system . As algorithms proven to be correct in this setting are guaranteed to converge under arbitrary system behavior ( e.g. , exercising failures or being attacked ) [ 9 , 24 ] , we study the fault tolerance of our proposed FRL framework using the Byzantine failure model . We design a gradient-based Byzantine filter on top of the variance-reduced federated policy gradient framework . We show that , when a certain percentage ( denoted by α < 0.5 ) of agents are Byzantine agents , the sample complexity of the FRL system is worsened by only an additive term of O ( α4/3/ 5/3 ) ( Section 4 ) . Therefore , when α→ 0 , ( i.e. , an ideal system with zero chance of failure ) , the filter induces no impact on the convergence . Contributions . In this paper , we study the federated reinforcement learning problem with theoretical guarantee in the potential presence of faulty agents . We introduce Federated Policy Gradient with Byzantine Resilience ( FedPG-BR ) , the first FRL framework that is theoretically principled and practically effective for the FRL setting , accounting for random systematic failures and adversarial attacks . In particular , FedPG-BR ( a ) enjoys a guaranteed sample complexity which improves with more participating agents , and ( b ) is tolerant to the Byzantine fault in both theory and practice . We discuss the details of problem setting and the technical challenges ( Section 3 ) and provide theoretical analysis of FedPG-BR ( Section 4 ) . We also demonstrate its empirical efficacy on various RL benchmark tasks ( Section 5 ) . 2 Background . Stochastic Variance-Reduced Gradient aims to solve minθ∈Rd [ J ( θ ) , 1B ∑B i=1 Ji ( θ ) ] . Under the common assumption of all function components Ji being smooth and convex in θ , gradient descent ( GD ) achieves linear convergence in the number of iterations of parameter updates [ 25 , 26 ] . However , every iteration of GD requires B gradient computations , which can be expensive for large B . To overcome this problem , stochastic GD ( SGD ) [ 27 , 28 ] samples a single data point per iteration , which incurs lower per-iteration cost yet results in a sub-linear convergence rate [ 29 ] . For a better trade-off between convergence rate and per-iteration computational cost , the stochastic variancereduced gradient ( SVRG ) method has been proposed , which reuses past gradient computations to reduce the variance of the current gradient estimate [ 30–33 ] . More recently , stochastically controlled stochastic gradient ( SCSG ) has been proposed for convex [ 34 ] or smooth non-convex objective function [ 35 ] , to further reduce the computational cost of SVRG especially when required is small in finding -approximate solution . Refer to Appendix A.1 for more details on SVRG and SCSG . Reinforcement Learning ( RL ) can be modelled as a discrete-time Markov Decision Process ( MDP ) [ 36 ] : M , { S , A , P , R , γ , ρ } . S represents the state space , A is the action space , P ( s′|s , a ) defines the transition probability from state s to s′ after taking action a , R ( s , a ) : S × A 7→ [ 0 , R ] is the reward function for state-action pair ( s , a ) and some constant R > 0 , γ ∈ ( 0 , 1 ) is the discount factor , and ρ is the initial state distribution . An agent ’ s behavior is controlled by a policy π , where π ( a|s ) defines the probability that the agent chooses action a at state s. We consider episodic MDPs with trajectory horizon H . A trajectory τ , { s0 , a0 , s1 , a1 , ... , sH−1 , aH−1 } is a sequence of state-action pairs traversed by an agent following any stationary policy , where s0 ∼ ρ. R ( τ ) , ∑H−1 t=0 γ tR ( st , at ) gives the cumulative discounted reward for a trajectory τ . Policy Gradient ( PG ) methods have achieved impressive successes in model-free RL [ 37 , 38 , , etc. ] . Compared with deterministic value-function based methods such as Q-learning , PG methods are generally more effective in high-dimensional problems and enjoy the flexibility of stochasticity . In PG , we use πθ to denote the policy parameterized by θ ∈ Rd ( e.g. , a neural network ) , and p ( τ |πθ ) to represent the trajectory distribution induced by policy πθ . For brevity , we use θ to denote the corresponding policy πθ . The performance of a policy θ can be measured by J ( θ ) , Eτ∼p ( ·|θ ) [ R ( τ ) |M ] . Taking the gradient of J ( θ ) with respect to θ gives ∇θJ ( θ ) = ∫ τ R ( τ ) ∇θp ( τ | θ ) dτ = Eτ∼p ( ·|θ ) [ ∇θ log p ( τ | θ ) R ( τ ) |M ] ( 1 ) Then , the policy θ can be optimized by gradient ascent . Since computing ( 1 ) is usually prohibitive , stochastic gradient ascent is typically used . In each iteration , we sample a batch of trajectories { τi } Bi=1 using the current policy θ , and update the policy by θ ← θ + η∇̂BJ ( θ ) , where η is the step size and ∇̂BJ ( θ ) is an estimate of ( 1 ) using the sampled trajectories { τi } Bi=1 : ∇̂BJ ( θ ) = 1B ∑B i=1∇θ log p ( τi | θ ) R ( τi ) . The most common policy gradient estimators , such as REINFORCE [ 39 ] and GPOMDP [ 40 ] , can be expressed as ∇̂BJ ( θ ) = 1 B B∑ i=1 g ( τi|θ ) ( 2 ) where τi = { si0 , ai0 , si1 , ai1 , . . . , siH−1 , aiH−1 } and g ( τi|θ ) is an unbiased estimate of ∇θ log p ( τi | θ ) R ( τi ) . We provide formal definition of g ( τi|θ ) in Appendix A.2 . SVRPG . A key issue for PG is the high variance of the estimator based on stochastic gradients ( 2 ) which results in slow convergence . Similar to SGD for finite-sum optimization , PG requires O ( 1/ 2 ) trajectories to find an -stationary point such that E [ ‖∇J ( θ ) ‖2 ] ≤ [ 19 ] . That is , PG typically requires a large number of trajectories to find a well-performing policy . To reduce the variance of the gradient estimator in PG ( 2 ) , SVRG has been applied to policy evaluation [ 41 , 42 ] and policy optimization [ 43 ] . The work of Papini et al . [ 18 ] has adapted the theoretical analysis of SVRG to PG to introduce the stochastic variance-reduced PG ( SVRPG ) algorithm . More recently , Xu et al . [ 19 ] has refined the analysis of SVRPG [ 18 ] and shown that SVRPG enjoys a sample complexity of O ( 1/ 5/3 ) . These works have demonstrated both theoretically and empirically that SVRG is a promising approach to reduce the variance and thus improve the sample efficiency of PG methods . Fault tolerance refers to the property that enables a computing system to continue operating properly without interruption when one or more of its workers fail . Among the many fault formalisms , the Byzantine failure model has a rich history in distributed computing [ 22 , 23 ] and is considered as the most stringent fault formalism in fault-tolerant FL system design [ 9 , 24 ] . Originated from the Byzantine generals problem [ 21 ] , the Byzantine failure model allows an α-fraction ( typical α < 0.5 ) of workers to behave arbitrarily and possibly adversarially , with the goal of breaking or at least slowing down the convergence of the algorithm . As algorithms proven to be resilient to the Byzantine failures are guaranteed to converge under arbitrary system behavior ( hence fault-tolerant ) [ 22 , 23 ] , it has motivated a significant interest in providing distributed supervised learning with Byzantine resilience guarantees [ e.g. , 44–49 ] . However , there is yet no existing work studying the correctness of Byzantine resilience in the context of FRL .

Summary: The authors propose a federated reinforcement learning approach that combines existing stochastic variance-reduced policy gradient method with a robust aggregation method to achieve fault tolerance. Theoretically, the authors provide convergence and sample complexity results under reasonable assumptions. The proposed approach is tested empirically on various benchmark tasks.