Sample and Communication-Efficient Decentralized Actor-Critic Algorithms with Finite-Time Analysis

1 INTRODUCTION . Multi-agent reinforcement learning ( MARL ) has achieved great success in various application domains , including control ( 66 ; 10 ; 51 ) , robotics ( 64 ) , wireless sensor networks ( 24 ; 67 ) , intelligent systems ( 71 ) , etc . In MARL , a set of fully decentralized agents interact with a dynamic environment following their own policies and collect local rewards , and their goal is to collaboratively learn the optimal joint policy that achieves the maximum expected accumulated reward . Classical policy optimization algorithms have been well developed and studied , e.g. , policy gradient ( PG ) ( 49 ) , actor-critic ( AC ) ( 23 ) and natural actor-critic ( NAC ) ( 37 ; 7 ) . In particular , AC-type algorithms are more computationally tractable and efficient as they take advantages of both policy gradient and value-based updates . However , in the multi-agent setting , decentralized AC is more challenging to design compared with the centralized AC , as the algorithm updates involve sensitive agent information , e.g. , local actions , rewards and policies , which must be kept locally in the decentralized learning process . In the existing designs of decentralized AC , the agents need to share either their local actions ( 70 ; 69 ; 8 ; 36 ; 72 ; 27 ; 19 ; 26 ; 11 ) or local rewards ( 15 ; 33 ; 32 ) with their neighbors , and hence are not desired . This issue is addressed by Algorithm 2 of ( 70 ) at the cost of learning a parameterized model to estimate the averaged reward , yet this approach requires extra learning effort and the reward estimation can be inaccurate . Moreover , existing decentralized AC algorithms are not sample and communication-efficient , and do not have finite-time convergence guarantee , especially under the practical Markovian sampling setting . Therefore , we aim to address the following important question . • Q1 : Can we develop a decentralized AC algorithm that is convergent , sample and communicationefficient , and does not require sharing agents ’ local actions and policies ? On the other hand , as an important variant of the decentralized AC , decentralized NAC algorithm has not been formally developed and rigorously analyzed in the existing literature . In particular , a major challenge is that we need to develop a fully decentralized and computationally tractable scheme to compute the inverse of the high dimensional Fisher information matrix , and this scheme must be both sample and communication efficient . Hence , we want to ask : • Q2 : Can we develop a computationally tractable and communication-efficient decentralized NAC algorithm that has a low finite-time sample and communication complexity ? In this study , we provide affirmative answers to the above two questions by developing fully decentralized AC and NAC algorithms that are sample and communication-efficient , and do not reveal agents ’ local actions and policies . We also develop rigorous finite-time analysis of these algorithms under Markovian sampling . Our contributions are summarized as follows . 1.1 OUR CONTRIBUTIONS . We develop fully decentralized AC and NAC algorithms and analyze their finite-time sample and communication complexities under Markovian sampling . Our results and comparisons to existing works are summarized in Table 1 1 . Our decentralized AC and NAC algorithms adopt the following novel designs to accurately estimate the policy gradient in an efficient way . • Noisy Rewards : In a decentralized setting , local policy gradients ( estimated locally by the agents ) involve the average of all agents ’ local rewards . To help agents estimate this averaged reward without revealing the raw local rewards , we let them share Gaussian-corrupted local rewards with their neighbor , and the variance of the Gaussian noise can be adjusted by each agent to reach its desired level . • Mini-batch Updates : We apply mini-batch Markovian sampling to both the decentralized actor and critic updates . This approach 1 ) helps the agents obtain accurate estimations of the corrupted averaged reward ; 2 ) significantly reduces the variance of policy gradient caused by Markovian sampling ; and 3 ) significantly reduces the communication frequency and complexity . Moreover , for our decentralized NAC algorithm , we additionally adopt the following design to compute the inverse of the Fisher information matrix in an efficient and decentralized way . • Decentralized Natural Policy Gradient : By reformulating the natural policy gradient as the minimizer of a quadratic program , we develop a decentralized SGD with Markovian sampling that allows the agents to estimate the corresponding local natural gradients by communicating only 1In this table , Õ ( · ) hides all logarithm factors . In ( 25 ) , the sample complexity has been established for various AC-type algorithms , and we compare with the best one . In ( 70 ) , the Algorithm 1 needs to share local actions while the Algorithm 2 does not . scalar variables with their neighbors . In particular , in order to minimize the sample complexity of the decentralized SGD , we set the batch size to be exponentially increasing . Theoretically , for the first time , we provide finite-time convergence analysis of decentralized AC and NAC algorithms under Markovian sampling . Specifically , we prove that our decentralized AC and NAC algorithms achieve the overall sample complexities O ( −2 ln −1 ) and O ( −3 ln −1 ) , respectively , and both match the state-of-the-art complexities of their centralized versions ( 60 ) . Moreover , both decentralized algorithms achieve a significantly reduced overall communication complexity O ( −1 ln −1 ) . In particular , our analysis involves new technical developments . First , we need to characterize the bias and variance of ( natural ) policy gradient and stochastic gradient caused by the noisy rewards and the inexact local averaging steps , and control them with proper choices of batch sizes and number of local averaging steps . Second , when using decentralized Markovian SGD to compute the inverse Fisher information matrix , we need to use an exponentially increasing batch size to achieve an optimized sample complexity bound . Such a Markovian SGD with adaptive batch size has not been studied before and can be of independent interest . 1.2 RELATED WORK . Convergence analysis of AC and NAC . In the centralized setting , the AC algorithm was firstly proposed by ( 23 ) and later developed into the natural actor-critic ( NAC ) algorithm ( 37 ; 7 ) . Specifically , ( 37 ) does not provide any convergence result , while ( 22 ; 5 ) and ( 20 ; 6 ; 7 ) establish the asymptotic convergence rate of centralized AC and NAC , respectively , which is weaker than our finite-time convergence results . Furthermore , ( 54 ; 25 ; 38 ; 61 ; 56 ) and ( 54 ) establish the finite-time convergence rate of centralized AC and NAC , respectively . Please refer to Table 1 for their sample complexities . Moreover , ( 60 ) improve the finite-time sample complexities of the above works to the state-of-theart result for both centralized AC and NAC by leveraging mini batch sampling , and our sample complexities match these state-of-the-art results . In the decentralized setting , a few works have established the almost sure convergence result of AC ( 15 ; 27 ; 47 ; 33 ) , but they do not characterize the finite-time convergence rate and the sample complexity . To the best of our knowledge , there is no formally developed decentralized NAC algorithm . Decentralized TD-type algorithms . The finite-time convergence of decentralized TD ( 0 ) has been obtained using i.i.d samples ( 52 ; 14 ; 53 ; 28 ) and Markovian samples ( 46 ; 53 ) , respectively , without revealing the agents ’ local actions , policies and rewards . Decentralized off-policy TD-type algorithms have been studied in ( 34 ; 45 ; 9 ; 12 ) . Decentralized AC in other MARL settings . Some works apply decentralized AC to other MARL settings that are very different from ours . For example , ( 44 ; 36 ; 17 ; 11 ; 57 ) studied adversarial game . ( 30 ) studied a mixed cooperative-competitive environment where each agent maximizes its own Q function ( 30 ) . ( 11 ) proposed Delay-Aware Markov Game which considers delay in Markov game . ( 68 ; 31 ) studied linear control system and linear quadratic regulators instead of an MDP . ( 55 ) studied sequential prisoner ’ s dilemmas . Policy gradient algorithms . Policy gradient ( PG ) and natural policy gradient ( NPG ) are popular policy optimization algorithms . ( 1 ) characterizes the iteration complexity ( i.e. , number of episodes ) of centralized PG and NPG algorithms by assuming access to exact policy gradient . They also established a sample complexity result O ( −6 ) in the i.i.d . setting for NPG , which is worse than the state-of-the-art result O ( −3 ln −1 ) of both centralized NAC ( 60 ) and our decentralized NAC with Markovian samples . ( 3 ) proposes decentralized PG in a simple cooperative MARL setting , where all the agents share one action and the same policy , and they establish a iteration complexity in the order of O ( −4 ) . ( 13 ; 73 ) apply decentralized PG to Markov games . ( 2 ) applies decentralized NPG to a different cooperative MARL setting where each agent observes its own state , takes its own action and has access to these information of its neighbors . Value-based algorithms . Value-based algorithms have also been develop for MARL . Specifically , ( 21 ; 18 ) develop distributed Q-learning in a simplified cooperative MARL setting , where the agents share a joint action . In particular , ( 18 ) characterizes the convergence rate of a value function-based convergence error , which is a different optimality measure from that of AC-type algorithms . ( 35 ) applies distributed Q-learning to another cooperative MARL setting , where each agent observes its own state and takes its own action . It establishes an asymptotic convergence guarantee , and no convergence rate is given . ( 40 ) develops a value propagation algorithm that uses primal-dual method to minimize a soft Bellman error in the MARL setting . Under an assumption that the variance of the stochastic gradient is uniformly bounded , it establishes a non-asymptotic convergence rate to an approximate stationary point . 2 REVIEW OF MULTI-AGENT REINFORCEMENT LEARNING . In this section , we first introduce some standard settings of RL . Consider an agent that starts from an initial state s0 ∼ ξ and collects a trajectory of Markovian samples { st , at , Rt } t ⊂ S × A × R by interacting with an underlying environment ( with transition kernel P ) following a parameterized policy πω with induced stationary state distribution µω . The agent aims to learn an optimal policy that maximizes the expected accumulated reward J ( ω ) = ( 1− γ ) E [ ∑∞ t=0 γ tRt ] , where γ ∈ ( 0 , 1 ) is a discount factor . The marginal state distribution is denoted as Pω ( st ) and the visitation measure is defined as νω ( s ) : = ( 1− γ ) ∑∞ t=0 γ tPω ( st = s ) , both of which depend on the policy parameter ω ∈ Ω and the transition kernel P . We also define the mixed transition kernel Pξ ( ·|s , a ) : = γP ( ·|s , a ) + ( 1− γ ) ξ ( · ) , whose stationary state distribution is known to be νω . In the multi-agent RL ( MARL ) setting , M agents are connected via a fully decentralized network and interact with a shared environment . The network topology is specified by a doubly stochastic communication matrix W ∈ RM×M . At any time t , all the agents share a common state st. Then , every agent m takes an action a ( m ) t following its own current policy π ( m ) t ( ·|st ) parameterized by ω ( m ) t . After all the actions at : = { a ( m ) t } Mm=1 are taken , the global state st transfers to a new state st+1 and every agent m receives a local reward R ( m ) t . In this MARL setting , each agent m can only access the global state { st } t , its own actions { a ( m ) t } t and rewards { R ( m ) t } t and policy π ( m ) t . Next , define the joint policy πt ( at|st ) : = ∏M m=1 π ( m ) t ( a ( m ) t |st ) parameterized by ωt = [ ω ( 1 ) t ; . . . ; ω ( M ) t ] , and define the average reward Rt : = 1M ∑M m=1R ( m ) t . The goal of the agents is to collaboratively learn the optimal joint policy that maximizes the expected accumulated average reward J ( ω ) : = ( 1− γ ) E [ ∑∞ t=0 γ tRt ∣∣∣s0 ∼ ξ ] . Throughout , we consider the setting that the agents keep interacting with the environment and observing a trajectory of MDP transition samples , which are further used to learn the optimal joint policy .

Summary: In this work, the authors propose two decentralized policy gradient-type algorithms for multi-agent reinforcement learning, namely, a decentralized actor-critic algorithm and a decentralized natural actor-critic algorithm. The stochastic updates of both algorithms preserve the agents' privacy information, including their local actions and local rewards. The authors analyze the finite-time convergence rate of both algorithms under Markovian sampling and linear function approximation, and prove that both algorithms achieve the state-of-the-art sample complexities and improved communication complexities.