Sample and Communication-Efficient Decentralized Actor-Critic Algorithms with Finite-Time Analysis

1 INTRODUCTION . Multi-agent reinforcement learning ( MARL ) has achieved great success in various application domains , including control ( 66 ; 10 ; 51 ) , robotics ( 64 ) , wireless sensor networks ( 24 ; 67 ) , intelligent systems ( 71 ) , etc . In MARL , a set of fully decentralized agents interact with a dynamic environment following their own policies and collect local rewards , and their goal is to collaboratively learn the optimal joint policy that achieves the maximum expected accumulated reward . Classical policy optimization algorithms have been well developed and studied , e.g. , policy gradient ( PG ) ( 49 ) , actor-critic ( AC ) ( 23 ) and natural actor-critic ( NAC ) ( 37 ; 7 ) . In particular , AC-type algorithms are more computationally tractable and efficient as they take advantages of both policy gradient and value-based updates . However , in the multi-agent setting , decentralized AC is more challenging to design compared with the centralized AC , as the algorithm updates involve sensitive agent information , e.g. , local actions , rewards and policies , which must be kept locally in the decentralized learning process . In the existing designs of decentralized AC , the agents need to share either their local actions ( 70 ; 69 ; 8 ; 36 ; 72 ; 27 ; 19 ; 26 ; 11 ) or local rewards ( 15 ; 33 ; 32 ) with their neighbors , and hence are not desired . This issue is addressed by Algorithm 2 of ( 70 ) at the cost of learning a parameterized model to estimate the averaged reward , yet this approach requires extra learning effort and the reward estimation can be inaccurate . Moreover , existing decentralized AC algorithms are not sample and communication-efficient , and do not have finite-time convergence guarantee , especially under the practical Markovian sampling setting . Therefore , we aim to address the following important question . • Q1 : Can we develop a decentralized AC algorithm that is convergent , sample and communicationefficient , and does not require sharing agents ’ local actions and policies ? On the other hand , as an important variant of the decentralized AC , decentralized NAC algorithm has not been formally developed and rigorously analyzed in the existing literature . In particular , a major challenge is that we need to develop a fully decentralized and computationally tractable scheme to compute the inverse of the high dimensional Fisher information matrix , and this scheme must be both sample and communication efficient . Hence , we want to ask : • Q2 : Can we develop a computationally tractable and communication-efficient decentralized NAC algorithm that has a low finite-time sample and communication complexity ? In this study , we provide affirmative answers to the above two questions by developing fully decentralized AC and NAC algorithms that are sample and communication-efficient , and do not reveal agents ’ local actions and policies . We also develop rigorous finite-time analysis of these algorithms under Markovian sampling . Our contributions are summarized as follows . 1.1 OUR CONTRIBUTIONS . We develop fully decentralized AC and NAC algorithms and analyze their finite-time sample and communication complexities under Markovian sampling . Our results and comparisons to existing works are summarized in Table 1 1 . Our decentralized AC and NAC algorithms adopt the following novel designs to accurately estimate the policy gradient in an efficient way . • Noisy Rewards : In a decentralized setting , local policy gradients ( estimated locally by the agents ) involve the average of all agents ’ local rewards . To help agents estimate this averaged reward without revealing the raw local rewards , we let them share Gaussian-corrupted local rewards with their neighbor , and the variance of the Gaussian noise can be adjusted by each agent to reach its desired level . • Mini-batch Updates : We apply mini-batch Markovian sampling to both the decentralized actor and critic updates . This approach 1 ) helps the agents obtain accurate estimations of the corrupted averaged reward ; 2 ) significantly reduces the variance of policy gradient caused by Markovian sampling ; and 3 ) significantly reduces the communication frequency and complexity . Moreover , for our decentralized NAC algorithm , we additionally adopt the following design to compute the inverse of the Fisher information matrix in an efficient and decentralized way . • Decentralized Natural Policy Gradient : By reformulating the natural policy gradient as the minimizer of a quadratic program , we develop a decentralized SGD with Markovian sampling that allows the agents to estimate the corresponding local natural gradients by communicating only 1In this table , Õ ( · ) hides all logarithm factors . In ( 25 ) , the sample complexity has been established for various AC-type algorithms , and we compare with the best one . In ( 70 ) , the Algorithm 1 needs to share local actions while the Algorithm 2 does not . scalar variables with their neighbors . In particular , in order to minimize the sample complexity of the decentralized SGD , we set the batch size to be exponentially increasing . Theoretically , for the first time , we provide finite-time convergence analysis of decentralized AC and NAC algorithms under Markovian sampling . Specifically , we prove that our decentralized AC and NAC algorithms achieve the overall sample complexities O ( −2 ln −1 ) and O ( −3 ln −1 ) , respectively , and both match the state-of-the-art complexities of their centralized versions ( 60 ) . Moreover , both decentralized algorithms achieve a significantly reduced overall communication complexity O ( −1 ln −1 ) . In particular , our analysis involves new technical developments . First , we need to characterize the bias and variance of ( natural ) policy gradient and stochastic gradient caused by the noisy rewards and the inexact local averaging steps , and control them with proper choices of batch sizes and number of local averaging steps . Second , when using decentralized Markovian SGD to compute the inverse Fisher information matrix , we need to use an exponentially increasing batch size to achieve an optimized sample complexity bound . Such a Markovian SGD with adaptive batch size has not been studied before and can be of independent interest . 1.2 RELATED WORK . Convergence analysis of AC and NAC . In the centralized setting , the AC algorithm was firstly proposed by ( 23 ) and later developed into the natural actor-critic ( NAC ) algorithm ( 37 ; 7 ) . Specifically , ( 37 ) does not provide any convergence result , while ( 22 ; 5 ) and ( 20 ; 6 ; 7 ) establish the asymptotic convergence rate of centralized AC and NAC , respectively , which is weaker than our finite-time convergence results . Furthermore , ( 54 ; 25 ; 38 ; 61 ; 56 ) and ( 54 ) establish the finite-time convergence rate of centralized AC and NAC , respectively . Please refer to Table 1 for their sample complexities . Moreover , ( 60 ) improve the finite-time sample complexities of the above works to the state-of-theart result for both centralized AC and NAC by leveraging mini batch sampling , and our sample complexities match these state-of-the-art results . In the decentralized setting , a few works have established the almost sure convergence result of AC ( 15 ; 27 ; 47 ; 33 ) , but they do not characterize the finite-time convergence rate and the sample complexity . To the best of our knowledge , there is no formally developed decentralized NAC algorithm . Decentralized TD-type algorithms . The finite-time convergence of decentralized TD ( 0 ) has been obtained using i.i.d samples ( 52 ; 14 ; 53 ; 28 ) and Markovian samples ( 46 ; 53 ) , respectively , without revealing the agents ’ local actions , policies and rewards . Decentralized off-policy TD-type algorithms have been studied in ( 34 ; 45 ; 9 ; 12 ) . Decentralized AC in other MARL settings . Some works apply decentralized AC to other MARL settings that are very different from ours . For example , ( 44 ; 36 ; 17 ; 11 ; 57 ) studied adversarial game . ( 30 ) studied a mixed cooperative-competitive environment where each agent maximizes its own Q function ( 30 ) . ( 11 ) proposed Delay-Aware Markov Game which considers delay in Markov game . ( 68 ; 31 ) studied linear control system and linear quadratic regulators instead of an MDP . ( 55 ) studied sequential prisoner ’ s dilemmas . Policy gradient algorithms . Policy gradient ( PG ) and natural policy gradient ( NPG ) are popular policy optimization algorithms . ( 1 ) characterizes the iteration complexity ( i.e. , number of episodes ) of centralized PG and NPG algorithms by assuming access to exact policy gradient . They also established a sample complexity result O ( −6 ) in the i.i.d . setting for NPG , which is worse than the state-of-the-art result O ( −3 ln −1 ) of both centralized NAC ( 60 ) and our decentralized NAC with Markovian samples . ( 3 ) proposes decentralized PG in a simple cooperative MARL setting , where all the agents share one action and the same policy , and they establish a iteration complexity in the order of O ( −4 ) . ( 13 ; 73 ) apply decentralized PG to Markov games . ( 2 ) applies decentralized NPG to a different cooperative MARL setting where each agent observes its own state , takes its own action and has access to these information of its neighbors . Value-based algorithms . Value-based algorithms have also been develop for MARL . Specifically , ( 21 ; 18 ) develop distributed Q-learning in a simplified cooperative MARL setting , where the agents share a joint action . In particular , ( 18 ) characterizes the convergence rate of a value function-based convergence error , which is a different optimality measure from that of AC-type algorithms . ( 35 ) applies distributed Q-learning to another cooperative MARL setting , where each agent observes its own state and takes its own action . It establishes an asymptotic convergence guarantee , and no convergence rate is given . ( 40 ) develops a value propagation algorithm that uses primal-dual method to minimize a soft Bellman error in the MARL setting . Under an assumption that the variance of the stochastic gradient is uniformly bounded , it establishes a non-asymptotic convergence rate to an approximate stationary point . 2 REVIEW OF MULTI-AGENT REINFORCEMENT LEARNING . In this section , we first introduce some standard settings of RL . Consider an agent that starts from an initial state s0 ∼ ξ and collects a trajectory of Markovian samples { st , at , Rt } t ⊂ S × A × R by interacting with an underlying environment ( with transition kernel P ) following a parameterized policy πω with induced stationary state distribution µω . The agent aims to learn an optimal policy that maximizes the expected accumulated reward J ( ω ) = ( 1− γ ) E [ ∑∞ t=0 γ tRt ] , where γ ∈ ( 0 , 1 ) is a discount factor . The marginal state distribution is denoted as Pω ( st ) and the visitation measure is defined as νω ( s ) : = ( 1− γ ) ∑∞ t=0 γ tPω ( st = s ) , both of which depend on the policy parameter ω ∈ Ω and the transition kernel P . We also define the mixed transition kernel Pξ ( ·|s , a ) : = γP ( ·|s , a ) + ( 1− γ ) ξ ( · ) , whose stationary state distribution is known to be νω . In the multi-agent RL ( MARL ) setting , M agents are connected via a fully decentralized network and interact with a shared environment . The network topology is specified by a doubly stochastic communication matrix W ∈ RM×M . At any time t , all the agents share a common state st. Then , every agent m takes an action a ( m ) t following its own current policy π ( m ) t ( ·|st ) parameterized by ω ( m ) t . After all the actions at : = { a ( m ) t } Mm=1 are taken , the global state st transfers to a new state st+1 and every agent m receives a local reward R ( m ) t . In this MARL setting , each agent m can only access the global state { st } t , its own actions { a ( m ) t } t and rewards { R ( m ) t } t and policy π ( m ) t . Next , define the joint policy πt ( at|st ) : = ∏M m=1 π ( m ) t ( a ( m ) t |st ) parameterized by ωt = [ ω ( 1 ) t ; . . . ; ω ( M ) t ] , and define the average reward Rt : = 1M ∑M m=1R ( m ) t . The goal of the agents is to collaboratively learn the optimal joint policy that maximizes the expected accumulated average reward J ( ω ) : = ( 1− γ ) E [ ∑∞ t=0 γ tRt ∣∣∣s0 ∼ ξ ] . Throughout , we consider the setting that the agents keep interacting with the environment and observing a trajectory of MDP transition samples , which are further used to learn the optimal joint policy .

Summary: This paper developed two sample and communication efficient decentralized actor-critic algorithms for multi-agent reinforcement learning. Specifically, the authors proposed decentralized AC and natural AC algorithms that can be private and efficient for different agents to learn. They added noise the local rewards of agents, which were shared among different agents. Moreover, the authors adopted mini-batch updates for the actors and critics in these two algorithms. Theoretically, the authors showed that both algorithms were able to achieve the state-of-the-art sample complexities. To validate the proposed algorithms, they used a simple environment to show the superiority over the existing decentralized AC algorithm. Overall, the investigated topic in this paper is absolutely interesting. The sample and communication complexities of decentralized MARL algorithms remain active research areas. While the mathematical analysis in this work looks good, the numerical results to me don't look very convincing. Additionally, there are quite a few points I have in mind regarding the proposed algorithm and the relevant analysis.