Finite-Time Convergence and Sample Complexity of Multi-Agent Actor-Critic Reinforcement Learning with Average Reward

In this paper , we establish the first finite-time convergence result of the actor-critic algorithm for fully decentralized multi-agent reinforcement learning ( MARL ) problems with average reward . In this problem , a set of N agents work cooperatively to maximize the global average reward through interacting with their neighbors over a communication network . We consider a practical MARL setting , where the rewards and actions of each agent are only known to itself , and the knowledge of joint actions of the agents is not assumed . Toward this end , we propose a mini-batch Markovian sampled fully decentralized actor-critic algorithm and analyze its finite-time convergence and sample complexity . We show that the sample complexity of this algorithm is O ( N2/ǫ2 log ( N/ǫ ) ) . Interestingly , this sample complexity bound matches that of the state-of-the-art single-agent actorcritic algorithms for reinforcement learning . 1 INTRODUCTION . 1 ) Background and Motivations : In recent years , multi-agent reinforcement learning ( MARL ) has found a wide range of applications in networked large-scale systems , such as power grid systems ( Riedmiller et al. , 2000 ) , autonomous driving ( Yu et al. , 2019 ; Shalev-Shwartz et al. , 2016 ) and strategic games ( Silver et al. , 2018 ; Foerster et al. , 2018 ) , to name just a few . Although empirical successes of MARL applications have been widely observed , the fundamental theoretical understanding of how to develop fast-converging and low sample-complexity MARL algorithms , two of the most important performance metrics for MARL , remains in its infancy so far ( see , e.g. , ( Zhang et al. , 2021a ) for an excellent survey ) . In particular , two important aspects of cooperative MARL algorithm designs deserve special attention : • First , in the multi-agent collaborative setting , the information structure ( i.e. , the assumptions of who have the knowledge of what ) is far more complex than its single-agent counterpart and care must be taken in MARL problem formulations . In the cooperative MARL literature so far , many existing works assume full knowledge of joint states and joint actions , which often do not hold true in practice . For example , in autonomous driving ( Yu et al. , 2019 ) , each vehicle can only observe/detect the actions of the surrounding vehicles that are within its communication range . As another example , in power grid networks ( Riedmiller et al. , 2000 ) , each power distributor generally does not know the resistor values set by other distributors . • Second , most MARL theoretical studies in the MARL literature are focused on the discounted total reward setting , where a hyperparameter γ ∈ ( 0 , 1 ) is introduced as the discount factor in the objective function . Although the discounted total reward setting captures the important aspect of diminishing return in the future , it may not be appropriate for many other applications where the long-term average reward is of interest . For example , in the optimization of distributed communication networks with MARL , the typical and natural performance metrics are long-term average throughput or latency in the steady-state . The lack of a fundamental understanding on how to develop efficient cooperative MARL algorithms that consider the above two aspects in terms of information structure , scalability , and communication and sample complexities motivates us to fill this gap by developing a fully decentralized cooperative MARL algorithm in the average reward setting , without assuming joint action knowledge . 2 ) Technical Challenges : Developing a fully decentralized cooperative MARL algorithm for the average reward setting without full joint action knowledge is highly non-trivial and several major technical challenges naturally arise . First , it is well-known that , even in the single-agent reinforcement learning ( RL ) setting , the average reward setting is more challenging to analyze compared to the discounted reward setting , which necessitates different proof techniques ( Tsitsiklis & Van Roy , 1999 ) . In MARL , the decentralized nature and the lack of joint action-state information further complicate the algorithm design and analysis in the average reward setting . Second , due to the lack of joint action knowledge , the communication costs among agents in MARL will be significantly increased to achieve a satisfying performance , which implies that low sample and communication complexities are even more challenging and critical for MARL without joint action information . 3 ) Main Results and Contributions : The main contribution of this paper is that we overcome the aforementioned challenges and develop a consensus-based Markovian sampled decentralized actorcritic algorithm for MARL . Our key results in this paper are summarized as follows : • We propose a batch-sampled actor-critic algorithm that uses consensus updates in TD-sharing among agents . This batch sampling approach enables more efficient communication compared to the classical fully decentralized MARL . Specifically , in order to converge to an ǫ neighborhood of the stationary , we requireO ( ǫ−1 log ( ǫ−1 ) ) rounds of communication while only needing O ( ǫ−2 log ( ǫ−1 ) ) samples . By contrast , the state-of-the-art MARL requires a communication round per sampling . Also , our algorithm allows the use of constant step-sizes in both the actor and the critic steps . • We provide the first-ever sample complexity analysis in the MARL average reward problem setting without joint action information . Our obtained complexity isO ( ǫ−2 log ( ǫ−1 ) ) , where ǫ is the closeness to the neighborhood of stationary point ( treating network size N as a fixed constant ) . It is worth noting that the order-wise sample complexity of our algorithm matches that of the state-of-the-art single-agent RL algorithms . 2 RELATED WORK . In this section , we provide a quick overview on the closely related work on MARL algorithms and their theoretical results , along with several notable related counterparts in single-agent RL . 1 ) MARL Theoretical Analysis and Algorithm Design : For recent advances in MARL algorithms and their theoretical results , Zhang et al . ( 2021a ) provided a comprehensive survey . Also , Lee et al . ( 2020 ) highlighted the evolution from single-agent to multi-agent RL from a distributed optimization perspective . In the broader area of MARL , a line of research has been focused on the MARL policy evaluation problem . These works analyzed the convergence Doan et al . ( 2019 ) and proposed various variance reduction of policy evaluation in decentralized MARL algorithms Zhang et al . ( 2021b ) . Doan et al . ( 2019 ) used i.i.d . sampling and has shown the sample complexity of Õ ( ǫ−1 ) for their TD ( 0 ) learning algorithm to reach a mean-square error convergence . However , these algorithms do not involve policy improvement and solely focus on the performance evaluation of given policies . In the areas of joint policy evaluation and improvement , Foerster et al . ( 2018 ) considered multi-agent actor-critic algorithm that has a centralized critic and decentralized actors , which is different from our fully decentralized actor-critic algorithm . In contrast , Zhang et al . ( 2018 ) established asymptotic convergence results for fully decentralized MARL actor-critic algorithms . Concurrent with our work , Chen et al . ( 2021 ) has recently studied the mini-batch Markovian sampling actor-critic algorithm for a class of discounted reward MARL problems , where the finite-time convergence result is obtained . They have applied batch sampling for both actor and critic steps and achieved a sample complexity of O ( ǫ−2 log ( ǫ−1 ) ) , which is the same as ours . We note that , together with our work , these are the first finite-time convergence results for MARL . However , there are several key differences between our work and ( Chen et al. , 2021 ) . First , we focus on the average reward problem , while Chen et al . ( 2021 ) studied the discounted reward setting . Second , in ( Chen et al. , 2021 ) , agents share a noisy version of the rewards with the neighbors , which requires a re-sampling process from every sampled reward instance . In contrast , we allow agents to share local TD-errors with their neighbors and no re-sampling is required . 2 ) Related Literature in Single-Agent RL : We note that single-agent RL can be viewed as a centralized approach , where a central controller collects joint actions , rewards and even designs policies for agents . For the single-agent average reward setting , Tsitsiklis & Van Roy ( 1999 ) first analyzed the asymptotic convergence of TD ( λ ) algorithm with function approximations in the policy evaluation problem . Also , Tsitsiklis & Van Roy ( 2002 ) provided insights in terms of differences and connections between average reward and discounted reward of TD-based learning algorithms with function approximations . Recently , Qiu et al . ( 2021 ) analyzed the sample complexity for an actorcritic algorithm for the average reward problem . In their actor-critic algorithm , they used batch sampling for the critic learning and i.i.d . sampling for the actor step , with sample complexity being O ( ǫ−3 log2 ( ǫ−1 ) ) . By applying mini-batch sampling update , we are able to improve the sample complexity by a factor of O ( ǫ−1 ) . Another closely related work on single-agent RL is ( Xu et al. , 2020 ) , where the authors studied the discounted reward problem . They used batch sampling for both actor and critic steps in their actor-critic algorithm and developed a new technique to handle bias error in the critic step , which we also adopted for the average approximation parameter analysis in our critic step . This achieved the state-of-the-art sample complexity of O ( ǫ−2 log ( ǫ−1 ) ) for singleagent RL . In addition , the global convergence of actor-critic algorithm to the optimal policy has been studied in the case of discounted setting with single time scale in ( Fu et al. , 2020 ) and linear quadratic regulator in ( Yang et al. , 2019 ) . However , we note that these settings are fundamentally different from the average reward setting and it will be an interesting future direction to consider global convergence possibility in the average reward setting . To conclude this section , we summarize the aforementioned related actor-critic and TD algorithms and their sample complexity results in Table 1 . 3 MULTI-AGENT REINFORCEMENT LEARNING WITH AVERAGE REWARD . 3.1 SYSTEM MODEL . Consider a multi-agent system with N agents , denoted by N = { 1 , · · · , N } , operating in a networked environment . Let E be the edge set for a given network G = ( N , E ) . To formulate our MARL problem and facilitate our subsequent discussions , we first define the notion of networked multi-agent MDP as follows . Definition 1 ( Networked Multi-Agent MDP ) . Let G = ( N , E ) be a communication network that connects N agents . A networked multi-agent MDP is defined by following tuple ( S , { Ai } i∈N , P , { R i } i∈N , G ) , where S is the global state space observed by all agents , A i is the action set for agent i , and P : S×A×S → [ 0 , 1 ] is a global state transition function , andRi : S×A is the local reward function for agent i . Let A = ∏ i∈N A i be the joint action set of all agents . In this paper , we assume that the global state space S and action space for agent Ai are finite . As a result , the joint action space A is also finite for finite N . We also note that at time t ≥ 0 , all agents can observe the current global state st . However , agent i can only observe its own action ait ∈ A i , which is the key difference between our model and that in ( Zhang et al. , 2018 ) , where it is assumed that the joint actions are observable to all agents . Moreover , each agent can only observe its own reward rit , i.e. , agents do not observe or share rewards with other agents at time t. The reward function Ri ( s , a ) is an expectation given s and a , and the instantaneous reward is denoted by ri ( s , a ) , i.e. , Ri ( s , a ) = E [ ri ( s , a ) ] . We consider policies that are stationary . In our MARL system , each agent chooses its action following its local policy πi that is conditioned on the current global state s , i.e. , πi ( ai|s ) is the probability for agent i to choose an action ai ∈ Ai . Then , the joint policy π : S × A → [ 0 , 1 ] can be written as π ( a|s ) = ∏ i∈N π i ( ai|s ) . Moreover , the policies at the agents are parameterized . Specifically , each agent i ’ s local policy can be written as πi θi , where θi ∈ Rmi denotes the parameter . We let θ , [ ( θ1 ) T , · · · , ( θN ) T ] T ∈ R ∑N i=1mi . Then , we can write the joint policy as follows : πθ ( a|s ) = ∏ i∈N πθi ( a i|s ) .

Summary: This paper studies a networked MARL problem based on the model in [Zhang et al 2018], where each agent can observe the global state, take local action and observe local rewards. The key difference in setting from [Zhang et al 2018] is that [Zhang et al 2018] assume the global action can be observed, but in this paper, only local action is known to each agent. To deal with this, an additional consensus loop is added to estimate the average TD error, which can be used to estimate the advantage function. Further, compared to [Zhang et al 2018], a finite time error bound is provided.