Taming Communication and Sample Complexities in Decentralized Policy Evaluation for Cooperative Multi-Agent Reinforcement Learning

√ n −2 ) sam- ple complexity , where m is the number of agents and n is the length of trajectories . To our knowledge , this paper is the first work that achieves O ( −2 ) in both sample and communication complexities in decentralized policy evaluation for cooperative MARL . Our extensive experiments also corroborate the theoretical results of our proposed decentralized policy evaluation algorithms . 1 Introduction . In recent years , multi-agent reinforcement learning ( MARL ) has found important applications in many scientific and engineering fields , such as robotic network [ 45 , 22 , 41 ] , sensor network [ 6 , 40 , 43 ] , and power network [ 5 , 11 , 15 , 16 ] , just to name a few . In MARL , multiple agents observe the current joint state over a network , perform their own actions based on the current state , and transition to ∗Xin Zhang and Zhuqing Liu contributed equally to this work . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the next joint state . Each agent can only observe its local reward , which is a function of the joint state and actions . One important paradigm in MARL is the cooperative MARL , where the agents share a common goal to find an optimal global policy to achieve the maximum global accumulative reward [ 63 , 62 , 13 , 37 ] . As in classical reinforcement learning ( RL ) problems , a key component in cooperative MARL algorithms is the policy evaluation ( PE ) problem , whose goal is to evaluate the expected long-term accumulative reward for a given global policy . In cooperative MARL , agents share a common value function based on the joint states . As a result , the parameterization of the value function is the same across all agents . Examples of cooperative MARL include but are not limited to traffic light control [ 56 ] , autonomous driving [ 21 ] , financial trading [ 26 ] ) , etc . In cooperative MARL , the PE problem emerges as a key step for the agents to find an optimal global policy in MARL tasks [ 3 , 23 ] . For example , in the actor-critic algorithmic framework for MARL , the actors conduct the policy improvement step , while the critics perform the policy evaluation step and estimates the value function . The overall actor-critic algorithm tries to find an optimal policy by iterating between the policy evaluation and improvement steps . Thus , developing efficient policy evaluation algorithms is critical to the success of RL algorithms based on the actor-critic framework . However , developing efficient PE algorithms for MARL is highly non-trivial . On one hand , the global accumulative reward is not directly observable in an MARL system . As a result , the PE problem of MARL can only be solved in a decentralized fahsion . On the other hand , modern MARL tasks have been increasingly complex and often not directly computable . As a result , MARL often use highly nonlinear parametric models ( e.g. , deep neural network ( DNN ) ) for policy approximation . In this paper , we focus on PE based on nonlinear function approximations due to the following reasons : 1 ) linear approximation schemes are based on their pre-defined basis space , which may not be able to approximate the non-linear value function with high accuracy ; 2 ) non-linear neural network approximation can handle the cases where the states space that is mixed with continuous and ( infinite ) discrete state values ; and 3 ) nonlinear neural network approximation usually have a better generalization performance than linear approximation [ 61 ] , [ 20 ] , [ 60 ] . However , it has been shown that the convergence performance of RL algorithms with nonlinear function approximations is not guaranteed [ 49 ] . In light of the growing importance of MARL , in this paper , we focus on addressing the above challenges . The key contributions of this paper are summarized as follows : • To our knowledge , this work is the first to investigate the decentralized PE ( DPE ) problem for MARL with nonlinear function approximations . Via the Fenchel ’ s duality and local reward decomposition , we first reformulate the DPE problem of MARL as a decentralized non-convexstrongly-concave minimax saddle point problem . To solve the minimax problem in a decentralized fashion , we propose a gradient-tracking-based gradient descent-ascent ( GT-GDA ) algorithm . We show that GT-GDA enjoys a convergence rate of O ( 1/T ) , which leads to an O ( mn −2 ) sample complexity and an O ( −2 ) communication complexity , where m is the number of agents , n is the data size , and is the convergence accuracy . • To further reduce the sample complexity , we develop two variance-reduced algorithms , namely gradient-tracking stochastic recursive variance reduction ( GT-SRVR ) algorithm and its variant with incremental batch size ( GT-SRVRI ) . We show that both algorithms achieve the same communication complexity O ( −2 ) as GT-GDA , but requiring a lower sample complexity O ( m √ n −2 ) . • It is worth noting that , in our theoretical analysis , we relax the commonly-used compactness conditions of the feasible set with some mild assumptions on objectives . Thus , the solutions found by our algorithms are exactly the stationary points for the original policy evaluation problem . This result may be of independent interest for general RL problems . The rest of the paper is organized as follows . In Section 2 , we first provide the preliminaries of the DPE problem of MARL and discuss related works . In Section 3 , we first introduce the GT-GDA algorithm , and then propose two stochastic variance reduced algorithms , namely GT-SRVR and GT-SRVRI . We present their theoretical properties in Section 4 . Section 5 provides numerical results to verify our theoretical findings , and Section 6 concludes this paper . 2 Problem formulation and related work . In this section , we first introduce the DPE problem formulation in in Section 2.1 . Then in Section 2.2 , we review the recent developments of PE algorithms and compare them with our work . In Sec- tion 2.3 , we highlight the challenges in designing efficient DPE algorithms with nonlinear function approximation and the significance of our work . 2.1 Problem formulation of decentralized policy evaluation for MARL . Consider a multi-agent network system G = ( N , L ) , where N and L denote the sets of agents and edges , respectively , with |N | = m. In the system , the agents cooperatively perform a learning task . The agents can communicate with each other through edges in L. An MARL problem is formulated based on the multi-agent Markov decision process ( MDP ) framework , which is characterized by a quintuple ( S , A , Pass′ , { Ri ( s , a ) } mi=1 , ζ ) , where S andA are the state and action spaces , respectively ; s ∈ S and a ∈ A are joint state and action ; Pass′ is the transition probability from state s to state s′ after taking action a ; Ri ( s , a ) is the local reward received by agent i after taking action a in state s ; and ζ ∈ ( 0 , 1 ) is a discount factor . Both the joint state s and action a are available to all agents , while the local rewardRi is private to agent i . In a multi-agent system , the global reward function is defined as the average of the local rewards 1m ∑m i=1Ri ( s , a ) . Moreover , a joint policy π specifies sequential decision rules for all agents . Policy π ( a|s ) is the conditional probability of taking joint action a given state s. The goal of PE is to estimate the value function of a given policy π , which is defined as the long-term discounted accumulative reward : Vπ ( s0 ) = E [ 1 m ∑∞ t=0 ζ t ∑m i=1Ri ( st , at ) |s0 , π ] , where the expectation is taken over all possible state-action trajectories and initial states . To determine Vπ ( · ) , one of the most effective methods is the temporal-difference ( TD ) learning algorithm , which focuses on solving the Bellman equation for Vπ ( · ) : V ( s ) =T πV ( s ) , 1m ∑m i=1Rπi ( s ) + ζ ∑ s′∈S Pπss′V ( s′ ) , where T π denotes the Bellman operator , Rπi ( s ) = Ea∼π ( ·|s ) R ( s , a ) andPπss′ = Ea∼π ( ·|s ) Pass′ However , Pπss′ is unknown in MARL and the size of the state space S could be infinite . To address this challenge , a widely adopted approach is to approximate Vπ ( · ) by a function Vθ ( · ) parameterized by θ ∈ Rp . According to the formulation in [ 38 , 50 , 10 , 9 , 30 ] , the Bellman equation can be solved by minimizing the following mean-squared projected bellman error ( MSPBE ) : MSPBE ( θ ) , 12 ∥∥Es∼dπ [ ( T πVθ ( s ) − Vθ ) ∇θVθ ( s ) > ] ∥∥2K−1θ , where Kθ = Es∼dπ [ ∇θVθ ( s ) ∇θVθ ( s ) > ] ∈ Rp×p and dπ is the stationary distribution of the MDP under policy π . From the Fenchel ’ s duality ‖x‖2A−1 = maxy∈Rp 2〈x , y〉 − y > Ay , we can reformulate the MSPBE minimization problem as the following primal-dual minimax problem : min θ∈Rp max ω∈Rp L ( θ , ω ) , E [ 〈δ · ∇θVθ ( s ) , ω〉 − 1 2 ω > [ ∇θVθ ( s ) ∇θVθ ( s ) > ] ω ] , ( 1 ) where the expectation is taken over s ∼ dπ ( · ) , a ∼ π ( ·|s ) , s′ ∼ Pas· , and δ = 1m ∑m i=1Ri ( s , a ) + ζVθ ( s′ ) − Vθ ( s ) . In practice , we only have access to a finite dataset with n-step trajectories D = { ( st , at , { Ri ( st , at ) } ni=1 , st+1 ) } n t=0 . By replacing the unknown expectation with the finite sample average , we have the following empirical minimax problem : min θ∈Rp max ω∈Rp F ( θ , ω ) = 1 n n∑ t=1 〈δt · ∇θVθ ( st ) , ω〉 − 1 2 ω > K̂θω , ( 2 ) where δt , 1m ∑m i=1Ri ( st , at ) + ζVθ ( st+1 ) − Vθ ( st ) and K̂θ , 1 n ∑n t=1∇θVθ ( st ) ∇θVθ ( st ) > . In this paper , we assume that both Kθ and its empirical estimate K̂θ are positive definite for all θ . Define J ( θ ) , F ( θ , ω∗ ) = maxω∈Rp F ( θ , ω ) , where ω∗ = arg maxω∈Rp F ( θ , ω ) . J ( θ ) can be viewed as the finite empirical version of MSPBE . Here , we aim to minimize J ( θ ) by finding a stationary point of F ( θ , ω ) . Recall that in MARL , the local reward is only observable for each individual agent . Thus , it is hard to obtain the global reward 1m ∑m i=1Ri ( st , at ) and δt in a multi-agent network . To address this challenge , we define δi , t = Ri ( st , at ) + ζVθ ( st+1 ) − Vθ ( st ) and decompose the minimax problem in ( 2 ) as follows : minθ∈Rp maxω∈Rp F ( θ , ω ) = 1m ∑m i=1 Fi ( θ , ω ) = 1 mn ∑m i=1 ∑n t=1 fit ( θ , ω ) , where fit ( θ , ω ) , 〈δi , t ·∇θVθ ( st ) , ω〉− 1 2ω > [ ∇θVθ ( st ) ∇θVθ ( st ) > ] ω . We call this step as local reward decomposition . In cooperative MARL , a key challenge is that the PE problem in ( 2 ) has to be solved in a decentralized fashion , which is due to the fact that i ) the locally observed rewards are private and can not be shared with the other agents/central server ; ii ) it is difficult to set up a central sever in many MARL applications while decentralized setting is more flexible ( e.g. , wireless network [ 59 ] , UAV network [ 7 ] ) ; and iii ) the central server is vulnerable to cyber-attacks and would be a significant communication bottleneck [ 57 ] , [ 27 ] . To solve Problem ( 2 ) in a decentralized fashion , we can rewrite it in the following equivalent form : min { θi } mi=1 max { ωi } mi=1 1 m m∑ i=1 Fi ( θi , ωi ) = 1 mn m∑ i=1 n∑ t=1 fit ( θi , ωi ) , subject to θi = θj , ωi = ωj , ∀ ( i , j ) ∈ L , ( 3 ) where θi and ωi are the local copies of the original primal-dual parameters at agent i . In ( 3 ) , the equality constraint ensures that the local copies at all nodes are equal to each other , so the formulation is also referred to as the “ consensus form. ” Clearly , Problems ( 2 ) and ( 3 ) are equivalent . For a fixed θ , each local function Fi ( θi , · ) is a strongly concave function of ω . For a fixed ω , Fi ( · , ω ) is a non-convex function of θ . Thus , Problem ( 3 ) is a decentralized non-convex-strongly-concave minimax consensus optimization problem . In this paper , we adopt two complexity metrics that are widely used in the decentralized optimization literature ( e.g. , [ 47 ] ) to measure the efficiency of an algorithm : Definition 1 ( Sample Complexity ) . The sample complexity is defined as the total number of incremental first-order oracle ( IFO ) calls required across all nodes until algorithm converges , where one IFO call evaluates a pair of ( fit ( θ , ω ) , ∇fit ( θ , ω ) ) at node i . Definition 2 ( Communication Complexity ) . The communication complexity is defined as the total rounds of communications required until algorithm converges , where each node can send and receive a p-dimensional vector with its neighboring nodes in one communication round .

Summary: This paper studies the problem of policy evaluation in a decentralized multi-agent reinforcement learning setting with non-linear function approximation. The authors present a gradient tracking based descent-ascent optimization algorithm GT-GDA that aggregates global gradients by a network-weighted aggregation, and consequently updates local parameters by descent/ascent. The authors establish convergence of the proposed algorithm and additionally analyse a reduced-variant extension. Experimental results establish convergence in several numerical benchmarks.