A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning

1 INTRODUCTION . Learning in multiagent settings is inherently more difficult than single-agent learning because an agent interacts both with the environment and other agents ( Buşoniu et al. , 2010 ) . Specifically , the fundamental challenge in multiagent reinforcement learning ( MARL ) is the difficulty of learning optimal policies in the presence of other simultaneously learning agents because their changing behaviors jointly affect the environment ’ s transition and reward function . This dependence on nonstationary policies renders the Markov property invalid from the perspective of each agent , requiring agents to adapt their behaviors with respect to potentially large , unpredictable , and endless changes in the policies of fellow agents ( Papoudakis et al. , 2019 ) . In such environments , it is also critical that agents adapt to the changing behaviors of others in a very sample-efficient manner as it is likely that their policy could update again after a small number of interactions ( Al-Shedivat et al. , 2018 ) . Therefore , effective agents should consider the learning of other agents and adapt quickly to non-stationary behaviors . Otherwise , undesirable outcomes may arise when an agent is constantly lagging in its ability to deal with the current policies of other agents . In this paper , we propose a new framework based on meta-learning for addressing the inherent non-stationarity of MARL . Meta-learning ( also referred to as learning to learn ) was recently shown to be a promising methodology for fast adaptation in multiagent settings . The framework by Al-Shedivat et al . ( 2018 ) , for example , introduces a meta-optimization scheme by which a meta-agent can adapt more efficiently to changes in a new opponent ’ s policy after collecting only a handful of interactions . The key idea underlying their meta-optimization is to model the meta-agent ’ s learning process so that its updated policy performs better than an evolving opponent . However , their work does not directly consider the opponent ’ s learning process in the meta-optimization , treating the evolving opponent as an external factor and assuming the meta-agent can not influence the opponent ’ s future policy . As a result , their work fails to consider an important property of MARL : the opponent is also a learning agent changing its policy based on trajectories collected by interacting with the meta-agent . As such , the meta-agent has an opportunity to influence the opponent ’ s future policy by changing the distribution of trajectories , and the meta-agent can take advantage of this opportunity to improve its performance during learning . Our contribution . With this insight , we develop a new meta-multiagent policy gradient theorem ( Meta-MAPG ) that directly models the learning processes of all agents in the environment within a single objective function . We start by extending the meta-policy gradient theorem of Al-Shedivat et al . ( 2018 ) based on the multiagent stochastic policy gradient theorem ( Wei et al. , 2018 ) to derive a novel meta-policy gradient theorem . This is achieved by removing the unrealistic implicit assumption of Al-Shedivat et al . ( 2018 ) that the learning of other agents in the environment is not dependent on an agent ’ s own behavior . Interestingly , performing our derivation with this more general set of assumptions inherently results in an additional term that was not present in previous work by Al-Shedivat et al . ( 2018 ) . We observe that this added term is closely related to the process of shaping the learning dynamics of other agents in the framework of Foerster et al . ( 2018a ) . As such , our work can be seen as contributing a theoretically grounded framework that unifies the collective benefits of previous work by Al-Shedivat et al . ( 2018 ) and Foerster et al . ( 2018a ) . Meta-MAPG is evaluated on a diverse suite of multiagent domains , including the full spectrum of mixed incentive , competitive , and cooperative environments . Our experiments demonstrate that Meta-MAPG consistently results in superior adaption performance in the presence of novel evolving agents . 2 PRELIMINARIES . Interactions between multiple agents can be represented by stochastic games ( Shapley , 1953 ) . Specifically , an n-agent stochastic game is defined as a tupleMn=〈I , S , A , P , R , γ〉 ; I= { 1 , . . . , n } is the set of n agents , S is the set of states , A=×i∈IAi is the set of action spaces , P : S ×A 7→ S is the state transition probability function , R=×i∈IRi is the set of reward functions , and γ ∈ [ 0 , 1 ) is the discount factor . We typeset sets in bold for clarity . Each agent i executes an action at each timestep t according to its stochastic policy ait∼πi ( ait|st , φi ) parameterized by φi , where st ∈S . A joint action at= { ait , a−it } yields a transition from the current state st to the next state st+1∈S with probability P ( st+1|st , at ) , where the notation−i indicates all other agents with the exception of agent i . Agent i then obtains a reward according to its reward function rit = Ri ( st , at ) . At the end of an episode , the agents collect a trajectory τφ under the joint policy with parameters φ , where τφ : = ( s0 , a0 , r0 , . . . , rH ) , φ= { φi , φ−i } represents the joint parameters of all policies , rt= { rit , r−it } is the joint reward , and H is the horizon of the trajectory or episode . 2.1 A MARKOV CHAIN OF POLICIES . The perceived non-stationarity in multiagent settings results from a distribution of sequential joint policies , which can be represented by a Markov chain ( Al-Shedivat et al. , 2018 ) . Formally , a Markov chain of policies begins from a stochastic game between agents with an initial set of joint policies parameterized by φ0 . We assume that each agent updates its policy leveraging a Markovian update function that changes the policy after every K trajectories . After this time period , each agent i adapts its policy to maximize the expected return expressed as its value function : V iφ0 ( s0 ) = Eτφ0∼p ( τφ0 |φi0 , φ−i0 ) [ H∑ t=0 γtrit|s0 ] = E τφ0∼p ( τφ0 |φ i 0 , φ −i 0 ) [ Gi ( τφ0 ) ] , ( 1 ) where Gi denotes agent i ’ s discounted return from the beginning of an episode with initial state s0 . The joint policy update results in a transition from φ0 to the updated set of joint parameters φ1 . The Markov chain continues for a maximum chain length of L ( see Figure 1a ) . This Markov chain perspective highlights the following inherent aspects of the experienced non-stationarity : Sequential dependency . The future joint policy parameters φ1 : L = { φ1 , . . . , φL } sequentially depend on φ0 since a change in τφ0 results in a change in φ1 , which in turn affects τφ1 and all successive joint policy updates up to φL . Controllable levels of non-stationarity . As in Al-Shedivat et al . ( 2018 ) and Foerster et al . ( 2018a ) , we assume stationary policies during the collection of K trajectories , and that the joint policy update happens afterward . In such a setting , it is possible to control the non-stationarity by adjusting the K and H hyperparameters : smaller K and H increase the rate that agents change their policies , leading to a higher degree of non-stationarity in the environment . In the limit of K =H = 1 , all agents change their policy every step . 3 LEARNING TO LEARN IN MULTIAGENT REINFORCEMENT LEARNING . This section explores learning policies that can adapt quickly to non-stationarity in the policies of other agents in the environment . To achieve this , we leverage meta-learning and devise a new meta-multiagent policy gradient theorem that exploits the inherent sequential dependencies of MARL discussed in the previous section . Specifically , our meta-agent addresses this non-stationarity by considering its current policy ’ s impact on its own adapted policies while actively influencing the future policies of other agents as well by inducing changes to the distribution of trajectories . In this section , we first outline the meta-optimization process in MARL and then discuss how the meta-policy gradient theorem of Al-Shedivat et al . ( 2018 ) optimizes for this objective while ignoring the dependence of the future policy of other agents on our current policy . Finally , we derive a new extension of this policy gradient theorem that explicitly leverages this dependence and discuss how to interpret the impact of the resulting form of the gradient . 3.1 GRADIENT BASED META-OPTIMIZATION IN MULTIAGENT REINFORCEMENT LEARNING . We formalize the meta-objective of MARL as optimizing meta-agent i ’ s initial policy parameters φi0 so that it maximizes the expected adaptation performance over a Markov chain of policies drawn from a stationary initial distribution of policies for the other agents p ( φ−i0 ) : max φi0 E p ( φ −i 0 ) [ L−1∑̀ =0 V iφ0 : ` +1 ( s0 , φ i 0 ) ] , ( 2 ) s.t . V iφ0 : ` +1 ( s0 , φ i 0 ) = Eτφ0 : ` ∼p ( τφ0 : ` |φi0 : ` , φ−i0 : ` ) [ E τφ ` +1∼p ( τφ ` +1 |φ i ` +1 , φ −i ` +1 ) [ Gi ( τφ ` +1 ) ] ] ( 3 ) where τφ0 : ` = { τφ0 , . . . , τφ ` } , and V iφ0 : ` +1 ( s0 , φi0 ) denotes the meta-value function . This meta-value function generalizes the notion of each agent ’ s primitive value function for the current set of policies V iφ0 ( s0 ) over the length of the Markov chain of policies . In this work , as in Al-Shedivat et al . ( 2018 ) , we follow the MAML ( Finn et al. , 2017 ) meta-learning framework . As such , we assume that the Markov chain of policies is governed by a policy gradient update function that corresponds to what is generally referred to as the inner-loop optimization in the meta-learning literature : φi ` +1 : = φ i ` + α i∇φi ` Eτφ ` ∼p ( τφ ` |φi ` , φ−i ` ) [ Gi ( τφ ` ) ] , φ−i ` +1 : = φ −i ` +α −i∇ φ −i ` E τφ ` ∼p ( τφ ` |φ i ` , φ −i ` ) [ G−i ( τφ ` ) ] , ( 4 ) where αi and α−i denote the learning rates used by each agent in the environment . 3.2 THE META-POLICY GRADIENT THEOREM . Intuitively , if we optimize the meta-value function , we are searching for initial parameters φi0 such that successive inner-loop optimization steps with Equation ( 4 ) results in adapted parameters φi ` +1 that can perform better than the updated policies of other agents with policy parameters φ−i ` +1 ( see Figure 1b ) . Algorithm 1 Meta-Learning at Training Time Require : p ( φ−i0 ) : Distribution over other agents ’ initial policies ; α , β : Learning rates 1 : Randomly initialize φi0 2 : while φi0 has not converged do 3 : Sample a meta-train batch of φ−i0 ∼ p ( φ −i 0 ) 4 : for each φ−i0 do 5 : for ` = 0 , . . . , L do 6 : Sample and store trajectory τφ ` 7 : Compute φ ` +1 = f ( φ ` , τφ ` , α ) from inner-loop optimization ( Equation ( 4 ) ) 8 : end for 9 : end for 10 : Update φi0 ← φi0 + β ∑L−1 ` =0 ∇φi0V i φ0 : ` +1 ( s0 , φ i 0 ) based on Equation ( 6 ) 11 : end while Algorithm 2 Meta-Learning at Execution Time Require : p ( φ−i0 ) : Distribution over other agents ’ initial policies ; α : Learning rate ; Optimized φi∗0 1 : Initialize φi0 ← φi∗0 2 : Sample a meta-test batch of φ−i0 ∼ p ( φ −i 0 ) 3 : for each φ−i0 do 4 : for ` = 0 , . . . , L do 5 : Sample trajectory τφ ` 6 : Compute φ ` +1 = f ( φ ` , τφ ` , α ) from inner-loop optimization ( Equation ( 4 ) ) 7 : end for 8 : end for In Deep RL , a very practical way to optimize a value function is by following its gradient . The work of Al-Shedivat et al . ( 2018 ) derived the meta-policy gradient theorem ( Meta-PG ) for optimizing a setup like this . However , it is important to note that they derived this gradient while making the implicit assumption to ignore the dependence of the future parameters of other agents on φi0 : ∇φi0V i φ0 : ` +1 ( s0 , φ i 0 ) = Eτφ0 : ` ∼p ( τφ0 : ` |φi0 : ` , φ−i0 : ` ) [ E τφ ` +1∼p ( τφ ` +1 |φ i ` +1 , φ −i ` +1 ) [ ( ∇φi0 logπ ( τφ0 |φ i 0 ) ︸ ︷︷ ︸ Current Policy + ∑ ` ` ′=0∇φi0 logπ ( τφ ` ′+1 |φ i ` ′+1 ) ︸ ︷︷ ︸ Own Learning ) Gi ( τφ ` +1 ) ] ] . ( 5 ) In particular , Meta-PG has two primary terms . The first term corresponds to the standard policy gradient with respect to the current policy parameters used during the initial trajectory . Meanwhile , the second term∇φi0 log π ( τφ ` ′+1 |φ i ` ′+1 ) explicitly differentiates through log π ( τφ ` ′+1 |φi ` ′+1 ) with respect to φi0 . This enables a meta-agent i to model its own learning dynamics and account for the impact of φi0 on its eventual adapted parameters φ i ` ′+1 . As such , we can see how this term would be quite useful in improving adaptation across a Markov chain of policies . Indeed , it directly accounts for an agent ’ s own learning process during meta-optimization in order to improve future performance .

Summary: This paper points out that a key challenge in MARL is the non-stationarity of other agents' policies, as opposed to previous papers which only account for non-stationarity of the environment. The paper extends (Al-Shedivat et al., 2018) by directly conditioning the meta-policy on a distribution of other agents' policies. In my opinion, the major contribution of this paper is a new multiagent meta learning theoretic framework that explicitly accounts  for the dynamics of all agents.