Sample-efficient policy learning in multi-agent Reinforcement Learning via meta-learning

1 INTRODUCTION . Applying Reinforcement Learning ( RL ) to multi-agent scenes requires carefully consideration about the influence of other agents . We can not simply treat other agents as part of the environment and apply independent RL methods ( Lanctot et al. , 2017 ) if the actions of them has impact on the payoff of the agent to be trained . For example , consider the two-person ultimatum bargaining game , where two players take part in . One player propose a deal to split a fixed amount of money for them two and the other player decides to accept it or not . If the second player accepts the proposal , they split the money , but if the proposal is refused , they both get zero . Experimental results ( Güth et al. , 1982 ) show that in actual life , the second player makes the decision according to whether he or she judge the final result fair , rather than makes the obvious rational decision . Thus , the first player needs to predict how the second player will react so as to make the proposal acceptable . In order to exploit the other agents and find the corresponding optimal policy , we need to understand these agents . Here in this paper , we call all the other agents “ opponents ” to distinguish our agent from them , even if they may have cooperative relationship with our agent . For simplicity , we only consider tasks with only one opponent . Extension to tasks with more opponents is straightforward . A general way to exploit an opponent is to build a model for it from observations . This model can characterize any needed feature of the opponent , such as next action or the final goal . Such a model can make predictions for the opponent and thus turns the two-agent task into a simple-agent decision making problem . Then we can apply various RL methods to solve this problem . It is necessary that we need to have an accurate model for the opponent to help make decision . Previous works ( He et al. , 2016 ; Raileanu et al. , 2018 ) propose some methods to model the opponent . Generally , it requires many observations to get a precise model for the opponent . This may cost many iterations to act with the opponent . What ’ s more , even if we can precisely model the opponent , there exists a main drawback of above process that the performance of the learned policy has no guarantee for any other opponent . Things are even worse if opponents have their private types which are unknown for us . New opponents with different types can have different policies or even different payoffs . Therefore , it seems that when a new opponent came , we have to learn a policy from the beginning . In some practical situations , the whole opponents follow a distributions over all these possible types . Let ’ s come back to the ultimatum bargaining game . Bahry & Wilson ( 2006 ) shows that people with different ethnicity may have different standards for fairness . Thus if we assume the type for player 2 to be its judgment for fairness , there can be a distribution for types dependent on the ethnic distribution . Given that opponents follows a distribution , it is possible that we can employ some given opponents to help us speed up the process of opponent modeling and policy improving for the current opponent . If we consider the policy learning against a specific opponent as a task , our goal can be considered as training a policy on various tasks so that it can efficiently adapt to a good policy on a new task with few training samples . This is exactly a meta-learning problem . We employ Model-Agnostic MetaLearning ( MAML ) ( Finn et al. , 2017 ) to conduct meta-learning . Rabinowitz et al . ( 2018 ) applied meta-learning to understand opponents , but this work doesn ’ t address the policy improvement for the agent to be trained . We apply meta-learning to opponent modeling and policy learning separately while training the two meta-learners jointly . Then we use the meta-learners to initialize the model and policy for the new opponent . Experimental results show that the agent can adapt to the new opponent with a small number of interactions with the opponent . 2 PRELIMINARY . In this section , we introduce some preliminaries of our work . We formalize our task based on stochastic games ( Littman , 1994 ) , which is a general framework for multi-agent problems , and Bayesian games ( Harsanyi , 1967 ) , which formalize the incomplete information of players . Next we introduce Model-Agnostic Meta-Learning ( MAML ) , a meta-learning algorithm that is applicable to various gradient-based methods . Our approach employs MAML as the meta-learning method to train across tasks . 2.1 FORMALIZATION . We first introduce stochastic games . Formally , a stochastic game , with N players , is defined as 〈N , S , { A1 , ... , AN } , T , { R1 , ... , RN } 〉 , where S is the set of states , Ai is the action set for player i , Ri : S×A1× ... ×AN → ∆ ( [ 0 , 1 ] ) is the reward function for player i ( ∆ ( C ) denotes the probability distributions over a set C ) , and T : S × A1 × ... × AN → ∆ ( S ) is the transition function . The goal for player i is to maximize its own cumulative reward E [ ∑T t=0 γ tri , t ] , where ri , t is the reward player i gets at time step t and γ is the discounting factor . πi : S → ∆ ( Ai ) denotes the policy for player i , which maps each state to a distribution over actions . Our agent takes the role of one player . With out loss of generality , we assume that our agent is player 1 . In this paper we only consider two-player stochastic games . Bayesian games further introduce types for players . Since our agent takes the role of player 1 , the different types of player 1 can be considered as different states . We only consider that player 2 has a set of types Θ . A specific opponent playing player 2 has its own θ ∈ Θ which is unknown to player 1 . There exists a prior distribution p ( θ ) over the population of opponents . Under this setting , each θ ∈ Θ has its corresponding reward function Ri ( s , a1 , a2 , θ ) for each i ∈ { 1 , 2 } , s ∈ S , a1 ∈ A1 and a2 ∈ A2 . Therefore combining the above concepts we formalize our tasks as Bayesian stochastic games 〈N , S , { A1 , ... , AN } , T , { R1 , ... , RN } , Θ , p〉 , where R1 , ... , RN are the modified reward functions dependent on θ ∈ Θ and p is the prior distribution over Θ for types of player 2 . 2.2 MODEL-AGNOSTIC META-LEARNING . Meta-learning considers the goal that aims to quickly train a model for a new task with the help of data on many similar tasks . Formally , we denote { T } N1i=1 the given N1 tasks used for training . Then N2 more new tasks { T } N1+N2i=N1+1 are used for testing . In the meta-learning setting , we assume that the tasks are sampled from a distribution p ( T ) over all possible tasks . For each task Ti , we want to identify a mapping fi , that maps each input x to its corresponding output y. Model-Agnostic Meta-Learning ( MAML ) is one of the best meta-learning algorithms that can be applied to models that are trained with gradient descent . Denote the parameter for each fi as ψ′i . The loss for fψ′i on task Ti is denoted as LTi ( fψ′i ) . A meta-learner with parameter ψ is used as the initialization for all ψ′is and the update for each specific task is : ψ′i = ψ − α∇ψLTi ( fψ ) , where α is the learning rate . The update for ψ is : ψ = ψ − β∇ψ N1∑ i=1 LTi ( fψ′i ) , where β is the learning rate for ψ . 3 OUR APPROACH . Before we dive into the Bayesian stochastic games , we first consider the stochastic game that we have a specific opponent with type θ ∈ Θ . We aim to explicitly model the opponent so that we can get predictions about it . The predicted value should be some character of the opponent that can help our agent improve policy . For example , in games where our reward is related to the final goal of the opponent , we can directly predict its goal . Formally , at some state st ∈ S , our agent predicting some estimated value ṽ ( ·|st , θ ) , where ṽ is the estimation of value v. v represent a character of the opponent such as goals , next actions or next positions . For convenience , we use vθ to denote v ( ·|θ ) . Then our agent can choose an action according to π1 ( ·|st , ṽθ ( st ) ) , while an agent unaware of its opponent can only take action according to π1 ( ·|st ) . Thus , the task has been divided into two subtasks : modeling ṽθ of the opponent and learning a policy π1 ( ·|ṽθ ) . The latter task can be considered as a general RL problem and we can apply various RL methods to solve it . Now assume that opponents have a prior distribution for their types . That is , an opponent with type θ ∈ Θ can be treated as a sample following θ ∼ p ( θ ) . If we can collect data from some opponents sampled from p , it is possible we can generalize the model and policy to the game with a new opponent . Thus we can consider the opponent modeling and policy learning as two metalearning tasks . The former can be considered as an imitation learning or supervised learning task while the second is a RL task . Both can apply MAML for meta learning . Since the learned policy needs the model to make prediction , we can not train the two meta-tasks independently . We jointly training the model and policy with some given opponents . We call our method as Meta-OpponentAgent learning ( MOA ) to indicate that the model and policy are jointly trained via meta-learning . Finally when a new opponent comes , we initialize our model and policy with the meta-model and meta-policy . The training procedure is shown in the figure 1 . 3.1 OPPONENT MODELING . The opponent in our game is considered as some player that won ’ t adapt its policy to our agent . This assumption can be true in many practical situations . For example , the consumers of a business man usually have stable preferences . Then we further assume that there exist a distribution over the policies of the opponents . This assumption can also be considered true for the business man situation , where a specific consumer is just a sample from the whole population . Our goal here is to model the current opponent with the help of data that are collected by playing with other opponents . Formally , we now aim to model the value function v for the opponent from the observations . As mentioned above , v here can represent many forms of characters of the opponent , such as the final goal , next action , next position or any other character we wish to predict . For each type θ ∼ p , the value function is specified as vθ . Now we have M opponents sampled from the population . Opponent i has a type θi ∼ p ( θ ) . For any given state s as an input for the opponent i , the model outputs ṽθi ( s ) . The task of minimizing ṽθi and vθi are treated as task T oi = { ( s , vθi ( s ) } with loss function LT oi = dist ( vθi , ṽθi ) , where dist ( · , · ) is the distance metric for two v functions . The distance function can vary for different problems . Following the framework of MAML , we use a network called opponent network ( OPNet ) f with parameter ψ to learn the function ṽ . Then for each opponent , we collect data Si = { ( s , vθi ( s ) } and then use this dataset to update ψ to get the adapted parameter ψ′i . New dataset S ′ i = { ( s , vθi ( s ) } is collected with fψ′i . { S ′ i } Mi=1 are together used to update ψ . Finally we use the learned ψ to initialize the parameter for the current opponent as fψ . The updating process for parameters follows the framework of MAML . Following Grant et al . ( 2018 ) , we are learning an empirical Bayesian model over the opponent population after training with the M opponents . When data for the new opponent are observed , it is easier to adapt to the new opponent with such a prior model .

Summary: The paper presents an approach to multi-agent learning based on the framework of model-agnostic meta learning. The originality of the approach lies in the decomposition of the policy in two terms, with applications to opponent modeling: the first part of the policy tries to predict some important characteristic of the agent (the characteristic itself is prior knowledge, the value it takes for a particular opponent is learnt from observations). The second part of the policy takes the estimated characteristic of the opponent as input, the current state and produces the action. All networks are trained within the MAML framework. The overall approach is motivated by the task of opponent modeling for multi-agent RL.