Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning

1 INTRODUCTION . In the long journey of creating artificial intelligent ( AI ) that mimics human intelligence , a hallmark of an AI agent is its capabilities of understanding and interacting with other agents ( Lake et al. , 2017 ) . At the cognitive level , the real-world intelligent entities ( e.g . rats , humans ) are born to be able to reason about various properties of interests of others ( Tolman , 1948 ; Pfeiffer & Foster , 2013 ) . Those interests usually indicates unobservable mental state including desires , beliefs , and intentions ( Premack & Woodruff , 1978 ; Gopnik & Wellman , 1992 ) . In everyday life , people use this inborn ability to reason about others ’ behaviors ( Gordon , 1986 ) , plan effective interactions ( Gallese & Goldman , 1998 ) , or match with the folk psychology ( Dennett , 1991 ) . It is known that people can use this reasoning ability recursively ; that is , they engage in considering what others believe about their own beliefs . A number of human social behaviors have been profiled by the recursion reasoning ability ( Pynadath & Marsella , 2005 ) . Behavioral game theorist and experimental psychologist believe that reasoning recursively is a tool of human cognition that is equipped with evolutionary advantage ( Camerer et al. , 2004 ; 2015 ; Goodie et al. , 2012 ; Robalino & Robson , 2012 ) . Traditional approach of constructing the models of other agents , also known as opponent modeling , has a rich history in the multi-agent learning ( Shoham et al. , 2007 ; Albrecht & Stone , 2018 ) . Even though equipped with modern machine learning methods that could enrich the representation of the opponent ’ s behaviors ( He et al. , 2016 ) , those algorithms tend to only work either under limited types of scenarios ( e.g . mean-field games ( Yang et al. , 2018 ) ) , pre-defined opponent strategies ( e.g . Tit-fot-Tat in iterated Prisoner ’ s Dilemma ( Foerster et al. , 2018 ) ) , or in cases where opponents are assumed to constantly return to the same strategy ( Da Silva et al. , 2006 ) . Recently , a promising methodology from game theory – recursive reasoning – has become popular in opponent modeling ∗The first two authors have equal contributions . Correspondence to Jun Wang . ( Gmytrasiewicz & Durfee , 2000 ; Camerer et al. , 2004 ; Gmytrasiewicz & Doshi , 2005 ; De Weerd et al. , 2013b ) . Similar to the way of thinking from humans , recursive reasoning refers to the belief reasoning process where each agent considers the reasoning process of other agents , based on which it expects to make better decisions . Importantly , it allows an opponent to reason about the modeling agent rather than being a fixed type ; the process can therefore be nested in a form as `` I believe that you believe that I believe ... '' . Despite some initial trails ( Gmytrasiewicz & Doshi , 2005 ; Von Der Osten et al. , 2017 ) , there has been little work that tries to adopt this idea into the multi-agent deep reinforcement learning ( DRL ) setting . One main reason is that computing the optimal policy is prohibitively expensive ( Doshi & Gmytrasiewicz , 2006 ; Seuken & Zilberstein , 2008 ) . In this paper , we introduce a probabilistic recursive reasoning ( PR2 ) framework for multi-agent DRL tasks . Unlike previous work on opponent modeling , each agent here is to consider how the opponents would react to its potential behaviors , before it tries to find the best response for its own decision making . By employing variational Bayes methods to model the uncertainty of opponents ’ conditional policies , we develop decentralized-training-decentralized-execution algorithms , PR2-Q and PR2-Actor-Critic , and prove the convergence in the self-play scenario when there is only one Nash equilibrium . Our methods are tested on the matrix game and the differential game . The games come with a non-trivial equilibrium where conventional gradient-based methods find challenging . We compare against multiple strong baselines . The results justify the unique value provided by agent ’ s recursive reasoning capability throughout the learning . We expect our work to offer a new angel on incorporating conditional opponent modeling into the multi-agent DRL context . 2 RELATED WORK . Game theorists take initiatives in modeling the recursive reasoning procedures ( Harsanyi , 1962 ; 1967 ) . Since then , alternative approaches , including logics-based models ( Bolander & Andersen , 2011 ; Muise et al. , 2015 ) or graphical models ( Doshi et al. , 2009 ; Gal & Pfeffer , 2003 ; 2008 ) , have been adopted . Recently , the idea of Theory of Mind ( ToM ) ( Goldman et al. , 2012 ) from cognitive science becomes popular . An example of ToM is the `` Recursive Modeling Method '' ( RMM ) ( Gmytrasiewicz et al. , 1991 ; Gmytrasiewicz & Durfee , 1995 ; 2000 ) , which incorporates the agent ’ s uncertainty about opponent ’ s exact model , payoff , and recursion depth . However , these methods follow the decisiontheoretic approaches , and are studied in the limited context of one-shot games . The environment is relatively simple and the opponents are not RL agents . The Interactive POMDP ( I-POMDP ) ( Gmytrasiewicz & Doshi , 2005 ) implements the idea of ToM to tackle the multi-agent RL problems . It extends the partially observed MDP ( Sondik , 1971 ) by introducing an extra space of models of other agents into the MDP ; as such , an agent can build belief models about how it believes other agents know and believe . Despite the added flexibility , I-POMDP has limitations in its solvability ( Seuken & Zilberstein , 2008 ) . Solving I-POMDP with N models in each recursive level with K maximum level equals to solving O ( NK ) PODMPs . Such inherent complexity requires high precision on the approximation solution methods , including particle filtering ( Doshi & Gmytrasiewicz , 2009 ) , value iteration ( Doshi & Perez , 2008 ) , or policy iteration ( Sonu & Doshi , 2015 ) . Out work is different from I-POMDP in that we do not adjust the MDP ; instead , we provide a probabilistic framework to implement the recursive reason in the MDP . We approximate the opponent ’ s conditional policy through variational Bayes methods . The induced PR2-Q and PR2-AC algorithms are model-free and can practically be used as the replacement to other multi-agent RL algorithms such as MADDPG ( Lowe et al. , 2017 ) . Our work can also be tied into the study of opponent modeling ( OM ) Albrecht & Stone ( 2018 ) . OM is all about shaping the anticipated movements of the other agents . Traditional OM can be regarded as level-0 recursive reasoning in that OM methods model how the opponent behaves based on the history , but not how the opponent would behave based on what I would behave . In general , OM methods have two major limitations . One is that OM tends to work with a pre-defined target of opponents ; for example , fictitious play ( Brown , 1951 ) and joint-action learners ( Claus & Boutilier , 1998 ) require opponents play stationary strategies , Nash-Q ( Hu & Wellman , 2003 ) require all agents play towards the Nash equilibrium , so do Correlated Q-learning ( Greenwald et al. , 2003 ) , Minimax-Q ( Littman , 1994 ) , and Friend-or-foe Q ( Littman , 2001 ) . These algorithms become invalid if the opponents change their types of policy . The other major limitation is that OM algorithms require to know the exact ( Nash ) equilibrium policy of the opponent during training . Typical examples include the series of WoLF models ( Bowling , 2005 ; Bowling & Veloso , 2001a ; 2002 ) or the Nash-Q learning ( Hu & Wellman , 2003 ) , both of which require the Nash Equilibrium at each stage game to update the Q-function . By contrast , our proposed methods , PR2-Q & PR2-AC , do not need to pre-define the type of the opponents . Neither do our methods require to know the equilibrium beforehand . Despite the recent success of applying deep RL algorithms on the single-agent discrete ( Mnih et al. , 2015 ) and continuous ( Lillicrap et al. , 2015 ) control problems , it is still challenging to transfer these methods into the multi-agent RL context . The reason is because learning independently while ignoring the others in the environment will simply break the theoretical guarantee of convergence ( Tuyls & Weiss , 2012 ) . A modern framework is to maintain a centralized critic ( i.e . Q-network ) during training , e.g . MADDPG ( Lowe et al. , 2017 ) , BiCNet ( Peng et al. , 2017 ) , and multi-agent soft Q-learning ( Wei et al. , 2018 ) ; however , they require strong assumptions that the parameters of agent policies are fully observable , letting alone the centralized Q-network potentially prohibits the algorithms from scaling up . By contrast , our approach employs decentralized training with no need to maintain a central critic ; neither does it require to know the exact opponents ’ policies . 3 PRELIMINARIES . For an n-agent stochastic game ( Shapley , 1953 ) , we define a tuple ( S , A1 , . . . , An , r1 , . . . , rn , p , γ ) , where Sdenotes the state space , p is the distribution of the initial state , γ is the discount factor for future rewards , Ai and r i = r i ( s , ai , a−i ) are the action space and the reward function for agent i ∈ { 1 , . . . , n } respectively . Agent i chooses its action ai ∈ Ai according to the policy πiθ i ( a i|s ) parameterized by θi conditioning on some given state s ∈ S. Let us define the joint policy as the collection of all agents ’ policies πθ with θ representing the joint parameter . It is convenient to interpret the joint policy from the perspective of agent i such that πθ = ( πiθ i ( a i|s ) , π−i θ−i ( a−i|s ) ) , where a−i = ( a j ) j 6=i , θ−i = ( θ j ) j 6=i , and π−iθ−i ( a −i|s ) is a compact representation of the joint policy of all complementary agents of i . At each stage of the game , actions are taken simultaneously . Each agent is presumed to pursue the maximal cumulative reward ( Sutton et al. , 1998 ) , expressed as max ηi ( πθ ) = E [ ∞∑ t=1 γtr i ( st , ait , a −i t ) ] , ( 1 ) with ( ait , a −i t ) sample from ( πiθ i , π −i θ−i ) . Correspondingly , for the game with ( infinite ) time horizon , we can define the state-action Q-function by Qiπθ ( st , a i t , a −i t ) = E [ ∑∞ l=0 γ lr i ( st+l , ait+l , a −i t+l ) ] . 3.1 NON-CORRELATED FACTORIZATION ON THE JOINT POLICY . In the multi-agent learning tasks , each agent can only control its own action ; however , the resulting reward value depends on other agents ’ actions . The Q-function of each agent , Qiπθ , is subject to the joint policy πθ consisting of all agents ’ policies . One common approach is to decouple the joint policy assuming conditional independence of actions from different agents ( Albrecht & Stone , 2018 ) : πθ ( ai , a−i|s ) = πiθ i ( a i|s ) π−i θ−i ( a−i|s ) . ( 2 ) The study regarding the topic of “ centralized training with decentralized execution ” in the deep RL domain , including MADDPG ( Lowe et al. , 2017 ) , COMA ( Foerster et al. , 2017 ) , MF-AC ( Yang et al. , 2018 ) , Multi-Agent Soft-Q ( Wei et al. , 2018 ) , and LOLA ( Foerster et al. , 2018 ) , can be classified into this category ( see more clarifications in Appendix B ) . Although the non-correlated factorization of the joint policy simplifies the algorithm , this simplication is vulnerable because it ignores the agents ’ connections , e.g . impacts of one agent ’ s action on other agents , and the subsequent reactions from other agents . One might argue that during training , the joint Q-function should potentially guide each agent to learn to consider and act for the mutual interests of all the agents ; nonetheless , a counter-example is that the non-correlated policy could not even solve the simplest two-player zero-sum differential game where two agents act in x and y with the reward functions defined by ( xy , −xy ) . In fact , by following Eq . 2 , both agents are reinforced to trace a cyclic trajectory that never converge to the equilibrium ( Mescheder et al. , 2017 ) . It is worth clarifying that the idea of non-correlated policy is still markedly different from the independent learning ( IL ) . IL is a naive method that completely ignore other agents ’ behaviors . The objective of agent i is simplified to ηi ( πθ i ) , depending only on i ’ s own policy πθ i compared to Eq . 1 . As Lowe et al . ( 2017 ) has pointed out , in IL , the probability of taking a gradient step in the correct direction decreases exponentially with the increasing number of agents , letting alone the major issue of the non-stationary environment due to the independence assumption ( Tuyls & Weiss , 2012 ) .

Summary: The paper proposes a new approach for fully decentralized training in multi-agent reinforcement learning, termed probabilistic recursive reasoning (PR2). The key idea is to build agent policies that take into account opponent best responses to each of the agent's potential actions, in a probabilistic sense. The authors show that such policies can be seen as recursive reasoning, prove convergence of the proposed method in self-play, a demonstrate it in a couple of iterated normal form games with non-trivial Nash equilibria where baselines fail to converge.