Title: PROBABILISTIC RECURSIVE REASONING FOR MULTI-AGENT REINFORCEMENT LEARNING

Abstract: Humans are capable of attributing latent mental contents such as beliefs, or intentions to others. The social skill is critical in everyday life to reason about the potential consequences of their behaviors so as to plan ahead. It is known that humans use this reasoning ability recursively, i.e. considering what others believe about their own beliefs. In this paper, we start from level-1 recursion and introduce a probabilistic recursive reasoning (PR2) framework for multi-agent reinforcement learning. Our hypothesis is that it is beneficial for each agent to account for how the opponents would react to its future behaviors. Under the PR2 framework, we adopt variational Bayes methods to approximate the opponents' conditional policy, to which each agent finds the best response and then improve their own policy. We develop decentralized-training-decentralized-execution algorithms, PR2-Q and PR2-Actor-Critic, that are proved to converge in the self-play scenario when there is one Nash equilibrium. Our methods are tested on both the matrix game and the differential game, which have a non-trivial equilibrium where common gradient-based methods fail to converge. Our experiments show that it is critical to reason about how the opponents believe about what the agent believes. We expect our work to contribute a new idea of modeling the opponents to the multi-agent reinforcement learning community.

1. INTRODUCTION:

In the long journey of creating artificial intelligent (AI) that mimics human intelligence, a hallmark of an AI agent is its capabilities of understanding and interacting with other agents (Lake et al., 2017) . At the cognitive level, the real-world intelligent entities (e.g. rats, humans) are born to be able to reason about various properties of interests of others (Tolman, 1948; Pfeiffer & Foster, 2013) . Those interests usually indicates unobservable mental state including desires, beliefs, and intentions (Premack & Woodruff, 1978; Gopnik & Wellman, 1992) . In everyday life, people use this inborn ability to reason about others' behaviors (Gordon, 1986) , plan effective interactions (Gallese & Goldman, 1998) , or match with the folk psychology (Dennett, 1991) . It is known that people can use this reasoning ability recursively; that is, they engage in considering what others believe about their own beliefs. A number of human social behaviors have been profiled by the recursion reasoning ability (Pynadath & Marsella, 2005) . Behavioral game theorist and experimental psychologist believe that reasoning recursively is a tool of human cognition that is equipped with evolutionary advantage (Camerer et al., 2004; 2015; Goodie et al., 2012; Robalino & Robson, 2012) .

Traditional approach of constructing the models of other agents, also known as opponent modeling, has a rich history in the multi-agent learning (Shoham et al., 2007; Albrecht & Stone, 2018) . Even though equipped with modern machine learning methods that could enrich the representation of the opponent's behaviors (He et al., 2016) , those algorithms tend to only work either under limited types of scenarios (e.g. mean-field games (Yang et al., 2018) ), pre-defined opponent strategies (e.g. Tit-fot-Tat in iterated Prisoner's Dilemma (Foerster et al., 2018) ), or in cases where opponents are assumed to constantly return to the same strategy (Da Silva et al., 2006) . Recently, a promising methodology from game theory -recursive reasoning -has become popular in opponent modeling (Gmytrasiewicz & Durfee, 2000; Camerer et al., 2004; Gmytrasiewicz & Doshi, 2005; De Weerd et al., 2013b) . Similar to the way of thinking from humans, recursive reasoning refers to the belief reasoning process where each agent considers the reasoning process of other agents, based on which it expects to make better decisions. Importantly, it allows an opponent to reason about the modeling agent rather than being a fixed type; the process can therefore be nested in a form as "I believe that you believe that I believe ...". Despite some initial trails (Gmytrasiewicz & Doshi, 2005; Von Der Osten et al., 2017) , there has been little work that tries to adopt this idea into the multi-agent deep reinforcement learning (DRL) setting. One main reason is that computing the optimal policy is prohibitively expensive (Doshi & Gmytrasiewicz, 2006; Seuken & Zilberstein, 2008) .

In this paper, we introduce a probabilistic recursive reasoning (PR2) framework for multi-agent DRL tasks. Unlike previous work on opponent modeling, each agent here is to consider how the opponents would react to its potential behaviors, before it tries to find the best response for its own decision making. By employing variational Bayes methods to model the uncertainty of opponents' conditional policies, we develop decentralized-training-decentralized-execution algorithms, PR2-Q and PR2-Actor-Critic, and prove the convergence in the self-play scenario when there is only one Nash equilibrium. Our methods are tested on the matrix game and the differential game. The games come with a non-trivial equilibrium where conventional gradient-based methods find challenging. We compare against multiple strong baselines. The results justify the unique value provided by agent's recursive reasoning capability throughout the learning. We expect our work to offer a new angel on incorporating conditional opponent modeling into the multi-agent DRL context.

2. RELATED WORK:

Game theorists take initiatives in modeling the recursive reasoning procedures (Harsanyi, 1962; 1967) . Since then, alternative approaches, including logics-based models (Bolander & Andersen, 2011; Muise et al., 2015) or graphical models (Doshi et al., 2009; Gal & Pfeffer, 2003; 2008) , have been adopted. Recently, the idea of Theory of Mind (ToM) (Goldman et al., 2012) from cognitive science becomes popular. An example of ToM is the "Recursive Modeling Method" (RMM) (Gmytrasiewicz et al., 1991; Gmytrasiewicz & Durfee, 1995; 2000) , which incorporates the agent's uncertainty about opponent's exact model, payoff, and recursion depth. However, these methods follow the decisiontheoretic approaches, and are studied in the limited context of one-shot games. The environment is relatively simple and the opponents are not RL agents.

The Interactive POMDP (I-POMDP) (Gmytrasiewicz & Doshi, 2005) implements the idea of ToM to tackle the multi-agent RL problems. It extends the partially observed MDP (Sondik, 1971) by introducing an extra space of models of other agents into the MDP; as such, an agent can build belief models about how it believes other agents know and believe. Despite the added flexibility, I-POMDP has limitations in its solvability (Seuken & Zilberstein, 2008) . Solving I-POMDP with N models in each recursive level with K maximum level equals to solving O(N K ) PODMPs. Such inherent complexity requires high precision on the approximation solution methods, including particle filtering (Doshi & Gmytrasiewicz, 2009) , value iteration (Doshi & Perez, 2008) , or policy iteration (Sonu & Doshi, 2015) . Out work is different from I-POMDP in that we do not adjust the MDP; instead, we provide a probabilistic framework to implement the recursive reason in the MDP. We approximate the opponent's conditional policy through variational Bayes methods. The induced PR2-Q and PR2-AC algorithms are model-free and can practically be used as the replacement to other multi-agent RL algorithms such as MADDPG (Lowe et al., 2017) .

Our work can also be tied into the study of opponent modeling (OM) Albrecht & Stone (2018) . OM is all about shaping the anticipated movements of the other agents. Traditional OM can be regarded as level-0 recursive reasoning in that OM methods model how the opponent behaves based on the history, but not how the opponent would behave based on what I would behave. In general, OM methods have two major limitations. One is that OM tends to work with a pre-defined target of opponents; for example, fictitious play (Brown, 1951) and joint-action learners (Claus & Boutilier, 1998) require opponents play stationary strategies, Nash-Q (Hu & Wellman, 2003) require all agents play towards the Nash equilibrium, so do Correlated Q-learning (Greenwald et al., 2003) , Minimax-Q (Littman, 1994), and Friend-or-foe Q (Littman, 2001) . These algorithms become invalid if the opponents change their types of policy. The other major limitation is that OM algorithms require to know the exact (Nash) equilibrium policy of the opponent during training. Typical examples include the series

: ⇡(a i , a i |s) ⇡ i (a i |s)⇢ i (a i |s, a i ) < l a t e x i t s h a 1 _ b a s e 6 4 = " L l 3 7 7 v l p y n L G x T h Y m n p + 5 7 r c o X I = " > A A A C L n i c b V D L S g M x F L 3 j 2 / q q u n Q T b I U K W m b c 6 L K g g k s F q 0 K n r X f S j A 1 m J k O S U c r Y L 3 L j r + h C U B G 3 f o b p V M H X g c D h n H O 5 u S d I B N f G d Z + c k d G x 8 Y n J q e n C z O z c / E J x c e l E y 1 R R V q d S S H U W o G a C x 6 x u u B H s L F E M o 0 C w 0 + B y d + C f X j G l u Y y P T S 9 h z Q g v Y h 5 y i s Z K 7 e J + 2 U 9 4 B V t 8 g 2 A r 2 + T 9 G 7 1 O f M F C g 0 r J a 2 L d V u 5 b 3 V d d m W c q X 9 E N S 3 h / v d w u l t y q m 4 P 8 J d 4 n K d X K Y X g O A I f t 4 o P f k T S N W G y o Q K 0 b n p u Y Z o b K c C p Y v + C n m i V I L / G C N S y N M W K 6 m e X n 9 s m a V T o k l M q + 2 J B c / T 6 R Y a R 1 L w p s M k L T 1 b + 9 g f i f 1 0 h N u N P M e J y k h s V 0 u C h M B T G S D L o j H a 4 Y N a J n C V L F 7 V 8 J 7 a J C a m z D B V u C 9 / v k v + R k q + q 5 V e / I K 9 X 2 Y I g p W I F V q I A H 2 1 C D A z i E O l C 4 h X t 4 h h f n z n l 0 X p 2 3 Y X T E + Z x Z h h 9 w 3 j 8 A Z x m p q g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " K z w B o 9 Q a h h k L S T 4 c W I S f r 9 R 8 G j M = " > A A A C L n i c b V D L S g M x F M 3 U V 6 2 v q k s 3 w V a o o G X G j S 4 L K r i s Y B / Q a c u d N G N D M 5 M h y S h l 7 B e 5 8 V d 0 I a i I W z / D d F p B q w c C h 3 P O 5 e Y e L + J M a d t + s T J z 8 w u L S 9 n l 3 M r q 2 v p G f n O r r k Q s C a 0 R w Y V s e q A o Z y G t a a Y 5 b U a S Q u B x 2 v A G p 2 O / c U O l Y i K 8 0 s O I t g O 4 D p n P C G g j d f P n R T d i J e i w A w y d 5 J C N 7 t Q + d j n 1 N U g p b r F x O 6 l v d F f 2 R Z o p f U c P D G G j / W I 3 X 7 D L d g r 8 l z h T U k B T V L v 5 J 7 c n S B z Q U B M O S r U c O 9 L t B K R m h N N R z o 0 V j Y A M 4 J q 2 D A 0 h o K q d p O e O 8 J 5 R e t g X 0 r x Q 4 1 T 9 O Z F A o N Q w 8 E w y A N 1 X s 9 5 Y / M 9 r x d o / a S c s j G J N Q z J Z 5 M c c a 4 H H 3 e E e k 5 R o P j Q E i G T m r 5 j 0 Q Q L R p u G c K c G Z P f k v q R + V H b v s X D q F y t m 0 j i z a Q b u o h B x 0 j C r o A l V R D R F 0 j x 7 R K 3 q z H q x n 6 9 3 6 m E Q z 1 n R m G / 2 C 9 f k F S t 2 o G g = = < / l a t e x i t > ⇡(a i , a i |s) ⇡ i (a i |s)⇢ i (a i |s, a i )

< l a t e x i t s h a 1 _ b a s e 6 4 = " L l 3 7 7 v l p y n L G x T h Y m n p + 5 7 r c o X

I = " > A A A C L n i c b V D L S g M x F L 3 j 2 / q q u n Q T b I U K W m b c 6 L K g g k s F q 0 K n r X f S j A 1 m J k O S U c r Y L 3 L j r + h C U B G 3 f o b p V M H X g c D h n H O 5 u S d I B N f G d Z + c k d G x 8 Y n J q e n C z O z c / E J x c e l E y 1 R R V q d S S H U W o G a C x 6 x u u B H s L F E M o 0 C w 0 + B y d + C f X j G l u Y y P T S 9 h z Q g v Y h 5 y i s Z K 7 e J + 2 U 9 4 B V t 8 g 2 A r 2 + T 9 G 7 1 O f M F C g 0 r J a 2 L d V u 5 b 3 V d d m W c q X 9 E N S 3 h / v d w u l t y q m 4 P 8 J d 4 n K d X K Y X g O A I f t 4 o P f k T S N W G y o Q K 0 b n p u Y Z o b K c C p Y v + C n m i V I L / G C N S y N M W K 6 m e X n 9 s m a V T o k l M q + 2 J B c / T 6 R Y a R 1 L w p s M k L T 1 b + 9 g f i f 1 0 h N u N P M e J y k h s V 0 u C h M B T G S D L o j H a 4 Y N a J n C V L F 7 V 8 J 7 a J C a m z D B V u C 9 / v k v + R k q + q 5 V e / I K 9 X 2 Y I g p W I F V q I A H 2 1 C D A z i E O l C 4 h X t 4 h h f n z n l 0 X p 2 3 Y X T E + Z

x Z h h 9 w 3 j 8 A Z x m p q g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " K z w

B o 9 Q a h h k L S T 4 c W I S f r 9 R 8 G j M = " > A A A C L n i c b V D L S g M x F M 3 U V 6 2 v q k s 3 w V a o o G X G j S 4 L K r i s Y B / Q a c u d N G N D M 5 M h y S h l 7 B e 5 8 V d 0 I a i I W z / D d F p B q w c C h 3 P O 5 e Y e L + J M a d t + s T J z 8 w u L S 9 n l 3 M r q 2 v p G f n O r r k Q s C a 0 R w Y V s e q A o Z y G t a a Y 5 b U a S Q u B x 2 v A G p 2 O / c U O l Y i K 8 0 s O I t g O 4 D p n P C G g j d f P n R T d i J e i w A w y d 5 J C N 7 t Q + d j n 1 N U g p b r F x O 6 l v d F f 2 R Z o p f U c P D G G j / W I 3 X 7 D L d g r 8 l z h T U k B T V L v 5 J 7 c n S B z Q U B M O S r U c O 9 L t B K R m h N N R z o 0 V j Y A M 4 J q 2 D A 0 h o K q d p O e O 8 J 5 R e t g X 0 r x Q 4 1 T 9 O Z F A o N Q w 8 E w y A N 1 X s 9 5 Y / M 9 r x d o / a S c s j G J N Q z J Z 5 M c c a 4 H H 3 e E e k 5 R o P j Q E i G T m r 5 j 0 Q Q L R p u G c K c G Z P f k v q R + V H b v s X D q F y t m 0 j i z a Q b u o h B x 0 j C r o A l V R D R F 0 j x 7 R K 3 q z H q x n 6 9 3 6 m E Q z 1 n R m G / 2 C 9 f k F S t 2 o G g = = < / l a t e x i t > Opponent Effect Best Response 1 2 3 4

Figure 1 : Probabilistic recursive reasoning framework. PR2 decouples the connections between agents by Eq. 3. 1 : agent i takes the best response after considering all the potential consequences of opponents' actions given its own action a i . 2 : how agent i behaves in the environment serves as the prior for the opponents to learn how their actions would affect a i . 3 : similar to 1 , opponents take the best response to agent i. 4 : similar to 2 , opponents' actions are the prior knowledge to agent i on estimating how a i will affect the opponents. Looping from step 1 to 4 forms recursive reasoning.

of WoLF models (Bowling, 2005; Bowling & Veloso, 2001a; 2002) or the Nash-Q learning (Hu & Wellman, 2003) , both of which require the Nash Equilibrium at each stage game to update the Q-function. By contrast, our proposed methods, PR2-Q & PR2-AC, do not need to pre-define the type of the opponents. Neither do our methods require to know the equilibrium beforehand.

Despite the recent success of applying deep RL algorithms on the single-agent discrete (Mnih et al., 2015) and continuous (Lillicrap et al., 2015) control problems, it is still challenging to transfer these methods into the multi-agent RL context. The reason is because learning independently while ignoring the others in the environment will simply break the theoretical guarantee of convergence (Tuyls & Weiss, 2012) . A modern framework is to maintain a centralized critic (i.e. Q-network) during training, e.g. MADDPG (Lowe et al., 2017) , BiCNet (Peng et al., 2017) , and multi-agent soft Q-learning (Wei et al., 2018) ; however, they require strong assumptions that the parameters of agent policies are fully observable, letting alone the centralized Q-network potentially prohibits the algorithms from scaling up. By contrast, our approach employs decentralized training with no need to maintain a central critic; neither does it require to know the exact opponents' policies.

3. PRELIMINARIES:

For an n-agent stochastic game (Shapley, 1953) , we define a tuple (S, A 1 , . . . , A n , r 1 , . . . , r n , p, γ), where S denotes the state space, p is the distribution of the initial state, γ is the discount factor for future rewards, A i and r i = r i (s, a i , a -i ) are the action space and the reward function for agent i ∈ {1, . . . , n} respectively. Agent i chooses its action a i ∈ A i according to the policy π i θ i (a i |s) parameterized by θ i conditioning on some given state s ∈ S. Let us define the joint policy as the collection of all agents' policies π θ with θ representing the joint parameter. It is convenient to interpret the joint policy from the perspective of agent i such that

π θ = (π i θ i (a i |s), π -i θ -i (a -i |s))

, where a -i = (a j ) j =i , θ -i = (θ j ) j =i , and π -i θ -i (a -i |s) is a compact representation of the joint policy of all complementary agents of i. At each stage of the game, actions are taken simultaneously. Each agent is presumed to pursue the maximal cumulative reward (Sutton et al., 1998) , expressed as

EQUATION

with

(a i t , a -i t ) sample from (π i θ i , π -i θ -i ).

Correspondingly, for the game with (infinite) time horizon, we can define the state-action Q-function by

Q i πθ (s t , a i t , a -i t ) = E ∞ l=0 γ l r i (s t+l , a i t+l , a -i t+l ) .

3.1. NON-CORRELATED FACTORIZATION ON THE JOINT POLICY:

In the multi-agent learning tasks, each agent can only control its own action; however, the resulting reward value depends on other agents' actions. The Q-function of each agent, Q i πθ , is subject to the joint policy π θ consisting of all agents' policies. One common approach is to decouple the joint policy assuming conditional independence of actions from different agents (Albrecht & Stone, 2018):

EQUATION

The study regarding the topic of "centralized training with decentralized execution" in the deep RL domain, including MADDPG (Lowe et al., 2017) , COMA (Foerster et al., 2017) , MF-AC (Yang et al., 2018) , Multi-Agent Soft-Q (Wei et al., 2018) , and LOLA (Foerster et al., 2018) , can be classified into this category (see more clarifications in Appendix B). Although the non-correlated factorization of the joint policy simplifies the algorithm, this simplication is vulnerable because it ignores the agents' connections, e.g. impacts of one agent's action on other agents, and the subsequent reactions from other agents. One might argue that during training, the joint Q-function should potentially guide each agent to learn to consider and act for the mutual interests of all the agents; nonetheless, a counter-example is that the non-correlated policy could not even solve the simplest two-player zero-sum differential game where two agents act in x and y with the reward functions defined by (xy, -x y). In fact, by following Eq. 2, both agents are reinforced to trace a cyclic trajectory that never converge to the equilibrium (Mescheder et al., 2017) .

It is worth clarifying that the idea of non-correlated policy is still markedly different from the independent learning (IL). IL is a naive method that completely ignore other agents' behaviors. The objective of agent i is simplified to η i (π θ i ), depending only on i's own policy π θ i compared to Eq. 1.

As Lowe et al. (2017) has pointed out, in IL, the probability of taking a gradient step in the correct direction decreases exponentially with the increasing number of agents, letting alone the major issue of the non-stationary environment due to the independence assumption (Tuyls & Weiss, 2012) .

4. MULTI-AGENT PROBABILISTIC RECURSIVE REASONING:

In the previous section, we have shown the weakness of the learning algorithms that build on the noncorrelated factorization on the joint policy. Here we introduce the probabilistic recursive reasoning approach that aims to capture how the opponents believe about what the agent believes. Under such setting, we devise a new multi-agent policy gradient theorem. We start from assuming the true opponent conditional policy π -i θ -i is given, and then move onward to the practical case where it is approximated through variational inference.

4.1. PROBABILISTIC RECURSIVE REASONING:

The issue on the non-correlated factorization is that it fails to help each agent to consider the consequence of its action on others, which could lead to the ill-posed behaviors in the multi-agent learning tasks. On the contrary, people explicitly attribute contents such as beliefs, desires, and intentions to others in daily life. It is known that human beings are capable of using this ability recursively to make decisions. Inspired by this, here we integrate the concept of recursive reasoning into the joint policy modeling, and propose the new probabilistic recursive reasoning (PR2) framework. Specifically, we employ the nested process of belief reasoning where each agent simulates the reasoning process of other agents, thinking about how its action would affect others, and then make actions based on such predictions. The process can be nested in a form as "I believe [that you believe (that I believe)]". Here we start from considering the level-1 recursion, as psychologist have found that humans tend to reason on average at one or two level of recursion (Camerer et al., 2004) , and levels higher than two do not provide significant benefits (De Weerd et al., 2013a; b; de Weerd et al., 2017) . Based on this, we re-formulate the joint policy by

π θ (a i , a -i |s) = π i θ i (a i |s)π -i θ -i (a -i |s, a i ) Agent i's perspective = π -i θ -i (a -i |s)π i θ i (a i |s, a -i )

The opponents' perspective .

(3) Similar ways of decomposition can also be found in dual learning (Xia et al., 2017) on machine translation. From the perspective of agent i, the first equality in Eq. 3 indicates that the joint policy can be essentially decomposed into two parts. The conditional part π -i θ -i (a -i |s, a i ) represents what actions would be taken by the opponents given the fact that the opponents know the current state of environment and agent i's action; this is based on what agent i believes other opponents might think about itself. Note that the way of thinking developed by agent i regarding how others would consider of itself is also shaped by opponents' original policy π -i θ -i (a -i |s), as this is also how the opponents actually act in the environment. Taking into account different potential actions that agent i thinks the opponents would take, agent i uses the marginal policy π i θ i (a i |s) to find the best response. To this end, a level-1 recursive procedure is established: a i → a -i → a i . The same inference logic can be applied to the opponents from their perspectives, as shown in the second equality of Eq. 3.

Albeit intuitive, Eq. 3 may not be practical due to the requirement on the full knowledge regarding the actual conditional policy π -i θ -i (a -i |s, a i ). A natural solution is that one approximates the actual policy via a best-fit model from a family of distributions. We denote this family as ρ -i φ -i (a -i |s, a i ) with learnable parameter φ -i . PR2 is probabilistic as it considers the uncertainty of modeling π -i θ -i (a -i |s, a i ). The reasoning structure is now established as shown in Fig. 1 . With the recursive joint policy defined in Eq. 3, the n-agent learning task can therefore be formulated as

a i a i Decentralized Execution s s a -i a -i π i π i π -i π -i Decentralized Training with Probabilistic Reasoning Q i Q i Q -i Q -i ρ -i ρ -i ρ i ρ i

EQUATION

arg max

EQUATION

With the new learning protocol defined in Eq. 4 and 5, each agent now learns its own policy as well as the approximated conditional policy of other agents given its own actions. In such a way, both the agent and the opponents can keep track of the joint policy by

π i θ i (a i |s)ρ -i φ -i (a -i |s, a i ) → π θ (a i , a -i |s) ← π -i θ -i (a -i |s)ρ i φ i (a i |s, a -i ).

Once converged, the resulting approximate satistfies:

π θ (a i , a -i |s) = π i θ i (a i |s)ρ -i φ -i (a -i |s, a i ) = π -i θ -i (a -i |s)ρ i φ i (a i |s, a -i

), according to Eq. 3.

4.2. PROBABILISTIC RECURSIVE REASONING POLICY GRADIENT:

Given the true opponent policy π -i θ -i and that each agent tries to maximize its cumulative return in the stochastic game with the objective defined in Eq. 1, we establish the policy gradient theorem by accounting for the PR2 joint policy decomposition in Eq. 3.

Proposition 1. In a stochastic game, under the recursive reasoning framework defined by Eq. 3, the update for the multi-agent recursive reasoning policy gradient method can be derived as follows:

EQUATION

Proof. See Appendix B.2.

Proposition 1 states that each agent should improve its policy toward the direction of the best response after it takes into account all kinds of possibilities of how other agents would react if that action is taken. The term of π -i θ -i (a -i |s, a i ) can be regarded as the posterior estimation of agent i's belief about how the opponents would respond to his action a i , given opponents' true policy π -i θ -i (a -i |s) serving as the prior. Note that compared to the direction of policy update in the conventional multiagent policy gradient theorem (Wei et al., 2018) ,

a -i π -i θ -i (a -i |s)Q i (s, a i , a -i ) da -i , the direction of the gradient update in PR2 is guided by the term a -i π -i θ -i (a -i |s, a i )Q i (s, a i , a -i ) da -i .

In practice, agent i might not have access to the opponents' actual policy parameters θ -i , it is often needed to approximate π -i θ -i (a -i |s, a i ) by ρ -i φ -i (a -i |s, a i ), thereby we propose Proposition 2. Proposition 2. In a stochastic game, under the recursive reasoning framework defined by Eq. 3, with the opponent policy approximated by ρ -i φ -i (a -i |s, a i ), the update for the multi-agent recursive reasoning policy gradient method can be formulated as follows:

EQUATION

Proof. Substituting the approximated model ρ -i φ -i (a -i |s, a i ) for the true policy π -i θ-i in Eq. 6.

Proposition 2 raises an important point: the difference between decentralized training (algorithms that do not require the opponents' policies) with centralized learning (algorithms that require the opponents' policies) can in fact be quantified by a term of importance weights, similar to the connection between on-policy and off-policy methods. If we find a best-fit approximation such that

ρ -i φ -i (a -i |s, a i ) → π -i θ -i (a -i |s, a i

), then Eq.7 collapses into Eq. 6. Based on Proposition 2, we could provide multi-agent PR2 learning algorithm. As illustrated in Fig. 2 , it is a decentralized-training-with-decentralized-execution algorithm. In this setting, agents share the experiences in the environment including state and historical joint actions, while each agent receive its rewards privately. Our method does not require the knowledge of other agents' policy parameters. We list the pseudo-code of PR2-AC and PR2-Q in Appendix A. Finally, one last piece missing is how to find the best-fit approximation of ρ -i φ -i (a -i |s, a i ).

4.3. VARIATIONAL INFERENCE ON OPPONENT CONDITIONAL POLICY:

We adopt an optimization-based approximation to infer the unobservable ρ -i φ -i (a -i |s, a i ) via variational inference (Jordan et al., 1999) . We first define the trajectory τ up to time t including the experiences of t consecutive time stages, i.e. τ = [(s 1 , a i 1 , a -i 1 ), . . . , (s t , a i t , a -i t )]. In the probabilistic reinforcement learning (Levine, 2018) , the probability of τ being generated can be derived as

EQUATION

Assuming the dynamics is fixed (i.e. the agent can not influence the environment transition probability), our goal is then to find the best approximation of π i θ i (a i t |s t )ρ -i φ -i (a -i t |s t , a i t ) such that the induced trajectory distribution p(τ) can match with the true trajectory probability p(τ):

EQUATION

In other words, we can optimize the opponents' policy ρ -i φ -i via minimizing the K L-divergence, i.e.

EQUATION

Besides the reward term, the objective introduces an additional term of the conditional entropy on the joint policy

H π i θ i a i t |s t ρ -i φ -i a -i |s t , a i t

that potentially promotes the explorations for both the agent i's best response and the opponents' conditional policy. Note that the entropy here is conditioning not only on the state s t but also on agent i's action. Minimizing Eq. 10 gives us: Theorem 1. The optimal Q-function for agent i that satisfies minimizing Eq. 10 is formulated as:

EQUATION

)

And the corresponding optimal opponent conditional policy reads:

EQUATION

Proof. See Appendix C. Theorem 1 states that the learning of ρ -i φ -i (a -i |s, a i ) can be further converted to minimizing the K L-divergence between the estimated policy ρ -i φ -i and the advantage function:

D KL ρ -i φ -i (a -i |s, a i ) exp(Q i (s, a i , a -i ) -Q i (s, a i )) .

We can obtain a solution to Eq. 12 by maintaining two Q-functions, and then iteratively update them. We prove the convergence under self-play when there is one equilibrium. This leads to a fixed-point iteration that resembles value iteration.

Theorem 2. In a symmetric game with only one equilibrium, and the equilibrium meets one of the conditions: 1) the global optimum, i.e.

E π * Q i t (s) ≥ E π Q i t (s) ; 2) a saddle point, i.e. E π * Q i t (s) ≥ E π i E π -i * Q i t (s) or E π * Q i t (s) ≥ E π i * E π -i Q i t (s)

; where Q * and π * are the equilibrium value function and policy, respectively. The PR2 soft value iteration operator defined by:

TQ i (s, a i , a -i ) r i (s, a i , a -i ) + γE s ,a i ∼ps,π i log a -i exp(Q i (s , a i , a -i )) da -i , (13)

is a contraction mapping.

Proof. See Appendix D.

4.4. SAMPLING IN CONTINUOUS ACTION SPACE:

In continuous controls, getting the actions from the opponent policy ρ -i φ -i (a -i |s, a i ) is challenging. In this work, we follow Haarnoja et al. (2017) to adopt the amortized Stein Variational Gradient Descent (SVGD) (Liu & Wang, 2016; Wang & Liu, 2016) in sampling from the soft Q-function. Compared to MCMC, Amortized SVGD is a computationally-efficient way to estimate ρ -i φ -i (a -i |s, a i ). Thanks to SVGD, agent i is able to reason about potential consequences of opponent bavhaviors

a -i π -i θ -i (a -i |s, a i )Q i (s, a i , a -i ) da -i

, and finally find the corresponding best response.

5. EXPERIMENTS:

We evaluate the performance of PR2 methods on the iterated matrix games, differential games, and particle world environment. Those games can by design have a non-trivial equilibrium that requires certain levels of intelligent reasonings between agents. We compared our algorithm with a series of baselines. In the matrix game, we compare against IGA (Infinitesimal Gradient Ascent) (Singh et al., 2000) . In the differential games, the baselines from multi-agent learning algorithms are MASQL (Multi-Agent Soft-Q) (Wei et al., 2018) and MADDPG (Lowe et al., 2017) . We also including independent learning algorithms implemented through DDPG (Lillicrap et al., 2015) . To compare against traditional method of opponent modeling, we include one baseline that is based on DDPG but with one additional opponent modeling unit that is trained in an online and supervised way to learn the most recent opponent policy, which is then fed into the critic. Similar approach has been implemented by Rabinowitz et al. (2018) in realizing machine theory of mind. Besides, we applied centralized Symplectic Gradient Adjustment (SGA) (Balduzzi et al., 2018) optimization for DDPG agents (DDPG-SGA), which has recently been found to help converge to a local equilibrium quickly.

For the experiment settings, all the policies and Q-functions are parameterized by the MLP with 2 hidden layers, each with 100 units ReLU activation. The sampling network ξ for the ρ -i φ -i in SGVD follows the standard normal distribution. In the iterated matrix game, we trained all the methods including the baselines for 500 iterations. In the differential game, we trained the agents for 350 iterations with 25 steps per iteration. For the actor-critic methods, we set the exploration noise to 0.1 in first 1000 steps, and the annealing parameters for PR2-AC and MASQL are set to 0.5 to balance between the exploration and acting as the best response. 

5.1. ITERATED MATRIX GAME:

In the matrix game, the payoffs are defined by: R 1 = 0 3 1 2 , and R 2 = 3 2 0 1 . These exists the only Nash Equilibrium at (0.5, 0.5). This game has been intensively investigated in multiagent studies (Bowling & Veloso, 2001a; b) . One reason is that in solving the Nash Equilibrium for this game, simply taking simultaneous gradient steps on both agent's value functions will present the rotational behaviors on the gradient vector field; this leads to an endlessly iterative change of behaviors. Without considering the consequence of one agent's action on the other agent beforehand, it is challenging for both players to find the equilibrium. Similar issue has been found on training the GANs (Goodfellow et al., 2014; Mescheder et al., 2017) The results are shown in Fig. 3 . As expected, IGA fails to converge to the equilibrium but rotate around the equilibrium point. On the contrary, our method can find precisely the central equilibrium with a fully distributed fashion (see Fig. 3b ). The convergence can also be justified by the agents' policies in Fig. 3c , and the opponent's policy that is maintained by each agent in Fig. 3d .

5.2. DIFFERENTIAL GAME:

We adopt the same differential game, the Max of Two Quadratic Game, as Panait et al. (2006) ; Wei et al. (2018) . The agents have continuous action space of [-10, 10] . Each agent's reward depends on the joint action following the equations:

r 1 a 1 , a 2 = r 2 a 1 , a 2 = max ( f 1 , f 2 ) , where f 1 = 0.8 × [-( a 1 +5 3 ) 2 -( a 2 +5 3 ) 2 ], f 2 = 1.0 × [-( a 1 -5 1 ) 2 -( a 2 -5 1 ) 2 ] + 10.

The task poses a great challenge to general gradient-based algorithms because gradient tends to points to the sub-optimal solution. The reward surface is shown in Fig. 4a ; there is a local maximum 0 at (-5, -5) and a global maximum 10 at (5, 5), with a deep valley staying in the middle. If the agents' policies are initialized to (0, 0) (the red starred point) that lies within the basin of the left local maximum, the gradient based methods would tend to fail to find the global maximum equilibrium point due to the valley blocking the upper right area. The pathology of finding a suboptimal Nash equilibrium is also called relative over-generalization (Wei & Luke, 2016) .

We present the results in Fig. 4b , PR2-AC shows superior performance that manages to converge to the global equilibrium, while all the other baselines fall into the local basin on the left, except that the MASQL has small chance to find the optimal point. On top of the convergence result, it is worth noting that as the temperature annealing is required for energy-based RL methods, the learning outcomes of MASQL are extremely sensitive to the way of annealing, i.e. when and how to anneal the temperature to a small value during training is non-trivial. However, our method does not need to tune the the annealing parameter at all because the each agent is acting the best response to the approximated conditional policy, considering all potential consequences of the opponent's response.

Interestingly, by comparing the learning path in Fig. 4a against Fig. 5(a-e ) where the scattered blue dots are the exploration trails at the beginning, we can tell that if the PR2-AC model finds the peak point in joint action space, the agents can quickly go through the shortcut out of the local basin in a clever way, while other algorithms just converge to the local equilibrium. This further justifies the effectiveness and benefits of conducting recursive reasoning with opponents. Apart from testing in the self-play setting, we also test the scenario when the opponent type is different. We pair PR2-AC with all four baseline algorithms in Fig. 5(f-i ). Similar result can be found, that is, algorithm that has the function of taking into account the opponents (i.e. DDPG_OM & MADDPG) can converge to the local equilibrium even though not global, while DDPG and MASQL completely fails due to the inborn defect from the independent learning methods.

5.3. PARTICLE WORLD ENVIRONMENTS:

We further test our method on the multi-state multi-player Particle World Environments (Lowe et al., 2017) . This includes four testing scenarios: 1) Cooperative Navigation with 3 agents and 3 landmarks. Agents are collectively rewarded based on the proximity of any agent to each landmark while avoiding collisions; 2) Physical Deception with 1 adversary, 2 good agents, and 2 landmarks. All agents observe the positions of landmarks and other agents. Only one landmark is the true target landmark. Good agents are rewarded based on how close any of them is to the target landmark, and how well they deceive the adversary; 3) Keep-away with 1 agent, 1 adversary, and 1 landmark. Agent is rewarded based on distance to landmark. Adversary is rewarded if it push away the agent from the landmark; 4) Predator-prey with 1 prey agent who moves faster try to run away from 3 adversary predator who move slower but are motivated to catch the prey cooperatively.

The PR2 methods are compared against a series of the centralized MARL methods in Fig. 6 . Under the fully-cooperative setting (the left plot), PR2AC achieves the best performance over all baselines, even though it is a decentralized algorithm that does not have access to the exact opponent policies. Under the competitive settings where PR2AC rivals against the a set of adversary baselines, we find that PR2AC learners can beat all the baselines, including DDPG, DDPG-OM, and MASQL. The only exception is MADDPG, as it is suggested by the drop-down arrow. PR2AC performs particularly bad on the physical deception task. We believe it is mainly because the centralized critic can access the full knowledge of the exact policies of PR2-AC, but PR2-AC cannot access the models of its opponents in the reversed way; this could place PR2-AC in an inferior position during testing time as its deceptive strategy has been found out by the opponents already during training.

6. CONCLUSION:

Inspired by the recursive reasoning capability of human intelligence, in this paper, we introduce a probabilistic recursive reasoning framework for multi-agent RL that follows "I believe that you believe that I believe". We adopt variational Bayes methods to approximate the opponents' conditional policy, to which each agent finds the best response and then improve their own policy. The training and execution is full decentralized and the resulting algorithms, PR2-Q and PR2-AC, converge in selfplay when there is one Nash equilibrium. Our results on three kinds of testing beds with increasing complexity justify the advantages of learning to reason about the opponents in a recursive manner. In the future, we plan to investigate other approximation methods for the PR2 framework, and test our PR2 algorithm for the coordination task between AI agents such as coordinating autonomous cars before the traffic light.

Update the actor using the sampled policy gradient:

∇ θ i η i ≈ 1 N N j=0 ∇ θ i µ i (s j ) ∇ a i 1 M M k=0 Q i µ i (s j , a i j , a -i k, j ); 18

Compute ∆ρ -i φ -i using empirical estimation:

19 ∆ρ -i φ -i (•|s, a i ) =E a -i t ∼ρ -i φ -i κ a -i t , ρ -i φ -i •; s t , a i t ∇ ã-i Q i s t , a i t , ã-i ã-i =a -i t + κ ã-i , ρ -i φ -i •; s t , a i t ∇ ã-i | ã-i =a -i t , 20

where κ is a kernel function;

21 Compute empirical gradient ∇φ -i J ρ -i ;

22:

Update φ -i according to ∇φ -i J ρ -i ; 23 end 24 Update target network parameters for each agent i:

25 θ i ← λθ i + (1 -λ)θ i ; ω i ← λω i + (1 -λ)ω i ; 26 end 27 end

The Algorithm 2 shows the variant of Decentralized Multi-Agent Probabilistic Recursive Reasoning. We can simply approximate the ρ -i (a -i |s, a i ) by counting:ρ -i (a -i |s, a i ) = C(a i , a -i , s)/C(a i , s) in tabular if the state-action space is small, where C is the counting function. It this case, an agent

ρ -i (a -i |s, a i ) = 1 Z exp(Q i (s, a i , a -i ) -Q i (s, a i ))

6

Select and sample action a i based on the Recursive Reasoning ρ -i (a -i |s, a i ); 7 softmax(

a -i ρ -i (a -i |s, a i )Q i (s, a i , a -i ))

8

Observing joint-action (a i , a -i ), reward r i , and next state s ;

9 Q i (s, a i , a -i ) ← (1 -α)Q i (s, a i , a -i ) + α(r i + γV i (s )) Q i (s, a i ) ← (1 -α)Q i (s, a i ) + α(r i + γV i (s ))

where,

V i (s) = max a i a -i ρ -i (a -i |s, a i )Q i (s, a i , a -i ) 10 end

only needs to learn a joint action Q-function, and if the game is static, our method would degenerate to Conditional Joint Action Learning (CJAL) (Banerjee & Sen, 2007) .

B MULTI-AGENT POLICY GRADIENT B.1 MULTI-AGENT NON-CORRELATED POLICY GRADIENT Since π θ a i , a -i |s = π i θ i a i π -i θ -i a -i |,a i = π -i θ -i a -i |s π i θ i a i |s, a -i

, π θ a i , a -i |s can be factorized as π i θ i (a i |s)π -i θ -i (a -i |s) if a i and a -i are non-correlated. We follow the policy gradient formulation (Sutton et al., 2000; Wei et al., 2018) using Leibniz integral rule and Fubini's theorem which can give us Multi-Agent Non-correlated Policy Gradient:

EQUATION

)

Suppose the π i (a i ) is parameterized by θ i , and we apply the gradient over the η i :

EQUATION

In practice, off-policy is more data-efficient. In MADDPG (Lowe et al., 2017) and COMA (Foerster et al., 2017) , the replay buffer is introduced in a centralized deterministic actor-critic method for off-policy training. They apply batch sampling to the centralized critic which gives the joint-action Q-values:

EQUATION

B.2 MULTI-AGENT RECURSIVE REASONING POLICY GRADIENT Proposition 1. In a stochastic game, under the recursive reasoning framework defined by Eq. 3, the update rule for the multi-agent recursive reasoning policy gradient method can be devised as follows:

EQUATION

)

Proof: As following.

If we apply the chain rule to factorize the joint policy to: π θ (a i , a -i |s) = π i θ i (a i |s)π -i θ -i (a -i |s, a i ). Then, we can have multi-agent recursive reasoning objective function as:

η i = s a i a -i π(a i , a -i |s)Q i (a i , a -i ) da -i da i ds = s a i π i (a i |s) a -i π -i (a -i |s, a i )Q i (s, a i , a -i ) da -i da i ds.

(18) Compare to Eq. 14, a -i in Eq. 18 is additionally conditioned on a i . We introduce agent i'a action a i into other agents's policies, leading to π -i (a -i |s, a i ). We now compute the policy gradient analytically. Following the single agent Policy Gradient Theorem with Leibniz integral rule and Fubini's theorem, we get the multi-Agent Recursive Reasoning Policy Gradient:

EQUATION

However, in practice, the agent may not get access to other agents' policies. We need to infer the other agents' policies. We let ρ -i φ-i (a -i |s, a i ) denotes the parameterized opponent conditional policy of agent i to approximate other agents policies, i.e, π -i (a -i |s, a i ). Then we have Decentralized Multi-Agent Recursive Reasoning Policy Gradient comes as:

EQUATION

In Eq. 20, the gradient for agent i is scaled by

Q i ρ -i φ -i (s, a i ) = a -i ρ -i φ-i (a -i |s, a i )Q i (s, a i , a -i ) da -i .

The trajectories generated by updated policy would help to train ρ -i φ-i (a -i |s, a i ) and Q i (s, a i , a -i ). These steps form a Expectation-Maximization style learning procedures: first, fix ρ -i φ-i and Q i (s, a i , a -i ) to improve π i θ i (a i |s); then, improve ρ -i φ-i and Q i (s, a i , a -i ) by the trajectories generated by π i θ i (a i |s). Furthermore, since PR2 method do not require opponents' actual private policies, Decentralized Multi-Agent Recursive Reasoning Policy Gradient can be decoupled from other agents' on-policies or target policies. In other words, the training can be conducted in an off-policy fashion by sampling mini-batches from the memory buffer D with the help of the learned ρ

-i φ-i (a -i |s, a i ) from Q i (s, a i , a -i ).

C OPPONENT CONDITIONAL POLICY INFERENCE VIA OPTIMAL TRAJECTORY:

Theorem 1. The optimal Q-function for agent i that satisfies minimizing Eq. 10 is formulated as:

EQUATION

)

And the corresponding optimal opponent conditional policy reads:

EQUATION

Proof. As following.

Follow the proof in Levine (2018) ; Haarnoja et al. (2017) , we first give the overall distribution by:

EQUATION

We can adopt an optimization-based approach to approximate the opponent conditional policy, in which case the goal is to fit an approximation π(a i t , a -i t |s t ) ≈ π i (a i t |s t )ρ -i (a -i t |s t , a i t ) such that the trajectory distribution,

EQUATION

has high likelihood to be observed. In the case of exact inference, as derived in the previous section, D KL ( p(τ) p(τ)) = 0. We can therefore view the inference process as minimizing the K L-divergence:

EQUATION

Negating both sides and substituting, we get:

EQUATION

where H is the entropy term. In the recursive case, we can rewrite the objective as follows:

EQUATION

This corresponds to a standard bellman backup with a soft maximization for the value function. choosing optimal opponent recursive reasoning policy

EQUATION

Then we can have the objective function:

EQUATION

Then the gradient is then given by: (30)

∇ φ -i J i (φ -i ) = T t=1 E (st,a i t ,a -i t )∼p(st,a i t ,a -i t ) [∇ φ -i log ρ -i φ -i (a -i t |s t , a i t )( T t =t r i (s t , a i t , a -i t )] + ∇ φ -i T t=1

The gradient of the entropy terms is given by: (32)

∇ φ -i H(ρ -i φ -i ) = -∇ φ E (

The -1 comes from the derivative of the entropy terms, and replacing -1 with a state and self-action dependent baseline b(s t , a i t ) we can obtain the approximated gradient for φ: 

∇ φ -i J i (φ -i ) = T t=1

where Qi t (s t , a i t , a -i t ) is is an empirical estimate of the Q-value of the policy.

D SOFT BELLMAN EQUATION AND SOFT VALUE ITERATION:

Theorem 2. In a symmetric game with only one equilibrium, and the equilibrium meets one of the conditions: 1) the global optimum, i.e. E π * Q i t (s) ≥ E π Q i t (s) ; 2) a saddle point, i.e.

E π * Q i t (s) ≥ E π i E π -i * Q i t (s) or E π * Q i t (s) ≥ E π i * E π -i Q i t (s)

; where Q * and π * are the equilibrium value function and policy, respectively. The PR2 soft value iteration operator defined by: TQ i (s, a i , a -i ) r i (s, a i , a -i ) + γE s ,a i ∼ps,π i log a -i exp(Q i (s , a i , a -i )) da -i , (34) is a contraction mapping.

Proof. As following:

Based on Eq. 11 & 12 in Theorem 1, we can have the PR2 soft value iteration rules shown as:

Q i π (s, a i , a -i ) = r i (s, a i , a -i ) + γE s ∼ps H(π i (a i |s)π -i (a -i |s, a i )) + E a -i ∼π -i (•|s ,a i ) [Q i π (s , a i , a -i )] = r i (s, a i , a -i ) + γE s ∼ps Q i π (s , a i ) .

(35) Correspondingly, we define the soft value iteration operator T: TQ i (s, a i , a -i ) r i (s, a i , a -i ) + γE s ,a i ∼ps,π i log a -i exp(Q i (s , a i , a -i )) da -i . (36)

In a symmetric game with either one global equilibrium or saddle equilibrium, it has been shown by Yang et al. (2018) (see condition 1&2 in Theorem 1) that the payoff at the equilibrium point is unique. This validates applying the similar idea in proving the contraction mapping of soft-value iteration operator in the single agent case (see Lemma 1 in Fox et al. (2016) ). We include it here to stay self-contained.

We first define a norm on Q-values as

EQUATION

Similarly, log a -i exp(Q i 1 (s , a i , a -i )) da -i ≤ -ε + log a -i exp(Q i 2 (s , a i , a -i )) da -i . Therefore

TQ i 1 -TQ i 2 ≤ γε = γ Q i 1 -Q i 2 .

:



Figures and Tables:

Figure 2: Diagram of multi-agent PR2 learning algorithms. It conducts decentralized training with decentralized execution. The light grey areas on two sides indicate decentralized execution for each agent. White areas give the decentralized learning procedures. All agents share the interaction experiences in the environment represented by dark area in the middle.

Figure 3: Learning paths on the iterated matrix game. a: IGA. b-d: PR2-Q.

Figure 4: Max of Two Quadratic Game.

Figure 5: The learning path of Agent 1 (x-axis) vs. Agent 2 (y-axis).

Figure 6: Performance of PR2-AC on the Particle World environment. Each bar shows the 0 -1 normalized score for agent in cooperative navigation task and the normalized advantage score (agent reward -adversary reward) in a set of competitive tasks. Higher score is better.

st,a i t )∼p(st,a i t ,a -i t ) [E a -i t ∼ρ -i φ -i (a -i t |st,a i t ) [log ρ -i φ -i (a -i t |s t , a i t )]] = -E (st,a i t ,a -i t )∼p(st,a i t ,a -i t ) [∇ φ log ρ -i φ -i (a -i t |s t , a i t )(1 + log ρ -i φ -i (a -i t |s t , a i t )].(31) We can do the same for ∇ φ -i H(π i θ i ), and substitute these back we have:∇ φ -i J i (φ -i ) = (s t , a i t , a -i t ) -log ρ -i φ -i (a -i t |s t , a i t ) -log π i θ i (a i t |s t ) -1)].

Table content: <table><tr><td/><td>T</td></tr><tr><td/><td>E (st,a i t ,a -i t )∼p(st,a i t ,a -i t ) [∇ φ -i log ρ -i φ -i (a -i t |s t , a i t )</td></tr><tr><td/><td>t=1</td></tr><tr><td>T</td><td/></tr><tr><td>(</td><td>r i</td></tr><tr><td>t =t</td><td/></tr></table>

(s t , a i t , a -i t ) -log ρ -i φ -i (a -i t |s t , a i t ) -log π i θ i (a i t |s t ) -1

Table content: <table><tr><td/><td/><td/><td/><td>E (st,a i t ,a -i t )∼p(st,a i t ,a -i t ) [∇ φ log ρ -i φ -i (a -i t |s t , a i t )</td></tr><tr><td/><td colspan="2">T</td><td/></tr><tr><td>(</td><td colspan="2">t =t</td><td colspan="2">r i baseline ignore</td><td>)]</td></tr><tr><td/><td>T</td><td/><td/></tr><tr><td>≈</td><td/><td colspan="3">E (st,a i</td></tr><tr><td colspan="3">t=1</td><td/></tr><tr><td/><td/><td/><td/><td>Q i t (st,a i t )-V i t (st )</td><td>Q i t (st,a i t ,a -i t )-Q i t (st,a i t )</td></tr><tr><td/><td/><td/><td>T</td></tr><tr><td colspan="2">+</td><td colspan="2">t =t+1</td><td>t (st+1,a i t +1 ,a -i t +1 ) r ≈Q i</td><td>)]</td></tr><tr><td/><td>T</td><td/><td/></tr><tr><td>=</td><td/><td colspan="3">E (st,a i t ,a -i t )∼p(st,a i t ,a -i t ) [∇ φ log ρ -i φ -i (a -i t |s t , a i t )</td></tr><tr><td colspan="3">t=1</td><td/></tr><tr><td colspan="5">(r i (s t , a i t , a -i t ) + Q i t (s t+1 , a i t+1 , a -i t+1 ) -Q i t (s t , a i t , a -i t ) + V i t (s t</td><td>))]</td></tr><tr><td/><td/><td/><td/><td>ignore</td></tr><tr><td/><td>T</td><td/><td/></tr><tr><td>=</td><td/><td colspan="3">E (st,a i t ,a -i t )∼p(st,a i t ,a -i t ) [(∇ φ -i Q i t (s t , a i t , a -i t ) -∇ φ -i Q i t (s t , a i t ))</td></tr><tr><td colspan="3">t=1</td><td/></tr><tr><td colspan="5">(r i (s t , a i t , a -i t ) + Q i t (s t+1 , a i t+1 , a -i t+1 ) -Q i t (s t , a i t , a -i t ) + V i t (s t</td><td>))]</td></tr><tr><td/><td/><td/><td/><td>ignore</td></tr><tr><td/><td>T</td><td/><td/></tr><tr><td>=</td><td/><td colspan="3">E (st,a i t ,a -i t )∼p(st,a i t ,a -i t ) [(∇ φ -i Q i t (s t , a i t , a -i t ) -∇ φ -i Q i t (s t , a i t ))</td></tr><tr><td colspan="3">t=1</td><td/></tr><tr><td/><td/><td/><td/><td>( Qi t (s t , a i t , a -i t ) -Q i t (s t , a i t , a -i t ))],</td></tr></table>

Peer review (iclr2019):
 The paper introduces a decentralized training method for multi-agent reinforcement learning, where the agents infer the policies of other agents and use the inferred models for decision making. The method is intuitively straightforward and the paper provides some justification for convergence. I think the underlying theory is okay (new but not too surprising, a lot of the connections can be made with single agent RL), but the paper would be much stronger with experiments that have more than two players, one state and one dimensional actions. ( Update: the new version of the paper addresses most of my concerns. There are a lot more experiments, and I think the paper is good for ICLR. However, I wonder if the reasoning for PR2 is limited to "self-play", otherwise Theorem 1 could break because of the individual Q_i functions will not be symmetric. This could limit the applications to other scenarios. Also, maybe explain self-play mathematically to make the paper self contained? ) 1. From the abstract, " PR2-Q and PR2-Actor-Critic, that are proved to converge in the self-play scenario". Theorem 2 only shows that under relatively strong assumptions (e.g. single Nash equilibrium), the soft value iteration operator is a contraction. This seems to have little to do with the actual convergence of PR2-Q and PR2-AC, especially AC which uses gradient-based approach. Also here the "convergence" in the abstract seem to imply convergence to the (single) global optimal solution (as is shown in the experiments), for which I thought you cannot prove even for single agent AC -- the best you can do is to show that gradient norm converges to zero, which gives you a local optima. Maybe things are different with the presence of the (concave) entropy regularization? 2. Theorem 2 also assumes that the opponent model will find the global optimal solution (i.e. (11, 12) can be computed tractably). However, the paper does not discuss the case where or in question is imperfect (similar to humans over/underestimate its opponents), which might cause the actual solution to deviate significantly from the (single) NE. This would definitely be a problem in more high-dimensional MARL scenarios. I wonder if one could extend the convergence arguments by extending Prop 2. 3. The experiments mostly demonstrates almost the simplest non-trivial Markov games, where it could be possible that (11, 12) are true for PR2. However, the effectiveness of the method have not been demonstrated in other (slightly higher-dimensional) environments, such as the particle environments in the MADDPG paper. It does not seem to be very hard to implement this, and I wonder if this is related to the approximation error in (11, 12). The success in such environments would make the arguments much stronger, and provide sound empirical guidance to MARL practitioners. Minor points: - Are the policies in question stationary? How is PR2 different from the case of single agent RL (conditioned on perfect knowledge of a stationary opponent policy)? - I have a hard time understanding why PR2 would have different behavior than IGA even with full knowledge of the opponent policy, assuming each policy is updated with infinitesimally small (but same) learning rates. What is the shape of the PR2 optimization function wrt agent 1? - I wonder if using 2 layer neural networks with 100 units each on a 1 dimensional problem is overkill. - Figure 4(a): what are the blue dots? - Does (11) depend on the amount of data collected from the opponents? If so, how? - I would recommend combining Prop 1 and prop 2 to save space. Both results are straightforward to prove, but the importance sampling perspective might be useful. - Have you tried to compare with SGA (Balduzzi et al) or Optimistic mirror descent? - I am also curious about an ablation study over the components used to infer opponent policies. A much simpler case would be action-dependent baselines, which seem to implicitly use some information about the opponents.