A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning
Authors: Dong-Ki Kim
Miao Liu
Matthew Riemer
Chuangchuang Sun
Marwa Abdulhai
Golnaz Habibi
Sebastian Lopez-Cot
Gerald Tesauro
JONATHAN P HOW
Venue:  (2021)
Keywords: Multiagent reinforcement learning
Meta-learning
Non-stationarity

Abstract: A fundamental challenge in multiagent reinforcement learning is to learn beneficial behaviors in a shared environment with other agents that are also simultaneously learning. In particular, each agent perceives the environment as effectively non-stationary due to the changing policies of other agents. Moreover, each agent is itself constantly learning, leading to natural nonstationarity in the distribution of experiences encountered. In this paper, we propose a novel meta-multiagent policy gradient theorem that directly accommodates for the non-stationary policy dynamics inherent to these multiagent settings. This is achieved by modeling our gradient updates to directly consider both an agent’s own non-stationary policy dynamics and the non-stationary policy dynamics of other agents interacting with it in the environment. We find that our theoretically grounded approach provides a general solution to the multiagent learning problem, which inherently combines key aspects of previous state of the art approaches on this topic. We test our method on several multiagent benchmarks and demonstrate a more efficient ability to adapt to new agents as they learn than previous related approaches across the spectrum of mixed incentive, competitive, and cooperative environments.

Under review as a conference paper at ICLR 2021
A POLICY GRADIENT ALGORITHM FOR LEARNING TO LEARN IN MULTIAGENT REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
A fundamental challenge in multiagent reinforcement learning learn beneficial behaviors in shared environment with other agents that are also simultaneously learning. In particular, each agent perceives the environment as effectively non- stationary due to the changing policies of other agents. Moreover, each agent is itself constantly learning, leading to natural nonstationarity in the distribution of experiences encountered. In this paper, we propose novel meta- multiagent policy gradient theorem that directly accommodates for the non-stationary policy dynamics inherent to these multiagent settings. This is achieved by modeling our gradient updates to directly consider both an agent's own non-stationary policy dynamics and the non-stationary policy dynamics of other agents interacting with it in the environment. We find that our theoretically grounded approach provides a general solution to the multiagent learning problem, which inherently combines key aspects of previous state of the art approaches on this topic. We test our method on several multiagent benchmarks and demonstrate a more efficient ability to adapt to new agents as they learn than previous related approaches across the spectrum of mixed incentive, competitive, and cooperative environments.
INTRODUCTION
Learning in multiagent settings is inherently more difficult than single-agent learning because an agent interacts both with the environment and other agents (Buşoniu et al., 2010). Specifically, the fundamental challenge in multiagent reinforcement learning (MARL) is the difficulty of learning optimal policies in the presence of other simultancously learning agents because their changing behaviors jointly affect the environment's transition and reward function. This dependence on non- stationary policies renders the Markov property invalid from the perspective of each agent, requiring agents to adapt their behaviors with respect to potentially large, unpredictable, and endless changes in the policies of fellow agents (Papoudakis et al. 2019). In such environments, it is also critical that agents adapt to the changing behaviors of others in very sample-efficient manner as it is likely that their policy could update again after small number of interactions (Al-Shedivat et al.) 2018). Therefore, effective agents should consider the learning of other agents and adapt quickly to non-stationary behaviors. Otherwise, undesirable outcomes may arise when an agent is constantly lagging in its ability to deal with the current policies of other agents.
In this paper, we propose new framework based meta-learning for addressing the inherent non-stationarity of MARL. Meta-learning (also referred to as learning to learn) was recently shown to be a promising methodology for fast adaptation in multiagent settings. The framework by Al-Shedivat et al. (2018), for example, introduces a meta-optimization scheme by which a meta-agent can adapt more efficiently changes in a new opponent's 's policy after collecting only a handful of interactions. The key idea underlying their meta- opttmizaiiiiiionn is to model the meta-agent's learning process so that its updated policy performs better than an evolving opponent. However, their work does not directly consider the opponent's 's learning process in the meta-optimization, treating the evolving opponent as an external factor and assuming the meta-agent cannot influence the opponent's 's future policy. As a result, their work fails to consider an important property of MARL: the opponent is also a learning agent changing its policy based on trajectories collected by interacting with the meta-agent. As such, the meta-agent has an opportunity to influence the opponent's future policy by changing the distribution of trajectories, and the meta-agent can take advantage of this opportunity to improve its performance during learning.

---

Under review as a conference paper at ICLR 2021
Our contribution. With this insight, we develop new meta-mulltagget policy gradient theorem (Meta-MAPG) that directly models the learning processes of all agents in the environment within a single objective function. We start by extending the meta-policy gradient theorem of [aaA--SShee Shedivat et al. 2018) based on the multiagent stochastic policy gradient theorem (Wei et al. 2018) to derive a novel meta-policy gradient theorem. This is achieved by removing the unrealistic implicit assumption of Al-Shedivat et al. (2018) that the learning of other agents in the environment is not dependent on an agent's own behavior. Interestingly, performing our derivation with this more general set of assumptions inherently results in an additional term that was not present in previous work by Al-Shedivat al. (2018). We observe that this added term is closely related to the process of shaping the learning dynamics of other agents in the framework of Foerster et al. (201 Ba). As such, our work can be seen as contributing theoretically grounded framework that unifies the collective benefits of previous work by Al-Shedivat et al. (20T8) 8) Foerster et al. (20T8a). Meta-MAPG is evaluated on diverse suite of multiagent domains, including the full spectrum of mixed incentive, competitive, and cooperative environments. Our experiments demonstrate that Meta-MAPG consistently results in superior adaption performance in the presence of novel evolving agents.
Agunt
$
46
- Own Leaming
Trapoctryy
Ed
Four Leaming Gradent
Addes Dependency
Porr Agents
+
LL
$
Johnt Policy the
Joint
JuinE Policy do
Pelicy di
Policy
Policy
Policy thet
Figure I: (a) A Markov chain of joint policies representing the inherent non non-stationarity of MARL. Each agent updates its policy leveraging a Markovian update function, resulting in a change to the joint policy. (b) A probabilistic graph for Meta-MAPG. Unlike Meta-PG, our approach actively influences the future policies of other agents as well through the peer learning gradient.
PRELIMINARIES
Interactions between multiple agents can be represented by stochastic games (Shapley 1953). Specif- ically, an n-agent stochastic game is defined as tuple (, S, A. R, 7); [= [I, n. is the set of n agents, S is the set of states, A= A the state transition probability function, R = X Xzez is the set of reward functions, and y [0. the discount factor. We typeset sets in bold for clarity. Each agent executes an action at each timestep t according to its stochastic policy of ~ ''(a; st. Ø²) parameterized by ø', where 8, S.
the set of action spaces, P:S x A S is
A joint action a, = = ") yields transition from the current state s; to the next state $1+1 with probability P((s+1 1st, az), where the notation - i indicates all other agents with the exception of agent Agent i then obtains reward according to its reward function ri R' at). At the end of an episode, the agents collect a trajectory To under the joint policy with parameters Φ where TΦ == (sp, ao, ro...., TH), Φ = { Φ- represents the joint parameters of all policies, Tt= {ri, re is the joint reward, and // is the horizon of the trajectory or episode.
2.1 A MARKOV CHAIN OF POLICIES
The perceived non non-stationarity in multiagent settings results from a distribution of sequential joint policies, which can be represented by Markov chain (Al-Shedivat al., 2018). Formally, a Markov chain of policies begins from a stochastic game between agents with an initial set of joint policies parameterized by do. We assume that each agent updates its policy leveraging Markovian update function that changes the policy after every K trajectories. After this time period, each agent i adapts its policy maximize the expected return expressed as its value function:
Vi 0u (8o) = E Too up(Tmo 10 $1 {
xd [8 = W ~p(T 10 (c(o)))) 1=
H Σ
(1)
2

---

Under review as a conference paper at ICLR 2021
where G' denotes agent i's discounted return from the beginning of episode with initial state so- The joint policy update results in a transition from Φo to the updated set of joint parameters Φ1- The Markov chain continues for maximum chain length of L (see Figure Tap. This Markov chain perspective highlights the following inherent aspects of the experienced non-stationarity:
Sequential dependency. The future joint policy parameters Φ1: ( ΦL) sequentially depend on Φo since a change in Too results in a change in Φ1, which in turn affects Tai and all successive joint policy updates up to ØL.
Controllable levels of non-stationarity. As in Al-Shedivat et al. 2018) and Foerster et al. 201 Sa). assume stationary policies during the collection of K trajectories, and that the joint policy update happens afterward. In such setting, it is possible to control the non-stationarity by adjusting the K and И hyperparameters: smaller K and I increase the rate that agents change their policies, leading to higher degree of non-stationarity in the environment. In the limit of K = H 1, all agents change their policy every step.
LEARNING TO LEARN IN MULTIAGENT REINFORCEMENT LEARNING
This section explores learning policies that adapt quickly to non-stationarity in the policies of other agents in the environment. To achieve this, we leverage meta-learning and devise new mela-multiagen policy gradient theorem that exploits the inherent sequential dependencies of MARL discussed in the previous section. Specifically, our meta-agent addresses this non-stationarity by considering its current policy's impact on its own adapted policies while actively influencing the future policies of other agents as well by inducing changes to the distribution of trajectories. In this section, we first outline the optimizziiiiioonnn process MARL, and then discuss how the meta-policy gradient theorem AShheeitt et al. (2018) optimizes for this objective while ignoring the dependence of the future policy of other agents on our current policy. Finally, we derive new extension of this policy gradient theorem that explicitly leverages this dependence and discuss how to interpret the impact of the resulting form of the gradient.
3.1 GRADIENT BASED OETTA-OPTMMIZATION IN MULTIAGENT REINFORCHMENT LEARNING
We formalize the meta-objective of MARL optimizing meta-agent i's initial policy parameters o so that it maximizes the expected adaptation performance over a Markov chain of policies drawn from a stationary initial distribution of policies for the other agents p(dg"):
max p(Oo 66
Σ t=
ViDu4 (su, bb).
(2)
s.t.
Vi Vi @ooe41 (s0, ob) = Ture ool e 00.1)
Twets ~p(Foet 10 2 (T
0]]
where Tou = (Tpo: Jo and V
0u.e44 (50, (5)) denotes the value function. This meta-value function generalizes the notion of each agent's primitive value function for the current set policies veo (so) over the length of the Markov chain of policies. In this work, in Al-Shedivat et al. (2018) we follow the MAML (Finn et al. 2017) meta-learning framework. As such, we assume that the Markov chain of policies is governed by a policy gradient update function that corresponds to what generally referred to as the inner-loop optimization in the meta-learning literature:
0 == Φ + a 63
E.
[ (Toz))
The ~ ppl 0
(4)
t Φ == の - a -iv
- plroe oi G (To)))
where a² and a - denote the learning rates used by each agent in the environment.
3.2 THE MRTA-POLICY GRADIENT THEOREM
Intuitively, if we optimize the meta-value function, are searching for initial parameters of such that successive inner-loop optimization steps with Equation (4) results adapted parameters 0 that can perform better than the updated policies of other agents with policy parameters Φ (see Figure 1b).
3

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Meta-Learning at Training Time
Algorithm 2 Meta-Learning at Execution Time
Require: p(@o ") Distribution over other agents' ini-
Require: p(0p") Distribution over other agents' ini-
tial policies; o, 9:: learning rates
tial policies: a: Learning rate; Optimized Φ6
1: Randomly initialize db
1: Initialize db f- %b'
2: while d6 has not converged do
2: Sample a meta-test batch of Φo p(
3: Sample a meta-train batch of 60 ~ "0'
p(
3: for cach Φ
for each 6, do 0,..., L do 5: for &
4: for f= O..... do
5: Sample trajectory Téy
6: Sample and store trajectory The
6:
Compute dbe+1 = f Taby a) from
7:
Compute Φ f0,. Tuk. from inner-loop optimization (Equation (4))
inner-loop optimization (Equation (4))
7: end for
8:
end for
9:
end for
10:
Update db + tb B V V
Voo.e1
(su, (ō) based on Equation (6)
end while
In Deep RL, a very practical way to optimize value function is by following its gradient. The work AShheiitt et al. (2018) derived the meta-policy gradient theorem (Meta-PG) for optimizing setup like this. However, it is important to note that they derived this gradient while making the implicit assumption to ignore the dependence of the future parameters of other agents on Φ6:
Vi d6 doee41 (80: (6) = E
Tooie" up(Tooe Bar Rorl B
46 loollooTaaa (8)
-2 De v loga (Tee ( G (TeeLz))
(5)
Current Policy
Learning
In particular, Meta-PG has two primary terms. The first term corresponds to the standard policy gradient with respect to the current policy parameters used during the initial trajectory. Meanwhile, the second term V og log (T ( explicitly differentiates through log (T (0)+1) with
respect to oi. This enables a meta-agent to model its own learning dynamics and account for the impact of oi on its eventual adapted parameters 0 As such,
see how this term would be quite useful in improving adaptation across a Markov chain policies. Indeed, it directly accounts for agent's 's own learning process during meta-optimization in order to improve future performance.
3.3 THE METTA-MULTIAGENT POLICY GRADIENT THEOREM
In this section, we consider doing away with the implicit assumption from Al-Shedivat et al. (2018) discussed in the last section that we can ignore the dependence of the future parameters of other agents on Φb- Indeed, meta-agents need to account for both their own learning process and the learning processes of other peer agents in the environment to fully address the inherent non-stationarity of MARL. We will now demonstrate that our generalized gradient includes a new term explicitly accounting for the effect agent's current policy has the learned future policies of its peers.
Theorem (Meta-Multiagee Multiagent Policy Gradient Theorem (leta-MAPG). For any stochastic game Mn, the gradient of the meta-objective function for agent i at state with respect to the current parameters Φ6 of stochastic policy π evolving in the environment along with other peer agents using initial parameters Φ5 is:
Vi 0 000e4:
(so. 66)
LE Fraen T <p(Tmee loÉ 1
(6)
Toous upl Thme 1 tbol)
V logar( T 10b) Ei
loga (To 100 10( Dp
oi log ( (T & p
Current Policy
Own Leaming
Peer Leaming
Proof. See Appendix A for detailed proof of Theorem 1.
4

---

Under review as a conference paper at ICLR 2021
Probabilistic model perspective. Probabilistic models for Meta-PG and Meta-MAPG are depicted in Figure lb As shown by the own learning gradient direction, a meta-agent optimizes og by accounting for the impact of % on its updated parameters $1:e+3 and adaptation performance (T However, Meta-PG considers the other agents as an external factor that cannot be influenced by the meta-agent, indicated by the absence of the dependence between Tibo.e and P1:241 in Figure lb As result, the meta-agent loses an opportunity to influence the future policies of other agents and further improve its adaptation performance. By contrast, the peer learning term in Theorem I aims to additionally compute gradients through the sequential dependency between the agent's 's initial policy % and the future policies of other agents in the environment 11:++ so that it can learn to change Too in a way that maximizes performance over the Markov chain of policies.
Interestingly, the peer learning term that naturally arises when taking the gradient in Meta-MAPG has been previously considered in the literature by Foerster et al. (2018a). In the Learning with Opponent Learning Awareness (LOLA)) approach (Foerster et al. (2018a), this term was derived in an alternate way following a first order Taylor approximation with respect to the value function. Indeed, it is quite surprising to see how taking a principled policy gradient while leveraging more general set of assumptions leads to unification of the benefits of past works (AI-Shedivat et al. 20188 Foerster 2018a) on adjusting to the learning behavior of other agents in MARL.
al.
Algorithm. We provide pseudo-code for Meta-MAPG in Algorithm I for meta-training training and Algo- rithm 2 for meta-testing. Note that Meta is centralized during meta-training as it requires the policy parameters of other agents to compute the peer learning gradient. However, for settings where a meta-agent cannot access the policy parameters of other agents during meta-training, we provide decentralized meta-training algorithm with opponent modeling, motivated by the approach used in Foerster et al. (2018a), in Appendix B that computes the peer learning gradient while leveraging only an approximation of the parameters of peer agents. Once meta- meta-trained in either case, the adaptation to new agents during meta-testing is purely decentralized such that the meta-agent can decide how to shape other agents with its own observations and rewards alone.
4 RELATED WORK
The standard approach for addressing stattiiioon in MARL is to consider information about the other agents and reason about the effects of their joint actions (Hernandez- et al., (017). The literature on opponent modeling, for instance, infers opponents' behaviors and conditions an agent's policy on the inferred behaviors of others (He et al. 2016 Raileanu et al. 2018 Grover et al. 2018). Studies regarding the centralized training with decentralized execution framework (Lowe ctt al. 2017 Foerster et al. 2018b Yang et al. 2018: Wen et al. 2019). which accounts for the behavior of others through centralized critic, can also be classified into this category. While this body of work alleviates non- non-stationarity, it is generally assumed that each agent will have a stationary policy in the future. Because other agents can have different behaviors in the future as result of learning (Foerster et al., 201 this incorrect assumption can cause sample inefficient and improper adaptation (see Example | in Appendix). In contrast, Meta-MAPG models the learning process of each agent in the environment, allowing meta-lcarning agent to adapt efficiently.
Our approach is also related to prior work that considers the learning of other agents in the environ- ment. This includes Zhang & Lesser (2010) who attempted to discover the best response adaptation to the anticipated future policy of other agents. Our work is also related, as discussed previously, to LOLA |Foerster et al., 2018a) and more recent improvements (Foerster et al. 2018c). Another relevant idea explored by Letcher et al. (2019) to interpolate between the frameworks of Zhang & way that guarantees convergence while influencing the opponent's s future policy. However, all of these approaches only account for the learning processes of other agents and fail to consider agent's 's own non-stationary policy dynamics as in the own learning gradient discussed in the previous section. Additionally, these papers do not leverage meta-learning. As result, these approaches may require many samples to properly adapt to new agents.
Lesser 2010) and Foerster et al. (2018a)
1992) has recently become very popular as a method for improving sample efficiency in the presence of changing tasks in the Deep RL literature (Wang et al,, 60; Duan et al. 2016b] Finn et al.]
Meta-learning (Schmidhuber 1987 Bengio et al.)
2017 Mishra et al.] 2017): Nichol & Schulman 2018). See Vilalta & Drissi (2002) Hospedales et al. 2020 for in-depth surveys of meta-learning In
5

---

Under review as a conference paper at ICLR 2021
IPD Wis Compositing Peer Agent
IPD With Defecting Peer Agret
RP5
2-Agert HafCleveak
18
13
50
67
Pulkcy
Joint Pulicy
(a)
(b)
(d)
Meta MAPG
Meta
LOLA
REINFORCE
Figure 2: Adaptation performance during meta-testing in mixed incentive ((a), (b)). competitive (c), and cooperative (d) environments. The results show that Meta-MAPG can successfully adapt to a new and learning peer agent throughout the Markov chain. Mean and 95% confidence interval computed for 10 random seeds for ((a). (b), (c)) and 5 random seeds for (d) shown in figures.
particular, our work builds on the popular model agnostic meta-learning (MAML) framework (Finn et al.] (217)) where gradient-based learning is used both for conducting so called inner-loop learning and improve this learning by computing gradients through the computational graph. When we train our agents so that the inner loop can accommodate for dynamic Markov chain of other agent policies, we are leveraging an approach that has recently become popular for supervised learning
called meta-continual learning (Riemer et al., 2019 Javed & White 2019 Spigler [019]] Beaulleu al., 2020 Caccia et al. 2020 Gupta et al. 2020). This means that our agent trains not just adapt to single set of policies during meta-training, but rather to adapt to a set of changing policies with Markovian updates. As a result, we avoid an issue of past work (Al-Shedivat et al.] (2018) that required
the use of importance sampling during meta-testing (see Appendix D.1 for more discussion).
5 EXPERIMENTS
We demonstrate the efficacy of Meta-MAPG on a diverse suite of multiagent domains, including the full spectrum of mixed incentive, competitive, and cooperative environments. To this end, we directly compare with the following baseline adaptation strategies:
1) Meta-PG (Al-Shedivat et al. 2018): j A meta-learning approach that only considers how to improve its own learning. We detail our implementation of Meta-PG and a low-level difference with the implementation in the original paper by Al-Shedivat al. (2018) in Appendix D.
2) LOLA-DICE (Foerster et al. 2018ch: An approach that only considers how to shape the learning dynamies of other agents the environment through the Differentiable Nonte-Carlo Estimator (DiCE) operation. Note that LOLA-DiCE is an extension of the original LOLA approach.
3) REINFORCE (Williams 1992): A simple policy gradient approach that considers neither an agents own learning nor the learning processes of other agents. This baseline represents multiagent approaches that assume each agent leverages a stationary policy in the future.
In our experiments, implement each method's policy leveraging an LSTM. The inner-loop updates are based on the policy gradient with a linear feature baseline (Duan et al., 2016a), and we use generalized advantage estimation (Schulman al. 2016) with learned value function for the meta- optimization. We also learn dynamic inner-loop learning rates during meta-training, as suggested in Al-Shedivat al. 2018). We refer readers to Appendices C, D, E, H, and the source code in the supplementary material for the remaining details including selected hyperparameters.
Question 1. Is it essential consider both an agent's own learning and the learning of others?
To address this question, we consider the classic iterated prisoner's dilemma (IPD) domain. In IPD, agents and j act by either (C)ooperating or (D)efecting and receive rewards according to the mixed incentive payoff defined in Table 1. As in Foerster et al. (2018a), we model the state space as 8q- = Ø and 8, = a for
Agent j
Agent
C
D
C
(0.5, 0.5)
(-1.5, 1.5)
D
(1.5,
(-0.5, -0.5)
Table 1: IPD payoff table
For meta-learning, we construct a population of initial personas p( that include cooperating
personas (i.e., having a probability of cooperating between 0.5 and 1.0 at any state) and defecting

---

Under review as a conference paper at ICLR 2021
Sumple Complesit Aralysss
IPD Population Visssiaation (I+ Detribution)
IPD Pepuieiion Visukaation (Out of Disribunion)
Ame Cime
Prlecton
Prtecton
*f OT Trajectorice Betwoon Policy Updates
Principal Companent
Principal Companent Wiarta
ROVIONA
Visr
(a)
(b)
(c)
Figure 3: (a) Adaptation performance with a varying number of trajectories. Meta-MAPG achieves the best AUC in all cases and its performance generally improves with a larger K. Mean and 95% confidence interval are computed for 10 seeds. (b) and (c) Visualization of j initial policy for in distribution and out of distribution meta-testing, respectively, where the out of distribution split has smaller overlap between the policies used for meta-training/validee validation and those used for meta-lesting.
personas (i.e., having probability of cooperating between 0 and 0.5 at any state). Figure 3bb shows the population distribution utilized for training and evaluation. An agent j is initialized randomly from the population and adapts its behavior leveraging the inner-loop learning process throughout the Markov chain (see Figure 6 in the appendix). Importantly, the initial persona of agent j is hidden to Hence, an agent should: 1) adapt differently initialized agent j with varying amounts of cooperation, and 2) continuously adapt with respect to the learning of j.
The adaptation performance during meta-testing when an agent i, meta-trained with either Meta- MAPG or the baseline methods, interacts with an initially cooperating or defecting agent is shown in Figure 2a and Figure 2b] respectively. In both cases, our meta-agent successfully infers the underlying persona of the other agent and adapts throughout the Markov chain obtaining higher rewards than our baselines. We observe that performance generally decreases as the number of joint policy update increases across all adaptation methods. This decrease in performance is expected as each model is playing with another agent that is also constantly learning. As a result, the other agent realizes it could potentially achieve more reward by defecting more often. Hence, to achieve good adaptation performance in IPD, an agent should attempt to shape j's future policies toward staying cooperative long as possible such that can take advantage, which achieved by accounting for both an agent's 's own learning and the learning of other peer agents in Meta-MAPG.
We explore each adaptation method more detail by visualizing the action probability dynamics throughout the Markov chain. In general, we observe that the baseline methods have converged to initially defecting strategies, attempting to get larger rewards than a peer agent j in the first trajectory
Téo While this strategy can result in better initial performance than j. the peer agent will quickly change its policy so that it is defecting with high probability as well (see Figures 9 to IT in the appendix). By contrast, our meta-agent learns to cooperatively in Toy and then take advantage by deceiving agent j as it attempts to cooperate at future steps (see Figure 12 in the appendix).
Question How is adaptation performance affected by the number of trajectories between changes? We control the level of stationtnnar by adjusting the number of trajectories K between updates (refer to Section 2.1). The results in Figure [3l shows that the area under the curve (AUC) (i.e., the reward summation during Φ1.1) generally decreases when K decreases in IPD. This result is expected since the inner-loop updates are based on the policy gradient, which can suffer from a high variance. Thus, with a smaller batch size, policy updates have a higher variance (leading to noisier policy updates). As result, it is harder anticipate and influence the future policies of other agents. Nevertheless, in all cases, Meta-MAPG achieves the best AUC.
Question 3. Can Meta generalize its learning outside the meta-training distribution?
We have demonstrated that a meta-agent can generalize well and adapt to new peer. However, we would like to investigate this further and see whether meta-agent can still perform when the meta-testing distribution drawn from a significantly different distribution in IPD. We thus evaluate Meta-MAPG and Meta-PG using both in distribution (as in the previous questions) and out of distribution personas for j's initial policies (see Figures 3b and Bc Meta-MAPG achieves an AUC of 13.77=0.25 and 11. 0..33 33 for the in and out of distribution evaluation respectively. On the other hand, Meta-PG achieves an AUC of 6. 63.05 and 7.60 for the and out of distribution evaluation respectively. Variances are based on 5 seeds and we leveraged K 64 for this experiment.
7

---

Under review as a conference paper at ICLR 2021
Opponent Modefing
RP5 Web Varving Namber Of Agonts
RPS Askarios Study
Unde The Cume
Aee Under The Carve
Nowsag Rowaed For Agest
76-
0/-
Adpttotion
Agers
Apgris Joint Philiyy
fggtts
Joint Philcy
Mcs-MATG
Man-WAPG (DN))
(a)
(b)
(c)
Figure 4: (a) Adaptation performance with opponent modeling (OM). Meta-MAPG with OM uses inferred policy parameters for peer agents, computing the peer learning gradient in decentralized manner. (b) Adaptation performance with a varying number of agents in RPS. Meta-MAPG achieves the best AUC in all cases. (c) Ablation study for Meta-MAPG. Meta-MAPG achieves significantly better performance than ablated baselines with no own learning gradient and no peer learning gradient.
The mean and 95% confidence interval are computed using 5 seeds in (a) and 10 seeds in (c).
We note that Meta-MAPG's performance decreases during the of distribution evaluation, but still consistently performs better than the baseline.
Question 4. How does Meta-MAPG perform with decentralized aata-training?
We compare the performance of Meta-MAPG with and without opponent modeling in Figure 4a We note that Meta-MAPG with opponent modeling can infer policy parameters for peer agents and compute the peer learning gradient in decentralized manner, performing better than the Meta-PG baseline. However, opponent modeling introduces noise in predicting the future policy parameters of peer agents because the parameters must be inferred by observing the actions they take alone without any supervision about the parameters themselves. Thus, as expected, meta-agents experience difficulty in correctly considering the learning process of peer agents, which leads lower performance than Meta-MAPG with centralized meta-training.
Question 5. How effective is Meta-MAPG in a fully competitive scenario?
We have demonstrated the benefit of our approach in the mixed incentive scenario of IPD. Here, we consider an- other classic iterated game, rock-paper-seissons (RPS) with
Agent j P
Agent
S
R
(0.0)
(-1,1) 1)
(1.-1)
a fully competitive payoff table (see Table 2). In RPS, at
P
(1,-1)
(0,0)
(-1.1)
each time step agents and] can choose an action of either (R)ock, (P)aper, or (S)cissors. The state space is defined as 80 and 8; = - for & 1.
S
(-1,1)
(1, -1)
(0.0)
Table 2: RPS payoff table.
Similar to our meta- meta-learning setup for IPD, we consider a population of initial personas p(00 including the rock persona (with a rock action probability between 1/3 and 1.0), the paper persona (with paper action probability between 1/3 and 1.0), and the scissors persona (with a scissors action probability between 1/3 and 1.0). As in IPD, an agent is initialized randomly from the population and updates its policy based on the policy gradient with linear bascline while interacting with
Figure 2c shows the adaptation performance during meta-testing, Similar to the IPD results, we observe that the baseline methods have effectively converged to win against the opponent j in the first few trajectories. For instance, agent has high rock probability when playing against j with a high initial scissors probability (see Figures [3 to [5] in the appendix). This strategy, however, results in the opponent quickly changing its behavior toward the mixed Nash equilibrium strategy of (1/3, 1/3, 1/3) for the rock, paper, and scissors probabilities. In contrast, our meta-agent learned to lose slightly in the first two trajectories Toa to achieve much larger rewards in the later trajectories Toair while relying on its ability to adapt more efficiently than its opponent (see Figure [6] in the appendix). Compared the IPD results, we observe that it is more difficult for our meta-agent to shape j's future policies in RPS possibly due to the fact that RPS has fully competitive payoff structure, while IPD has a mixed incentive structure.
Question 6. How effective is Meta-MAPG in settings with more than one peer?
We note that the meta-men aetta-mlttiagent policy gradient theorem is general and can be applied to scenarios with more than one peer. To validate this, we experiment with 3 p-payerr and 4-player RPS, where consider sampling peers randomly from the entire persona population. Figure sbows against the Meta-PG baseline. We generally observe that the peer agents change their policies
shows a comparison
8

---

Under review as a conference paper at ICLR 2021
toward the mixed Nash equilibrium more quickly the number of agents increases, which results in decreased performance for all methods. Nevertheless, Meta-MAPG achieves the best performance in all cases and can clearly be easily extended to settings with a greater number of agents.
Question 7. Is it necessary to consider both the own learning and peer learning gradient? Our meta-multiagent policy gradient theorem inherently includes both the own learning and peer learning gradient, but it important to consider both terms? To answer this question, we conduct an ablation study and compare Meta-MAPG to two methods: one trained without the peer learning gradient and another trained without the own learning gradient. Note that not having the peer learning term is equivalent Meta-PG, and not having the own learning term is similar LOLA-DiCE but alternatively trained with a meta-optimization procedure. Figure [4c shows that a meta-agent trained without the peer learning term cannot properly exploit the peer agent's learning process. Also, meta-agent trained without the own learning term cannot change its own policy effectively in response to anticipated learning by peer agents. By contrast, Meta-MAPP achieves superior performance by accounting for both its own learning process and the learning process of peer agents.
Question 8. Does considering the peer learning gradient always improve performance?
To answer this question, we experiment with fully cooperative setting from the multiagent-MuloCo benchmark (de Witt et al.) 22220). Specifically, we consider the 2- 2-Agent HalfCheetah domain, where the first and second agent control three joints of the back and front leg with continuous action spaces, respectively (see Figure [5). Both agents receive a joint reward corresponding to making the cheetah robot run to the right as soon as possible. Note that two agents are coupled within the cheetah robot, so accomplishing the objective requires close cooperation and coordination between them.
For meta-learning, we consider a population of teammates with vary- ing degrees of expertise in running the left direction. Specifically, we pre-train teammate j and build a population based on checkpoints of its parameters during learning (see Figure Zn in Appendix). Then, during meta-learning, j is randomly initialized from this population of policies. Importantly, the teammate must adapt its behavior in this setting because the agent has achieved the opposite skill compared to the true objective of moving to the right during pre-training. Hence, meta-agent i should succeed by both adapting to differently initialized teammates with varying expertise in moving the opposite direction, and guiding the teammate's learning process in order to coordinate eventual movement to the right.
Figure 5: 2-Agent HalfChee- tah domain, where two agents are coupled within the robot and control the robot together. Graphic credit: de Witt et al. (2020).
Our results are displayed in Figure 2d There are two notable observations. First, influencing peer learning does not help much in cooperative settings and Meta-MAPG performs similarly to Meta-PG. The peer learning gradient attempts to shape the future policies of other agents so that the meta-agent can take advantage. In IPD, for example, the meta-agent influenced to be cooperative in the future such that the meta-agent can act with high probability of the defect action and receive higher returns. However, in cooperative settings, due to the joint reward, the teammate is already changing its policies in order to benefit the meta-agent, resulting in a less significant effect with respect to the peer learning gradient. Second, Meta-PG and Meta-MAPG outperform the other approaches of LOLA-DiCE and REINFORCE, achieving higher rewards when interacting with a new teammate.
6 CONCLUSION
In this paper, we have introduced Meta-MAPG which is a meta-learning algorithm that can adapt quickly to non-stationarity in the policies of other agents in a shared environment. The key idea underlying our proposed meta-optimization is directly model both an agent's 's own learning process and the non-stationary policy dynamics of other agents. We evaluated our method on several multiagent benchmarks, including the full spectrum of mixed incentive, competitive, and cooperative environments. Our results indicate that Meta-MAPG is able to adapt more efficiently than previous state of the art approaches. We hope that our work can help provide the community with a theoretical foundation to build off for addressing the inherent non-stationarity of MARL in principled manner. In the future, we plan to extend our approach to real-world scenarios, such as those including collaborative exploration between multiple agents CCan et al., 2019).

---

144-2---4435-6... Under review as a conference paper at ICLR 2021
REFERENCES
Maruan Al-Shedivat, Trapit Bansal, Yura Burda, llya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environ- ments. In International Conference on Learning Representations (ICLR), 2018. URLhttps:
//openreview. Htwpp//wwnee/ooommmd------- /
0-
Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O Stanley, Jeff Clune, and
Nick Cheney. Learning to continually learn. arXiv preprint arXiv:2002.09571, 2020. URL
https://arxiv: http///xxww..rrr///////22222 09571
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Geesei. On the optimization of a synaptic learning rule. In Preprints Conf. Optimality in Artificial and Biological Neural Networks, volume 2. Univ. of Texas, 1992.
Lucian Busoniu, Robert Babuška, and Bart De Schutter. Multi-agent Reinforcement Learning: An Overview, pp. 183-221. Springer Berlin Heidelberg, Berlin, Heidelberg, 2010. ISBN 978-
3-642-14435-6 doi: 10. 10.1007/978-3-422
978-3-642-144355
Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Caccia, Issam Laradji, Irina Rish, Alexande Lacoste, David Vazquez, et Online fast adaptation and knowledge accumulation: new approach to continual learning. Neural Information Processing Systems (NeurIPS), 2020.
L. Chan, D. Hadfield-Menell, S. Srinivasa, and A. Dragan. The assistive multi-armed bandit. In 2019 14th ACM/IEEE International Conference on Human-Robe Interaction (HRI), pp. 354-363, 2019. doi:
Christian Schroeder Witt, Bei Peng, Plerre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. arXiv preprint arXiv:2003.06709 2020.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Picter Abbeel. Benchmarking deep reinforcement learning for continuous control. volume 48 Proceedings of Machine Learning Research, pp. 1229-1338, New York, New York, USA, 20-22 Jun 2016a. PMLR. URL //proceedings. mlr prress/ v48/ ddann1. 6. html
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Picter Abbeel. RL2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint 1r11v:/611.022 201
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. ICML, 2017.
Jakob Foerster, Richard Y. Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-lcarning awareness. In International Conference on Au- tonomous Agents and MultiAgent Systems (AAMAS), pp. 122-130, 2018a.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Association for the Advancement of Artificial Intelligence (AAAI), February 2018b. URL /www cs ox ac uk people/ shimon pdf
Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric Xing, and Shimon Whiteson. DiCE: The infinitely differentiable Monte Carlo estimator. In International Conference on Machine Learning (ICML), volume 80, pp. 1533, 10-15 Jul 2018c. URL http://
proceedings.co www. wwwppppssseeeee wwwppppwssss/eeeeeeeeee hatB0//foettteee 8a. atml
Aditya Grover, Maruan Al-Shedivat, Jayesh Gupta, Yuri Burda, and Harrison Edwards. Learning policy representations in multiagent systems. In International Conference on Machine Learning (ICML), volume 80, pp. 1802-1811, 10-15 Jul 2018. URL latp:://proceedings.tt
pre3s/v80/grom 18.. html
10

---

Under review as a conference paper at ICLR 2021
Gunshi Gupta, Karmesh Yadav, and Liam Paull. La-maml: ahekd meta learning for continual learning. 2020. URL htllhttss///xxxiaa www.or/aaoooooo abs 22000000..200 13904
He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep reinforce- ment learning. In International Conference on Machine Learning (ICML), volume 48, pp. 1804 1813, 20-22 Jun 2016. URL http::/proocced.ggg mlz, www.sppessss......... //////////eee //eele htmll
Pablo Hernandez-Leal. Michael Kaisers, Tim Baarslag, and Enrique Munoz de Cote. A survey of learning in multiagent environments: Dealing with non-stationarity. CoRR, 0os//777700183, 2017. URL
org. abs ab91177709118 09183
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey, Meta-learning in neural
networks: A survey, arXiv preprint arXiv:2004.05433) 2020.
Khurram Javed and Martha White. Meta-Icarning representations for continual Iearn- ing. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Neural Information Processing Systems (NeurlPS), pp- 1818 1828. Curran Associates, Inc., 2019. URL http: /pppppaappeespesssaaeeeaaaaaaeaa cct.pperrcpppprr/ 8458 meta rep.armiiing-eeeestttttaattoo forss-fo--oonnnnn -ewtontnnnnnll.ee.gnnnn.....
Alistair Letcher, Jakob Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. Stable opponent shaping in differentiable games. In International Conference on Learning Representations (ICLR), 2019. URL httpp::/
net
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competiive environments, In Neural Information Processing Systems (NeurIPS), pp. 6382-6393, 2017.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. arXiv preprint arXiv:1707.0314), 2017.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning (ICML), 2016. URL http:// proceedings . mmr.. www...press...m v48 ht/.//mna......m/mmmmm.mmmmmm.ea
Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999, 2018
Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, and Stefano V. Albrecht. Dealing with non-stationarity in multi-agent deep reinforcement learning, CoRR, abs/1906.04437, 2019. URL
http://arxiv. org/ abs 1etp/9966.0000 04737
Roberta Raileanu, Emily Denton, Arthur Szlam, and Rob Fergus. Modeling others using oneself in multi-agent reinforcement learning. In International Conference Machine Learning (ICML), volume 80, pp. 4257-4266, 10-15 Jul 2018. URL http:: http:://ppooeeeengg.iiee /tttttppp//ppoeeeiinn.. p
v80 1800/rallee irranllllll htmml
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. International Conference on Learning Representations (ICLR), 2019.
Jurgen Schmidhuber. Evolutionary principles in self-referential learning. on learning now to learn: The meta-metas -hok Diploma thesis, Technische Universitat Munchen, Germany, 14 May 1987. URL http:: idsia ch/ www.chhwwtteceeeeeeeeee diip 1 oma html
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimension continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
L. S. Shapley. Stochastic games. Proceedings of the National Academy of Sciences, 39(10):109- 1100. 1953. ISSN 0027, doi: 10. 10.10073/paa.39.100.r0 10.00.10955 URL https: /www pnas bre/ccontemt/33/1/1055

---

Under review as a conference paper at ICLR 2021
Giacomo Spigler. Meta-learnt priors slow down catastrophic forgetting in neural networks. arXiv preprint arXiv:1909.041700 2019. URL http:::://axivv.oo////////o////
04170.pdr
Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981.
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial intelligence review, 18(2):77-95, 2002.
Jane X Wang. Zeb Kurth-Nelson Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv: /6/ 1.05763, 2016a.
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Rémi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficientt actor-critic with experience replay. CoRR, abs/ 1.01224 2016b. URL htttt:://axxi....itteetttt wwt.aa.www.....a...a..... abs/ 01224
Ermo Wei, Drew Wicke, David Freelan, and Sean Luke. Multiagent soft Q-learning. CoRR, abs/1804.00 184118804191, 2018. UtRphhtt:///xxxi
absa 00880098818777
Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning. In International Conference on Learning Representations (ICLR), 2019. URL httpp::/ htgp:p/reeeeeeeetteoooorrrrrr... wow.ooommm-iiliaeeeerreee
net
Ronald Williams. Simple statistical gradient-following algorithms for connectionist rein- forcement learning. Machine Learning 8(3):229-256, May 1992. ISSN 1573-0565. doi:
10.1007/BF00992666
Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning. In International Conference Machine Learning (ICML), volume 80, pp. 5571-5580, 10-15 Jul 2018. URL htttp://prooeeedingg. mlr wtt..perrssvv/yyggllll. html
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint crXiv:2001.06782 2020.
Chongjie Zhang and Victor R. Lesser. Multi-agent learning with policy prediction. In Association for the Advancement of Artificial Intelligence (AAAI), 2010.
12

---

Under review as a conference paper at ICLR 2021
A DERIVATION OF META-MULTIAGENT POLICY GRADIENT THEOREM
Theorem Multiagntt Policy Gradient Theorem (Meta-MAPG)). For any stochastic game Mns the gradient of the meta- meta-objective function for agent i at state so with respect to the current parameters ob of stochastic policy π evolving in the environment along with the other peer agents using initial parameters Do is:
Vi % 0ure+1
(50: (6)
E Thes1 ppplaees 1 10 +17i)) a7t1)
Tooie" ppl o Pi vorl)
V %o loga (1 To 100) +
loga (og The 108 +
% loga log @ - ( (
Current Policy
Own Leaming
Poor Leamnngg
Proof. We begin our derivation from the value function defined in Equation (3). We expand the meta-value function with the state-action value and joint actions, assuming the conditional independence between agents' actions (Wen al. 2019):
Vi
Po.e:1 (80,
Tao pl oaze vwel
E
Thets mp(roets 10 0et1)
|G' (T I]
Vi Betz (80)
Tooe ~ pl Pin mole
=E
E Iaja 30: 0f
22t1) (so, an)
Touut" pll o oo1)
on
where O'ees (50,a0) denotes the state-action value under the joint policy with parameters Φe+1
state 8p with joint action ag- In Equation (7), we note that both Φi and Φ1 depend di- Considering the joint update from Φo to Φ1. for simplicity, we can write the gradients in the inner-loop (Equation (4)) based on the multiagent stochastic policy gradient theorem (Wei et al. 2018)
V 60 To pilro t. "
E
G (TOW)
= Σ Poo (s) Σ
V
ग ( oi) ET(a ls, Φ5 b (s,a),
(8)
.E 6 Do ~pl 1i ")
G (T
-Σ Poo (s) ΣV EV
Ta saaa-ip $|s, o5' T(a'| s, Qz) (s,a),
where Pdo denotes the stationary distribution under the joint policy with parameters Φo- Importantly, the inner-loop gradients for an agent i and its peers a function of oi Hence, the updated joint policy parameter Φ1 depends on Φ Following Equation (8), the successive inner-loop optimization until Φet1 results in dependencies between Φb and Φi:c+1 and between of and Φ1:411 (see Fig- ure lb). Having identified which terms are dependent on Φ/ continue from Equation (7) and derive the gradient of the meta-value function with respect to di by applying the product rule:
Vi Viu.er (so: b6)
%
Vog
E
Σ (Tablss 2641) Σ
T
-1 2 (so, ao) |so,
ob
V d6
Σ pl Toole Bhee va.i)
Σ
T |so, Dbtr)
Σ T T
80, $ズ 1 (80, (80.
40
V
Σ p(Tenu D vati)
Σ
n ( so. )mm)
Σ
son
- Do
(5o. ao)
Tohoe
ao
Term
Σ P(Toueel D vark)
Σ 06
(abloo pbt1)
Σ π (
1800 W
(80. ao) +
Topu
a
Torm B
Σ p(Tooul Doxe v5.i) E
(aablo o
Σ V T (s..://ss. ) Doru. (so. ao) + ap
Toooe
ab
Torm C
13

---

Under review as a conference paper at ICLR 2021
Σ p(T oe voil) Σ π (agl500 (241) Σ T (00000411 (80. ao) (o) (9) Toou
ao
Term D
We first focus on the derivative of the trajectories Toboe in Term A:
Σ
p(Teme
& Pore )o.e)
Σ p(Toulob 10: 00 E
p(Tor 1i. 0T x x...X Ep(Torlo' Toe
vb
Tio
TMp
Σ V VopD(Toolle (6.) 45
[I .0100) E p(Tou 1o +
(10)
-eo
Σ
o: pT 101, $1
Ivr
p(Ter 10 $E +...+
T01
Σ ( Φ;
Σ p(T [I 10 $F*)
Tue
where the probability of collecting a trajectory under the joint policy with parameters Φe is given by:
p( 0, p(so) I (aa) st, ( St, Φ P at). Using Equation (11) and the log derivaavve trick, Equation (10) can be further expressed as:
(11)
E To pl
00 V ob log N (T (66) IT Ivve E( 1 &E
The
E Ta, ~p(oo 10 11 Vag llogT( (0)) log ( Ire Tap
Ep(Tol 18 Φ5
+...+
(12)
T [T up(To (0) ) (logT (To + toogT(TT, 10)))) Ep(Ton [vve &E
Tee
where the summations of the log-term fer-terms, such as Vay v5 (log T ( (T +lg (T (rrex(00")) 2) are inherently included due to the sequential dependencies between di and dr We use the result of Equation (12)
and organize terms to arrive at the following expression for Term A in Equation (9):
IE Tooie ~ P(Too.e & Bo
V o6 log ( (6)) + Σ og log ( 10b4)) +
Σ
Vog log ( (o+z))
F==0
Σ [a %
pb+1) n ( - 180 sz
Σ
(80 ao)
Q (80,
(13)
ao
Coming back Term B-D in Equation (9), repeatedly unrolling the derivative of the Q-function dg P (80,a0) an) by following Sutton & Barto 1998) yields:
Qt V
E Tooi upl 1 tt
Σ Porea (s) VV
(
(a'ls,
Σ T(a ET T(a क (2)
(14)
E Toou Te ( @b.e: )
Σ Edee (s)
Σ
T
- o E
D (a 1 (8, (s)
which adds the consideration of future joint policy det1 to Equation (13). Finally, summa- rize Equations (1) and (14) together and express in expectations:
V Vi Voueee (so, Hb)
F.
"oooe ~pl pooTTooeele 18b.
E
Toees up(Teeea 08 $
V logt (Too 106)+
logt (T (
loga (Toee +
(T (
)==0)
Current Policy
Own Leaming
Poer Learning
14

---

Under review as a conference paper at ICLR 2021
B META-MAPG WITH OPPONENT MODELING
Algorithm Meta-Learning at Training Time with Opponent Modeling
Require: p(6,"): Distribution over other agents' initial policies; 8, å: Learning rates
I: Randomly initialize
2: while Φ has not converged do
Sample a meta-train batch of Pu p(0o')
4: for each Φ5 do
5: Randomly initialize O for f o..., do
6:
Sample and store trajectory Tiex
8:
Approximate क- f (6 Tée â) using opponent modeling (Algorithm 4)
9:
Compute Φe+1 = (e. Tok: o:) from inner-loop optimization (Equation (4))
Compute Φ f' -Tee: a) from inner-loop optimization (Equation (4))
11:
end for
12:
end for
Update db db + BE ob V Vouees (so, Φb) based Equation (6) and Φ
14: end while
Algorithm 4 Opponent Modeling
I: procedure OPPONENT MODELING( Toee ài)
2: while O has not converged do
3: Compute log- liglliieldhoo Likeinood f(0, -Te) based Equation (15)
4: Update &5- 35¹ &V 8n Lmaethond
end while return oz
7: end procedure
In this section, explain Meta-MAPG with opponent modeling for settings where meta-agent cannot access the policy parameters of its peers during meta-training. Our decentralized meta- training method in Algorithm 3 replaces the other agents' true policy parameters Φ with inferred parameters Φ in computing the peer learning gradient. Specifically, follow Foerster al. (0188)) for opponent modeling and estimate ox from Tæe using log-likelihood Likelihood (Line in Algorithm 3):
H
Likellibood Σ
log "(a,"'aa1,00 Z5²)
(15)
where St, a₂ Tée A meta-agent can obtain S- 4 by iteratively applying the opponent modeling procedure until the maximum chain length of L. We also apply the inner-loop update with the Differentiable Monte- Carlo Estimator (DiCE) (Foerster et al. 2018c) to the inferred policy parameters of peer agents (Line 10 in Algorithm 3). By applying DICE, we can save the sequential dependencies between di and updates to the policy parameters of peer agents Φ in computation graph and compute the peer learning gradient efficiently via cilomatic-differeniin (Line 13 Algorithm 3).
C ADDITIONAL IMPLEMENTATION DETAILS
C.1 NETWORK STRUCTURE
Our neural networks for the policy and value function consist of a fully-connected input layer with 64 units followed by single-layer LSTM with 64 units and fully-connected output layer. We reset the LSTM states to zeros at the beginning of trajectories and retain them until the end of episodes. The LST policy outputs a probability for the Berouuli distribution in the iterated games (i.e., IPD,
15

---

Under review as a conference paper at ICLR 2021
RPS). For the 2-Agent HalfCheetah domain, the policy outputs and variance for the Gaussian distribution. We empirically observe that no parameter sharing between the policy and value network results in more stable learning than sharing the network parameters.
C.2 OPTIMIZATION
We detail additional important notes about our implementation:
We apply the linear feature baseline (Duan et al. 201 20) and generalized advantage estimation (GAE) (Schulman al., 2016) during the inner-loop and outer-loop optimization, respectively, to reduce the variance in the policy gradient.
We use DiCE (Foerster et al. 2018c) to compute the peer learning gradient efficiently. Specifically, we apply DiCE during the inner-loop optimization and save the sequential dependencies between ob and &1-1 in a computation graph. Because the computation graph has the sequential dependencies, we can compute the peer learning gradient by the backpropagation of the meta-value function via toolbox.
the
Learning from diverse peers can potentially cause conflicting gradients and unstable learning. In IPD, for instance, strategy to adapt against cooperating peers can be completely opposite to the adaptation strategy against defecting peers, resulting in conflicting gradients. To address this potential issue, outer-loop optimization. We also have tested the baseline methods with PCGrad.
the projecting conflicting gradients (PCGrad) (Yu et al. (2020) during the
We use a distributed training to speed up the meta-optimization. Each thread interacts with a Markov chain of policies until the chain horizon and then computes the meta-optimization gradients using Equation (6). Then, similar to Mnih et al. (2016), each thread asynchronously updates the shared meta-agent's 's policy and value network parameters.
D ADDITIONAL BASELINE DETAILS
We train all adaptation methods based meta-training set until convergence. We then measure the adaptation performance on a meta-testing set using the best-learned policy determined by meta-validation
D.1 META-PG
We have improved the Meta-PG baseline itself beyond its implementation in the original work (AI: Shedivat et al. 2018) to further isolate the importance of the peer learning gradient term. Specifically, compared to Al-Shedivat Al--SShhiii
al. (2018). we make the following thcoretical contributions build on:
Underlying problem statement. Al-Shedivat et al. (2018) bases their problem formulation off that of multi-task/ / continual single-agent RL. In contrast, ours is based on general stochastic game between n agents (Shapley 1953).
Markov chain joint policies. Al-Shedivat al. (2018) treats an evolving peer agent as an external factor, resulting in the absence of the sequential dependencies between a meta-agent' 's current policy and the peer agents' future policies in the Markov chain. However, our important insight is that the sequential dependencies exist in general multiagent settings as the peer agents are also learning agents based on trajectories by interacting with meta-agent (see Figure lb).
Meta-objective. The meta-objective defined in Al-Shedivvt et al. (018)) is based on single-agent settings. contrast, our meta-objective based on general multiagent settings (see Equations ? to
ootiiiiptmiz gradient. Compared to Al-Shedivat et al. (2018), our meta-optimization gradient inherently includes the additional term of the peer learning gradient that considers how an agent can directly influence the learning process of other agents.
Importance sampling. Compared to Al-Shedivat et al. (2018). we avoid using the importance sampling during meta-testing by modifying the meta-value function. Specifically, the framework uses meta-value function on pair consecutive joint policies, denoted V Venees (so, g6). which assumes
16

---

Under review as a conference paper at ICLR 2021
initializing every 6) from oy. However, noted in Al-Shedivat et al. 2018), this assumption requires interacting with the same peers multiple times and is often impossible during meta-testing, To address this issue, the framework uses the importance sampling correction during meta-testing. However, the correction generally suffers from high variance (Wang et al. (0))))). As such, we effectively avoid using the correction by initializing from % only once at the beginning of Markov chains for both meta-training and meta-testing.
The above theoretical differences have resulted in an improved meta-agent that can learn addition- ally affect future policies of other peer agents, achieving better results than the Meta-PG baseline in our experiments.
D.2 LOLA-DICE
We used open-source PyTorch implementation for LOLA-DiCE We make minor changes to the code, such as adding the LSTM policy and value function.
E ADDITIONAL EXPERIMENT DETAILS
E.1 IPD
Population Initial Persona p(oaa)
Agent
Cooperative Persona
Trajectory
Agent
Defective Persona
Joint Policy tbe
Joint Policy 4
Joint Policy Ór.
Figure 6: IPD meta-learning setup. An agent j's policy is initialized randomly from the initial persona population p(Ø5") that includes various cooperating and defecting personas. The agent then updates its policy throughout the Markov chain, requiring agent adapt with respect the learning of j.
We choose to represent the peer agent j's policy as a tabular representation to effectively construct the population of initial personas p( (5-") for the meta-learning setup. Specifically, the tabular policy has a dimension probability between 0.5 and 1.0 and probability between 0 and 0.5 each state to construct the cooperating and defecting population, respectively. As such, the tabular representation enables us to sample as many as personas but also controllable distribution p(Ø5²) by merely adjusting the probability range. We sample total of 480 initial personas, including cooperating personas and defecting personas, and split them into 400 for meta-traing meeaa-taiinnn, 40 for meta- valad-aaii,tin,, and 40 for meta-testing. Figure(3b) F visualizes the distribution, where we used the principal component analysis (PCA) with two components.
that corresponds to the number states in IPD. Then, we randomly sample a
E.2 RPS
In RPS, we follow the same meta-learning setup as IPD, except we sample a total of 720 initial opponent personas, including rock, paper, and scissors personas, and split them into 600 for meta- training. 60 for meta-validation, and 60 for meta-testing. Additionally, because RPS has three possible actions, we sample rock preference probability between 1/3 and 1.0 for building the rock persona population, where the rock probability is larger than the other two action probabilities. We follow the same procedure for constructing the paper and scissors persona population.
Available https: / thub. com/a lexi jacq/ L.O
17

---

Under review as a conference paper at ICLR 2021
E.3 2-AGENT HALFCHEETAH
We used an open source implementation for multiagent-MuloCo benchmark P Agents in our ex- periments receive state observations that include in- formation about all the joints. For the meta-learning setup, we pre-train a teammate with an LSTM pol- icy that has varying expertise moving to the left direction. Specifically, we train the teammate up to 500 train iterations and save a checkpoint at each eration. Intuitively, as the number of train iteration increases, the teammate gains more expertise. We then use the checkpoints from 50 to 300 iterations the meta- train/val and from 475 and 500 iterations the meta-fest distribution (see Figure 7 We con- struct the distribution with the gap to ensure that the meta-testing distribution has sufficient difference the meta-train/val so that we can test the generaliza- tion of our approach. Lastly, the teammate agent j updates its policy based on the policy gradient with the linear feature baseline as in IPD and RPS.
2-Agent HallCheetah Population Visualization
Average Reward
ham
Train Iteration
Moto Moatn/oann/Sa
Kota Thxt
Figure 7: Visualization a teammate j's ini- tial expertise in the 2-Agent HalfCheetah do- main, where the meta-test distribution has a sufficient difference meta veta-train/val.
F IMPORTANCE OF PEER LEARNING
Example 1. Failure to consider the learning process of the other agents can result in divergence of learning objectives.
For example, consider stateless zero-sum game playing hetween two agents. Agents and max- imize simple value functions V = 66 & and V -Ø'o>"" respectively, where o² od E R. In this game, there exists a unique Nash equilibrium at the origin (i.e., { d {0, 0})) We compare: the standard approach that optimizes the value function in Equa- tion (1) with the stationary assumption and 2) an approach that considers the learning process of oth- ers, such as the LOLA method. As Figure g shows, the standard approach diverges further from the equi- librium, resulting in worse results for both agents. The cause of the failure in this example is due the stationary assumption that each agent assumes its op- ponent has the same behavior in the future (Letcher et al. 2019). In contrast, by considering the learn- ing process of the opponent, the LOLA approach converges to the equilibrium. As such, it is impor- tant consider the learning of the other agents highlighted by this example.
LOLA
Figure 8: Learning paths on the zero-sum game. The standard approach with the station- ary assumption diverges, resulting in worse performance for both agents. In contrast, an approach that considers the learning process of the other agents, such as LOLA (Foerster) et 2018a), converges to the equilibrium.
Available http:///w.aoeee.oooeeee/eeeeeeae
18

---

Under review as a conference paper at ICLR 2021
G
ANALYSIS ON JOINT POLICY DYNAMICS
G.1
IPD
Meta-PG Agent Markov Chain Dynamics
Peer Agent Markov Chain Dynamics
Action Probability
0.8
Probability
0.8
0.6-
0.6
0.4
11.4-
00
4/2
02
0%
03
41
62
0%
(3
Joint Policy
Joint Policy
Dofect
Conperate
Figure 9: Action probability dynamics with Meta-PG in IPD with cooperating persona peer
LOLA-DICE Agent Markov Chain Dynamics
Peer Agent Markov Chain Dynamics
1.0
Action Probability
0.8
Action Probability
0.8
0.6
0.6
0.4
0.4
0.21
0.2
90
61
64
6:
01
66
41
07 9-
61
6-
05
Joint Pollicy
Joint Policy
Delect
Cooserate
Figure 10: Action probability dynamics with LOLA-DiCE in IPD with a cooperating persona peer
REINFORCE Agent Markov Chain Dynamics
Peer Agent Markov Chain Dynamics
Action Probability
0.8
Action Probabilit
066
0.6
0.4
0.4
0.2
0.2
0.0
0.0
96
01
03
0. Joint Policy
0%
94
O7
01
82
6%
0-
05
04
Joint Policy
Defecc
Cooperate
Figure 11: Action probability dynamics with REINFORCE in IPD with cooperating persona peer
Meta-MAPG Agent Markov Chain Dynamics
Peer Agent Markov Chain Dynamics
1.0
Action Probability
0.8
Action Probability
0.8
0.5
0.8
0.4
0.0
00
64
67
Joint Policy
Joint Policy
Defect
Cooperate
Figure 12: Action probability dynamics with Meta-MAPG in IPD with cooperating persona peer
19

---

Under review as a conference paper at ICLR 2021
G.2 RPS
Meta PG Agent Markov Chain Dynamics
Peer Agent Markov Chain Dynamics
Action Probabiilty
0.8
Action Probability
0.8-
0.6
0.6
041
0.81
42
61
62
6/
5:
66
67
Joint Palicy
Joint Policy
Rock
Pawer
Scisors
Figure 13: Action Probability Dynamics with Meta-PG RPS with a scissors persona opponent
LOLA-DICE Agent Markov Chain Dynamics
Peer Agent Markov Chain Dynamics
1.0
Action Probability
Action Probability
0.2
0.2
0.0
Joint Pollcy
Joint Policy
Scisoos
Figure 14: Action Probability Dynamics with LOLA-DiCE in RPS with scissors persona opponent
REINFORCE Agent Markov Chain Dynamics
Peer Agent Markov Chain Dynamics
Action Probability
Action Probability
06-
0.6
0.4
0.4
0.2
0.2
0.0
0.0
01
011
05
Oc.
00
02
Oc
Joint Policy
Joint Palicy
Paper
Sciesors
Figure 15: Action Probability Dynamics with REINFORCE in RPS with a scissors persona opponent
Meta-MAPG Agent Markov Chain Dynamics
Peer Agent Markov Chain Dynamics
Action Probability
0.8
Action Probability
0.5
066
AU6
0.1
0.1-
0.2
0.2
0.0
0.0
090
00
11 Joint Policy
00
09
01
45
65
06
00
Joint Policy
Rock
Paper
Sciwerr
Figure 16: Action Probability Dynamics with Meta-MAPG in RPS with scissors persona opponent
20

---

Under review as a conference paper at ICLR 2021
H
HYPERPARAMETER DETAILS
We report our hyperparameter values that we used for each of the methods in our experiments:
H.1 META-MAPG AND META-PG
Hyperparameter
Value
Trajectory batch size K
4, 8, 16, 32, 64 5
Number of parallel threads
Actor learning rate (inner)
1.0, 0.1
Actor learning rate (outer)
le-4
Critic learning rate (outer)
1.5e-4
Episode horizon H
150
Max chain length I.
GAE À
0.95
Discount factor y
0.96
Table 3: IPD
Hyperparameter
Value
Trajectory batch size K
64
Number of parallel threads
5
Actor learning rate (inner)
0.1, 0.01
Actor learning rate (outer)
le-5
Critic learning rate (outer)
1.5e-5
Episode horizon H
50
Max chain length L
GAE X
0.95
Discount factor y
0.90
Table 4: RPS
Hyperparameter
Value
Trajectory batch size K
64
Number of parallel threads
5
Actor learning rate (inner)
5e-3
Actor learning rate (outer)
5e-5
Critic learning rate (outer)
5.5e-5
Episode horizon
200
Max chain length
GAE
0.95
Discount factor ?
0.99
Table 5: 2-Agent HalfCheetah
21

---

Under review as a conference paper at ICLR 2021
H.2 LOLA-DICE
Hyperparameter
Value
Trajectory batch size
4, 8, 16. 1.0
32, 64
Actor learning rate
Critic learning rate Episode horizon H Max chain length
1.5e-3
150
7
Number of Look-Ahead
3,
Discount factor
0.96
Table 6: IPD
Hyperparameter
Value
Trajectory batch size K
64
Actor learning rate
0.1, 0.01
Critic learning rate
1.5e-3
Episode horizon
50
Max chain length L
7
Number of Look-Ahead
Discount factor y
0.90
Table 7: RPS
Hyperparameter
Value
Trajectory batch size K
64
Actor learning rate
5e-3
Critic learning rate
1.5e-4
Episode horizon H
200
Max chain length L
Number of Look- Ahead Discount factor ?
0.99
Table 8: 2-Agent HalfCheetah
H.3 REINFORCE
Hyperparameter
Value
Trajectory batch size K
4, 8, 16, 32, 64 1.0, 0.1
Actor learning rate
Episode horizon H
150
Max chain length L
5
Discount factor y
0.96
Table 9: IPD
Hyperparameter
Value
Trajectory batch size K
64
Actor learning rat
0.1,
0.01
Episode horizon /
50
Max chain length L
7
Discount factor y
0.90
Table 10: RPS
22

---

Under review as a conference paper at ICLR 2021
Hyperparameter
Value
Trajectory batch size K
64
Actor learning rate
5e-3
Episode horizon H
200
Max chain length L
2
Discount factor 7
0.99
Table 11: 2-Agent HalfCheetah
23


Reviews:
[{"review_id": "x3DMpk4zHB5", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies meta-learning in multi-agent reinforcement learning. It proposes a meta multi-agent policy gradient method that considers the learning processes of other agents in the environment for fast adaptation. This method can be seen as a unified framework of previous methods (Al-Shedivat et al. (2018) and Foerster et al. (2018a)). The method outperforms previous methods in two matrix games and 2-agent HalfCheetah.\n\nPros:\n- The method is simple and well motivated. It additionally takes into consideration peer learning, comparing to Al-Shedivat et al. (2018).  \n- The method unifies the benefit of Al-Shedivat et al. (2018) and Foerster et al. (2018a). \n- The method greatly outperforms these two methods in two matrix games. \n\nCons:\n- Like LOLA, the method needs the access to policy parameters of other agents, while Al-Shedivat et al. (2018) do not. This may be impossible in mixed and competitive environments. How to deal with this? \n- In experiments, most questions are answered by the two matrix games. It is not fully convinced since the state space is very limited. Why not choose RoboSumo in Al-Shedivat et al. (2018) as an experiment? \n- For two matrix games, opponent policy is limited compared to complex environments, for example, halfcheetah. Although the out of distribution is tested, it is less informative for generalization. Why not test the out of distribution for halfcheetah? \n- \"The out of distribution has a smaller overlap between meta-train/val and meta-testing distribution.\" What exactly is the out of distribution?\n- The experimental results need to be more elaborated. Why do Meta-PG and LOLA perform similarly to REINFORCE?\n\n---\n**After rebuttal**\n\nThe responses address my main concerns. I have increased the score to 6. But, I also agree with other reviewers that the novelty of this paper is somewhat limited. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Pnmdmj--je7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of meta-learning in a multi-agent environment under the assumptions that:\n* the learning agent's policy evolves over time as a function of the other agent's actions\n* the other agents' policies evolve potentially using the learning agent's actions.\nThe policy learning problem is assumed to be Markovian. The meta-learning problem is considered to be that\nof finding the best initial policy parameters (that will subsequently be evolved according to the learning dynamics) as to maximize the agent's cumulative marginal payoff.\n\nThe paper is very well written, easy to read and relatively straightforward in its exposition. I do not have any big remarks about writing except that the authors may want to rethink the term \"defective persona\" to avoid the weird double meaning. A sufficient amount of related work presented and the lineage of the ideas is traced convincingly well. \n\nThe main contribution of this paper is to extend the ideas of Al-Shedivat et al. in a way that exposes the other agent's learning dynamics to the policy optimization (as opposed to treating them as a non-stationarity). The policy gradient form corresponding to this setting is derived in Theorem 1. The approach is evaluated in a synthetic experiment using iterated games as well as a somewhat less synthetic experiment on a quarter-cheetah problem (each agent controls a leg of the half-cheetah).\n\nI think that while the paper is incremental, the point that is raised within is rather intriguing. If anything my main criticism is that the authors could have gone for a more challenging setting that iterated games. E.g. recent results (https://arxiv.org/pdf/1901.08654.pdf) indicate that in settings like collaborative exploration, being aware of the other player's learning dynamics is important for achieving a better outcome.  Perhaps the policy gradient approach can solve issues that cannot be addressed straightforwardly within the bandit framework. Another question is whether the approach can be used successfully to tune the inner learning process, e.g. by incorporating the policy gradient step size and other hyper-parameters into phi_0.\n\nOverall I think this is a solid paper, which would benefit significantly from more ambitious problems.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4He0guKRGZP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper points out that a key challenge in MARL is the non-stationarity of other agents' policies, as opposed to previous papers which only account for non-stationarity of the environment. The paper extends (Al-Shedivat et al., 2018) by directly conditioning the meta-policy on a distribution of other agents' policies. In my opinion, the major contribution of this paper is a new multiagent meta learning theoretic framework that explicitly accounts  for the dynamics of all agents. \n\nStrengths of the paper:\n1) A new perspective in MARL that considers nonstationarity of MARL in terms of dynamics of the other agents' policies\n2) A new theoretically grounded algorithm that explicitly models the policy dynamics of all agents\n\nWeaknesses of the paper:\n1) Except for the new perspective of incorporating policy dynamics of other agents, the backbone of the paper (i.e., meta-RL based framework to mitigate non-stationarity of MARL) is inherently the same as (Al-Shedivat et al., 2018). The novelty is somewhat limited. \n2) In experiments the paper answers several questions that show the effectiveness of the new algorithm. However, this is subject to the two-agent setting. It is questionable whether such a framework can perform well in settings where there are multiple agents. \n\nQuestion: does the proposed framework generalize to >2 agents scenarios? if yes, what is the reason that the authors did not conduct empirical evaluations in these scenarios? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "43J4Xef9sIB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper makes a full derivation of the meta-learning gradient estimation for multi-agent adaptation.\nThe resulting algorithm combines the meta-learning of the opponent's updates (existing in LOLA) and of oneself's futur updates (existing in Meta-PG).\n\nWhile the theoretical part of the paper is clear and well explained, the experimental setup is missing a lot of details to be interpreted:\n- In each experiment, it seems (but never explicitly formulated) that \"agent i\" (agent 1, since all experiments are involving 2 players) is doing the meta-learning algorithm (meta-MAPG, meta-PG or LOLA) while the other (agent 2) is a naive agent initialised with defective/cooperative policies.\n- In that case: how are naive agent updated? with simple policy-gradient?\n- How many lookahead are used (denoted by L in algorithms)?\n- Why did LOLA failed at learning to cooperate with the cooperative opponents? (it should have learned to cooperate, unless naive agents are still doing selfish PG updates --and in that case, meta-MAPG results are very impressive)?\n- Are the opponent's policies given or learned (i.e. with opponent modelling)? \n\nAlso, I would have been interesting to see an ablation study showing the importance of the \"own learning\" and \"peer learning\" terms in equation 6 (from the same implementation with fixed HP). Does the authors have tried it?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of multi-agent meta-learning. It can be viewed as extending Al-Shedivat et al. (2018) by incorporating the dynamics of other agents. The reviewers praised clear writing and theory. There were two main concerns. The first concern is the novelty when compared to Al-Shedivat et al. (2018). The second concern are experiments, which could be more ambitious and are not always clearly described.

The reviews of this paper were borderline and this was not enough to get accepted.
Decision: Reject