Influencing Long-Term Behavior in Multiagent Reinforcement Learning
Authors: Dong-Ki Kim
Matthew Riemer
Miao Liu
Jakob Nicolaus Foerster
Michael Everett
Chuangchuang Sun
Gerald Tesauro
JONATHAN P HOW
Venue: NeurIPS 2022 Accept (2022)
Keywords: multiagent reinforcement learning
active Markov game
active average reward formulation

Abstract: The main challenge of multiagent reinforcement learning is the difficulty of learning useful policies in the presence of other simultaneously learning agents whose changing behaviors jointly affect the environment's transition and reward dynamics. An effective approach that has recently emerged for addressing this non-stationarity is for each agent to anticipate the learning of other agents and influence the evolution of future policies towards desirable behavior for its own benefit. Unfortunately, previous approaches for achieving this suffer from myopic evaluation, considering only a finite number of policy updates. As such, these methods can only influence transient future policies rather than achieving the promise of scalable equilibrium selection approaches that influence the behavior at convergence. In this paper, we propose a principled framework for considering the limiting policies of other agents as time approaches infinity. Specifically, we develop a new optimization objective that maximizes each agent's average reward by directly accounting for the impact of its behavior on the limiting set of policies that other agents will converge to. Our paper characterizes desirable solution concepts within this problem setting and provides practical approaches for optimizing over possible outcomes. As a result of our farsighted objective, we demonstrate better long-term performance than state-of-the-art baselines across a suite of diverse multiagent benchmark domains.

Influencing Long-Term Behavior in Multiagent Reinforcement Learning
Dong-Ki Kim 1,3 dkkim93@mit.edd edu
Matthew Riemer mdriemer@us, 1mm. com
Miao Liu miao. liul@us 1mm. com
Jakob N. Foerster
Michael Everett mfe@mit.ed
Chuangchuang Sun
j jakob foerste r@eng. wx.ac.uk uk
ccsunl@mit.edu
Gerald Tesauro gtesauro@us .immeuus........mm..m. com
Jonathan P. How 1,3
how@@iit.cm
Abstract
The main challenge multiagent reinforcement learning is the difficulty of learn- ing useful policies in the presence of other simultancously learning agents whose changing behaviors jointly affect the environment's transition and reward dynamics. An effective approach that has recently emerged for addressing this non-stationarity is for each agent to anticipate the learning of other agents and influence the evolu- tion of future policies towards desirable behavior for its own benefit. Unfortunately, previous approaches for achieving this suffer from myopic evaluation, considering only finite number of policy updates. As such, these methods can only influence transient future policies rather than achieving the promise of scalable equilibrium selection approaches that influence the behavior at convergence. In this paper, we propose principled framework for considering the limiting policies of other agents time approaches infinity. Specifically, develop new optimization objective that maximizes each agent's average reward by directly accounting for the impact of its behavior the limiting set of policies that other agents will converge to. Our paper characterizes desirable solution concepts within this problem setting and provides practical approaches for optimizing our farsighted objective, we demonstrate better long-term performance than state-of-the-art baselines across suite of diverse multiagent benchmark domains.
possible outcomes. As result
Introduction
Learning in multiagent reinforcement learning (MARL) is fundamentally difficult because an agent interacts with other simultancously learning agents in shared environment [1]. The joint learning of agents induces non-stationary environment dynamics from the perspective of each agent, requiring an agent to adapt its behavior with respect to potentially unknown changes in the policies other agents [2]. Notably, non-stationary policies will converge recurrent of joint policies by the end of learning. In practice, this converged joint policy can correspond to a game-theoretic solution concept, such Nash equilibrium [3] or
generally cyclic correlated equilibrium [4], but multiple equilibria can exist for single game with some of these Pareto dominating others [5]. Hence, a critical question in addressing this non-stationarity is how individual agents should behave to influence convergence of the recurrent set of policies towards more desirable limiting behaviors.
Our key idea in this work is to consider the limiting policies of other agents time approaches infinity. Specifically, the converged behavior of this dynamic multiagent system is not due to some
MIT-LIDS PIBM-Research MIT-IBM Watson AI Lab MMila University Oxford
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

arbitrary stochastic processes, but rather each agent's 's underlying learning process that also depends on the behaviors of the other interacting agents. As such, effective agents should model how their actions can influence the limiting behavior of other agents and leverage those dependencies to shape the convergence process. This farsighted perspective contrasts with recent work that also considers influencing the learning of other agents [6-11]. While these approaches show improved performance over methods that neglect the learning of other agents entirely [12-14], they suffer from myopic evaluation: only considering few updates to the policies of other agents or optimizing for the discounted return, which only considers a finite horizon time of 1 (1-7) for discount factor y [15].
Our contribution. With this insight, we make the following primary contributions in this paper:
Formalization of multiagent non-stationarity (Section 2). We introduce an active Markov game setting that formalizes MARL with simultaneously learning agents as a directed graphical model and captures the underlying non-stationarity over time. We detail how such a system eventually converges to stationary periodic distribution. As such, the objective is to maximize its long-term rewards over this distribution and, if each agent maximizes this objective, the resulting multiagent system settles into a new and general equilibrium concept that call an active equilibrium.
Practical framework for optimizing an active Markov game (Section 3). We outline a practical approach for optimization in this setting, called FUlly Reinforcing acTTive influence witH averagE Reward (FURTHER). Our approach is based on policy gradient and Bellman update rule tailored to active Markov games, Moreover, we show how variational inference can be used to approximate the update function of other agents and support decentralized execution and training.
Comprehensive evaluation of our approach (Section 4). We demonstrate that our method consistently converges more desirable limiting distribution than baseline methods that either neglect the learning of others [14] or consider their learning with a myopic perspective [8] in various multiagent benchmark domains. We also demonstrate that FURTHER provides flexible framework such that it can incorporate recent advances in multiagent learning and improve performance in large-scale settings by leveraging the mean-field method [16].
2 Problem Statement: Active Markov Game
This work studies general multiagent learning setting, where each agent interacts with other independently learning agents in shared environment. Agents in this setting update their policies based recent experiences which are affected by the joint actions of all agents. As such, while an agent cannot directly modify the future policies of other interacting agents, the agent can actively influence them by changing its own actions. In this section, we first formalize the presence of this causal influence in multiagent interactions by introducing the new framework of an active Markov game. We then formalize solution concepts and objectives for learning within this framework. Finally, we discuss dependence on initial states and policies, detailing choices that we can make to minimize the impact of these initial conditions on behavior after convergence.
Stationary Markov Game
Active Markov Game
State Transition Dynamics
St
Ece1
Sc+2
St
9--1
9c+2
at
ace1
a
Bets
Joint Policy Dynamics
0
02+z
Figure I: Within the stationary Markov game setting, agents wrongly assume that other agents will have stationary policies into the future. In contrast, agents in an active Markov game recognize that other agents have non -on-stationary policies based on the Markovian update functions.
2.1 Directed Graphical Model of Active Markov Game
We define an active Markov game as tuple M, (I, 0,4); I= (1....,7) n. is the the
set of n agents; S is the state space; A AzzzA' the joint action space; T: T SSA-
state transition function; R= R is the joint reward function; θ x O is the joint policy

---

parameter space; and U = Xterll' is the joint Markovian policy update function. We typeset sets in bold for clarity. Compared to the stationary Markov game that effectively represents MARL with wrongly assumed stationary policies in the future, the active Markov game considers how policies change over time (see Figure 1). Specifically, at each timestep L, each agent i executes an action at a current state st E S according to its stochastic policy of ~ n² (-1s,; 0%) parameterized by 02 E O'. A joint action = {ai, az") yields a transition from 8t to 8t411 with probability T T(s-11 at). where the notation -i indicates all other agents except agent i. Each agent i then obtains a reward according to its reward function r = (st, az) and updates its policy parameters according to 4' 4'( f r where 7i- {s,; az rr. St+1} denotes agent i's transition. This process continues until the convergence of non-stationary policies. Notably, the joint policy update function 24 is function of at. which affects the state transitions and rewards, so agent can actively influence future joint policies by changing its own behavior. Modeling this influence rather than ignoring it is the main advantage of using active Markov games rather than the stationary Markov game formalism.
2.2 Solution Concepts in Active Markov Games
The formalism of active Markov games provides a principled framework for each agent to model the impact of its behavior on joint future policies. In this section, we study the theoretical convergence properties of an active Markov game and develop relevant terminology that will help us characterize this convergence. We begin by formalizing the limiting behavior as a stationary periodic distribution.
Definition 1. (Stationary k-Periodic Distribution). The limiting behavior of an active Markov game can be represented by a stationary periodic probability distribution over the joint space of states and policies, defined as stationary conditional distribution with respect to a period of onder k:
(s O so. So C) = p(s; 02 0 E)
Vt s, 80 S, O,
where t11% with % denoting the modulo operation. The stationary k- k-periodic distribution satisfies the following property as its time averaged expectation stays stationary in the limil:
1 Σ
E (m Oct1 s0, Oo 1+
Σ (se Oc|so, Co 6) Σ [T<aelse Oe) 02
(2)
T(st+1l82, a²) ((0 Te)
V9 02
Our notion a stationary k-periodic distribution provides a flexible representation for characterizing the limiting distribution, generalizing from fully stationary fixed-point convergence (when k= = 1) to the extreme case of totally non-stationary convergence (when is oo).
Having defined the joint convergence behavior of an active Markov game, we can now develop an objective that each agent can optimize to maximize its long-term rewards. Our key finding is that the average reward formulation, developed for single-agent learning [17, 18), is well suited for studying the limiting behavior of other interacting agents in multiagent learning. In particular, the average reward formulation maximizes the agent's average reward per step with equal weight given to immediate and delayed rewards, unlike the discounted return objective. Once the joint policy converges to the stationary periodic distribution, rewards collected by this recurrent set of policies govern each agent's 's average reward as 1- xc. Thus, optimizing for the average reward in an active Markov game encourages agents to consider how to influence the limiting set of policies after convergence rather than transient policies that are only experienced momentarily.
Definition 2. (Active Average Reward Objective). Each agent i in an active Markov game aims to find policy parameters 6 and update function 't' that maximize its expected average reward p² ER:
Oo=0. ag (T S 20 (0.T))
max (s, = max lim E 0',1
-
E (s:, ar)
T
T
1
Eu Ocls, 0,0) (aglsr 0z) Oe) at);
max 0
T E=1 0z.02
where T denotes the time horizon. It is important to note that Equation (3) has no preference over the large equivalence class of update functions that eventually converge an optimal limiting behavior, and we only require finding an update function in this class even if the convergence rate slow. This is advantageous for our discussion to come about solution concepts in active Markov games. However,
3

---

in our practical approach to optimization, we also optimize over the transient distribution, pushing towards solutions with lower regret by modeling our value function based on the differential returns as in Proposition 2. We also note that even if a single agent maximizes this objective, agents will not necessarily arrive at any kind of equilibrium. This is because other agents may have sub-optimal or biased update functions beyond the agent's 8 control, and rational agent can potentially seek to converge to an average reward that is better for it than that of any equilibrium as a result. Additionally, whether an agent just seeks to optimize its policy maximize its update function as well depends on the kind of solution concept that is desired. For example, finding a fixed stationary policy equates to using an update function that arrives a fixed point, whereas function if we seek to find an optimal non -oatstationary policy as in the meta-learning literature [9, 19].
also optimize over the update
If all agents maximize the active average reward objective, we arrive at a new and general equilibrium concept that we call an active equilibrium, where no agents can further optimize its average reward:
Definition 3. (Active Equilibrium). In an active Markov game, an active equilibrium is joint policy parameters 0 = -0'+ (** with associated joint update function u =0= - u=} such that:
psss, 8 0 U u (==")))) p ps, 00'0 ui i+) ViEI, EO' EU'
(4)
where U denotes the space of agent i's update functions. Our active equilibrium definition is related to non-stationary solution concepts in game theory, such as the non-stationary Nash equilibrium [20], that search for a sequence of best-response joint policies. However, these non-stationary solutions are generally intractable to compute due to the unconstrained sequence over the infinite horizon and the resulting large policy search space size. By contrast, the active equilibrium provides a more refined and practical notion than these solution concepts by having constraint on the sequence based on the update functions. We also note the generality of active equilibrium that it can correspond to other standard solution concepts as we impose restrictions relevant variables:
Remark (Connection to Existing Solution Concepts), Stationary Nash [3] and correlated equilibria [21] are special kinds of active equilibria when 1 and joint action distributions are independent and correlated, respectively. Cyclic Nash and cyclic correlated equilibria [4] are also special cases of active equilibrium if k>> 1, the joint update function is deterministic, and joint action distributions are independent and correlated, respectively.
2.3 Addressing Sensitivity to Initial Conditions
The recurrent set of converged joint policies is generally dependent on initial states and policies, as specified by the conditioned initial variables in Equations (1) to (3). This initial condition dependence implies that there can be instances of poor convergence performance simply due to undesirable initial states and policies (see Appendix A for an example). In this paper, we address this sensitivity to initial conditions by considering the stochastically stable periodic distribution, which is a special case of the stationary periodic distribution. The stochastic distribution describes the limiting joint behavior when each agent has communicating strategies (i.c., for every pair of policy parameters 0¹, 8 E O', 0² transitions to l² in a finite number of steps with non-zero probability and vice versa) by adding noise to its update function UE and noise -> O over time (i.e., lim; U = = Importantly, the stochastic distribution provides an important analytical benefit of independent convergence with respect to the initial conditions. Specifically, assuming communicating state transitions T, if only agent i's update function is perturbed, then we arrive at the notion of self-stable periodic distribution:
Definition& 4. (Self-Stable Periodic Distribution). Given communicating state transition T, if noise is added only to the agent i's update function UI we achieve the stationary k-periodic distribution independent of the initial state and the agent i's initial policy
0 over time:
1 k 1o Σ 00111 10000%ee11))
1 k
(+1)=
Σ Σ pulse, 0e/00 Σ Taelss O2)
1=186.0c
at
(5)
T(5+11132, ae) u ((0e01000.11
Vs 0.+10115
Similarly, if the full joint update function is perturbed with noise Ue: this induces a unique stationary periodic distribution independent of the initial state and initial joint policy:
Definition 5. (ointly-Stable Periodic Distribution). Given communicating state transition T, if noise e is added to the joint update function Us we achieve the same stationary k-periodic distribution

---

independent of the initial state and the initial policies as c-+ over time:
1 k Σ na (se+1, 00+11e+1)=
Σ (a Ozle) Σ x(aelse;
i= 02
(6)
az
T(se411se, at) ((n))(((0+)) 1(0 TE) A8441 ES, 0
Proposition 1. (Uniqueness of Jointly-Stabi Stable Periodic Distribution). Given communicating state transition and perturbed joint update function with noise e, the jointly- periodic distribution is unique as c-+ O over time.
Proof. See Appendix B for details.
The jointly-stable periodic distribution is induced in many cases interest to multiagent learning, including when all policies employ update functions leveraging the Greedy in the Limit with Infinite Exploration (GLIE) property [18]: 1) all state-action pairs are visited
infinitely often and 2) In particular, a broad class of action exploration or noisy policy update functions lead to this kind of distribution [22- 26). Indeed, MARL algorithms generally rely on persis- tent exploration and thus satisfy GLIE Lastly, demon- strated in Figure 2, although maximizing over the space of jointly-stable periodic distributions confines the search space of stationary periodic distributions, the best possible active equílibria still lie within this smaller space while also allowing for optimization robust to initial conditions. We focus on designing learning algorithm that can find an equilibrium in the practical and confined search space of the jointly-stable distributions in the following section.
+ OC, the behavior policy converges to the greedy policy.
Sell Stable Pariodic Ditributions
jointly Stable Periodic Ditributions
Activn Equilibna
Cydli
Equilibria
Figure Venn diagram describing rela- tionships between the proposed distribu- tions and equilibrium concepts.
3 FURTHER: Practical Method for Solving Active Markov Game
In this section, we develop a practical method, called FURTHER, for learning beneficial policies in the space of the jointly-stable periodic distributions. We first outline practical version of the average reward objective and derive its policy gradient. We then detail our model-free implementation that builds top of soft actor-critic [27] and variational inference [28] to learn policies that efficiently optimize for the average reward objective in decentralized manner.
3.1 Formulation and Policy Gradient Theorem of FURTHER
While the objective in Equation (3) ideally maximizes over the space of update functions and learns non-stationary policy, addressing the computational difficulty of long horizon meta-learning still remains an active area of research [9, 29, 30]. As such, in FURTHER, we take a practical step forward and learn the optimal fixed point policy that influences joint policy behavior to maximize its long-term average reward Prz E IR at a state E and policy parameters of other agents 0 -
50 - 0 + E Ri ar) ( ). T T
max Pa: 0 max lim E
). wn( No. ( (1 1.10," (T.."
(7)
T-
where the subscript 0 notation denotes the implicit dependence on the learning of agent i's fixed stationary policy. As discussed in Section 2.3, useful result under the jointly-stable periodic distribution is that the average reward becomes independent of the initial states and policies:
p (s, 0 ( 00* P.. "*4s'004400
(8)
We now derive the Bellman equation in active Markov game that defines the relationship between the value function and average reward.
Proposition 2. (Active Differential Bellman Equation). The differential value function uy represents the expected total difference between the accumulated rewards from and 0 and the average reward
5

---

Pos [18]. The differential value function inherently includes the recursive relationship with respect to dg. at the next state s' and the updated policies of other agents 0 0-
T
= 00 0
V/g (s. 0- = lim
E
Σ
(R) ar) - -Pzi) ob por(( (-rr(1110777 00 1. - a (-1ea:
T0000
S0t1 (e) 0 -e (-10 :p)))
T(a''s Σ a 10;00) IT
0 1
(9)
0-
R' - P + u (5', 0-1)
Proof. See Appendix C for a derivation.
Finally, we derive the policy gradient based on the active differential Bellman equation:
Proposition 3. (Active Average Reward Policy Gradient Theorem). The gradient of active average reward objective in Equation (7) with respect to agent i's policy parameters 8 is:
(0'))(00)= k
Σ Σ 0z (8t, 0c1012 V o (a) Σ 0' <(aa"1ss 0g (82, 02 ae), (10)
s1.00
oi
with 1 (se, O2 az)= ((senll ae) Σu (0 02', TEt) (se, ae) pg.. + o (se+1: 02t1)
81-1
0
Proof. See Appendix D for a derivation.
3.2 Soft Actor-Critic Implementation with Variational Inference
Algorithm overview. FURTHER broadly consists of inference and reinforcement learning modules. In practice, each agent has partial observations about others and cannot directly observe their true policy parameters 0 and policy dynamics U The inference learning module predicts this hidden information about other agents leveraging variational inference [28] modified for sequential prediction. The inferred information becomes the input to the reinforcement learning module, which extends the policy gradient theorem in Equation (10) and learns active average reward policies sample efficiently by building on the multiagent soft actor-critic (MASAC) framework [14, 27, 31]. We note that each agent interacts and learns these modules by only observing the actions of other agents, so our implementation supports decentralized execution and training. We provide further details, including
implementation for k 1 and psuedocode, in Appendix E.
For simplicity, we consider the period k= 1 and develop corresponding soft reinforcement learning optimizations in Equations (12) to (14).
Inference learning module. This module aims infer the current policies of other agents and their learning dynamics based on an approximate variational inference [28]. Specifically, we optimise a tractable evidence lower bound (ELBO), defined together with an encoder p( -(2it11 1 T/i Plme)) and a decoder p(a," (1,, 2, ddcc)) parameterised by dine and dice respectively:
Jaho E (orttt))p((1t Ird (
Σ log log / oz DDDK((((2 i oin lpp
(11)
Reconstruction loss
KT. divergence
where latent strategies 2 represents inferred policy parameters of other agents 0, the encoder represents the policy dynamics of other agents 4 with parameters d' and Th (ro..., 7 denotes i's transitions up to timestep We refer to Appendix F for a detailed ELBO derivation. By optimizing the reconstruction term, the encoder aims to infer accurate next latent strategies of other agents. Also, by imposing the prior through the KL. divergence, where we set the prior to the previous posterior with initial prior p( =0 1), the inferred policies from the encoder are encouraged to be sequentially consistent across time (i.e., no abrupt changes in policies of others).
Reinforcement learning module. This module aims to learn a policy that can maximize the agent's average reward based on the inferred information about other agents. Each agent maintains its policy (-1ss, 2 0') parameterized by 0³, two f-functions dg (s, 2 az wh) and bo (s, 2 "a; v

---

S
U
D
1) (0.0) S
U (4.4) (0.0)
( (1,2)
D (0,0) (8,8)
(a) Bach/Stravinsky
(b) Coordination
H (11-1) (L.I)
(c) Matching Pennies
(d) MuloCo RohoSumo
(e) MAgent Battle
Figure 3: (a)-(c) Payoff tables for Bach Stravinsky (general-sum), coordination (cooperative), and matching pennies (competitive) games. (d) A competitive RoboSumo domain [19] with two agents
fighting each other. (e) A mixed cooperative-comppeiive battle domain [32] with 25 vs 25 agents.
parameterized by v v and learnable average reward Pb. R. We train the q-functions and pl. by minimizing the soft Bellman residual:
2 a; (a:a)))))))))))))) v))) y=r'~ 1ogls's 2-;;; wg), (12) J (1 Pg. (,, ar yo (V- (8,
where =11,2, denotes i's replay buffer, and vi denotes the target q-network parameters. The soft value function v/j calculates the state value with the policy entropy H. and entropy weight a:
ugi(s, 2 = ",0") ET(e "|s; min (s, 2 a + ++)))) (13)
8-1.2
Finally, the policy is trained to maximize:
= = (a Σ क (a'l,, : min (s, & (a)(((1,,,,
(14)
B=122
We note that Equations (13) and (14) are for discrete action space, and we detail optimizations for continuous action space in Appendix E.
Mean-Field FURTHER. FURTHER provides flexible framework such that can easily integrate recent advances in multiagent learning. For example, by reconstructing and predicting the mean action and latent strategy of neighbor agents in Equation (11), we can incorporate the mean- feeld framework to improve performance in large-scale learning settings. Appendix E details the mean-field version of FURTHER with pseudocode.
Evaluation
We demonstrate FURTHER's ' efficacy diverse suite of multiagent benchmark domains. We refer to Appendix for experimental details and hyperparameters. The code available at https: //bit.ly/3fftt and video highlights available at https //bit. 1y/37IWebb, The mean and 95% confidence interval computed for 20 seeds are shown in each figure.
Baselines.
We compare FURTHER with the following baselines (see Appendix G.2 for details):
LILI [8]: approach that considers the learning dynamics of other agents but suffers from myopic evaluation bias by optimizing the discounted return objective (see Equation (25)).
MASAC [14]: An approach that extends SAC [27] to multiagent learning setting by having centralized critics [12]. This baseline assumes other agents will have stationary policies in the future and thus neglects their learning (see Equation (26)).
We note that these selected baselines are closely related to FURTHER, optimizing different objectives with respect - + particular, LIIII and MASAC optimize the discounted return objective with and without modeling 24 respectively. such, our baseline choices enable us to separately analyze the effect of FURTHER'S ss novel average reward objective. For completeness, we also consider additional baselines of an opponent modeling method (DRON) and an incentive MARL. method (MOA). These results are shown in Appendix H.
7

---

Avge Rewerd for Agost
2,00
IBS Comparison against Q- Q-Loaaner
Avoa Avorage Rovard for Agost
IC Comparison with Self- Sell-Play Setting
Aczamulate Rewsed
63030
IMP Comparison
0002
2000
0.00
0.20
10030
X000
1000
4200
200000
400000
800000
1000000
FURTHER
MASAC
FURTHER
MASAC
FURTHER UU
FURTHER vs MASAC
(b)
(c)
Figure 4: (a) Convergence in IBS. The FURTHER agent achieves convergence to its optimal pure strategy Nash equilibrium. (b) Convergence in IC with self-play. The FURTHER team shows better converged performance than baselines. (c) A competitive play in IMP between FURTHER and baseline methods. FURTHER receives higher rewards than LILI and MASAC over time.
Question 1. How do methods perform when playing against a q-learning agent?
We consider playing the iterated Bach or Stravinsky game (IBS; see Figure 3a). This general-sum game involves conflicting elements with two pure strategy Nash equilibria, where convergence to (B,B) and (S,S) equilibrium are more preferable from agent i's and j's perspective, respectively. Suppose agent plays against a naive learner j. such as q-learner [33], whose initial g-values are set to prefer action (S). In this experimental setting, it is ideal for agent to change j's influence behavior to select (B) such that they converge i's optimal pure strategy Nash equilibrium of (B.B).
The average reward performance when an agent i, trained with either FURTHER or the baseline methods, interacts with the q-learner j is shown in Figure 4a. There two notable observations. First, the FURTHER agent consistently converges to its optimal equilibrium of (B,B), while the LILI agent often converges to the sub-optimal equilibrium of (S,S). The FURTHER agent learns to select (B) while j selects (S), receives the worst rewards of zero, and waits until j's q-value for (S) is updated be lower than the q-value for (B). With the limiting view, learns that the waiting process is only temporary, and receiving the eventual rewards of by converging to (B,B) is optimal. By contrast, LILI suffers from myopic evaluation and shows decreased performance upon convergence because the agent prefers simply converging to the sub-optimal equilibrium rather than waiting indefinitely. Figure 5a also shows that LILI achieves sub-optimal performance for any value of y and shows unstable learning as 7 1. Second, FURTHER and LILI outperform the other approach of MASAC, showing the benefit of considering the active influence on future policies of other agents.
Question 2.
Which equilibrium do methods converge to in self-play setting?
We experiment with self-play setting in which both agents learn with the same algorithm in an iterated cooperative (IC) game with identical payoffs (see Figure 3b). This game has two pure strategy Nash equilibria of (U,U) and (D,D), in which the (D.D) equilibrium Pareto dominates the other. Figure 4b shows the average reward performance as the training iteration increases. First we find that LILI performs better than MASAC by considering the learning of agents. However, similar the IBS results, we observe that FURTHER consistently converges to the hest equilibrium of (D.D) while the baseline methods converge the sub-optimal equilibrium of (U,U) due the myopic view.
Question 3. How does FURTHER's ' limiting optimization perform directly against baselines?
We consider FURTHER agent i directly competing against either LILI or MASAC opponent j in the iterated matching pennies (IMP) game (see Figure 3c). To show that FURTHER has a long-term perspective and thus can collect more rewards than the opposing method over time, we evaluate using metric of relative accumulated reward summed up the current timestep: r Figure 4c shows that the relative accumulated reward for FURTHER is positive for both settings, meaning that FURTHER receives higher rewards than LILI and MASAC over time. This result suggests that FURTHER is more effective than LILI by employing the limiting view via the average reward formulation. This result also conveys that it is beneficial to consider the underlying learning dynamics rather than ignoring them because FURTHER can more easily exploit the MASAC opponent and achieve higher accumulated l rewards than when competing against the LIIII opponent.
8

---

IDS
Factors
Reletive Accumatarc Roword
2000
RoboSumo Comparison
Relasivve Aucumelate Roward
80000
Battle Comparison
100
3000
50000
TD baror
400002
500002
100
13
1.4
2.0
20400)
20000
100000 120000
190000
200030
(O0KE)
Average Reward for Aguni
Timastop
0.15
+ y 0.9
Hity ELOORN
PUREII
FURTHER UILI
FURTHER va MASAC
FURTHER LILAMF
(a)
(b)
(C)
Figure 5: (a) Convergence performance and corresponding TD errors with varying in LILI As 7->1, LILI shows unstable learning (i.e., large TD error). (b) competitive play in the RoboSumo
domain, showing that FURTHER can learn beneficial behavior in an environment with complex interactions (c) A mixed cooperative-competiie play in the battle domain. FURTHER-M can solve a large-scale learning settings.
Question 4. How does FURTHER scale more complex environment?
To answer this question, we use the MuJoCo RoboSumo domain ([19]; see Figure 3d), where two ant robots compete with each other with the objective of pushing the opponent out of the ring. The reward function consists of a sparse reward of for winning against the opponent and shaped rewards of moving towards the opponent and pushing the opponent further from the center of the ring. This environment has complex interactions because an agent must learn how to control its joints with continuous action space to move around the ring while learning to push the opponent. Similar to the setup in Question 3, FURTHER agent i directly competes against either LILI or MASAC opponent j. We note that each agent has only partial observations about its opponent. As such, an agent infers its opponent's 's hidden policies and learning dynamics based on partial observations. We show the RoboSumo results in Figure 5b. Consistent with results in the iterated matrix games, we observe that FURTHER gains more rewards than the baselines over time and wins against MASAC more often than against LILI. The averaged winning rate across the entire interaction shows that FURTHER wins against LILI and MASAC with 60.6% and 63.9%, respectively. Therefore, FURTHER provides a scalable framework that can learn policies in an environment with complex interactions and continuous action space.
Question 5. How does FURTHER scale to a large number of agents?
Finally, we show the scalability of our method regarding the number of agents using the battle domain ([32]; see Figure 3e). In this large-scale mixed agents and a blue team of 25 agents interact in gridworld, where each agent collaborates with its teammates to eliminate the opponents. Specifically, we compare when red and blue agents learn with the feead version of FURTHER (i.e., FURTHER-MPT and LILI (i.e., LILI-MF), respectively, where they predict the mean actions of neighboring agents. We note that all 50 agents learn a decentralized manner without sharing parameters with one another. It is evident that FURTHER-M M outperforms LILI-MF, which shows the effectiveness of having the limiting perspective. This result also conveys that FURTHER can easily incorporate other techniques in multiagent learning and show improved performance in large-scale settings.
setting, a red team of 25
5 Related Work
Stationary MARL.
The standard approach for addressing non-stationarity MARL to consider information about other agents and reason about joint action effects [34]. Example frameworks include centralized training with decentralized execution, which accounts for the actions of other agents through centralized critic [12-14, 16, 35-37J. Other related approaches include opponent modeling frameworks that infer opponent policies and condition an agent's 's policy on this inferred information about others [38-41]. While this does alleviate non-stationarity. each agent learns its policy by assuming that other agents will follow the same policy into the future. This assumption is incorrect because other agents can have different behavior in the future due to their learning [6], resulting in instability with respect to their changing behavior. In contrast, FURTHER models the learning processes of other agents and considers how to actively influence limiting behavior.

---

Learning-aware MARL. Our framework is closely related to prior work that considers the learning of other agents in the environment. The framework by [42], for instance. learns the best response adaptation to the other agent's anticipated updated policy. Notably, LOOAA [6] and its more recent improvements [7, 43] study the impact of behavior on one or a few of another agent's 's policy updates. Our work is also related to frameworks that leverage the inferred policy dynamics of other agents to impact their future policies by maximizing the discounted return objective [8, 10]. Meta-learning frameworks are also related that directly account for the non-stationary policy dynamics in multiagent settings based on the inner-loop and outer-loop optimization [9, 11, 19, 44]. Lastly, the field of incentive MARL [45-48] is related, where agents additionally optimize incentive rewards and learn successful policies in solving sequential social dilemma domains [49, 50]. However, all of these approaches only account for a finite number of updates to the policies of other agents, so we observe that these methods can converge to a less desirable solution. FURTHER addresses this issue by optimizing for the average reward objective in the active Markov game setting.
theeethhcoretic MARL. Another effective approach to addressing the non-stationarity is learning equilibrium policies that correspond to game-theoretic solution concepts [4, 51-54]. These frame- works predict stationary joint action values by the end of learning and can guarantee convergence to Nash [3] or correlated [21] equilibrium values under certain assumptions. However, noted in [55], this convergence is guaranteed only while ignoring the actual learning dynamics of other agents, and each agent assumes all agents will play the same joint equilibrium strategy. As such, equilibrium learners can fail to learn best-response policies when others choose to play different equilibrium strategies in the future as result of their learning. By contrast, FURTHER considers convergence to a recurrent set of joint policies by inferring the true policy dynamics of other agents.
Conclusion
In this paper, we have introduced FURTHER to address non-stationarity by considering cach agent's s impact on the converged policies of other agents. The key idea is to consider the limiting policies of other agents through the average reward formulation for newly proposed active Markov game framework, and we have developed a practical model-free and decentralized approach in this setting. We evaluated our method various multiagent settings and showed that FURTHER consistently converges to more desirable long-term behavior than state-of sfate-of-the-at of-the-art baseline approaches.
Acknowledgments
Research funded by IBM (as part of the MIT-IBM Watson AI Lab initiative).
References
[1] Lucian Buşoniu, Robert Babuška, and Bart De Schutter. Multi-agent Reinforcement Learning: An Overview, pages 183-221. Springer Berlin Heidelberg, Berlin, Heidelberg, 2010.
[2] Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, and Stefano V. Albrecht. Dealing with non-stationarity in multi-agent deep reinforcement learning. CORR, Miss/190.0447 2019.
[3] John F. Nash. Equilibrium points in n-person games. Proceedings of the National Academy of Sciences, 36(1):48-49, 1950.
[4] Martin Zinkevich, Amy Greenwald, and Michael Littman. Cyclic equilibria in markov games. In Neural Information Processing Systems (NeurIPS), volume 18. MIT Press, 2006.
[5] Ann Nowé, Peter Vrancx, and Yann-Michaäl De Hauwere. Game Theory and Multi-agent Reinforcement Learning, pages 441-470 1400. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012.
[6] Jakob Foerster, Richard Y. Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), page 122-130, Richland, SC, 2018
[7] Alistair Letcher, Jakob Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. Stable opponent shaping in differentiable games. International Conference on Learning Representations (ICIR), 2019.
10

---

[8] Annie Xie, Dylan Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent representations to influence multi-agent interaction. In Conference on Robot Learning (CoRL), 2020.
[9] Dong Ki Kim, Miao Liu, Matthew D Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sebastian Lopez-Cot, Gerald Tesauro, and Jonathan How. A policy gradient algorithm for learning to learn multiagent reinforcement learning. In International Conference on Machine Learning (ICML), volume 139, pages 5541-5550, PMLR, 18-24 Jul 2021.
[10] Woodrow Zhouyuan Wang, Andy Shih, Annie Xie, and Dorsa Sadigh. Influencing towards stable multi-agent interactions. In Conference on Robot Learning (CoRI.), 2021.
[11] Christopher Lu, Timon Willi, Christian A Schroeder De Witt, and Jakob Foerster. Model-free opponent shaping. International Conference on Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Research, pages
PMLR, 2022.
[12] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperativecoompeittv environments. In Neural Information Processing Systems (NeurIPS), pages 6382-6393, 2017.
[13] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. Association for the Advancement of Artificial Intelligence (AAAI), 32(1), Apr. 2018.
[14] Shariq Iqbal and Fei Sha. cetor-attention-critic for multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), volume 97, pages 2961-2970. PMLR, 09-15 Jun 2019.
[15] Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning, 49(2), 2002.
[16] Yaodong Yang. Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), volume 80, pages 5571-5580, 10-15 Jul 2018
[17] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition, 1994.
[18] Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018
[19] Maruan Al-Shedivat, Trapit Bansal, Yura Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. In International Conference on Learning Representations (ICLR), 2018.
[20] Constantinos Daskalakis, Noah Golowich, and Kaiqing Zhang. The complexity of markov equilibrium in stochastic games. CoRR, abs/2204.0399) 2022.
[21] Robert J. Aumann. Correlated equilibrium as an expression of bayesian rationality. Economet- rica, 55(1):1-18, 1987.
[22] Dean Foster and Peyton Young. Stochastic evolutionary game dynamics. Theoretical Population Biology, 38(2):219-23 2222, 1990.
[23] M.I. Freidlin, Szües, and A.D. Wentzell. Random Perturbations of Dynamical Systems. Grundlehren der mathematischen Wissenschaften. Springer, 2012.
[24] Georgios Chasparis and Jeff S. Shamma. Distributed dynamic reinforeement of efficient outcomes in multiagent coordination and network formation. Dynamic Games and Applications, 2(1):18-50, 2012.
[25] Georgios C. Chasparis. Stochastic stability of perturbed learning automata in positive-utility games. IEEE Transactions on Automatic Control, 64(11) 1):4444 4469 2019,
11

---

[26] John R. Wicks and Amy Greenwald. An algorithm for computing stochastically stable distribu- tions with applications to multiagent learning in repeated games. In Conference on Uncertainty in Artificial Intelligence (UAI), UAI005, page 623 2005.
[27] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning (ICML), volume 80, pages 1861-1870. PMIR, 2018.
[28] David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association,
Apr 2017.
[29] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Neural Information Processing Systems (NeurIPS), 32, 2019.
[30] Tristan Deleu, David Kanaa, Leo Feng, Giancarlo Kerg, Yoshua Bengio, Guillaume Lajoie, and Pierre-Lue Bacon. Continuous-time meta-learning with forward mode differentiation. arXiv preprint arXiv:2203.01443; 2022.
[31] Petros Christodoulou. Soft actor-critic for discrete action settings. CoRR, abs/1910.07207, 2019.
[32] Lianmin Zheng, Jiacheng Yang, Han Cai, Ming Zhou, Weinan Zhang, Jun Wang, and Yong Yu. Magent: A many-agent reinforcement learning platform for artificial collective intelligence. In Association for the Advancement of Artificial Intelligence (AAAI), 2018.
[33] Christopher J. C. H. Watkins and Peter Dayan. Q-learning. In Machine Learning, pages 279-292, 1992.
[34] Pablo Hemandez-Leaa, Michael Kaisers, Tim Baarslag, and Enrique Munoz de Cote. A survey of learning in multiagent environments: Dealing with non-stationarity. CoRR, abs/ 700,09987.. 2017.
[35] Shayegan Omidshafiei, Dong- Ki Kim, Miao Liu, Gerald Tesauro, Matthew Riemer, Christopher Amato, Murray Campbell, and Jonathan P. How. Learning to teach in cooperative multiagent reinforcement learning. In Association for the Advancement of Artificial Intelligence (AAAI).
AAAI Press, 2019.
[36] Samir Wadhwania, Dong-Ki Ki Kim, Shayegan Omidshafiei, and Jonathan How. Policy distilla- tion and value matching in multiagent reinforcement learning. In International Conference on Intelligent Robots and Systems (IROS), pages 8193-8200 2019.
[37] Dong- Ki Kim, Miao Liu, Shayegan Omidshafiei, Sebastian Lopez-Cot, Matthew Riemer, Golnaz Habibi, Gerald Tesauro, Sami Mourad, Murray Campbell, and Jonathan P. How. Learning hier- archical teaching policies for cooperative agents. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), AAMAS '20, page 620-628, Richland, SC, 2020. International Foundation for Autonomous Agents and Multiagent Systems.
[38] He He, Jordan Boyd-Graber Kevin Kwok, and Hal Daumé III. Opponent modeling in deep reinforcement learning. In International Conference Machine Learning (ICML), volume 48, pages 1804-1813, 20-22 Jun 2016.
[39] Roberta Raileanu, Emily Denton, Arthur Szlam, and Rob Fergus. Modeling others using oneself in multi-agent reinforcement learning. In International Conference on Muchine Learning (ICML), volume 80, pages 4257-4266, 10-15 Jul 2018.
[40] Aditya Grover, Maruan Al-Shedivat, Jayesh Gupta, Yuri Burda, and Harrison Edwards. Learning policy representations in multiagent systems. In International Conference on Machine Learning (ICMI.), volume 80, pages 1802-1811, 10-15 Jul 2018
[41] Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning. In International Conference on Learning Representa- tions (ICLR), 2019.
12

---

[42] Chongjie Zhang and Victor R. Lesser. Multi-agent learning with policy prediction. In Associa- tion for the Advancement of Artificial Intelligence (AAAT), 2010.
[43] Jakob Foerster. Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric Xing, and Shimon Whiteson. DiCE: The infinitely differentiable Monte Carlo estimator. In International Conference Machine Learning (ICML), volume 80, pages 1524-1533, PMLR, 2018.
[44] Jan Balaguer, Raphael Koster, Christopher Summerfield, and Andrea Tacchetti. The good shepherd: An oracle agent for mechanism design. arXiv preprint arXiv:2202. 10135, 2022.
[45] Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, Dj Strouse, Joel Z. Leibo, and Nando De Freitas Social influence as intrinsie motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning (ICML), volume 97, pages 1040-3049 PMLR, 2019.
[46] Jane X. Wang, Edward Hughes, Chrisantha Fernando, Wojciech M. Czarnecki, Edgar A. Dućñezz Guzmán, and Joel Z. Leibo. Evolving intrinsic motivations for altruistic behavior. In International Conference Autonomous Agents and MultiAgent Systems (AAMAS), page 683-692, Richland, SC, 2019.
[47] Jiachen Yang, Ang Li, Mehrdad Farajtabar, Peter Sunehag, Edward Hughes, and Hongyuan Zha. Learning to incentivize other learning agents. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Neural Information Processing Systems (NeurIPS), volume 33, pages 5208-15219, Curran Associates, Inc., 2020.
[48] Jiachen Yang, Ethan Wang, Rakshit Trivedi, Tuo Zhao, and Hongyuan Zha. Adaptive incentive design with multi-agent meta- gett-gradient reinforcement learning. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), page 1436-1445 Richland, SC, 2022.
[49] Joel Z. Leibo, Vinicius Zambaldi, Mare Lanctot, Janusz Marecki, and Thore Graepel. Multi- agent reinforcement learning in sequential social dilemmas. In International Conference on Autonomous Agents and MulriAgent Systems (AAMAS), page 464-473, Richland, SC, 2017. International Foundation for Autonomous Agents and Multiagent Systems.
[50] Weixun Wang, Jianye Hao, Yixi Wang, and Matthew Taylor. Achieving cooperation through deep multiagent reinforcement learning in sequential prisoner's dilemmas. In International Conference on Distributed Artificial Intelligence (DAI), 2019.
[51] Michael L. Littman. Markov games as framework for multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), pages 157-163. Morgan Kaufmann Publishers Inc., 1994.
[52] Michael L. Littman. Friend-or-foe q-learning in general-sum games. In International Confer- ence on Machine Learning (ICML), page 322-328, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc
[53] Xiaofeng Wang and Tuomas Sandholm. Reinforcement learning play optimal nash equilibrium in team markov games. In Neural Information Processing Systems (NeurIPS), page 1603-1610. MIT Press, 2002.
[54] Amy Greenwald and Keith Hall. Correlated-O learning. In International Conference on Machine Learning (ICMI., page 242-249. AAAI Press, 2003.
[55] Michael Bowling. Convergence and no-regret in multiagent learning. In Neural Information Processing Systems (NeurIPS), pages 209-216. MIT Press, 2005.
[56] Luisa Zintgraf. Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta- learning. In International Conference on Learning Representations (ICI.), 2020.
[57] He He, Jordan L. Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep reinforcement learning. CoRR, abs/1609 htts/1609.0559,, 2016
[58] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix I.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Ap- pendix I.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendices B D and
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix G.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] The mean and 95% confidence interval computed for 20 seeds are shown each figure.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix G.
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you're 'oure using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "g8OIiceVzqi", "reviewer": "Reviewer_hBd2", "summary": "This paper considers the issue of non-stationarity in decentralized multi-agent reinforcement learning by developing a new framework, called active Markov game, which explicitly models changing parameters in the environment transition. Then, following the concept of active Markov game, a more general solution concept considering the average reward is discussed. This solution concept is further implemented within the SAC algorithm with variational inference adopted to model the changing policy parameter. Experiments are conducted over matrix games, a zero-sum MuJoCo battle environment and a grid-world mixed cooperative and competitive game. ", "questions": "1. From Fig.1, we cannot see any multi-agent specific factors. So what if the number of agents degenerates to 1? Does an active Markov game degenerate into a stationary Markov game? I feel so. Some discussions will be appreciated.\n2. In the second line of equation (3) (above line 109), there is a summation over all the states. Isn't a density term over states missing?\n3. Again in equation (3), how do we choose $k$ in practice?\n4. In line 169, the authors state that ``_while maximizing over the space of jointly-stable periodic distributions confines the search space of stationary periodic distributions, this still allows convergence to the **best possible active equilibria**._''. So why could the method converge to the *best* equilibrium? Also, isn't the equilibrium unique under the Jointly-Stable Periodic Distribution? What's the gap between the two? ", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength:\nThis paper proposes a novel framework with rigorous theoretical studies. I like the idea of introducing the long-term effect into the MARL framework without simply considering the stationary converged policies. The implementation of the method is also generic. The experiments are conducted over 3 domains, which is great, even from an empirical perspective.\n\n\n## Weakness\nThe paper is generally great, but the clarity and evaluation can be possible further improved. Some of the major concerns are listed below. More detailed questions can be found in the next section.\n1. **Formulation of active Markov game**. I feel the design of the active Markov game is a bit less motivated. I think there are multiple ways to incorporate the changing policy into the formulation but why is the active Markov game finally chosen? The most confusing part is that the current formulation of the active Markov game, at least from the diagram in Fig.1, looks like a meta-learning formulation, i.e., each player adapts within a single episode. Such an impression comes from a lot of definitions: for example, $\\mathcal{U}^i$ (line 79) is defined over a single transition. However, from the content of the paper, it seems that the correct understanding is that the active Markov game intends to model the overall learning process. I think the authors could possibly improve the paper by better clarifying the insights of the active Markov game.\n2. **Evaluation of competitive games**. In general, I do feel the evaluation results on the matrix games is sufficient. However, the authors use cross-play between policies from different methods as the evaluation metric on RoboSumo and MA-Battle, which looks less convincing to me. Frankly, I would feel more necessary to show the learned strategies (or converged equilibria) by different algorithms than to show the cross-play results. There are many factors influencing the ad hoc play performance of a policy and there are lots of straightforward techniques that can help improve the zero-shot generalization capabilities, such as population training, fictitious co-play, adversarial training, etc. The current results may suggest that the FURTHER policy is more robust but I don't feel the present results could sufficiently imply the conclusion that the FURTHER converges to _a better equilibrium_. So the paper can be stronger if the evaluation of complex games is more comprehensive. \n\n## Minor Issues\nMany equations are split over two lines, e.g., Eq (2) (5) (6). I would suggest the authors put the right-hand side of the equation into a single line.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CeL-njMpTuu", "reviewer": "Reviewer_h5DN", "summary": "The paper propose a multiagent learning framework by explicitly modeling the long-term behavior of other interacting agents in the environments and show good convergence performance by playing against different types of opponents or self-play.\n", "questions": "see pros and cons section", "limitations": "see pros and cons section", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper provides a comprehensive and formal modeling and theoretical analysis of the mutiagent learning problem with opponent modeling and also gives a practical implementation version using VAE with good convergence performance.\n\nOne concern is the marginal contribution of the proposed framework. The idea of explicitly modeling the behavior of opponents is not new and it seems the main novelty of this work is to explicitly modeling the long-term behavior of opponents instead of myopic prediction. Another weakness is that the authors compared with two baselines which are not representative of the literature. There are quite a number of highly related works the authors may discuss but should compare with directly if possible, to name a few, [1-3]\n\n[1] Achieving cooperation through deep multiagent reinforcement learning in sequential prisoner's dilemmas, DAI 2019\n[2] Opponent Modeling in Deep Reinforcement Learning, 2016\n[3] A Deep Bayesian Policy Reuse Approach Against Non-Stationary Agents. NeurIPS2020.\n\nAnother suggestion is to include the sequential PD benchmarks proposed in  AAMAS17 paper\" Multi-agent Reinforcement Learning in\nSequential Social Dilemmas\", which would strengthen the evaluation results. I would be willing to increase the score if the concerns can be addressed in the rebuttal phase.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "C8VT-4pZFrH", "reviewer": "Reviewer_kSMF", "summary": "The authors propose an interesting framework to consider the limiting policies of other agents as time approaches infinity. The average reward of each agent is maximized by developing a new optimization objective that directly considers the impact of its behavior on the set of limiting set policies that the other agents will converge on. Better long-term performance than the state-of-the-art baseline is demonstrated over a range of different multi-agent benchmark domains.", "questions": "* From my point of view, the perspective of this work is very similar to the concept of incentivize, but I don't see any discussion by the authors about incentivize related, so why not? Also why no comparison? For example.\n    * Learning to Incentivize Other Learning Agents\n    * Adaptive Incentive Design with Multi-Agent Meta-Gradient Reinforcement Learning\n", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths：\n* Motivation of the paper was interesting and very important\n* The writing was very clear and the drawings helped with understanding\n\nWeaknesses：\n* The comparison method in the experimental part seems to be relatively less\n* No consideration of \"incentivize\" MARL-related work", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "2u029lqk8Hq", "reviewer": "Reviewer_FNVy", "summary": "This paper aims to address the non-stationarity coming from the active learning agents in multi-agent reinforcement learning. It proposes a novel framework for considering the convergence of policies by accounting for the influence on others. This paper also develops a new algorithm for optimizing agents under the proposed framework. Empirical results show desirable performance compared with SOTA.", "questions": "Please see Weaknesses 1, 3, 4.", "limitations": "I do not see any potential societal impact not discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\nThis paper addresses one of the most important issues in MARL, namely the non-stationarity of the environment from the perspective of individual agents. The framework is novel from my understanding. The experimental environments are well selected.\n\n\n**Weaknesses:**\n1. FURTHER uses VAE to bypass the issue of unobservable policy parameters and updating dynamics. It is a convenient choice but not immediately clear to me how to go from proposition 3 (Active Average Reward Policy Gradient Theorem) to Equation (12) which is using the latent variable. It seems that a very complex policy gradient suddenly reduces to something extremely simple. If the latent variable represents the inferred policy parameters $\\boldsymbol{\\theta}_{t}^{-i}$, where is the updating dynamics $\\mathcal{U}^{-i}$? I hope the authors could provide more explanation.\n2. VAE is notoriously difficult to train and usually requires very delicate hyperparameter fine-tuning, even for stationary data like images. I suspect it is even more difficult for non-stationary distribution. If the quality of the learned latent space is bad (e.g. mode collapse), the algorithm could break down easily.\n3. How is Equation (14) generalized to continuous action space? There is a continuous action space experiment (RoboSumo) but I did not see an algorithm for that.\n4. Although there is some discussion about the relationships between the solution concepts in Section 2.3, I still do not quite understand the intuition of why searching for Active Equilibria would result in more optimal equilibrium as argued in the experiment analysis. If LILI suffers from myopic evaluation due to the discount factor $\\gamma$, would it be helpful to set $\\gamma \\to 1$ in the episodic setting (assuming a finite horizon)?\n\n\n**Summary:**\nOverall I enjoy reading this paper as an attempt to solve a significant problem. However, I found it is hard to understand the whole picture given the gap and missing information as discussed in the weaknesses. I am willing to increase my score if the authors could address my concerns.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept