Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Games
Authors: The Viet Bui
Tien Anh Mai
Thanh Hong Nguyen
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-agent Reinforcement Learning
Imitation Learning
Primary area: reinforcement_learning
TL;DR: An approach to utilize imitation learning techniques to improve multi-agent reinforcement learning.

Abstract: Training agents in multi-agent games presents significant challenges due to their intricate nature. These challenges are exacerbated by dynamics influenced not only by the environment but also by strategies of opponents. Existing methods often struggle with slow convergence and instability.
To address these challenges, we harness the potential of imitation learning (IL) to comprehend and anticipate actions of the opponents, aiming to mitigate uncertainties with respect to the game dynamics.
Our key contributions include:
(i) a new multi-agent IL model for predicting next moves of the opponents - our model works with hidden actions of opponents and local observations;
(ii) a new multi-agent reinforcement learning (MARL) algorithm that combines our IL model and policy training into one single training process;
and (iii) extensive experiments in three challenging game environments, including an advanced version of the Star-Craft multi-agent challenge (i.e., SMACv2).
Experimental results show that our approach achieves superior performance compared to state-of-the-art MARL algorithms.

Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Games
The Viet Bui
Singapore Management University, Singapore theviet. bui. 20..00022@0@hh smu edu. sg
Tien Mai
Singapore Management University, Singapore atmai smu. edu.sg
Thanh Hong Nguyen
University of Oregon Eugene, Oregon, United States thanhhng@cs, uoregon.edo
Abstract
Training agents in multi-agent games presents significant challenges due to their intricate nature. These challenges are exacerbated by dynamics influenced not only by the environment but also by strategies of opponents. Existing methods often struggle with slow convergence and instability. To address these challenges, we harness the potential of imitation learning (II.) comprehend and anticipate actions of the opponents, aiming to mitigate uncertainties with respect to the game dynamics. Our key contributions include: (i) new multi-agent IL model for predicting next moves of the opponents - our model works with hidden actions of opponents and local observations; (ii) new multi-agent reinforcement learning (MARL) algorithm that combines our IL model and policy training into one single training process; and (iii) extensive experiments in three challenging game environments, including an advanced version of the Star-Crate Craft multi-agen agent challenge (i.e., SMACv2). Experimental results show that our approach achieves superior performance compared to state-of-the-an MARL algorithms.
Introduction
Recent works MARL have made a significant progress developing new effective algorithms that can perform well in complex multi-agent environments including SMAC [3 25). Among these works, centralized training and decentralized exccution (CTDE) (5) has attracted a great attention from the RL community due to its advantage of leveraging global information to train centralized critic (i.e., actor-critic methods (88) or a joint Q-function (i.e., value-decomposition methods [23] 291). This approach enables
efficient and stable learning process while allowing agents to act in a decentralized manner. Under this CTDE framework, off-policy methods such as MADDPG [8 and QMIX (23 have become very popular due to their data efficiency and saateof-thhe-ae art (SOTA) results on a wide range of benchmarks. On the other hand, on-policy gradient methods have been under-explored in MARL due their data consuming and difficulty in transferring knowledge from single-agent to multi-agent settings. However, a recent work shows that on-policy methods (such as MAPPO, a multi-agent version of proximal policy optimization) outperforms all other SOTA methods including MADDPG and QMIX in various multi-agent benchmarks, and especially works
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

well in complex SMAC settings [4]. performance of policy gradient methods in MARL.
Motivated by this promising result, we focus on improving the
We consider a partially observable MDP environment in which there are agents attempting to form an alliance to play against team of opponents, where allied agents have to make decision independently without communicating with other members. We aim to enhance the performance of PPO in MARL with the introduction of a novel opponent-imitation component. This new component is then integrated into the MAPPO framework to enhance the policy learning of allied agents. A key challenge in our problem setting is that allied agents are unaware of actions taken by their opponents. In addition, each allied agent only has local observations of opponents locating in the current neighborhood of the agent - the locations and neighborhoods of all players are changing over time depending on actions taken by players and the dynamics of the environment. Lastly, learning to imitate opponents occurs during the policy learning process of the allied agents. The dependency between these two learning components makes the entire learning process significantly challenging.
We address these challenges while providing the following key contributions. First, we convert the problem of imitating the opponent policy into predicting their next states. The outcome of this next state prediction is an indirect implication of the opponent policy. We then cast the problem of opponent next-state prediction as a new multi-agent imitation learning (IL) problem. We propose a new mullti-agent IL algorithm, which is an adaptation of IQ-Learn 9 (a SOTA IL algorithm), that only considers local opponent-state- policy, our IL algorithm targets the prediction of next states of the neighboring opponents. Second, we provide a comprehensive theoretical analysis which provides bounds on the impact of the changing policy of the allied agents (as a result of the policy learning process) on our II. outcomes.
observations. Especially, instead of imitating the opponents'
Third, we present a unified MARL algorithmic framework in which we incorporate our IL component into MAPPO. Our idea is combine each allied agent's local observations with the next-state prediction of neighboring opponents of that agent, creating an augmented input based on which to improve the decision making of the allied agent at every state. This novel integration results in new MARL algorithm, which we name mmitation-enhanee
Multi-Agent EXtended PPO (IMAX-PPO).
Finally, we conduct extensive experiments in several benchmarks ranging from complex to simple ones, including: SMACv2 (an advanced version of the Star-Craft multi-agent challenge) [4. Google research football (GRF) [13,, and Gold Miner (7). Our empirical results show that our new algorithm consistently outperforms SOTA algorithms significantly accross all these benchmarks.
2 Related Work
MARL. The literature on MARL includes both centralized and decentralized algorithms. While centralized algorithms (2) learn a single joint policy to produce joint actions of all the agents, decen- tralized learning [17 optimizes each agent's local policy independently. There are also algorithms based on centralized training and decentralized execution (CTDE), For example, methods in [18,5] adopt actor-critic structures and learn centralized critic that takes global information as input. value-decomposition (VD) is a class of methods that represent the joint Q-function as a function Alternativll,, the use of graadett methods, such as PPO 26), has also been investigated in multi-agent RL. For example, (3] propose independent PPO (IPPO), decentralized MARL, that can achieve high success rates in several hard SMAC maps. IPPO is, however, overall worse than QMIX [23] a method based on factorizing Q function to facilitate CTDE. Later methods based on factorized Q-learning include QTRAN (27] and QPLEX [0, where QPLEX has been shown to achive better perforamance than QMIX and QTRAN. Recently, [34] develop MAPPO, a PPO-based MARL algorithm that outperforms QMIX and QPLEX on some popular multi-agent environments such SMAC [25] 4 and GRF [15) To the best of our knowledge, MAPPO is currently a SOTA method for MARL. Our work integrates a new opponent imitation model into MAPPO, resulting in a new MARL algorithm that outperforms SOTA methods on various challenging game tasks.
of agents' local Q-functions 29
Imitation Learning (IL). In this study, we employ IL to anticipate the opponents' moves. IL is known as a compelling approach for sequential decision-making 20 []. In II, a collection of expert trajectories is provided, with the objective of learning a policy that emulates behavior similar to the expert's policy. One of the simplest II. methods is Behavioral Cloning (BC), which aims to maximize the likelihood of the expert's 's actions under the learned policy. BC disregards environmental

---

dynamics, rendering it suitable only for uncomplicated environments. Several advanced IL techniques, encompassing environmental dynamics, have been proposed [24 8 12]. While these methods operate in complex and continuous domains, they involve adversarial learning. making them prone to instability and sensitivity to hyperparameters. The IQ-learn [9] stands as a cutring-edge II. algorithm with distinct advantages, specifically its incorporation of dynamics awareness and non-adversarial training. It's important to note that all the aforementioned IL methods were designed for single-agent RL. In contrast, the literature on multi-agent RL is limited, with only handful of studies addressing IL in multi-agent RL. For instance, [28] presents an adversarial training- -aaeed algorithm, named Multi-agent Generative Adversarial IL. It's worth noting that all the IL algorithms mentioned above are established on the premise that expert (aka. opponent in our case) actions are either observable or can be accessed via sampling, which implies that no existing algorithm can be directly applied to multi-agent game settings with local state-only observations.
Opponent Modeling, Many existing works in MARL attempt to capture the learning process of For example, the LOLA algorithm 6 considers the impact of one agent's policy on the parameter update of other opponents while Meta-MAPG [14] combines LOLA with meta-learning, accounting for continuous adaptation. Another important line of research opponent modelling follows hierarchical reasoning, considering each agent holds a belief about the other agents according to varying levels of reasoning ability (B2) 31 B5 19]. As example, in [32], they introduce a probabilistic recursive reasoning framework in which variational Bayes methods are used to approximate the opponents' conditional policies. Depart from these two lines of research, there are many other works that attempt to learn representation for the opponent's policy or to consider the prediction of opponents' actions as an an
opponents and incorporate into the learning of the agent's policy G 14 33
auxiliary task that can be trained simultaneously with the RL part [I 22
21
10] For example,
21 use variational encoder to model the opponents' fixed policies while 10 apply behavioral
cloning together with agent identification to learn a hybrid annerativeddisermminatv representation for the opponents' policy, All the aforementioned related works require having access to opponent's observations and actions during training and/or execution. In our work, on the other hand, the opponent modeling can be only trained based on local observations of each agent in the allied team. These local observations contain limited information about current states of nearby opponents while opponents' actions are unobservable. As result, existing methods on opponent modeling are not applicable in our multi-agen setting.
3 Multi-Agent POMDP Setting
We consider a multi-playe Markov game in which there are multiple agents forming an alliance to play against some opponent agents. We present the Markov game as tuple (S,N,,, A", .A",, P, R). where S is the set of global states shared by all the agents, N₂ and NS, are the set of ally and enemy agents, A ILE AF is the set of joint actions and A jex; Ař, is the set of joint actions of all the ally agents, P is the transition dynamics of the game environment, and R is reward function that takes inputs as states and actions of all agents and returns the corresponding rewards. At each time step where the global state is S, each ally agent i E .V makes action of according to policy ग. (af )of), where of is the observation of ally agent given state S. The joint action of allied agents can be now defined A =aaft E N.), and the joint policy is defined accordingly:
(A*1)) = II iev' 7f (af
The enemy agents, at the same time, make joint action A = {ağl j E N.) with the probability:
II ( S)
II jev- T5(05109)
After all agents make decisions, the global state transits to a new state S' with the probability P(S'| A", A", In our setting, the enemies' policies II are fixed and thus can be treated as a part of the environment dynamics, as follows:
P(S S) = S)=
= (SSA) 9"))
Our goal is to find policy that optimizes the allies' expected joint reward, formulated as follows maxдa E E( SS- RR Aa)]
'Environment dynamics are implicitly involved in sampling.
3

---

The game dynamics involve both the environment dynamics and the joint policy of enemies, making the training costly to converge. We aim migrate uncertainties associated with these game dynamies by first predicting the opponent policy based on the allies' past observations and leveraging this prediction into guiding the policy training for the allies.
Opponent Policy Imitation
The key challenge in our problem is that actions taken by opponents are hidden from allied agents. Moreover, each allied agent has limited observations of other agents; they can only obtain information about nearby opponents. For example, in the SMAC environment, for each allied agent, besides information about the agent itself, the allied agent is also aware of the relative position and health point, etc. of the neighboring opponents P Therefore, instead of directly predicting opponents' next moves, we focus on anticipating next states of opponents - this next-state prediction can be used as an implication of what actions have been taken by the neighboring opponents. Our key contributions include: (i) a novel representation of the opponent next-state prediction in the form of multi-agent IL; (ii) new adaptation of IQ-Learn to solve our new IL problem; (iii) comprehensive theoretical analysis on the influence of policy learning of allied agents on the next-state prediction outcomes; and (iv) a practical multi-agent II. algorithm which is tailored to local observations of allied agents.
Here, it is important to note that prior works on II. with state-only observations all assume that actions are not available in the expert demonstrations but can be accessed via sampling, which is not the case in our context. Alternatively, one could apply standard supervised learning for this opponent-nent -eetstttte prediction task. However, well-known drawback of this approach is that it disregards environment dynamics and often struggles with distribution shifts [16]. As shown later in experiments, our IL approach significantly outperforms this supervised-leamien approach.
4.1 Multi-Agent IL with Unobservable Actions
We now present our IL formulation and our adaptation of IQ-Learn for solving our new IL problem. For the sake of theoretical analysis, this section focuses on IL with global state-only observations. We then introduce a new practical algorithm later which addresses local observations of allied agents.
Opponent Next-State Prediction as an IL. To formulate the problem as an IL that accounts for the action-unobservable issue, we introduce new notion of the "expert" state in our IL problem as a pair W = (S, A2) which comprises of the original state S' and the joint action of the allies A taken in the previous step that leads state S. The action space of the "expert" is equivalent to the original state space S. We then introduce a new notion of reward function for the expert as R (W, S'). This action (S') of the expert is basically a resulting state of joint actions of the allies A and hidden joint actions of the enemies A taken at state S. Altogether with the reward function RE (W, S'), we introduce a new notion of joint policy for the expert, (S" (W) (or (S')S, A which is essentially the probability of ending up at global state S' from state S. The dynamics in this setting becomes P(W W, S') = P(("", A) (S, AS), S') = IIo [A) (which is the allies" policy) where W' (S', A) and A is the action taken by the allies at state S.
VS, S, E Aa) be the support set of the imitating policy. We now introduce the maximum-entrog inverse RL framework 12 w.r.t the new notions the expert's reward and policy (R" (T)):
Let II {II S S A + [0, 1],
max min Hell
L(TI, R Epr. R (W,S") - E,n.. (W. 5') Ep..n In (((W))
(2)
where p is the occupancy measure of (W, S') given by the expert policy II and the ally joint policy II and p the occupancy measure of (W,'') given by the imitation and ally policies. In particular, peac can be computed follows:
(S', W) - (I W) 7 PPW, = W [I
An Adaptation of IQ-Learn. Drawing inspiration from the SOTA IL algorithm, IQ-learn, we construct our IL algorithm which is an adaptation of IQ-Learn tailored to our multi-agent environment.
Theee neighbors change over time depending on actions of all agents and the environment dynamics.

---

The main idea of IQ-learn is to convert reward learning problem into Q-function learning one. To apply IQ-Learn to our setting, we present the following new soft and inverse soft Bellman operators. which is as adaption from the original ones introduced in [
B (W, 12 (W, 8 + >'E [Vi(W")
(W, (W 5* - 7 Ew (Vi((("))))
where the state value function is computed as follows:
Fr(W)=Es--il ( (W)S") (W, IIT(( W))
ft is contractive, thus defining a unique fixed point solution Q' such that B =Q" Let us further define the following function of and Q" for the Q-function learning:
First, it is clear that B
JTT, = p (T (W) - E. (T (W)) + p (In (W))
We obtain theoretical result on connection between the learning reward and learning Q-functions Proposition 1. For any reward function It,, let Q" be the unique fixed point solution to the soft Bell- man equation aQ. = Q", then: L(TI, Rt) = J[TT,Q Q*), and for any Q. J(T,, Q+) = L(I,,
Proposition T indicates that the II. problem in ? equivalently to the Q-value-based II. problem:
maxq: ming (11,
(5)
Suppose Q" is solution to (5) then rewards be recovered by taking (W, (W,"")
yEW:, (W')) Under this viewpoint, Prop. shows that key properties of original IQ- still hold in our multi-agent setting with missing observations, making our IL algorithm convenient to use.
Proposition 2. The IL problem (5) is equivalent to the maximization maxqr (119, Q*) where the imitation policy can be computed based on Qf follows:
ex S')
(S Es" exp( (""(W,""))
Moreover, the function J J Q*) is concave in Q
4.2 Affect of Allies' Policies on Imitation Learning
The above IL algorithm is trained during the training of the allies' policies, which means the dynamics P(W"IW,"' S') = [14 the impact of these changes on the imitation policy. According to Proposition given Q', we can compute corresponding optimal imitation policy as II? Therefore, the value function (W) can be alternatively write (W). We now can denote the loss function of the imitation model (Eq. 8) as function of the allies' joint policy explicitly: & (TT+'Q(') JII,, Q*).
(Aa) (S) change during the IL process. Therefore, we aim to analyze
We first present our results about bounds on the impact of the allies' changing policies on the IL loss function (Prop. 3). Based on these results, we then provide a bound on the IL learning outcome
accordingly (Corollary 4b. Let us denote by Q = =max Q" (W, S') an upper bound of Q"
Proposition 3. Given two allies' joint policies 11 and Î1 such that (LI (-IS)][[T s for any state S E S, the following inequality holds:
( (Q*) ( (Q) < (aQ (l) V2
where 2 +
2 I- + (3 7), and 3 = [1-b] +3 -3-7
72
Let ([ ,00,, IT) be the objective of II. J(II, Q*), written function of the allies' joint policy ITo The following proposition establishes bound for the variation of (II" II) < | IT) as
function of (L [[I for any pair of allies' joint policies
Prop. 3 allows us to establish upper bound for the IL when the allies' joint policy changes.
All proof's are in the appendix.
5

---

(15( (4/.
Imilation Learning
non-neighbor enamy (maskiod))
MAPPP
Enermy %
ally
Actor
Ally palkyy Mg( (of (
(Decentrakized)
Mull-aagett
cnemy
enamy sf.
ghobo enery (mackedd)
L-function @ lof.ag.)
Cittic (Cantralized)
ally Si
%
Locall observation
enemy next states (S(..0))
Indicate applles aly
Global state
Figure 1: An overview of our IMAX-PPO algorithm. Each local observation of of an ally agent i includes information about itself, as well enemy and ally agents in its neighborhood (which changes over time). The output of the IL component is the predicted next states of neighboring enemy agents (predictions for the non-neighbor enemies will be masked out).
Corollary 4. Given two allies' policies 11 and I1 with KL(II" (11)) s V, then:
( maxo (*(T(111)1))))))) s o(vi)
Since the allies' joint policy II will be changing during our policy learning process, the above result implies that the imitating policy will be stable if 11 becomes stable, and if 11 is converging to target policy 11 then the imitator's policy also converges to the one that is trained with the target
ally policy with a rate of / [ I That is, if the actual policy within a O(c) neighborhood
of the target policy (i.e., KL(III < E) then the expected return of the imitating policy is within (vV)) neighborhood of the desired "expected return" given by the target policy.
IMAX-PPO: Imitation-enhanced Multi-Agent EXtended PPO Algorithm
We present our MARL algorithm for the competive game setting. We first focus on a practical implementation of an II. algorithm taking into account local observations. We then show how to integrate this into our MARL algorithm. We call our algorithm as IMAX-PPO, standing for Imitation-enhancee Multi-Agent EXtended PPO algorithm.
5.1 Imitation Learning with Local Observations
In previous section, we present our new IL algorithm (which is an adaptation of IQ-Learn) to learn an expert policy [i' up at state S' when the current global state S and the allies' joint action is A From the allies' perspective, to run this IL algorithm, it requires the allies to have access to the global state S. However, each ally agent can only observe local states of its neighboring enemies (such as their locations, speeds, etc.). Therefore, goal is to predict next states of enemies in the ncighborhood (wf"), where local information w = (of of with of is an observation vector of agent containing the local states of the agent itself and of all the agents in the neighborhood that observable by agent i. In particular, given that sf is the local state of an ally agent in and sy is of an enemy agent je and NN (i) is the neighborhood of i, we have:
(S' Aa) that behaves similarly to the probabilities of ending
adapt our IL algorithm in accordance with such local information. The
each allied agent i, denoted by
of } U (8f EN)) JA (s), J.EN(())NT)
To apply II algorithm to this local observation setting, build common policy network îT. and Q network Ø for all the agents where 2h and the the network parameters. The IL objective

---

function can be reformulated according to local observations of the allies as follows:
JHH (oo) ) >iiEve (5 mrrop
(555)) next
w
(6)
iEvo
-B " nme (w next
- (1 - --7) B 2 P (w
where "f" = (of, aof -.,-, ueq (og. at) (a is the action taken by agent i at observation of resulting next observation of of Se.aos { :Jk E N is the next states of enemies in the current neighhorhood of the agent i. In addition, the value functions are re-formulated as follows:
Vit (wff) (sgaan 1 (St wff) In (St [w]))]
In the end, we can update īi; and 2i00 by the following actor-critic rule: for fixed Digg" we update do to maximize y Diog), and for a fixed [iç we apply soft actor-critic (SAC) to update va.
5.2 IMAX-PPO Algorithm
We now combine the local observation of of each allied agent i with the next-state prediction St St. of its neighboring enemies (obtained by our IL algorithm) create an augmented input. This augmented input is used to improve the policy learning of the allied agent i. That is, we aim to optimize the allies' policy I1 (af S ny iEWa) that optimizes the long-term expected joint reward:
maxπi Baag of S.10))1111
EIEN' (o .q)
where of is an observation vector of agent i, S. S.c.nest is the information derived from the imitator for agent i, of E A is local action of agent i. To facilitate the training and integration of the imitation learning policy into the MARL algorithm, for every ally agent i, we gather game trajectories following the structure (o;, of. S, n. These gathered observations are then stored in replay buffer to train the imitation policy i (S 10f", (of)
In the IMAX-PPO framework, at each game state S, considering the current actor policy II and the imitating policy Il' for each agent Ma we draw sample for the allied agents' joint action A IIo Corresponding local observation of and action a. of each agent i are then fed as inputs into the imitation policy to predict the subsequent state S. ÎT. (-lof, 7,). Once the predicted local states {5t E Na) are available, is used as input to the actor policy IIş in order to generate new actions for the allied agents. In simpler terms, we select a next local action a ~II (-Joß, S. .. Beside the allies' policy network, we also use a centralized value network Vi (S) and update it together with the policy network in actor-critic manner, similarly to MAPPO. The actor network is trained by optimizing the following objective:
(0) = EiENa Bog Sr min (0) (0), - 1- +c)A)]
where r0( IE [of s my me (a lop spmin and À is the advantage function, calculated by Generalized Advan- tage Estimation (GAE). The Critic network is trained by optimizing
Φh Es
max [1V (S) R(S)) V dediip
(S) [())){]]]
where R((S) = A+ Via Vi (S) and 00,0% 1,, (S) =s)==cipp((>> clip(VV. (S), Va Ve... (S) - Vi (S) + +). We
provide the key stages in Algorithm] [ Additionally, Fig. serves as an illustration of our IMAX-PPO.
6 Experiments
We evaluate the performance of our IMAX-PPO algorithm (Algo. 1 in comparison with some standard and SOTA multi-agent RL algorithms: IPPO, MAPPO, QMIX and QPLEX. In addition, to examine the impact of our multi-agent II. model on the performance of IMAX-PPO, we include two versions of IMAX-PPO where opponent's 's next states are predicted by (i an adaption of the GAIL
7

---

Algorithm IMAX-PPO Algorithm
Input: Initial allies' policy network TTg initial allies' value network Vg initial imitator's policy network 0i imitial imitator's Q network Tep learning rates Ki NQr kV.
Output: Trained allies' policy IIg
1: for = 0, 1........ do
2: # +Updating imitator:
3: #q.1-1 &Q niQ V vi (1ha)) Train Q function using the objective in 6
4: #7,1+1 = ,n, Ki V t,, g. ar (S, # Update policy IT.. (for continuous domains)
5: # UUdating allies policy:
6: 8141 = 8₁ + ドー V (
# Update allies' actor by maximizing (0)
7: 0g.e+1 Out xiy Ve, Ta (0.)
# Update allies' critic by minimizing 4h (0,,)
8: end for
9: return policy solution for allied agents
Table I: Win-rates (percentage).
Sup MAPPO
IMAX-PPO
Tasks
Scenarios
MAPPO
IPPO
QMIX
QPLEX
GAIL
InQ
5_vs_5
58.0
54.6
70.2
53.3
71.8
68.1
78.7
10_ vivss_00 10
58.3
58.0
69.0
53.7
67.3
59.6
79.8
SMAC Protoss
10_vs_11
18.2
20.3
42.5
22.8
36.7
21.3
48.7
20_vs_ 20_
38.1
44.5
69.7
27.2
71.1
76.3
80.6
20 40_vs_20 23
5.1
4.1
16.5
4.8
21.9
11.8
24.2
5_vs__
52.0
56.2
58.4
70.0
55.8
53.3
69.9
10_vs_10
58.1
57.3
65.8
66.
54.1
58.4
72.2
SMAC
10_vs_11
31.0
39.4
41.4
26.9
53.9
Terran
20_vs_20
52.8
49.6
57.6
23.9
38.6
35,9
65.4
20 vs. 23
11.2
10.0
10.0
7.0
11.2
4.7
17.7
5_ys_5
41.0
37.2
37.2
47.8
52.5
48.6
55.0
10 vs 10
39.1
49.4
40.8
41.6
57.4
50.6
57.6
SMAC
10 11
31.2
26.0
28.0
31.1
38.1
34.8
41.5
Zerg
20 vs.
31.2
26.7
43.3
v0_vs_23
15.8
8.3
10.1
6.7
13.6
8.2
21.3
easy
49.3
54.5
61.8
Gold Miner
medium
40.6
39.5
47.3
50.4
39.4
39.3
55.0
hard
31.2
31.2
41.7
43.5
31.3
29.7
49.8
3_ys_1
88.0
82,7
8.1
90.2
96.1
96.4
98.1
GRF
casy
87.8
84.1
16.0
94.9
89.7
64.1
95.0
hard
77.4
70.9
3.2
95.1
10.7
15.2
97.3
algorithm 12). denoted as IMAX-PPO (GAIL) and (ii) the IQ-learn adaption (i.e. Algorithm [ denoted as IMAX-PPO (InQ). Moreover, to compare out IL-based approach with supervise learning. include approach based MAPPO where the opponent's 's next states learned and predicted by standard supervising learning. We denote this approach as Sup-MAPPO.
The details of Sup-MAPPO and IMAX-PPO (GAIL) are provided in the appendix. We run extensive experiments three multi-agent competitive environments: SMACv2, Google Research Football (GRF), and Miner. Each reported value is computed based on 32 different rounds of game playing (each corresponds to different random seed).
SMACv2. SMACv2 4 is an advanced variant of SMAC, driven by the aim to present a more challenging setting the assessment of cooperative MARL algorithms. In SMACv2, scenarios are procedurally generated, which require agents to generalize to previously unseen settings (from the same distribution) during evaluation. This benchmark consists of 15 sub-tasks where the number of agents varies from 5 to 20. The agents can play with opponents of different difficulty levels. In

---

MAPPO
QMTX
SUpMAPPO
THAX-PPO (GAIL)
1MAX-PPO (InQ)
0.54
0.6-
0.4
SH
protoss___ __vs_
protoss 10 vs_ 10
peoloss 10_ys_11 yx_
protoss _0_vs_22 vs_20
protoss 20, vs
0.6-
0.-
0.2
3N termnn 10_w vs_
10M
2008
5K teran vs_20
20M
teraa 5_ys_5
terran 11
terrai, 20__e_ vs_
0.2
SH
100
5M
10M
S31
3010
10M
SM
30M
zerg_5 5
zerg. _0 10_
zerg vs_
zerg 20____0_v
zerg
Figure 2: Win-rate curves on SMACv2 environment.
comparison to SMACv1 25 SMACv2 stands apart by permitting randomized team compositions, varied starting positions, and an emphasis on augmenting diversity.
Figure 2 shows the performance of the five algorithms during the training process across 15 sub-tasks. The x-axis is the number of training steps and the y-axis is the winning rates averaged over 32 rounds of evaluations. In Figure 2 IMAX-PPO (InQ) consistently and significantly outperforms other baselines. Our algorithm frequently attains quicker convergence; it achieves high win rates at earlier training stages. This could be attributed to the incorporation of our IL component, which facilitates faster comprehension of opponents throughout the game. In partieular, the IMAX-PPO (InQ) outperforms the two other variants IMAX-PPO (GAIL) and Sup-MAPPO, indicating the advantage of our inverse- approach over other IL (i.e. GAIL)) and traditional supervising learning methods. Details of win rates at the end of training are shown in Table [
Google Research Football (GRF). This is a challenge on Kaggle competitions made by Google Research team [15]. We focus on three main sub-tasks, sorted based on increasing difficulty levels: (i) academy- --v-mmy--------ii-e -with----i---eeerr three allies try to score against a goal-keeping opponent; (ii) academy- counterattack- easy: four allies versus a counter-attack opponent and goal-keeping opponent; and four allies versus two counter-attack opponents and goalkeeper.
(iii) conudemny-cunnterattuck--a
By default, the representations of all agents' observations RGB pixels in GRE, so we pre-process this information by distilling some important features such as object positions, object directions, distances between objects, etc. The final win rates are in Table [ which shows that IMAX-PPO (InQ) achieves nearly 100% win-rates, and significantly outperforms other baselines
Gold Miner [7]. This is another competitive multi-agen game for evaluating our methods, orig- inating from MARL competition. Multiple miners navigate in a 2D terrain containing obstacles and repositories of gold. Players get points according to the volume of gold they successfully extract. This game challenging to win as the agents have to learn playing against extremely well-designed heuristic-based enemies. In this game, the ally agents win if the allied team's 's average mined gold is higher than that of the enemy team.
We customized the original environment into three sub-tasks (between two allies against two enemies) 2_aiy___s_2__0000 of three difficulty levels: (i) Easy (easy_2_nn vs_s_2 2)= The enemies' greedy strategy is to find the shortest way to the golds; (ii) Medium (medium_ 2_vs_2): One enemy is greedy, and the other follows the algorithm of the sacond-ranking team in the competition; and (iii) Hard (hard 2 vs 2): The enemies are the first- and second-rane rackkng teams in the competition.
'The win rate curves during the training for GRF and Gold Miner environments in the appendix.

---

For this environment, the win rates are in Table Again, IMAX-PPO (InQ) obtained superior win rates across all three tasks. Especially, in the hard-level task, our algorithm manages win more than 50% of the time against the first and second-ranking teams in the competition.
7 Conclusion
We introduced a novel principled framework for enhancing agent training in multi-agent environments through II.. Our new IL. model, adapted from IQ-learn, can predict opponents' policy using only local state observations. By integrating this model into multi-agent PPO algorithm, our IMAX-PPO algorithm consistently outperforms previous SOTA algorithms such as QMIX and MAPPO. This improvement is observed across various challenging multi-agent tasks, including SMACv2 and GRF. possible limitation of our work is that relies on the assumption that the enemies do not update their policies during training (even though this is a standard setting in cooperative multi-agent reinforcement learning (55). A future direction would be to delve into these aspects to develop efficient MARL algorithms for cooperative-compettiiv multi-agent RI..
Acknowledgments
This research/project is supported by the National Research Foundation Singapore and DSO National Laboratories under the Al Singapore Programme (AISG Award No: AISG2-RP-2220-0077,
References
[1] Abbeel, P. and Ng, A. Y. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 1, 2004.
[2] Claus, C. and Boutilier, C. The dynamics of reinforcement learning in cooperative multiagent systems. AAAI/IAAI, 1998(746-752);2, 1998.
[3] de Witt, C. Gupta, T., Makoviichuk, D., Makoviychuk, V., Torr, P. H., Sun, M., and Whiteson, independent learning all you need the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533 2020.
[4] Ellis, B., Moalla, S., Samvelyan, M., Sun, M., Mahajan, A., Foerster, J. N., and Whiteson, Smacv2: An improved benchmark for cooperative multi-agen reinforcement learning, arXiv preprint arXiv:2212.07440 2022.
[5] Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[6] Foerster, N., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., and Mordatch, I. Learning with opponent-learning awareness. arXiv preprint arXiv: 2017 2017.
[7] FPT. Fpt reinforcement learning competition. r e1comp2020 2020.
https://github.cm com/
htt.g/thhwb.omm///hhggggngn xphongvn/
[8] Fu, J., Luo, K., and Levine, S. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.1 1248, 2017.
[9] Garg, D... Chakraborty, S., Cundy, C., Song, J., and Ermon, S. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34:4028 4039, 2021.
[10] Grover, A., Al-Shedivat, M., Gupta, J., Burda, Y., and Edwards, H. Learning policy representa- tions in multiagent systems. In International conference on machine learning, pp. 1802-1811. PMLR, 2018.
[11] He, H., Boyd- J., Kwok, K., and Daumé III, H. Opponent modeling in deep reinforcement learning. In International conference machine learning, pp. 1804-1813. PMLR, 2016.
10

---

[12] Ho, J. and Ermon, S. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[13] Hong, Z.-W., Su, S.-Y., Shann, T.-Y., Chang, Y.-H,, and Lee, C.-Y. A deep policy inference q-network for multi-agent systems. arXiv preprint arXiv:1712.0780 2017.
[14] Kim, D. K., Liu, M., Riemer, M. D., Sun, C., Abdulhai, M.. Habibi, G., Lopez-Cot, S., Tesauro, G., and How, J. policy gradient algorithm for learning to learn in multiagent reinforcement learning. In International Conference on Machine Learning, pp. 5541-5550. PMLR, 2021.
[15] Kurach, K., Raichuk, A., Stanczyk, P., Zajac, M., Bachem, O., Espeholt, L.,, Riquelme, C.. Vincent, D., Michalski, M., Bousquet, O., et al. Google research football: A novel reinforce- ment learning environment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 4501-4510, 2020.
[16] Li, Z., Xu, T., Qin, Z., Yu, Y., and Luo, Z.-Q. Imitation learning from imperfection: Theoretical justifications and algorithms. Advances in Neural Information Processing Systems, 36, 2024.
[17] Littman, M. L. Markov games as framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157-163. Elsevier, 1994.
[18] Lowe, R., Wu, I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, I. Multi-agent actor-critic for mixed cooperative.compttiii environments. Advances in neural information processing systems, 30, 2017.
[19] Ma, X., Isele, D., Gupta, J. K., Fujimura, K., and Kochenderfer, J. Recursive reasoning graph for multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7664-7671, 2022.
[20] Ng, A. Y., Russell, S., et al. Algorithms for inverse reinforcement learning. Ieml, volume 1, pp. 2, 2000.
[21] Papoudakis, G. and Albrecht, S. V. Variational autoencoders for opponent modeling in multi- agent systems. arXiv preprint arXiv:2001. 10829, 2020.
[22] Rabinowitz, N., Perbet, F., Song, F., Zhang, C., Eslami, S. A., and Botvinick, M. Machine theory of mind. In International conference on machine learning, pp. 4218-4225 PMLR, 2018.
[23] Rashid, T., Samvelyan, M., De Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, S. Mono- tonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21(1):7234-7284 2020.
[24] Reddy, S., Dragan, A. D., and Levine, S. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint arXiv: 1/008 2019.
[25] Samvelyan, M., Rashid, T., De Witt, C. Farquhar, G., Nardelli, N., Rudner, G., Hung, C.-M., Torr, P. H., Foerster, J., and Whiteson, S. The starcraft multi-agent challenge. arXiv preprint 10X.::192.000000 2019.
[26) Schulman, J., Wolski, F. Dhariwal, P., Radford, A., and Klimov, Proximal policy optimiza- tion algorithms. arXiv preprint arXiv: 1707.06777. 2017.
[27] Son, K., Kim, D., Kang, W. J., Hostallero, D. E., and Yi, Y. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pp. 5887-5896. PMLR, 2019.
[28] Song, J., Ren, H., Sadigh, D., and Ermon, S. Multi-agent generative adversarial imitation learning, Advances in neural information processing systems, 31, 2018
[29] Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv: 1 052966 2017.
[30] Wang, J., Ren, 7., Liu, T., Yu, Y., and Zhang, C. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint 20f.:2000800022 2020.
2020.

---

[31] Wen, Y. Yang, Y., Luo, R., and Wang, J. Modelling bounded rationality in multi-agent interactions by generalized recursive reasoning. arXiv preprint arXiv:1901.09216, 2019.
[32] Wen, Y. Yang, Y. Luo, R., Wang, J., and Pan, W. Probabilistic recursive reasoning for multi-agent reinforcement learning. arXiv preprint arXiv:1901.09207, 2019..
[33] Willi, T., Letcher, A. H., Treutlein, J., and Foerster, J. Cola: consistent learning with opponent- learning awareness. In International Conference on Machine Learning, pp. PMLR, 2022.
[34] Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Bayen, A., and Wu, Y. The surprising effec- tiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 15:24611-24622 2022.
[35] Yu, X., Jiang, J., Zhang, W., Jiang, H., and Lu, Z. Model-based opponent modeling. Advances in Neural Information Processing Systems, 35:28208-2822 2022.
12

---

Appendices
A Missing Proofs
A.1 Proof of Proposition
Proposition I
For any reward function Re, let Q be the unique fixed point solution to the soft BlI. = =Q", then we have L(II, It5) J(1,, )²). and for any Q', JII, Q3)
Bellman equation oQ
L(II, TT))
Proof. The proof is similar to that given in 9
that if Q" is solution to the soft
Bellman equation, then
Q"(W (W 8 + PBw (VH()))
where
(W) = (V.5)
which implies
RW = - -W (V](")) TQ
which validates L(II, = J[TI, Q*). The second inequality is just a trivial result from the definition of J and L.
A.2 Proposition 5
Our following Proposition 5 shows that the function J(T,, Q*) can be reformulated in a more compact form that is convenient for training.
Proposition 5. The function J(-) can be written as follows:
(11, Q*) = B(W.S (W, S') -1Bw (VH(
- (1 Ewo n. (W))
(8)
where So is an initial state and po is the initial state distribution in the original MDP.
Proof. From the definition of J (II, Q*) we write function J as
J(II, Q*) = Epr... (Tt (W,
(W, S - E Ep..u (W;, (W, S')] + E [In [I('S
S -ESSAA S)) (W, - (W"))
(W [V((")
=
+ Epo.n [In S" W))
We consider the second and third terms of 9 and write
E(s,1g, (00 S', -ll Vii W } - Epnin n [In [T)
> (Q+(((4,, -r [EW.IIAA Σ Evr((wan))
(1 - Bs
7 In IIII(S(+ (w:)
Esr. S1111ll1
(Q WT) -y ES Mil Σ (S wal)
(1 (1-7) ,
Σ
-7 ES.4I -I
A S1 Σ
> In I
(1 --)),, Wo (11 Wo) In [[)
= (1 EW,,, Vi (W)).
(10)
Thus,
J(II, Q?) ((S.A2.5')~ (W,S') >EE Vi(WW)) (1 po Vi(
(11)
as desired.
13

---

A.3 Proof of Proposition 2
Proposition 2 The problem maxq ming (II, Q?) is equivalent the maximization
max Q*)
where
exp(Q
(S" Es (ap(Q"(W)S"))
Moreover, the function J(II Q")) is concave in Qr.
Proof. To simplify the proof, we first consider the following simpler optimization problem. Let P1, .... PN [0, 1] and Ti, 2N are N real numbers. Consider the maximization problem
N
max pE(0.1]
Σ
J.P:
ps InIp
(P1)
subject
Σ
P
P
Using the KKT condition, p" is an optimal solution the above convex problem, p" needs to satisfy the following conditions: there exists À such that
x lm
0, Vi
P
which implies
Pj exp(x; - 1)
Combine this with the condition E. we should have exp(-1- A) Σ, exp(x.) and exp(x::
P. Σ, exp(1z))
(*). In addition, when p = p". the objective function of (P1) can be written as
f(x) Σ app pt In(p))
Σ (x: In(p)))
Σ pi + In
Σ Eexp(as)
In Σ exp(x;)
We then that f(x has log-sum-exp form, thus it is convex in (**).
We now return to the minimax problem maxq- minn (II,, Q+). For any fixed Q can be seen that the problem minn JIII, Q+) is equivalent to
1B [V]] + (VF
max
We first consider the problem maxn Vi (W). recalling that (W) can be written as
Σ IT W) x (Q", (og(
Then it can be seen that each term Σ or ((S x (Q"(W)") log(I[(") has the form of P1). thus from (**) we see that (S) is maximized at
cxp(Q"(W,S")
119(S'))) ESt exp (WW,
Moreover, according the result (**) proved above, when II I1, (i( is convex in Q Consequently, the loss function of the IQ-learn
(II, (*) ={{,. (') 'Q" (W, (WWS')-a - Vi(("") + (1-7) po (W0))
concave in Q", which completes the proof.
14

---

A.4 Corollary 6
Corollary 6. Ifll 11 (defined in Pruposition 2 then we can write (f((S) as follows:
Vi(W) = VV(W) = In exp (Q (Q,
Note that Corollary is just a direct result from the proof Proposition 2
A.5 Proof of Proposition
Proposition Given two allies' joint policies IT and I1x such that (I( for
any state S E ES the following inequality holds:
((II" - - S (aQ (ln|s]) 22
where Q = max Q" (W, is an upper bound of Q", a (1-7)² + - + (3 - 7), and (W,S")
+ 7.
Proof. We first write the loss function as
(TTIQQ) = E(s.A M') { { (W, B VG(W")] - -1 (W))
We first write the difference between two loss values as
4 (Tī"QQ9) = [S s (Q, 8') p
-7 (p VS(W") Bw (2(W"))
(Bwo-Po,1- (V5(W)) (2()))
(12)
We first consider the first term of 12.. Let us denote the following function
r(S" E (s. f
7 (Q' Wi)) (5%. Wo) = (W.S')
and write the first term of 12
(S), Wi)) ] (8).)))(m... Σ 7d (S) (w))
(1 7) E ((S
1=0
= =(1- 11-7)
E B(s,W.)-(TT.1)))))
r' (Q) Wa))
= =(1 (1-00 7) ((S(W))((TTTT))
r' (Q'(S", W.)) (PS [(S)
((S))W) - B(s (r(S(Wo))])
+ (1 (4) (BS6,.W))<(P)))))))
= (1 7) ((STW.)-(TT.I))
Σ
xa (Q'(S) W.)) + B(ss...)MIT.TT Σ
-
+ (1 -(1-7) (E(s6.Wo)<<ea nnn3
((())W)) - (allWo)BB(s,,aaa (MM. [r(ss/w6)]])
= (1 E(s
(Q*(S), w,) + + W -((s))(((())))))))
+ (1 (1-7) (E(s).Mo)-(PM.TT)) ((S))W) - (
(13)
We first see that
Q
T(S = E ( (s).Wi fi
*' (Q" W:)) (S) Wo) = (W.S') s
1-7
15

---

Thus, we can bound the second term of 13 as
(1 7) (E (Bs)) [TS.]
1B (sg (o)p,. (T((
< (1 - S.W
[ (15)) - i1
{ (21n2x1,((T((/))))(1(1111 8
$ <Qmax
(14)
We also that
r(S Q W:) + t (. Mi Jw.. [T(5+1|W6+1)
Thus, we further can bound the first term
(33 as
(.s1) 11 w. (00 W:) + (W -T(S)))(()))))))
$ > (-1s) - -.
22 [ (-15))
7HH max
8
where:
= max (T())WW)) (W,S')
max (W,8')
E (S) Nis)
Q
1-7
where Q is an upper bound of Q*(- Therefore, we can bound 13 as
7t (Q) WT) + ( -
+ (1 (FIS),W)~(PPMII))) ((S E (s (P. II.)
[T(S](6)]]]
(15)
Q max {
7
22 (
(1 71 +
(16)
For the second term of 112 we can bound it as
? (B(W.S".W')pprr (VV((W)) -
Σ Σ
= (1 71
EwinIr,1-
VV((W))
E B Tio
= - -11-))E(.wIr..e
+ (1 71) (WW))-(PP, -
(17)
Σn' (vg(wa)) Wo-w) w It then can be seen that
where U(W) = E E (w )
U(W)== VV (W;)
7B (w. fi (W (W(
(18)
16

---

and
U(W) 1 1-7 max V@(W)
< max In Σ ?
(Ee))(((WS))))
gr
$ (In (In|S] +Q)
Thus, given any W,, we have
E 1191 VV(W)) -
-B w. (U(W++1)] - - (W(W-
s I (+1.) (-15el)
s maxx V 22
(19)
where K
1 (In |S| Q) maxw (U(W)) So, (77) can be bounded
(1 BW, (TT,TT IT.
(V(
(I |S| Q)
(1 (1-)) max
(12)
(20)
Moreover,
(1 (1-7) 7) (E(ri)
(U(W)) (U(W)) E (ion(po,fo))) _o, (W)
(11)))) (19))1))))))) K
$ (1 ?) max
S,Aa
< 2(1 | - Āम (-1S:)ll
$ 2(m S + 2)
max 3
(21)
Combine 17 and
20 and 21 we bound the second term of [12 as
E(w,s' a [V5((")] E(w,s B( [1Vo (W')]
+2 (In [|||s| + Q) max -(1- (a- - 7)
V (T
(22)
The last term of
12 can be bounded simply as
Ewg~Po (V (W)
2
< (1 7) max VV (W))
max
W
S
max S (1 (In |S]
+ Q)
2n (II
(23)
5
Combine 16
22
and 23
we get
( 10)) क @Q)
(1 (1->)² + ? |S| Q)
7Q
+ (3 - (1)
+ 2)
max (1.)))(())))))))(((((
(1 (1-7)
(1 - 7) + 1- + (3 7) Q+ (1--)) +
72
In (is) xtui (2
which completes the proof.
17

---

B Continuous Action Space
We extend the results in Section 4.2 provide bounds for the of continuous action space. In this scenario, an actor-critic method is used since the policy IT cannot be computed exactly as in Prop. 2 In this case, we can use an explicit policy II to approximate 11 instead. We then iteratively update Q and II alternatively using the loss function in Prop. [ In particular, for fixed Q", soft actor update max [g-- (W", S*) In [[[[WW will bring the imitation policy II closer I1
Let (TT*) IT) be the objective of II. in Eq. (8) written as function of the allies' joint policy ITo. The following proposition establishes bound for the variation of [ |Q", II) +(11 [Q+; II) as
function of kI.( [|TT)) for any pair of allies' joint policies (ITo, IT²).
Proposition 7 (Continuous state space S). Given two allies' joint policies IT and iT such that for every state S E S, (L() S the following inequality holds:
((II" III 4 (IT" Q", m)) (aQ (aQ 3H) v2ln 2c
where H= 17=inff iinfw Es In W). a lower bound of the entropy of the actor policy IT.
Proof. We can follow the same arguments as in the proof of Proposition 3 above, with the only difference being when we bound V/ Here, VV) replaced by Vii and can be bounded
as
Vi(W) = Σ (P 9'
log (((((()))))))
< inf (Q inf II
<Q- 11
(24)
We now see that the entropy term Es II(S"WW) (og()())WW)) is minimized when ((((W) = 1/1S), where H = infw ( (W)), and H(W) is the entropy of II(-WW), i.e. Es (I (S)).... Therefore, the overall bound can be established by replacing the term In S in the discrete case by - H. We then obtain
(1I III QQ 1n/ (al BH) max { V (1 (115))
Prop. 3 & 7 allow us to establish an upper bound for the IL when the allies' joint policy changes.
Corollary 8. Given two allies' policies I1 and πo with KL(II* (11))(()))))))) S e, VS then: (continuous)
maxe minn (TT (Q, maxq minn (TT)Q. m) s O(ve)
Proof. To simplify the notation, we first prove the following result:
Lemma 9. Given > and two functions (xx, g(x) such that (f(x) g(x) S for any X (X is the feasible set of x). The following hold true
max f(x) - max ((x)) Se
min f(x) min g(x) Se
To prove the above lemma, let o x be optimal solutions max, fxx and ming g(z), respectively. We consider 2 cases
If max, (xx 2 minz (xx), then we have
max f(x) min g(x)) = f(x²) - g(x9) < f(x)) -(xa))-))'))))) (g)
where (a) is because [f(z) -x(x)) < for any X
18

---

iii If max, f(x)= I ming g(z), then we have
max (a) min g(x)] 9(x9) - f(x()=)+))))))))()) S g(x9) - f(x0)
Thus max f(x) - max, g(x) 2 c. The inequality mmi, f(x) ming g(x)) 1 c can be validated in the same way.
We now get back to the main proof. Since ((II")"") - ((ī (4) - O(vc) (Proposition 3 Lemma above implies that
maxo ( - m - O(ve)
Moreover, from Proposition 7 applying Lemma Sasmma@twer twice, we have the following
minn {111 (0",)1"11))))))")" minn { mmn{ , s so(v)
and
maxq minn (0,, maxqr minn ( (((E)
which completes the proof.
C
Other Experimental Settings and Results
C.1 MAPPO with Supervised Learning to Predict Opponent's Next States
In this approach, instead of using IQ-learn to predict opponent's next states, we employ a simple supervised learning approach. Specifically, create neural network of parameters o: (o att), taking inputs an observation and an action of allied agent i, and predict the next states of enemy agents in the observable area agent The loss function can be defined as minimize the MSE between the actual next enemy states and predicted ones, as follows:
min Jsup(()
Σ
Σ Sg. MM(O,, (0))
2
(Staaes (op a Vi)-butter
C.2 IMAX-PPO with GAIL
To predict enemies' next states, the GAIL algorithm by 122 can be adapted in similar way as our main IMAX-PPO algorithm. Building upon our new MDP framework, which is necessary handle the unobservable-actto discriminator D(W, SS)) that distinguishes between generated transitions (W, S') and those collected from interacting with the actual enemy agents. The GAIL loss function can be formulated as
issue, the aim is also learn a policy WW) by creating
min max (Jaan D) Ess ())ra))= (log D(S', W)) Bp... [log(1 D(W,S') (H(II)) II where (1I) is the entropy of II. In practice with partial observations, we can model D and 1I as neutral networks D., (Se.eett off af) and [S , where w, 0 are trainable parameters. The objective can be formulated as follows:
min max (EaAI.(0, (ss off aaf [log D ,
En of ofuufeee [log(1 D(S off af()) AH
C.3 Experimental Settings
For a fair comparison between our proposed algorithm and existing methods, we use the same model architecture and hyperparameters shown in Tables 2 and respectively. We also use
19

---

SMACv2
GRF
Miner
Runner
Parallel
Workers
8
32
Total steps
10e6
0.5e6
2e6
Mini steps
1024
Evaluation
32
Mini batch
1
Device
Cuda
Mini epochs
5
Learning rate
5e-4
2c-4
Epsilon
1e-5
Weight decay
0.0
Clip range
0.2
Entropy coefficient
0.01
0.0
0.01
Value coefficient
0.5
Weight gain
0.01
Weight initializer
Orthogonal
Max gradient norm
10.0
Gamma (GAE)
0.99
Lambda (GAE)
0.95
Table 2: Hyperparameters
Value Normalization method and Recurrent Neural Network mechanism proposed by MAPPO 34 which used as an improvement method to speed up the training process. All sub-tasks (SMACv2, GRF, Miner) are trained concurrently in GPU-accelerated HPC (High Performance Computing). Therefore, running times reported might not be accurate. In average, each sub-task requires 4-7 days of training depending
its difficulty. As result, due the limitation of our computing resources, we do not test our method's 's performance with other settings such as disabling Value Normalization, using MLP instead of Recurrent Neural Network, tuning learning rate, tuning clip range, etc.
com/ Regarding GRF tasks, there are 11 academy scenarios (depicted at httoc///ccccroo..oooo.. .m).. gpogle-research/ footballl//ooooooooo
Although testing all these sub-tasks is interesting, but we only evaluate on top three importance scenarios: academy_3 33 vs with keeper, academy__cunnteratinn cww.teraaaaccckkcccc easy, and ceademm_couuterattaace h_hard. All are the hardest sub-tasks with the highest number of agents except the almost full football scenarios academy_ satnimysssigggeeee single_goal_wee sooua__verus_lare sazzy. We report the win-rate curves of our IMAX for the three tasks in Figure|4] We do not show the performance curves of the other baselines methods (QMIX. QPLEX, and MAPPO) as they are not available in their papers, noting that the final win-rates of all the methods considered are already reported in the main paper. Finally, Figure shows the win-rate curves on the Gold Miner tasks.
x( (G-1)
(G)
x (o)a)
V(S,
V(S))
V(S,.))
MLP
MLF
MLP
A
GRU
GRU
GRU
Ag-3
Aprilo
App
GRU
GRU
GRU
MI.P
MLP
MLP
MLP
MLP
MLP
21-1
S,
S
(b) Critic architecture
(a) Actor architecture
Figure 3: PPO model architecture
20

---

MAPPO
SupMAPPO
TMAX-PPO (GATL)
TMAX-PPO (TnQ)
400k
3 3_vs__
counter _aseeee_aayy__
counter hard
Figure 4: Win-rate curves on GRF environment.
MAPPO
QALEX
SUpMAPPO
IHAX-PPO (GAIL)
IMAX-PPO (InQ)
0.2-
easy easy_2_M medium_2_vs_2
hard_2_vs___
Figure 5: Win-rate curves on Gold Miner environment.
D Ablation Study
In this section, we present an ablation study to evaluate the effectiveness of our IMAX framework when integrated with traditional multi-agent reinforcement learning algorithms. Specifically, we introduce a variant called QMIX-IMAS which incorporates our IMAX framework into the conven- tional QMIX algorithm. We conducted experiments using the SMACv2 benchmark, focusing on three different factions: Protoss, Terran, and Zerg. For each faction, we tested two team configurations: 5_vs_5 and 10_vs_10.
The performance of QMIX-IMAX was compared against the original, non-IMAX versions of popular algorithms such as MAPPO and QMIX. The results of these experiments are summarized in Table which illustrates the winning rates achieved by each algorithm across the different scenarios. Additionally, Figure displays the learning curves, providing a visual representation of the performance dynamics over the training period.
Our findings indicate that the QMIX-IMAX variant consistently outperforms the traditional QMIX algorithm across all tested scenarios. However, does not achieve the same level of performance as MAPPO-IMAX variant. This suggests that while the IMAX framework enhances the capabilities of QMIX, the integration with MAPPO yields superior results. These observations underscore the potential of the IMAX framework to improve bascline algorithms, offering promising direction for future research in multi-agent reinforcement learning.
MAPPO-IMAX (ours)
QMIX-IMAX (ours)
Task
MАPPO
QMIX
5_vs__
21.6629.05
33.8424.90
48.22+7.87
40.75+2.99
protoss
10_vs_10
9.11+2.04
18.6445.97
46.03+5.93
18.65+5.05
5_ 5_vs.5 10. vs 10
23.9524.89
41.4825.64
50.5624.84
48.4414.11
terran
13.5342.83
34.1243.22
45.1642.87
38.7545.99
5 vs. 5
13.7426.10
30.3744.24 13.1347.76
36.3943.96 41.53+6.53
29.50+2.03
zerg
10. vs. 10
6.06+1.76
25.5243.81
Table 3: Wining rate (percentage) of two baseline methods MAPPO and QMIX and our improvement algorithm on SMACv2 tasks.
21

---

_____s
protoss_ 10_ vs_
QeIX
0.6-
0.6-
MAPPO-INAX (ouns)
win rate
0.2
0.5M
0.5M
steps
steps
terran_5 _______
terran, terran_10_vv___ 10__v_11000
0.6-
0.6-
win rate
0.4-
0.2
0.2
0-
0.9M
1M
0.5M
1N
steps
zerg_5_vv _____
zerg__10_vv vs_10
0.3
win rate
0.2
0.5M
steps
steps
Figure 6: Learning curves with different methods SMACv2.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Our abstract includes main claims reflecting main contributions and finding.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations of the work in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

All the proofs of the theorems and propositions stated in the main paper are provided in the appendix with clear references.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide details on the environments and hyper-parameter settings in the appendix. We also uploaded our source code for re-productivity purposes.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The data we used, along with our source code, has been uploaded with the main puper. We have also provided sufficient instructions for their use.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/
pub / gu des / CodeeSuom ss i onoool cy) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
wttp://ccpppcclcc.....ce.coccppa http:/iddeeeoeeuueeieeeeee Submi hisbbmisssa. as ciiooooolll ooyyl for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
reproducible, they
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the details are provided in the main paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Following standard practices, we reported win-rate scores, computed via several independent evaluations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provides all the details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper develops a general algorithm for multi agent RI, which we have tested only in simulated environments. As such, we do not foresee any direct societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
We have provided clear citations the source code and data used in the paper.
Justification:
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, (papersw haperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our source code is submitted alongside the paper, accompanied by sufficient instructions. We will share the code publicly for re-producibility or benchmarking purposes.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "aqbQqDFjZM", "reviewer": "Reviewer_bLoo", "summary": "The paper studies cooperative-competive MARL. It utilizes imitation learning to comprehend and anticipate the next actions of the opponent agents (enemies), aiming to mitigate uncertainties of the controlled agents (allies) with respect to the game dynamics.", "questions": "The questions have been intergrated into the weaknesses section.", "limitations": "Limitations of the work have been discussed in the weaknesses section. The authors also highlight some limitations in the conclusion.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- The paper studies a very interesting problem, central to the MARL community.\n- The paper proposes a novel and interesting method, combining imitation learning techniques and opponent modelling for enhancing the agents' individual policies.\n- The paper provides many experiments on three benchmarks (smacv2, grf, gold miner)\n- The proposed method significantly improves performance in many tasks over the most important baselines: the backbone MAPPO and the supervised baseline, SUP-MAPPO.\n- The authors provide interesting theoretical analysis of the proposed framework.", "weaknesses": "- Related work needs improvement. In opponent modelling, the authors claim that: \"All the aforementioned related works require having access to opponent’ observations and actions during training and/or execution\". This is not the case in most opponent/agent modelling works (e.g., see Papoudakis et al.), as they model the other allies, not the enemies (which is allowed under the CTDE schema). I believe that the proposed method belongs to the category of opponent/agent modelling in MARL. Moreover, some important references are missing, see [1], [2], [3].\n- The presentation needs improvement, heavy notation in many parts. The authors should remind more often what some quantities represent. Furthermore, more intuition of the proposed framework and the method is needed in sections 4 and 5 and the related work. In a nutshell, why can IL solve the agent modelling problem, why is it important and how? Also, how does the goal of the method (i.e., predicting the next states of the enemies) is connected to the IL objective of section 4.1? \n- Ablation study is missing. Also, it would be interesting if the authors provide experiments of the proposed method on top of other MARL algorithms as well. \n- The framework can be impractical as it is now, as it may need a lot of hardcoding to be implemented to any environment, since it leverages enemy state information within the individual (allies) agents' observations (e.g., information regarding the neighborhoods). In other words, one may need to be able to decompose manually the agents' observations into different parts, some of them are related to enemies.\n\n[1] Papoudakis, Georgios, Filippos Christianos, and Stefano Albrecht. \"Agent modelling under partial observability for deep reinforcement learning.\" Advances in Neural Information Processing Systems 34 (2021): 19210-19222.\n\n[2] J. Sun, S. Chen, C. Zhang, Y. Ma, and J. Zhang, “Decision-making with speculative opponent 388 models,” IEEE Transactions on Neural Networks and Learning Systems, 2024.\n\n[3] R. Raileanu, E. Denton, A. Szlam, and R. Fergus, “Modeling others using oneself in multi-agent reinforcement learning,” in International conference on machine learning. PMLR, 2018, pp. 4257–4266.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for carefully reading our paper and providing us with constructive feedback.\n> Related work needs improvement. In opponent modelling, the authors claim that: \"All the aforementioned related works require having access to opponent’ observations and actions during training and/or execution\". This is not the case in most opponent/agent modelling works ... \n\nThank you for your feedback! Yes, our work falls under the category of opponent modeling in MARL. We will revise the section to clearly differentiate between opponent-agent modeling and allied-agent modeling. In our competitive MARL setting in particular, access to opponents’ observations and actions during both training and execution is not allowed, thus making our opponent-prediction significantly more challenging. \n\nWe will incorporate discussions on suggested references. Specifically, Reference [1] relies on having access to the observations and actions of the modeled agents during training to develop accurate representations. In contrast, our setting does not provide such inputs, making our task more challenging. Reference [2] involves training the opponent model in conjunction with the learning agents’ policies to mainly maximize the agents’ expected return, without reasoning about the opponents’ long-term goals and plans. On the other hand, our approach takes a more sophisticated perspective by utilizing information about opponents derived from our agents’ local observations. We employ imitation learning to uncover and understand the long-term, complex policies of the opponents, providing a deeper insight into their behavior. Finally, reference [3] assumes that all agents have full visibility of the environment, while in our scenario, each agent is limited to local observations.\n\n> The presentation needs improvement, heavy notation in many parts. The authors should remind more often what some quantities represent. Furthermore, more intuition of the proposed framework and the method is needed in sections 4 and 5 and the related work. In a nutshell, why can IL solve the agent modeling problem, why is it important and how? Also, how does the goal of the method (i.e., predicting the next states of the enemies) be connected to the IL objective of section 4.1? \n\nWe will add more intuitive explanations and revise the notations as suggested. Our rationale for employing imitation learning (IL) to predict opponents' behavior stems from IL’s proven effectiveness in understanding and replicating the long-term goals and complex policies of agents based on historical trajectory observations. Especially, in our MARL setting, IL has demonstrated its significant advantages in predicting the opponents’ behavior solely based on partial observations about these opponents, resulting in a remarkable performance of our MARL algorithm, IMAX-PPO, substantial outperforming other SOTA algorithm in complex MARL environments such as SMACv2 and Google Research Football.  \n\nThe objective in Section 4.1 is to learn an IL policy $\\Pi(S'|S, A^\\alpha)$ from demonstrations by treating next-state prediction as an IL problem. This policy allows for the prediction of the next state of the enemies given the current state $S$ and ally action taken in the previous round. This information is then used in our IMAX-MAPPO algorithm to support the MARL.\n\nWe hope the above responses are clear. If you have any other questions, we are happy to address and clarify them.\n\n> Ablation study is missing. Also, it would be interesting if the authors provide experiments of the proposed method on top of other MARL algorithms as well.  \n\nIn the current paper, we have included an ablation study comparing the performance of our IMAX-MAPPO algorithm with other IL-based and supervised-learning-based approaches (Sup-MAPPO, GAIL-MAPPO). Additionally, in response to your question about incorporating our IL-based approach with other MARL algorithms, we have provided additional experiments for QMIX-IMAX, as reported in the attached 1-page PDF. The results generally show that our IL-based framework can improve QMIX, but this combination is generally less effective than our main algorithm, IMAX-MAPPO. We will include such results in the paper.\n\n> The framework can be impractical as it is now, as it may need a lot of hardcoding to be implemented to any environment, since it leverages enemy state information within the individual (allies) agents' observations (e.g., information regarding the neighborhoods). In other words, one may need to be able to manually decompose the agents' observations into different parts, some of them related to enemies.\n\nSince we have more components to learn (imitation learning to mimic the opponent's policy), we need an additional step to decompose the agents’ observations into different parts. This ensures that the appropriate information is used for imitation learning or main policy learning. So far, this decomposition step has not required significant effort and can be done with just a few lines of code. However, it would be ideal to have an automated algorithm that can perform the decomposition automatically. We will investigate this in future work. \n\n*We hope the above responses are clear. If you have any other questions, we are happy to address and clarify them.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "psoQ3QbLDt", "reviewer": "Reviewer_4odM", "summary": "This paper presents a new framework of multi-agent reinforcement learning (MARL) by modeling opponents’ behaviors through imitation learning.", "questions": "If I understand correctly, an important feature of the proposed framework is that all ally agents jointly learns a single joint model of the enemies. On the other hand, the SupMAPPO agents learn the enemies’ next states individually. What if the same information sharing is performed for supervised prediction of enemies’ next state?", "limitations": "It is assumed that opponents don’t learn.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The motivation and the method is well described and the performance is tested in extensive experiments with challenging tasks against SOTA methods.", "weaknesses": "It is assumed that each agent performs just based on the present observation o_i, but coordinated behaviors often require memory coding the group strategy or opponents’ game plan.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reading our paper and for the positive  feedback.\n\n> If I understand correctly, an important feature of the proposed framework is that all ally agents jointly learns a single joint model of the enemies. On the other hand, the SupMAPPO agents learn the enemies’ next states individually. What if the same information sharing is performed for supervised prediction of enemies’ next state? \n\nIn response to your question, we would like to clarify that in both of our main algorithms, IMAXPPO and SupMAPPO, we use a common joint model for all the enemies. Although each enemy agent provides different inputs, leading to different outputs, this common joint model helps reduce the size of the learning model and is more efficient than learning individual models for each enemy agent.\n\n>It is assumed that opponents don’t learn.\n\nYes, all the enemy policies are fixed in the environment. This is a standard setting employed in other state-of-the-art MARL algorithms such as MAPPO and QMIX. Allowing enemies to learn and adapt their policies during training would be an interesting configuration, but it would require significant effort, as the current environmental setup (such as SMACv2) does not support this. Additionally, from an algorithmic design perspective, a new MARL algorithm would be needed to handle this situation. We plan to explore this in future work.\n\n*We hope the above responses are clear. If you have any other questions or concerns, we are happy to address and clarify them.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7YSOZFKIOF", "reviewer": "Reviewer_ZqFd", "summary": "This paper addresses the issue of training instability and slow convergence in MARL caused by the changing strategies of other agents. It proposes reducing the uncertainty faced during training by imitating the opponents' strategies. To address the challenge that opponents' actions are usually unobservable, the paper further proposes predicting the opponents' next state, providing corresponding derivation and analysis. Finally, the authors tested their approach in multiple experimental environments, achieving significant improvements in final rewards.", "questions": "1. The performance of many methods in the experiments fluctuate. How many seeds were used in the experiments? Previous experiments on SMACv2 usually uses more than 5 random seeds.\n2. Are the strategies of the enemies fixed in the environment?", "limitations": "The training process may be unstable.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-organized, with clear motivation and straightforward method introduction, making it easy to follow. The adaptation of IQ-Learn is easy to understand, and the derivation process is quite clear and technically sound.\n\nThe experiments included SOTA MARL algorithms and compared results under different imitation learning frameworks. In the majority of tasks across three different environments, the proposed IMAX-PPO algorithm achieved the highest expected rewards, demonstrating the effectiveness of the method.", "weaknesses": "I have not found any obvious flaws. My concern lies in the experiments. I noticed that the training curves in the SMACv2 and Gold Miner environments are very unstable, i.e., the performance flucuates. Is this due to adversarial imitation learning? The stability of the algorithm needs further validation. Additionally, the default settings have fixed enemy strategies, making the learning easier. If the enemies are also learning agents, the instability may be exaggerated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reading our paper and providing positive feedback.\n\n> The performance of many methods in the experiments fluctuate. How many seeds were used in the experiments? Previous experiments on SMACv2 usually uses more than 5 random seeds. \n\nThe main reason the win-rate curves are not very stable and fluctuate at some points is that MARL tasks are highly complex, particularly SMAC_v2, which is considered one of the most challenging tasks in MARL. This is why all the curves of all the methods are not as stable as those typically seen in single-agent training. Additionally, the inclusion of more components in our method (such as the IL component) can also contribute to instability. Despite this instability, our method consistently converges to better policies compared to other baselines.\n\nRegarding the number of seeds, we followed the standard setting in prior MARL papers to train and test our algorithm. We used 5 seeds for training and 32 seeds to evaluate the win rate for each given policy. We will clarify this in the paper.\n\n> Are the strategies of the enemies fixed in the environment? \n\nYes, all the enemy policies are fixed in the environment. This is a standard setting employed in other state-of-the-art MARL algorithms such as MAPPO and QMIX. Allowing enemies to learn and adapt their policies during training would be an interesting configuration, but it would require significant effort, as the current environmental setup (such as SMACv2) does not support this. Additionally, from an algorithmic design perspective, a new MARL algorithm would be needed to handle this situation. We plan to explore this in future work.\n\n*We hope the above responses are clear. If you have any other questions or concerns, we are happy to address and clarify them.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for carefully reading our paper and providing constructive feedback, which we have made efforts to address. Please find a summary of our responses below.

**Reviewer bLoo** mentioned some related work in opponent/agent modeling. In response, we have explicitly provided a discussion on the differences between our IL-based method and those in the opponent/agent modeling literature. The reviewer also raised questions about why IL is useful for solving the agent modeling problem and for effective MARL, as well as the practicality of our method when decomposing the information to facilitate learning. We have provided detailed responses to these questions. Additionally, the reviewer asked about applying the IL-based framework on top of other MARL algorithms. This point is well taken, and in the attached 1-page PDF, we have provided additional experiments for QMIX+IMAX. The experiments generally show that incorporating IMAX can improve QMIX, but this combination is generally outperformed by our main algorithm (MAPPO-IMAX). 

**Reviewer 4odM** asked whether SupMAPPO agents learn the enemies' next states using a joint enemy agent model. In response, we clarified that both SupMAPPO and our IMAX-MAPPO use a single joint model of the enemies. This approach is more practical than using an individual model for each agent.

**Reviewer ZqFd** questioned why our learning/evaluation curves are not very stable and how many seeds were used in the experiments. We have provided detailed responses to these questions and will add clarifications to the paper.

We thank all the reviewers for their comments and positive feedback, which we have tried to address and clarify. If you have any further questions, we are more than happy to discuss and clarify them.

Meta-review: All reviewers agree this is a good paper, on a relevant topic, with substantial contributions. There is nevertheless an opportunity to improve clarity and presentation, as raised in some concrete instances by multiple reviewers, for the camera-ready.
Decision: Accept (poster)