Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Games

Authors: viet  the
  bui
tien  mai
hong  thanh
  nguyen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:23:25

Abstract: Training agents in multi-agent games presents significant challenges due to their intricate nature. These challenges are exacerbated by dynamics influenced not only by the environment but also by strategies of opponents. Existing methods often struggle with slow convergence and instability.
To address these challenges, we harness the potential of imitation learning (IL) to comprehend and anticipate actions of the opponents, aiming to mitigate uncertainties with respect to the game dynamics.
Our key contributions include:
(i) a new multi-agent IL model for predicting next moves of the opponents - our model works with hidden actions of opponents and local observations;
(ii) a new multi-agent reinforcement learning (MARL) algorithm that combines our IL model and policy training into one single training process;
and (iii) extensive experiments in three challenging game environments, including an advanced version of the Star-Craft multi-agent challenge (i.e., SMACv2).
Experimental results show that our approach achieves superior performance compared to state-of-the-art MARL algorithms.

Main paper: 
 introduction: 
 Recent works in MARL have made a significant progress in developing new effective algorithms that can perform well in complex multi-agent environments including SMAC [34, 25] . Among these works, centralized training and decentralized execution (CTDE) [5] has attracted a great attention from the RL community due to its advantage of leveraging global information to train a centralized critic (i.e., actor-critic methods [18] ) or a joint Q-function (i.e., value-decomposition methods [23, 29] ). This approach enables a more efficient and stable learning process while allowing agents to act in a decentralized manner. Under this CTDE framework, off-policy methods such as MADDPG [18] and QMIX [23] have become very popular due to their data efficiency and state-of-the-art (SOTA) results on a wide range of benchmarks. On the other hand, on-policy gradient methods have been under-explored in MARL due to their data consuming and difficulty in transferring knowledge from single-agent to multi-agent settings. However, a recent work shows that on-policy methods (such as MAPPO, a multi-agent version of proximal policy optimization) outperforms all other SOTA methods including MADDPG and QMIX in various multi-agent benchmarks, and especially works well in complex SMAC settings [34] . Motivated by this promising result, we focus on improving the performance of policy gradient methods in MARL.We consider a partially observable MDP environment in which there are agents attempting to form an alliance to play against a team of opponents, where allied agents have to make decision independently without communicating with other members. We aim to enhance the performance of PPO in MARL with the introduction of a novel opponent-imitation component. This new component is then integrated into the MAPPO framework to enhance the policy learning of allied agents. A key challenge in our problem setting is that allied agents are unaware of actions taken by their opponents. In addition, each allied agent only has local observations of opponents locating in the current neighborhood of the agent -the locations and neighborhoods of all players are changing over time depending on actions taken by players and the dynamics of the environment. Lastly, learning to imitate opponents occurs during the policy learning process of the allied agents. The inter-dependency between these two learning components makes the entire learning process significantly challenging.We address these challenges while providing the following key contributions. First, we convert the problem of imitating the opponent policy into predicting their next states. The outcome of this next state prediction is an indirect implication of the opponent policy. We then cast the problem of opponent next-state prediction as a new multi-agent imitation learning (IL) problem. We propose a new multi-agent IL algorithm, which is an adaptation of IQ-Learn [9] (a SOTA IL algorithm), that only considers local opponent-state-only observations. Especially, instead of imitating the opponents' policy, our IL algorithm targets the prediction of next states of the neighboring opponents. Second, we provide a comprehensive theoretical analysis which provides bounds on the impact of the changing policy of the allied agents (as a result of the policy learning process) on our IL outcomes.Third, we present a unified MARL algorithmic framework in which we incorporate our IL component into MAPPO. Our idea is to combine each allied agent's local observations with the next-state prediction of neighboring opponents of that agent, creating an augmented input based on which to improve the decision making of the allied agent at every state. This novel integration results in a new MARL algorithm, which we name Imitation-enhanced Multi-Agent EXtended PPO (IMAX-PPO).Finally, we conduct extensive experiments in several benchmarks ranging from complex to simple ones, including: SMACv2 (an advanced version of the Star-Craft multi-agent challenge) [4] , Google research football (GRF) [15] , and Gold Miner [7] . Our empirical results show that our new algorithm consistently outperforms SOTA algorithms significantly accross all these benchmarks.2 Related Work MARL. The literature on MARL includes both centralized and decentralized algorithms. While centralized algorithms [2] learn a single joint policy to produce joint actions of all the agents, decentralized learning [17] optimizes each agent's local policy independently. There are also algorithms based on centralized training and decentralized execution (CTDE). For example, methods in [18, 5] adopt actor-critic structures and learn a centralized critic that takes global information as input. Value-decomposition (VD) is a class of methods that represent the joint Q-function as a function of agents' local Q-functions [29, 23] . Alternatively, the use of policy-gradient methods, such as PPO [26] , has also been investigated in multi-agent RL. For example, [3] propose independent PPO (IPPO), a decentralized MARL, that can achieve high success rates in several hard SMAC maps. IPPO is, however, overall worse than QMIX [23] , a method based on factorizing Q function to facilitate CTDE. Later methods based on factorized Q-learning include QTRAN [27] and QPLEX [30] , where QPLEX has been shown to achive better perforamance than QMIX and QTRAN. Recently, [34] develop MAPPO, a PPO-based MARL algorithm that outperforms QMIX and QPLEX on some popular multi-agent environments such as SMAC [25, 4] and GRF [15] . To the best of our knowledge, MAPPO is currently a SOTA method for MARL. Our work integrates a new opponent imitation model into MAPPO, resulting in a new MARL algorithm that outperforms SOTA methods on various challenging game tasks.Imitation Learning (IL) . In this study, we employ IL to anticipate the opponents' moves. IL is known as a compelling approach for sequential decision-making [20, 1] . In IL, a collection of expert trajectories is provided, with the objective of learning a policy that emulates behavior similar to the expert's policy. One of the simplest IL methods is Behavioral Cloning (BC), which aims to maximize the likelihood of the expert's actions under the learned policy. BC disregards environmental dynamics, rendering it suitable only for uncomplicated environments. Several advanced IL techniques, encompassing environmental dynamics, have been proposed [24, 8, 12] . While these methods operate in complex and continuous domains, they involve adversarial learning, making them prone to instability and sensitivity to hyperparameters. The IQ-learn [9] stands as a cutting-edge IL algorithm with distinct advantages, specifically its incorporation of dynamics awareness and non-adversarial training. It's important to note that all the aforementioned IL methods were designed for single-agent RL. In contrast, the literature on multi-agent RL is limited, with only a handful of studies addressing IL in multi-agent RL. For instance, [28] presents an adversarial training-based algorithm, named Multi-agent Generative Adversarial IL. It's worth noting that all the IL algorithms mentioned above are established on the premise that expert (aka. opponent in our case) actions are either observable or can be accessed via sampling, which implies that no existing algorithm can be directly applied to our multi-agent game settings with local state-only observations. Opponent Modeling. Many existing works in MARL attempt to capture the learning process of opponents and incorporate it into the learning of the agent's policy [6, 14, 33] . For example, the LOLA algorithm [6] considers the impact of one agent's policy on the parameter update of other opponents while Meta-MAPG [14] combines LOLA with meta-learning, accounting for continuous adaptation. Another important line of research on opponent modelling follows hierarchical reasoning, considering each agent holds a belief about the other agents according to varying levels of reasoning ability [32, 31, 35, 19] . As an example, in [32] , they introduce a probabilistic recursive reasoning framework in which variational Bayes methods are used to approximate the opponents' conditional policies. Depart from these two lines of research, there are many other works that attempt to learn a representation for the opponent's policy or to consider the prediction of opponents' actions as an an auxiliary task that can be trained simultaneously with the RL part [11, 13, 22, 21, 10] . For example, [21] use variational encoder to model the opponents' fixed policies while [10] apply behavioral cloning together with agent identification to learn a hybrid generative-discriminative representation for the opponents' policy. All the aforementioned related works require having access to opponent's observations and actions during training and/or execution. In our work, on the other hand, the opponent modeling can be only trained based on local observations of each agent in the allied team. These local observations contain limited information about current states of nearby opponents while opponents' actions are unobservable. As a result, existing methods on opponent modeling are not applicable in our multi-agent setting. 
imax-ppo with gail: 
 To predict enemies' next states, the GAIL algorithm by [12] can be adapted in a similar way as our main IMAX-PPO algorithm. Building upon our new MDP framework, which is necessary to handle the unobservable-action issue, the aim is also to learn a policy Π(S ′ |W ) by creating a discriminator D(W, S ′ ) that distinguishes between generated transitions (W, S ′ ) and those collected from interacting with the actual enemy agents. The GAIL loss function can be formulated asmin Π max D J GAIL (Π, D) = E (S ′ ,Q)∼ρ Π,α [log D(S ′ , W )] + E ρe,α [log(1 -D(W, S ′ ))] -λH(Π)] ,where H(Π) is the entropy of Π. In practice with partial observations, we can model D and Π as neutral networksD w (S e,next i , o α i , a α i ) and Π θ (S e,next i , o α i , a α i ), where w, θ are trainable parameters. The objective can be formulated as follows:min θ max w J GAIL (θ, w) = E (S e,next i ,o α i ,a α i )∼ρ Π,α log D(S e,next i , o α i , a α i ) + E (S e,next i ,o α i ,a α i )∼buffer log(1 -D(S e,next i , o α i , a α i )) -λH(Π θ ) .Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss limitations of the work in the conclusion section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
multi-agent pomdp setting: 
 We consider a multi-player Markov game in which there are multiple agents forming an alliance to play against some opponent agents. We present the Markov game as a tuple {S, N α , N e , A α , A e , P, R}, where S is the set of global states shared by all the agents, N α and N e are the set of ally and enemy agents, A α = i∈N A A α i is the set of joint actions and A e = j∈Ne A e j is the set of joint actions of all the ally agents, P is the transition dynamics of the game environment, and R is a reward function that takes inputs as states and actions of all agents and returns the corresponding rewards. At each time step where the global state is S, each ally agent i ∈ N α makes an action a α i according to a policy π α i (a α i |o α i ), where o α i is the observation of ally agent i given state S. The joint action of allied agents can be now defined as A α = {a α i | i ∈ N α }, and the joint policy is defined accordingly:Π α (A α |S) = i∈N α π α i (a α i |o α i ).The enemy agents, at the same time, make a joint action A e = {a e j | j ∈ N e } with the probability: Π e (A e |S) = j∈N e π e j (a e j |o e j ).After all agents make decisions, the global state transits to a new state S ′ with the probability P (S ′ |A e , A α , S). In our setting, the enemies' policies Π e are fixed and thus can be treated as a part of the environment dynamics, as follows:P (S ′ |A α , S) = A e Π(A e |S)P (S ′ |A e , A α , S)Our goal is to find a policy that optimizes the allies' expected joint reward, formulated as follows: 1maxEQUATIONThe game dynamics involve both the environment dynamics and the joint policy of enemies, making the training costly to converge. We aim to migrate uncertainties associated with these game dynamics by first predicting the opponent policy based on the allies' past observations and leveraging this prediction into guiding the policy training for the allies.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The data we used, along with our source code, has been uploaded with the main paper. We have also provided sufficient instructions for their use.Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: We provides all the details in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
opponent policy imitation: 
 The key challenge in our problem is that actions taken by opponents are hidden from allied agents. Moreover, each allied agent has limited observations of other agents; they can only obtain information about nearby opponents. For example, in the SMAC environment, for each allied agent, besides information about the agent itself, the allied agent is also aware of the relative position and health point, etc. of the neighboring opponents. Here, it is important to note that prior works on IL with state-only observations all assume that actions are not available in the expert demonstrations but can be accessed via sampling, which is not the case in our context. Alternatively, one could apply standard supervised learning for this opponent-next-state prediction task. However, a well-known drawback of this approach is that it disregards environment dynamics and often struggles with distribution shifts [16] . As shown later in our experiments, our IL approach significantly outperforms this supervised-learning approach.We now present our IL formulation and our adaptation of IQ-Learn for solving our new IL problem.For the sake of theoretical analysis, this section focuses on IL with global state-only observations. We then introduce a new practical algorithm later which addresses local observations of allied agents.Opponent Next-State Prediction as an IL. To formulate the problem as an IL that accounts for the action-unobservable issue, we introduce a new notion of the "expert" state in our IL problem as a pair W = (S, A α -) which comprises of the original state S and the joint action of the allies A α taken in the previous step that leads to state S. The action space of the "expert" is equivalent to the original state space S. We then introduce a new notion of a reward function for the expert as R e (W, S ′ ). This action (S ′ ) of the expert is basically a resulting state of joint actions of the allies A α and hidden joint actions of the enemies A e taken at state S. Altogether with the reward function R e (W, S ′ ), we introduce a new notion of joint policy for the expert, Π e (S ′ |W ) (or Π e (S ′ |S, A α -)), which is essentially the probability of ending up at a global state S ′ from state S. The dynamics in this IL setting becomes P (W ′ | W, S ′ ) = P ((S ′ , A) | (S, A α -), S ′ ) = Π α (A | S) (which is the allies' policy) where W ′ = (S ′ , A) and A is the action taken by the allies at state S.Let Π Π Π = {Π : S × S × A α → [0, 1], S ′ ∈S Π(S ′ |S, A α -) = 1, ∀S, S ′ ∈ S, A α ∈ A α }be the support set of the imitating policy. We now introduce the maximum-entropy inverse RL framework [12] w.r.t the new notions of the expert's reward and policy (R e (S ′ |W ), Π e (S ′ |W )):EQUATIONwhere ρ e,α is the occupancy measure of (W, S ′ ) given by the expert policy Π e and the ally joint policy Π α , and ρ Π,α the occupancy measure of (W, S ′ ) given by the imitation and ally policies. In particular, ρ e,α can be computed as follows:ρ e,α (S ′ , W ) =(1 -γ)Π e (S ′ | W ) ∞ t=0 γ t P (W t = W | Π e , Π α )An Adaptation of IQ-Learn. Drawing inspiration from the SOTA IL algorithm, IQ-learn, we construct our IL algorithm which is an adaptation of IQ-Learn tailored to our multi-agent environment.The main idea of IQ-learn is to convert a reward learning problem into a Q-function learning one. To apply IQ-Learn to our setting, we present the following new soft and inverse soft Bellman operators, which is as adaption from the original ones introduced in [9] :EQUATIONEQUATIONwhere the state value function is computed as follows:V e Π (W ) = E S ′ ∼Π Q e (W, S ′ ) -ln(Π(S ′ |W )) First, it is clear that B Π,R e Q eis contractive, thus defining a unique fixed point solutionQ * such that B Π,R e Q * = Q * .Let us further define the following function of Π and Q e for the Q-function learning:J(Π, Q e ) = E ρ e,α [T Π Q e (W, S ′ )] -E ρ Π,α [T Π Q e (W, S ′ )] + E ρ Π,α [ln Π(S ′ |W )]We obtain a theoretical result on a connection between the learning reward and learning Q-functions: 3Proposition 1. For any reward function R e , let Q * be the unique fixed point solution to the soft Bellman equationB Π,R e Q * = Q * , then: L(Π, R e ) = J(Π, Q * ), and for any Q e , J(Π, Q e ) = L(Π, T Π Q e ).Proposition (1) indicates that the IL problem in ( 2) is equivalently to the Q-value-based IL problem:EQUATIONSuppose Q * is a solution to (5), then rewards can be recovered by taking R e (W,S ′ ) = Q * (W, S ′ ) - γE W ′ [V e Π (W ′ )].Under this viewpoint, Prop. 2 shows that key properties of original IQ-learn still hold in our multi-agent setting with missing observations, making our IL algorithm convenient to use. Proposition 2. The IL problem ( 5) is equivalent to the maximization max Q e J(Π Q , Q e ) where the imitation policy can be computed based on Q e as follows:Π Q (S ′ |W ) = exp(Q e (W, S ′ )) S ′′ exp(Q e (W, S ′′ )) Moreover, the function J(Π Q , Q e ) is concave in Q e .The above IL algorithm is trained during the training of the allies' policies, which means the dynamics P (W ′ |W, S ′ ) = Π α (A α |S) change during the IL process. Therefore, we aim to analyze the impact of these changes on the imitation policy. According to Proposition 2, given Q e , we can compute corresponding optimal imitation policy as Π Q . Therefore, the value function V e Π (W ) can be alternatively write as V e Q (W ). We now can denote the loss function of the imitation model (Eq. 8) as a function of the allies' joint policy explicitly:Φ(Π α |Q e ) ≡ J(Π, Q e ).We first present our results about bounds on the impact of the allies' changing policies on the IL loss function (Prop. 3). Based on these results, we then provide a bound on the IL learning outcome accordingly (Corollary 4). Let us denote byQ = max (W,S ′ ) Q e (W, S ′ ) an upper bound of Q e .Proposition 3. Given two allies' joint policies Π α and Π α such that KL(Π α (•|S)|| Π α (•|S)) ≤ ϵ for any state S ∈ S, the following inequality holds:Φ(Π α |Q e ) -Φ( Π α |Q e ) ≤ αQ + β ln |S| √ 2 ln 2ϵ where α = γ (1-γ) 2 + γ 2 1-γ + (3 -γ), and β = γ 2 (1-γ) + 3 -γ.Let Φ(Π α |Q e , Π) be the objective of IL J(Π, Q e ), written as a function of the allies' joint policy Π α .The following proposition establishes a bound for the variation of|Φ(Π α |Q e , Π) -Φ( Π α |Q e , Π)| as function of KL(Π α || Π α ), for any pair of allies' joint policies (Π α , Π α ).Prop. 3 allows us to establish an upper bound for the IL when the allies' joint policy changes.Figure 1 : An overview of our IMAX-PPO algorithm. Each local observation o α i of an ally agent i includes information about itself, as well as enemy and ally agents in its neighborhood (which changes over time). The output of the IL component is the predicted next states of neighboring enemy agents (predictions for the non-neighbor enemies will be masked out).Π α and Π α with KL(Π α (•|S)|| Π α (•|S) ≤ ϵ, ∀S ∈ S, then: maxQe Φ(Π α |Q e ) -maxQe Φ( Π α |Q e ) ≤ O( √ ϵ)Since the allies' joint policy Π α will be changing during our policy learning process, the above result implies that the imitating policy will be stable if Π α becomes stable, and if Π α is converging to a target policy Π α * , then the imitator's policy also converges to the one that is trained with the target ally policy with a rate of KL(Π α || Π α ). That is, if the actual policy is within a O(ϵ) neighborhood of the target policy (i.e., KL(Π α || Π α ) ≤ ϵ) then the expected return of the imitating policy is within a O( √ ϵ) neighborhood of the desired "expected return" given by the target policy. 
imax-ppo: imitation-enhanced multi-agent extended ppo algorithm: 
 We present our MARL algorithm for the competive game setting. We first focus on a practical implementation of an IL algorithm taking into account local observations. We then show how to integrate this into our MARL algorithm. We call our algorithm as IMAX-PPO, standing for Imitation-enhanced Multi-Agent EXtended PPO algorithm.In previous section, we present our new IL algorithm (which is an adaptation of IQ-Learn) to learn an expert policy Π e (S ′ |W ) = Π e (S ′ |S, A α ) that behaves similarly to the probabilities of ending up at state S ′ when the current global state is S and the allies' joint action is A α . From the allies' perspective, to run this IL algorithm, it requires the allies to have access to the global state S. However, each ally agent i can only observe local states of its neighboring enemies (such as their locations, speeds, etc.). Therefore, we adapt our IL algorithm in accordance with such local information. The goal is to predict next states of enemies in the neighborhood of each allied agent i, denoted by Π e (S e,next i |w α i ), where local information w α i = (o α i , a α i,-) with o α i is an observation vector of agent i, containing the local states of the agent i itself and of all the agents in the neighborhood that are observable by agent i. In particular, given that s α i k is the local state of an ally agent i k and s e j k is of an enemy agent j k and N (i) is the neighborhood of i, we have:o α i = {s α i } ∪ {s α i k : i k ∈ N (i) ∩ N α } ∪ {s e j k : j k ∈ N (i) ∩ N e }To apply our IL algorithm to this local observation setting, we build a common policy network Π e ψπ and Q network Q e ψ Q for all the agents where ψ π and ψ Q are the network parameters. The IL objective function can be reformulated according to local observations of the allies as follows:J( Π e ψπ , Q e ψ Q ) = i∈N α E (S e,next i ,w α i )∼ρ e,α Q e (S e,next i , w α i ) (6) -γE w α,next i [V e Π (w α,next i )] -(1 -γ)E w α i0 ∼P 0 ,Π α V e Π (w α i0 )wherew α i = (o α i , a α i,-), w α,next i = (o α,next i , a α i ) (a α i is the action taken by agent i at observation o α i , resulting in next observation o α,next i ), S e,next i = {s ′e j k : j k ∈ N (i)∩ N e } is the next states of enemies in the current neighborhood of the agent i. In addition, the value functions are re-formulated as follows:V e Π (w α i ) = E (S e,next i )∼ Π e ψπ Q e ψ Q (S e,next i , w α i ) -ln( Π e ψπ (S e,next i |w α i ))In the end, we can update Π e ψπ and Q e ψ Q by the following actor-critic rule: for a fixedQ e ψ Q , we update ψ Q to maximize J( Π e ψπ , Q e ψ Q ), and for a fixed Π e ψπ , we apply soft actor-critic (SAC) to update ψ π .We now combine the local observation o α i of each allied agent i with the next-state prediction S e,next i of its neighboring enemies (obtained by our IL algorithm) to create an augmented input. This augmented input is used to improve the policy learning of the allied agent i. That is, we aim to optimize the allies' policy Π α θ (a α i |o α i , S e,next i ), i ∈ N α } that optimizes the long-term expected joint reward:max Π α θ E (a α i ,o α i ,S e,next i )∼Π α θ i∈N α R α i (o α i , a α i )where o α i is an observation vector of agent i, S e,next i is the information derived from the imitator for agent i, a α i ∈ A α i is a local action of agent i. To facilitate the training and integration of the imitation learning policy into the MARL algorithm, for every ally agent i, we gather game trajectories following the structure (o i , a α i , S e,next i). These gathered observations are then stored in a replay buffer to train the imitation policy Π e ψπ (S e,next i |o α i , a α i ). In the IMAX-PPO framework, at each game state S, considering the current actor policy Π α and the imitating policy Π e , for each agent i ∈ N α , we draw a sample for the allied agents' joint action A α ∼ Π α . Corresponding local observation o α i and action a α i of each agent i are then fed as inputs into the imitation policy to predict the subsequent state S e,next i ∼ Π e (•|o α i , a α i ). Once the predicted local states {S e,next i , i ∈ N α } are available, it is used as input to the actor policy Π α θ in order to generate new actions for the allied agents. In simpler terms, we select a next local actiona ′α i ∼ Π α θ (•|o α i , S e,next i). Beside the allies' policy network, we also use a centralized value network V α θv (S) and update it together with the policy network in an actor-critic manner, similarly to MAPPO. The actor-network is trained by optimizing the following objective:EQUATIONwherer i (θ) = Π α θ (a α i |o α i ,S e,next i ) Π α θ old (a α i |o α i ,S e,next i) and A is the advantage function, calculated by Generalized Advantage Estimation (GAE). The Critic network is trained by optimizingΦ α (θ v ) = E S max [V α θv (S) -R(S)] 2 , [V clip θv,θ v,old (S) -R(S)] 2 where R(S) = A + V α θ v,old (S) and V clip θv,θ v,old (S) = clip(V α θv (S), V α θ v,old (S) -ϵ, V α θ v,old (S) + ϵ).We provide the key stages in Algorithm 1. Additionally, Fig. 1 serves as an illustration of our IMAX-PPO. 
experiments: 
 We evaluate the performance of our IMAX-PPO algorithm (Algo. 1) in comparison with some standard and SOTA multi-agent RL algorithms: IPPO, MAPPO, QMIX and QPLEX. In addition, to examine the impact of our multi-agent IL model on the performance of IMAX-PPO, we include two versions of IMAX-PPO where opponent's next states are predicted by (i) an adaption of the GAIL # Updating imitator:3: ψ Q,t+1 = ψ Q,t + κ e Q ∇ ψ Q [J(ψ Q )]# Train Q function using the objective in (6) 4: # Updating allies' policy:ψ π,t+1 = ψ π,t -κ e π ∇ ψπ E S e,next i [V e Π (S6:θ t+1 = θ t + κ α π ∇ θ L α (θ)# Update allies' actor by maximizing L α (θ)7: algorithm [12] , denoted as IMAX-PPO (GAIL) and (ii) the IQ-learn adaption (i.e. Algorithm 1, denoted as IMAX-PPO (InQ). Moreover, to compare out IL-based approach with supervise learning, we include an approach based on MAPPO where the opponent's next states are learned and predicted by standard supervising learning. We denote this approach as Sup-MAPPO.θ v,t+1 = θ v,t -κ α V ∇ θv Φ α (θ v ) #The details of Sup-MAPPO and IMAX-PPO (GAIL) are provided in the appendix. We run extensive experiments in three multi-agent competitive environments: SMACv2, Google Research Football (GRF), and Miner. Each reported value is computed based on 32 different rounds of game playing (each corresponds to a different random seed).SMACv2. SMACv2 [4] is an advanced variant of SMAC, driven by the aim to present a more challenging setting for the assessment of cooperative MARL algorithms. In SMACv2, scenarios are procedurally generated, which require agents to generalize to previously unseen settings (from the same distribution) during evaluation. This benchmark consists of 15 sub-tasks where the number of agents varies from 5 to 20. The agents can play with opponents of different difficulty levels. In comparison to SMACv1 [25] , SMACv2 stands apart by permitting randomized team compositions, varied starting positions, and an emphasis on augmenting diversity.Figure 2 shows the performance of the five algorithms during the training process across 15 sub-tasks. The x-axis is the number of training steps and the y-axis is the winning rates averaged over 32 rounds of evaluations. In Figure 2 , IMAX-PPO (InQ) consistently and significantly outperforms other baselines. Our algorithm frequently attains quicker convergence; it achieves high win rates at earlier training stages. This could be attributed to the incorporation of our IL component, which facilitates faster comprehension of opponents throughout the game. In particular, the IMAX-PPO (InQ) outperforms the two other variants IMAX-PPO (GAIL) and Sup-MAPPO, indicating the advantage of our inverse-Q approach over other IL (i.e. GAIL) and traditional supervising learning methods. Details of win rates at the end of training are shown in Table 1 .. This is a challenge on Kaggle competitions made by Google Research team [15] . We focus on three main sub-tasks, sorted based on increasing difficulty levels: (i) academy-3-vs-1-with-keeper: three allies try to score against a goal-keeping opponent; (ii) academycounterattack-easy: four allies versus a counter-attack opponent and a goal-keeping opponent; and (iii) academy-counterattack-hard: four allies versus two counter-attack opponents and a goalkeeper.By default, the representations of all agents' observations are RGB pixels in GRF, so we pre-process this information by distilling some important features such as object positions, object directions, distances between objects, etc. The final win rates are in Table 1 , which shows that IMAX-PPO (InQ) achieves nearly 100% win-rates, and significantly outperforms other baselines. 4Gold Miner [7] . This is another competitive multi-agent game for evaluating our methods, originating from a MARL competition. Multiple miners navigate in a 2D terrain containing obstacles and repositories of gold. Players get points according to the volume of gold they successfully extract. This game is challenging to win as the agents have to learn playing against extremely well-designed heuristic-based enemies. In this game, the ally agents win if the allied team's average mined gold is higher than that of the enemy team.We customized the original environment into three sub-tasks (between two allies against two enemies) of three difficulty levels: (i) Easy (easy_2_vs_2): The enemies' greedy strategy is to find the shortest way to the golds; (ii) Medium (medium_2_vs_2): One enemy is greedy, and the other follows the algorithm of the second-ranking team in the competition; and (iii) Hard (hard_2_vs_2): The enemies are the first-and second-ranking teams in the competition.For this environment, the win rates are in Table 1 . Again, IMAX-PPO (InQ) obtained superior win rates across all three tasks. Especially, in the hard-level task, our algorithm manages to win more than 50% of the time against the first and second-ranking teams in the competition. 
conclusion: 
 We introduced a novel principled framework for enhancing agent training in multi-agent environments through IL. Our new IL model, adapted from IQ-learn, can predict opponents' policy using only local state observations. By integrating this model into a multi-agent PPO algorithm, our IMAX-PPO algorithm consistently outperforms previous SOTA algorithms such as QMIX and MAPPO. This improvement is observed across various challenging multi-agent tasks, including SMACv2 and GRF. A possible limitation of our work is that it relies on the assumption that the enemies do not update their policies during training (even though this is a standard setting in cooperative multi-agent reinforcement learning [4, 25] ). A future direction would be to delve into these aspects to develop efficient MARL algorithms for cooperative-competitive multi-agent RL. e , J(Π, Q e ) = L(Π, T Π Q e ).Proof. The proof is similar to that given in [9] , as we can see that if Q * is a solution to the soft Bellman equation, thenQ * (W, S ′ ) = R e (W, S ′ ) + γE W ′ [V * Π (W ′ )] where V * Π (W ) = E S ′ ∼Π [Q * (W, S ′ )-ln(Π(S ′ |W ))] which implies R e (W, S ′ ) = Q * (W, S ′ ) -γE W ′ [V * Π (W ′ )] = T Π Q * (W, S ′ ) which validates L(Π, R e ) = J(Π, Q * ).The second inequality is just a trivial result from the definition of J and L. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA]Justification: The paper develops a general algorithm for multi-agent RL, which we have tested only in simulated environments. As such, we do not foresee any direct societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: [TODO]Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have provided clear citations to the source code and data we used in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: Our source code is submitted alongside the paper, accompanied by sufficient instructions. We will share the code publicly for re-producibility or benchmarking purposes.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Our following Proposition 5 shows that the function J(Π, Q e ) can be reformulated in a more compact form that is convenient for training. Proposition 5. The function J(•) can be written as follows:EQUATION) where S 0 is an initial state and P 0 is the initial state distribution in the original MDP.Proof. From the definition of J(Π, Q e ) , we write function J asEQUATIONWe consider the second and third terms of (9) and writeEQUATION) as desired.Proposition 2: The problem max Q e min Π J(Π, Q e ) is equivalent to the maximization max Q e J(Π Q , Q e ) whereΠ Q (S ′ |W ) = exp(Q e (W, S ′ )) S ′′ exp(Q e (W, S ′′ )) Moreover, the function J(Π Q , Q e ) is concave in Q e .Proof. To simplify the proof, we first consider the following simpler optimization problem. Let p 1 , p 2 , ..., p N ∈ [0, 1] and x 1 , ..., x N are N real numbers. Consider the maximization problemmax p∈[0,1] N N i=1 x i p i -p i ln p i (P1) subject to i p i = 1Using the KKT condition, if p * is an optimal solution to the above convex problem, p * needs to satisfy the following conditions: there exists λ such thatx i -ln p * i -1 -λ = 0, ∀i i p * i = 1 which implies p * i = exp(x i -1 -λ) Combine this with the condition i p * i = 1 we should have exp(-1 -λ) = i exp(x i ) and p * i = exp(xi) j exp(xj ) (*).In addition, when p = p * , the objective function of (P1) can be written asf (x) = i x i p * i -p * i ln(p * i ) = i p i (x i -ln(p * i )) = i p i x i -x i + ln i exp(x i ) = ln i exp(x i )We then see that f (x) has a log-sum-exp form, thus it is convex in x (**).We now return to the minimax problem max Q e min Π J(Π, Q e ). For any fixed Q e , it can be seen that the problem min Π J(Π, Q e ) is equivalent tomax Π γE W [V e Π (W )] + (1 -γ)E W0∼P 0 [V e Π (W 0 )]We first consider the problem max Π V e Π (W ), recalling that V e Π (W ) can be written asV e Π (W ) = S ′ Π(S ′ |W ) × [Q e (W, S ′ ) -log(Π(S ′ |W ))]Then it can be seen that each term S ′ Π(S ′ |W ) × [Q e (W, S ′ ) -log(Π(S ′ |W ))] has the form of (P1), thus from (**) we see that V e Π (S) is maximized atΠ Q (S ′ |W ) = exp(Q e (W, S ′ )) S ′′ exp(Q e (W, S ′′ ))Moreover, according to the result (**) proved above, when Π = Π Q , V e Π (W ) is convex in Q e . Consequently, the loss function of the IQ-learnJ(Π, Q e ) = E (S,A α -,S ′ )∼(ρ e,α ) [Q e (W, S ′ )-γE W ′ V e Π (W ′ )] + (1 -γ)E W 0 ∼P 0 V e Π (W 0 ) is concave in Q e ,which completes the proof.Corollary 6. If Π = Π Q (defined in Proposition 2), then we can write V e Π (S) as follows:V e Π (W ) = V e Q (W ) = ln S ′ exp Q e (W, S ′ )Note that Corollary 6 is just a direct result from the proof of Proposition 2.A.5 Proof of Proposition 3 Proposition 3: Given two allies' joint policies Π α and Π α such that KL(Π α (•|S)|| Π α (•|S)) ≤ ϵ for any state S ∈ S, the following inequality holds:Φ(Π α |Q e ) -Φ( Π α |Q e ) ≤ αQ + β ln |S| √ 2 ln 2ϵwhere Q = max(W,S ′ ) Q e (W, S ′ ) is an upper bound of Q e , α = γ (1-γ) 2 + γ 2 1-γ + (3 -γ), and β = γ 2(1-γ) + 3 -γ..Proof. We first write the loss function asΦ(Π α |Q e ) = E (S,A α -,S ′ )∼(ρ e,α ) Q e (W, S ′ )-γE W ′ ∼Π α V e Q (W ′ ) -(1 -γ)E W 0 ∼P 0 ,Π α V e Q (W 0 )We first write the difference between two loss values asEQUATIONWe first consider the first term of (12) . Let us denote the following functionΓ(S ′ |W ) = E (S ′ t ,Wt)∼Π e , Π α t=0 γ t (Q e (S ′ t , W t )) (S ′ 0 , W 0 ) = (W, S ′ )and write the first term of ( 12) asEQUATIONWe first see thatΓ(S|W ) = E (S ′ t ,Wt)∼Π e , Π α t=0 γ t (Q e (S ′ t , W t )) (S ′ 0 , W 0 ) = (W, S ′ ) ≤ Q 1 -γThus, we can bound the second term of ( 13) asEQUATIONWe also see thatΓ(S ′ t |W t ) = Q e (S ′ t , W t ) + γE (S ′ t+1 ,Wt+1)∼Π e , Π α |S ′ t ,Wt Γ(S ′ t+1 |W t+1 )Thus, we further can bound the first term ( 13) asE (S ′ t+1 ,Wt+1)∼Π e ,Π α |S ′ t ,Wt Q e (S ′ t , W t ) + γΓ(S ′ t+1 |W t+1 ) -Γ(S ′ t |W t ) = γE (S ′ t+1 ,Wt+1)∼Π e ,Π α |S ′ t ,Wt Γ(S ′ t+1 |W t+1 -γE (S ′ t+1 ,Wt+1)∼Π e , Π α |S ′ t ,Wt Γ(S ′ t+1 |W t+1 ≤ γH||Π α (•|S t ) -Π α (•|S t )|| ∞ ≤ γH max S 2 ln 2KL(Π α (•|S)|| Π α (•|S))where:H = max (W,S ′ ) {Γ(S ′ |W )} = max (W,S ′ ) E (S ′ t ,Wt)∼(Π e , Π α ) t=0 γ t (Q e (S ′ t , W t )) ≤ Q 1 -γwhere Q is an upper bound of Q e (•|•). Therefore, we can bound (13) asEQUATIONFor the second term of ( 12), we can bound it asEQUATIONwhereEQUATIONandU (W ) ≤ 1 1 -γ max W V e Q (W ) ≤ 1 1 -γ max W ln S ′ exp(Q e (W, S ′ )) ≤ 1 1 -γ (ln |S| + Q)Thus, given any W t , we haveEQUATION)where 17) can be bounded asK = 1 1-γ (ln |S| + Q) ≥ max W {U (W )}. So, (EQUATIONMoreover,EQUATIONCombine ( 17) and ( 20) and ( 21), we bound the second term of ( 12) asEQUATIONThe last term of ( 12) can be bounded simply as asEQUATIONCombine ( 16), ( 22) and ( 23) we getΦ(Π α |Q e ) -Φ( Π α |Q e ) ≤ γQ (1 -γ) 2 + γ 2 (ln |S| + Q) (1 -γ) + (3 -γ)(ln |S| + Q) max S 2 ln 2KL(Π α (•|S)|| Π α (•|S)) = γ (1 -γ) 2 + γ 2 1 -γ + (3 -γ) Q + γ 2 (1 -γ) + 3 -γ ln |S| max S 2 ln 2KL(Π α (•|S)|| Π α (•|S))which completes the proof.We extend the results in Section 4.2 to provide bounds for the case of continuous action space. In this scenario, an actor-critic method is used since the policy Π Q cannot be computed exactly as in Prop. 2.In this case, we can use an explicit policy Π to approximate Π Q instead. We then iteratively update Q e and Π alternatively using the loss function in Prop. 5. In particular, for a fixed Q e , a soft actor update max Π E S ′ ∼Π(•|W ) [Q e (W, S ′ ) -ln Π(S ′ |W )] will bring the imitation policy Π closer to Π Q .Let Φ(Π α |Q e , Π) be the objective of IL in Eq. ( 8), written as a function of the allies' joint policy Π α .The following proposition establishes a bound for the variation of |Φ(Π α |Q e , Π) -Φ( Π α |Q e , Π)| as function of KL(Π α || Π α ), for any pair of allies' joint policies (Π α , Π α ).Proposition 7 (Continuous state space S). Given two allies' joint policies Π α and Π α such that for every state S ∈ S, KL(Π α (•|S)|| Π α (•|S)) ≤ ϵ, the following inequality holds:Φ(Π α |Q e , Π) -Φ( Π α |Q e , Π) ≤ αQ -βH √ 2 ln 2ϵwhereH = inf W S ′ Π(S ′ |W ) ln Π(S ′ |W ), is a lower bound of the entropy of the actor policy Π.Proof. We can follow the same arguments as in the proof of Proposition 3 above, with the only difference being when we bound V e Q (W ). Here, V e Q (W ) is replaced by V e Π (W ) and can be bounded asEQUATIONWe now see that the entropy term S ′ Π(S ′ |W ) log(Π(S ′ |W )) is minimized when Π(S ′ |W ) = 1/maxQe minΠ Φ(Π α |Q e , Π) -maxQe minΠ Φ( Π α |Q e , Π) ≤ O( √ ϵ) (continuous)Proof. To simplify the notation, we first prove the following result:Lemma 9. Given ϵ > 0, and two functions f (x), g(x) such that |f (x) -g(x)| ≤ ϵ for any x ∈ X (X is the feasible set of x). The following hold true| max x f (x) -max x g(x)| ≤ ϵ | min x f (x) -min x g(x)| ≤ ϵTo prove the above lemma, let x f , x g be optimal solutions to max x f (x) and min x g(x), respectively. We consider 2 cases (i) If max x f (x) ≥ min x g(x), then we have| max x f (x) -min x g(x)| = f (x f ) -g(x g ) ≤ f (x f ) -g(x f ) (a) ≤ ϵ where (a) is because |f (x) -g(x)| ≤ ϵ for any x ∈ X (ii) If max x f (x) ≤ min x g(x), then we have | max x f (x) -min x g(x)| = g(x g ) -f (x f ) ≤ g(x g ) -f (x g )≤ϵ Thus | max x f (x) -max x g(x)| ≤ ϵ. The inequality | min x f (x) -min x g(x)| ≤ ϵ can be validated in the same way.We now get back to the main proof. Since Φ(Π α |Q e ) -Φ( Π α |Q e ) ≤ O( √ ϵ) (Proposition 3), Lemma 9 above implies thatmax Q e Φ(Π α |Q e ) -max Q e Φ( Π α |Q e ) ≤ O( √ ϵ)Moreover, from Proposition 7, applying Lemma 9 twice, we have the followingmin Π Φ(Π α |Q e , Π) -min Π Φ( Π α |Q e , Π) ≤ O( √ ϵ) and max Q e min Π Φ(Π α |Q e , Π) -max Q e min Π Φ( Π α |Q e , Π) ≤ O( √ ϵ)which completes the proof.In this approach, instead of using IQ-learn to predict opponent's next states, we employ a simple supervised learning approach. Specifically, we create a neural network of parameters δ: M i δ (o α i , a α i ), taking inputs as an observation and an action of allied agent i, and predict the next states of enemy agents in the observable area of agent i. The loss function can be defined as to minimize the MSE between the actual next enemy states and predicted ones, as follows:min    J Sup (δ) = ( S e,next i , o α i , a α i ,∀i)∼buffer i S e,next i -M i δ ( o α i , a α i ) 2    C.For a fair comparison between our proposed algorithm and existing methods, we use the same model architecture and hyperparameters as shown in Tables 2 and 3 respectively. We also use [34] , which used as an improvement method to speed up the training process. All sub-tasks (SMACv2, GRF, Miner) are trained concurrently in a GPU-accelerated HPC (High Performance Computing). Therefore, running times reported might not be accurate. In average, each sub-task requires 4-7 days of training depending on its difficulty. As a result, due to the limitation of our computing resources, we do not test our method's performance with other settings such as disabling Value Normalization, using MLP instead of Recurrent Neural Network, tuning learning rate, tuning clip range, etc.Regarding GRF tasks, there are 11 academy scenarios (depicted at https://github.com/ google-research/football/blob/master/gfootball/doc/scenarios.md).Although testing in all these sub-tasks is interesting, but we only evaluate on top three importance scenarios: academy_3_vs_1_with_keeper, academy_counterattack_easy, and academy_counterattack_hard. All are the hardest sub-tasks with the highest number of agents except the almost full football scenarios academy_single_goal_versus_lazy. We report the win-rate curves of our IMAX for the three tasks in Figure 4 . We do not show the performance curves of the other baselines methods (QMIX, QPLEX, and MAPPO) as they are not available in their papers, noting that the final win-rates of all the methods considered are already reported in the main paper. Finally, Figure 5 shows the win-rate curves on the Gold Miner tasks. 0 5M 10M 0 5M 10M 0 5M 10M 0 5M 10M 0 5M 10M 0.4 0.8 0 5M 10M 0 5M 10M 0 5M 10M 0 5M 10M 0 5M 10M 0.4 0.8 0 5M 10M 0 5M 10M 0 5M 10M 0 5M 10M MAPPO IPPO QMIX QPLEX SupMAPPO IMAX-PPO (GAIL) IMAX-PPO (InQ)In this section, we present an ablation study to evaluate the effectiveness of our IMAX framework when integrated with traditional multi-agent reinforcement learning algorithms. Specifically, we introduce a variant called QMIX-IMAX, which incorporates our IMAX framework into the conventional QMIX algorithm. We conducted experiments using the SMACv2 benchmark, focusing on three different factions: Protoss, Terran, and Zerg. For each faction, we tested two team configurations: 5_vs_5 and 10_vs_10.The performance of QMIX-IMAX was compared against the original, non-IMAX versions of popular algorithms such as MAPPO and QMIX. The results of these experiments are summarized in Table 3 , which illustrates the winning rates achieved by each algorithm across the different scenarios. Additionally, Figure 6 displays the learning curves, providing a visual representation of the performance dynamics over the training period.Our findings indicate that the QMIX-IMAX variant consistently outperforms the traditional QMIX algorithm across all tested scenarios. However, it does not achieve the same level of performance as our MAPPO-IMAX variant. This suggests that while the IMAX framework enhances the capabilities of QMIX, the integration with MAPPO yields superior results. These observations underscore the potential of the IMAX framework to improve baseline algorithms, offering a promising direction for future research in multi-agent reinforcement learning. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: [TODO]Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Environment dynamics are implicitly involved in sampling.These neighbors change over time depending on actions of all agents and the environment dynamics.All proofs are in the appendix.The win rate curves during the training for GRF and Gold Miner environments are in the appendix. 


Decision: All reviewers agree this is a good paper, on a relevant topic, with substantial contributions. There is nevertheless an opportunity to improve clarity and presentation, as raised in some concrete instances by multiple reviewers, for the camera-ready.

Reviews: review: paper_summary: paper_summary: The paper studies cooperative-competive MARL. It utilizes imitation learning to comprehend and anticipate the next actions of the opponent agents (enemies), aiming to mitigate uncertainties of the controlled agents (allies) with respect to the game dynamics.
strength_weakness: strength_weakness: - Related work needs improvement. In opponent modelling, the authors claim that: "All the aforementioned related works require having access to opponent’ observations and actions during training and/or execution". This is not the case in most opponent/agent modelling works (e.g., see Papoudakis et al.), as they model the other allies, not the enemies (which is allowed under the CTDE schema). I believe that the proposed method belongs to the category of opponent/agent modelling in MARL. Moreover, some important references are missing, see [1], [2], [3].
- The presentation needs improvement, heavy notation in many parts. The authors should remind more often what some quantities represent. Furthermore, more intuition of the proposed framework and the method is needed in sections 4 and 5 and the related work. In a nutshell, why can IL solve the agent modelling problem, why is it important and how? Also, how does the goal of the method (i.e., predicting the next states of the enemies) is connected to the IL objective of section 4.1? 
- Ablation study is missing. Also, it would be interesting if the authors provide experiments of the proposed method on top of other MARL algorithms as well. 
- The framework can be impractical as it is now, as it may need a lot of hardcoding to be implemented to any environment, since it leverages enemy state information within the individual (allies) agents' observations (e.g., information regarding the neighborhoods). In other words, one may need to be able to decompose manually the agents' observations into different parts, some of them are related to enemies.

[1] Papoudakis, Georgios, Filippos Christianos, and Stefano Albrecht. "Agent modelling under partial observability for deep reinforcement learning." Advances in Neural Information Processing Systems 34 (2021): 19210-19222.

[2] J. Sun, S. Chen, C. Zhang, Y. Ma, and J. Zhang, “Decision-making with speculative opponent 388 models,” IEEE Transactions on Neural Networks and Learning Systems, 2024.

[3] R. Raileanu, E. Denton, A. Szlam, and R. Fergus, “Modeling others using oneself in multi-agent reinforcement learning,” in International conference on machine learning. PMLR, 2018, pp. 4257–4266.
questions: questions: The questions have been intergrated into the weaknesses section.
limitations: limitations: Limitations of the work have been discussed in the weaknesses section. The authors also highlight some limitations in the conclusion.
paper_summary: paper_summary: This paper presents a new framework of multi-agent reinforcement learning (MARL) by modeling opponents’ behaviors through imitation learning.
strength_weakness: strength_weakness: It is assumed that each agent performs just based on the present observation o_i, but coordinated behaviors often require memory coding the group strategy or opponents’ game plan.
questions: questions: If I understand correctly, an important feature of the proposed framework is that all ally agents jointly learns a single joint model of the enemies. On the other hand, the SupMAPPO agents learn the enemies’ next states individually. What if the same information sharing is performed for supervised prediction of enemies’ next state?
limitations: limitations: It is assumed that opponents don’t learn.
paper_summary: paper_summary: This paper addresses the issue of training instability and slow convergence in MARL caused by the changing strategies of other agents. It proposes reducing the uncertainty faced during training by imitating the opponents' strategies. To address the challenge that opponents' actions are usually unobservable, the paper further proposes predicting the opponents' next state, providing corresponding derivation and analysis. Finally, the authors tested their approach in multiple experimental environments, achieving significant improvements in final rewards.
strength_weakness: strength_weakness: I have not found any obvious flaws. My concern lies in the experiments. I noticed that the training curves in the SMACv2 and Gold Miner environments are very unstable, i.e., the performance flucuates. Is this due to adversarial imitation learning? The stability of the algorithm needs further validation. Additionally, the default settings have fixed enemy strategies, making the learning easier. If the enemies are also learning agents, the instability may be exaggerated.
questions: questions: 1. The performance of many methods in the experiments fluctuate. How many seeds were used in the experiments? Previous experiments on SMACv2 usually uses more than 5 random seeds.
2. Are the strategies of the enemies fixed in the environment?
limitations: limitations: The training process may be unstable.

Comments: comment: We would like to thank the reviewer for reading our responses and for maintaining a positive view of our paper!
I would like to thank the authors for addressing my concerns and questions. I have increased my rating score.
We thank the reviewer for the prompt reply.

> I wonder whether IMAX-PPO framework needs carefully hyper-parameters tuning or the help of complex training tricks.

Thank you for the question. We would like to clarify that we did not conduct much hyper-parameter tuning for the IMAX-PPO. Instead, we adopted some default parameter settings from the MAPPO paper. For the IL component, we also utilized the  default parameters from the IQ-learn paper. We would also like to note that the inclusion of the IL component can make training more computationally expensive and more unstable compared to other baselines. However, we believe that the gains in the recovered policy are worth these trade-offs.

*We hope the above answers your question. If you have any further questions, we are happy to clarify them.*
Thanks for the reply. My concern on the stability issue has not been fully dispelled. Although I agree with the authors' response that the inclusion of more components may result in instability, I wonder whether IMAX-PPO framework needs carefully hyper-parameters tuning or the help of complex training tricks.