Efficient Multi-Agent Cooperation Learning through Teammate Lookahead
Authors: Feng Chen
Xinwei Chen
Rong-Jun Qin
Cong Guan
Lei Yuan
Zongzhang Zhang
Yang Yu
Venue: Accepted by TMLR ()

Abstract: Cooperative Multi-Agent Reinforcement Learning (MARL) is a rapidly growing research field that has achieved outstanding results across a variety of challenging cooperation tasks. However, existing MARL algorithms typically overlook the concurrent updates of teammate agents. An agent always learns from the data that it cooperates with one set of (current) teammates, but then practices with another set of (updated) teammates. This phenomenon, termed as ``teammate delay'', leads to a discrepancy between the agent's learning objective and the actual evaluation scenario, which can degrade learning stability and efficiency. 
In this paper, we tackle this challenge by introducing a lookahead strategy that enables agents to learn to cooperate with predicted future teammates, allowing the explicit awareness of concurrent teammate updates. This lookahead strategy is designed to seamlessly integrate with existing policy-gradient-based MARL methods, enhancing their performance without significant modifications to their underlying structures. The extensive experiments demonstrate the effectiveness of this approach, showing that the lookahead strategy can enhance the cooperation learning efficiency and achieve superior performance over the state-of-the-art MARL algorithms.

Published in Transactions on Machine Learning Research (03/2025)
Efficient Multi-Agent Cooperation Learning through Team- mate Lookahead
Feng Chen Xinwei Chen', Rongjun Qin Cong Guan', Lei Yuant.2, Zongzhang Zhang"; Yang Yu1.2
National Key Laboratory for Novel Software Technology, Nanjing University School of Artificial Intelligence, Nanjing University
Polioi Technologies
chenf@lamdo. edu.cn, on, zinwei, chen www.nnpoliiir..ttt [qinrj, guane, yuant] @lamda. njj.. edu. en, (zzzhang, yuu) @nnj)eeeeeduccr
Reviewed on OpenReview: https: // openreview. net/ forum? id= CeNNIQGGG
Abstract
Cooperative Multi-Agent Reinforcement Learning (MARL) is a rapidly growing research field that has achieved outstanding results across variety of challenging cooperation tasks. However, existing MARL algorithms typically overlook the concurrent updates of tcammate agents. An agent always learns from the data that it cooperates with one set of (current)) teammates, but then practices with another set of (updated) teammates. This phenomenon, termed as "teammate delay", loads to discrepancy between the agent's 's learning objective and the actual evaluation scenario, which can degrade learning stability and efficiency. In this paper, we lackle this challenge by introducing a lookahead strategy that enables agents to learn to cooperate with predicted future teammates, allowing the explicit awareness of concurrent tcammate updatos. This lookahcad strategy is designed to scamlessly integrate with existing bolicy-gradient-basee MARL methods, enhancing their performance without significant modifications to their underlying structures. The extensive experiments demon- strate the effectiveness of this approach, showing that the lookahead strategy enhances cooperation learning efficiency and achieves competitive performance compared to state-of- the-art MARL algorithms.
1 Introduction
Cooperative Multi-Agent Reinforcement Learning (MARL) techniques focus on replicating the collaborative intelligence observed huraan teams (Oroojlooy & Hajinevhad 2023), and advancements recent years have showcased its remarkable potential in various application domains, including robotics (Wang et all, 2022), games (Berner et al. 2019), and social networks (Lcibo et al. 2017). Among them, multi-agent policy gradient methods stand out with the capability to handle ionuu control lasks and with polential to solve intricate cooperative problems (de Witt et al.) 2020aa Yu et al. 2022a) Despite the ongoing progresses in this category of methods, we point out that they typically suffer from a "teammate delay" issue. Specifically, this issue occurs when an agent learns from the data that it cooperates with current teammates, but then practices with updated teammates due to the concurrent teammate updates. It. worth noting that the "teammate delay" phenomenon related to, yet distinct from, the general non-stationarity issue (Yuan al. 2023) in multi-agent systems. Whereas non-stationarity broadly refers to learning challenges arising from dynamic policy changes across all agents (including both teammates and opponents), the identified "teammate delay" specifically characterizes learning inefficiency in cooperative settings caused by agents neglecting tcammate
* *Corresponding Author

---

Published in Transactions on Machine Learning Research (03/2025)
Reward Vegetable Salad
Cook
Purchaser
Complete Incomplete
Cooking Pottern
Cook
Vegetable Salad
Cook
30
15
Hamburgers
Complete Incomplete
Policy Update
10
5
Cook hamburgers
Cook hamburgers
Figure 1: One simple example to show "teammate delay" issue, where Purchaser is expected to buy corre- sponding ingredients for Cook to cook.
policy updates. This problem is particularly pronounced in on-policy MARL algorithms, where agents optimize their policies based on outdated teammate policies.
For more intuitive illustration, one concrete example is shown in Figure [ In this example, Cook initially wants to cook hamburgers, and Purchaser adjusts its policy to buy corresponding ingredients after one round of policy update. However, at this moment, Cook also improves its policy to cook vegetable salad. Thereby, their updated policies fail to cooperate well. This simple example reveals that updating the agents to cooperate with current. teammates would lead to a training-test mismatch because the teammates update their policies as well. This gap between policy training and evaluation in each round of update can lead to severe learning inefficiency.
Although not explicitly pointing out this teammate delay issue, there exist works aiming to resolve similar problems arising from concurrent updates of teammate policies. Opponent modeling methods soek to alle- viate the non-stationarity in multi-agent scenarios through explicitly modeling the leammate policies (Yuan) They either introduce an auxiliary task of predicting teammate behaviors (Hernandez Leal
et al.
2023).
2019) or learn teammate representations as extra policy conditions (Papoudakis & Albrecht] 2020; ct al.
Cao et al. 2023). Despite their effectiveness in many problem scenarios, these methods necessitate ex- tra Leammate modeling efforts and lack theoretical analysis support. On the other hand, recent. works, LOLA [Foerster et al. 2018a) and COLA (Willi et al. 2022), explicitly acknowledge the learning behavior of other agents and propose learning rules with opponent- learning awareness. However, these works are limited to two-player simple problems and face challenges to exlend to practical cooperative scenarios. The most promising approach is one recent progress of multi-agent policy gradient method, HAPPO (Kuba et al. 2021). This method proposes a sequential policy update scheme with theoretical guarantees for joint policy improvement. However, its actual implementation involves an approximation utilizing importance sampling, potentially influencing the actual performance due to large variance in policy gradients.
Despite all these previous efforts, how the teammate delay issue influences the cooperative policy learning and how to better mitigate its negative impact are still open questions. To answer these two questions, in this paper, we both provide a formal analysis about the impact of this issue on the policy update, which motivates us to predict future teammate policics, and propose a model-based MARL algorithm where we approximate the future leammates via conducting policy updates within the environment model. In summary, our main contributions are:
We offer rigorous formal analysis policy-gradient MARL algorithms by investigating the regret of the updated policy, which unveils the impact of "teammate delay" issue on cooperative policy learning.
Furthermore, we introduce practical model-based MARL algorithm explicitly designed to address the challenges posed by the "teammate delay" issue. By leveraging insights from our formal analysis, our algorithm aims to enhance cooperative policy learning.
To validate the effectiveness of our proposed approach, we conduct empirical studics on various benchmarks. These benchmarks include complex problems with continuous action spaces, as well as challenging multi-agent cooperative tasks. The empirical results validate the effectiveness of our
2

---

Published in Transactions on Machine Learning Research (03/2025)
method, showcasing its ability to match exceed the performance of existing approaches across diverse scenarios with performance gains.
2 Preliminaries
In this section, will introduce some basic preliminaries concerning our problem formulation and analysis. We will firstly introduce these concepts in single-agent setting for brevity, and they can be simply extended to multi-agent setting.
In single-agent setting, the sequential decision-making problem can be modeled as a Markov decision process (MDP). can be defined as tuple (S, 7, Po), where S the state space, A the action space, P(-|s, a) means the transition function, R(s, a) is the reward function, 7 E (0, is the discounted factor, while Pu is the initial state distribution. The agent policy is defined as (alss) and the goal of Reinforcement Learning (RI.) is to maximize the expected discounted return:
( (a 1.0--)
Σ 7 ((s(
(1)
Morcover, for a given agent policy и, we can define the stationary state distribution at timestep t as dj (s; = 8). Then, the discounted state visitation distribution can be defined as ds(ss) = (1-?) Er= ddd (s); and the discounted state-action visitation distribution can be defined as d= (s,a) = (s) Based on these definitions, the expected discounted return of policy a can be rewritten as:
(TT) =
1
Σ (s) Σ x(als)R(s.a). SES
(2)
.EAA
For the sake of clarity in the subsequent derivations, we additionally define p=(s) = Σe (s) (d+)), which can be regarded the unnormalized version of d, (s). Then the expected return further oquals to nT(x)= = Eess (s( EaeA ((s,))((s, a). We define the agent policy that can maximizo 17(x)) ( as the optimal policy x", which means that 7/(=T) v()r for all Markovian agent policy
Moreover, for two different agent policy T' and n, their difference in discounted return is given by the famous performance discrepancy lemma:
7(T') - nT(T)= Σ (s)
Σ ('(ass)
a) = - V=(s,a),
(3)
SES
LEEA
Qx( = Ex
= s.00
R
V=(s)= Σ ((cls)Q=(s,a)
(4)
REA
where a) and Vs( are respectively the state-action value function and state value function.
For multi-agent setting, most definitions are similar. The main difference lies in the fact that there are N agents in the multi-agent setting, leading to the concepts of joint action space A and joint policy л. Accordingly, the discounted return and performance discrepancy lemma (Kakade & Langford 2002) become:
n(T) = =Ep(s
(s)(s) Σ
TT(a ((s,s)
(5)
aEA
n( - TT(T) = Σ (s)
Σ aaA
(aaaass) ((ss)))Arss
(6)
#ES
Notably, we focus on the fully cooperative setting, where the cnvironmental reward R(ss, is shared among all agents. Additionally, her in the single-agent or multi-agent setting, the problem formulation only considers Markovian policies, which means the agent policy makes decisions based on the state information at the current timestep. There some slight abuse of notation, as the subscript will also denote the i-th agent in subsequent derivations. For example, n, represents the policy of the i-th agent. More details about the notations listed in Appendix A.1
3

---

Published in Transactions on Machine Learning Research (03/2025)
2.1 Single-Agent Policy Gradient
In single-agent setting, the goal of reinforcement learning is to maximize the discounted return 77()) of the agent policy x, which according to the performance discrepancy lemma equals to:
(()) n(TTT) + Σ (s)
Σ AEA
x(als)),,
Ag*(s, = (xt(s,a) V=*(s),
(7)
#ES
where nh is the k-th round agent policy. As it hard to sample trajectorics corresponding the state distribution (s); in practice, we lypically use n& to sample trajectories instead. This implies that the actual learning objective is:
- (T)=-((T)) + Epae(s P
Σ (s, LaEA
(8)
#ES
In fact, J(x) is related to x² but for brevity we omit it in input. The same applies to the following context. Due to the state distribution change, can not be updated too far away from nh for which traditional actor- critic algorithms, e.g., A3C (Mnih et al. 2016). conduct only few policy gradient ascend while trust-region algorithms, such as PPO Schuuman et al. 2017), conforms to the trust-region optimization.
2.2 Multi-Agent Policy Gradient
When it comes to the multi-agent setting, the problem can be defined as a tuple AN,S,A,,,T 7: Po),, where N is the number of agents. It should be noted that in some scenarios, the agents need to make decisions based on local observations. For the sake of simplicity and without loss of generality, we have omilled the partial observability in derivation. Moreover, we assumee that the joint policy л can be decomposed into the product of individual policies ((a s) = Il= (a;[s), where the individual policy for agent and the joint action a [a1, a2, ... .axl= E A is decomposed of individual actions {a,)
In this case, the learning objective for cach agent multi-agent policy gradient methods is typically defined as:
Jz( (T,
= - Px)(ss)
(=(asss) Σ aEEA
(=((a)ls) iE (1, 5#1
(9)
#ES
More detailed derivation for this learning objective is provided in Appendix A.2 Compared with that in single-agent setting, the data distribution here is also influenced by the teammate policies. In other words, the i-th agent updates its policy associated with the current tearmates (mf in this learning objective. Overall, the learning objective for the joint policy can be formalized as:
N
J(T)) )7(*** + Σ (s)
Σ
(a(asss) (1),)))
(10)
N 1=1
SES
571
3 Method
In this work, we identify the teammate delay phenomenon in the common practice of multi-agent policy gradient methods. The direct negative impact of this issue can be analyzed and how to solve this issue deserves further study. In this section, we firstly discuss how the teammate delay issue can cause a negative impact on the cooperative policy learning through analyzing the regret of the updated joint policy at the next round. Motivated by this analysis, we then propose a practical algorithm that exploits the futurr teammate information to facilitate the cooperation learning.
4

---

Published in Transactions on Machine Learning Research (03/2025)
3.1 Analysis Motivates Predicting Future Teammates
From Equation [10), we know that in typical multi-agent policy gradient methods, the learning objective of the agent policy involves computing an expectation with respect to the current teammate policies {Tf h Consoquently, the current policy distribution of the tcammates will have an impact on the policy update. In order to provide further analysis on this impact, we replace the teammate policies with a general notation (1y)azti which means that the trajectories are sampled associated with a sampling policy p.. In this way, the learning objective is transformed into:
( m) n(TT') + P(()s)
E (a(asss Σ A=1 aEA
[(s(ayss)AA jti
(11)
SES
where n* still denotes the joint policy at the k-th round. In existing multi-agent policy-gradient methods, the sampling policy zi is typically selected to be n² which means that we expect n, to collaborate well with the k-th round teammate policies through maximizing Eoes Pmt (x)+(8) (aa)))(( [1 m5 (ay]s)A (s, a).
We wonder what would happen when we adjust from n* to other distributions. To answer this question, we firstly propose the following lemma that estimates the upper bound of discrepancy between the learning objective J(T,, μ)) and the actual policy return (T()).
Lemma 1 Assume that we update the joint policy Th to kh with sampling policy μ. Given the measurement of distance between sampling policy and the updated policy n as Os max, Drv (m. k+1 ((1))llu(((1s)) we have:
N
2
N 2c(N -
4ey
[J( ,,) 11 ((()+)))+
Σ
a
a4
(12)
(1 - 7)2
N
where y is the discount factor and maxx, |Ax as)
11,k+1,p0 For proof see Appendix A.2 The estimated upper bound of the discrepancy between J( (e) and (11)
in Lemma 1 can aid us analyzing the regret of xh+1 leading to the following theorem:
Corollary 1 Suppose that we update joint policy T to k+1 with sampling policy ,e, then the regret of the updated joint policy k+1 has the following upper bound:
2
4ey
2c(N 1) +
1i(T") - ( S <T(T")
J( 14) +
Σ
(13)
7)
N
1=1
For proof see Appendix A.2 The right hand side of Inequality 13 sheds light on the elements that can influence the cooperative policy learning. Totally, we expect minimize the regret of the updated policy via minimizing the overall upper bound expression. Other than the first term 17(=)) that constant value, the upper bound is composed of -J(x++11, 11) and one extra term (c). In fact, the second term -J(+++1 14) is exactly the loss function that the algorithm aims to minimize at each update round, which typically serves as a surrogate function for the regret πT(=") (+1). However, Inequality (13) reveals that the regret can not be bounded by -J( k+1 (1) alone, and an extra term (c) relatively caplures the gap between this surrogate function and the actual regret.
This extra term (c), function of the sampling policy i and the updated policy *+1 can not be optimizod by the previous learning algorithms, but it can have impact on the cooperation learning. When given large term (c), the surrogate function would be far from the actual regret, which can result in low learning efficiency. In fact, it is casy to observe that term (c) will be reduced to zero when μ is equivalent to π k+1
The TV distance measures the distance between two distributions via calculating Drv + Σ, (P(x) Q()) Cover
1999
5

---

Published in Transactions on Machine Learning Research (03/2025)
Algorithm 1 Multi-Agent Policy Gradient Learning with Lookahead
Input: The number of agent N, max iteration number K, trajectory batch size M Output: Obtained multi-agent cooperation policy
1: Initialize replay buffer B:
2: Initialize a joint policy π = {mi randomly;
for iteration k = to K do Sample batch of transitions from B and update the environment model by minimizing loss moddll:
Sample batch of trajectories {f)m in the environment model with sampling policy x² and obtain * = ((r",, x²) using the training trajectories (F)M;
6: Sample batch of trajectories {T)M in the real environment with sampling policy # and obtain
π k+I = (+++1, т²) using the training trajectorics {T]M;
7: Add trajectories T to the buffer B;
end for
That is, the extra term (c) would disappear if we trained the agents with the information of futuree teammates. This outcome motivates to replace the sampling policy μ with approximation of the future teammates, thus to reduce the regret. A more comprehensive analysis on how approximating future teammates reduces the regret upper bound is provided in Appendix A.3. In Section 3.2 and Section 3.3 we will show how we approximale the future teammates.
3.2 Future Teammate Approximation
Based on the above analysis, we are motivated to replace the sampling policy fe with the future teammate policy x+1 in each round of policy update. However, achieving this goal is not easy in practice, because in each round of policy update, the updated policy *h+1 is affected by the sampling policy μe that is, л x++1 and a are coupled. Thus, to serve this goal, we propose that the future leammate policy can be obtained by solving a bi-level optimization problem below:
Theorem Suppose the sampling policy μ can derive the same updated policy, which means that u arg max, n J( )²). If it exisls, it will the solution of the following bi-level optimization problem:
arg min (u]l7**+)) s.l. k+1 arg max J (T)
(14)
Its proof can be found in Appendix A.2 This theorem inspires that can obtain the expected μ² by solving the consistent bi-level optimization problem. The solution of this problem to some extent contains the information of future teammate policy. However, this problem typically follows a form of Stackelberg Game (Friedman 1971 and is not easy to solve.
In this casc, we propose to perform one-step approximation of this optimization problem, which means that with z initialize as n², we firstly solve the inmer-loop optimization with x²-1= arg max [Tt,, 14) and then assign the obtained x+1 to the sampling policy μ1, thus obtaining the approximation of the solution n. This one-step approximation is commonly utilized for stackelberg-game- a trade-off between the solution accuracy and the computation cost. In brief, for feasible future teammate approximation, we perform an additional round of optimization before each algorithm iteration, using the previous round's policy r² as the sampling policy, and the obtained policy # serves as the approximation of future teammate policy.
problems, and it achieves
3.3 Practical Algorithm Implementation
Model-based Approximation The above analysis motivates us to conduct extra training to estimate the future teammate policy. However, straight-forward implementation not practical because it wastes near half of the online samples for estimating the future teammate policy, and those samples are not utilized for the actual policy training, which result will lead to very low sample efficiency of the algorithm. To

---

Published in Transactions on Machine Learning Research (03/2025)
Multivaratee Gaussan Distaibution
r
r
Gaussin Distriution
Concatomation
0
MLL Network
OOO
S a
a
agent
Model rchitectuee MDP Environment
Model rchitecture POM Environment
Figure 2: Network architectures for environment model in practical implementation.
avoid this issuc, we propose to learn an environment model, and put the teammate policy estimation process within it, thus avoiding the waste of a large number online samples.
In practice we build the environment model utilizing multiple Multi-Layer Perceptrons (MLPs), as shown in Figure 2 It takes the environment state s, joint action of next state s' and reward signal Specifically, the next state is modeled as multivariate Gaus- sian distribution s' (s,,(s, al0), (, al0)), and the reward is modeled as univariate Gaussian -~ NN, a)),, (s, al0)²). To train the environment model, we construct a replay buffer B and store the online samples into it throughout the training process. At each iteration, we will sample data batches from buffer B to minimize the following negative logelikelihood of the true data:
imput, and predicts the distributions
Lmodel (0) Lirans()) Creward
(15)
1 ((s' - - '
Ltrans (0) (s,,,,,' |- logp(s' Ma. E.)] = E(s.a,s'))
Us) + log [2:1]]
(16)
2
1 (r (r- m,,) + log oz )
Creward (f) ((s,,,'))) [- log (ogg(())mr, [f]] (s.a,,''))
(17)
o²
where Mss Es, Mr: o, are short for va(s, (s, a (0) Mr (8, a (0), as, al0) that are the predictions of envi- ronment model fa.
Moreover, considering that many multi-agent scenarios are partially observable, where agents can only ob- serve a portion of the environment's state, for such POMDP environments, we additionally introduce one projection network he that predicts individual observations o; from the global state s. Thus, for these POMDP cases, the training loss includes one additional term of observation prediction:
Lmodel (8, 6) Liraus (0) + Creward (0) + Eprojection (ø).
(18)
Eprojection (Ø) = (ss. AB
Σ log lo) En Ex,,,
(19)
where (s olo), (s, are the outputs of the projection network hig-
Off-policy Value Estimation The intuition of our work is to modify the sampling policy μ in Equa- tion 11), thus to derive better optimization objective that can bring smaller upper bound of the regret for the updated policy. However, term Ayt is expected be maintained which means that we want estimate the advantage with the policy of the last round. For previous methods, it easy to achieve because
7

---

Published in Transactions on Machine Learning Research (03/2025)
Optimal Goal
1.0
3.905
Lookahcad
3.256
Start Point
MAPPO
0.8
2.607
1.958
0.6
1.309
Lookahead
ppruximation
Agent I's Best-Responsee (BR)
0.660
0.4
policy-eurve agaimst Agent
0.011
0.638
Score
0.2
Poliey adjustment based lookahead
24
Leokahead
1.287
1-
MAPPO
0.0
1.935
Optimal
0.0
0.2
0.4
0.6
0.8
1.0
a1
Number of samples
(b)
(c)
Visualization results on the toy environment. (a) Landscape of the two-variable function, where darker the color is, higher function score the region obtains; (b) Algorithm learning score curve; (c) Opti- mization process Lookahead and MAPPO.
Figure 3:
the trajectories are sampled by nh and we can estimate the advantage directly. While in our algorithm de- sign, the sampling policy is replaced with the lookahead policy # which means that we need conduct off-policy estimation for Azt. Specifically, adopt the V-trace (Espeholt. et al. 2018 trick estimate A, with the trajectories sampled by *h+1
Overall Flow of the Algorithm Combining all the algorithmic design techniques that have raised, propose practical algorithm that can enhance the underlying multi-agent policy gradient method. The overall flow of our algorithm has been presented in Algorithm | In line 5, we obtain the estimated future teammate policy * within the environment. model, while in line 6 we utilize # to aid in actually updating the policy in the real environment. Besides, we update the environment model in line
4 Experiments
In this section, we substantiate the efficacy our proposed approach through empirical validation via experiments conducted on diverse benchmarks. These benchmarks include a toy environment, which serves illustrate the algorithmic process of our approach, and two intricate cooperative multi-agent scenarios that provide practical validation of our approach's effectiveness. Specifically, results to investigate the following questions: 1) How does algorithm work and
aim to utilize these experimental
analyze the underlying mechanism through one simple task (Section 4.1 2) Can our algorithm actually enhance the cooperation learning in complex multi-agent cooperative tasks (Section 4.2 ? 3) Does the phenomenon
exhibited by our algorithm in complex cooperative tasks still align with our analysis (Section 4.3
4.1 Algorithm Analysis in Toy Environment
To visually reveal how our method works, we devised a toy environment involving a two-variable function optimization problem. As depicted in Figure a), this problem comprises two agents with continuous action spaces in the range [0, 1]. Whenever the agents execute a joint action [ag, 01], the environment yields reward:
R ag 2(a0 an )2 3ag-

---

Published in Transactions on Machine Learning Research (03/2025)
jounts divided into aggents
(a) 2x3-Agemt HaltCheetah
academy vs with Keeper
(c) MMM2
Figure 4:
Some example task scenarios of Multi-Agent MuJoCo (MA-MuJoCo), Google Football Rescarch
(GRF), and StarCraft Multi-Agenl Challenge (SMAC) environments.
Evaluation results of various methods on MA-MuJoCo tasks, providing average scores across 5 seeds with standard errors. The highest score for each task scenario bolded and the top-2 scores are marked in blue The average rank denotes the average ranking across all task scenarios of each method.
Table 1:
Algrrithm
2x4
4x2
HalfChhetah
HallCheotah 3x2
Walker2d
Walker2d 3x2
Average
Lookabead
3393.39(336 (336 98)
2858. (22
3315.83(346-10 46)
3687.10 1997.101200 301.90 98)
20-18. 30/30 27)
(1(133774)
1.33
2471.15/201.00
18(168-48)
2910.18 (839.18(99388)
3016.20(50.2))) (80030))
25444.6(2272.0)
2780.65(76.22)
2.00
TAPPO
2055.21 (18.25)
1550366((207.c6 .66)
2154.73(443.24
(447752(7115) 1.66)
(11527)))))
1445.56(350.34)
MАPPO
1034.19(18.98)
1002. 15(18.30)
(560029/003.777 76)
(350032(447..22
1852 8(41.8)))))8)
1812.54(139444
4.33
884.93(46.24)
875.8(20.84)
(335.0)
2477.98(100.12)
2021.89(07.66)
1775.2(122.08)
4.33
MADDPG
1866:08(9:77) (131.08(13.45)
1558.54(251.6)
1295.5(384.87)
71.68(15:36)
100.72(35.19)
5.33
Specifically, initialize the joint policy [0.1, and both adopt the algorithms of MAPPO and lookahcad strategy to investigate how they converge to the optimal policy [1 As shown in Figure 3 without our lookahead strategy, agent 1 always shows a large gap from the Best-Response (BR) against the updated agent O, revealing the phenomenon of "teammate delay". While our lookahead strategy can help agent predict the updated policy of agent 0, leading to a shorter optimization path. From the learning curve in Figure ()), we also find that our lookahead strategy helps converge the optimal policy using much fewer samples, enhancing the learning efficiency.
c).
The algorithm analysis in this straightforward objective optimization task helps provide intuitive expla- nation about the algorithm mechanism and motivation behind our approach. In the subsequent sections, we explore whether the proposed lookahead strategy can indeed enhance the cooperative learning in more complex task scenarios.
4.2
Main Results in Complex Cooperative Tasks
4.2.1
Experiment Setup
To investigate the effectiveness of our approach in more practical task scenarios, this section focuses several prevalent cooperative benchmark environments, including continuous control tasks from Multi- AAetii MuJoCo (MA-MuJoCo) fde Witt et al.) 20206) and Google Research Football (GRF) (Kurach et al. 2020 games with discrete action spaces. Besides, we also include experiments on StarCraft Multi-Agent Challenge (SMAC), which enables complex tasks involving larger team sizes. Examples of task scenarios from these three environments are illustrated in Figure The introduction to each environment is provided below.
The MA-MuJoCo environment is built upon the MuJoCo physics engine to create realistic simulations for MARL research. In specifie, MA-MuJoCo partitions the
Multi-Agent MuJoCo (MA-MuJoCo)
9

---

Published in Transactions on Machine Learning Research (03/2025)
Table 2: Evaluation results of various methods on GRF tasks. GRF 3vs1, CA(hard) and Corner are respec- tively short for maps of academy 3 1 with keeper, academy counterattack hard and academy corner GRF environment.
Algorithm
GRF 3vs1
GRF CA (hard)
GRF Corner
Average Rank
Lookahead
0.82(0.02)
0.50(0.07)
0.60))
1.33
HАPPO
0.86()
0.46(0.09)
(.49().07)
2.00
ТАPPO
0.77(0.03)
(.49().03)
0.28(0.10)
3.00
MAPPO
0.66(0.04)
(.37().08)
0.48(0.11)
3.67
CDS
0.49(0.11)
0.21(0.07)
0.02(0.01)
5.00
Table 3: Evaluation results of various methods on SMAC tasks, providing average scores across 5 seeds with standard errors.
Algorithu
3s5z
5m vs.
6m
MMM2
Average Rank
LookAhead
(2.2().31)
90.5(2.51)
90.6(1.04)
1.00
НАРРО
92.2 (1..4
87.5(2.04)
87.1(2.20)
4.00
TAPPO
(5.3().60)
89.1(2.51)
88.2(2.90)
3.33
MАPPO 91.9(1.42)
88.2(2.35)
89. (2.79)
2.67
IPPO
91.9(2.51)
87.5(2.93)
87.5(2.51)
3.67
QMIX
88.3(2.90)
75.8(3.70)
87.5(2.60)
5.33
body graph in MuJoCo into disjoint sub-graphs, one for each agent, e.g., 2x4-Agent Ant means dividing the joints in Ant into 2 agents, each controlling joints.
Google Rescarch Football (GRF) The Google Research Football (GRF) novel benchmark environ- ment offering simulations of soccer matches, enabling the study of multi-agent behaviors and reinforcement learning. It introduces challenging cooperation learning tasks it has the property of heterogeneity and sparse rewards.
StarCraft Multi-Agent Challenge (SMAC) The SMAC environment is also popular benchmark for MARL, set in micromanagement scenarios with multi-agent and multi-unit battles. The goal is for the ally units to cooperate together to debate the enemy units. SMAC allows for the design of complex and challenging cooperative tasks due to the diversity of unit types and maps.
To thoroughly explore the cooperative performance that our approach can potentially bring about, we integrate our lookahead strategy with HAPPO, one of the current state-oftthe-art multi-agent policy gradient algorithms, in all experiments of this section. For comparison, critic algorithms baselines. An opponent modeling approach, TAPPO, is also included, which learns teammate representations to incorporate additional policy conditions like in previous methods (Papoudakis & Albrechl 2020 Cao et al. 2023). We adopt this baseline lo rast our approach with traditional opponent modeling approaches in mitigating non-stationarity issue arising from teammate co-learning. Moreover, in the GRF environment, we add one additional bascline CDS (Li ct al. 2021), a valuo-based MARL algorithm designed specifically for solving GRF games, for a more comprehensive comparison. More details about baselines be found in Appendix B.
select several popular multi-agent actor-
10

---

Published in Transactions on Machine Learning Research (03/2025)
4.2.2 Results Analysis
MA-MuJoCo As shown in Table [ in multiple task scenarios of MA-MuJJCo, our approach Lookahead has achieved superior cooperative performance compared to other baseline algorithms. For tasks of Ant and HalfCheetah, our approach has consistently achieved the highest scores across all methods. These results imply that in these task scenarios, through predicting the potential future policies of other agents controlling their respective joints, our algorithm can help agents learn to manipulate their own joints in coordination with other agents better, finally enhancing the cooperative performance. Despite not achieving the highest score, our approach also attains top-2 performance in the Walker2d scenarios. We hypothesize that the slight performance loss might be due to the nature of the Walker2d task, which requires not only proficient walking but also maintaining the balance of the mechanical legs at all times. The possible failure to maintain the legs' balance may pose great challenges to the learning of the environment model, consoquently having a negative impact on the performance of our algorithm. Actually, the selection of environment model learning methods is orthogonal to our algorithm. In the future, we will consider designing better model learning methods to further enhance the performance of our approach.
Google Research Football (GRF) Similar to the results in MA-MuJoCo, the results in Table 2 demon- strate that our approach also achieves superior performance in GRF problems. In particular, our approach is the only one that attains a success rale exceeding 60% in the academy corner scenario, achieving a no- table improvement of over 25% compared to the second-best algorithm. Moreover, the average rank of our approach also stands out as the best in the GRF environment, the same as that in MA-MuJoCo.
StraCraft Multi-Agent Challenge (SMAC) The results in Table 3 highlight the win rate performance of different approaches in SMAC, where our approach consistently achieves the best performance across three different task scenarios. These results, especially those on MMM2 task which involves ten ally agents, demon- strate the ability of our approach to adapt to more complex cooperative tasks, supporting the generality and adaptability of our approach.
Ablation Study The comparison in Tables D to between the Lookahead and HAPPO algorithms on both benchmark environments can be seen as ablation study to assess the impact of our introduced lookahead strategy. Across the majority of task scenarios, the incorporation of our lookahead strategy results in enhanced cooperative performance compared to the original HAPPO algorithm, e.g., exceeding the second- best algorithm by around 1000 points in 2x4-Agent Ant, which effectively validates the efficacy of the proposed lookahead strategy.
4.3 Analysis of Lookahead Policy in Complex Problems
While we have analyzed the algorithmic mechanism in a toy environment, applying the algorithm in complex tasks more intricate due to the involvement of model learning. In this section, we measure the policy distances to investigate whether our lookahead approximation still provides right direction information in MA-MuJoCo. In specific, we compute the difference between DKL ( x²) and (TI( **+*), which equals to DKL(m*+1 π²* DKL( 7 I*.)). As can see from Figure 5 that this metric consistently keep positive throughout the training process, the results reveal that # is actually closer to the future teammate policy *+1, forming a relatively good approximation. These results support our motivation and more analytical experimental results can be found in Appendix C
5 Related Work
The related work of this paper mainly covers three aspects: multi-agent policy gradient, multi-agent model learning, and opponent modeling. Below, we provide the introduction to related works in these three aspects respectively. Among them, opponent modeling approaches are typically proposed to address the general non- stationarity issue in multi-agent systems, considering the dynamic policy changes of both tcammates and opponents. While related, our identified "teammate delay" problem is particularly pronounced in on-policy MARL algorithms for cooperative settings, which is issue more related to algorithm design.
11

---

Published in Transactions on Machine Learning Research (03/2025)
Ant
HalfCheetah
KL Distance Difference
0.60
KL Distance Difference
0.060
2x4-Agent Ant
Ax3-Atent HalfChoctah
0.45
4x2-Agent Ant
0.045
3x2-Agent HalfChectah
0.30
0.030
0.15
0.015
0.00
0.000
0.0M
1.0M
2.0M
3.0M 4.0M 5.0M Timesteps
0.0M
1.0M
2.0M 3.0M 4.0M 5.0M Timesteps
Figure 5: Measuring policy distances in Multi-Agent MuJoCo environments. The y-axis calculates the difference in distances from तb and # to xh+1, i.c., DKL(T**1, "x)++,"x*)_)_))_ - DKL ( ***1), and the plot illustrates the changes in this metric throughout the policy training process.
Multi-Agent Policy Gradient The multi-agent policy gradient algorithms hold better convergence sta- bility compared to the value-based algorithms, and they provide the ability to handle continuous control tasks. IA2C (Chu et al. 2019) introduces the A2C method to the multi-agent sctting, and adopts an independent learning paradigm. Subsequently, COMA (Foerster el al. 2018bb proposes the paradigm of centralized critic with decentralized actors, which tries to conduct credit- assigmmettttt for each agent via introducing a counter- factual baseline. MAAC (Igbal & Sha 2019) and DOP (Wang al.) 2020) respectively improve the policy gradient methods by introducing the attention mechanism to the critic network and conducting value decom- position for the centralized critic. On the other hand, IPPO (de Witt et al.) 2020a and MAPPO (Yu et al. 2022a) extend the trust-region policy optimization scheme to the multi-agent setting, and obtain remarkable performance. However, all the policy gradient methods above directly optimize the agent policy associated with the current teammates, and may suffer from the "teammate delay" issue. Recently, HAPPO (Kuba) ett al. 2021) introduces sequential update scheme to the multi-agent policy gradient algorithm, which con- siders the mutual influences between different agents' policy update. Nevertheless, it adopts importance sampling technique which suffers from high variance, and it is orthogonal to algorithm. More discussion is provided in Appendix B.2 Besides, there exist algorithms introducing deterministic policy gradient to the multi-agent setting [Lwwe et al. 2017), while we mainly consider stochastic policy gradient methods here.
Aglti-Agent Model Learning Model-based reinforcement learning enjoys higher sample efficiency. How- ever, multi-agent model learning faces significant challenges due the exponential growth of the state-action space and the non-stationary in multi-agent scenarios. Adopting the Dreamer (Hafner et al.
2019 architec-
ture, MAMBA [Egorov& & Shpilman 2022) sustains a world model for each agent with necessary communica- tion, thus to scale gracefully with the number of agents. Another work (Mahajan et al. 2021] shows utilizing tensor decomposition in multi-agent model learning can significantly improve the sample efficiency when the enviromment transition and reward functions are of low CP-rank. Krupnik (Krupník et al.) 2020) adopts generative models to learn a multi-step world model which can consider the delayed effects of the previous actions. Besides, considering the characteristics of multi-agent settings, AORPO (Zhang et al., 2021] and CTRL (Par et al. 2019) incorporate the opponent modeling into the model learning in order to roll-out opponent-wise trajectories. When the environment model has been obtained, dyna-style algorithms (Zhang 2022: Willemsen et al. 2021) conduct data augmentation to enhance policy lcarning. MBVD (Xu
ct al.
2022 evaluates the current state value via imagining future stales within the model. Han (Han et al., conducts credit assignment by computing the shapley value (Winter) (002) using the samples rolled out in the model. In the future, a promising direction is to use generative world models for multi-agent environment modeling, enhancing the learning of multi-agent policies, these generative world models have
el al.
2022
already shown considerable potential (Hu et al.] 2023
Bruce al.] 2024 Qiao al. 2024
12

---

Published in Transactions on Machine Learning Research (03/2025)
Opponent Modeling Opponent modeling is well-studied topic in the field of MARL. Some previ- ous works utilize opponent modeling to alleviate the non-stationariit issue in MARL. Among them, some works (Hong et al. 2018 Papoudakis & Albrecht 2020; Xie et al.
2021 Cao et al. 2023) involve utiliz- ing the opponent representations as additional imputs to the policy network, thereby enhancing the policy learning. While AMS-A3C and AFS-A3C [Hernandez-Leal et al.) 2019) treat the opponent modeling as an auxiliary task to guide the network optimization. Besides, another series of works assume that opponents are uncertain may change, and they aim to help recognize and adapt to the opponents. DPN-BPR+ (Zheng et al. 2018) and MBOM (Yu et al.) 2022b) estimate the most probable types of opponents from statistical perspective, while Fastap (Zhang et al. 2023 further considers that the changes of teammates may happen within one episode and learn a instantaneous representation to achieve fast recognition of teammate changes. Moreover, there exist other series of works (Foerster et al., 2018a; Willi et al. 2022 Lu et al. 2022) that propose a better update operator for general-sum games by modeling the influences agents' policies on the other agent. However, these methods are limited two-player simple problems, while our work focuses complex cooperative tasks.
6 Closing Remarks
This paper introduces a pioneering approach to enhance cooperative MARL by anticipating future teammate policies. Alleviating the prevalent issue of "teammate delay", our proposed lookahead strategy bridges the gap between the learning objective and the real evaluation scenario, significantly boosting the lcarning cffi- ciency. Through seamless integration with existing gradient-based MARL methods, our approach surpasses state-of-the-art algorithms, exhibiting good performance in complex cooperative multi-agent benchmarks. Currently, our method mainly relics on the environment model to predict the future teammates. Thus, the practical algorithu performance to some exlent limited by the model learning error. How to better esti- mate the future teammates and whether there exist other ways to harness the predicted information of future teammates deserve further investigation. We believe researches in this topic can bring great advancement in the MARL domain.
Acknowledgments
This work is supported by the National Science Foundation of China (62276126, 62495093, 62250069) and the Natural Science Foundation of Jiangsu (BK2024119, BK20243039).
13

---

Published in Transactions on Machine Learning Research (03/2025)
References
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemys"law D'kebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. Xiv preprind arXiv:191 2. 06680, 2019.
Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In International Conference on Machine Learning, pp. 4603 4623, 2024.
Jiahan Cao, Lei Yuan, Jianhao Wang, Shaowei Zhang, Chongjie Zhang, Yang Yu, and De-Chuan Zhan. Tinda: Multi-agent local information decomposition for awareness of teammates. Science China Information Sciences, 66(8):1 17, 2023.
Tianshu Chu, Jie Wang, Lara Codecà, and Zhaojian Li. Multi-agent deep reinforcement Icarning for large- scale traflic signal control. IEEE Transactions on Intelligent Transporlation Systems, 21(3):1086 1005- 2019.
Thomas M Cover. Flements of information theory. John Wiley Sons, 1999.
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXic preprint Xiw:2011.00558 2020a.
Christian Schroeder de Witt, Bei Peng, Pierre-Alxxanrre Kamienny, Philip Torr, Wendelin Böhmer, and Shi- mon Whiteson. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. arXiv preprinl ar Xic:2000000000 06709, 19, 2020b.
Vladimir Egorov and Alexei Shpilman. Scalable multi-agent model-based reinforcement learning. In Inter- national Conference on Aulonomous Agents and Multingent. Systems, pp. 381-390, 2022.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Tain Dumning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference Machine Learning, pp. 1407 1416, 2018.
Jakob Focrster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Picter Abboel, and Igor Mordatch. Learning with opponent-learning awareness. In International Conference on Autonomous Agents and Mullingent Systems, pp. 122 130, 2018a.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Coun- terfactual multi-agent, policy gradients. In AAAT Conference on Artificial Intelligence, pp. 2974-2909 2982, 2018b.
James W Friedman. non-cooperative equilibrium for supergames. The Review of Economic Studies, 38 (1):1 12, 1971.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2019.
Dongge Han, Chris Xiaoxuan Lu, Tomasz Michalak, and Michael Wooldridge. Multiagent model-based credit assignment for continuous control. In International Conference on Autonomous Agents and Multiagent Systems, pp. 571-579, 2022.
Pablo Hernandez-Lent Bilal Kartal, and Matthew E Taylor. Agent modeling as auxiliary task for deep rein- forcement learning. In AAAT Conference Artificial Intelligence and Interactive Digital Entertainment, pp. 31 37, 2019.
Zhang-Wa Wei Hong, Shih- Shih-Yang Su, Tzu-Yun Shann, Yi-Hsiang Chang, and Chun-Yi Yi Lec. A deep policy inference Q-network for multi-agent systems. In International Conference on Autonomous Agents and Multingent Systems, pp. 1388-13888 1396, 2018.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoscev, Alex Kendall, Jamic Shotton, and Gianluca. Corrado. Gaia-l: A generative world model for autonomous driving. or Xiv preprinl arXiv:2309.1003 2023.
Shariq Tqbal and Fei Sha. Actor-attention-critie for multi-agent reinforcement learning. Tn International Conference Machine Learning, pp. 2961 2970, 2019.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Interna- tional Conference on Machine Learning, pp. 267 274, 2002.
Orr Krupnik, Igor Mordatch, and Aviv Tamar. Multi-agent reinforcement learning with multi-step generative models. In Conference on Robol Learning, pp. 776 790, 2020.
Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang, Trust. region policy optimisation in multi-agent reinforcement learning. In International Conference on Learning Representations, 2021.
Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, and Sylvain Gelly. Google research football: A novel reinforcement learning environment. In AAAI Conference on Artificial Intelligence, pp. 4501 4510, 2020.
Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent reinforce- ment. learning in sequential social dilemmas. In International Conference on Autonomous Agents and Multiogent Systems, pp. 464 0644473,45 2017.
Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang, Celebrating diversity in shared multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 3991 4002, 2021.
Ryan Lowe, Yil I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-compeliive environments. In Advances in Neural Information Processing Systems, pp. 6379-6390, 2017.
Christopher Lu, Timon Willi, Christian A Schroeder De Witt, and Jakob Foerster. Model-free opponent shaping. In International Conference on Machine Learning, pp. 14398 14411, 2022.
Anuj Mahajan, Mikayel Samvelyan, Lei Mao, Viktor Makoviychuk, Animesh Garg, Jean Kossaifi, Shimon Whiteson, Yuke Zhu, and Animashree Anandkumar. Tesseract: Tensorised actors for multi-agent rein- forcement learning. In International Conference Machine Learning, pp. 7301-7312, 2021.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukeuoglu. Asynchronous methods for deep reinforcement learning. In Inter- national Conference on Machine Learning, pp. 1928-1937, 2016.
Afshin Oroojlooy and Davood Hajinezhad, A review of cooperative multi-agent deep reinforcement learning. Applied Intelligence, 53(11))13677-13722 2023.
Georgios Papoudakis and Stefano V Albrecht. Variational autoencoders for opponent modeling in multi-agent systems. In AAAI Workshop on Reinforcement Learning in Games, 2020.
Young Joon Park, Yoon Sang Cho, and Seoung Bum Kim. Multi-agent reinforcement learning with approx- imate model learning for competitive games. PloS one, 14(9):c0222215, 2019.
Shuofei Qiao, Runnan Fang, Ningyu Zhang, Yuqi Zhu, Xiang Chen, Shumin Deng, Yong Jiang, Pengjun Xic, Fei Huang, and Huajun Chen. Agent planning with world knowledge model. arXiv preprint arXiv:2405.1420 14205, 2024.
John Schulman, Sergey Levine, Picter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. Tn International Conference on Machine Learning, pp. 1899-1897, 2015.
15

---

Published in Transactions on Machine Learning Research (03/2025)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. Xir preprint arxi: NN7:1707.03477. 2017.
Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. DOP: Off-policy multi-agent decomposed policy gradients. In International Conference on Learning Representations, 2020.
Yutong Wang, Mehul Damani, Pamela Wang, Yuhong Cao, and Guillaume Sartoretti. Distributed reinforce- ment learning for robot teams: A review. Current Robotics Reports, 3(4):239 257, 2022.
Daniël Willemsen, Mario Coppola, and Guido CHE de Croon. Mambpo: Sample-efficient multi-robot re-
inforcement learning using learned world models. In IEEE/RSJ International Conference on Intelligent Robols and Systems, pp. 5635 5640, 2021.
Timon Willi, Alistair Hp Letcher, Johannes Treutlein, and Jakob Foerster. COLA: Consistent learning with opponent-learning awareness. In International Conference on Machine Learning, pp.
2022.
Eyal Winter. The shapley value. Handbook of Game Theory with Economic Applications, 3:2025 2054, 2002.
Annie Xie, Dylan Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent representations to 2021.
influence multi-agent interaction. In Conference on Robol. Learning, pp-
Zhiwei Xu, Dapeng Li, Bin Zhang, Yuan Zhan, Yumpeng Bai, and Guoliang Fan. Mingling foresight with imagination: Model-based cooperative multi-agent reinforcement learning. In Advances in Neural Infor- mation Processing Systems, pp. 11327-11340, 2022.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. In Advances in Newral Information Processing Systems, pp. 24611-24624, 2022a.
Xiaopeng Yu, Jiechuan Jiang, Wanpeng Zhang, Haobin Jiang, and Zongqing Lu. Model-based opponent modeling. In Advances in Neural Information Processing Systems, pp. 28208 28221, 2022b.
Lei Yuan, Ziqian Zhang, Lihe Li, Cong Guan, and Yang Yu. A survey of progress on cooperative multi-agent reinforcement learning in open environment. arXiv preprint or arvi::2312 2tt::331200000 01058, 2023.
Qizhen Zhang, Chris Lu, Animesh Garg, and Jakob Foerster. Centralized model and exploration policy for multi-agent RL. In International Conference on Autonomous Agents and Multiagent Systems, pp. 1500-1508, 2022.
Weinan Zhang, Xihuai Wang, Jian Shen, and Ming Zhou. Model-based multi-agent policy optimization with adaptive opponent-wise rollouts. In International Joint Conference Artificial Intelligence, pp. 3384-3391, 2021.
Ziqian Zhang, Lei Yuan, Lihe Li, Ke Xue, Chengxing Jía, Cong Guan, Chao Qian, and Yang Yu. Fast teammate adaptation in the presence of sudden policy change. arXiv preprint ar arXiv:2305.0500 2023.
Yan Zheng, Zhaopeng Meng, Jianye Hao, Zongzhang Zhang, Tianpei Yang, and Changjie Fan. A deep Bayesian policy reuse approach against non-stationary agents. In Advances Neural Information Pro- cessing Systems, pp. 960 970, 2018.
16

---

Published in Transactions on Machine Learning Research (03/2025)
Appendix
A Notations and Theoretical Analysis
A.1 Notations
In Table 4
we list the main notations in our paper.
Table 4: Notation list.
Symbol
Mcaning
S,s
S denotes the state space for cither the single-agent problem multi-agent problem, while S is an instance of the state.
A,a
A denotes the action space for the single-agent problem, while a is an instance of the action.
N
Number of the agents in multi-agent problems.
A, (A.))
A is the joint action space for the multi-agent problem, A, is the action space for agent
a, {a
a [01, a2,,02,-.- a.. .an] is an instance of the joint action, where is the action for agent i.
P
Transition function for either the single-agent problem or multi-agent problem.
R
Reward function for either the single-agent problem or multi- agent problem.
?
Discount factor. π denotes the policy for single-agent problems, where (a]s) means the probability of taking action under state s. т denotes the joint policy for multi-agent problems, while H; indicates the policy for agent x(als) = ( (a,lss)) means the probability of taking action a under state 8. The obtained joint policy after the k-th round of policy up- date.
n, (.TT 1
*
The discounted return of joint policy æ in multi-agent prob- lems, i.e., 7)()) = Ex 1
ds), a))
The stationary state visitation distribution and state-action visitation distribution when given the fixed joint policy æ.
(s)
The unnormalized stationary state distribution derived by the joint policy л. Specifically, Px (*(s) d=)d(s))
(5,4)) 0s a)
Q (8, a) represents the Q-function in single-agent problems, defined as the expected cumulattve reward obtained by tak- ing action a in state and then following policy x thereafter, i.e., Qx*(s, for multi-agent. problems. Vx (s) represents the state value function in single-agent problems, indicating the expected cumulative reward start- ing from state a and following policy thereafter, i.e., V (s) = Ext IE2o r'ra|30 = s]. Va+ (s) for multi-agent problems.
E, 1 7 s, 00 a). Qu (s, a) is
(s), (s)
a, = max, Dry (I/ (-1s))l (-(ss)) is utilized to denote the distance between the sampling policy and the updated policy (the policy at the next round).
17

---

Published in Transactions on Machine Learning Research (03/2025)
A.2 Proofs of Main Theoretical Results
In this section, we provide the proofs of the main theoretical results in our paper. In specific, we begin by outlining the primary theoretical results below, followed by their respective proofs one-by-one, Among them, Lemma 1, Theorem 1, and Theorem 2 are introduced in the main text, while Theorem is introduced the supplementary discussion in Appendix A.3
Statement In previous multi- ageti policy gradient methods, the learning objective for each agent in multi- agent policy gradient methods is typically defined
(ATTlaa5) n((TT) + Σ
Px
Σ aEA
(ails) (I iE (1,2, N..
(20)
SES
iti
Lemma 1 Assume that update the joint policy x² to nhl with sampling policy μ. the measurement of distance between sampling policy and the updated policy th+ a; max, Drv (x/+ (1s))lla))))))) we have:
Given
[J( .4) 1) (1
(1 7)2
4ey
Σ as
+
2c 2c(N
N
1)
N Σ
a;
(21)
where is the discount factor and e = max. |A (s, a)l.
Corollary 1 Suppose that we update joint policy Th to kht with sampling policy μ,, then the regret of the updated joint policy k+1 has the following upper bound:
4Ey
N
2
2c(N
1)
N
n((T') - 9 (+t)tt S <((T")
J(z 11) + (1 7)
ai
(22)
N
Theorem Suppose the sampling policy μ² can derive the same updated policy, which means that μ* arg max, n (.). If it erists, it will the solution of the following bi-level oplimization problem:
arg min DKL k+1), (+1] s.t. n k+1 = arg max ((T (1)
(33)
T
Theorem Let be the ego-mar- naa-marroppeee oaertoperttor We suppose that pt denotes the lookahead policy which means that it can derive the same updated policy, i.e., μ* = Vuu", T²); and nt' (0(T* x²) denotes the updated policy when using nh as the sampling policy. We eapress the trust region as that DTV (z(((10)11+(((1))) S B..
E.es Put (s))(,,A,)) we have J(p ,1)) 2 J(xt'=""
In this case, when Bg S 2) 2 Par (A)
Below are proofs.
Statement In previous multi-agent policy gradient methods, the learning objective for each agent is typi- cally defined as:
Mm = ( +
Σ Pm (s)
Σ n, (arls)
(I)) (s.0)) iE (1,2,...... ... (24)
SES
aEA
jtt
Proof. Based on the previous literature, have already know that for single-agent setting, the learning objective is typically defined as:
J(T)= = n(=)) + Epnt Σ [((a)))A,4(55))
(25)
#ES
LaeA
2 "Assuming a function defined over the state and joint action space, ego-max-operator defined &ff.8
EN ma Ea.re II (s).))))(ss.a).
18

---

Published in Transactions on Machine Learning Research (03/2025)
However, since is the policy are optimizing, can not obtain the training data belonging to its distribution perfectly. Thus, in practice, the objective is typically transformed into:
Σ T (als) {{ass) (<ass))
J(z) - -1==") Σ Pax (a)(8)
A (8, (s,a)
(26)
#ES
LaEd
T(a|s)
= {)(x") + E (saa)rt
A (8,
(27)
15 (a)s)
This allows us to optimize the training objective through sampling trajectories using n² which is the policy at the last round. For multi-agent cascs, the situation is similar. The provious multi-agent policy gradient methods typically sample data utilizing the joint policy n² to sample training data and the ideal goal is optimize the following objective:
J(T) = '1(+T* + >'Exrte Por
n (als) T(als)
A (s.
(28)
SES
LOEA
+)((+²)) + EE (s
T (als) A (s,
T (a)s)
(29)
However, since the candidate space of joint policy ग is huge and it is hard to optimize directly. The previous multi-policy gradient methods actually conduct a trade-off, and optimize a decomposed objective for each agent (Yu et al. 2022a):
(a;ls) A (s, (9.0a) ie (1,2,..,
Ti(ar(s)
J. (m.) n)(TT* + E (s,
(30)
Equation (30p can actually expressed as:
J.(Ti ((mrlkm5) (izs) n(TT")++ + Σ
Part (8)
Σ
((aasss) (7)( ie (1, ,,,
(31)
sES
aEA
ffi
Lemma 1 Assume that. we update the joint policy T to kt1 with sampling policy μ. the measurement of distance between sampling policy and the updated policy rktl as as max, DTV (T. ( ( ((1)))le((1ss)), we have:
Given
[[( .p) nn(T (t1]+
dey
Σ O4
+
2e(N - 1)
Σ 0;
N
(32)
(1 - 7
where 7y is the discount, factor and maxx.a A (s)aa)
19

---

Published in Transactions on Machine Learning Research (03/2025)
Proof. Firstly, according to the performance difference lemma (Kkkaee & Langford 2002). we have:
1T( 1
17(
-[ 1 .4)
Σ P (8) Σ π
(a))) (8,
8ES aEA
Σpu(s)
(a:a)s)) II [T((a)ls)Amm
N
$ES
im
aEA
jti
Σ
Σ
π
* (al (8, Epu(s)
Σ
π (a1s)
#ES
aEA
SES
aEA
(s((s) Σ
Th+ (als)
Σ Pm(s)
Ti(a:)s)
II [Ps(ayls)An
N
SES
i=
jtt
(33)
< Σ
P (5) Σ n x (als) Epu(s) Σ
* (als)))
.885
acA
SES
aEA
Epu(s) Σ n (n FES
+
aEA
Σ Ps())
(r(a)ss) [[s(ayss)AAn(s..)) it
N
i=1 aEA
4ey
[1 7)2
a²
+
Σ (s))s
Σ
π
h+1 (als))
+ES
GEA
Σ (s))))
(a(a)s))
II (ss(ay)o)AA))saaa)
N
1=1
aEA
where (I) holds because of the conclusion that has already been obtained in TRPO Theorem 1). Besides, we further have:
Schulman et al. 2015
(see
Σ Pu(s) Σ n (nlss asa
N
(ns)
Σ aEA
k+1
(a.[s) II
SES
<
c
(s)
1
a. (a:ls) Σ
[ms(as]s) itt
II n 321
(a,ls)
N
SES
a.E.:
R-IEA-
(34)
N
(II)
(s)
2
Σ
(a.ls) (1)7(0))))))
SES
2E
Σ (s((s
Σ
Drv (1
2E N
<
max Dry (1s)(l
N
i=1
20

---

Published in Transactions on Machine Learning Research (03/2025)
where (II) holds according to the definition of TV distance. Thus, we finally have:
N
YEY
2E
1( - J k+1 1 14) -
a
+
max DTv ( (11s)]
(1 7-
N
4ey
2N
1)
N
7)2
+
N
a4
(35)
i=1
4ey
N
2c(N
1)
N
7)²
a2
N
1=
(a)
where (III) is because S Ex= as Specifically, term (a) in the final upper bound is due to the state distribution mismatch of the training trajectories, while term (b) reveals the impact of the "teammate delay" phenomenon on the learning objective.
Corollary 1
Suppose that we update joint policy nh to nh+* with sampling policy μ, then the regret of the
updated joint policy π k+1 has the following upper bound:
N
48y
2c - 1)
'(TT) - n(( S <)((T))
J( ,11) + (1-7)2
Σ
Ci
+
Σ
0g.
(36)
N
1=1
1=1
Proof-
7(TT') (11)
1(TT) )) -[ 11) + J [[TT k+1 11) - [7(TT++)) 111
S n(TT) - -1 .1) + [ kt1 11) - -,(TT=
(37)
(IV) S ((TT)) - J( .1) + (1 (1-
4e7
N
2
2c(N - 1)
N
Σ
Σ
+
06
N
1=1
where (TV) is obtained due to Lemma 1.
Theorem Suppose the sampling policy μº' can derive the same updated policy, which means that μ¹ arg max, ( v.). If it exists, it will be the solution of the following bi-level optimization problem:
arg min DKL (11 s.t. n k+1 = arg max (
(38)
Proof. This theorem implicitly conveys a twofold meaning. First, we will prove that fx is one optimal solution of this optimization problem. Second, we will prove that each solution a for this optimization problem can derive the same updated policy when serving as the sampling policy.
First, for n², we know that the corresponding *+1 = arg max J(T 11) is also μe, due to the definition of μ². Then we have:
DKL (MIl *+1) = DKI (1w²)) =
(39)
Moreover, since DKL. (ull ++1) 2 O, we know that μ² achieves the lowest value of (KIl( Thus, μ²* is one solution for this bi-level optimization problem.
Sccond, suppose μe is one solution for this bi-level optimization problem. Then according to the definition of this problem, we should have:
(RT((E arg max [TT, (²))) S (1"""""))") 0,
(40)
It can be inferred that DKL (pt'l argmaxx J(TT, μz) )) 0, implying that arg max, J(T, pe)) and pt' have the same distributions. Thus we know 4t' arg max J(T,, (t')
21

---

Published in Transactions on Machine Learning Research (03/2025)
Theorem Let & be the ego-maz-operator. We suppose that μe denotes the lookahead policy which means that it can derive the same updated policy, i.e., 1a" = ( ,r);; and n' = ((*", x²) denotes the updated policy when using πh as the sampling policy. We eapress the trust region as that Drv (T(())))](f(((ss)) s B..
In this case, when 3, S E..s Pu (s))(
we have [1 ,10)) 2 ((x'xxx))
Pat (s)0(s)AA,))) (t
Proof. To begin with, for the updated policy n' (b( T²) using πt as the sampling policy, we have:
J( = "1(T²) +
Σ Pxt (8)
TH(alss) (ayl(s)))
N
SE8
1=1 aEA
3#1
|J(T', )²*) -J(TT TT))
Par (s)
(#<(a=|s) (a:ls)) II [=
N
(41)
(s)
N
(10)-m(p (a:ss) II [ [r(asss)=
N
jøi
Pr (8)
Σ Div((((1s)ll)
max E a-iEAiii
II
N
We have trust region condition that Vs S, Siy s 3z. We assume Bg has an upper bound for each agent i, then we further have:
N
[( - J[ x S 26 Σ (x)
Σ
(
max
N
a, EA
EES
e-IEAjEi
N
+ 1 S J( xh + 2c Σ
Park (8)
max a.EA
Σ
(42)
N
SES
a ii
1
+ J(T', (() S (T()*) + 2c Σ Park (s)
Σ
max
N
G,EA
We know that J(x,, 14") arg max J (TT, n²), where Ball(u") means the trust region of f². Further, this optimization problem means that:
1
arg max H(T, = n7(T** + Epw =EBall(u)
(s)
Pk (s)
(a,ss)
Il (I)(a jz1
(43)
N
#ES
aEA
Thus, we are actually to optimize 1 MI' EnEA mi(a:ls) II Li (ā As) (s,a) for each s E S; and when we find the optimized results are actually p. it means that for each s S S, ((-ss) is a nash equilibrium for the cooperative game where the utility of action is defined as A With proper updating scheme, it is reasonable that we can obtain equilibrium that satisfics:
Σ (arls) II (IG = Σ (a(lss)) aEA
Σ
Il
0.EA
a a
(44)
Σ
II
max a.E.A
EAjEA
22

---

Published in Transactions on Machine Learning Research (03/2025)
Thus we have that:
J( )- =n7(T*)
Σ PM (s)
Σ
M (agls) [T
N
SES
i=1 aEA
(45)
2 1)(TT)) + Σ Pk- (8)
max Σ (a)
N i=1
EA
sES
E-iEAj
We define ({.s)
Di max EA Ea EEA Il
Then when B. S C
N according to Equation (42) and Equation 045), it is obvious that "(p,,²²
EvcoPu- (s)0(e
2 22 Patk Pat
J(T',
A.3 Extra Analysis Upper Bound
Theorem has lold us that when the extra term (c) the upper bound disappears when we train the agents with future teammate information, which motivates us predict future teammates, However, we still retrain question whether climinating term (c) can indeed reduce the overall upper bound. For this question, one potential risk is that eliminating term (c) might influence the optimization of -JT,, μ4), thus making the second term -J(x**1 (4) larger. To solve this concern, we prove that under certain conditions it is at least better than the previous algorithms, as described below.
To begin with, we introduce the policy update operotor &b and ego-mar- moar-opeerttr & We suppose that μe denotes the lookahead policy which means that it can derive the same updated policy, i.e., μ = ( ,));; and T' = ( , x denotes the updated policy when using Th as the sampling policy. We express the trust region that Drv (1((-18)]1 (-16)) S Bc. In this case, when Bg EvcePu Put (s) 2 Pork (ot(e))0e
Theorem 2
we have Hpx 1x)) 2 J(T''= )
For proof see Appendix A.2 This theorem shows that when we replace μ with an approximation of future teammate policies, under certain conditions can at least obtain smaller upper bound compared to the previous typical algorithms. Specifically, the required conditions are relevant to the trust-region setting.
B More Implementation Details
B.1 More Details about Baselines
To conduct performance comparison in our experiments, we firstly select the main multi-agent actor-critic hmgs as baselines, including MADDPG (Lowe et al., 2017], TPPO (de Witt et al. 2020a), MAPPO (Yu et al. 2022a) and HAPPO (Kuba et al.) 2021). Besides, we additionally design an opponent modeling algorithm TAPPO (abbreviated for Teammate-Awail MAPPO) further compare our approach with tra- ditional opponent modeling techniques. In specific, TAPPO learns teammate representations for extra policy conditions like in previous works (Papoudakis & Albrecht, 2020; Cao et al., 2023 and is incorporated into MAPPO. Furthermore, in the GRF enviromment, we additionally include the CDS algorithm (Li et al. 2021). which is value-basee algorithm specifically designed for the GRF problems. It mainly designs mechanism to enhance policy diversity among agents and surpasses typical value-batl based algorithms in the GRF environment in its experiments.
S1 xh) means the result of round of policy update starting from x using μ the sampling policy, Le., ((, arg max x J(mr. 11) within trust. region x for MAPPO. Assuming a function defined over the state and joint action space, the ego-max-operator defined as Kf.8)
EN Ea Ea.re Ig 75 (s).))))(ss.a).
23

---

Published in Transactions on Machine Learning Research (03/2025)
B.2 Incorporating Lookahead Strategy into HAPPO
Heterogencouu--Aeent Proximal Policy Optimisation (HAPPO) (Kuba et al. 2021 | is a recent work that introduced the sequential policy update scheme to the multi-agent policy gradient algorithm. It provides a monotonic improvement guarantee in theory based on the finding of the multi-agent advantage decomposition lemma. The core idea of this work is to update the agents' policies in sequence. This approach empowers subsequent agents to adapt their policies based the updated strategies of preceding agents, thereby miligating to some extent the impact of the "Leammate delay" phenomenon. However, it has two main issues that might impact its effectiveness:
1) Despite the sequential policy update scheme, the preceding agents in the sequence still learn to cooperate with the previous round teammates, which means the teammate delay issue persists for the preceding agents. This results in a critical importance placed on the order of agents' update (e.g., for the example the Introduction section, the order is Cook first, the updates are more efficient; otherwise, sequential update yields no benefits). However, HAPPO adopts random update orders, which poses significant limitation.
2) Since in practice, the training trajectories are sampled by the policy of the previous round, НАРРО adopts Importance Sampling to help the subsequent agents learn to cooperate with updated previous agents. This approach can lead higher variance the policy gradients as we need multiply it by importance sampling ratio to correct the objective. This issue becomes exacerbated when dealing with larger number of agents, need to accunulate the product of importance ratios for all preceding agents.
Due to these two main issues, the effectiveness of HАРPО in practice may be influenced and it can not fully resolve the "teammate delay" issue Actually, in practice, our lookahead strategy can be seamlessly integrated with НАРPO, further enhancing its effectiveness, which has been validated by our empirical experiments. The detailed process is introduced in the Algorithm 2 where the text highlighted in rod emphasizes the uniqueness introduced by HAPPO.
Algorithm Heterogencoous-AAget Proximal Policy Optimisation with Lookahcad Input: The Number of agent N
Output: A cooperation policy for a multi-agent system
Initialize replay buffer B; Initialize policy π randomly;
3: for each iteration k do
4: Sample a batch of transitions from B and update the environment model by minimizing loss Loodet;;
5: Sample a batch of trajectories 7 in the environment model with sampling policy n², and obtain +++ via maximizing J(T, x²) within trust region in a sequential update scheme using the training trajectories F;
Sample batch of trajectories in the real environment with sampling policy *b+1, and obtain * k+1 via maximizing J(T,, **+1) within trust region in sequential update scheme using the training trajectories T;
7: Add trajectories t to the buffer B;
8: end for
B.3 Details about Hyper-parameter
In this section, we firstly introduce the hyper-parameter configurations of our method in the experiments, and then illustrate how tune the hyper-parameters.
24

---

Published in Transactions on Machine Learning Research (03/2025)
used across task scenarios of multi-agent MuJoCo. Note that lka is short for Lookahead, mlearn short for "model learning", and ppo stands for Proximal Policy Optimization (PPO) algorithm.
Table 5: Common
hyper-parameter
value
hyper-paranneter
value
hyper-parameter
value
critic Ir
3e-4
max grad norm
10
Ika episode length
20
actor Ir
3c-4
num rollouts
40
lka num mini-batches
10
gamma
0.99
ppo num mini-batches
10
lka entropy coef
0.001
optimizer
Adam
entropy coef
0.01
mlearn batch size
512
optim ops
lc-5
stacked-frames
1
B.3.1 Hyper-pant parameter configuration
Here, list the configuration of hyper-paramettrss that was utilized in experiments to facilitate re- producing our experimental results. Note that for the common hyper-parameters, both the Tookahead algorithm and HAPPO adopted the same value in our experiments. Hence, the hyper-parameter configura- tions provided in this section are also applicable to our experimental results of the HAPPO algorithm.
Multi-agent MuJoCo
For typre-parameters that were set to the same values across all task scenarios, Additionally, the varying hyper- ppprr-pareette configurations across
the configuration provided in Table 5
different tasks are provided in Table i
Table 6: Different hyper- parameetrss used across task scenarios of multi-agent MuJoCo. Note that lka is short for Lookahead, mlearn is short for "model learning", and ppo stands for Proximal Policy Optimization (PPO) algorithm.
hyper-paramcter
Ant
HalfCheetah
Walker2d
episode length
200
400
200
ppo num epochs
10
20
10
Ika num rollouts
2000
4000
2000
Ika num epochs
10
20
10
mlearn num epochs
4000
2000
2000
Google Rescarch Football (GRF) In the three task scenarios of Google Research Football (GRF), we employed identical hyper-parameler configurations, which are detailed in Table 7
Table 7: Common hyper-parameters used across task scenarios of Google Research Football (GRF). Note that Ika short for Lookahead, mlcarn short for "model learning", and ppo stands for Proximal Policy Optimization (PPO) algorithm.
seper-parameter
value
hyper-parameeter
value
hyper-parameter
value
critic Ir actor ir gamma
5e-4
episode length
100
Ika num rollouts
100
5e-4
num rollouts
10
Ika num mini- mani-bacches
1
0.99
ppo num mini-batches
1
Ika num epochs
15
optimizer
Adam
ppo num epochs
15
Ika entropy coef mlearn batch size
0
optim eps
le-5
entropy coef
5e-3
1024
max grad norm
10
Ika episode length
100
mlearn num epochs
800
25

---

Published in Transactions on Machine Learning Research (03/2025)
KL Distance Difference
0.60
2x4-Ageut
Ant
KL Distance Difference
0.060
HalsCheetah
2x3-Agont HallCheetak 3x2-Agont HallChortak
KL. Distance Difference
Walker2d
Age-Agent WalkerZd
0.45
4x2-Agent
0.045
0.012
3x2-Agem' Wallerdd
0.30
0.030
0.006
0.15
0.015
0.003
0.00
0.000
0.000
0.0M
1.0M
2.0M 3.0M 4.0M
5.0M
0.0M
1.0M
2.0M
3.0 Timesteps
4.0M
5.0M
0.0M
1.0M
2.0M
3.0M
4,0M
5.0M
Timesteps
Timesteps
(a) Ant Task
(b) HalfCheetah Task
(c) Walker2d Task
Figure 6: Measuring policy distances in multi-agent MuJoCo environment.
B.3.2 hypee--arameter tuning strategy
Multi-agent MuJoCo For the practical implementation efficiency of the algorithm, we employed JAX implement our Lookahead algorithm. Additionally, to ensure fair and effective comparison of the efficacy of our added lookahead strategy, the underlying HAPPO algorithm also utilized the same codebase. Furthermore, the fundamental hyper-parameters for both Lookahead and HAPPO were kept consistent. Consequently, mainly tune the parameerrss to align the performance of the HAPPO algorithm of our codebase with that disclosed in the original paper.
Google Research Football (GRF) Similar to the case in multi-agent MuJoCo, to ensure a fair com- parison, we maintained consistent foundational hyper-paramoters for both Lookahoad and the underlying HAPPO algorithm. While luning these hyper-paramelers, we conducted a search within certain ranges to fine-tune the underlying HAPPO algorithm for reasonably good performance results. Specifically, we explored learning rate ir (including critic_lr and actor_ _r) within the range of {1e-4, 5e-4, 1e-3), number of learning epochs apo_num__ -pochs within {10, 15, 20), and entropy regularization coefficient entropy_coet coef within {1e-3, 5e-3}.
C More Experimental Results
C.1 More Results about Lookahead Policy Analysis
In Section 4.3 measure the KL distance difference of policies Ant and HalfCheetah tasks of multi- agent MuJoCo. Here, we additionally provide the results on the Walker2d task. As we can see, the results on three different types of tasks all validate that our approach can empirically obtain positive results for D +*) - ( **+1), which means that #*+1 can to some extent provide the direction infor- mation of future teammate policy *+1 and is relatively a good approximation.
C.2 Study about Multiple Steps of Lookahead Approximation
In the practical implementation of the algorithm in this work, we employ a one-step approximation to estimate future teammate policies. We wonder whether we can obtain better approximation of future tcammates through more rounds of lookahcad training. To answer this question, we conduct additional experiments, comparing with the baselines that perform more rounds of policy update when obtaining the lookahead policy. The results are depicted in Figure where "Lookahead Step" represents the number of policy update rounds conducted for approximating future teammates, i.c., "Lookahead Step=1" corresponds to the results in the maintext.
From the results, we can see that when we increase the Lookahead Step, the quality of the lookahead approximation does not increase and we obtain lower performance. It is reasonable because when we conduct more iterations for lookahead training within the model, the newly updated policy would appear unfamiliar to the model, as the environmental model has been trained data. sampled from the old policies. This
26

---

Published in Transactions on Machine Learning Research (03/2025)
4000
Lookahead Scep-1
4000
3200
Lookahead Sxep-2
Lookahead Sxep74
3200
Score
2400
Score
2400
1600
1600
800
800
8.0M
1.0M
2.0M 3.0M
4.0M
5.0M
d.M 1.0M
2.0M
3.0M
4.0M
5.0M
Timestops
Timestops
(a) 2x4-Agent Ant
(b) 2x3-Agent HalfCheetah
Figure 7: Experimental results for study about multiple steps of lookahead approximation.
necessitates refraining from employing excessively off-policy policies for trajectory rollout within the model. However, despite achieving lower convergence performance, it seems to learn faster in the early stage in the task of 2xe-Agent HalfCheetah when we increase the Lookahead Step. This encourages us to design better model learning algorithm in the future, thus to further improve the effectiveness of our approach.
27


Reviews:
[{"review_id": "i7vhBcS0k2", "reviewer": "Reviewer_7aYm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The method used to deal with the “teammate delay” issue is very interesting. However, I find several concerns regarding the problem's validity, theoretical clarity, and overall presentation. My detailed comments are as follows:\n\nThe “teammate delay” described in the paper seems to be a potentially artificial issue that arises due to specific assumptions about the learning setup. Its significance is questionable for the following reasons:\n\nThe multi-agent environment is inherently dynamic because each agent’s strategy influences others. This non-stationarity arises naturally from agents’ concurrent updates rather than a specific \"delay\" mechanism.\n1.If agents update their strategies simultaneously, there is no actual \"lag.\" Agents will adapt to updated teammate policies during interaction. In distributed or asynchronous optimization, delays are more related to computation or data synchronization, not learning.\n2.In cooperative MARL, agents dynamically adjust to others’ behaviors through feedback mechanisms (e.g., rewards or shared information). While “teammate delay” might temporarily influence progress, agents’ strategies are expected to converge over time.\n3.The paper builds on Equation (4) without introducing the Performance Discrepancy Lemma in the preliminaries. Readers unfamiliar with this result might find the subsequent theoretical analysis difficult to follow. More background on this lemma should be provided.\n\nThe definition of  (state distribution under the discounted return) is missing from the main text and is instead deferred to the appendix, increasing the reading difficulty. However, even in the appendix, there is no explicit mathematical definition of it.\n\nWhile the result in Equation (2) can be supported by prior literature, the authors fail to rigorously justify Equation (3). The derivation of Equation (3) is not presented, and the paper does not provide a strict proof. Please include a detailed derivation to validate this result.\n\nThe rationale behind Equation (5) is insufficiently explained. The text vaguely states it is for “further analysis,” but this does not justify its introduction. Moreover, introducing time indices could improve the clarity of this equation.\n\nThe significance of the bound presented in Lemma 1 is not well explained. Please elaborate on the practical implications of this result.\n\nThe paper refers to a policy update operator and an ego-max operator, but these terms are not formally defined in the main text. Definitions should be provided for clarity.\n\nThe proof of Theorem 2 is written ambiguously. Specifically, the statement “ \\pi^k+1 can be defined” lacks justification, and the use of subscript k is inconsistent throughout the proof. This inconsistency makes the proof difficult to follow and undermines its rigor. Additionally, the explanation for why \\pi^k+1 = \\mu^* holds is not clear. This part of the proof requires more formal reasoning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the “teammate delay” issue in Cooperative Multi-Agent Reinforcement Learning (MARL), where agents experience discrepancies due to concurrent teammate updates. By introducing a lookahead strategy to predict future teammates, the approach improves learning stability and efficiency, achieving superior performance over state-of-the-art MARL methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see my comments on weaknesses.", "broader_impact_concerns": "Nan", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Wq5fVnTX1p", "reviewer": "Reviewer_XofD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThis paper motivates a practical lookahead based approach to policy learning in MARL through theoretical intuitions. \n\n1. _Regret analysis to quantify the impact of teammate delay_: Lemma 1 computes an upper bound for the difference between the surrogate loss $J(\\pi^{k+1},\\mu)$ used during training update (at $\\pi^k$) and the actual return $\\eta(\\pi^{k+1})$ evaluated in practice. The bound depends on the total variation distance between the sampling $(\\mu)$ and updated $(\\pi^{k+1})$ policies. Theorem 1 establishes an upper bound for the overall regret of the updated joint policy $\\pi^{k+1}$, showing that this bounding term can be minimized by reducing the mismatch between $\\mu$ and $\\pi^{k+1}$. \n\n2. _Future teammate approximation through one-step lookahead_: Due to the coupling between $\\mu$ and $\\pi^{k+1}$, the authors propose a bilevel optimization problem that solves for the sampling policy $\\mu^*$ which would minimize the TV distance between $\\mu$ and $\\pi^{k+1}$ in the previous regret analysis. Following similar approaches to bi-level leader follower problems with Stackelberg games, this paper proposes an optimization step that first solves for a sampling policy $\\tilde{\\pi}^{k+1}$ which approximates future teammate policy and uses $\\tilde{\\pi}^{k+1}$ to estimate $\\pi^{k+1} = \\mathop{argmax}_{\\pi} J(\\pi, \\tilde{\\pi}^{k+1})$.\n\n3. _Leveraging model based RL to improve training sample efficiency_: To estimate future teammate policies efficiently, the lookahead approximation method is integrated into a model-based learning framework. Trajectories collected by the sampling policy during policy update are stored in a replay buffer and used to train the environment model.\n\n\n4. _Ablation studies for analyzing benefits of lookahead over baselines_: Comparisons with baseline methods like MADDPG, MAPPO and HAPPO validate the effectiveness of the lookahead strategy. The proposed method easily integrates with existing algorithms like MAPPO and HAPPO without requiring substantial architectural changes, making it accessible for adoption. \n\n\n**Weaknesses**\n\n1. _Performance limited by model error_: As the authors also admit, the \"practical algorithm performance is to some extent limited by the model learning error\". Assuming the authors have used a predictive neural network model of the environment (note request for architecture and training details under Requested Changes), it would have to be retrained to model different environments. The authors have not discussed the potential of using their proposed approach with methods from generative modeling of the environment (eg. maybe drawing from the ongoing research in generative world models). \n\n2. _Scalability with larger teamsizes_: The experiments are limited to settings with almost 3 agents. The current set of results do not provide sufficient evidence of performance in environments with larger teamsizes where approximating the future teammate policy would be challenging. Particularly, there is a lack of discussion of this method's limitations with stochastic policies. \n\n3. _Applicability to POMDPs_: The ability to leverage a well trained environment model would be particularly important when the environment is partially observable. In this paper, authors focus primarily on MDP environments and propose model learning methods for future work. \n\n4. _Performance in long horizon tasks_: This paper does not focus on tasks that require multi-agent cooperation over long horizons and/or against strategic opponents, so it is difficult to argue for the practical utility of the proposed method in realistic application settings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on the drawback of existing cooperative MARL algorithms where agents update their individual policy without accounting for the concurrent updates in their teammates' policies. This phenomenon is termed as \"teammate delay\". In this work, a lookahead based approach is proposed that accounts for concurrent teammate updates by updating each agent's policy to learn to cooperate with predicted future teammates. It is a flexible algorithmic framework that can integrate with existing policy gradient based MARL algorithms, like HAPPO. Experiments with MA-MuJoCo and Google Research Football indicate that lookahead based policy update is competitive with prior MARL baselines and achieves similar evaluation performance with fewer training update steps.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. [Critical for decision] SMAC is another popular environment for testing MARL algorithms. Could the authors compare the proposed approach against baselines in SMAC? \n\n2. [Critical for decision] I would request the authors to provide more details of their environment model - eg. the network architecture, training algorithm, etc. \n\n3. [Critical for decision] Please fix the typos in the paper (eg. page 10: roll-outed -> rolled out; Abstract: learn to cooperative -> learn to cooperate).\n\n4. [Critical for decision] The motivating example in Fig 1 and accompanying text in the first paragraph of page 2 is confusing to me. This particular example changes the cook's task or goal (from cooking hamburgers to cooking salad) which causes the mismatch between the teammates policies. This, to me, does not seem like a convincing example of \"teammate delay\" because the rest of the paper focuses on settings where all teammates are updating their policy towards the same goal.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "szoSLXxMgh", "reviewer": "Reviewer_N2BW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The concept of predicting teammate policy updates is interesting. The provided algorithm is not computationally heavy, and can be seemlessly integrated with existing MARL frameworks. \n\nWeaknesses on Theoretical Results: \n- Theorem 1 doesn't sufficiently motivate us to solve for a sampling policy $\\mu$ that minimizes term $(c)$. In Equation (7), with $\\pi^{k+1}= \\arg\\max_{\\pi}J(\\pi,\\mu)$ being a function of $\\mu$, a natural target will be to find $\\mu$ so that $J(\\pi^{k+1}, \\mu) - (c)$ is maximized. There is no reason to believe a $\\mu$ that minimizes $(c)$ will maximize the two terms as a whole, unless the authors provide more intuitions. \n- Theorem 2 is not rigirous. The definition of operator $\\psi$ is unclear. In footnote (2), the ``trust region`` is ambiguous. The reviewer has checked the cited MAPPO paper, but it is not mentioned in the paper. Moreover, in the proof, the authors state that *$\\psi$ typically means ... with some cinditions*. The terms ``typically`` and  ``some conditions`` are way too vague for a rigorous proof. Furthermore, the authors need to address the existence of such solution $\\mu^*=\\psi(\\mu^*,\\pi^k)$ in the proof. \n- Aside from the aforementioned issues, the reviewer finds Theorem 2 trivial. Since $\\pi^{k+1}$ is defined as $\\psi(\\mu, \\pi^k)$ (footnote 2), an obvious $\\mu$ closest to $\\pi^{k+1}$ is simply the solution to $\\mu = \\psi(\\mu,\\pi^k)$, if such a solution exists. The reviewer suggests that the reviewer expressing the $\\mu$ design in different way.\n\nOther Weaknesses:\n- Theorem 1 is not very informative. It appears to be a natural result that follows by Lemma 1 and a triangle inequality. It is better presented as a corollary.\n- Some equations need to be checked. For example, In Equation (1), it seems there should be a factor $\\frac{1}{1-\\gamma}$ in the second term, if $\\rho_{\\pi}(s)$ is a probability distribution. In Equation (8), it seems $\\min_{\\mu}$ should be $\\arg\\min_{\\mu}$.\n- Some notations are confusing. The following are some examples.\n    - At the beginning of Section 2.2, the reward function $\\mathcal{R}$ is not explained. From the context, it seems the agents share the same reward function and this fact deserves a clarifying sentence. \n    - $\\eta(\\pi)$ in Equation (1) is not formally defined throughout. According to the context, it should be the expect value $E[V_{\\pi}(s)]$ over the initial state distribution. However, the initial state distribution is also not defined. \n    - In Theorem 1, $\\pi^*$ is not defined. We can not be sure whether it is Markovian or non-Markovian.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates cooperative MARL. To improve learning stability and efficiency, this paper proposes an algorithm with a lookahead strategy, which enables the learning agents to predict the future behavior of teammates and make better policy updates. The authors provide theoretical results that justify the necessity of this lookahead strategy. The algorithm is tested in MuJoCo and Google Research Football experiments.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The aforementioned Weaknesses should be addressed. All of them are critical to securing my recommendation.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the “teammate delay” issue in Cooperative Multi-Agent Reinforcement Learning (MARL), where agents experience discrepancies due to concurrent teammate updates. By introducing a lookahead strategy to predict future teammates, the approach improves learning stability and efficiency, achieving superior performance over state-of-the-art MARL methods.

All the reviewers acknowledge the contribution of the work, while there are several minors raised by reviewers should be addressed in final version:

- Some claims in the paper is over-claiming, "The empirical results unequivocally demonstrate the superiority of our method,..", however, there exists competitors on par with the proposed algorithm. 

- The terminology "teammate delay problem" is related to the instability issues of multi-agent training environments, which should be discussed to avoid potential confusion.
Decision: Accept (poster)