Intent-Aware Planning in Heterogeneous Traffic via Distributed Multi-Agent Reinforcement Learning
Authors: Xiyang Wu
Rohan Chandra
Tianrui Guan
Amrit Bedi
Dinesh Manocha
Venue: CoRL 2023 Oral (2023)
Keywords: Autonomous Driving
Multi-agent Reinforcement Learning
Representation Learning

Abstract: Navigating safely and efficiently in dense and heterogeneous traffic scenarios is challenging for autonomous vehicles (AVs) due to their inability to infer the behaviors or intentions of nearby drivers. In this work, we introduce a distributed multi-agent reinforcement learning (MARL) algorithm for joint trajectory and intent prediction for autonomous vehicles in dense and heterogeneous environments. Our approach for intent-aware planning, iPLAN, allows agents to infer nearby drivers' intents solely from their local observations. We model an explicit representation of agents' private incentives: Behavioral Incentive for high-level decision-making strategy that sets planning sub-goals and Instant Incentive for low-level motion planning to execute sub-goals. Our approach enables agents to infer their opponents' behavior incentives and integrate this inferred information into their decision-making and motion-planning processes. We perform experiments on two simulation environments, Non-Cooperative Navigation and Heterogeneous Highway. In Heterogeneous Highway, results show that, compared with centralized training decentralized execution (CTDE) MARL baselines such as QMIX and MAPPO, our method yields a $4.3\%$ and $38.4\%$ higher episodic reward in mild and chaotic traffic, with $48.1\%$ higher success rate and $80.6\%$ longer survival time in chaotic traffic. We also compare with a decentralized training decentralized execution (DTDE) baseline IPPO and demonstrate a higher episodic reward of $12.7\%$ and $6.3\%$ in mild traffic and chaotic traffic, $25.3\%$ higher success rate, and $13.7\%$ longer survival time.

Intent-Aware Planning in Heterogeneous Traffic via Distributed Multi-Agent Reinforcement Learning
Xiyang Wu University of Maryland wuxiyang@umd, edu
Rohan Chandra
Tianrui Guan University of Maryland rayguanSumd. edu
University of Texas at Austin
rchandra@utexas.c edu
Amrit Singh Bedi University of Maryland amri tod@umd. edu
Dinesh Manocha
University of Maryland
dmanocha@umd. edu
Abstract: Navigating safely and efficiently in dense and heterogeneous traffic scenarios is challenging for autonomous vehicles (AVs) due to their inability to infer the behaviors or intentions of nearby drivers. In this work, we introduce a distributed multi-agent reinforcement learning (MARL) algorithm for joint trajec- tory and intent prediction for autonomous vehicles in dense and heterogeneous en- vironments. Our approach for intent- planning, iPLAN, allows agents to infer nearby drivers' intents solely from their local observations. We model an explicit representation of agents' private incentives: Behavioral Incentive for high-level decision decision-making strategy that sets planning sub-goals and Instant Incentive for low-level motion planning to execute sub-goals. Our approach enables agents to infer their opponents' behavior incentives and integrate this inferred information into their dociiion-mmking and motion-planning processes. We perform exper- iments two simulation environments, Non-Cooperative Navigation and Het- erogeneous Highway. In Heterogeneous Highway, results show that, compared with centralized training decentralized execution (CTDE) MARL baselines such as QMIX and MAPPO, our method yields a 4.3% and 38.4% higher episodic re- ward in mild and chaotic traffic, with 48. 1% higher success rate and 80.6% longer survival time in chaotic traffic. We also compare with decentralized training de- centralized execution (DTDE) baseline IPPO and demonstrate a higher episodic reward of 12.7% and 6.3% in mild traffic and chaotic traffic, 25.3% higher success rate, and 13.7% longer survival time.
Keywords: Autonomous Driving, Multi-agent Reinforcement Learning, Repre- sentation Learning
Introduction
In this work, consider the task of trajectory planning for autonomous vehicles in dense and het- erogeneous traffic. High density is typically measured in the number of vehicles per square meter and high heterogeneity refers to a large variance in agents' driving styles ranging from aggressive to conservative, vehicle dynamics, and vehicle types [1]. For example, these agents may include two-wheelers, cars, buses, and trucks. The key challenge to efficient trajectory planning in such environments to be able to accurately infer the behavior of these heterogeneous agents [2]. There- fore, many solutions perform trajectory planning by jointly predicting the agents' future trajectories along with their intent [3].
Trajectory prediction is the task of predicting the future states of an agent [4] which typically con- sists of spatial coordinates, and heading angle, but may also include first-order information such as
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

velocity. Intent prediction focuses on inferring neighboring agents' behavior using local informa- tion [5]. In the context of autonomous driving, some studies have approached intent prediction by classifying driving behaviors into predefined classes [6, 2] such as aggressive or conservative.
Although many methods for joint trajectory and intent prediction [7, 8, 3, 5] have been extensively studied for planning in both industry and academia, most of the existing approaches
trained and evaluated on datasets like The Waymo Open Motion Dataset [9] and the NuScenes dataset [10), which primarily consist of homogeneous traffic and lack variation in driver behavior [3], As a result, these methods [7, 8, 3, 5] often struggle to reliably predict the intentions of heterogeneous agents in unstructured and dense traffic [11].
On the other hand, simulators such as CARLA are designed generate traffic agents with diverse, kinodynamically feasible behaviors [12], addressing the lack of diverse behavior in datasets. Most of the joint trajectory and intent prediction methods evaluated on the datasets discussed above can be used with such simulators [13,4] But these methods typically require generating and collecting data in offline storage, which defcats the purpose of a simulator [14]. Complementary to these offline approaches, simulators [12] also offers the capability to model multiple agents and their interac- tions simultaneously via multi-agent reinforcement learning (MARL), where the learning algorithm can engage with the simulation environment [15]. MARL has demonstrated remarkable success in many different multi-agent domains such as Go [16], chess [17]. poker [18]. Dota2 [19], and StarCraft [20]. However, their applicability to autonomous driving has been relatively sparse [21].
Deep MARL for trajectory plamning in autonomous driving only recently achieved significant mo- mentum with the Highway-Env environment [22] proposed the author's doctoral thesis [23]. Since then, several deep MARL approaches have been proposed [24, 25] for trajectory planning, but these methods do not extend to heterogeneous traffic and also assume agents can communicate and share information with each other. To the best of our knowledge, there is prior decentralized training decentralized execution (DTDE) MARL approach for joint intent and trajectory prediction for AVs in heterogeneous traffic. More related works are included in Appendix B.
Main Contributions: In this paper, we propose new intent-aware trajectory planning algorithm for autonomous driving in dense and heterogeneous traffic environments. We cast the autonomous hidden parameter partially observable stochastic game (HiP-POSG) [26, 27] and solve it using a DTDE MARI. framework, called iPL.AN, built around a joint intent and tra- jectory prediction encoder-decoder architecture. Given the current traffic conditions and historical observations, iPLAN computes the optimal multi-agent policy for each agent in the environment, relying solely on local observations without
driving problem
or communication.
Our main contributions include:
1. To the best of our knowledge, we propose the first DIDE MARL algorithm for joint trajectory and intent prediction for autonomous vehicles in dense and heterogeneous environments. Our algorithm is fully decentralized without weight sharing, communication, or centralized critics, and can handle variable agents across episodes.
2. We model an explicit representation of agents' private incentives that include (i) Behavioral Incentive for high-level drcision-making strategy that sets planning sub sub-goals and (it) Instant Incentive for low-level motion planning to execute sub-goals. These incentives enable behavior- aware motion forecasting, which is more suited for heterogeneous traffic.
3. We perform experiments on two simulation environments, Non-Cooperative Navigation [28] and Heterogeneous Highway [22]. The results show that, compared to centralized training decen- tralized execution (CTDE) MARL baselines like QMIX and MAPPO, our method yields a 4.3% and 38.4% higher episodic reward in mild and chaotic traffic and is 48.1% more successful with an 80.6% longer survival time in chaotic traffic Heterogeneous Highway. Compared to the DTDE baseline IPPO, we demonstrate a higher episodic reward of 12.7% and 6.3% in mild and chaotic traffic, 25.3% higher success rate, and 13.7% longer survival time in the Heterogeneous Highway.

---

2 Problem Formulation
Problem Setting and Assumptions: We consider a multi-agent scenario with N 2 2 non- cooperative agents [29], i.e., agents are controlled by individual policies that maximize their own reward without weight sharing or communication. In each episode, agents interact with one an- other and gain general experience without any prior knowledge about specific agent from previous episodes. Agents' strategies remain the same within one episode, though strategies may evolve be- tween episodes. We assume that all agents are driven by motivations behind their actions. These motivations can arise from instantaneous reactions to environmental changes or more enduring pref- erences. We denote them as incentives for agents' strategies. While these incentives are private and not explicitly known to other agents, they can be discerned through observing agents' strate- gies that offer insights into the incentives behind agents' actions. In this work, we explicitly model these private incentives with hidden parameters representing latent states. Therefore, formu- late this problem as multi-agent hidden parameter partially observable stochastic game [30], or HiP-POSG
Task and objective: We consider the tuple
(4)
1. (fl T, (r.)):))))
(I)
where N the number of agents. S the set of states. A, the set of actions for agent i. O the observation set of agent i of the global state E S, generated by agent i's observation function Ω4 : 8 O, our problem, agent i's observation of at time t could be further specified as of = f be,, where N₁ refers the set of agents j in the neighborhood of i. The bold of denotes the set of agent i's observation of its neighbors at time t. We denote the sequence of agent i's historical observations Oz,j of opponent j up to time & as hi, = {of jo The bold h =o denotes agent i's observation history of its neighbors. Here, we indicate that agent i's observation history of agent only consists of its observation of agent j's states, while agent j's actions and rewards are unobservable information by others. Z, denotes the latent state space that represents the incentive of agent i's strategy. St O x O x... x O x 2, + 2, is agent i's incentive inference function that makes an estimation ž., of its opponent j's actual incentive 2j from its observation history of opponent h&, up to time t and its past estimation 2n. Here, we assume agent i's estimations of agent j's incentive 2,g belongs to the same latent state space Zj agent j's actual incentive 2j. T S x Ai Az x x AN - A(S) is the (stochastic) transition matrix between global states. Ta S x A1 Ag x x Av + R is the reward function for agent i. 7 is the reward discount factor. Agent decides its action a; E A, with policy Of x Og X x O'2 x Z x Zn x... x ZN + A(A:) with its observations of. own incentive 2; and estimated opponents' incentives _ at time t.
The objective of agent is find the optimal policy x;. maximizing its --disoounted cumulative rewards over episode of length T. The objective equation is given by
T Σ Y'' (s', (0.())))))))
π arg max ET W,
where r; is the reward function of agent i.
Incentive Latent Representation. In this work, we assume that agents' actions are motivated by (i) long-term planning tied to an agent's driving behavior or personality and (ii) short-term collision avoidance related to the current traffic state. To this end, we decouple agent i's incentive 2 into a vector 2; = (3, Our formulation related to the task and motion planning literature [31] where the behavior incentive follows a high-level decision-making strategy with the goal of setting planning sub-goals whereas the instant incentive refers to the low-level motion planning with the goal of executing the sub-goals. The behavior incentive biases the motion forecasting in behavior- aware manner such that it is better suited for heterogeneous traffic.
an extension of the HiP- HiP-POMDP [26. 27]
3

---

Behavioral Incentive 3 models drivers' driving styles which are deeply rooted in their personal- ities [32]. Given the observations for the previous few seconds, behavior incentive performs high- level decision makiing and plans actions, or sub-goals, and asks, "What's the most likely action of this driver to lake nex?"". The answer is encoded via 8. This tells an agent whether it should speed up in empty traffic or slow down in dense traffic. also is able recognize conservative drivers and the possible need to overtake. Therefore, this incentive is able to reason between aggressive and conservative drivers.
Instant Incentive ç signifies drivers' instantaneous responses to proximate traffic, taking into ac- count the positions and speeds of neighboring vehicles. Instant incentive then asks, "How should I execute this sub- sub-goal/highh-eeee action/plan using my controller so that I I'm safe and still on track towards my goal?". Instant incentive measures classical efficiency metries defined in robotics liter- ature such collision avoidance (safety), distance from goal, and smoothness.
Incentive Inference To cater to two different incentives, we split agent i's incentive inference function fi into two distinct functions, fi,9 and faç: Big ~ (..((1ht),,8,,) uses agent i's historical observation hig of opponent up to time t and its previous estimation of opponent j's behavioral incentive B to estimate opponent j's new behavioral incentive Bi at time Pi.j
Ci fr.c(-lo,g. Bi (t,") uses agent i's observation of.j of opponent j at time its current estimation over opponent j's behavioral incentive Bi, and its previous estimation of opponent j's instant incentive 2,, to estimate opponent j's new instant incentive Čt, at time L. With the esti- mation of opponents" incentives, agent i's policy af ~ (-100 Bi, 2.) decides its action ag with its local observation, ego incentive, and estimations over opponents' incentives. Here, 3 denotes the combination of agent i's hehavioral incentive 3 and its estimations over all its opponent agents' be- havioral incentives (
==1, at time ? denotes the combination of agent i's instant incentive and its estimations over all its opponent agents' instant incentives 12r, 1 at time t.
iPLAN: Methodology
We demonstrate the overall architecture of our proposed framework in Figure Agents interact with the environment with continuous state space S. Here, we denote that an agent's 's state includes its ID, current position, and current velocity. An agent's ' observation includes the states of its neighbors within its observation scope. An agent records its historical observations of its opponents' states for incentive inference. With historical observations h and intermediate observations of, agent i estimates opponent j's behavioral incentive B, and instant incentive Çy- The controller of agent decides action at based on its local observation of ego, and opponents' estimated behavioral incen- tives B.. and instant incentives 2/- The action space A of the environment is discrete and consists of the following high-level actions: {lane left, idle, lane faster, slower) in our Heterogeneous Highway environment, or (idle, up, down, left, right} in our Non-cooperative Navigation environ- ment (details in Appendix C), while a low-level motion controller (e.g., IDM model [33]) converts the high-level actions into a sequence of x, y coordinates.
3.1 Behavioral Incentive Inference
The behavioral incentive inference module intends estimate opponents' behavioral incentives by generating latent representations from their historical states. At time step f, agent i queries a sequence of historical observations ht for opponent from its observation history profile as the input of the behavioral incentive inference module. For case of computing, we truncate the full historical interaction sequence into fixed-length sequence that includes the observation history from the previous ta steps. We introduce an encoder & to update opponents' behavioral incentive estimation and a decoder D, to predict opponents' state sequences in the next t,, steps with current historical observations and behavioral incentive estimation. In practice, parameterize encoder E,

---

Traffic state at time
Behavioral Incentive Inforence
Controller
Historica cal
Behavioral Incerttiee Encoder
Predicted
Current Osservation
Aggressive Vehicle
Conservative Vehicle
D
0
A
Bellaiioral Incentive
Neutral Vehicle
Predictdd Trajectory
Past Observations
A
Behasioral
g
Behaviard oral becoder
2
Incentive
Vehiele's Mind
Corrent
What current position
Incentive Inference
and verocity of the redt
Controller
sebicle?
Behavioral
Incentvve ts the ariver of the real rericle oggressive conservativ??
ff
GAT-RNN
RNN
6 Predicred Next State
Action
WWl the rea vehicle
accelenate and overtalle in the next move?
Current
Incentiic
Prediction
At time t, we show current vehicle states in solid colors: ego vehicles (solid yellow vehicle), aggressive vehicles (solid red), conservative vehicles (solid green), neutral vehicles (solid bluc). The future states of each vehicle are shown with dotted colors. time step i, the ego-agent observes nearby vehicles incentive inference (red block) uses agent i's historical observations h of other vehicle states (stacked gray boxes of current observations. o.) to infer their behavioral incentives and predict future state sequences with behavioral incentive inferences. The instant incentive inference (blue block) uses agent i's current observations
Figure 1: Intent-aware planning in heterogeneous traffic:
infers their behavioral and instant incentives. The behavioral
of (single gray box) and its inference of others' behavioral incentives B3 (single red box) infer other vehicles' instant incentives ? for trajectory prediction. Agent i's controller (yellow block) selects its action af with its current observations of (gray) and its inference of others' behavioral incentives B (red) and instant incentives ? (blue).
with fgg, and decoder DD with PD, Hence, the encoder & approximates the behavioral incentive inference function Big ~ fa( P1,
To capture the sequential nature within opponents' state observation sequences, the encoder & em- ploys recurrent network that processes My time series. This produces behavioral incentive of opponent j. As insights from cognitive science suggest, the human social focus remains relatively stable [34]. Thus, we interpret the behavioral incentive inference for oppo- nents as gradual process, converging towards the true behavioral incentives of opponents without abrupt transitions between updates. Starting with an initial neutral estimation of opponents' behav- ioral latent states, agents propose new estimates for opponents' behavioral incentives each time step. However, they employ gentle update strategy, using an additional coefficient 7), to refine the behavioral incentive estimates. This approach allows agents produce more accurate estimates of opponents' behavioral incentives, managing the variability between consecutive updates, which in turn ensures more stable agent policies.
estimate of the
= nE.,,, (1- n)
The decoder D, uses another recurrent network that concatenates agent i's historical observations hiy of opponent j with its current behavioral incentive estimation S The output the predicted state sequence h of opponent j from to + th. We train our encoder and decoder with behav- ioral incentive inference loss JR.. given by average Ll-norm enror between the predicted state sequence hith p f B and the ground truth 24th
Jo. = min
Di(hi -
(4)
D. Nl.
5

---

3.2 Instant Incentive Inference for Trajectory Prediction
The instant incentive inference module intends to estimate opponents' instant incentives from cur- rent observations of surrounding agents and their behaviors, which is used for trajectory prediction. Similar to the behavioral incentive inference, we introduce another encoder-decoder structure with encoder o, parameterized by 8g. and decoder thy parameterized by Pv. The encoder Φ; approx- imates the instant incentive inference function Çi ~ fi,c(-lo%.. At. 2t-1) from agent i's current observations of of agent current behavioral incentive estimations B and previous instant incentive estimations _i- The instant latent state encoder Φ; uses a sequential structure with two networks. The first network a Graph Attention Network (GAT) [35]. For agent i, GAT reads its observation of time and the current behavioral incentive estimation B The output of GAT is fed to an undirected graph G that represents instantaneous interactions among agents time t. Every node in g represents an agent in the environment, while the attention weight over the edge between node i and node encodes the interaction between agent and j with its relative importance. The second part of the encoder Φ is recurrent neural network (RNN) to extract the temporal information from interaction history. The RNN uses the graphical representation G of interactions as the input and 2 previous instant incentive estimation č; as the hidden state. The output hidden state of this RNN ? the updated instant incentive estimation over all opponents of agent
The decoder dh predicts all opponents" trajectories over pre-defined length &p from instant incentive estimations č We use another RNN that takes agent i's current observation of as the input and its current instant incentive estimation C as the hidden state. The first output of this RNN is the prediction of opponents' states g at the next time step + 1. Then we use o, as the new input of RNN and iteratively predict opponents' states. The sequence of opponents' state predictions (o
{ō, Jk=1 (O, ()) is the trajectory prediction from / 1 to lp for all opponents of agent We train our encoder and decoder with instant incentive inference loss JG. given by an average LI-norm error between predicted trajectories ( 1 jk=1 and ground truth trajectories {o + J
(1 B. (1))) of o
Jc. min 0.. Nlp j=1
(5)
3.3 Implementation
The pseudocode (Algorithm 1) and flow graph (Figure 3) of our algorithm are given in Appendix A. For each environmental step in the execution (line 4), agent gathers its current and historical observations of and hę (line 6), and uses this information infer their opponents' behavioral in- centives 8 and instant incentives ç (lines 7 and 8). After that, agent i's policy म, selects action of (line 9). The backbone algorithm for each agent's controller is PPO [36], which includes policy network ग. and critic network Q, For each gradient step in training, agent updates its policy m and critic Qi (line 15) with sampled trajectories, computes the behavioral incentive inference loss Js. (line 16) to update its behavioral incentive inference encoder 0g, and decoder 0p, with JR., and uses instant incentive inference loss Jt. (line 17) to update its instant incentive inference encoder 0g. and decoder 0g..
Empirical Results and Discussion
We perform experiments over two non- non-cooperative environments, Von-Cooperative Navigation [28] and Heterogeneous Highway [22]. Experiments are designed from two perspectives. The first is to compare our approach's performance with other CTDE and DTDE MARL approaches in non- cooperative environments. this paper, we compare our method with two CTDE MARL baselines, QMIX [37] and MAPPO [38], and one DTDE MARL baseline, IPPO [39]. QMIX uses central network to assign credits among agents with respect to their Q-values and global states. MAPPO

---

Eas Scoraro
Hard Sowsers
000
302
603
40-
1000
36
00 02
14
3.0
0.0
02
0-5
1.0
0.80
0.25
0.50
0.75
1.00
1.25
1.50..00
0.20
0.50
0.25
125
1.00
Time
1e0
OMI
IPPO
IPLANGGT
GIXX
IPFO
IPLAN-GAT
MAPPO
IPLAN-SN
IPLAN
MAPPO
IPLAN-S
IPLAN
(a) Con-Cooperative Navigation: with 3 agents (b) Heterogencous Highway: with agents (left) (left) easy and (right) hard scenarios. 50 steps/episode. mild and (right) chaotic scenarios. 90 steps/episode.
Comparison of average episodic reward in the Non-Cooperative Navigation and Heterogeneous Highway environments. Conclusion: iPLAN (orange) outperforms CTDE approaches like QMIX (blue) and MAPPO (brown) as well as IPPO (green) in hetcrogeneous traffic environments.
Figure 2:
uses a central critic that reads the observation of all agents and generates a critic value update distributed actors. IPPO uses distinct PPO policy control each agent without any centralized training. weight-sharing, communication, inference module. The other perspective is to show the necessity of instant and behavioral incentive inference, especially under highly heterogeneous scenarios. We further design two scenarios with different heterogeneity levels in both environ- ments and perform ablation studies over two variants of our method, including iPLAN-BM a vanilla IPPO controller without the instant incentive inference module, and iPLAN-GAN a vanilla IPPO controller without behavioral incentive inference module. Details regarding the experiment envi- ronment design given in Appendix C. Further details regarding implementation, visual results, module design, and aaper-prreeeter study are given in Appendix D, E, F, and G.3, respectively.
4.1 Results on Non-Cooperatiie Navigation
Figure 2a compares episodic rewards in easy and hard scenarios. iPLAN outperforms other methods with low deviation. iPLAN-GAT and vanilla IPPO have larger deviations, indicating the benefit of behavioral incentive inference in stabilizing strategies. QMIX and MAPPO perform poorly with negative episodic rewards in both scenarios. In Non-Cooperative Navigation, agents are attracted to the closest landmark at each time step, allowing multiple agents to target the same landmark simultaneously. As there is no consensus in destination assignment, agents must observe and infer others' strategies to modify their own. This reliance on observations and inference contributes to the superior performance of DTDE MARL approaches over CTDE MARL approaches in Non- Cooperative Navigation.
4.2 Results on Heterogeneous Highway
Figure 2b compares episodic rewards in the mild and chaotic traffic scenarios of the Heterogeneous Highway. We find that iPLAN has the best episodic reward in both the mild and chaotic traffic. iPL.NNGAT, iPLAN-BM, and vanilla IPPO have similar performances in mild traffic scenarios, but iPLAN-GAT is slightly worse than iPLAN in the chaotic traffic. Notably, two CTDE MARL baselines have much lower episodic rewards than DTDE MARL approaches in chaotic traffic, and QMIX has a significant collapse compared with its performance in mild traffic.
In addition to the episodic reward curve comparison, evaluate method and baselines several navigation metrics, including:
Episodic Average Speed. Agents' average speed during their lifetime an episode. Agents are encouraged to drive faster when driving between 20 and 30 m/s.
Average Survival Time. The average time steps passed over all agents before they collide reach the end of this episode. Longer survival time reflects agents' better ability to avoid collisions.
7

---

Success Rate. The percentage of vehicles that still stay collision-fii when an episode ends.
Table shows navigation metrics for mild and chaotic traffic. High speed (closer to 30) correlates with low survival time and success rate. This is because aggressive reward-exploitng policies in- crease collision risk, reducing long-ferm reward. Approaches like iPLAN and iPLAN-GAT drive slower (closer to 20) for safety and higher episodic reward. Instant incentive inference improves episodic reward and success rates, especially in chaotic traffic. iPLAN maintains similar success rates but a higher average speed in mild traffic, being more conservative and dependent in heteroge- neous traffic. Comparing iPLAN and iPLAN-GAT, iPLAN drives faster both scenarios for higher episodic reward. iPLAN-GAT has longer survival time mild traffic, but the opposite in chaotic traffic. This indicates that agents are more dependent on their instant incentive inference in mild traffic when opponents' trajectories incentive inference in chaotic traffic due aggressive vehicles' unpredictable behaviors. QMIX performs well in mild traffic but poorly in chaotic traffic (success rate 20%) due to environmental heterogeneity effect on its credit assignment.
more predictable, and more dependent their behavioral
5 Conclusion, Limitations, and Future Work
This paper presents a novel intent- aware distributed MARL algorithm tailored for planning and navigation in heterogeneous traffic. We model two distinct incentives, the behavioral incentive and the instant incentive, for agents' strategies. Our approach enables agents to infer their oppo- nents' behavior incentives and inte-
Avg. Speed (m./s)
Avg. Survival Time
Success Rate
Approach
(# Time Steps) f
(%)
QMIX [37]
21.24 :: 0.09
75.98 +: 3.67
67.50 :: 6.34
MAPPO [38]
27.85
0.40
48.94 3.11
32.81 + 5.22
IPPO [39]
22.63 1.:3:00:
66.13
49.06 7.35
iPLAN GAT
22.05 1: 0.11
75.54 = 3.61
68.44 : 6.64
iPLAN-BM
22.61 = 0.16
61.11 = 4.28
45.63 : 6.33
iPLAN
22.91 0.15
70.56 3.81
68. + 5.86
QMIX [37]
27.06 0.47
39.38 2.64
19.69 + 3.72
Chuntic
MAPPO [38]
29. 29.46=22 0.05
42.31
16.25 = 3.76
grate this inferred information into their decision-making and motion- planning processes. Results show that our approach shows promis- ing result in the two environments we use, Non- Non-Cooperative Navigation and Heterogeneous Highway, with a better performance in episodic re- ward curves and navigation metrics
IPPO [39]
22.28 0.13
3.64
42.50
iPLAN-GTT
20.91 0.13
71.21 3.83
61.88 = 6.41
iPLAN-BA BM
21.65 0.28
63.20 3.51
35.31 = 5.66
iPLAN 21.61 0.16
76.20 3.33
5.91
Table 1: Navigation metrics in Heterogencous Highway: Met- rics are averaged over 64 episodes with 0.95 confidence. iPLAN outperforms all other approaches in its highest success rate and sur- vival time, though it tends to be conservative in its average speed.
than baselines. Nonetheless, is important to acknowledge certain limitations within our research:
Firstly, our evaluation has been confined to simulation environment. Real-world autonomous driving scenarios present diverse array of traffic situations and driving behaviors, potentially posing challenges for the generalization of our approach to unfamiliar strategies unseen during training. Furthermore, attempting to predict the complete state of multi-agent system real-workd contexts may introduce the risk of significant and potentially hazardous errors, as agents might inaccurately reconstruct or predict states.
Secondly, the design of our framework centers on the use of two incentives for representing and inferring the objectives of other drivers in motion planning. However, in certain scenarios, such as in more straightforward or mixed conditions, the necessity of dual incentives remains ambiguous i.e. it might be that a singular incentive set is adequate. Future research could delve deeper into the design of inference models. Additionally, while our contributions are empirically substantiated, they currently lack a solid theoretical foundation. Therefore, forthcoming research initiatives should prioritize the establishment of theoretical safety and convergence bounds for our approach.
8

---

Acknowledgments
We would like thank Caroline Wang for helpful discussions and feedback during the course of this paper. This research supported by Army Cooperative Agreement W911NF2120076.
References
[1] R. Chandra. Towards Autonomous Driving in Dense, Heterogeneous, and Unstructured Traffic. PhD thesis, University of Maryland, College Park, 2022.
[2] R. Chandra, U. Bhattacharya, T. Mittal, Bera, and Manocha. Cmetric: driving behavior measure using centrality functions. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2035-2042. IEEE, 2020.
[3] R. Chandra, T. Guan, S. Panuganti, T. Mittal, U. Bhattacharya, A. Bera, and D. Manocha. Forecasting trajectory and behavior of road-agents using spectral clustering in graph-Istms. IEEE Robotics and Automation Letters, 2020.
[4] N. Lee, W. Choi, P. Vernaza, C. B. Choy, P. H. S. Torr, and M. Chandraker. Desire: Distant fu- ture prediction in dynamic scenes with interacting agents. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2165 2174, 2017.
[5] S. Qi and S.-C. Zhu. Intent-aware multi-agent reinforcement learning. In 2018 IEEE interna- tional conference on robotics and automation (ICRA), pages 7533-7540. IEEE, 2018.
[6] R. Chandra, U. Bhattacharya, T. Mittal, X. Li, A. Bera, and D. Manocha. Graphrqi: Classifying driver behaviors using graph spectrums. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4350-4357. IEEE, 2020.
[7] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data. In A. Vedaldi, H. Bischof, T. Brox, and J.- M. Frahm, editors, Computer Vision ECCV 2020, pages 683-700, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58522-5.
[8] R. Chandra, U. Bhattacharya, A. Bera, and D. Manocha. Traphic: Trajectory prediction in dense and heterogeneous traffic using weighted interactions. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8483-8492, 2019.
[9] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp, C. R. Qi, Y. Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9710-9719, 2021.
[10] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631) 2020.
[11] R. Chandra, X. Wang, M. Mahajan, R. Kala, R. Palugulla, C. Naidu, A. Jain, and D. Manocha. Meteor: A dense, heterogeneous, and unstructured traffic dataset with rare behaviors. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9169-9175. IEEE, 2023.
[12] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1-16. PMLR, 2017.
[13] N. Rhinehart, R. Mcallister, K. Kitani, and S. Levine. Precog: Prediction conditioned on goals in visual multi-agent settings. In 2019 IEEE/CVF International Conference Com- puter Vision (ICCV), pages 2821-2830, Los Alamitos, CA, USA, nov 2019. IEEE Computer

---

Society. 10t (00:100..10//CCY22099900000 URL 10. 1w19//199II.../ 201.
[14] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 2011.
[15] J. Zhang, A. S. Bedi, M. Wang, and A. Koppel. Multi-agent reinforcement learning with general utilities via decentralized shadow reward actor-critic. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 9031-9039, 2022.
[16] D. Silver, A. Huang, C. Maddison, Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser,
I. Antonogiou, V. Pannecrshelvam, M. Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
[17] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al. Mastering chess and shogi by self with general reinforce- ment learning algorithm. arXiv preprint arXiv: 1711v01712.0181/ 2017.
[18] N. Brown and T. Sandholm. Superhuman ai for multiplayer poker. Science, 365(6456):885- 890, 2019.
[19] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019.
[20] O. Vinyals, T. Ewalds, S. Bartunov, P. Georgiev, A. S. Vezhnevets, M. Yeo, A. Makhzani, H. Küttler, J. Agapiou, J. Schrittiieser al. Starcraft A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04785 2017.
[21] R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez. Deep reinforcement learning for autonomous driving: A survey, IEEE Transactions Intelligent Transportation Systems, 23(6):4909 4926, 2021.
[22] E. Leurent. An environment for autonomous driving decision-making https://githii:
com/eeleureet/hhihaaayyeee, 2018
[23] E. Leurent. Safe and efficient reinforcement learning for behavioural planning autonomous driving. PhD thesis, Université de Lille, 2020.
[24] H. Zhang, W. Chen, Z. Huang, M. Li, Y. Yang. W. Zhang, and J. Wang. Bi-level actor-critic for multi-agent coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7325-7332, 2020.
[25] B. Chen, Xu, Z. Liu, I.. Li, and D. Zhao. Delay-aware multi-agent reinforcement learning for cooperative and competitive environments. arXiv preprint arXiv:2005.05441,, 2020.
[26] F. Doshi-Veles and G. Konidaris. Hidden parameter markov decision processes: A semipara- metric regression approach for discovering latent task parametrizations. In IJCAI: proceedings of the conference, volume 2016, page 1432. NIH Public Access, 2016.
[27] J. Song, H. Ren, D. Sadigh, and Ermon. Multi-agent generative adversarial imitation learn- ing. Advances in neural information processing systems, 31, 2018.
[28] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and Mordatch. Multi-agent actor-critic for mixed cooperative-compettiiee environments. Neural Information Processing Systems (NIPS), 2017.
10

---

[29] J. Nash Jr. Non-cooperative games. In Essays on Game Theory, pages 22-33, Edward Elgar Publishing, 1996.
[30] E. A. Hansen, D. S. Bernstein, and S. Zilberstein. Dynamic programming for partially observ- able stochastic games. In AAAI, volume 4, pages 709-715, 2004.
[31] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:265-293. 2021.
[32] E. Cheung, A. Bera, E. Kubin, K. Gray, and D. Manocha. Identifying driver behaviors using trajectory features for vehicle navigation. In 2018 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages
IEEE, 2018.
[33] M. Treiber, A. Hennecke, and D. Helbing. Congested traffic states in empirical observations and microscopic simulations. Physical review E, 62(2):1805, 2000.
[34] Swettenham, S. Baron-Cohen, T. Charman, A. D. Cox, G. Baird, A. Drew, L. M. Rees, and J. Wheelwright. The frequency and distribution of spontaneous attention shifts between social and nonsocial stimuli in autistic, typically developing, and nonautistic developmentally delayed infants. Journal of child psychology and psychiatry, and allied disciplines, 39 5:747- 53, 1998.
[35] Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio. Graph attention networks. arXiv preprint arXiv: 1090, 2017.
[36] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint 100.:v:1777.66... 2017.
[37] T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson. Mono- tonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research,
2020.
[38] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu. The surprising effectiveness of ppo cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611-2466
2022.
[39] C. S. de Witt, T. Gupta, D. Makoviichuk, V. Makoviychuk, P. H. S. Torr, M. Sun, and S. White- son. Is independent learning all you need in the starcraft multi-agent challenge?, 2020.
[40] W. Luo, Park, A. Commman, B. Sapp, and D. Anguelov. JFP: Joint future prediction with interactive multi-agent modeling for autonomous driving. In 6th Annual Conference Robot Learning, 2022. URL htttps://ppereviiee
[41] A. Farid, S. Veer, B. Ivanovic, K. Leung and M. Pavone. Task-relevant failure detection for trajectory predictors in autonomous vehicles. In 6th Annual Conference on Robot Learning, 2022. URL pettpa://openreviie het.:/eeeetttooorridddddonnooo
[42] P. Bhattacharyya, C. Huang, and K. Czarnecki. SSL-lanes: Self-supervised learning for motion forecasting in autonomous driving. In 6th Anmual Conference on Robot Learning, 2022. URL https://openreviex. wwt.wriiiei.eteorriueeeeee www-
[43] R. Chandra, U. Bhattacharya, C. Roncal, A. Bera, and D. Manocha. Robustip: End-to-end trajectory prediction for heterogeneous road-agents in dense traffic with noisy sensor inputs. In Proceedings of the 3rd ACM Computer Science in Cars Symposium, pages 1-9, 2019.
[44] H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp. B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid, C. Li, and D. Anguelov. Tnt: Target-driven trajectory prediction. In Conference on Robot Learning, 2020.
11

---

[45] V. Lefkopoulos, M. Menner, A. Domahidi, and M. N. Zeilinger. Interaction-awat aware motion prediction for autonomous driving: A multiple model kalman filtering scheme. IEEE Robotics and Automation Letters, 6(1):80-87) 87, 2020.
[46] K. Okamoto, K. Berntorp, and S. Di Cairano. Driver intention-based vehicle threat assessment using random forests and particle filtering.
2017.
[47] Joseph, F. Doshi- Velez, A. S. Huang, and N. Roy. A bayesian nonparametric approach to modeling motion patterns. Autonomous Robots, 31:383-4000 40033-400000 2011.
[48] A. S. Bedi, D. Peddireddy, V. Aggarwal, and A. Koppel. Efficient gaussian process bandits by believing only informative actions. arXiv preprint arXiv:2003. 2020.
[49] J. Li, W. Zhan, Y. Hu, and M. Tomizuka. Generic tracking and probabilistic prediction frame- work and its application in autonomous driving. IEEE Transactions on Intelligent Transporta- tion Systems, 21(9):333--3649 2019.
[50] C.-J. Hoel, K. Driggs-Campbell K. Wolff, L. Laine, and M. J. Kochenderfer. Combining planning and deep reinforcement learning in tactical decision making for autonomous driving. IEEE transactions on intelligent vehicles, 5(2):294-305 2019.
[51] N. Deo and M. M. Trivedi. Convolutional social pooling for vehicle trajectory prediction. In Proceedings of the IEEE conference computer vision and pattern recognition workshops, pages 1476-147, 2.
[52] S.-W. Yoo, C. Kim, J. Choi, S.-W. Kim, and S.-W. Seo. Gin: Graph constraint policy optimization for autonomous driving. IEEE Robotics and Automotion Letters, 8(2):464 471, 2022.
Graph-based based interaction-awer awwae
[53] Z. Cao, E. Biyik, G. Rosman, and D. Sadigh. Leveraging smooth attention prior for multi-agent trajectory prediction. In 2022 International Conference on Robotics and Automation (ICRA), pages 10723-10730. IEFF, 2022.
[54] A. Liniger and J. Lygeros. A noncooperative game approach to autonomous racing. IEEE Transactions Control Systems Technology, 28(3):884 897, 2019.
[55] Y. Wang, F. Zhong, J. Xu, and Y. Wang. Tom2c: Target-oriented multi-agent communication and cooperation with theory of mind. arXiv preprint arXiv:2111.09189, 2021.
[56] N. Rabinowitz, F. Perbet, F. Song, C. Zhang, S. A. Eslami, and M. Botvinick. Machine theory of mind. In International conference on machine learning, pages
PMLR, 2018.
[57] A. Vemula, K. Muelling, and J. Oh. Social attention: Modeling attention in human crowds. In 2018 IEEE international Conference on Robotics and Automation (ICRA), pages 4601-4607. IEEE, 2018
[58] Z. Dai, T. Zhou, K. Shao, D. H. Mguni, B. Wang, and H. Jianye. Socially-atenntive policy optimization in multi-agent self-driving system. In Conference on Robot Learning, pages 946- 955. PMLR, 2023.
[59] H. Wu, Sequeira, and D. V. Pynadath. Multiagent inverse reinforcement learning via theory of mind reasoning, arXiv preprint arXiv:2302. 10238, 2023.
[60] R. Chandra, R. Maligi, A. Anantula, and J. Biswas. Socialmapf: Optimal and efficient multi- agent path finding with strategic agents for social navigation. JEEE Robotics and Automation Letters, 2023.
[61] H. He, J. Boyd- K. Kwok, and H. Daumé III. Opponent modeling in deep reinforcement learning. In International conference machine learning, pages 1804-1813, PMLR, 2016.
12

---

[62] Z. Zhu, E. Bıyık, and D. Sadigh. Multi-agent safe planning with gaussian processes. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6260- 6267. IEEE, 2020.
[63] G. Papoudakis, F. Christianos, and S. Albrecht. Agent modelling under partial observability for deep reinforcement learning. Advances in Neural Information Processing Systems, 34: 19210-19222, 2021.
[64] D. P. Losey, M. Li, J. Bohg, and D. Sadigh. Learning from my partner's actions: Roles in decentralized robot teams. In Conference on robot learning, pages 752-765. PMLR, 2020.
[65] Parekh, Habibian, and D. P. Losey. Rili: Robustly influencing latent intent. 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 01-08. IEEE, 2022.
[66] B. Von Der Osten, M. Kirley, and T. Miller The minds of many: Opponent modeling in a stochastic game. In IJCAI, pages 3845-3851, 2017.
[67] K. K. Ndousse, D. Eck, S. Levine, and N. Jaques. Emergent social learning via multi-agent reinforcement learning. In International Conference on Machine Learning, pages 7991-8004. PMLR, 2021.
[68] A. Xie, D. Losey, R. Tolsma, C. Finn, and D. Sadigh. Learning latent representations to influence multi-agent interaction. In Conference robot learning, pages 575-588. PMLR, 2021.
[69] W. Z. Wang, A. Shih, A. Xie, and D. Sadigh. Influencing towards stable multi-agent interac- tions. In Conference robot learning, pages 1132-1144 1143. PMLR, 2022.
[70] N. Jaques, A. Lazaridou, E. Hughes, C. Gulcehre, Ortega, D. Strouse, J. Z. Leibo, and N. De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International conference on machine learning, pages 3040-3049. PMLR, 2019.
[71] D.-K. Kim, M. Riemer, M. Liu, J. Foerster, M. Everett, C. Sun, G. Tesauro, and J. How. Influencing long-term behavior in multiagent reinforcement learning. Advances in Neural Information Processing Systems, 18:88888818811, 2022.
[72] A. Mavrogiannis, R. Chandra, and D. Manocha. B-gap: Behavior-rich simulation and naviga- tion for autonomous driving. IEEE Robotics and Automation Letters, 7(2):4718-4725, 2022.
[73] Polack, F. Altché, B. d' Andréa Novel, and de La Fortelle. The kinematic bicycle model: A consistent model for planning feasible trajectories for autonomous vehicles? In 2017 IEEE intelligent vehicles symposium (/V), pages 812-818. IEEE, 2017.
13

---

A Algorithm
A.1 Pseudocode
Algorithm iPLAN: Intent-aware Planning Hcterogeneous Traffic via Distributed MARL
Require: Number of agents N, Number of experiences K for experience replay, Length of histori- cal observation sequence th. Length of trajectory prediction tp
1: Initialize: Agent i's network parameters on, lq. 0E,, 0D. Pø.. Pg.. i 1,2,...... ...
2: Initialize: Replay Buffer B <- 0. Incentive Inferences B. 0. S; f 0 for i 1, . N
3: for each environmental step do NN do 5: Gather current and historical observations of and h
4: for i
Infer behavioral incentives B, with & (h,, B3 2-11
6:
Infer instant incentives č with (o 0 21-1
7:
Select action ot with m ( 2 9: end for
8:
end for
11: for each gradient step do
Sample K experiences from the replay buffer B
13:
for k = 1,2..... K do
for
N do
15:
// Update PPO controller
Perform experience replay on experience & Update policy Oz and critic fq. of the PPO controller // Update hehavioral incentive inference module for each step t in experience k do
17:
19:
Gather historical observation sequence h from experience k
20:
Infer behavioral incentives 3 with E, BB -,-²1))
21:
Predict future observation sequence h; -1- with Di(h)" )
22:
Use predicted h +t*+ and ground-truth truth h +he compute Ts, in (4) Update behavioral incentive encoder 0g, and decoder 0D, with Jg. end for
23:
24:
25:
26:
Update instant incentive inference module for each step t in experience i do
27:
Gather current observation of and behavioral incentives 3 from experience -11
28:
Infer instant incentives ? with di ,3, t
29:
C:
Predict future trajectories {o, + 1/ with ui 5;
30:
31:
j==1 to compute Jr, in (5) Update instant incentive encoder 0g2 and decoder 0,, with Tt. end for end for
Use predicted {6: } and ground-truth fo +
32:
34:
end for
36:
end for
Output: 8 o3 n. for each
14

---

A.2 Flow Diagram
To illustrate our algorithm in Algorithm 1, we create a flow diagram to visualize the execution and training procedure performed by agent in iPLAN, Details could found Fig.
Initialization
N
Updating Network?
Behavioral
Gather New Observation
Incentive Module
Sample Replay
Controller
Buffer
Instant Incentive
Module
Infer Behavioral Incentive
Query Historical Observation Seq
Query Current Observation
Experience Replay
Infer Instant Incentive
Infer Behavioral Incentive
Infer Instant Incentive
Update Controller Network
Predict Future Observation Seq
Predict Future Trajoctories
Actions
Plug into Replay Buffer
Update Behavioral Incentive Network
Update Behavioral Incentive Network
N
Terminate?
Output Networks
Figure 3: Flow Diagram for Algorithm
15

---

B Related Work
Trajectory and Intent Prediction for Autonomous Driving. Trajectory prediction is a fundamen-
tal task in autonomous driving [40, 41, 42]. TraPHic and RobustTP [43, 8] use an LSTM-CNN framework to predict trajectories in dense and complex traffic. TNT [44] uses target prediction, motion estimation, and ranking-based trajectory selection to predict future trajectories. DESIRE [4] uses sample generation and trajectory ranking for trajectory prediction. PRECOG [13] combines conditioned trajectory forecasting with planning objectives for AVs. Additionally, many methods focus intent prediction gain better understanding of interactions between vehicles when pre- dicting trajectories. Intent prediction be done by physical- based methods like Kalman filter [45] or Monte Carlo [46], classical machine learning like Gaussian processes (GP) [47, 48), Hidden Markov Model (HMM) [49], and Monte Carlo Tree Search (MCTS) [50], or deep learning- methods such as Trajectron++ and CS-LSTM [7, 51]. [52] uses a Seq2Seq framework to encode agents' observations over neighboring vehicles as their social context for trajectory forecasting and decision-making. [53] uses temporal smoothness attention modeling for interactions and a se- quential model for trajectory prediction. However, most methods overlook variations in driving behaviors, which deteriorates their reliability in heterogeneous traffics.
Intent-aware Multi-agent Reinforcement Learning. As a large-scale and non-cooperative [54] scenario, the awareness of opponents' incentives is quite important when implementing MARL in autonomous driving. Intent-aware multi-agent reinforcement learning [5] estimates an intrin- sic value that represents opponents' intentions for communication [55] or pecision-makknn. Many intent inference modules are based on Theory of Mind (ToM) [56] reasoning or social attention- related mechanisms [57, 58). [59] uses ToM reasoning over opponents' reward functions from their historical behaviors in performing multi-agent inverse reinforcement learning (MAIRL). [60] uses game theory ideas to reason about other agents' incentives and help decentralized planning among strategic agents. However, many prior works oversimplify the intent inference and make some prior assumptions about the content of intent. In the real world, agents' incentives are more complex and intractable during interactions among large groups of agents, so more general and high-level incentive representation is needed in intent-aware MARL.
Opponent Modeling, Opponent modeling [61] in multi-agent reinforcement learning usually de- ploys various inference mechanisms to understand and predict other agents' policies. Opponent modeling could be done by either estimating others' actions and safety via Gaussian Process [62] or by generating embeddings representing opponents' observations and actions [63]. Inferring op- ponents' policies helps to interpret peer agents' actions [64] and makes agents more adaptive when encountering partners [65]. Notably, many works [66, 67] reveal the phenomenon whereby ego agents' policies also influence opponents' policies. To track the dynamic variation of opponents' strategies made by an ego agent's influence, [68, 69] propose the latent representation to model opponents' strategies and influence based on their findings on the underlying structure in agents' strategy space. [70] provides causal influence mechanism over opponents' actions and defines an influential reward over actions with high influence over others' policies. [71] proposes an optimiza- tion objective that accounts for the long-term impact of ego agents' behavior in opponent modeling. A considerable limitation of many current methodologies is the underlying assumption that agents continually interact with consistent set of opponents across episodes. This assumption is a mis- fit for real-world autonomous driving contexts. On roads, drivers constantly come across different vehicles and drivers, necessitating the ability to infer the intentions of new opponents with minimal prior knowledge.
C Experiment Details
C.1 Non-Cooperative Navigation
Non- Coo-erooeeee Navigation is developed based the Multi-agent Particles Environment (MPE) [28]. n agents required to maximize their coverage landmarks without any ex-
16

---

plicit cooperation or inter- aggntt communication mechanism. Instead of being assigned some pre- determined landmarks as their destinations, agents are attracted to the immediate closest landmark at each time step. This indicates that an agent's s destination is not fixed in an episode and that mul- tiple agents can be attracted to a specific landmark simultaneously. Agents should properly select their intended landmarks, reach and stay their intended landmarks, and avoid any conflicts with other agents. The length of each episode is 50 steps. Agents and landmarks randomly initialized within 2 x world space. All plots in Non-Cooperative Navigation are averaged over 5 random seeds.
In Con-Coooeeettive Navigation, there are three different kinds of agents that are controllable by MARL policies and one kind of agent that is controlled by the pre-defined random policy taking random actions each time step. Table shows the parameters of different kinds of agents; their major differences come from their sizes and acceleration values:
Agent Type
Size
Acceleration
Normal
0.08
1.0
Tiny 0.06
1.1
Bulky 0.10
0.9
Random 0.08
1.0
Table 2: Parameters for Agents in Non-cooperative Navigation
Scenarios. Two scenarios with different heterogeneity levels are included this paper:
Easy: Normal agent, 1 Tiny agent, and Bulky agent.
Hard: 1 Normal agent, Tiny agent, Bulky agent, and Random agent.
Note that all agents in the easy scenario are controllable. One uncontrollable agent exists along with three controllable agents in the hard scenario, which makes this scenario more heterogeneous.
Observation Space. Coo-Coopeee Navigation is a fully-observable environment with con- tinuous observation space for each agent. The observation vector of agent is composed of state vectors of all entities within the world space, including the states of all agents and landmarks. Here, we denote the state of an entity in Non- Von-Coopeeative Navigation as vector with its ID, current posi- tion, and velocity. Within agent i's observation vector, the positions of all entities are their positions with respect to agent Agent i's ego state vector locates at the top of its observation vector and uses its own absolute position in the world space. For those CTDE MARL algorithms requiring the global state, the global state is the collection of all entities' state vectors composed of their IDs, absolute positions, and velocities in the world space.
Action Space. Non-Cooperative Navigation has a discrete action space with 5 identical high-level actions, (idle, up, down, left. right). Taking action in any direction (i.e., all actions except idle) makes this agent accelerate by one step size that direction. The acceleration step size varies in different kinds of agents.
Reward. Each agent has an individual reward function in Non-Cooperative Navigation. An agent gets penalty that equals its distance from the closest landmark in the environment at each time step. Notably, multiple agents may get this penalty with respect to their distances to a specific landmark if this landmark is the closest to all of them. If a collision happens between two agents, both will receive a penalty of -5. If an agent reaches the scope with a distance of less than 0.1 to any landmarks, this agent receives positive reward of 10. We denote this scope as the rewarding scope. If all controllable agents reach and stay within the rewarding scope without conflicts, they all receive positive reward of 100.
17

---

C.2 Heterogeneous Highway
Heterogeneous Highway is developed based on Highway-eny [22], which is 2D autonomous driv- ing simulator based on PyGame. Traffic scenarios in our environment are designed based on the Highway scenario given by Highway-env with simulated vehicles driving on a multi-lane highway. The objective of vehicles controlled by MARL algorithms is maintain collision-free trajectory with proper speed between 20 and 30 m/s when driving through heterogeneous traffic. Uncon- trollable vehicles are controlled by three different behavior-driven vehicle models modified from models proposed in [72], and we denote them as Normal, Aggressive, and Conservative vehicles. Their major differences come from their kinematic features, given in Table 3.
Kinematic Parameters
Normal
Aggressive
Conservative
Max Speed (m/s)
40
50
40
Default Speed Range (m /s)
[23, 25]
[35, 40]
[23, 25]
Max Acceleration (m/s* 2²
6.0
9.0
5.0
Desired Acceleration (m/s"
3.0
6.0
2.0
Desired Deceleration (m ss))
-5.0
-9.0
-4.0
Desired Front Distance (m)
5.0 +1
0.5
8.0 +
Time Wanted (Before Stop) (s)
1.5
1.2
1.8
Table 3: Kinematics for the behavior-drivens vehicle model used in Heterogeneous Highway scenarios. All
vehicles are assumed to have the same size 1.
The length of each episode is 90 steps. Initially, vehicles are randomly placed throughout the world space with a density of 1. All results Heterogeneous Highway have averaged over random seeds.
Scenarios. Two scenarios under mild and chaotic traffic included in this paper. Each scenario has 5 controllable vehicles and 50 behavior-driven vehicles uniformly distributed over an 8-lane high- way. The compositions of different behavior-driven vehicles relate to the heterogeneity of traffic. The mild traffic has mostly -ehavine vehicles, so we consider this scenario more homoge- neous. In the chaotic traffic scenario, more aggressive vehicles exist, which makes the environment more heterogeneous. Here are the propositions of each kind of behavior-driven vehicle in mild and chaotic traffic scenarios:
Mild: 80% Normal vehicles 10% Aggressive vehicles + 10% Conservative vehicles. Chaotic: 40% Normal vehicles 30% Aggressive vehicles + 30% Conservative vehicles.
Observation Space. Heterogeneous Highway is partiall environment in that agents can only observe 15 other vehicles within their predefined observation scope. The observation scope for each agent is 100 m in both directions of the x-axis and 20 m in both directions the y-axis. Each agent has a continuous observation space. The observation vector of an agent is composed of stacked state vectors of all vehicles within its observable scope. Here, we denote state vector of a vehicle i's observation vector, its ego state vector locates it at the top of its observation vector and uses its own absolute position in the world space. The remaining state vectors are state vectors of vehicles observed by agent using their positions relative to agent i. The global state for CTDE MARL. baselines is made up of concatenated state vectors of all controllable and uncontrollable vehicles within the environment.
vector with its ID, current position, and velocity in the world space. For agent
Action Space. The action space for each controllable agent is discrete with 5 distinct actions, {lane left, idle, lane right, faster, slower}. Vehicles convert their high-level discrete action orders into sequence of x, coordinates when taking actions. All vehicles' low-level motion models follow the Kinematic Bicycle Model [73], and their kinematic parameters are given in Table 3.
Reward. For DTDE MARL algorithms, each agent receives an individual reward, while for CTDE MARL approaches, all agents receive global reward by summing their individual rewards together. Once agent collides with other vehicles, this agent gets -1 penalty. Agents are encouraged to
18

---

keep right, and an agent gets a linear reward from 0 0.1 with respect to its distance to the rightmost lane. Agents are encouraged to keep a speed within the rewarding speed range of 20 to 30 m/s. At each time step, an agent is rewarded with respect to its speed within the reward speed range. an agent can reach a speed of 30 or higher at this time step, it gets reward of 0.4. an agent keeps speed of 20 lower at this time step, gets reward of O.
D
Visual Results
(a) (Mild) iPLAN: All 5 agents (green) are successful.
(b) (Chaotic) İPLAN: 5 agents (green) are successful.
(c) (Mild) MAPPO: 2 agents (green) are successful. The first 3 agents crash (red vehicles).
(d) (Chaotic) MAPPO: agents (green) are successful. The first, second, fourth crash (red vehicles).
(c) (Mild) QMIX: agents (green) are successful. The first and the last crash (red vehicles).
(1) (Chaotic) QMIX: agents (green) are successful. The third vehicle crashes (red vehicle).
We visually comparc the performance of iPLAN with QMIX and MAPPO. Each baseline tested with multiple learning agents shown in green, and each figure above shows such learning agents from their respective viewpoints. each figure, we show cases when the green agents succeeded versus when they crashed. Conclusion: All 5 agents succeed using iPLAN as shown in Figures 4a and 4b whereas on average 2 more agents crash using QMIX or MAPPO.
Figure 4: Qualitative results Heterogeneous Highway:
19

---

E Implementation Details
Behavioral Incentive Inference. The encoder of the behavioral incentive inference module uses 1-layer GRU network with size of 32 and generates an 8-length vector as latent representation of the behavioral incentive. The decoder uses another 1-layer GRU network with a size of 64 to predict future state sequences, with a dropout rate of 0.1. The truncated length Ea of the observation history is 10 in the Heterogeneous Highway and 5 in Non-Cooperative Navigation. The learning rate for behavioral incentive inference is 1 x 10-1.
Instant Incentive Inference. The encoder the instant incentive inference module uses a GAT with a hidden-layer size of and a 1-layer GRU with hidden-layer size 32. The decoder uses another 32-size GRU predict the trajectory, with dropout of 0.1. The trajectory prediction length ip is 5 in the Heterogeneous Highway and 2 in Non-Cooperative Navigation. The learning rate for instant incentive inference is 2 10
IPPO Controller: The input of the PPO controller for an agent the flattened vector its ob- servation all entities' (vehicles in Heterogeneous Highway; other agents and landmarks Non- Cooperative Navigation) states and the inference of all other agents' (or other vehicles') behavioral incentive and instant incentive. The PPO controller has buffer size of 256 and learning rate of 5 x 10-4 for its actor and critic. All fully-connected and recurrent layers in the actor and critic of
PPO have dimension of 64.
20

---

700
Episodic Reward
800
800
80
40
200
100
0.0
02
0.4
0.6
0.5
1.0
-0.0
0.2
0,4
0.5
0.8
1.0
Time Steps
Time Steps
IPPO-BMFC
PPC-BM
IPLAN-FC
IPLAN
Figure 5: Non--oonooeee Cooperooree Navigation with recurrent and folly-connected behavioral incentive inference modules: Comparing the episodic reward in the (left) easy and (right) hard scenarios. Conclusion: iPLAN (orange) performs better than others in the easy scenario. iPLAN-BM (green) outperforms iPLAN-BMFC (blue) in the hard scenario.
F
Supplementary Experiments: Behavioral Incentive Inference Module
F.
Choice of Behavioral Incentive Inference Module
During our design process for the behavioral incentive inference module, we experimented with different architectures the encoder-decoder framework. Specifically, we tested the usage of recurrent layer and fully-connected layer. While the latter design has been utilized in prior works for similar tasks [68, 69, 65], want address the temporal relationship presented in the historical observation sequences. To evaluate the performance of these two designs, we conduct experiments on the comparison between iPLAN and an alternative approach that uses fully-connected behavioral incentive inference module.
In this module, we take the flattened historical observation sequence as input and employed a 3-layer fully-connected network with hidden layer dimension of 64 the encoder. This encoder generates an 8-length latent representation of the behavioral incentive. Additionally, fully-connected network with the same hidden layer dimension as the decoder to predict future state sequences for opponents. The learning rate for this alternative behavioral incentive inference module set to x 10
another 3-layer
We depict the episodic rewards over both environments in Figure 5 and Figure 6. In these figures, the approach employing the fully-connected network in the behavioral incentive inference module is denoted as iPLAN-FC, and the same notation applies to iPLAN-BMFC The results indicate that incorporating the recurrent layer improves the performance of the behavioral incentive inference module. Specifically, our approach (iPLAN, orange curve) demonstrates better performance than iPLAN-FC (red curve). Similarly, iPLAN-BM (green curve) outperforms iPLAN-BMFC (blue curve) in general.
F.2 Soft Updating Policy
Another important aspect to consider in our behavioral incentive inference module design is the updating policy for behavioral incentives. Drawing inspiration from previous works [68, 69, 65], divide the behavioral incentive inference within an episode into multiple sub-episodes. We aim to update the behavioral incentive inferences at the end of each sub-episode. This updating pol- icy is referred to the hard-updatte policy, in contrast to the soft-updating policy, which treats the behavioral incentive inference as converging procedure and iteratively updates the behavioral incentive inferences.
In our experiments, we evaluate the performance of iPLAN and an alternative method, iPLAN-Hard, which employs hard-updating policy. In iPLAN- Hard, the behavioral incentive inference module updates the behavior incentives at specific time intervals (e.g., f 10, 20, 30...., while the behav-
21

---

80
Episodic Reward
65
65
-0
65
60
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2,00
Time Steps
Time Steps
IPPO-BMFC
IPPO-BM
IPLAN-FC
IPLAN
Figure 6: Heterogeneous Highway with recurrent and folle-connected behavioral incentive inference modules: Comparing the episodic reward in the (left) mild and (right) chaotic traffic scenarios. Conclusion: Approaches using recurrent behavioral incentive inference modules, including iPLAN (orange) and iPLAN-BM (green), outperform those using fully-connected behavioral incentive inference modules.
600
Episodic Reward
600
50
400
20
200
0.0
02
041
0.8
1.0
-8.0
0.2
04
0.6
0.8
10
Time Steps
1w8
Time Steps
168
iPP.BMM-ard
IPPO-M
IPLAN-Hard
IPLAN
Figure 7: Von-Cooperative Navigation with and without soft-updating policy: Comparing the episodic reward in the (left) easy and (right) hard scenarios. Conclusion: iPLAN-BN BM-Hard (bluc) performs the best in the easy scenario and the worst in the hard scenario. iPLAN (orange) has better performance in general.
ior incentive inferences remain unchanged between these updating points (i.e., between t = 10 and t = 20). All other hyperparameters used the behavioral incentive inference module remain the same.
Figure and Figure 8 illustrate the results obtained with different behavior incentive updating poli- cies. In Non-Cooperative scenario but performs the worst in the hard scenario. This significant gap between scenarios may stem from its inability to capture heterogeneity, considering that all agents in the easy scenario are controllable. On the other hand, iPLAN exhibits overall better performance, ranking second in the hard scenario. This outcome demonstrates that the soft-updating pol-
Mit.aatiion.LLNNNM----E achieves the best performance in the easy
easy scemario and first icy helps address heterogeneity and stabilize agents' strategies. Heterogeneous Highway, iPLAN- Hard denotes the approach that notation applies to iPLAN-BM- M-BM-rardd The results reveal that despite the difference in updating poli- cies, their performances remain relatively close in mild traffic for both comparison pairs (iPLANN iPLAN-Hard, iPLAN-BM v.s. iPLAN-BM-Hard inference not available, the use the soft-updating policy leads to substantial improvement iPLAN. As agents become more reliant on their inference of others' behaviors and intentions in highly heterogeneous environment, the reliability and flexibility of their behavioral incentive inference become crucial, enabling them gain better understanding of their surroundings.
hard-updating policy for behavioral incentives, and the same
However, in chaotic traffic, where instant incentive
F.3 GAT-version Behavior Incentive Encoder
We perform an additional experiment that includes an alternative approach that uses a GAT module after the behavior incentive encoder to discuss the possibility of incorporating graphical network in behavior incentive inference that may be helpful in addressing the changing observation set. We
22

---

80
Episodie Reward
65
56
80
60
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2,00
Time Steps
Time Steps
IPPO-BM-Hard
IPO-BM
IPLAN-Hard
IPLAN
Figure 8: Heterogeneous Highway with and without soft-updating policy: Comparing the episadic reward the (left) mild and (right) chaotic traffic scenarios. Conclusion: iPLAN (orange) that uses soft-updating policy for behavioral incentive inference module greatly outperforms its alternative approach iPLAN-Hard (red) that uses hard-updating policy.
350
Episodic Reward
300
290
90
200
40
.00
100
0.0
02
0.1
0.6
0.8
1.0
-1
0.2
0,1
0.6
0.8
1.0
Time Steps
1w8
Time Steps
LPAN
IPLAN-dusi-GAT
Figure 9: Non-Cooperative Navigation with and without graphical attention network in the behavioral Comparing the episodic reward the (left) easy and (right) hard
incentive inference module iPLAN: scenarios. Conclusion: Using a graphical attention network in the behavioral incentive inference module of iPLAN does not help to improve performance.
compare this alternative approach with iPLAN in the easy and hard scenarios of Non-Coopen Non-Cooperative Navigation.
Fig. 9 presents the comparison between the two approaches. According to the result in both easy and hard scenarios, we find that iPLAN (orange) outperforms the alternative approach that uses GAT module inside the behavior incentive encoder, an approach named iPLAN-dual-GAT (blue), with a clear margin between two episodic reward curves. Besides, using GAT behavior incentive inference also leads to slower execution speed due to the additional complexity in the behavior inference module. The result shows that using GAT in the behavior incentive inference does not help to address the changing observation and additional network parameters introduced by the GAT module deteriorate the performance of iPLAN in practice.
23

---

55
IPLAN-L2
Episadic Reward
45
35
Time Steps
Figure 10: Using 1.2 norm in loss function of behavioral and instant incentive inference module of iPLAN: Comparing the episodic reward curve of iPLAN (orange) and iPLAN-L22 (bluc) under the chaotic scenario of Hetterogeneous Highway, with testing episodes results generated frozen models. Conclusion: iPLAN using LI norm loss function in two incentive inference modules performs better than iPLAN-L2 with clear margin between two episodic reward curves.
G
Supplementary Experiments: Inference Module Design
G.1 L2-Norm Loss Function
Regarding the possibility of using alternative loss function design with different L-p norm, we modify the loss function in Eq. (4) and Eq. (5) by using L2-norm, instead of L1 norm, in the loss function for both incentive inference modules. We name this alternative approach as iPI.AN-L22. The new loss functions for both incentive inference modules are:
Behavior incentive inference loss function:
N
Ja. D. Nt.
pi 1
(6)
5=1
Instant incentive inference loss function:
N 4
(5. - of o++++]
Jg. = min mitth Ntp 1=1
(7)
Algorithm
Mean
Std
iPLAN
53.321
9.490
iPLAN-L2
48.182
11.921
Table 4: Standard statistical test: Standard statistical test over the episodic reward of iPLAN and iPLAN- 1.2 under the chaotic scenario of Heterogeneous Highway when training step 950, 000. Perform standard statistical test iPLAN and iPLAN-L2 (iPLAN using 1.2 norm in loss function) Conclusion: iPLAN using norm in the loss function in two incentive inference modules performs better than iPLAN-L2
We perform evaluation experiments under the chuotic scenario of the Heterogeneous Highway. We train iPLAN and iPLAN-L2 models and test both models' performance over frozen models. We perform 32 testing episodes at the testing phase each time. The random seed we use is 59582679.
Fig. 10 shows the episodic reward curve over testing phases performed, while Table. provides the standard statistical test results over frozen models of İPL.AN and iPL.AN-L.2 after 950, 000 training steps. From the result, could conclude that the current loss function design of iPLAN, i.c. using Ll-norm in both incentive inference modules, leads to better performance than the alternative approach using L2-norm in loss functions of both incentive inference modules.

---

80
85
Episodic Reward
85
0.3
35
0.0
02
0.4
0.8
0.8
1.0
1.4
1.8
0.0
0.2
0.4
0.8
1.0
1.2
16
Time Steps
Time Steps
IPLAN-large
IPLAN
Figure 11: Heterogeneous Highway with different learning rates for instant incentive inference module: Comparing the episodic reward in (left) mild and (right) chaotic traffic scenarios (with 1.6M training time steps). Conclusion: Using a smaller learning rate in instant incentive inference (iPLAN, orange) has a better performance in the mild traffic
G.2 Weight Sharing
Regarding allowing weight sharing in iPLAN modules, we design an alternative approach of iPLAN that shares weights between different agents' behavior and instant incentive inference modules. We name this alternative approach Wei---wwight--haringg We perform evaluation experiments under the chaotic scenario of the Heterogeneous Highway. We provide standard statistical test results over at 1, 200, 000 training steps. We perform 32
frozen models of iPLAN and WeiLNN-eight-shaaiee testing episodes at the testing phase each time. The random seed we use 59582679. We compute the p-values of the results by comparing the results of alternative approaches with iPLAN.
Algorithm
Mean
Std
p-value
iPLAN
56.540
10.141
iPLAN-weight-sharing
52.876
11.848
0.195
Standard statistical test over the episodic reward of iPLAN and iPLAN- weight-sharing under the chaotic scenario of Heterogeneous Highway when training step 1, 200, 000. Per- form standard statistical test of iPLAN and iPLAN-weight-sharing.
Table 5: Standard statistical test:
Table. shows the standard statistical test results performed over frozen models after 200, 000 training steps. Results show that performing weight-sharing over inference modules degrades per- formance. This is primarily due the inherent challenges in harmonizing policies within diverse agent team. As discussed in response to Weakness 2.2, even subtle disparities in controller policies can lead to significant variances in incentive inference modules. Weight sharing does not rectify this discrepancy. Furthermore, upholding distinct incentive inference modules without re- sorting to weight sharing effectively manages the innate diversity of the multi-agent system, making the approach more adept for intricate, heterogencous systems.
G.3 Supplementary Experiments: Hyper-Parameter Study
G.3.1 Learning Rate in Instant Incentive Inference
Figure compares episodic rewards when using different learning rates for instant incentive inference. iPLAN (orange curve) uses learning rate of 10 and iPLAN-large (blue curve) uses learning of 14-4, The result shows that using a smaller learning rate in instant incentive inference has better performance in practice.
G.3.2 Hidden Layer Dimension in Behavioral Incentive Inference
Figure 12 presents comparison of the effect of hidden layer dimensions used in behavior incentive inference. In this figure, denote the alternative approach iPLAN that utilizes a hidden layer
25

---

700
Eplsodic Reward
800
600
80
400
40
800
200
100
0.0
0.2
0.4
0.6
0.5
-.00
0.2
04
0.5
0.8
1.0
Time Steps
Time Steps
IPO-BMM-28
PPO-BM
IPLAN-126
IPLAN
Figure 12: Non-Cooperative Navigation with different hidden layer dimensions for behavioral incentive inference module: Comparing the episodic reward in the (left) easy and (right) hard scenarios. Conclusion: Approaches like iPLAN-BM-128 (blue) and iPLAN-128 (red) that use a larger hidden layer dimension for be- havioral incentive inference do not address the heterogeneity well and suffer from the overfitting problem.
dimension of 128 as iPL,NN-28, and the same notation applies to the alternative approach iPLAN-- BM-128 of iPLAN-BM.
In the easy scenario, both iPLAN-BM-128 (blue curve) and iPLAN-128 (red curve) exhibit signifi- cantly better performance than their counterparts using a hidden layer dimension of 64 in the first half of training. However, their episodic rewards experience a substantial decline in the second half, resulting in a lower ultimate episodic reward compared to iPLAN. This observation suggests that these models are overfitting in the easy scenario.
In the hard scenario, iPLAN (orange curve) outperforms iPLAN-128 (red curve) and iPLAN- BLLAN-B--22 (blue curve), the episodic reward of iPLAN-BM-128 begins to decrease when iPLAN's curve is still increasing. This phenomenon demonstrates that using a larger hidden layer dimension does not necessarily lead to performance improvement, as can exacerbate the overfitting problem. Addi- tionally, a larger hidden layer dimension may not effectively address the heterogeneity in a more complex and heterogeneous environment, such as the hard scenario.
Overall, the results indicate that carefully selecting the hidden layer dimension is crucial. While larger dimension may offer some benefits, it can also lead to overfitting and failure in addressing the challenges posed by heterogeneity in certain scenarios.
26

---

60
60
IPLAN-MAPPO
MAPPO
Episodie Reward
Episodic Reward
600
45
40
40
IPLAN-QMIX
30
0.0 02
044
0.6
0.8
1.0
12
1.4
0.0
02
0.4
0.6
0.8
1.0
1.2
14
Time Steps
Time Steps
(a) IPLAN-QMIX
(b) iPLAN-MAPPO
Figure 13: Heterogeneous Highway with the CTDE version of iPLAN: Comparing the episodic reward in the chaotic scenario with approaches incorporating iPLAN (orange) with QMIX (left) and MAPPO (right), Con- clusion: Incorporating iPLANN with centralized credit assignment MARL approaches does not help to achieve better performance.
H Supplementary Experiments: CTDE Version of iPLAN
Regarding evaluating iPLAN with centralized critic (MAPPO) and mixing network (QMIX) in the controller, we perform supplementary experiments that incorporate iPLAN incentive inference module with two CTDE MARL baselines, QMIX and MAPPO. We name the alternative approach combining QMIX and iPLANN as iPL.NNQMIX. Similarly, we name the alternative approach combin- ing MAPPO and iPLAN as iPLAN-MAPPO We evaluate the performance alternative approaches under the chaotic scenario of the Heterogeneous Highway. We compute and visualize the episodic reward curve over rigorous testing phase performed 32 testing episodes using frozen models. The random seed use is 59582679.
Avg. Survival Time (# Time Steps)
Avg. Speed (m/s)
Algorithm
Success Rate (%)
Avg. Reward
QMIX
38.13 + 8.37
54.29 3.12
46.01 5.23
23.50 + 0.30
iPLAN-QMIX
54.38 + 7.79
50.46 = 3.40
64.96 + 3.65
23.88 + 0.19
iPLAN
64.38 + 9.12
56.54 = 3.51
74.92 4.86
21.99 + 0.17
Table Navigation metrics QMIX, iPLAN-QMIX, and iPLAN under chaotic scenario of Heterogeneous Highway: Comparing the navigation metrics of QMIX, iPLAN-QMIX, and iPLAN acquired the chaotic sce- nario over frozen models after 1, 200, 000 training time steps. Conclusion: iPLAN shows better performance than the other approaches, terms of success rate, average episodic reward, and average survival time.
Avg. Survival Time (# Time Steps)
Avg. Speed (m/s)
Algorithm
Success Rate (%)
Avg. Reward
MAPPO
26.88 + 7.06
43.70 3.50
44.60 3.71
29.93 0.02
iPLAN-MAPPO
23.75 + 5.86
42.22 + 3.09
42.20 + 3.29
29.93 + 0.02
iPLAN
64.38 + 9.12
56.54 + 3.51
74.92 4.86
21.99 0.17
Table 7: Navigation metries of MAPPO, iPLAN-MAPPO. and iPLAN under chaotic scenario of Heteroge-
neous Highway: Comparing the navigation metrics of MAPPO, iPLAN-MAPPO, and iPLAN acquired in the chaotic scenario frozen models after 1, 200, 000 training time steps. Conclusion: iPLAN shows better performance the other two approaches. in terms of success rate, average episodic reward, average survival time.
Fig. presents the episodic reward variation throughout training. We train all models for 1.5 million time steps. Fig. 13 (a) presents the result of iPLAN (orange), QMIX (red), and iPLAN- QMIX (blue). The result shows that iPLAN achieves a better overall performance, compared with the other two approaches, and the CTDE version of iPLAN, iPLAN-QMIX, does not achieve better
27

---

performance, compared with QMIX. Fig. 13 (b) presents the result of iPLAN (orange), MAPPO (red), and iPLAN-MPPO (blue). The result shows that iPLAN outperforms the other two approaches with a large margin between episodic reward curves, and iPLAN-MAPPO does not have a better performance compared with vanilla MAPPO.
We also compute navigation metrics QMIX, iPLAN-QMIX, and iPLAN, and MAPPO, iPLAN- MAPPO, and iPLAN under the chaotic scenario. We compute results generated by 32 testing episodes over frozen models after 1, 200, 000 training time steps. Table. 6 shows the results for QMIX, iPLAN-QMIX, and iPLAN and Table. shows the results for MAPPO, iPLAN-MAPPO, and iPLAN. The result shows that iPLAN shows a much better performance than the other two approaches, in terms of success rate, average episodic reward, and average survival time, when evaluating the frozen model after 1, 200, 000 training time steps.
28

---

80
IPLAN
QMIX
55
Episadic Reward
IPPO
50
45
Time Steps
Figure 14: Episodic reward curve of iPLAN and baselines with rigorous testing phase. Comparing the episodic reward curve of iPLAN (orange) and baselines, including QMIX (blue), MAPPO (red), and IPPO (green) under the chaotic scenario of the Heterogeneous Highway, with testing episodes results generated over frozen models. Conclusion: iPLAN shows better performance reward.
other baselines in terms episodic
I Supplementary Experiments: Standard Statistical Test
To address concerns regarding performing the standard statistical tests over iPLAN and baselines, in- cluding QMIX, MAPPO, and IPPO, and perform the rigorous testing phase, we refined our codebase and performed a rigorous testing phase over frozen models every 50, 000 training step. We present the results over testing phases over 32 testing episodes throughout the training. We perform evalua- tion experiments under the chaotic scenario of the Heterogeneous Highway. The random seeds we use 59582679, 763887655, and 312261940.
Fig. 14 shows the Episodic reward curves of iPLAN and baselines when performing rigorous testing phase. From the result, we find that iPLAN (orange) shows better performance than all other baselines in terms of episodic reward and there a clear margin between iPLAN and other baselines without overlap their error bar.
To better present the results, provide standard statistical test. results frozen models of iPLAN and basclines, QMIX, MAPPO, and IPPO, at 200, 000 (Table. 8), 500, 000 (Table. 9), 1, 000, 000 (Table. 10), and 1, 450, 000 (Table. 11) training steps. We compute the p-values of the results by comparing the results of alternative approaches with iPLAN (Tables could be found on the next page).
Standard statistical test results show that iPLAN outperforms all baselines included in our paper in terms of episodic reward, and p-values (< 0.05) suggest results statistically significant.
29

---

Algorithm
Mean
Std
p-value
iPLAN
45.866
11.863
QMIX
44.700
12.022
0.5017
MАPPO
43.317
10.990
0.1261
IPPO
46.463
11.700
0.7271
Table Standard statistical test: Standard statistical test over the episodic reward generated by frozen models iPLAN and baselines, QMIX, MAPPO, and IPPO, under chaotic scenario Heterogeneous Highway when training step 200, 000. Perform standard statistical test of iPLAN and iLLAN-weighhtssaaiig..
Algorithm
Mean
Std
p-value
iPLAN
50.568
10.379
QMIX
44.455
13.008
4.357 x 10 10m4
MAPPO
41.855
10.785
5.150 x 10 10
IPPO
53.986
12.287
3.968 x 10-2
Table Standard statistical test: Standard statistical test over the episodic reward generated by frozen models of iPLAN and baselines, QMIX, MAPPO, and IPPO, under the chaotic scenario of the Heterogeneous Highway when training step 500, 000. Perform standard statistical test of iPLAN and iPLAN- weiigt--hariing.
Algorithm
Mean
Std
p-value
iPLAN
54.445
11.718
QMIX
46.579
13.555
2.978 x 10-5
MАPPO
41.911
10.192
2.681
10-13
IPPO
50.836
12.023
3.751 x 10
Table 10: Standard statistical test: Standard statistical
the episodie reward generated by frozen mod- els iPLAN baselines, QMIX, MAPPO, and IPPO, under chaotic scenario the Heterngeneous High- way when training step = 1, 000, 000. Perform standard statistical test of iPLAN and iPLAN--eight-shaarin
Algorithm
Mean
Std
p-value
iPLAN
53.514
11.252
QMIX
47.695
13.002
1.162
10-3
MAPPO
42.903
9.401
3.160
10-11
IPPO
49.502
10.207
1.080 x 10
Table 11: Standard statistical fest: Standard statistical test over the episodic reward generated by frozen mod- els of iPLAN and baselines, QMIX, MAPPO, and IPPO, under the chaotic scenario of the Heterogencous High- way when training step = 450, 000. Perform standard statistical test of iPLAN and iPLAN-weight--harinng
30

---

IPLAN
MAPPO
Episodic Reward
34
32
30
26
26
Time Steps
Figure 15: iPLANN evaluation under an advanced chaotic scenario of Heterogemeous Highway: Comparing the episodic reward of iPLAN (blue) (denote iPLAN- VH) in the chaotic- VH scenario with the episodic reward of iPLAN (orange) and MAPPO (red) in the chaotic scenario. Conclusion: iPLAN shows converging trend in the chaotic- VH scenario with lower episodic reward than the other two.
J Supplementary Experiments: More Complex Traffic
Regarding the possibility of implementing iPLAN under a more complex domain, we perform supplementary experiment that evaluates iPLAN under a more challenging traffic scenario. This advanced setting, which we have termed chaotic-VH,, mirrors the existing traffic distribution of behavior-driven vehicles (Normal: Aggressive: Conservative scenario, but with vehicle density that is twice dense previously studied chaotic scenario. Due to computational constraints during the rebuttal phase, our exploration limited to single random seed over million time steps. Despite this, the results of iPLAN evaluated under chuotic-Vil are promising.
3) in the current chaotic
Fig. 15 shows the episodic reward curve for all three experiments, while Table. 12 provides the nav- igation metrics over these approaches evaluated over 32 testing episodes frozen models trained for million time steps. We observe that though having lower episodic reward curve than it used to be in chaotic, iPLAN outperforms MAPPO in iPLAN- VH. in terms of episodic reward curve, average episode length, and success rate.
Avg. Survival Time (# Time Steps)
Avg. Speed (m/s)
Algorithm Success Rate (%)
Avg. Reward
iPLAN
50.63 + 9.33
41.15 + 4.38
56.19 + 6.62
19.77 + 0.88
MАPPO
18.13 + 4.37
32.49 + 2.80
35.80 + 3.42
24.02 +: 0.92
Table 12: Navigation metrics iPLAN and MAPPO under an advanced chaotic scenario Hetero-
geneous Highway: Comparing the navigation metrics of iPLAN and MAPPO acquired in the chaotie-VE VH scenario Conclusion: iPLAN shows promising performance under the chaotic- VH scenario as it has better performance than MAPPO.
31

---

K Further Discussion of Empirical Results
Centralized versus Decentralized Training Regime. In this work, we operated the decentral- ized training regime, based on the assumption that agents should learn navigation policies in DTDE manner without centralization in training. Empirically, we find that CTDE MARI. approaches per- form worse as the environmental heterogeneity increases due to the absence of consensus among agents in heterogeneous environments. On the other hand, the awareness of opponents' strategies becomes more important in agents' decision-making when the environment is heterogeneous, espe- cially the awareness of agents' instant reactions to surroundings. This need for increased awareness makes intent-aware distributed MARL algorithms perform better in these environments.
To further investigate the empirical performance of CTDE and DTDE approaches under our prob- lem setting, conduct experiments integrating two incentive inference modules of iPLAN with two CTDE approaches, QMIX and MAPPO, and compare its performance with iPLAN and other base- lines. We include the experiment details and results in Appendix H. Results show that integrating iPLAN inference module in CTDE approaches does not help to achieve a better performance in the chaotic scenario the Heterogeneous Highway than the current DTDE version of ÍPLAN.
Decoupled Incentive Inference. Individually, the incentives yield some benefit over a baseline controller. For example, find that both the behavior and instant incentive inference modules individually help to achieve higher reward, especially in more heterogeneous environments (See Figure 2). However, our system works best when both incentives are jointly activated, for example in Table 1, we find that the success rate drops significantly for iPLAN-GAT, compared to iPLAN (61.88% versus 67.8 81).. This clearly indicates autonomous vehicles need the behavior incentive module to survive the more heterogeneous chaotic traffic scenario.
32


Reviews:
[{"review_id": "ZYD5HDfLyr", "reviewer": "Reviewer_htJq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a MARL approach for driving where each agent acts based on a combination of 'behavioral' incentives, which govern preferences over driving style, and 'instant' incentives, which govern the local preferences over avoiding collisions. Agents learn to predict the incentives of other agents while they learn to act. They evaluate in multi-agent driving domains and demonstrate performance improvements.", "summary_of_recommendation": "Overall, I think the idea has promise, but I worry that the presentation/evaluation is not quite where is should be. If the authors can address the presentation issues, I think the paper should be accepted. I think a more clear demonstration on a more complex domain could also improve the paper's impact. \n\nUpdate after rebuttal/discussion: I appreciate the author’s efforts during the discussion to address my concerns about presentation and to better scope the claimed contributions. As a result, I am happy to raise my score and confidence level in the result.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "## Can you describe an example of what the behavior and instant incentives are?\n\nI had trouble following this and would benefit from an example. I think this something like this would be good to include in the paper to make it more accessible. \n\n## Please address the clarity issues I raised in my review. \n\nI think the idea presented is promising, but I think that some issues with the presentation need to be addressed before I recommend acceptance.\n\n## Are there any experiments you could run to test the value of using hierarchical incentives?\n\nI'm curious how this would compare to a simpler method that infers a single set of incentives for each agent.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Strengths\n\n## Originality\n\nThe paper proposes an interesting idea: where agents in a heterogenous MARL environment learn opponent models. Given that there is lots of work on driver modeling for autonomous driving, this is a promising approach for their domain. I would be curious to see how this handles human driving behavior.\n\n## Significance\nThe underlying idea has promise and so I think there is some upside for a high-impact paper.\n\n## Quality\n\nAlthough there is some room for improvement in presentation, the experiments seem to be a reasonable test of the method. \n\n# Weaknesses\n\n## Clarity\n\n[major] I had trouble following the overall presentation of the inference training method. I think that a more clear example of the method and edits to Algorithm 1 would help the reader follow and/or replicate the method. \n\n[minor] the typesetting in lines 171-182 is a bit strange due to the inline math.\n\n[major] I had trouble understanding the practical difference between the behavioral and instant incentive. Maybe a better description of the method would help, or a more clear caption. E.g., consider rephrasing the sentence \"The instance incentive inference... [uses data] .. to infer other vehicles instance incentives.\" It would be helpful to make it extra clear what an example of the instant incentives is. \n\n[major] there's not enough description of the different environments in the main paper, I recommend restructuring the paper so you can include a paragraph describing the environments in more detail", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "YIOBep5sffl", "reviewer": "Reviewer_qQsj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper focuses on path planning for autonomous vehicles, by training agents (individual vehicles) to learn the driving styles (termed “incentives”) of neighboring vehicles, to both make short-term path predictions about other agents and finally reason about their own path decisions. The work is demonstrated in simulation on two different traffic scenarios (each with two levels of difficulty), where it is compared to three standard algorithms (QMIX, MAPPO, and independent PPO (iPPO)) as well as with a few ablation versions of the proposed approach.", "summary_of_recommendation": "While this paper considers a relevant problem, which it approaches along an interesting and novel axis, I worry that the current writing is still built upon shaky premises and inaccurate statements that weaken its story. Some of the key design considerations are also debatable and may merit to be more thoroughly explained and/or investigated for further performance improvement. Results are remain questionable, as no actual testing phase was carried out and algorithms were only tested via their trend at the end of the training phase, thus on potentially widely different scenarios. Performance differences also remain rather marginal. All in all, while I believe this work has a lot of potential, I also think that it needs more time to be polished, and I sadly cannot recommend it for publication at CORL 2023 in its present form.\n\nEDIT: Authors have made significant improvements to the paper, and after rebuttal I believe it is now ready for publication.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Have authors tried their proposed framework with weight sharing? If so, how do the performance compare to the proposed approach without weight sharing? What are the potential reasons behind these differences, in your opinion?\n\n2. How about testing the proposed framework with a centralized critic (in the controller), and/or a mixing network (also in the controller)? Nothing in the proposed approach actually limits such implementation, and this could also help shed light into the potential advantages of extending the agents’ state with incentive information.\n\n3. Can authors present results of a standard statistical test between the performance results of the different baselines? Current confidence intervals often overlap, thus lowering the confidence the reader can have in the significance of these differences.\n\n4. More importantly: can authors present results of a proper/rigorous testing phase, where frozen models are compared on the same set of randomized experiments, with random seeds fixed? This should also include the same statistical tests for completeness.\n\n5. Can authors discuss the choice of reconstructing a very high-dimensional raw state in the incentive inference modules, and the advantages/limitations of such an approach? Are there other options there, and/or potential directions from the (still very early) field of model-based MARL?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work considers a relevant problem for the community, and focuses on an interesting approach to tackle it: driving style estimation (i.e., essentially agent modelling) to enable and rely on predictions of the other agents’ short-term actions. I found the paper overall pleasant to read, and in particular appreciated the rigor (yet readability) of Section 3. That being said, I have several concerns with the actual premises of the work and with the results, which I believe call for more time to refine the overall story and the evaluation.\n\nIn the introduction, authors state that existing MARL algorithms can be classified as either centralized (e.g., QMIX or MAPPO) or distributed (e.g., iPPO). However, this does not seem to be what is generally accepted in the field, as far as I know. “Centralized” approaches like QMIX/MAPPO only rely on information centralization during training (e.g., mixing network, or centralized critic) but remain entirely decentralized at runtime. Instead, one generally refers to QMIX/MAPPO as “centralized training, decentralized execution (CTDE),” a paradigm that could be well-suited to many AV scenarios since training is often done in simulation (where centralization is not an issue), while offering the desired purely decentralized execution capabilities and scalability at deployment time. Therefore, I was first confused by this statement and premise.\n\nAuthors then follow up by stating that distributed approaches like iPPO often rely on tricks like parameters sharing, which cannot apply in AV path planning (but without substantiating why). However, the whole approach then described in this work, which consists of three modules (behavioral incentive inference, instant incentive inference, and controller/policy), has two of these modules completely symmetric and agent-agnostic, thus naturally allowing (and likely even benefitting) from parameter sharing! That is, while each vehicle’s policy network (controller) may justifiably want to assume independent weights for heterogeneity, the inference modules could and likely should share weights, since these modules all ask the same question, e.g., “based on what your neighbors did recently, how would you classify their intention?” for the first module. More to this point, one could even argue that the policy network (controller) could also use parameter sharing without drastically restricting performance/generality…\nThere again, I am confused by the overall premise of the work, which drastically weakens its contribution. It seems to me that the paper relies on “perceived”/”stated” requirements of the proposed solution (no “centralization” during training, no parameters sharing) that I am not convinced have reasons to be. This distracts the reader from appreciating the actual contribution of the work, i.e., the proposed planning framework, which does not really need/builds upon any of these requirements.\n\nResults then compare the proposed approach and its ablation variants to standard baselines (MAPPO, QMIX, and iPPO), but does so in random training scenarios instead of freezing the models and running them through a rigorous, fixed set of randomized experiments. As a result, I am not convinced by the results, where many of the curves also seem to overlap and where algorithms were tested on potentially widely different scenarios. This is not the way algorithms are/should be compared in our field, and I believe a more rigorous testing phase is necessary.\n\nFinally, I also have major concerns with the choice of design for the incentive inference modules, as learning to reconstruct/predict the full raw state seems like a rather flawed process: since each agent’s state also contains information about neighboring agents, learning to simply L1-reconstruct/predict such state may not allow agents to really focus on the most important/key portions of it. As a result, agents may often learn to grossly reconstruct/predict states, which may involve “small” yet very impactful mistakes, leading to very poor incentive learning and downstream policies. This is a common issue in many (single-agent) model-based learning methods that rely on similar encoder-decoder structure, and I believe that the fact that this paper considers a multi-agent problem only makes this issue more important. I also believe that this aspect of the work may be the one that, once properly addressed, could lead to the largest performance improvement.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "IWDx70JWFY", "reviewer": "Reviewer_vQKv", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper considers autonomous driving in a decentralized multi-agent setting where agents vary in their instant and behavioral incentives. The ego agent is unaware of these incentives but can estimate them from observations. A two-stream architecture for incentive inference is proposed. Experiments in two non-cooperative simplified autonomous driving domains compare to several baselines and ablations.\n", "summary_of_recommendation": "Overall, the paper is very well written and carefully done. If the results were a bit stronger, particularly in terms of including more trials (random seeds), I would consider recommending strong accept.\n\n### After Rebuttal\n\nThe original submission was solid, and the paper was improved substantially during the rebuttal period. I appreciate the engagement of the authors and the other reviewers.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Can you clarify the relationship between the environments used in experiments and those in prior work? To what extent are the environments new or changed in this work?\n2. Is my understanding about the fixed set of agent observations mentioned above correct?\n3. The main disadvantage of centralized approaches versus decentralized ones seems to be practical infeasibility. In the experiments though, feasibility concerns can be cast aside because we’re in a simple simulation setting. Nonetheless, we see that the proposed decentralized approach actually does better than centralized baselines. What should we conclude from this? As a thought experiment, if we ignore feasibility concerns, would it be possible to make a centralized version of iPlan that actually performs better than any other method in this paper? Or is there something about decentralization that actually improves performance?\n4. Would it be accurate to say IPPO-GAT-BM = iPlan? I ask because I am wondering if IPPO-GAT and IPPO-BM are truly ablations of iPlan, or more like extensions of IPPO. My preference would be true ablations.\n5. Can you comment on the proposed approach from a Theory of Mind perspective? My understanding is that the ego agent reasons about opponent’s and opponent’s opponents. Does the “recursion” stop there? Or is the system somehow able to reason about opponents reasoning about opponents reasoning about opponents and so on?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n\n* The problem setting is very clearly stated. I like the framing of the problem as a HiP-POSG. Overall, the formalisms are precise and the notation, while extensive, is helpful in my opinion.\n* The problem setting is also incredibly difficult. Getting something to work at all with multiple agents and partial observability is a feat.\n* The related work is very thorough and the paper seems well positioned in the literature.\n* Besides a few details mentioned below, the figures are very helpful.\n\n\n## Weaknesses\n\n* 3 random seeds is not sufficient to draw meaningful conclusions.\n* Relatedly, the empirical results provide only tentative support for the main hypothesis that decoupling behavioral and instant incentive inference is better than a unified inference scheme. In particular, the performance of iPlan versus IPPO-GAT does not seem convincingly better.\n* Although a graph attention network is used to model instant interactions during instant incentive inference, my understanding is that each agent’s observation set is otherwise fixed in terms of the opponents that it is tracking. A more realistic setting would have the observation set change, and the model would need to be updated accordingly -- for example, the behavior incentive inference model may also need to be a graph neural network. I don’t think the model needs to be changed for this paper, but I do think this assumption should be clarified. I also think that this criticism of other works should be applied to this work as well: “Other works… require the involvement of the same group of agents between episodes, which is not practical in the real world.”\n* In Figure 1 and part of the text, the behavior incentive inference model is said to “reconstruct” historical states. I think the use of the word “reconstruct” here is quite confusing. As far as I understand (from L203 in particular), the model is actually _predicting_ future states from earlier states. Alternatively, we could say that it is reconstructing _trajectories_, rather than states, which made me think that individual states are being auto-encoded (which wouldn’t make sense). I was especially confused in Figure 1 because it appears that single time steps are being reconstructed, since only $t$ appears in the orange box.\n* The GIFs in the supplementary material are helpful to understand the environments tested, but they are also a bit overwhelming without context, especially given that multiple animations are concatenated together and each individually is quite busy. It would be very helpful to have the video draw the viewer’s attention to the important parts.\n* I like the separation between behavioral incentives and instant incentives, but there is a third component of the hidden state -- vehicle type / kinematics -- and it’s not clear where that component comes into play. I wonder if a third “stream” would make sense. For example, incentives drift over time (fast and slow), but kinematic constraints are almost always constant. I also want to better understand where this component manifests in the current architecture. Is it one or both of the incentive streams?\n* (I wrote the following, but then I realized that my concern was addressed in the experiments with the inclusion of randomly-behaving agents. I’m including the concern anyway in case the authors want to head it off earlier in the paper.) Other decentralized approaches that use parameter sharing or global rewards are criticized in the introduction because they “introduce strong and unrealistic assumptions for applications like autonomous driving.” I am not completely convinced that these mechanisms require stronger assumptions than the ones required here. This work assumes that all of the agents are being trained using the same MARL approach. This posits that all cars on the road are autonomous and that there is one autonomous driving company that determined the software on those cars. As far as I understand, parameter sharing and global rewards would only require the same assumption. Perhaps the approach in this work could be extended to a setting where not all agents are using the same MARL algorithm, but we would need to see experimental evidence for that (nevermind!).\n\n### Minor\n* Equation 1 uses $\\mathcal{F}_i$ but the subsequent text uses $f_i$. Similarly there is some switching between $\\mathcal{R}_i$ and $r_i$.\n* There are a few awkward word choices in the paragraph L207-216: “temporal relation”, “blank prior”, “authentic estimation”. ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a multi-agent RL approach in the context of path planning of autonomous vehicles to learn behavior of neighboring vehicles. 

The reviewers have found that the proposed approach is interesting and technically sound. The reviewer concenrs on readability and contribution has been sufficiently resolved during the rebuttal phase.
Decision: Accept (Oral)