Adversarial Driving Policy Learning by Misunderstanding the Traffic Flow
Authors: Dongkun Zhang
Jintao Xue
Jingke Wang
Yunkai Wang
Yuxiang Cui
Eryun Liu
Wei Jing
Rong Xiong
Junbo Chen
Yue Wang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Acquiring driving policies that can transfer to unseen environments is essential for driving in dense traffic flows. Adversarial training is a promising path to improve robustness under disturbances. Most prior works leverage few agents to induce driving policy's failures. However, we argue that directly implementing this training framework into dense traffic flow degrades transferability in unseen environments. In this paper, we propose a novel robust policy training framework that is capable of applying adversarial training based on a coordinated traffic flow. We start by building up a coordinated traffic flow where agents are allowed to communicate Social Value Orientations (SVOs). Adversary emerges when the traffic flow misunderstands the SVO of driving agent. We utilize this property to formulate a minimax optimization problem where the driving policy maximizes its own reward and a spurious adversarial policy minimizes it. Experiments demonstrate that our adversarial training framework significantly improves zero-shot transfer performance of the driving policy in dense traffic flows compared to existing algorithms.

Under review as a conference paper at ICLR 2023
ADVERSARIAL DRIVING POLICY LEARNING BY MISUNDERSTANDING THE TRAFFIC FLOW
Anonymous authors Paper under double-blind review
ABSTRACT
Acquiring driving policies that can transfer to unseen environments is essential for driving in dense traffic flows. Adversarial training is a promising path to im- prove robustness under disturbances. Most prior works leverage few agents to induce driving policy's failures. However, we argue that directly implementing this training framework into dense traffic flow degrades transferability in unseen environments, this paper, we propose a novel robust policy training framework that is capable of applying adversarial training based on coordinated traffic flow. We start by building up a coordinated traffic flow where agents are allowed to com- municate Social Value Orientations (SVOs). Adversary emerges when the traffic flow misunderstands the SVO of driving agent. We utilize this property to formu- late minimax optimization problem where the driving policy maximizes its own reward and a spurious adversarial policy minimizes it. Experiments demonstrate that our adversarial training framework significantly improves zero-shot transfer performance of the driving policy in dense traffic flows compared to existing al- gorithms.
INTRODUCTION
Policy learning in dense traffic flows is a progressively active area for both academia and industry community in autonomous driving (Dosovitskiy et al., 2017:Suo et al. (202I). Since training driving policy in real world is costly, researchers aim build dense traffic flows simulation as an alter- native (Cai et al. 2020 Pal et al. 2020) Wu et al.) 1021). Peng et al. (202T) develops traffic flow that exhibits altruistic behaviors and training driving policy in such coordinated flow also performs well. However, the internal dynamics of different traffic flows are varied, making it difficult to train driving policy in one flow and transfer it into unseen traffic patterns. Hence, is indispensable to develop robust driving policies that can transfer among different traffic flows.
An appealing technical route to improve the robustness driving policy is adversarial attack (Pinto) et al. 2017), which models differences between training and evaluating environments extra dis- turbances towards driving policy (Wachi 2019) Chen et al., 2021 Liu et al.) 2021; Huang et al.] 2022]. To exert disturbances driving policy, these works leverage few agents deliberately in- duce driving policy's failures. Although working well in sparse traffic situations, this pipeline cannot extend to dense traffic flows. On the one hand, increasing the number of attacking agents makes ad- versarial attacks easier, yet is harder for the driving policy to resist such strong disturbances, which severely harms policy learning. On the other hand, attacking agents mainly concentrate on producing adversarial behaviors towards driving policy, while overlooking the modeling of altruis- tic behaviors among them. Therefore, the key to construct coordinated traffic flow which still generates adversarial behaviors.
We develop coordinated traffic flow with communication and propose misunderstanding- adversarial training pipeline based on this flow. Specifically, for building a coordinated traffic flow, we introduce the concept of Social Value Orientation (SVO) (Liebrand) 1984) in social psychology which balances egoistic and altruistic behaviors for each agent. SVO can be regarded as the hidden information of one agent, which typically cannot be accessed by other agents. However, in this paper, allow agents in traffic flow communicate genuine SVOs with each other. Since the traffic flow is served as a testbed for training and evaluating driving policies, the coordination mechanism within the traffic flow is invisible driving policies.

---

Under review as a conference paper at ICLR 2023
Coordinated Traffic Flow via Communication
Adversary via Misunderstanding
Bottlerack
Marge
Without
SVD
Intersection
Roundabout
Adversary via Misunderstandinn
0
0-
((t-
Driving Agent
Baskground Agents
Adversarial Agent
Figure I: Overview of our training framework. Left: We build up coordinated traffic flow in which agents communicate SVOs coordinate with each other. Right: By disturbing the SVO of driving agent, our traffic flow exhibits adversarial behaviors towards the driving policy.
In other words. when placing a driving policy to interact with the traffic flow, the traffic flow requires receiving driving policy's SVO while the driving policy is unaware of traffic flows' SVOs. This property offers neat approach to induce misunderstandings between driving policy and our traffic flow, making it adversarial towards driving policy. We reserve a spurious adversarial agent to disturb the SVO delivery from the driving agent to other agents and formulate minimax optimization problem where the driving policy maximizes its own reward while the spurious adversarial policy minimizes it, as shown in Figure
Contributions. We propose novel adversarial training framework based on a coordinated traffic flow to obtain driving policies that can transfer across various traffic flows. We develop a coordinated traffic flow where agents exhibit egoistic, prosocial, and altruistic behaviors based on communicat- ing SVOs with each other. Based on this traffic flow, we apply adversarial driving policy training by adversarially misunderstanding the traffic flow, which is disturbed to produce improper coordi- nated behaviors towards driving policy. We investigate characteristics of several traffic flows in four challenging scenarios and carry out comprehensive comparative studies evaluate the robustness of driving policy. Results show that our traffic flow achieves the highest success rate and the proposed adversarial training pipeline significantly improves the transfcrability of driving policy compared to existing al algorithms.
RELATED WORK
Dense traffic flows. Prior works explore different methodologies to simulate dense traffic flows
including rule design (Behrisch al. 2011: Dosovitskiy al. [017] Cai et al. 2020 Zhou et al.) 2021]. Imitation Learning (IL) (Zhao et al. 2021: Gu et al.) [022]] Wang et al. 2022). and Multi- Agent Reinforcement Learning (MARL. (Pal et al. 2020: Palanisamy] 2020 Wu al. 2021). II. naturally leverages numerous human expert data but suffers from severe distribution shift and poor closed-loop performance even in simple scenarios. Most rule- and MARL-based algorithms aim to simulate individual behaviors of distinct agents, which overlooks complex interactions among agents. Similar to our work, |eng et al. (2021) also builds coordinated traffic flow based on SVO. However, agents in their traffic flow have no access to other agents' SVOs, leading to conservative behaviors.
Adversarial attack. A common way to acquire robust policy is applying Robust Adversarial Re- inforcement Learning (RARL) (Pinto et al.) 2017]: Pan et al., 2019 Vinitsky al. 2020, Oikarinen
et al.
202T). Researchers in autonomous driving also follow this pipeline (Wachi] 2019 Ding et al.]
2020
Chen et al. 2021 1 Sharif & Marijan 202 Huang et al.) (022). Adversarial policies in Ding
ct al.
)020);; Chen et al. (2021) Sharif & Marijan (2021): Huang al. (2022) are optimized to collide with driving agent, while Wachi (2019) Huang et al. (002)) attempt to expel ego agent from drivable areas. Using attacking agents interfere with driving policy deliberately, such pipeline provides large adversarial disturbance for driving policy. However, excessively concerning rarely happened scenarios harms the robustness of driving policy in unseen environments since it fails
2

---

Under review as a conference paper at ICLR 2023
to capture simpler yet non-trivial interactive patterns. In this work, we apply adversarial training framework on a coordinated traffic flow with communication to solve this problem.
TRAFFIC SIMULATION CONSTRUCTION
3.1 PROBLEM SETTING
Partially Observable Stochastic Game (POSG). Traffic simulation systems are typically formulated as a POSG (Olichock & Amato 2016). Formally, POSG is a tuple G R, Po: O. nn.y,") n IS the number of agents. I denotes the set 10.1...., S is the state space. A is the joint action space of n agents and A = X,ezAA,- P A A(S) the state transition probability. R. = Ro, R1...., R denotes the set of agent-specific reward functions and Rg S A + R is bounded for all E I. Note that each agent i receives distinct reward from its own reward function r; Rs a). Po E A(S) is the initial state distribution. O the joint observation space and O xzezO,- Y E (0, is the discount factor, and T is the time horizon. In POSG, each agent i maximizes its own expected cumulative reward via policy 3 O, + (A.)) When n becomes large, it is time-and-space consuming to optimize a set of poli- cies B = [33. .r...... 3n-1) To solve this problem, we simply adopt parameter sharing strategy (Terry et al. 2020p, i.e., B. = B, with the help of neural network which has powerful representation ability.
Incorporating Social Value Orientation (SVO). From the perspective of social psychology, agents should consider surrounding agents" rewards to achieve coordinated driving. Following Schwarting et al. (2019); Buckman et al. (2019); Peng et al. (202T) we introduce the concept of SVO to model coordinated behaviors among agents and build up coordinated traffic simulation. By incorporating SVO, each agent maximizes reward with the consideration of other surrounding agents:
R cos(cc)) R, Ris
where Rs. = EjETS, R5/[Is, I. Is, is the set of surrounding agents w.r.t. agent i. c, E [] is the SVO of agent and kept fixed during each episode. Given equation embedded POSG G" (I, A, R', C, Po. O, 7, T". R' (Ry , R,,, set of SVO-embedded reward functions. C = (Co, C1, Cn 3 is set of all SVOs.
[ we formulate a SVO- denotes the
Problem formulation. As one can see, SVO determines the trade-off between egoistic and altruistic behaviors. For each agent, it is necessary to recognize SVOs of itself and other agents, which provides the ability to infer other agents' reward structures. Therefore, we design policy as , : O, x (Xjezs, C₁)= ((A.). And we use single policy / to optimize the sum of optimization objectives SVO-embedded POSG:
max Eg P a B
y (R (p 4)
44 E C
3.2 KEY COMPONENTS
State space. Agents driving in dense traffic flow need to continually interact with surrounding agents. Besides, road structures also influence agents' decisions. Therefore, state space S needs cover a collection of static and dynamic elements. The set of static elements E include lane centerlines, sidelines, agents' global paths, i.e., E = (centerline, sideline, path The set of dynamic elements Ed include current and historical poses and velocities (trajectories) of all agents, i.e., Fd = [(rajecloryy Irajeclory,.... ... trajecloryn-1} We utilize vectorized representation based on Gao et al. (2020), which is computation- and memory-efficient. in E and E are sets of points containing corresponding features. Specifically, static element J. E E V, = P,, 1,,4, 4,4,, i. where P, = (z, y, 0) is the pose of point j in element and ( is the lane width of element i. For points in dynamic elements, vy (p,. C. i, where pj = (a. y. 0, v) and e; denotes the SVO of agent i.
our work, elements
ex = ( U1...., U.......
( denotes the set of prohahility distribution over X.
3

---

Under review as a conference paper at ICLR 2023
Observation space. In POSG, each agent could only receive perceptual information locally, L2 norm to define locality, i.e. agent could only receive points (Ze, Me) that (I yo) -z)e ye Be) S d, in which (z;, y4) is the current location of agent i.
Design of R. The goal of each agent in dense traffic flow is homogeneous, for instance, all agents want successfully finish the task fast possible. Besides, since each agent receives O, design- ing R3 upon O; rather than S benefits policy training. In our work, we use self-motivated reward R. : O; A; + R and Rt R for all E Z. However, designing self-motivated reward function still remains an open problem. Designing fine-grained dense reward accelerates training procedure but relies too heavily on human knowledge, while training with coarse sparse reward requires much more data. To combine both benefits, we design a near---ppae sparse reward function containing dense re- ward for incentive driving fast and a sparse reward for penalizing catastrophic failures. Catastrophic failures include collision with other agents, deviation from drivable area, driving too far from global path, and crashing into wrong lane. Coordinated behaviors could be produced by incorporating SVO.
Policy training. We apply Independent Policy Learning (IPL) TTan 1993) to solve Equation Although IPL is prone to generate egoistic suboptimal behaviors (de Witt et al. 2020 Yang et al., 2020), we could alleviate this problem by incorporating SVO, which forces the algorithm to consider other agents' goals.
3.3 POLICY ARCHITECTURE
To better extract static and dynamic features and capture relations among them, we utilize a hier- archical feature extraction framework. We use DeepSet (Zaheer et al. 2017) aggregate homoge- neous information within dynamic and static elements, followed by Multi-Head Attention (MHA)
Vaswani al. 2017) further extract heterogeneous information among different elements.
Homogeneous feature aggregation. Consider the elements set e C E, E = {E", Edd, and the function processing on the set needs retain the adjacency between elements and permutation- invariant to the order of objects in the element. Based on theorem 2 in (Zaheer et al. 2017). the propagation function f is defined as:
f(e) = Σ )(v) DEE
(3)
And we obtain the element level features l, [e). where is the input elements set, the nodes transformed into a representation d(v). The sum of representations is processed using the p network defined by Multi-Layer Perception (MLP) network. In our implementation, DeepSet can extract polyline-level features while not introducing too many parameters.
Heterogeneous feature aggregation. The static element level features 1 = 118 Ie....., (iy.....,....... .......
and the dynamic element features l [td, l2 "e.,,.. dd .... go through MHA layer which takes into account their inter-relations to output the final action for the agents. Given arbitrary fea- ture matrices w, 2 and their linear projections wQ, Wk wy and zq, 2k, 20, the Self Attn(w) and ProssAttn(us 2) defined as:
Self All((()) Softmar (wQw*)
Vdk
WV
CrossAAt So Softmax (wQ=*) 2V
(4)
Vdh
where ddl is the dimension of the key vectors. We leverage the one-layer cross-attention network to model the interaction between dynamic and static segments. The dynamic elements features l and static elements features 15 are fused by the Sel Attn and CrossAttn operation:
Alln(1d g Sel Altn((d) + Cross ClossAAll([1,, 12)
15 SelfAUln(( + Cross Alln(1 (1,,, (5)
(5)
1, 1:) is the final output of MHA. We then decode the agents' action from lo:
(16)
(6)
where pp.) is the action decoder, and a E A. For simplicity,
ML. the decoder function.
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Miisuddrsstandnng-bane Adversarial Reinforcement Learning
Input: SVO-embedded POMDP M containing traffic flow policy Output: Driving policy x, adversarial policy Te. Initialize: Learnable parameters 0z. Or.
for 1.2..... .N do
/+ Stage 1: Given Te optimizo do Collect a set of transition tuples {(o, a, r)] trajectories by rolling out n and we on M; Optimize parameters 9, of ι using any RL algorithms;
for n1 = 2,.,...,
Stagge for no = 2,2,..., N₂ do
Given optimize We
Collect a set of transition tuples {o, ca, -r) trajectories by rolling out т and Te on M;
Optimize parameters 0r. of nee using any RL algorithms;
MISUNDERSTTANDNGGBASER ADVERSARIAL LEARNING
4.1 PROBLEM SETTING
For single-agent driving task, we formulate SVO-embedded Partially Observable Markov Decision Process (POMDP) as M (S, A, P. C, Po: Po: O, 7, T, (3). Note that , is the policy that controls the traffic simulation and affects state transition probability P. Following the tradition, define policy x : O + A(A) to solve M. Ca E = [0, ブ is the SVO of driving policy which taken by 3. The genuine SVO of driving agent Cr is always since existing single-agent algorithms fully self-interested. Adversary emerges when 03 and C differ:
T
max min Eg Pa.eg Aeou
r R(ss at)
t=
In section 3.1 and equation 7 the SVOs are invariant during one episode for the reason of stabilizing training. However, in adversarial training, we aim to destabilize policy training. Therefore, we introduce a spurious policy Te : O + A(C) to produce eæ which is allowed to change across time steps, changing equation] into:
max min E. Reg. Peg.r Panaa TI
R(ss, at)
(8)
Note that equation 8 relates to three policies including driving policy т, background policy 3, and spurious adversarial policy Th. Since driving policy maximizes its own reward, it is egoistic from the perspective of social psychology. Background policy controls the whole traffic flow to exhibit egoistic and altruistic behaviors. The spurious policy is the only that aims to generate adversarial behaviors by minimizing driving policy's 's reward.
We highlight that agents in our traffic flow try to coordinate with each other, which is fundamental difference compared to previous attacking agents. Instead of deliberately inducing failures of x, we keep 5 non-adversarial and leverage an extra ne to disturb the SVO of taken by B.
4.2 ADVERSARIAL POLICY TRAINING
Algorithm I outlines our training framework. Given background policy 8, we alternatively optimize both driving policy и and adversarial policy Te- The parameters 02 of and 0z, of n. are randomly initialized before training. In each of N iterations, first optimize 0₂ and keep 0,._ fixed, followed by optimizing 0x and keeping 0g fixed.
In Stage I, we iterate N1 times to optimize driving policy By sampling POMDP M, we collect a set transition tuples {(o, a, d.rr). where o, E O, E A, and r = R(o, a). We then apply standard RL algorithms to optimize 0z. In Stage 2, we iterate N₂ times to optimize adversary policy т. Similar to Stage 1, we sample and get another set of transition tuples and apply RI. algorithms. Differently, we reverse the sign of since adversary policy aims to decrease the
5

---

Under review as a conference paper at ICLR 2023
Intersection
Bottieneck
Merge
Roundabout
Safety
Safety
Safety
Safety
Success
Success
Success
Success
Efficinnc
Efficiency
Efficiency
Efficiency
IDM
FLOW
CoPO
FailMaker
Social Comm (Ours)
Figure 2: Performance of different traffic flows. The radar graphs demonstrate three essential features of different traffic flows. Safety is calculated by taking the complement catastrophic failures.
Intersection
SocialComm
CoPO
Bottleneck
85
96
84
62
773
18
27
45
72
50
18
36
65
SVO
(degree)
SVO
(degree)
Figure 3: Success rates of CoPO and SocialComm (Ours). The figure reports the percentage of success rates in intersection and regular intervals. All agents in traffic flows are given the same SVO.
We assign fixed SVO from 0 to 90º at
performance of driving policy. Note that the action adversarial policy is ca. the action of driving policy which is used to compute r. This alternating procedure is repeated for N iterations.
The main difference between standard and misunderstanding- tive of B. In standard adversarial learning, 3 controls background agents to attack the driving agent. Background agents know exactly which one the driving agent. While in misunderstanding based adversarial learning, background agents aim to coordinate with each other, including the driving agent. background agent cannot distinguish which surrounding agent the driving agent. There- fore, the spurious agent which produces eg applies adversary from the perspective of driving agent. the spurious agent and driving policy take the same observation.
adversarial learning is the objec-
In Algorithm [
RESULTS
In this section, we pursue to answer three seminal questions. (1) Can our proposed traffic simulation produce more coordinated behaviors? (2) Does the spurious adversarial policy degrade driving policy's 's performance? (3) Does our adversarial training framework improve driving policy's 's zero- shot transfer ability? Before discussing these questions, we first explain some preliminary details.
Settings. We evaluate our proposed method using our internal driving simulator which supports various maps and scenarios. Similar to Peng al. (2021), we select several highly interactive sce- During training, randomly place 8 to 20 vehicles in each scenario the beginning of each episode. After training, we randomly place 20 vehicles and evaluate all relevant methods. See more details of our simulator in Appendix A.4
narios including intersect inn, bettleneck merge, and roundabous
Metrics. We consider three kinds of widely-accepted and general metrics. Firstly, success rate of the whole traffic simulation (Success). Secondly, catastrophic failure rates of the whole traffic sim- ulation. Catastrophic failures include collision between agents (Collision), deviation from drivable area (Off Road), driving too far from global path (Off Route), and crashing into wrong lane (Wrong

---

Under review as a conference paper at ICLR 2023
Quaue
Rush
Cut-in
Yield
Figure 4: Coordinated behaviors in bott The figure highlights that our traffic flow with communication produces diverse coordinated behaviors such as queueing at the narrow crossing. rushing at open areas and yielding avoid crashes.
Lane). Third, driving efficiency represented by average speed of the whole traffic simulation (Effi- ciency). As for single-agent training, these metrics are calculated from the perspective of ego agent. More disccusions can be found Appendix A.5]
Traffic flows. We denote our traffic flow as SocialComm and implement four representative traffic flows to carry out comparative studies on different traffic flows and training pipelines. (1) Intelligent Driver Model (IDM) (Treiber & Kesting
2013) is a rule- saeebbaed controller which uses one single differential equation to model longitudinal movements for all agents. Each agent strictly follows its global path. (2) FLOW (Wu et al. 2021) is a MARL-based method where each agent aims to maximize its own reward. (3) CoPO (Peng et al.) 202T) is MARL-based method that also incorporates SVO. CoPO has stages. First, similar to traffic flow, CoPO applies IPL and SVO to train a background policy. Then, CoPO additionally trains a meta-controller to select all agents' SVOs that the success of the whole population is maximized. Note that agents in CoPÔ have no access to other agents' SVOs. (4) FailMaker (Wachi) (019) is method to generate attacking behaviors. Attacking agents are rewarded if they successfully induce driving agent' catastrophic failures while avoiding personal failures except for collision.
Training pipelines. We use vanilla RL (VRL), existing robust adversarial RL (RARL), and our proposed misunderstanding-ben can be applied in IDM, FLOW, CoPO, and SocialComm, RARL and M-RARL can only be applied in FailMaker and SocialComm (with the spurious adversarial agent) respectively.
adversarial learning (M-RARL) to train driving policies. VRL
5.1 PERFORMANCE OF TRAFFIC FLOWS
We demonstrate the performance of different traffic flows. Figure and Figure show quantitative results of different traffic flows. As one can see, our proposed SocialComm achieves the highest success rates and average speeds across all scenarios. Compared with CoPO, agents in SocialComm can recognize other agents' SVOs and produce coordinated behaviors, therefore achieving collab- oration and high efficiency of the whole system. FailMaker achieves the lowest success rate and highest collision rate (lowest safety) due to its adversarial nature. Note that in me rge, our traffic flow outperforms other methods by large margin. The in merge are much closer than these in other scenarios, which makes it harder for the agents to coordinate. Qualitative results are shown in Figure 4 See Appendix A.2 for more results.
is that the initial poses of all agents
5.2
MISUINDERSTAANDDNGG-AASSS ADVERSARY WITH SOCIALCOMM
Our robust policy learning framework applies adversarial training on coordinated traffic flow. In this part, demonstrate that misunderstanding based adversary successfully degrades driving pol- icy's performance and has the ability to impede the driving agent.
Performance. We first train driving policy using vanilla RL in four non-adversarial traffic flows (in- cluding IDM, FLOW, CoPO, and SocialComm) and deploy these well-trained driving policies into SocialComm with the spurious adversarial agent. Results are shown in Table Data in parentheses the change of performances under the spurious adversarial policy. Success rates and efficiencies of all driving policies decrease. This reveals that our spurious adversarial policy could impede the

---

Under review as a conference paper at ICLR 2023
Table 1: Effect of misunderstanding-baees adversary with our coordinated traflic flow So- cialComm. The table reports the percentage of different metrics in intersection and bottleneck. Results in parentheses indicate the performance change under adversary. Results marked in red indicate the performance degradation under adversary while results blue indicate the performance increase. A "f" indicates proposed traffic flow.
Int Intersecti ion
Methods
Success (f)
Collision (4)
Road (4)
Route (L)
Wrong Lane (L)
Efficiency (t)
VRL/IDM
77.0 (-2.5)
10.0 (+4.5)
11.0 (-3.5)
1.0 (+1.))
0.0 (+0.0)
47.1 (-1.2)
VRL/FLOW
(-1.5)
13.5 (+1.5)
1.0 (+1.0)
(-0,5)
(-0.5)
50.1 (-0.5)
VRI/CaPO
31.5.(33.0
14.5 (+4.0)
1.0 (+0.5
2.0 (-1.0)
0.0 (+0.0)
48.8 (-1.3)
VRL/SocialComm'
(-2.0
7.0 (+2.0)
(+1.5)
(-1.0)
0.5 (+0.0)
51.9 (-1.1)
Bottleneck
Methods
Success (f)
Collision ())
Off Road (L)
Off Route (L)
Wrong Lane (L)
Efficiency (f)
VRL/IDM
52.5 (-2.0)
26.0(+1.00
(+1.0+
0.5 (+0.0)
0.0 (+0.0)
58.3 (-1.3)
VRL/FLOW
19.5 (+4.5)
(-2.5)
0.5 (+0.0)
0.0 (+0.0)
74.6 (-0.8)
VRL/CaPO
71.5 (-2.0)
13.5 (-2.5)
15.5 (+3.5)
1.0 (+0.0)
0.0 (+0.0)
70.9 (-1.0)
VRL/SocialComm'
(-6.0)
4.0 (43.0)
5.0 (43.0)
0.0 (+0.0)
0.0.0)
81.3
Withoul Adversary
Adversary via Misunderstanding
Slow down
Cut-in
0 Uara 0:49
25 0,09 0.92
= 5 0,00 0.88
Figure 5: Adversarial behaviors towards driving agent generated by coordinated traffic flow. Driving policy controls the red vehicle. Vehicles with blue and purple boxes two background agents that exhibit adversarial behaviors towards driving policy while maintaining coordination.
driving policy. Note that catastrophic failures of driving policy still increase under adversary in our traffic flow. The reason is that although highly coordinated, our traffic flow cannot eliminate catas- trophic failures of the whole traffic system. And not unallowable for the traffic flow incur driving policy's catastrophic failures due to the optimization objective in Equation g
Adversarial behaviors on coordinated traffic flow. Figure demonstrates some qualitative results on how our traffic flow impedes driving policy to finish its task. Driving policy controls the red car. At t = Os, the driving policy aims to pass through the bottleneck efficiently and keeps high speed. When driving policy approaches the bottleneck, where interaction frequently happens, the agent with blue box slows down (t = 2.5s) to regulate the speed of driving policy and pass through the bottleneck. After that, the agent with purple box cut in the agent ahead of driving agent (t 5.0s). These agents impede the driving policy and degrade its efficiency and explicit coordinated behaviors among each other.
5.3 ZERO-SHIOT TRANSFER PERFORMANCE OF DRIVING POLICIES
To evaluate the robustness in unseen environments, we deploy all driving policies in all traffic flows. In Figure elements in primary diagonals obtained by evaluating in their training environments (the traffic flow used in training and evaluating is the same) and typically achieve highest success rates. Therefore, off-diagonal elements reveal zero-shot transfer performances and colors in the
8

---

Under review as a conference paper at ICLR 2023
Intersection
Bottleneck
DM
79.5
65.5
69.5
77
74.5
91.5
51.0
48.5
52.5
50.5
67.0
87.5
75.5
84
82.5
53.0
79.0
61.0
75.5
73.5
CerO
69.0
76.0
61.5
81
78.5
64.5
60.5
89.0
71.5
69.5
69.0
79.0
74.5
87
85.0
67.0
72.5
65.0
91.0
85.0
NJNAR
Sorth
SconE.omm
Soctaloer
Sodt
SocalComm
Figure 6: Zero-shot transfer performance in intersection and bottleneck. The heatmap reports the percentage success rate for different methods different traffic flows. Deeper color represents higher success rate. Primary diagonals indicates that training and evaluating environments are the same. " indicates our proposed misunderstanding- -aased adversarial training.
lowest row are column-win -iiue deepest, indicating that driving policy trained with our proposed M- RARL acquires highest transferability.
RARL/FailMaker shows the worst zero-shot transfer performance (which is removed from Figure for clarity and can be found Appendix A.3) since background agents FailMaker deliberately induce catastrophic failures of driving policy. This means that RARL/FailMadl
has no way to see non-adversarial traffic behaviors. Therefore, although robustness under adversarial attack im- proved, RARerFailMkeer is fragile unseen traffic patterns. Based on this observation. VRL/IDM, VRL/FLOW, VRL/CoPO, and VRL/SocialComm demonstrate superior zero-shot transfer perfor- mances in non-adversarial traffic flows compared to RARI/Fail
Comparing VRL/SocialComm and M-RARL.SocialComm, one can see that injecting adversaries properly in dense traffic flow significantly improves robustness in unseen non-adversarial environ- ments. Note that all methods except RARL/FailMaker act poorly in FailMaker since it is extremely easy for background agents in FailMaker attack driving policies, matter how driving agents shrewdly. See Appendix A.3 for more results.
CONCLUSION
In this paper, we propose a novel adversarial training framework based on a coordinated traffic flow with communication. Driving policies trained with this framework exhibit robust behaviors across various traffic flows. We report characteristics of several traffic flows in scenarios including i intersectin bottleneck merge, and roundabout. We carry out numerous compar- ative studies to evaluate the transferability of driving policy. Results show that our traffic flow achieves the highest success rate and adversarial learning on our traffic flow significantly improves driving policy's 's zero-shot transfer performance compared to existing algorithms.
REFERENCES
Michael Behrisch, Laura Bieker, Jakob Erdmann, and Daniel Krajzewicz. Sumo-simulation of urban mobility: overview. In Proceedings of SIMUL 2011, The Third International Conference on Advances in System Simulation. ThinkMind, 2011.
Noam Buckman, Alyssa Pierson, Wilko Schwarting, Sertac Karaman, and Daniela Rus. Sharing is caring: Socially-complian autonomous intersection negotiation. 2019 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 6136-6143. IEEE, 2019.
Panpan Cai. Yuanfu Luo, Aseem Saxena, David Hsu, and Wee Sun Lee. Lets-drive: Driving in a crowd by learning from tree search. In Proceedings of Robotics: Science and Systems, FreiburgimBreisgauu, Germany, June 2019. doi: 10.15607/RSS.2019XXY000

---

Under review as a conference paper at ICLR 2023
Panpan Cai, Yiyuan Lee, Yuanfu Luo, and David Hsu. Summit: A simulator for urban driving in massive mixed traffic. In 2020 JEEE Interaational Conference on Robotics and Automation (ICRA), pp. 4023 023-4029.100 IEEE, 2020.
Baiming Chen, Xiang Chen, Qiong Wu, and Liang Li. Adversarial evaluation of autonomous vehi- cles in lane-change scenarios. IEEE Transactions on Intelligent Transportation Systems, 2021.
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. Con- ference Robot Learning, pp. 66-75. PMLR, 2020.
Jianyu Chen, Bodi Yuan, and Masayoshi Tomizuka. Model-free deep reinforcement learning for urban autonomous driving. In 2019 IEEE intelligent transportation systems conference (ITSC), pp. 2765 27711. IEEE, 2019a.
Yilun Chen, Chiyu Dong, Praveen Palanisamy, Priyantha Mudalige, Katharina Muelling, and John M Dolan. Attention-based hierarchical deep reinforcement learning for lane change be- haviors in autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019b.
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint 1.00::201 2020.
Wenhao Ding, Baiming Chen, Minjun Xu, and Ding Zhao. Learning to collide: An adaptive safety- critical scenarios generating method. In 2020 JEEE/RS/ International Conference Intelligent Robots and Systems (IROS), pp. 2243-2250 IEEE, 2020.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pp. 1-16. PMLR, 2017.
Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
11525-15 11533, 2020.
Junru Gu, Chen Sun, and Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15303- 15312, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018.
Peide Huang, Mengdi Xu, Fei Fang, and Ding Zhao. Robust reinforcement learning stackelberg game via adlattivel-reegularizee adversarial training. In Proceedings of the Thirty-First Interna- fional Joint Conference on Artificial Intelligence, İJCAI-22, pp. 3099-39990 3106. International Joint Conferences on Artificial Intelligence Organization, 2022.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR (Poster), 2015.
Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng. and Raquel Urtasun. Learning lane graph representations for motion forecasting. In European Conference on Computer Vision, pp. 541-556. Springer, 2020.
Wim BG Liebrand. The effect of social motives, communication and group size on behaviour in an n-person multi-stage 1984.
game. European journaal of social psychology. 14(3):239-264
Weilin Liu, Ye Mu, Chao Yu, Xuefei Ning. Zhong Cao, Yi Wu. Shuang Liang, Huazhong Yang, and Yu Wang. Multi-agent vulnerability discovery for autonomous driving with hazard arbitration reward. arXiv preprint 12. 2021.
10

---

Under review as a conference paper at ICLR 2023
Majid Moghadam, Ali Alizadeh, Engin Tekin, and Gabriel Hugh Elkaim. An end-to-end deep reinforcement learning approach for the long-term short-term planning on the frenet space. arXiv preprint arXhv:2011.13309 2020.
Tuomas Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, and Tsui-Wei Weng. Robust deep reinforcement learning through adversarial loss. Advances in Neural Information Processing Systems, 34:26156-28 2616667, 2021.
Frans A Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs. Springer, 2016.
Avik Pal, Jonah Philion, Yuan- Liao, and Sanja Fidler. Emergent road rules in multi-agent driving environments. International Conference on Learning Representations, 2020.
Praveen Palanisamy. Multi-agent connected autonomous driving using deep reinforcement learning. In 2020 International Joint Conference Neural Networks (IJCNV), pp. 1-7. IEEF, 2020.
Xinlei Pan, Daniel Seita, Yang Gao, and John Canny. Risk averse robust adversarial reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8522-8528. IEEE, 2019.
Zhenghao Peng. Quanyi Li, Ka Ming Hui, Chunxiao Liu, and Bolei Zhou. Learning to simulate self- driven particles system with coordinated policy optimization. Advances in Neural Information Processing Systems, 14:10784-10797, 2021.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In International Conference Machine Learning, pp. 2817-2826 PMLR, 2017.
Nicholas Rhinehart, Rowan McAllister, and Sergey Levine. Deep imitative models for flexible inference, planning, and control. In International Conference on Learning Representations, 2019.
Wilko Schwarting, Alyssa Pierson, Javier Alonso-Mora, Sertac Karaman, and Daniela Rus. Social behavior for autonomous vehicles. Proceedings of the National Academy of Sciences, 116(50): 24972-24978, 2019.
Aizaz Sharif and Dusica Marijan. Evaluating the robustness of deep reinforcement learning for autonomous and adversarial policies in a multi-agent urban driving environment. arXiv preprint arXiv:2112.11947 2021.
Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. Trafficsim: Learning simulate realistic multi-agent behaviors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10400-10409, 2021.
Ming Tan. Multi ageti reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pp. 330-337, 1993.
Justin K Terry, Nathaniel Grammel, Ananth Hari, Luis Santos, and Benjamin Black. Revisiting parameter sharing in multi-agent deep reinforcement learning. arXiv preprint 2tti::0005116255 2020.
Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde.
End-to-end model-free reinforcement learning for urban driving using implicit affordances. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7153-7162, 2020.
Martin Treiber and Arne Kesting. Traffic flow dynamics. Traffic Flow Dynamics: Data, Models and Simulation, Springer- peringer-Verlag Berlin Heidelberg, pp. 983-1000, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz. Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Eugene Vinitsky, Yuging Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen. Robust reinforcement learning using adversarial populations. arXiv preprint arXiv:2008.01825, 2020.
11

---

Under review as a conference paper at ICLR 2023
Akifumi Wachi. Failure-scent scenario maker for rule-based agent using multi-agent adversarial rein- forcement learning and its application to autonomous driving. In International Joint Conference on Artificial Intelligence. International Joint Conferences Artificial Intelligence, 2019.
Jingke Wang, Tengju Ye, Ziqing Gu, and Junbo Chen. Lip: Lane-based trajectory prediction for tonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 77134-1712,, 2022.
Cathy Wu, Abdul Rahman Kreidieh, Kanaad Parvate, Eugene Vinitsk,, and Alexandre M Bayen. Flow: A modular learning framework for mixed autonomy traffic. IEEE Transactions on Robotics, 2021.
Jiachen Yang. Ang Li, Mehrdad Farajtabar, Peter Sunehag, Edward Hughes, and Hongyuan Zha. Learning incentivize other learning agents. Advances in Neural Information Processing Sys- lems, 33:15208-15219, 2020.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Ben Sapp. Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, al. Tnt: Target-driven trajectory prediction. In Con- ference on Robot Learning, pp. 895-904. PMLR, 2021.
Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Mont- gomery Alban, IMAN FADAKAR, Zheng Chen, al. Smarts: An open-source scalable multi- agent rl training school for autonomous driving. In Conference on Robot Learning, pp. 264-285. PMLR, 2021.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 IMPLEMENTATION DETAILS
Intelligent Driver Model (IMM). IDM is given by Equation 9 and Equation T0 The model de- scribes the acceleration Pback of the back agent, as function of the agent's s velocity bback, the reference velocity vo. the difference between the agent velocity and the velocity the agent in front Sback- Here, front
front + Liength. the position of the front agent, Sback denotes the position of the back agent, and Liength. denotes the length of the front agent. The physical interpretation of the parameters the minimum following time, T. the minimum following gap, $p, the maximum acceleration, a, the minimum following gap, 8o, the maximum acceleration, a, and the comfortable braking deceleration, b.
Ae Uback Ufront, and the following distance
Uback a [
Vback no
)
"-((
(
(Ubacks Av)
27
Φ (Wback, Av) = + Uback T Dhack Av
(10)
2v ab
Policy learning parameters. For Independent Policy Learning (IPL.) and single-agent reinforce- ment learning algorithms, we utilize Soft-Actor-Critic (SAC) (Haarnoja et al., 2018)) and Adam optimizer (Kingma & Ba 2013). Detailed parameters are shown in Table
A.2 RESULTS ON OUR COORDINATED TRAFFIC FLOW
More results are shown in Table
Figure
Figure Figure
Figure 10 Figure 11
Figure 12
Figure 13 and Figure 14
A.3
RESULTS ON ZERO-SHOT TRANSFER
More results are shown in Table 5 Table Table
Table
A.4 DETAILS OUR SIMULATOR
Our internal driving simulator is 2D and aims to investigate single- and multi-agent driving behav- iors, especially in dense traffic flows. Inspired by the trajectory prediction community (Liang al, 2020 Zhao et al. 2021 Gu et al.) 2021). our simulator utilizes sparse (vectorized) representation to capture the structural information of high-definition maps and agents. Compared rasterized encoding which rasterizes the HD map elements together with agents into an image, vectorized rep- resentation is computation- and memory-efficient (Gao et al.) 2020). The critical components of simulator are built on top of this vectorized representation. For designing RL-oriented sim- ulator, there are three critical components including scenario initialization, step forward, and done condition.
scenario initialization. We choose one scenario from intersection, bott leneck, merge, and roundabout and load the bre-built vectorized map for this scenario. After that, assign global path, initial pose, and SVO for each agent in the scenario. A vectorized map contains two parts including the centerline and sideline. Each part 3D tensor which contains different ele- ments. Each element contains a sequence of points. Each point v is a vector [p, !] where = (x,, 0) 0). The average distance of adja- cent points is 2.0m. Given this map, we build a graph G on top of centerline where each element is a node of G and an edge exists only when two elements are connected end end. For each scenario, we manually pick up two bunch of points as initial and terminational poses respectively. For each agent, we randomly select an initial and terminal pose (Pinstial and Pterminal) and use A* algorithm to search a list of points from Pinitial Pterminal on G. This list of points is the global path of the agent, in which the first point is the initial pose. Finally, we assign SVO E [o. 3 to this agent. For all agents in the scenario, the above procedure is repeated. We further clarify what the term "agent" means in our paper. Agents can be divided into two categories: foreground agents and background agents. Background agents NPCs that have unchanged policies like IDM and
the pose and represents the lane width (for sideline, is always
13

---

Under review as a conference paper at ICLR 2023
learned NNs. Background agents are part of the environment. For single-agent environment, there only one single foreground agent (driving agent). For multi-agent environment, there exist mul- tiple foreground agents. "Foreground agent" is exactly the meaning of "agent" in RL community. Currently, when there are n vehicles in our simulator, the number of foreground and background agents (1, 1) for single-agent settings and (n, 0) for multi-agent settings.
step forward. We use bicycle model as the vehicle dynamic model, where the inputs of the model acceleration and steer and the state is (Ir, y, 0, v). To guarantee that the agent will not exceed its maximum speed Umaz 6m/ /s substantially, we introduce PID controller (K₂ 1.0, K; 0.01, Ka 0.05) to regulate a given the reference speed vy and current speed v. Therefore, the action is (u,, 5) where 2r E [0, Uman and 8 E [-45%, 45%]. As explained in Section 3.2 the state and observation space contains a collection of static and dynamic elements and is vectorized, the dimension of state and observation space is inherently not fixed. The length of static elements is not fixed and has no upper bound. The upper bound of dynamic elements' length 10.
done condition. In our simulator, one agent is done if it reaches its destination, encounters catas- trophic failures, or survives until timeout. Once an agent is done, it will be removed from the scenario. When all foreground agents are done, this episode ends. Catastrophic failures include col- lision with other agents, deviation from drivable area, driving too far from global path, and crashing into wrong lane. The maximum steps for one episode steps in the environment, we call it "timeout". In this paper, Imaz = 100. An agent marked success only when it passes the interaction zone (as shown in Figure [5] Note that name each scenario with its interaction zone. For instance, the interaction zone in bottlenech is the bottleneck.
tmaz and when agent survives maz
A.S
METRICS
Note that most metrics we use are widely used in prior works. Success is used in Dosovitskiy et al.
2017 :
Chen al. (2020)
Wu et al. [002]: Peng et al.] (202T); Rhinehart et al. 2019); Chen et
2019b Cai et al. (2019) Collision is used in Chen et al. (2020); Suo et al. (2021): Chen et al.
2019b;;
Cai et al.
2019) Off Road is used Rhinehart et al. (2019) Chen et al. (2019b). Off
Route our design, but Chen et al. (2019a;
Toromanotf et al. (2020) uses it reward term.
Wrong Lane is used in Rhinehart et al. (2019). Efficiency is used in Wu et al. (2021) Chen et al.
2019h
Moghadam et al. (2020) Cai al. 2019
A.6 ADDITIONAL RESULTS
The comparison of training efficiency between CoPO and SocialComm (Ours) is shown in Figure 16
The impact of varying the number of agents in each scenario is shown in Figure 17
Table 2: Parameters of IDM.
Parameter
Value
Desired speed Up Time gap T
6 m// 1.0 8
Minimum gap So
2 m
Acceleration exponent
Acceleration a
5.0 m/s²
Comfortable deceleration b
5.0 m/s²
14

---

Under review as a conference paper at ICLR 2023
Table 3: Hyperparameters of SAC.
Parameter
Value
optimizer
Adam
actor learning rate
10
critic learning rate
5 10
tune learning rate discount (7)
1 10-4
0.9
batch size
128
replay buffer size
10
nonlinearity
ReLU
target smoothing coefficient (r)
0.005
target update interval
200
Table 4: Quantitative performance of traffic flows. The table reports the percentage of different metries intersection, bott lenekk,,,, merge, and roundabout. "t" indicates our proposed traffic flow.
Intersection
Traffic Flows
Success (f)
Collision ())
Off Road ())
Off Route (L)
Wrong Lane (L)
Efficiency (t)
IDM
70.4 + 0.0
29.6 + 0.0
0.0
0.0 +
+ 0.0
42.1 +
FLOW
79.6 0.5
15.9 0.6
=
1.2
=
17.4-.0.2
CoPO
79.6 0.3
17.6 0.4
1.3 0.1
1.4
0.1
46.8 0.1
FailMaker
45.3 45.3
52.5
0.0 0.0
0.0 1 0.0
0.0 0.0
28.7 0.1
SocialComm
86.9 1 0.5
9.0 1 0.4
2.8 0.1
1.1
0.2 0.1
$1.0 0.2
Bottleness
Traffic Flows
Success (1)
Collision (L)
Off Road (L)
Off Route (1)
Wrong Lane (L)
Efficiency (t)
33.0
=
0.0
0.0
FL.OW
76.2 + 0.5
9.8 0.4
14.3 + 0.6
0.0 +
0.0
75.1 0.3
CoPO
80.3 1 0.6
9.3 1.3100
11.4 0.4
0.0 1 0.0
0.0
0.0
74.5 1 0.3
FailMaker
+.2
78.6 0.1
0.1 0.0
0.0 + 0.0
0.0 = 0.0
29.0 + 0.2
SocialComm
83.4 + 0.4
9.4 + 0.3
7.3 0.3
0.0 + 0.0
0.0 + 0.0
76.4 0.1
Merge
Traffic Flows
Success (f)
Collision ())
Off Road ())
Off Route (1)
Wrong Lane (L)
Efficiency (t)
IDM
60.0 + 0.0
40.0 + 0.0
0.0 0.0
0.0 +
0.0
46.9 0.0
FLOW
66.2 : 0.4
25.4 0.5
8.5 0.2
0.0 :
0.0 0.0
55.1 0.2
CoPO
69.3 + 0.5
26.9 = 0.6
3.8 0.3
0.0 + 0.0
0.0 = 0.0
54.9 0.2
FailMaker
16.0 0.2
80.9 1 0.2
3.4 0,2
0.0 1 0.0
0.0 1 0.0
16.2 1 0.1
SocialComm
83.1 1 0.5
16.2 1 0.5
0.6 0.1
0.2 1 0.1
0.0 1 0.0
60.0 1 0.2
Roundabout
Traffic Flows
Success (t)
Collision (L)
Off Road ())
Route (L)
Wrong Lane (L)
Efficiency (1)
IDM
73.6 1 0.0
26.4 1 0.0
0.0
0.0
0.0 1 0.0
0.0 1 0.0
38.1 0.0
FLOWW
72.7 + 0.6
22.4 + 0.4
4.8 0.2
0.0 + 0.0
0.1 + 0.1
39.2 39.2+00 0.1
CoPO
81.2 0.6 0.6
14.3 0.5
4.0 = 0.2
0.0 : 0.0
0,5 0.1
39.0
FailMaker
21.3 + 0.3
77.6 0.3
0.5 0.0
0.0 + 0.0
0.6 0.1
15.7 15.7000 0.1
SocialComm
84.6 + 0.5
11.5 + 0.3
3.6 = 0.3
0.1 + 0.0
0.5 0.1
42.2 + 0.1
15

---

Under review as a conference paper at ICLR 2023
SocialComm CaPO
Success
80.0
77.5
18
27
45
54
63
81
16
Collision
24
10
Off Road
18
81
2.50
1.75
Off Route
1.50
18
27
36
45
54
53
81
0.4
Wrong Lane
0.1
Efficiency
50
4B
44
18
45
63
51
99
SVO (degree)
Figure 7: The performance of CoPO and SocialComm (Ours) in intersection.
16

---

Under review as a conference paper at ICLR 2023
SocialComm CaPO
85
Success
80
18
27
45
54
63
81
Collision
15
Off Road
18
61
Off Route
0.04
18
27
36
45
54
63
81
0.04
Wrong Lane
0.50
0.50
-0.00
-0.04
Efficlency
18
45
63
51
SVO (degree)
Figure 8: The performance of CoPO and SocialComm (Ours) in bott leneck.
17

---

Under review as a conference paper at ICLR 2023
SocialComm CaPO
Success
80
18
27
36
54
63
81
Collision
35
15
Off Road
18
61
1.0
Off Route
0.4
18
27
36
45
54
63
81
0.04
Wrong Lane
0.50
0.50
-0.00
-0.04
60
Efficiency
15
18
36
45
63
51
99
SVO (degree)
Figure 9: The performance of CoPO and SocialComm (Ours) in merge.
18

---

Under review as a conference paper at ICLR 2023
85.0
SocialComm CaPO
77.5
75.0
725
18
27
54
63
81
20
Collision
25
Off Road
18
81
10
Off Route
18
27
36
45
54
63
81
Wrong Lane
3.5
12
Efficiency
41
40
18
45
63
51
99
SVO (degree)
Figure 10: The performance of CoPO and SocialComm (Ours) in roundabout.
19

---

Under review as a conference paper at ICLR 2023
Agent Cut-in
Agent 2: Bypass
Agent 5: Queue
Agent 6: Rush
Agent 3: Yield
Agent 4: Bypass
Figure 11: Coordinated behaviors in intersection.
Queue
Rush
Cut-in
Yield
Figure 12: Coordinated behaviors in bottleneck.
Agent 3: Queue
Agent 1:Yield
Agent 4:Rush
Agent Cut-in
Figure 13: Coordinated behaviors in merge.
20

---

Under review as a conference paper at ICLR 2023
Agent 3: Yield
Agent 2:Creep
Agent 1:Rush
Agent 4:Bypass
Figure 14: Coordinated behaviors in roundabout.
Bottlenack
Marga
Intersection
Roundabout
Figure 15: The initial poses (makred as black arrows) and interaction zone (marked as gray rectangles) of each scenario.
Internaction
Bettereck
Roundabed
EsPO Socia Comm (Oun)
Episado
Figure 16: Training success rates of CoPO and SocialComn (Ours).
21

---

Under review as a conference paper at ICLR 2023
Table 5: Zero-shot transfer performance intersection. Each subtable stores results of
different driving policies in the same traffic flow. A indicates our proposed method.
Evaluate
Intersection:
IDM
Success (?)
Collision ())
Off Road ())
Off Route ())
Wrong Lane (J)
Efficiency (t)
VRL/IDM
79.5
6,0
2.0
0.0
0.0
47.1
VRL/FLOW
67.0
30.5
0.5
0.5
0,5
39.5
VRL/CoPO
27.0
0.0
0.0
43,0
RARL/FaiMakkr
22.5
20.5
19.5
11.0
0.0
25.2
VRI JSacialCComm
69.0
28.5
3.5
2.0
0.0
40.8
M-RARL/SocialCome
71.5
21.5
6.0
1.0
0.0
42.1
Evaluate in
Intersection:
FLOW
Success (1)
Collision (1)
Off Road (I)
Off Route (J)
Wrong Lane (1)
Efficiency (t)
VRL/IDM
1.5
42.8
VRIJFLLOW
87.5
6.5
1.5
0.0
0.0
51.5
VRL/CoPO
76.0
12.0
2.0
3.0
0.0
46.5
RARL/FailMake
13.5
13.0
16.0
0.5
25.7
VRL/SocialComm
79.0
11.0
4.5
0.0
47.9
M-RARL/SocialComm
83.5
8.0
3.0
0.5
49.7
Evaluate in CoPO
Intersect or
Success (*)
Collision
Off Road
Off Route (
Wrong Lane
Efficiency
VRL/IDM
69.5
20.0
8.0
1.0
0.5
45.0
VRL/FLOW
75.5
20.0
0.0
0.0
46.1
VRL/CoPO
81.5
16.0
1.0
0.0
0.0
48.7
RARL/FaiMaker
25.5
11.5
15.5
14,5
0.5
26.0
VRI /SocialCComm
74.5
19.0
1.0
2.0
0.0
45.8
79.5
16.5
1.0
0.0
47.7
Evaluate
Intersection:
FailMaker
Success (1)
Collision (1)
Off Road (1)
Off Route ())
Wrong Lane ({)
Efficiency (f)
VRL/IDM
55.0
22.5
22.0
0.0
0.5
37.3
VRL./FOWW
52.0
37.0
12.0
0.0
0.0
36.0
VRL/CoPO
1.0
0.5
34.7
RARL/FalMaake
26.5
11.5
17.5
14.5
0.0
25.9
VRL/SocialComm
51.5
33.0
16.5
1.5
0.0
36.5
M.RAAR//ScialCome/
51.0
40.5
8.0
0.0
0.0
35.9
Bvaluate in SocialComm
Intersect ior
Success (*)
Collision ())
Off Road ())
Off Route (.)
Wrong Lane (L)
Efficiency (t)
VRL/IDM
77.0
10.0
1.0
0.0
47.1
VRL/FLOW
84.0
13.5
0.5
0.5
50.1
VRL/CoPO
81.5
14.5
1.0
2.0
0.0
48.8
RARL/FailMaker
21.5
12.5
21.0
15.0
0.5
24.9
VRI /SocialCComm
1.5
0.5
L-RAARL/ScialComm
85.5
11.0
0.5
0.0
50.9
Intersectiion
Evaluate
SocialComm (with Adv)
Success (*)
Collision (4)
Road (4)
Route (J)
Wrong Lane (1)
Efficiency (f)
VRLJIDM
74.5
14.5
7.5
2.5
0.0
45,9
/RIL/FLOW
15.0
0.0
0.0
49.6
VRL/CoPO
78.5
18.5
1.0
0.0
47.5
RARL/FilMaker
24.0
12.5
15.5
18.0
0.5
25.3
VRL/SocialComm
85.0
9.0
3.0
1.0
0.5
50.8
86.0
10.5
0.5
1.5
0.0
51.4
22

---

Under review as a conference paper at ICLR 2023
Table Zero-shot transfer performance bottleneck. Each subtable stores results of differ- ent driving policies in the same traffic flow. A indicates our proposed method.
Evaluate
Bottleneck
IDM
Success (?)
Collision ())
Off Road ())
Off Route ())
Wrong Lane (J)
Efficiency (t)
VRL/IDM
91.5
6,0
2.5
0.0
0.0
72.8
VRL/FLOW
53.0
25.5
19.5
2.5
0.0
51.7
VRL/CoPO
64.5
24.0
2.5
0.0
59,6
RARL/FaiMakkr
50.0
10.0
37.5
3.0
0.0
33.9
VRI JSaciallao
67.0
20.5
12.5
0.0
0.0
55.8
M-RARL/SocialCome
74.5
21.0
5,5
0.0
0.0
S7.6
Evaluate in
Bottieneck
FLOW
Success (1)
Collision (1)
Off Road (I)
Off Route (J)
Wrong Lane (1)
Efficiency (t)
VRL/IDM
24.0
0.5
0.0
54.8
VRI,FLLWW
79.0
18.5
2.0
0.5
0..
76.1
VRL/CoPO
60.5
27.5
12.5
0.0
0.0
65.5
RARL/FailMake
39.5
0.0
0.0
25.6
VRL/SocialComm
72.5
24.0
0.0
0.0
68.7
M-RARL/SocialComm
73.0
25.5
0.0
0.0
68.4
Evaluate in CoPO
Bott 1 laneck
Success (*)
Collision
Off Road
Off Route (
Wrong Lane
Efficiency (f)
VRL/IDM
48.5
16.5
33.0
2.0
0.0
57.2
VRL/FLOW
0.0
0.0
64.6
VRL/CoPO
89.0
9.0
1.5
0.5
0.0
81.8
RARL/FaiMaker
$1.0
11.0
34.5
3.5
0.0
35.1
VRI /SocialCComm
65.0
30.5
0.0
0.0
67.2
74.0
25.5
0.0
0.0
71.0
Evaluate
FailMaker
Success (1)
Collision (1)
Off Road (1)
Off Route ())
Wrong Lane ({)
Efficiency (f)
VRL/IDM VRL.OFOWW
20.5
59.0
26.0
0.0
0.0
34.5
4.5
94.0
3.0
0.0
0.0
24.2
VRL/CoPO
0.0
0.0
27.5
RARL/FalMakee
38.0
23.0
38.5
0.5
0.0
26.1
VRL/SocialComm
2.5
87.5
13.5
0.0
0.0
24.3
M.RAAR//ScialCome/
4.5
94.0
2,0
0.0
0.0
24.1
Bvaluate in SocialComm
Bott leneck
Success (1)
Collision ())
Off Road ())
Off Route (.)
Wrong Lane (L)
Efficiency (t)
VRL/IDM
26.0
0.5
0.0
58.3
VRL/FLOW
75.5
19.5
0.5
0.0
74.6
VRL/CoPO
71.5
13.5
15.5
1.0
0.0
70.9
RARL/FailMaker
36.0
23.5
42.0
0.5
0.0
26.6
VRI /SocialCCom
0.0
0..
81.3
R-RAALL/ScialComm
89.5
8.5
0.0
0.0
79.6
Evaluate
Bottleness
SocialComm (with Adv)
Success (1)
Collision (4)
Road (4)
Off Route (J)
Wrong Lane (1)
Efficiency (f)
VRLJIDM
50.5
27.0
22.0
0.5
0.0
$7.0
/RIL/FLOW
73.5
24.0
0.5
0.0
73.8
VRL/CoPO
69.5
11.0
19.0
1.0
0.0
RARL/FilMaker
36.5
19.0
43.5
1.0
0.0
27.7
VRL/SocialComm
85.0
7.0
8.0
0.0
0.0
78.3
88.5
11.5
0.0
0.0
0.0
79.8
23

---

Under review as a conference paper at ICLR 2023
Table 7: Zero-shot transfer performance in merge. Each subtable stores results of different driving policies in the same traffic flow. A "I" indicates our proposed method.
Evaluate IDM
Mezge
Success (*)
Collision ())
Off Road ())
Off Route ())
Wrong Lane (J)
Efficiency (t)
VRL/IDM
92.0
6.0
2.0
0.0
0.0
62.3
VRL/FLOW
41.5
53.5
7.5
0.5
0.0
29.1
VRL/CoPO
19.0
9.0
0.0
38.0
RARL/Failakkrr
66.5
5.0
14.5
14.0
0.0
45,9
VRI JSacialCComm
67.5
18.5
14.5
0.0
0.0
43.6
M-RARL/SocialCome
85.0
15.0
0.0
0.0
0.0
53.2
Evaluate in FLOW
Mezge
Success (1)
Collision (1)
Off Road (4)
Off Route (J)
Wrong Lane (1)
Efficiency (t)
VRL/IDM
38.0
0.0
0.0
40.5
VRIJFLLOW
69.5
25.0
5.5
0.0
0..
56.0
VRL/CoPO
52.0
35.0
12.0
1.0
0.0
48.2
RARL/FailMake
40.5
9.5
0.0
24.2
VRL/SocialComm
62.5
33.5
0.0
0.0
54.5
M-RARL/SocialComm
67.5
21.5
11.0
0.0
0.0
56.0
Evaluate in CoPO
Me rge
Success (*)
Collision
Off Road
Off Route (
Wrong Lane
Efficiency
VRL/IDM
36.5
44.5
19.0
0.0
0.0
40.0
VRL/FLOW
44.0
0.5
0.0
46.6
VRL/CoPO
70.5
24.5
5.0
0.0
0.0
55.4
RARL/FaiMakeer
17.0
39.0
34.0
10.0
0.0
23.5
VRI /SocialCComm
57.0
37.5
0.0
0.0
49.9
59.5
40.0
0.0
0.0
52.0
Evaluate
Mexge
FailMaker
Success (1)
Collision (1)
Off Road (1)
Off Route ())
Wrong Lane (1)
Efficiency (f)
VRL/IDM /RL.OFOWW
74.5
18.0
0.0
0.0
15.1
9.0
81.0
11.5
0.0
0.0
17.5
VRL/CoPO
0.0
0.0
19.2
RARL/FalMakke
19.5
37.5
39.5
6.5
0.0
21.6
VRL/SocialComm
8.5
88.0
4.0
0.0
0.0
16.8
M.RAAR//ScialCome/
12.0
78.5
11.0
0.0
0.0
18.5
Bvaluate in SocialComm
Me rge
Success (*)
Collision ())
Off Road ())
Off Route (.)
Wrong Lane (L)
Efficiency (t)
VRL/IDM
16.0
19.0
0.0
0.0
51.3
VRL/FLOW
71.5
21.0
0.0
0.0
55.0
VRL/CoPO
70.0
24.5
5.5
0.5
0.0
55.1
RARL/FailMaker
18.5
29.0
47.5
6.5
0.0
27.6
VRI /SocialCCom
14.0
0.0
0..
60.7
R-.AARL/SocalComm
83.5
12.5
0.5
0.0
61.2
Evaluate
Merge
SocialComm (with Adv)
Success (1)
Collision (4)
Road (4)
Off Route (J)
Wrong Lane (1)
Efficiency (f)
VRLJIDM
64.5
18.0
17.5
0.0
0.0
51.1
/RI./FLOW
69.5
22.5
0.0
0.0
$4.1
VRL/CoPO
66.0
23.5
10.5
0.5
0.0
53.4
RARL/FilMaker
18.5
25.0
50.5
7.0
0,0
27.2
VRL/SocialComm
83,5
15.0
1.5
0.0
0.0
61.0
85.0
13.5
1.5
0.0
0.0
61.3
24

---

Under review as a conference paper at ICLR 2023
Table Zero-shot transfer performance roundabout. Each subtable stores results of differ- ent driving policies in the same traffic flow. A "f" indicates our proposed method.
Evaluate
Roundabout
IDM
Success (?)
Collision ())
Off Road ())
Off Route ())
Wrong Lane (J)
Efficiency (t)
VRL/IDM
83,0
9.0
5.5
0.0
2.5
38.7
VRL/FLOW
44.5
51.5
4.0
1.0
0.0
23.8
VRL/CoPO
33.0
0.0
0.0
31.5
RARL/FaiMakkr
32.5
16.0
55.5
6.0
0.5
19.7
VRI JSacialCComm
40.5
30.5
31.0
0.55
1.0
21.1
M-RARL/SocialCome
63.0
18.5
17.0
0,0
1.5
32.3
Evaluate in FLOW
Roundabout
Success (1)
Collision (1)
Off Road (I)
Off Route (J)
Wrong Lane (1)
Efficiency (t)
VRL/IDM
0.0
0.5
32.7
VRIJFFLOW
84.0
11.5
4.5
0.0
0.0
42.7
VRL/CoPO
61.5
29.5
9.0
0.0
0.0
38.4
RARL/FailMake
24.0
5.0
0.5
17.7
VRL/SocialComm
66.5
25.5
0.0
0.0
38.6
M-RARL/SocialComm
70.5
26.5
0.0
0.5
40.1
Evaluate in CoPO
Roundabout
Success (*)
Collision
Off Road
Off Route (
Wrong Lane
Efficiency
VRL/IDM
55.5
18.5
23.0
0.5
2.5
32.4
VRL/FLOW
12.0
0.0
2.0
35.7
VRL/CoPO
86.5
12.0
1.5
0.0
0.0
41.8
RARL/FaiMaker
32.0
25.0
46.5
4.5
0.5
17.2
VRI /SocialCComm
70.0
26.5
0.0
0.5
38.8
76.5
17.5
5.5
0.0
0.5
39.9
Evaluate
Roundabout
FailMaker
Success (1)
Collision (1)
Off Road (1)
Off Route ())
Wrong Lane ({)
Efficiency (f)
VRL/IDM
19.5
64.0
20.5
0.0
0.5
20.2
VRL.OFOWW
29.5
57.5
14.0
0.0
0.0
25.0
VRL/CoPO
70.5
0.0
0.5
22.7
RARL/FalMakke
29.5
17.0
51.5
5.0
0.0
14.5
VRL/SocialComm
17.0
63.0
24.5
0.0
0.0
19.9
M.RAAR//ScialComo/
18.5
$7.5
25.5
0,0
0.0
20.5
Bvaluate in SocialComm
Roundabout
Success (*)
Collision ())
Off Road ())
Off Route (.)
Wrong Lane (L)
Efficiency (t)
VRL/IDM
75.5
13.0
0.0
2.0
38.0
VRL/FLOW
74.0
12,0
14.0
0.0
0.0
39.7
VRL/CoPO
82.0
13.5
1.5
1.5
2.5
42,4
RARL/FailMaker
30.0
16.5
54.5
5.0
0.0
17.7
VRI /SocialCComm
0.0
0.0
43.2
L-RAARL/ScialComm
86.0
12.0
0.0
0.0
42.8
Evaluate
Roundabout
SocialComm (with Adv)
Success (*)
Collision (4)
Road (4)
Route (J)
Wrong Lane (1)
Efficiency (f)
VRLJIDM
72.5
13.0
13.0
0.0
1.5
37.8
/RI./FLOW
81.0
14.5
0.5
0.0
41.2
VRL/CoPO
80.5
17.5
1.5
1.0
0.0
42.2
RARL/FilMaker
31.0
15.5
56.0
4.5
0.5
18.2
VRL/SocialComm
84.5
11.5
5.0
0.0
0.0
42.3
87.0
10.5
2.0
0.0
0.5
42.9
25

---

Under review as a conference paper at ICLR 2023
intersection
100
bottleneck
merge
roundabout
95
Success
90
85
80
10
12
14
16
18
Number of Agents
Figure 17: Success rates with different number of agents.
26


Reviews:
[{"review_id": "oZRYGHnDRi", "reviewer": "Reviewer_7Hpn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors build a traffic simulator in which each agent drives in order to maximize its reward as well as the mean reward of nearby agents. The extent to which the vehicle optimizes for the reward of other agents $c_\\beta$ can be adjusted. The authors then show that if a new agent is trained in such a simulator and during training the parameter for controlling $c_\\beta$ is adversarially optimized, the resultant policy transfers better than baselines to new traffic densities and test-time out-of-distribution NPC controllers (e.g. IDM).", "summary_of_the_review": "I'm currently unclear on how to interpret some of the main claims of the paper, namely whether the authors are claiming that they have a general-purpose driving policy, or a general-purpose simulator in which robust driving policies can be trained. I also think that qualitative videos would make it much easier to judge the quality of the contribution, since most of the metrics are designed by the authors. If the authors clarify their contribution, I am willing to increase my score.", "strength_and_weaknesses": "One strength of this paper is that it addresses multiple impactful problems in self-driving simulation. Generally speaking, adversarial NPCs exhibit unrealistic aggressive behavior. The idea of controlling the extent to which the NPCs optimize for the reward of nearby agents seems like a reasonable way to parameterize the space of plausible vehicle behaviors that an AV may interact with on real roads.\n\nThat being said, the main weakness for me reading this paper was that I found it hard to understand precisely what the authors claim as a contribution in this paper. My sense is that the main contribution is the use of $c_\\beta$ as the variable to adversarially optimize. However, my sense is another contribution is maybe the construction of NPCs that have such a $c_\\beta$? Or is the main contribution the policy that is ultimately trained using Algorithm 1 which they show can be transferred to new test-time simulators with out-of-distribution NPC controllers? I'm not seeing yet what is novel about Algorithm 1 - do the authors define what is different between \"misunderstanding-based adversarial learning\" and standard adversarial learning?\n\nI list my other minor edits and questions below:\n\n- state space - no notion of length or width?\n\n- \"Although IPL is prone to generate egoistic suboptimal behaviors\" - is there a citation for this claim?\n\n- Figure 3 - do the authors compare against the baseline of training the policy on a hard-coded uniform distribution for $c_\\beta$ instead of the adversarially optimized distribution that they find for $c_\\beta$?\n\n- Section 5.1 - clarify why would the initial poses being close together imply that it's harder for the agents to coordinate?\n\n- Figure 5 - I think it's easier to read if the same coordinate frame is used for each column\n\n- Figure 6 - I find this figure difficult to read. I think it would be better if the authors use the same colormap for all grid cells instead of using blue only for the diagonal.\n\n- Do the authors have videos they can share? It's hard to understand the effect of different $c_\\beta$ from frame shots.\n\n- Section 5.3 - I'm not sure what it means for the evaluation environment to be \"IID with training environment\"?\n\n- Figures and Tables - I feel \"ours\" is being used to sometimes denote NPC policies, sometimes ego policies, and sometimes training environments. Is there a way the authors could disambiguate each of these contributions?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gIZ2TY_Sev8", "reviewer": "Reviewer_hLos", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel adversarial framework to train driving policies under learned traffic flow in simulation. It first extends traffic flow learned with reinforcement learning by exposing each other’s intrinsic social value orientations (SVO). Next, it uses this exposed SVO to train adversarial agents as well as a policy. In particular, the traffic flow accesses the driving policy’s SVO but not vice versa (misunderstanding). Experiments show that 1. The new traffic flow with exposed SVO to each other learns better behavior 2. The proposed adversarial training effectively reduces driving policy’s performance and 3. Driving policy has better zero-shot transferability across different traffic flows when trained with the proposed method. \n", "summary_of_the_review": "The paper presents a novel framework to train driving policies under coordinated traffic flow. Experiments are thorough and demonstrate **constructive** usefulness of the presented approach. I personally think this paper is worth acceptance.\n", "strength_and_weaknesses": "## Strengths \n+ The paper is well-written and the presentation is strong.\n+ The proposed method demonstrates convincing empirical results.\n+ The paper proposes a novel way to adversarially train driving policies in traffic flow, and actually demonstrates that agents trained with the proposed method drive (transfers) better, which is a big plus.\n\n## Weaknesses\n- I can’t seem to find sample complexity information on traffic flow training in the paper. I am curious to see whether exposing SVO improves/decreases sample complexity with respect to the metrics in Table 1, and how it compares to the baseline CoPO.\n- Table 1 compares how the proposed method impedes the driving policy. I am curious to see whether this result generalizes across different driving policies/architectures. \n- I would like to see an ablation that justifies the “misunderstanding” part of the claim/story. For example, an ablation in which the driving policy also sees the SVO of the nearby traffic flow.\n- The adversarial training currently strictly comes from the adversarial agent. Since the background traffic already coordinates with each other, I am wondering why not directly alternate between training between the driving policy and background traffic flow, and have the traffic flow perform multi-agent coordinated attacks to the driving policy.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OYb1lprf_Y", "reviewer": "Reviewer_o4m3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work builds a framework for traffic simulation and propose a form of adversarial training to improve driving policies.  \nIn their traffic flow, they assume the agents allowed to communicate *social value orientations* (SVOs) and introduce an adversarial agent that tries to minimize the overall reward for the driving agent by only modifying the SVO.\nThey show strong improvements on their coordinated traffic flow task on a variety of subtasks (intersection, bottleneck, merge, roundabout).", "summary_of_the_review": "This work introduces a fairly novel approach and overall shows improved performance across various scenarios in their simulator. The major factor in my current decision is on reproducibility.", "strength_and_weaknesses": "### Strengths\n\n* Experiments show clear improvements over IDM, FLOW, COPO\n* Overall writing is good, mathematical notion is for the most part clear\n\n### Weaknesses\n\n* Reproducibility - all experiments were performed on \"our internal driving simulator\". Can the authors give more details on this? This is a major factor in my decision.\n* The experiments were all done in this one simulator and it is unclear if the effectiveness of this strategy translates to other domains.\n* Writing/figure quality (more details below) - a couple figure/tables are quite hard to understand and could be presented better.\n\n### Minor Issues / Questions\n\n* What is the mathematical equation for the reward function? Were any other reward functions tried?\n* Algorithm 1: in my opinion this is described well by Sec 4.2 and can be taken out.\n* Table 1: Highlighted colors seem arbitrary - why is the \"0.0\" colored in \"Ours + Wrong Lane\"?\n* Figure 6 took quite a while to understand what I was looking at. FailMaker seems bad and can be taken out and just described in text.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CBxSmB3iVCA", "reviewer": "Reviewer_MJTo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a robust RL policy training framework which applies adversarial training using coordinated traffic flow. For building the traffic flow, it uses the so-called “Social Value Orientations (SVOs)” (which is a weight that balances the proportion of each agent’s own driving reward and its surrounding agents driving rewards in the agent’s aggregated optimization reward) and allows the communication of SVOs among the background agents in the traffic flow. Adversary emerges when the SVOs of the driving agent sent to the traffic flow agents are perturbed. The problem can then be turned into a min-max optimization problem where the driving policy maximizes its reward and a spurious adversarial policy minimizes it through changing the SVOs seen by the agents. The proposed method is evaluated in terms of the safety, speed, and success rate of the generated traffic flows, the effect of adversary based on coordinated traffic flow, and the zero-shot transfer of the driving policy.", "summary_of_the_review": "Overall the approach is moderately interesting and the results seem to be promising. However, some clarification needs to be made regarding the novelty of the method and the details of evaluation for a better evaluation of the paper.", "strength_and_weaknesses": "## Pros:\n\n1.Leveraging SVOs for adversarial attack are moderately novel and its effect on improving zero-shot transfer of the driving policy seems promising.\n\n2.generalization of the training policy is an important problem\n\n## Cons:\n\n1.Communicating SVOs among agents seem to be a relatively incremental improvement over [Peng 2021].\n\n2.One issue is that the realism evaluation of the settings is missing \nAre the generated trajectories realistic and initial positions realistic? One way to evaluate if the trajectories are realistic is to look into acceleration (both longitudinal and latitudinal) and jerk distributions and compare them against some similar scenarios in real world datasets (e.g., nuScenes and Waymo Open).\n\n3.some details are missing\nsee “Clarity, Quality, Novelty And Reproducibility” below.\n\n4.effect of adversary is not that large\nTable 1 success rates are not influenced much percentage wise. One usually expects the attack to have a stronger effect.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject