BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction
Authors: Zikang Zhou
Haibo HU
Xinhong Chen
Jianping Wang
Nan Guan
Kui Wu
Yung-Hui Li
Yu-Kai Huang
Chun Jason Xue
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-Agent Systems
Transformers
Generative Models
Autonomous Driving
Primary area: robotics
TL;DR: A fully autoregressive Transformer with next-patch prediction mechanism for multi-agent behavior simulation in autonomous driving

Abstract: Simulating realistic behaviors of traffic agents is pivotal for efficiently validating the safety of autonomous driving systems. Existing data-driven simulators primarily use an encoder-decoder architecture to encode the historical trajectories before decoding the future. However, the heterogeneity between encoders and decoders complicates the models, and the manual separation of historical and future trajectories leads to low data utilization. Given these limitations, we propose BehaviorGPT, a homogeneous and fully autoregressive Transformer designed to simulate the sequential behavior of multiple agents. Crucially, our approach discards the traditional separation between "history" and "future" by modeling each time step as the "current" one for motion generation, leading to a simpler, more parameter- and data-efficient agent simulator. We further introduce the Next-Patch Prediction Paradigm (NP3) to mitigate the negative effects of autoregressive modeling, in which models are trained to reason at the patch level of trajectories and capture long-range spatial-temporal interactions. Despite having merely 3M model parameters, BehaviorGPT won first place in the 2024 Waymo Open Sim Agents Challenge with a realism score of 0.7473 and a minADE score of 1.4147, demonstrating its exceptional performance in traffic agent simulation.

BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction
Zikang Zhou
Haibo Hu
Xinhong Chen'
Jianping Wang'
Nan Guan Chun Jason Xue
Kui Wu²
Yung-Hui Li
Yu-Kai Huang
City University of Hong Kong University of Victoria
'Hon Hai Research Institute
CCarnegie Mellon University Mohamed bin Zayed University of Artificial Intelligence fzikanzhou2-c, haibohu2-c)amy cityu. edu. hk (xinhong. chen, jianwang, nanguan} t.u. wkui@uvic.co yunghui li@foxcoon. com yukaih2@andrew. cmu. edu jason. xuetmbzuai.ac.o. ac
Abstract
Simulating realistic behaviors of traffic agents is pivotal for efficiently validating the safety of autonomous driving systems. Existing data-driven simulators pri- marily use encoder-decoder architecture encode the historical trajectories before decoding the future. However, the heterogeneity hetween encoders and decoders complicates the models, and the manual separation of historical and future trajectories leads to low data utilization. Given these limitations, we pro- pose BehaviorGPT, homogeneous and fully autoregressive Transformer designed to simulate the sequential behavior of multiple agents. Crucially, our approach discards the traditional separation between "history" and "future" by modeling each time step the "current" one for motion generation, leading to a simpler, more parameter- and data-efficien agent simulator. We further introduce the Next- Patch Prediction Paradigm (NP3) mitigate the negative effects of autoregressive modeling, in which models are trained reason at the patch level of trajectories and capture long-range spatial-temporal interactions. Despite having merely 3M model parameters, BehaviorGPT won first place in the 2024 Waymo Open Sim Agents Challenge with realism score of 0.7473 and minADE score of 1.4147, demonstrating its exceptional performance in traffic agent simulation.
Keywords: Multi-Agent Systems, Transformers, Generative Models, Autonomous Driving
Introduction
Autonomous driving has emerged unstoppable trend, with its rapid development increasing the demand for faithful evaluation autonomy systems' reliability [32]. While on-road testing can measure driving performance by allowing autonomous vehicles (AVs) interact with the physical world directly, the high testing cost and the scarcity of safety-critich scenarios the real world have hindered large-scale and comprehensive evaluation. As alternative, validating system safety via simulation has become increasingly attractive [14 diverse driving scenarios simulated at low
as it enables rapid testing in cost. This work focuses smart agent simulation. i.c., simulating the behavior of traffic participants such as vehicles, pedestrians, and cyclists the digital world, which is critical for efficiently validating and iterating behavioral policies for AVs.
48 53
good simulator should be realistic, matching the real-world distribution multi-agent behaviors to support the assessment of AVs' ability to cocxist with humans safely. To this end, researchers started
*Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

by designing naive simulators that mainly replay the driving logs collected in the real world [27 29 When testing new driving policies that deviate from the ones during data collection, agents in such simulators often exhibit unrealistic interactions with AVs, owing to the lack of reactivity to AVs'
behavior changes. To simulate reactive agents, traditional approaches [14 (8] apply traffic rules to control agents heuristically [45 [26], which may struggle to capture real-world complexity. Recently,
the availability of large-scale driving data [6. [5 50], the emergence of powerful deep learning tools [9 47 (0), and the prosperity of related fields such motion forecasting 116 46 59 42 (58),
have spurred the development of data-driven agent simulation
44 4 24] 52 (66) towards more precise
matching of behavioral distribution. With the establishment of standard benchmarks like the Waymo Open Sim Agents Challenge (WOSAC) [2), which systematically evaluates the realism of agent simulation in terms of kinematics, map compliance, and multi-agent agent interaction, the research on data-driven simulation approaches has been further advanced 49 35].
mainly mirror the techniques and opt for an encoder-decoder architecture, presumably due to the similarity between the two fields. Typically, these models use encoder to extract historical information and decoder to predict agents' future states leveraging the encoded features. This paradigm requires manually splitting the multi-agent time series into historical and future segment, with the two segments being processed by separate encoders and decoders with het- erogeneous architecture. For example, MVTA [49] constructs training samples by randomly selecting a "current" timestamp to divide sequences into historical and future components. Others [52] 35 use fixed-length agent trajectories as historical scene context, conditioned on which the multi-agent future is sampled from the decoder. Nonetheless, the benefit of employing heterogeneous modules to separately encode the history and decode the future, at the cost of significantly complicating the architecture, is unclear. Moreover, the manual separation of history and future leads to low utilization of data and computation: as every point in the sequence can be used for the separation, we believe a sample-efficient framework should be able to learn from every possible history-future pair from the sequence in parallel, which cannot be easily achieved by encoder-decoder solutions owing their heterogeneous processing for the historical and the future time steps.
Existing learning-based agent simulators 44
52
56
49
35
from motion forecasting 16 46 59
Inspired by the success of decoder-only Large Language Models (LLMs) B3 38 5), introduce a fully autoregressive Transformer architecture, dubbed BehaviorGPT, into the field of smart agent simulation to overcome the limitations of previous works. By applying homogeneous Transformer blocks [47 to the complete trajectory snippets without differentiating history and future, we arrive at a simpler, more parameter-efficient and more sample-effcient solution for agent simulation. Utilizing relative spacetime representations (58]. BehaviorGPT symmetrically models each agent state in the sequence if it were the "current" one and tasks each state with modeling subsequent states' distribution during training. As result, our framework maximizes the utilization of traffic data for autoregressive modeling, avoiding wasting any learning signals available in the time series.
Autoregressive modeling with imitation learning, however, suffers from compounding errors [39] and causal confusion [... Concerning the behavior simulation task, we observed that blindly mimicking LLMs' training paradigm of next-token prediction 355 regardless of the difference in tokens' semantics across tasks, will make these issues more prominent. For next-token prediction model embedding tokens at 10 Hz, a low training loss can be achieved by simply copying and pasting the current token as the next one without performing any long-range interaction reasoning in space or time. To mitigate this issue, we introduce the Next-Patch Prediction Paradigm (NP3) that enables models to reason the patch level of trajectories, illustrated in Figure | By enforcing models to autoregressively generate the next trajectory patch containing multiple time steps, which requires understanding the high-level semantics of agent behaviors and capturing long-range spatial- temporal interactions, we prevent models from leveraging trivial shortcuts during training. We equip BehaviorGPT with NP3 and attain superior performance on WOSAC 32] with merely 3M model parameters, demonstrating the effectiveness of our modeling framework for smart agent simulation.
Our main contributions are three-fold. First, we propose fully autoregressive architecture for smart agent simulation, which consists of homogeneous Transformer blocks that process multi-agent long sequences with high parameter and sample efficiency. Second, develop the Next-Patch Prediction scheme to enhance long-range interaction reasoning, leading to more realistic multi-agent simulation over long horizon. Third, we achieve remarkable performance on the Waymo Open Motion Dataset, winning first place in the 2024 Waymo Open Sim Agents Challenge.

---

patch
Figure 1: Next-Patch Prediction Paradigm with patch sizes of 1, 5, and 10 time steps for trajecto- ries sampled 10 Hz. The capsules in dark red represent the agent states at the current time step t, while the faded red capsules indicate agents' past states. The grey circles represent the masked agent states required for generation. Our approach groups multi-step agent states as patches, demand- in each patch to predict the subsequent patch dur- ing training.
Monk
Mark
Mark
step (0.3s) stept 0.16)
patch
5 steps
steps (0.5k)
patch size 10
Mank
O 000 O
Maak
0
10 steps (1s)
10 stepa (14)
2 Related Work
2.1 Multi-Agent Traffic Simulation
Multi-agent traffic simulation is essential for developing and testing autonomous driving systems. From early systems like ALVINN [36] to contemporary simulators such as CARLA [14] and SUMO 28 these platforms have used heuristic driving policies to simulate agents' reactive be-
haviors 8 I0]. However, they struggle to capture real-world complexity since policies based on simple heuristics are not robust enough to handle all sorts of scenarios. With the availability of lcrge-scale data and deep learning approaches, generative models like VAEs [44), GANs (24), Diffusion [56], and autoregressive models [49] 41 B5] have gained success in generating multi-agent motions, which greatly enhance the realism of simulations. Given the temporal dependency of agent trajectories, autoregressive models naturally fit the simulation task, while others require extra designs to capture such dependencies. Among the existing autoregressive models, two representatives are MotionLM [41] and Trajeglish (33). Both of them adopt encoder-decoder paradigm, designing complicated scene context encoders to extract historical information before autoregressive decoding. In contrast, our approach fully autoregressive similar decoder-only LLMs [37 38 [5), which
eliminates the need for using heterogeneous modules to process the historical and future time steps and achieves higher efficiency in terms of data and parameters via simpler architectural design.
2.2 Patching Operations in Transformers
The application of patches in Transformer models has demonstrated significant potential across various data modalities. For instance, BERT [2 employs subword tokenization [40] for natural language processing, while VIT [13] segments images into 2D patches for visual understanding. The patching design has also found applications in time-series forecasting 57 34,, aiming at retaining
local semantics and reducing computational complexity (34 Moreover, it has shown the effectiveness self-supervised learning, which has significantly facilitated representation learning and contributed to excellent fine-tuning results on large datasets 22
21 3 Since the task of agent simulation also involves time-serics data, we expect the patching mechanism to help models effectively capture the peatialtemporal interactions in driving scenarios and enhance the realism of the generated motion. Our proposed Next-Patch Prediction Paradigm (NP3) utilizes patch-level tokens in autoregressive modeling and trains each token to generate the next patch that comprises multi-step motions, which shares some similarities to multi-token prediction in LLMs 17
3 Methodology
This section presents the proposed BehaviorGPT for multi-agent behavior simulation, with Figure 2 illustrating the overall framework. To begin with, we provide the formulation of our map-conditioned, multi-agent autoregressive modeling. Then, we detail the architecture of BehaviorGPT, which adopts a Transformer decoder with a triple-attention mechanism to operate sequences at the patch level. Finally, we present the objective for model training.
3

---

Prs
Agent Embreding
Agent Data
Vel
...
8P @
Traffic Seaario
Eabedding
Map Embeedinng
Dreeder
Head
State
Takua
Takm
Prodicte Podticn
Tredited Nar Probaldlt
Sitooc
Probubility
Figure 2: Overview of BehaviorGPT. The model takes as input the agent trajectories and the map elements, which are converted into the embeddings of trajectory patches and map polyline segments, respectively. These embeddings are fed into Transformer decoder for autoregressive modeling based on next-patch prediction, in which the model is trained to generate the positions, velocities, and yaw angles of trajectory patches.
3.1 Problem Formulation
In multi-agent traffic simulation, we aim simulate agents' future behavior dynamic and complex environments. Specifically, we define scenario as the composite of vector map M and the states of Nagent agents over time steps. At each time step, the state of the i-th agent S; includes the agent's position, velocity, yaw angle, and bounding box size. The semantic type of agents (e.g.. vehicles, pedestrians, and cyclists) are also available. Given the sequential nature of agent trajectories, we formulate the problem sequential predictions patch will affect the subsequent patches. We define an agent-level trajectory patch as
trajectory patches, where the prediction of each
PT = -(-
i (1,...,,Nagunl
NNll...,,uuuuhhi
where is the number of time steps covered by patch, Npacch = T/e indicates the number of patches, and PT represents the 7-th trajectory patch of the i-th agent, with (((==x))(+1)(xxx) denoting the states within the patch. On top of PT, we use p (((==))xx+()((xxx) denote the r-th multi-agent patch, where pr incorporates all agents' states the 7-th patch. Next, we factorize the multi-agent joint distribution over patches along the time axis according to the chain rule:
1:Nagen
Npurt
Pr
T1:T 1:Naas
I
II Pr (PT 7=1
plo(t-1),
(2)
where P I.Naen M) is the joint distribution of agents' states over all time steps conditioned on the map M. Further, we factorize over agents the conditional distribution of multi-agent patches based the assumption that agents plan their motions independently within the horizon of patch:
Naser:
Pr (P" 11) M)
Pr |
Considering the multimodality of agents' behavior within the horizon a patch, we assume Pr(P. pl:(T-1), M) to be mixture model consisting of Nmode modes:
Nasste
Pr PT pl:(r- M)
T
P. pl:(T-11) 1)M))
A=1
where TH, is the probability of the k-th mode. Given the sequential nature of the states within patch, we further conduct factorization over the states per mode using the chain rule:
TXE
Pr PT p1:(t-1),
II
Pr (StN
S ((((-1)))+1)((-1)) 1):(r-1), .M) (5)
[=(--))x(+1
Such an autoregressive formulation can be interpreted as planning the patch-level behavior of each agent independently (Eq. (3)), freezing agents' behavior mode per time steps (Eq. (4)).

---

and autoregressively unrolling the next state under specific behavior mode (Eq. 5 Under this formulation, we can flexibly adjust the replan frequency during inference to control the reactivity of agents. For example, we can let agents execute E {1...., L) steps of the planned motions and choose a new behavior mode after a steps to react to the change in environments.
3.2 Relative Spacetime Representation
In our autoregressive formulation, we treat each trajectory patch as the "current" patch that is responsible for estimating the next-patch distribution during training, contrasting many existing approaches that designate one current time step per sequence [52 49 23
As a result, it is inefficient to employ the eeal-sstablished agent- polyline-centric representation from the field of motion given that these representations are computed under the reference frames determined by one current time step per sequence. For this reason, we adopt the relative spacetime representation introduced in QCNet (58) to model the patches symmetrically in space and time, achieving simultaneous multi-agent prediction when implementing Eq. (3) and allowing parallel next-patch prediction for the modeling of Eq. 2 Under this representation, the features of each map element and agent state are derived from e.g., the semantic category of map element and the speed of an agent state. On top of this, we effectively maintain the spatial-temporal relationships between input elements via relative positional embeddings. Specifically. we use and index two different input elements and compute the relative spatial-temporal embedding by
forecasting [46 59
33
42
25
54
43
perdinate-independdee
attributes,
Rz-i MLP (I|d)~ull, 2 (ni, dy-i), A05 A ATJ-s).
(6)
where Ry-;; is the relational embedding from j to i, ||d]==|| || is the Euclidean distance between them, L(n,, d)-=s) is the angle between n, (i.e., the orientation of i) and d, (i.e., the displacement vector from j to i), A05=;//Zz== is the relative yaw/height from j to i, and ATJ is the time difference.
3.3 Map Tokenization and Agent Patching
Before performing spatial-tempont relational reasoning among the input elements of a traffic scenario, we must convert the raw information into -igheiiiensoonal embeddings. We first embed map information by sampling points along map polylines every 5 meters and tokenizing the semantic category of each 5-meter segment (e.g., lane centerlines, road edges, and crosswalks) via learnable embeddings. The i-th polyline segment's embedding is denoted by M,, which does not include any information about coordinates. On the other hand, process agent states using attention- based patching to obtain patch-level embeddings of trajectories. For the i-th agent's state S at time step f, employ MLP transform the speed, the velocity vector's angle relative to the bounding box 's heading. the size of the bounding box, and the semantic type of the agent. into feature vector Sy To further acquire patch embeddings, we collect the feature vectors of l consecutive agent states and apply the attention mechanism with relative positional embeddings them:
PT MHSA(Q = STT*, K= = {[S], R (+x())))
where Pr is the patch embedding of the i-th agent at the 7-th patch, MHSA(-) denotes the multi-head self-attention 477 [:, :] denotes concatenation, and R (xx)( indicates the positional embedding of St relative to ST computed according to Fq. 6 Such an operation can be viewed as aggregating
the features of s! (((=-))x4t)(fxxx-1) into ŠT and using the embeddings fused with high-level semantics as the agent tokens in the subsequent modules.
3.4 Triple-Attention Transformer Decoder
After obtaining map tokens and the patch embeddings of agents, we employ Transformer de- coder [47] with the triple-attention mechanism to model the spatial-temporal interactions among scene elements. As illustrated in Figure 33 the triple-attention mechanism considers three distinct sources of relations in the scene, including the temporal dependencies over the trajectory patches per agent, the regulations of the map elements on the agents, and the social interactions among agents.
Temporal Self-Attention. This module captures the relationships among the trajectory patches of each individual agent. Similar to decoder-only LLMs [37 38 55, it leverages the multi-head
5

---

Attention
Atteation
Attention
stego
Vehicle
Map
Vehicle
(R) Temporal Self-Attention
Agent-Map Cross-Attention
(c) Agent-Agent Self-Attention
Figure 3: Triple Attention applies attention mechanisms to model (a) agents' sequential behaviors, (b) agents' relationships with the map context, and (c) the interactions among agents.
self-attention (MHSA) with a causal mask to enforce each trajectory patch to only attend to the preceding patches of the same agent, accommodating our autoregressive formulation. The temporal MHSA is equipped with relative positional embeddings:
F2 MHSA(Q = P., p K= = {[P. R. (xx)) (E(1.....
(8)
where F.21, and PT are the temporal-aware feature vector and the patch embedding of the i-th agent the r-th patch, respectively, and R R embeds the relative position from sext S Six which represents the spatial-tempores relationship between the patches P and P..
Agent-Map Cross- Attention. Unlike natural language which only has sequence dimension, must also conduct spatial reasoning to consider the environmental influence on agents' behavior. To facilitate the modeling of agent-map interactions, we apply the multi-head cross-attention (MHCA) to each trajectory patch in the scenario. Considering that scenario may comprise an explosive number of map polyline segments and that an agent would not be influenced by map elements far away, we filter the key/value map elements in MHCA using the n-neersst neighbors algorithm 42 54 The agent-map cross attention formulated as
F #2m,i
MHCA(Q = -F K= V= -[M,, RTSI))EEN())))
(9)
where F 22mm, is the map-aware feature vector for the i-th agent at the 7-th patch, My is the embedding of the j-th map polyline segment, R5z0 is the relative positional embedding between the agent state SiXE and the j-th map polyline segment, and (ii, r) denotes the k-nearest map neighbors of S? x
Agent-Agent Self-Attention. We further capture the social interactions among agents by applying the MHSA to the space dimension of the trajectory patches. In this module, also utilize the locality assumption induced by the k-nearest neighhor selection for better computational and memory efficiency. Specifically, the map-aware features of trajectory patches are refined by
F2 = MHSA(Q = Fizm, K
(1Faam.. R RIx( R.LlDbe(())))
(10)
where 3 is the feature vector enriched with spatial interaction information among agents for the i-th agent at the 7-th patch, 3- contains the relative information between the i-th and the j-th agent the 7-th patch, and (ii, -) filters the k-nearest agent neighbors of ST*e.
Overall Decoder Architecture. Each of the attention layers above is enhanced by commonly used components in Transformers [47), including feed-forward networks, residual connections [19], and Layer Normalization M a pre-norm fashion. To enable higher-order relational reasoning, we stack multiple triple-attention blocks by interleaving the three Transformer layers. We denote the ultimate feature the i-th agent at the 7-th patch as F7, which will serve as the input of the prediction head for next-patch prediction modeling.
3.5 Next-Patch Prediction Head
Given the interaction-aware patch features output by the Transformer decoder, we develop next- patch prediction head to model the marginal multimodal distribution of agent trajectories, which estimates the distributional parameters of each patch's 'ss successor.

---

The following describes the process of next next-patch prediction regarding the 7-th patch of the i-th agent. Based on the attention output F7. we intend to estimate the parameters of the next patch mixture model pre-defined with modee modes. First, we use an MLP to transform F into n. T E R Non the mixing coefficient of the modes. In each mode, the conditional distribution of the next agent state, as depicted in Eq. ( considered multivariate marginal distribution that parameterizes the position and velocity components Laplace distributions and the yaw angle as a von Mises distribution. Based on this formulation, we employ a GRU-based autoregressive RNN [ to unroll the states within the next patch step by step, with each step being conditioned on the previously predicted states. Specifically, The hidden state T. ii,k of the RNN is initialized with F at / = 1 for vk E {1,.... Mmodess) ). At each step of the rollout, parameters of the next agent state's 's position and velocity based on the hidden state. On the other hand, the MIP also estimates the location and concentration parameters of the next yaw angle. The location parameters of the newly predicted state, including the 3D positions, the 2D velocities, and the yaw angle, are used to update the RNN's hidden state directly without relying on the predicted scale/concentration parameters for sampling. The whole process is summarized as follows:
MLP to estimate the location and scale
Th M Zz)),
FT, h
1.
(11)
7xt+1 Eext+t 1i, Lik
x x(+ iik
MIP((M Zk)),
T. i,k
RNN(M 1,kt MLP(ML TxE+t))
where { location, scale, and concentration parameters in the k-th mode, and Zg is the k-th learnable mode embedding.
Txl+t E R Je
(6 E IR heE ()....) and {Kix E ().,(),....)) are the
3.6 Training Objective
To train BehaviorGPT, we apply the negative log- log-likelihood loss LNLL to the factorized distribution of Pr(S):T MM as formulated previously:
I:Napm
Npw N Naaa
Naste
TXt
CNL. Σ
log
Tik
II 1)x+1)
Pr (St. 1
()(+-)))) pittrt-))ppp((t))) pli(r=1), .M)
(12)
Note that each ground-truth trajectory patch is transformed into the viewpoint of its previous patch. During training, we utilize teacher forcing to parallelize the modeling of next-patch prediction and ease the learning difficulty, but we do not use the ground-truth agent states when updating the RNN's hidden states, intending to train the model to recover from its mistakes made in next-state prediction.
4 Experiments
This section first introduces the dataset and the evaluation metrics used in our experiments, followed by presenting the implementation details and the rollout results obtained by BehaviorGPT on the Waymo Open Sim Agents Benchmark [32]. Finally, we conduct ablation studies to further compare and analyze the performance of BehaviorGPT under various settings.
4.1 Dataset and Metrics
Our experiments conducted on the Waymo Open Motion Dataset (WOMD) [15.. The dataset comprises 188,995/44,097/44 094/449974499000 trainin//aliddationttssstt step observations sampled 10 Hz, totaling 9.1 seconds. Given 1 11-step initial states of the scenarios, we simulate up to 128 agents and generate 80 simulation steps per agent at 0.1-second intervals in an autoregressive and reactive manner. Each agent requires 32 simulations comprising x/y/z centroid coordinates and a heading value. The results on the test set are obtained by utilizing the full training set, while the performance on the validation set is based on 20% of training data unless specified.
scenarios. Each scenario includes 91-
We use various metrics for evaluation. The minADE measures the minimum average displacement error over multiple simulated trajectories, assessing trajectory accuracy. REALISM is the meta-metric
7

---

Table 1: Test set results in the 2024 Waymo Open Sim Agents Challenge.
Medel
#Pam
minADE
REALISM
LINEAR SPEED
(1)
LINEAK ACCEL.
ANG SPERD
ANG ACCRI
TO
COLLISION
(T)
DISUTO ROAD EDGE
(T)
OFFROAD
(7)
Linear Intmppllttoon
7,5148
0.9983
0.9336
0.0061
0.2522
0A393
0.215
0.195
0,7555
0.4801
0.6426
Trafiidioee1 1 SS
10M
18825
0,8883
0.3361
0:1497
04512
0.5844
0.359
0.8083
0.8209
016423
0.9134
YBD DS
12M
1.4723
0.7200
0.3991
0.3664
0.4997
0.5222
0.3653
0.341
0.3153
0.6305
0.5788
MYTE
265M
1.670
0.7302
0.5305
0.2531
0.4974
0.6000
0.3743
03659
0.8310
0.6655
0U9071
GUMP
52.5
1.001]
0.7431
0.3367
0.111
0.5089
0.653
0.3757
0.900
0.8276
0.6086
0,9025
BeluvtorGPP (Ours)
0,7473
0.3615
0.9537
NN Seevwto
Prodiced Scmure
Nedicted Stewe
Nedireed Stewer
+100
+339
+338
-138
-335
-335
-315
-315
-109
-082
-000
-100
Figure 4: High-quality simulations produced by BehaviorGPT, where multimodal behaviors of agents are simulated realistically.
that expects the simulations to match the real-world distribution. LINEAR SPEED and LINEAR ACCEL evaluate the realism regarding speed and acceleration. Similarly, ANG SPEED and ANG ACCEL measure the realism of angular speed and acceleration. DIST TO OBJ considers the distances to objects, while COLLISION and TTC assess the simulation performance in terms of collision and time to collision. Finally, DIST TO ROAD EDGE and OFFROAD focus on map compliance.
4.2 Implementation Details
The optimal patch size we experimented with is 10, corresponding to 1 second. All hidden sizes are set to 128. Each attention layer has 8 attention heads with 16 dimensions per head. To save training resources, we limit the maximum number of agents per scenario to 128 and restrict the maximum number of neighbors in kNN attention layers 32. The prediction head produces 16 modes per agent and time step. We train the models for 30 epochs on 8 NVIDIA RTX 4090 GPUs with a batch size of 24, utilizing the AdamW optimizer [3. The weight decay rate and dropout rate are both set to 0.1. The learning rate is initially set to 5 x 10-1 and decayed to O following a cosine annealing schedule [30]. Our results in the 2024 WOSAC are obtained using a single model with 2 decoding blocks and total of 3M parameters. To produce 32 replicas of rollouts, modes from agents' next-patch distributions until completing the 8-second multi-agent trajectories, and we repeat this process with different random seeds. The final results on the leaderboard are based on a replan rate of Hz, while the ablation studies are based on a 1-Hz replan rate unless specified.
randomly sample behavior
4.3 Quantitative Results
We report the test set results in Table Notably, BehaviorGPT achieves the lowest minADE and the best REALISM, underscoring the model's ability to match the real-world distribution. Its excellent performance on COLLISION and OFFROAD also indicates that the model has successfully captured the agent-agent and agent-map interactions in driving scenarios. Besides the benchmarking results, we also compare the number of model parameters in BehaviorGPT and other baselines. Table [] demonstrates that BehaviorGPT, with only 3M parameters, achieves more realistic simulation than significantly larger models like MVTE [49 and GUMP [22,, which demonstrates the parameter efficiency our approach. Without employing tricks like data augmentation, model ensemble, or post-processing steps, BehaviorGPT won first place in the 2024 Waymo Open Sim Agents Challenge.
8

---

-118
1009
-1230
1010
5715
472
4008
-202
-0.
-443 3408 -115 1019-11---000 -100-177 -120
+300
-142 -00--375--00 1100-1155 2260-175-330
-006
-445
+105
+030
Figure 5: A typical failed case produced by BehaviorGPT, where offrood trajectories are generated owing to the compounding error caused by autoregressive modeling.
4.4 Qualitative Results
Figure 4 visualizes some qualitative results of the rollouts produced by our model. In this scenario, BehaviorGPT can generate multiple plausible futures given the same initial states agents, which demonstrates its capability of simulating diverse yet realistic agent behavior. However, we also note that autoregressive models still suffer from accumulated errors some cases, shown in Figure [ the vehicle in orange gradually goes out of the road as time goes by, which indicates the inherent limitations autoregressive generation.
4.5 Ablation Studies
We conduct some ablation studies to gain a more in-depth understanding of our approach.
Impact of patch size. Table presents the results of BehaviorGPT with varying patch sizes. According to the results, it is evident that using a patch size of 5, i.e., training and predicting with 2-Hz tokens, significantly outperforms the baseline without patching. Moreover, increasing the patch size to 10 further enhances the overall performance. These results demonstrate the benefits of incorporating the NP3 into agent simulation. However, changing the patch size also leads to a variation in replan frequency, which also has influence simulation. Next, we investigate the impact of replan frequency on the test set using the model submitted to the 2024 WOSAC.
Table 2: Impact of patch size on the validation set.
Replan Frequency
minADE (1)
REALISM (1)
LINEAR (1)
LINEAR (7)
ACCEL (1)
TNST
COLLISION
DIST EDGE
OFFROAD
(1)
(1)
(1)
Hx
2.3752
0.6733
0.2550
0.2088
0.4022
0.5094
0.3201
0.9002
0.8015
0.149
0.8432
Hz
1,5500
0.7273
0,3543
0.3218
0.4625
0.5435
0.3768
09181
018339
0.6544
0.9077
10
Hi
1,5203
0.7335
0,3517
0.3023
0.4734
0,5432
0.3797
0.9358
0.8329
0.6645
0.9132
Impact of replan frequency. During inference, vary the replan frequency of the model with patch size of 10 by discarding a portion the predicted states at each simulation step. As shown in Table increasing the replan frequency from 1 Hz 2 Hz even improve the overall performance, which may benefit from the enhanced reactivity. This phenomenon demonstrates that the performance gain is not merely due the lower replan frequency, that with patch size of 5 even harder if using the same replan frequency of 2 Hz. However, using an overly high replan frequency harms the performance, as indicated by the third row of Table 3 Overall, we conclude that using a larger patch indeed helps long-term reasoning, but moderate replan firequency is important for temporal stability, which may be neglected by prior works.
the model with patch size of 10 beats
Table 3: Impact of replan frequency on the test set.
Pasch Size
Replan Frequency
minADE (1)
REALISM (7)
LINEAR SPEED
LINEAR ACCEL
ANG SPEED
ANG ACCEL
DIST TO
COLLISION
TTC (1)
DIST TO ROAD EDGE (1)
OFFROAD
(1)
(1)
(f)
(")
10
1.5405
0.7414 0.7342
0.3553
0,3153
0.5313
0.3772
118285
0.6664
0.3834
1.5693
0.3472
0.3/22

---

Impact of multi-agent interaction modeling. We remove all agent-agent self-attention layers in the first row of Table 4 to show that mod- eling the interactions among agents can boost minADE and REALISM. In particular, the real- ism in terms of collision is improved by 34.66% when employing agent-agent self-attention.
Table 4: Impact of agent-agent self-attention the validation set.
Agent- Aggest Self Attencion
minADE (1)
REALISM (1)
COLLISION
(71
(?)
(
2.1489 1.6247
0.6659
0.3539
0,6067
0.9070
0.7349
0,37X3
0,9409
0U3330
Table 5: Effects of training data Table 6: Effects of model depth Table Effects of model width
on the validation set.
on the validation set.
on the validation set.
Trin Data
minADE
REALISM
Model Depth
minADE
REALISM
Model Width
minADE
REALISM
#Param
(1)
(1)
#Param
())
(*)
#Param
(
(t)
20%
SM
14881
0.7396
2
3M
1.6247
0.7349
64
800K
196377
0,7251
50%
5M
14060
0.7427
3
4M
1,5381
0.7387
128
3M
1.6447
0.7349
100%
5M
1,3804
0.7438
4
5M
1.4881
0.7396
192
7M
1,4993
0.7382
Table 8: Extrapolation ability to generate longer sequences.
Training
Inference
minADE (1)
REALISM (1)
LINEAK SPRED
ACCEL
SPEED (1)
ANG ACCRI
TD.RI (?)
COLLISION (1)
TIC <t)
TO ROAD (1)
OFFROAD (1)
(*)
(1)
(?)
9.1 scc
9.1 sec
1,6347
0.7349
0.3546
0.3105
0,4689
0.5363
0.3783
0,9409 0.9375
0.8320 0.8273
0.668S 0.6585
0.9163
5.0 SEC
9.1 sec
L.6991
0,7333
0.3565
0,3471
0.4613
0.5293
0.3813
0
Scaling with data. We train our models with different proportions of training data. All the models have 4 decoding blocks and a hidden size of 128, totaling 5M parameters. As shown in Table 5 BehaviorGPT is able to achieve remarkable performance with merely 20% of training data, which is attributed the high data efficiency of our approach. Increasing the proportion of training data from 20% to 50% further improves the performance on minADE and REALISM, and training on 100% of the data continues to gain enhancement. Judging from the trend in Table 5 more data for model training will continuously achieve better simulation performance.
we believe that feeding
Scaling with model size. We investigate the effects of scaling up the model size based on some preliminary experiments with 20% of training data. In Table we vary the number of decoding blocks while fixing the hidden size as 128. On the other hand, we fix the number of decoding blocks as 2 and vary the hidden size, as depicted in Table 7 Based on the experimental results, we can summarize that enlarging the model consistently leads more realistic simulation, which showcases the potential of BehaviorGPT for scaling up.
Extrapolation ability. We tried training a model on 5-second sequences and generating 9. Del-second sequences during inference. The results in Table 8 show that this model achieves similar perfor- mance compared with the baseline trained with 9.1-second sequences, demonstrating approach's extrapolation ability to generate longer sequences.
5
Conclusion
This work introduced BehaviorGPT, fully autoregressive architecture designed to enhance smart agent simulation for autonomous driving. By applying homogeneous Transformer blocks to entire trajectory snippets and utilizing relative spacetime representations, BehaviorGPT simplifies the modeling process and maximizes data utilization. To enable high-level understanding and long-range interaction reasoning in space and time, we developed the Next-Patch Prediction Paradigm, which tasks models with generating trajectory patches instead of single-step states. Experimental results on the Waymo Open Sim Agents Challenge demonstrate that BehaviorGPT achieves outstanding performance with merely 3M model parameters, highlighting its potential to further improve the realism of agent simulation with more data and computation.
Limitations. First, BehaviorGPT is currently inferior in kinematics- performance, which can be enhanced by incorporating kinematic model, e.g., the bicycle model. Second, the current version of BehaviorGPT does not support controlling agent behavior with specific prompts such as language and goal points. However, achieving controllable generation should be trivial given a powerful base model. Finally, we have not verified whether BehaviorGPT will facilitate the development of motion planning, which we leave future work.
10

---

Acknowledgement
This project is supported by a grant from Hong Kong Research Grant Council under GRF project 11216323 and CRF C1042-23G.
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization, arXiv preprint 1oXiv:607.006500 2016.
[2] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems, 2020.
[3] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers, In International Conference on Learning Representations, 2022.
[4] Luca Bergamini, Yawei Ye, Oliver Scheel, Long Chen, Chih Hu, Luca Del Pero, Blažej Osiński, Hugo Grimmett, and Peter Ondruska. Simnet: Learning reactive drivin simulations from real-world observations. In 2021 1 IEFE International Conference on Robotics and Automation (ICRA), pages 5119-5125. IERE, 2021.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information P Processing Systems, 2020.
[6] Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8748-8757 2019.
[7] Dian Chen, Vladlen Koltun, and Philipp Krähenbühl. Learning to drive from world on rails. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15590-15599, 2021.
[8] Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Conference on Robot Learning, pages 66-75. PMLR, 2020.
[9] Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn en- coder-decoder for statistical machine translation. In Conference Empirical Methods in Natural Language Processing, 2014.
[10] Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End-to-end driving via conditional imitation learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 4693-4700. IEEE, 2018.
[11] Pim De Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. In Advances Neural Information Processing Systems, 2019.
[12] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlshy. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[14] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference Robot Learning, pages 1-16. PMLR, 2017.

---

[15] Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai Ben Sapp. Charles Qi. Yin Zhou, Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9710-9719, 2021.
[16] Jiyang Gao, Chen Sun, Hang Zhao, Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 11525-11533, 2020.
[17] Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Syn- naeve. Better faster large language models via multi-token prediction. arXiv preprint 24X4:24004.19777 2024.
[18] Junru Gu, Chen Sun, and Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15303-1533 2021.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[20] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
[21] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3466 2021.
[22] Yihan Hu, Siqi Chai, Zhening Yang, Jingyu Qian, Kun Li, Wenxin Shao, Haichao Zhang, Wei Xu, and Qiang Liu. Solving motion planning tasks with a scalable generative model. In European Conference on Computer Vision, 2024.
[23] Zhiyu Huang, Zixu Zhang, Ameya Vaidya, Yuxiao Chen, Chen Lv, and Jaime Fernández Fisac. Versatile scene-consistent traffic scenario generation as optimization with diffusion. arXiv preprint arXiv:2404.02: 02524, 2024.
[24] Maximilian Igl, Daewoo Kim, Alex Kuefler, Paul Mougin, Punit Shah, Kyriacos Shiarlis, Dragomir Anguelov, Mark Palatucci, Brandyn White, and Shimon Whiteson. Symphony: Learning realistic and diverse agents for autonomous driving simulation. In 2022 International Conference on Robotics and Automation (ICRA), pages 2445 2451. IEEE. 2022.
[25] Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. Hdgt: Heteroge- neous driving graph transformer for multi-agent trajectory prediction via scene encoding. IEEE transactions on pattern analysis and machine intelligence, 2023.
[26] Ame Kesting, Martin Treiber, and Dirk Helbing. General lane-changing model mobil for car-following models. Transportation Research Record, 1999(1):86-99) 2007.
[27] Parth Kothari, Christian Perone, Luca Bergamini, Alexandre Alahi, and Peter Ondruska. Drivergym: Democratising reinforcement learning for autonomous driving. arXiv preprint arXiv:2111.0688 2021.
[28] Daniel Krajzewicz, Georg Hertkorn, Christian Rössel, and Peter Wagner. Sumo (simulation of urban mobility)-an open-source traffic simulation. In Proceedings of the 4th middle East Symposium on Simulation and Modelling (MESM20002), pages 183-187, 2002.
[29] Quanyi Li, Zhenghao Peng. Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3461-3475 2022.
[30] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
12

---

[31] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[32] Nico Montali, John Lambert, Paul Mougin, Alex Kuefler, Nicholas Rhinehart, Michelle Li, Cole Gulino, Tristan Emrich, Zoey Zeyu Yang, Shimon Whiteson, Brandyn White, and Dragomir Anguelov. The waymo open sim agents challenge. In Advances in Neural Information Process- ing Systems Truck on Datasets and Benchmarks, 2023.
[33] Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Ben- jamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 2980-2987. IEEE, 2023.
[34] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. The Eleventh International Conference on Learning Representations, 2023.
[35] Jonah Philion, Xue Bin Peng, and Sanja Fidler. Trajeglish: Traffic modeling as next-token prediction. In The Twelfth International Conference on Learning Representations, 2024.
[36] Dean A Pomerleau. Alvinn: An autonomous land vehicle neural network. Advances in Neural Information Processing Systems, 1988.
[37] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. OpenAl blog, 2018.
[38] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog. 2019.
[39] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 627-635, 2011.
[40] Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5149- 5152. IEEE, 2012.
[41] Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled Refaat, Rami AI-Rfou, and Benjamin Sapp. Motionlm: Multi-agent motion forecasting as language modeling. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8579-85900 2023.
[42] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. In Advances in Neural Information Processing Systems, 2022.
[43] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bert Schiele. Mir+: Multi-agent motion prediction with symmetric scene modeling and guided intention querying. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[44] Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. Trafficsim: Learning to simulate realistic multi-agent behaviors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10400-10409, 2021.
[45] Martin Treiber, Ansgar Hennecke, and Dirk Helbing. Congested traffic states in empirical observations and microscopic simulations. Physical Review E, 62(2): 1805, 2000.
[46] Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S Refaat, Nigamaa Nayakanti, Andre Cormman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, et al. Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction. In 2022 International Conference on Robotics and Automation (ICRA), pages 7814-7821. IEEE, 2022.
13

---

[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- lion Processing Systems, 2017.
[48] Eugene Vinitsky, Nathan Lichtlé, Xiaomeng Yang, Brandon Amos, and Jakob Nicolaus Foerster. Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world. In Advances in Neural Information Processing Systems Track on Datasets and Benchmarks, 2022.
[49] Yu Wang, Tiebiao Zhao, and Fan Yi. Multiverse transformer: 1st place solution for waymo open sim agents challenge 2023. arXiv preprint arXiv:2306. / 1868, 2023.
[50] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In Advances in Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
[51] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting, In Advances in Neural Information Processing Systems, 2021.
[52] Danfei Xu, Yuxiao Chen, Boris Ivanovic, and Marco Pavone. Bits: simulation. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 2929-2936. IEEE, 2023.
Bi-level imitation for traffic
[53] Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang. and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1389-13999 2023.
[54) Zhejun Zhang, Alexander Liniger, Christos Sakaridis, Fisher Yu, and Lue Van Gool. Real-time motion prediction via heterogeneous polyline transformer with relative pose encoding. In Advances in Neural Information Processing Systems, 2023.
[55] Zhejun Zhang, Christos Sakaridis, and Lue Van Gool. Traffiebots v1. 5: Traffic simulation via conditional vacs and transformers with relative pose encoding. arXiv preprint arXiv:2406. 10898, 2024.
[56] Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone. Guided conditional diffusion for controllable traffic simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3560-3566. IEEE, 2023.
[57] Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, 2021.
[58] Zikang Zhou, Jianping Wang, Yung-Hui Li, and Yu-Kai Huang. Query-centric trajectory prediction. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 17863-17873, 2023.
[59] Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, and Kejie Lu. Hivt: Hierarchical vector transformer for multi-agent motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8823-8833, 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have clearly outlined the scope and the targeted task of this work, and presented concise introduction of approach. The important contributions in detail, and the reported results obtained by our model are tested in the Waymo Open Sim Agents Challenge. This open challenge allows for fair comparisons with other baselines.
described
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of this work in the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification: BehaviorGPT is an application-oriented approach for autonomous driving, No new theory has been proposed in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We've described our approach in detail, including the model architecture, loss function, datasets, and metries used for training and testing. We will also make our code publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The data used in this work is the Waymo Open Motion Dataset, which is publicly available. Our code will also be made public after the paper is published.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
7 //nips. //tips.cc/pub.l.. wwtpc//pppp.guudeeeeoeepppppppee httpi//gguidee//ddeuubiiiiseee SrbodeSubmissionool11..
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: As the data used in this work is the Waymo Open Motion Dataset, the training and testing details are available on the website of Waymo Open Dataset Challenge. We have also clarified other details in Section 3, Section 4, and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: While not conducting significance tests over results, our experiments are conducted on the Waymo Open Motion Dataset, which has large data scale. Thus, the experimental results are stable across multiple trials, and the reported results are reliable and authentic.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The resources used for model training have been introduced clearly in the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp////eaarrrsseccpppp//hhhaaa
Answer: [Yes]
Justification: There violations against the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
18

---

The proposed BehaviorGPT is particularly designed for smart agent simulation, which can benefit the verification of autonomous driving systems and is expected to promote the development of autonomous driving. While autonomous driving itself may be double- sided sword, its negative side is out of the scope this work and the developed BehaviorGPP will not directly bring negative societal impacts.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper has cited the original paper that produced the Waymo Open Motion Dataset, and the information about the dataset is available online.
Guidelines:
19

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
20

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
21


Reviews:
[{"review_id": "yB6YLKvBiK", "reviewer": "Reviewer_HMYm", "summary": "This paper presents BehaviorGPT, a model for multi-agent traffic simulation. BehaviorGPT's architecture is based on a decoder-only transformer that autoregressively predicts patches of trajectories. The key insight is that predicting patches forces the model to learn longer-horizon reasoning. Then, to predict states from patches, BehaviorGPT uses an RNN that outputs a mixture of Gaussians/Laplacians for each agent's x, y, and heading. BehaviorGPT achieves state-of-the-art performance on the Waymo Open Sim Agents Challenge.", "questions": "1. During inference, how do you generate 32 rollouts? Do you randomly sample with a different seed 32 times?\n2. What is the significance of not using ground truth state when unrolling the RNN? Do you have experiments to illustrate why it's necessary?\n3. Can the authors discuss limitations of their method in a more meaningful manner than \"performance can be better\"?", "limitations": "Limitations are mentioned (in the checklist section only) but limited to \"performance could be better\". The paper can be improved with a more thorough discussion of limitations; e.g., what are the reasons why BehaviorGPT still underperforms in certain metrics on Table 1?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed architecture is simple yet incorporates well-reasoned inductive biases like relative positional encodings and factorized self-attention. Despite its simplicity, BehaviorGPT achieves state-of-the-art performance on the Waymo Open Sim Agents Challenge. Moreover, BehaviorGPT does this with 10x fewer parameters than the previous state-of-the-art, Trajeglish.\n- The paper ablates the efficacy of patch-based tokens, convincingly demonstrating that it is critical to having strong performance. To my knowledge, this design choice is novel within multi-agent traffic simulation and its adjacent fields. \n- Code will be made available after publication.\n- The paper is generally well-written and easy-to-follow.", "weaknesses": "- The paper could be strengthened with stronger ablation studies to highlight the design choices that make BehaviorGPT outperform prior work, like Trajeglish. A number of design choices in BehaviorGPT set it apart from Trajeglish, but it is not clear which ones actually contribute to its ability to outperform Trajeglish even with significantly less parameters; e.g., is it the patch-based tokens? is it the factorized attention with relative positional encodings? is it the specific mixture of Gaussians/Laplacians output distribution? Such analysis would give the reader more insight into what design choices are important for multi-agent traffic simulation.\n- The paper does not provide sufficient detail to reproduce the model; e.g., basic hyperparameters like the number of transformer layers. That said, this weakness is mitigated by the authors' intention to publish code upon acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments and constructive suggestions. In the following, we attempt to address your critical concerns by giving more analyses and clarifying the implementation details.\n\n**Q1: What makes BehaviorGPT outperform SOTA with fewer parameters?** \n\n**A1:** Indeed, the most notable design choices that set BehaviorGPT apart from other autoregressive behavior models lie in three perspectives, including (1) the patch-based tokens, (2) the decoder-only Transformers based on relative spacetime representation, and (3) the use of continuous distributions. Let us understand these design choices better with more in-depth analyses.\n\n(1) **Patching mechanism**: Our paper argues that the Next-Patch Prediction scheme can enhance models’ capability of long-range reasoning. In Table 3, we show that using a larger patch (e.g., increasing the patch size from 5 to 10) can improve the performance. However, some may doubt that the performance gain is merely due to the lower replan frequency (e.g., the 1 Hz for a patch size of 10 compared with the 2 Hz for a patch size of 5) as discovered by Waymo [30], which may enhance the temporal stability of trajectories. To clarify this, we try to increase the replan frequency by discarding a portion of the predicted states at each simulation step. The test set results produced by the model with a patch size of 10 are shown as follows. From the table, we can see that increasing the replan frequency from 1 Hz to 2 Hz can even improve the overall performance, which may benefit from the enhanced reactivity. This phenomenon demonstrates that the performance gain is not merely due to the lower replan frequency, as the model with a patch size of 10 beats that with a patch size of 5 even harder if using the same replan frequency. However, we found that an overly high replan frequency harms the performance, as indicated by the third row of the table. Overall, we conclude that using a larger patch indeed helps long-term reasoning, but a moderate replan frequency is important for temporal stability, which may be neglected by prior works.\n| Patch Size | Replan Frequency | minADE $\\downarrow$ | Realism $\\uparrow$ | Offroad $\\uparrow$  |\n| :--- | :---: | :---: | :---: | :---: |\n| 10 | 1 Hz | 1.5405 | 0.7414 | 0.9308 |\n|10 | 2 Hz | **1.4147** | **0.7473** | **0.9349** |\n| 10 | 5 Hz | 1.5693 | 0.7342 | 0.9089 |\n\n(2) **Decoder-only Transformers with relative spacetime representation**: Decoder-only Transformers with relative spacetime representation can utilize training data more efficiently. Compared with encoder-decoder models where each scenario is treated as one pair of history and future, BehaviorGPT models each time step as the current one and requires each state to model subsequent states’ distribution during training, which is equivalent to constructing training samples from every possible history-future pair of the time series. To illustrate the sample efficiency of BehaviorGPT, we evaluate the models (5M parameters, hidden size = 128, #Decoder layer = 4) trained with different proportions of training data. The table below shows that using merely 20% of the data for training has already enabled the model to surpass many SOTA. The high data efficiency allows small models to achieve incredible performance.\n| Training Data | minADE $\\downarrow$ | Realism $\\uparrow$ | Offroad $\\uparrow$  |\n| :--- | :---: | :---: | :---: |\n| 20% | 1.4881 | 0.7396 | 0.9207 |\n| 50% | 1.4060 | 0.7427 | 0.9250 |\n| 100% | 1.3804 | 0.7438 | 0.9268 |\n\n(3) **Choice of distributions**: Unlike standard autoregressive models, we choose continuous distributions for modeling since we prefer more end-to-end solutions. To model traffic scenarios with categorical distributions, we must first conduct discretization, which involves many essential implementation details (e.g., choosing the proper vocabulary size). Given that the code of Trajeglish is not publicly available, it is inappropriate to recklessly conclude something like \"using continuous distributions is better\" by comparing our model with a discretized variant that is not well-tuned. However, we indeed found that the choice of distributions is crucial. Prior to using Laplace distributions, we used Gaussian distributions for parameterization and noted that the model could not converge normally (see Figure c in the general response). Moreover, we observed that directly optimizing the full mixture models is better than using the winner-take-all training strategy (Realism score: 0.7396 vs 0.7146). These are the most critical things to make a continuous autoregressive model work well.\n***\n**Q2: What about the implementation details?**\n\n**A2**: All hidden sizes are 128. All attention layers have 8 attention heads with 16 dimensions per head. We train our models for 30 epochs with a batch size of 24 using the AdamW optimizer. The weight decay rate and the dropout rate are both 0.1. Using the cosine annealing scheduler, we decay the learning rate from $5 \\times 10^{-4}$ to 0. Our main results are produced by a single model with 2 decoder layers and 3M parameters. To obtain 32 replicas of rollouts, we randomly sample a behavior mode from each agent's next-patch distribution using different random seeds.\n***\n**Q3: What is the significance of not using the GT when unrolling the RNN?**\n\n**A3**: We have tried a non-autoregressive GRU that does not rely on the predicted states when unrolling the next states, which can achieve lower minADE (1.5203 vs 1.6554) but performs worse on higher-order kinematic metrics (speed likelihood: 0.3517 vs 0.3544, acceleration likelihood: 0.2630 vs 0.2873).\n***\n**Q4: What are the limitations?**\n\n**A4**: (1) The kinematic performance can be enhanced by incorporating a kinematic model (e.g., bicycle model); (2) BehaviorGPT does not support controlling agent behavior with prompts (e.g., language, goal points); (3) We have not verified whether BehaviorGPT will facilitate the development of motion planning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xzrtJhPJKx", "reviewer": "Reviewer_b92q", "summary": "The paper proposes a decoder only model for traffic simulation. The paper proposes to not predict the next state/token but to predict the next patch a small trajectory segment. The network can select several possible patch generators which are based on RNNs. The resulting architecture achieves strong results with a small transformer based architecture only using about 3M parameters.", "questions": "-Is it correct that you have as many RNNs as modes, with N_mode different parameters?\n\n-How do you select the 32 trajectories for the submission? \n\n-If you keep all modes with p>=0.9 do you not have issues with an excessive amount of trajectories? Or do you just keep the top-1?\n\n-Are there more tricks in the post processing, this is normally important to achieve good results for the sim agent challenge.\n\n-Is it correct that your input tokens are continuous?\n\n-Do you run the patch generation in a receding horizon fashion, where the model would run at 10Hz even though you generate 1s outputs. If not, would it be possible to do so?\n\n-Why is the training so slow? I believed that one of the advantages of a decoder only model compared to more policy learning based methods such as TraffiBot would be training speed. Since there is not autoregressive rollout during the training\n\n-Why do you use a RNN for the patch generation, could a simple MLP not do the same job and be faster?", "limitations": "Limitations could be discussed in more details.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Proposed approach is interesting and works well. The method combines several ideas that have been shown to be efficient and combines them in a network for traffic simulation. The results of the model given its size are impressive and it would be interesting how it performed if the size was scaled up.", "weaknesses": "-The ablation study is not fully clear to me, I can not see a base model, and I also do not know what parameters are used in the final model. \n\n-Most of the ideas in the paper are known in other related fields. Not just predicting the action/next state is known in imitation learning to help with compounding errors. It is also getting used in LMMs [1]. The transformer architecture design is similar to HPTR. \n\n-A lot of details to reproduce the results are not clear (see questions)\n\n[1] Better & Faster Large Language Models via Multi-token Prediction, Gloeckle. et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the valuable feedback. In the following, we answer your questions to resolve some critical concerns.\n\n**Q1: Details of final models and ablation studies.**\n\n**A1**: We list the detailed configurations of each group of experiments as follows, which will be added to the revised version.\n|  Table | Training Data | Evaluation Set | #Param | #Max Agent | #Decoder Layer | Hidden Size | Patch Size | RNN Head | #Neighbor | #Mode | top-p |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| 1 | 100% | Test | 3M |128 | 2 | 128 | 10 | Autoregressive | 32 | 16 | 1.0 |\n| 3 | 20% | Val | 3M | 128 | 2 | 128 | Ablated | Non-Autogressive | 32 | 16 | 1.0 |\n| 4 | 20% | Val | 3M | 64 | 2 | 128 | 10 | Non-Autogressive | Ablated | 8 | 0.95 |\n| 5 | 20% | Val | 3M | 100 | 2 | 128 | 10 | Non-Autogressive | 32 | Ablated | Ablated |\n***\n**Q2: Relevant ideas in related fields.**\n\n**A2**: Thank you for mentioning the relevant ideas in related fields, such as the concurrently proposed multi-token prediction in LLMs and the KNN design of HPTR, which will be discussed in the revised version. However, using patch-wise tokens in decoder-only Transformers is a new attempt to our knowledge. \n***\n**Q3: Implementation Details.**\n\n**A3**: All hidden sizes are set to 128. All attention layers have 8 attention heads with 16 dimensions per head. We train our models for 30 epochs with a batch size of 24 using the AdamW optimizer. The weight decay rate and the dropout rate are both 0.1. Using the cosine annealing scheduler, we decay the learning rate from $5 \\times 10^{-4}$ to 0. Our main results are produced by a single model with 2 decoder layers and 3M parameters. Other details that you are interested in are discussed below.\n\n**(1) Is it correct that you have as many RNNs as modes, with N_mode different parameters?**\n\nThe parameters of the RNN are shared across all modes.\n\n**(2) Sampling strategy.**\n\nWe randomly sample one behavior mode from each agent's next-patch distribution until we complete the 8-second trajectories of all agents. To obtain 32 replicas of rollouts, we repeat this process using different random seeds.\n\n**(3) Post-processing tricks.**\n\nWe do not apply any other post-processing tricks.\n\n**(4) Is it correct that your input tokens are continuous?**\n\nYes, our input and output tokens are all continuous.\n\n**(5) Do you run the patch generation in a receding horizon fashion?**\n\nWe try to increase the replan frequency by discarding a portion of the predicted states at each simulation step. The test set results produced by the model with a patch size of 10 are shown as follows. From the table, we can see that increasing the replan frequency from 1 Hz to 2 Hz can improve the overall performance, which may benefit from the enhanced reactivity. However, we found that an overly high replan frequency harms the performance, as indicated by the third row of the table. The results in Table 1 are based on 1-Hz replan frequency.\n| Patch Size | Replan Frequency | minADE $\\downarrow$ | Realism $\\uparrow$ | Offroad $\\uparrow$  |\n| :--- | :---: | :---: | :---: | :---: |\n| 10 | 1 Hz | 1.5405 | 0.7414 | 0.9308 |\n|10 | 2 Hz | **1.4147** | **0.7473** | **0.9349** |\n| 10 | 5 Hz | 1.5693 | 0.7342 | 0.9089 |\n\n**(6) Why is the training so slow?**\n\nCompared with typical encoder-decoder models, the sequence length in our approach is much longer. For example, most motion forecasting models on the Waymo Open Motion Prediction Benchmark utilize 1-second history to predict 8-second future trajectories, while our decoder-only Transformers simultaneously utilize 1-second, 2-second, ..., and 8-second history to predict the next patch during training.\n\n**(7) Why do we use RNNs for the patch generation?**\n\nWe hope to comply rigorously with the closed-loop requirements of agent simulation via autoregressive RNNs. We have not tested a simple MLP head yet.\n***\n**Q4: What are the limitations?**\n\n**A4**: (1) Currently, BehaviorGPT underperforms in kinematics-related performance (e.g., linear/angular speed likelihood), which can be enhanced by incorporating a kinematic model (e.g., bicycle model); (2) BehaviorGPT does not support controlling agent behavior with prompts (e.g., language, goal points). Future work on agent simulation may consider controllable generation; (3) We have not verified whether BehaviorGPT will facilitate the development of motion planning.\n***\n**Q5: Scaling experiments.**\n\n**A5:** We try to conduct some scaling experiments under the constraints of time and computing budget.\n\n(1) **Hidden size**: We vary the hidden size to obtain models with different parameters. The experiments use 20% of the training data and 2 layers of the Transformer decoder. As depicted below, increasing the hidden size consistently improves the performance.\n| Hidden Size | #Param | minADE $\\downarrow$ | Realism $\\uparrow$ | Linear Speed  $\\uparrow$  | Collision $\\uparrow$  | Offroad $\\uparrow$  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| 64 | 800K | 1.9637 | 0.7251 | 0.3369 | 0.9229 | 0.9056 |\n|128 | 3M | 1.6247 | 0.7349 | 0.3546 | 0.9409 | 0.9163 |\n|192 | 7M | 1.4993 | 0.7382 | 0.3646 | 0.9439 | 0.9185 |\n\n(2) **Number of decoder layers**: We also try to fix the hidden size as 128 and vary the number of decoder layers, obtaining models by training on 20% of the data. Based on the experimental results below, we can conclude that increasing the depth of the models can benefit the performance.\n| #Decoder Layer | #Param | minADE $\\downarrow$ | Realism $\\uparrow$ | Linear Speed  $\\uparrow$  | Collision $\\uparrow$  | Offroad $\\uparrow$  |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| 1 | 2M | 1.7318 | 0.7319 | 0.3465 | 0.9319 | 0.9149 |\n| 2 | 3M | 1.6247 | 0.7349 | 0.3546 | 0.9409 | 0.9163 |\n| 3 | 4M | 1.5381 | 0.7387 | 0.3570 | 0.9450 | 0.9199 |\n| 4 | 5M | 1.4881 | 0.7396 | 0.3633 | 0.9481 | 0.9207 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cO5gHYBiit", "reviewer": "Reviewer_mLGo", "summary": "This work presents BehaviorGPT, a model for trajectory prediction which is decoder only and respects temporal causality by employing a autoregressive sequence model.  They opt for a coarser time resolution of the sequence they call \"patching\" for reasons of efficiency and larger context, in analogy to word-level instead of byte-level representations for language sequence models. After predicting a coarser time segment (\"patch\") embedding, they then decode this into the finer time resolution sequence of states also respecting temporal causality with a RNN decoder within the \"patch\".\n\nThe model is decoder only, in the sense that they can process an infinite temporal stream where agent input representation is the same as agent output representation.  For the static map information, this is encoded once and cross-attended to in the rest of the decoder-only model.  The inner architecture is interleaved attention between map, agents, and time dimension.\n\nThey report exceptional performance results on the Waymo Open Sim Agents Challenge (WOSAC), which requires the models obey temporal causality to respect a sim agents use case. At the same time, their model is 92% smaller than competitive methods, which is substantially more parameter efficient.", "questions": "Q1. I have a big pet peeve with the terminology \"patching\", which is why I mostly keep it in quotes throughout this writeup.  To me (as a native english speaker), the notion of a \"patch\" only makes sense to describe a 2D or 3D region.  Calling a subsequence or segment of a 1D sequence a \"patch\" is very unintuitive.  In language models and in computer science when referring to 1D sequences in general, more common terms are subsequence, (sub)segment, or chunk.  In my opinion, \"chunk\" is nice, and gives me intuition that your process is analogous to this: https://chunkviz.up.railway.app/\n\nSo, not a question, but please consider this!\n\nQ2. How does this model do on the Waymo Open Motion Dataset's Motion Forecasting benchmark? Many papers in Table 1 also report results there, so I am curious.", "limitations": "None", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This is a significantly novel decoder-only model, with reasonable complexity, with very strong results.\n\nThe ideas like \"triple attention\", QCNet's positional encoding are present in other papers.  Predicting coarser state subsequences is a straightforward idea.  But piecing these ideas together into the full decoder only model results in a significant new model (in the specific area of multi-agent trajectory prediction).\n\nWith caveats, the paper is mostly very clearly explained, and will definitely have impact for other practitioners to replicate/extend.", "weaknesses": "The biggest doubt I have about the impact of this paper is: is the impact of the paper \nA. the decoder only architecture or \nB. \"patching\", aka lower resolution modeling for efficiency reasons\n\nWithout patching, the model performs significantly worse than other top methods (compare minADE with patchsize=1 in Table 3 to minADEs in Table 1).  Thus,\n - does decoder-only really matter?\n -  if one added the patching idea to Trajeglish or MVTE or MTR++, would those also perform much better?\n\nMy hunch is patching is doing the heavy lifting here, but that is not clear in the story.\n\nIf this hunch is true, one story of this paper is \"other methods faithfully adhere to the overly-fine 10hz native processing, but BehaviorGPT found a way to bypass that inefficiency\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments! We fully understand your concerns and hope to explain our motivation as well as some subtle details regarding your doubt.\n\n**Q1: Does decoder-only architecture really matter?**\n\n**A1**: Indeed, it is possible to achieve very good results with an encoder-decoder model, but our paper intends to convey the message that \"decoder-only models are sufficient to do very well.\" In other words, it may not be necessary to design some intricate modules to encode the history and employ some well-designed DETR-like decoders for future prediction as done by the most advanced trajectory prediction models, such as Wayformer, MTR, and QCNet. Given that using a much more concise decoder-only architecture can attain super strong results, there is no reason for us to pursue a more complicated solution due to the principle of Occam's razor. Besides the conciseness, combining decoder-only Transformers with relative spatial-temporal positional embeddings has advantages in engineering, as we can utilize the key-value cache technique to reuse past computations. By contrast, typical encoder-decoder models need to encode the history and run the full decoder at each simulation step, which is quite inefficient.\n\nMoreover, we also found that decoder-only models have very high sample efficiency. Compared with encoder-decoder models where each scenario is treated as one pair of history and future, BehaviorGPT models each time step as the current one and requires each state to model subsequent states’ distribution during training, which is equivalent to constructing training samples from every possible history-future pair of the time series. To illustrate the sample efficiency of BehaviorGPT, we evaluate the models (5M parameters, hidden size = 128, #Decoder layer = 4) trained with different proportions of training data. The table below shows that using merely 20% of the data for training has already enabled the model to surpass many SOTA. The high sample efficiency allows small models to achieve incredible performance, which is also an essential reason for using decoder-only architecture.\n| Training Data | minADE $\\downarrow$ | Realism $\\uparrow$ | Offroad $\\uparrow$  |\n| :--- | :---: | :---: | :---: |\n| 20% | 1.4881 | 0.7396 | 0.9207 |\n| 50% | 1.4060 | 0.7427 | 0.9250 |\n| 100% | 1.3804 | 0.7438 | 0.9268 |\n***\n**Q2: If one added the patching idea to other models or lowered the replan frequency, would those also perform much better?**\n\n**A2**: We believe the patching idea can be applied to other autoregressive models (e.g., Trajeglish). But one thing we want to point out is that a lot of top methods on the WOSAC are adapted from SOTA motion prediction models (e.g., MTR). These methods do not necessarily comply with the closed-loop requirements of agent simulation and often produce 8-second trajectory points in one shot. Thus, these open-loop methods have already been operating at an extremely low frequency (i.e., 0.125 Hz, which is forbidden). However, our approach outperforms them by a large margin when operating at a moderate replan frequency.\n\nTo help readers understand how the replan frequency may affect the simulation results, we varied the replan frequency of the same model (patch size = 10) during inference, which can be achieved by discarding a portion of the predicted states at each simulation step. The test set results are shown as follows. From the table, we can see that increasing the replan frequency from 1 Hz to 2 Hz can improve the overall performance, which may benefit from the enhanced reactivity. This phenomenon demonstrates that the performance gain is not merely due to the lower replan frequency, as the model with a patch size of 10 beats that with a patch size of 5 even harder if using the same replan frequency (i.e., 2 Hz). However, we found that an overly high replan frequency harms the performance, as indicated by the third row of the table. Overall, we conclude that using a larger patch indeed helps long-term reasoning, but a moderate replan frequency is important for temporal stability, which may be neglected by prior works.\n| Patch Size | Replan Frequency | minADE $\\downarrow$ | Realism $\\uparrow$ | Offroad $\\uparrow$  |\n| :--- | :---: | :---: | :---: | :---: |\n| 10 | 1 Hz | 1.5405 | 0.7414 | 0.9308 |\n|10 | 2 Hz | **1.4147** | **0.7473** | **0.9349** |\n| 10 | 5 Hz | 1.5693 | 0.7342 | 0.9089 |\n***\n**Q3: Is it appropriate to use the terminology \"patching\"?**\n\n**A3**: Thanks for your comment, and we agree that the notion of a \"patch\" is more suitable for describing 2D or 3D stuff. In fact, BehaviorGPT uses a multi-agent formulation to simulate all agents' states simultaneously. In this sense, our formulation involves multiple dimensions, including the agent dimension, the time dimension, and the state dimension (3D position + 2D velocity + 1D yaw angle). Thus, we think it appropriate to use the notion of a \"patch\" in our case.\n***\n**Q4: What about the motion forecasting results?**\n\n**A4**: As mentioned above, many approaches on the WOSAC leaderboard are adapted from typical marginal motion forecasting models, so they can be tested on the Waymo motion prediction benchmark without any effort. In contrast, BehaviorGPT is a generative model that considers the joint distribution of all agents in the scene, which is misaligned with the objective of the Waymo motion prediction benchmark. Thus, it is more reasonable to compare our approach with other joint multi-agent motion prediction models. In fact, the WOSAC leaderboard also evaluates minADE, the most commonly used metric for motion prediction. We also noted that QCNeXt, a joint multi-agent prediction model that has won 1st place in the CVPR 2023 Argoverse 2 multi-agent motion forecasting challenge, is also on the WOSAC leaderboard. We found that QCNeXt performs much better on minADE (1.08 vs our 1.54), but its closed-loop performance lags far behind most simulation-oriented models. It seems to be a trade-off.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VEZtM1VJa3", "reviewer": "Reviewer_QXEW", "summary": "This paper proposes a new decoder-only learning scheme for autonomous driving dynamics. Rather than using an encoder-decoder type architecture, the model uses spatial, temporal, and “social” attention between map-agent, time-agent, and agent-agent respectively in a time-autoregressive manner. Further, the work explores the usage of “next patch prediction”, where multiple timesteps are bundled, transformed, and unrolled using an RNN. This allows for easier modelling of long-range dependencies and more efficient processing.", "questions": "What happens when scaling the model up? Does the need for patching vanish?\n\nWhat are the results after 20% of training? (particularly interesting because you criticize the data-inefficiency of prior art)\n\nWhy are mixing coefficients only predicted once per patch?\n\nWhy does training take so much time? (I expect a 3M timeseries model to train a lot more quickly)\n\nDoes training time change significantly with larger patches? (due to RNNs needing to be unrolled)\n\nWhy only autoregressive across time and not also across e.g. space?\n\nYou claim that \n> Developed the Next-Patch Prediction scheme to enhance models’ capability of long-range interaction reasoning, leading to more realistic multi-agent simulation over a long horizon;\n\nCan you support this by plotting e.g. accuracy vs. prediction horizon?\nAs is, it is unclear whether the higher performance is actually due to long-horizon performance or just an improvement in the short term and then matching the long-horizon performance.\n\nMore generally: How is the generalisation to longer sequence lengths? Generalisation to longer sequences is a known problem for transformers.", "limitations": "All models considered are really small, making it hard to judge whether the individual components of the model are necessary or just an artifact of the expressivity of small models being low.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Originality**: To our knowledge, this is the first work in Traffic Simulation utilizing decoder-only autoregressive prediction, so there is a significant degree of novelty. \n\n**Quality** The different “arbitrary” selections, such as the number of neighbors to consider, the top-p to sample and patch size are explicitly ablated, which is nice to see. The modelling of the multiple domains present in autonomous driving seems to be sound.\n\n**Clarity**: The paper is generally structured well and readable.\n\n**Significance**: To our knowledge this is the first fully autoregressive model for autonomous driving simulation. The work is also a clear step up from the previous solutions in the WASAC 2023 challenge.", "weaknesses": "**Originality**:  The comparison against previous timeseries-transformers is rather lacking. I would like to see Autoformer and informer mentioned, and, possibly the most relevant here the Space-Time transformer (https://arxiv.org/pdf/2109.12218) which models local interactions within a timestep using graphs (similar to your k-nearest neighbor all2all attention).\n\n**Quality**: Something I would like to see though is a scaling comparison: Transformers tend to perform exponentially better as they scale larger, so seeing something like a 30M model to match the “Trajeglish” work would be nice. It is unclear whether, for instance, the patching is actually necessary to the degree described, or whether this is just an artifact of the model being rather small and shallow. Prior work on the expressivity of transformers, such as https://arxiv.org/abs/2205.11502 has shown that one can map K reasoning steps onto a K-deep model, so it might be that a patching of 10 items is only necessary because the model is too shallow to learn the combination itself. I'm aware that scaling a method might not be possible due to the expense of training a larger model, but without it one cannot be sure of the contributions of patching. I will not see it as a big negative if this is impossible because the performance is good enough to argue for the efficacy of the entire system as a whole, but I'm unsure about the importance of each individual component.\n\n**Clarity**: The notation of the patches is a little hard to parse. I would write \n$$S_i^{((\\tau-1)\\times (l+1))\\ :\\ (\\tau \\times l)}$$\n to make the grouping more clear (It took me way too long to mentally group the terms together…). Specifically, it’s hard to visually group the two sides of the “:” together. Figure 1 is slightly confusing: I understand what you want to show, but maybe put boxes to group the timesteps into tokens such that it becomes clear the transformer predicts the next token which is composed of multiple timesteps.\nSomething I did not quite understand was the tokenization of the map: You say that you sample every 5m and then assign a class to that sample. Does that mean you assign a class to the span from the last sample to the next one (i.e. you assign a class for [0m, 5m], [5m,10m],...) or just at that sample point (i.e. you assign for 0m, 5m, 10m). In the latter case I would assume you can easily jump over crucial information like center lines since those are less than 5m wide.\n\n**Significance**: The small size of BehaviorGPT is nice, but not too interesting for many real world problems: Generally, I would rather have a more accurate 30M model (that e.g. closes the gap to MVTE in acceleration), than a less accurate 3M variant. There are obviously advantages to a model being small, but the runtime difference between a 3M and a 30M parameter model is rarely big enough to make the former worth it (you can run a ResNet 50 with 25M parameters on a raspberry pi in real time…).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments. We will revise the statements, notations, and figures according to your suggestions. In the following, we attempt to address your critical concerns by giving more analyses and clarifying the implementation details.\n\n**Q1: Comparisons with timeseries/spatial-temporal Transformers.**\n\n**A1**: Thank you for pointing out these relevant works. We will discuss them in the revised version.\n***\n**Q2: Scaling experiments.**\n\n**A2**: Experimenting with models as large as Trajeglish is beyond the reach of our computing resources, but we try our best to conduct some scaling experiments under the constraints of limited time and computing budget. We conduct two groups of experiments, including (1) varying the hidden size when fixing the number of decoder layers to be 2 and (2) varying the number of decoder layers when fixing the hidden size to be 128. Although the experimental results show that larger models can achieve better results, the gain in performance also seems to plateau at the scale of 5-7M parameters. we can expect from the trend that continuing to enlarge the model would not bring too significant improvement.\n| Hidden Size | #Param | minADE $\\downarrow$ | Realism $\\uparrow$ | Collision $\\uparrow$  | Offroad $\\uparrow$  |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| 64 | 800K | 1.9637 | 0.7251 | 0.9229 | 0.9056 |\n|128 | 3M | 1.6247 | 0.7349 | 0.9409 | 0.9163 |\n|192 | 7M | 1.4993 | 0.7382 | 0.9439 | 0.9185 |\n\n| #Decoder Layer | #Param | minADE $\\downarrow$ | Realism $\\uparrow$ | Collision $\\uparrow$  | Offroad $\\uparrow$  |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| 1 | 2M | 1.7318 | 0.7319 | 0.9319 | 0.9149 |\n| 2 | 3M | 1.6247 | 0.7349 | 0.9409 | 0.9163 |\n| 3 | 4M | 1.5381 | 0.7387 | 0.9450 | 0.9199 |\n| 4 | 5M | 1.4881 | 0.7396 | 0.9481 | 0.9207 | \n***\n**Q3: Tokenization of the map.**\n\n**A3**: Each map token spans 5 meters, and a semantic category is assigned to the 5-meter segment. We will clarify this in the revised version.\n***\n**Q4: Significance of small models.**\n\n**A4**: We understand that today is the era of large models, but we must point out that many industrial applications, such as autonomous driving, are still using small models (some false advertising on large models might somewhat mislead consumers). As also indicated by the Reviewer gNhc, autonomous driving companies need to run simulations at massive scales before real-world deployment, so simulation efficiency determines how fast we can upgrade the autonomous driving system. In industry, pursuing 1% more realistic simulation with 10x larger models may not be a wise strategy. Thus, we believe small models can also make a great impact on real-world applications.\n***\n**Q5: Does an extremely large model need the patching design?**\n\n**A5**: Since it is impossible for us to train a super-large model, we are unable to reach a conclusion. However, we have shown that the patching design helps small models beat up very large models, demonstrating the value of the patching mechanism--if autonomous driving companies can attain satisfying simulation quality with a small model, there is no reason for them to spend way more money on training and deploying an extremely large model. \n***\n**Q6: Data efficiency.**\n\n**A6**: We evaluate our models (5M parameters, hidden size = 128, #Decoder layer = 4) trained with different proportions of training data. Our model is able to achieve very decent performance when trained on merely 20% of training data, which is attributed to the high data efficiency of our approach.\n| Training Data | minADE $\\downarrow$ | Realism $\\uparrow$ | Linear Speed  $\\uparrow$  | Linear Acceleration $\\uparrow$  | Offroad $\\uparrow$  |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| 20% | 1.4881 | 0.7396 | 0.3633 | 0.3181 | 0.9207 |\n| 50% | 1.4060 | 0.7427 | 0.3637 | 0.3203 | 0.9250 |\n| 100% | 1.3804 | 0.7438 | 0.3655 | 0.3227 | 0.9268 |\n***\n**Q7: Why are mixing coefficients only predicted once per patch?**\n\n**A7**: Because we desire the mixing coefficients to represent the likelihood of *multi-step* behavior.\n***\n**Q8: Why does training take so much time?**\n\n**A8**: Besides model size, the training time is also determined by the computational complexity. On the one hand, our model is trained on 9-second sequences at 10 Hz. On the other hand, the agent simulation task is more than a timeseries problem, as most traffic scenarios involve hundreds of agents. Last but not least, the Waymo Open Motion Dataset is one of the largest datasets in autonomous driving, involving 574 driving hours and a 1.4 TB download size.\n***\n**Q9: Does training time change significantly with larger patches?**\n\n**A9**: We did not notice a significant change in training time when using a patch size of 10, which may be due to the light weight of the RNN head (1 GRU layer with a hidden size of 128).\n***\n**Q10: Why only model the space dimension autoregressively?**\n\n**A10**: While it is natural to model the time dimension autoregressively, it is difficult to determine the order of agents in the chain. Image autoregressive models (e.g., PixelCNN) also face this problem, where the order of pixels in the chain may significantly affect the performance.\n***\n**Q11: Long-term vs short-term.**\n\n**A11**: Please note that scenarios are generated autoregressively rather than in one shot. Without good short-term performance, it is unlikely to have good long-term performance owing to compounding errors.  \n***\n**Q12: How about the extrapolation ability to longer sequences?**\n\n**A12**: We tried training a model on 5-second sequences and generating 9-second sequences during inference. The results below demonstrate the extrapolation ability of our approach.\n| Training | Inference | minADE $\\downarrow$ | Realism $\\uparrow$ | Collision $\\uparrow$ | Offroad $\\uparrow$ |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| 9s | 9s | 1.6247 | 0.7349 | 0.9409 | 0.9163 |\n| 5s | 9s | 1.6294 | 0.7333 | 0.9375 | 0.9100 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9t34mEYS6O", "reviewer": "Reviewer_pxxg", "summary": "This work focuses on multi-agent simulation for autonomous driving. Instead of commonly used encoder-decoder structure, the authors propose a decoder-only autoregressive architecture for better data utilization, and achieve SOTA on Waymo SimAgents Benchmark.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The low data utilization problem using common encoder-decoder structure is identified, which requires a sequence be split into history and future. In proposed autoregressive architecture, each time step is treated as current, resulting in higher data utilization.\n* Next-Patch Prediction Paradigm is introduced to force the model perform long-range interaction, preventing the shortcut learning in next-token prediction.", "weaknesses": "* It seems that no solid evidence is provided to prove the model performance \"scales seamlessly with data and computation\". I do not think Table 4 can support this assertion.\n* Though parameter-efficient, it may not be superior in terms of computation or inference latency, providing more details about this would be better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank your valuable feedback. To resolve your concerns, we try our best to conduct some scaling experiments under the constraints of limited time and computing budget.\n\n**Q1: Can BehaviorGPT scale with data and computation?**\n\n**A1:** We answer this question regarding the quantity of training data, the hidden size, and the number of decoder layers.\n\n(1) **Quantity of data**: We evaluate our models (5M parameters, hidden size = 128, #Decoder layer = 4) trained with different proportions of training data provided by the Waymo Open Motion Dataset. As shown in the table below, our model is able to achieve very decent performance when trained on merely 20% of training data, which is attributed to the high data efficiency of our approach. Increasing the proportion of training data from 20% to 50% further improves the performance across various metrics. Moreover, training on 100% of the data continues to make the model more powerful. Judging from the trend indicated in the table, we believe that feeding more data for model training will continuously improve the overall performance.\n| Training Data | minADE $\\downarrow$ | Realism $\\uparrow$ | Linear Speed  $\\uparrow$  | Linear Acceleration $\\uparrow$  | Offroad $\\uparrow$  |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| 20% | 1.4881 | 0.7396 | 0.3633 | 0.3181 | 0.9207 |\n| 50% | 1.4060 | 0.7427 | 0.3637 | 0.3203 | 0.9250 |\n| 100% | 1.3804 | 0.7438 | 0.3655 | 0.3227 | 0.9268 |\n\n(2) **Hidden size**: We vary the hidden size to obtain models with different parameters. The experiments use 20% of the training data and 2 layers of the Transformer decoder. As depicted by the table below, increasing the hidden size consistently improves the performance.\n| Hidden Size | #Param | minADE $\\downarrow$ | Realism $\\uparrow$ | Linear Speed  $\\uparrow$  | Collision $\\uparrow$  | Offroad $\\uparrow$  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| 64 | 800K | 1.9637 | 0.7251 | 0.3369 | 0.9229 | 0.9056 |\n|128 | 3M | 1.6247 | 0.7349 | 0.3546 | 0.9409 | 0.9163 |\n|192 | 7M | 1.4993 | 0.7382 | 0.3646 | 0.9439 | 0.9185 |\n\n(3) **Number of decoder layers**: We also try to fix the hidden size as 128 and vary the number of decoder layers, obtaining models by training on 20% of the data. Based on the experimental results below, we can conclude that increasing the depth of the models can benefit the performance.\n| #Decoder Layer | #Param | minADE $\\downarrow$ | Realism $\\uparrow$ | Linear Speed  $\\uparrow$  | Collision $\\uparrow$  | Offroad $\\uparrow$  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| 1 | 2M | 1.7318 | 0.7319 | 0.3465 | 0.9319 | 0.9149 |\n| 2 | 3M | 1.6247 | 0.7349 | 0.3546 | 0.9409 | 0.9163 |\n| 3 | 4M | 1.5381 | 0.7387 | 0.3570 | 0.9450 | 0.9199 |\n| 4 | 5M | 1.4881 | 0.7396 | 0.3633 | 0.9481 | 0.9207 |   \n\nIt is a pity that we do not have more computing resources to experiment with even larger models, and we welcome researchers with sufficient computing resources to examine the scalability of our architecture after we contribute our code to the open-source community.\n\n***\n\n**Q2: What about the inference latency?**\n\n**A2**: The average latency per simulation step is less than 10 ms on an NVIDIA L20 GPU (seq length: 9 secs).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uqwPURrZp6", "reviewer": "Reviewer_gNhc", "summary": "BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction introduces a model architecture for multi-agent simulation of dynamic traffic actors.  Improved simulation capabilities are essential to the safe and rapid development of autonomous vehicles.  BehaviorGPT structures multi-agent simulation a next-patch-predition problem.  Given a history of trajectory patches (embeddings of trajectory subsequences of fixed length), the model predicts the next patch's set of states for each actor independently.  These predictions can then be treated as a patch and the process can be repeated in an autoregressive rollout. Through use of relative encodings (in both time and space) and a \"decoder-only\" architecture, BehaviorGPT can be trained in parallel any continuous subset of patches.  BehaviorGPT is state-of-the-art on the Waymo Sim Agents Benchmark with impressive scores for both trajectory accuracy (minADE) as well as realism.  Furthermore, BehaviorGPT accomplishes these SOTA results with only 3M parameters (an order of magnitude lower than other competing approaches).", "questions": "It wasn't clear how many patches were available as history during training or inference?", "limitations": "No concerns.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper has several empirical strengths.  Most notably it is SOTA on a challenging benchmark (the Waymo Sim Agents Benchmark) despite being an surprisingly small model (3M parameters).  Improvement on this benchmark is meaningful as simulation (and the existing sim2real gap) is a major challenge for development and deployment of autonomous vehicles.\n\n- The manuscript does an excellent job of contextualizing this work within the growing cannon of motion forecasting and simulation literature.\n\n- The manuscript does a nice job providing intuitive motivations for their architectural choices.  For example, the RNN in the prediction head is clearly motivated by the sequential nature of trajectory simulation.", "weaknesses": "At a very high level, I think this paper presents a model which produces impressive results on an important benchmark (which represents an important problem).  However, I think that the current manuscript falls short in multiple dimensions.  I do not expect or think that the authors must make substantive changes with respect to all of the points below.  However if the authors were able to expand or address a couple of the weaknesses described below it would significantly improve the strength of the submission.\n\n## 1. Scope down claims slightly\nThe results are super impressive, but I would scope down some of the adjacent claims.  For example, there is the assertion that there is a big sample efficiency gain over MTVA and Trajeglish but there is no evidence to support that.  I also think that the claim of being the first \"decoder only\" model is a bit overstated; this is largely semantic.  Other authors writing the same paper may have called your decoder the encoder and the prediction head the decoder.  Additionally the paper Query-Centric Trajectory Prediction had an awesome ablation in their paper where the reduced the encoder to zero layers and still had excellent results).  Instead of focusing on being decoder only, focus on the ability for further parallelize training.  I also find the statements around \"next patch prediction\" to be a bit strange because you predict states not a patch ... it only becomes a patch when it is embedded by the decoder.\n\n## 2. Reproducibility\nI don't find the current manuscript to be sufficiently detailed to reproduce the model.  (Code release will fix a lot of this).  Some specific examples include:\n- More details are need around map tokenization - I would not know how to translate your text into code here.\n- In the prediction head it is not clear to me if there is a single RNN and a single MLP which is outputing all the mixture components or there are multiple copies (I assume the former ... but its not clear).\n- Embedding sizes?\n\n## 3. Manuscript quality\n- The figures and captions could use work.  In general the captions are far to brief and do not provide enough information to a reader skimming the paper.  The figures themselves, with the exception of figure 2 provide little explanatory value.  Figure 3 is a poor use of space as most of the figure is an identical cartoon repeated three times.  It would be great to replace some of the non-useful figures with visual representations of the model output (i.e. real examples).\n- References need to be cleaned up ... I only skimmed but a couple examples include [18] where Densetnt -> DenseTNT and [48] should cite the NeurIPS 2023 manuscript rather than arxiv.\n- I would steer away from introducing the concept of a scene level patch $P^{\\tau}$ as it is never actually constructed.\n\n## 4. Forecasting metrics/leaderboards\nAs noted in the paper, the simulation problem and the forecasting problem are quite similar.  The minADE numbers suggest that this method would also perform well on motion forecasting benchmarks.  I would love to see numbers there.  I do believe the Waymo Sim Benchmark is an important one, but it is also new and has not had as many competitive submissions as the Waymo or Argoverse forecasting benchmarks.  How does this model compare with QCNet on those benchmarks?\n\n## 5. Model introspection\nIt is not clear from this analysis, which elements of the architecture are actually important for its performance.  I would like to see more ablations.  What happens when we drop out one of the factorized attention modules?  Does the model really get any worse if we don't to  the a2a attention?  Does the model improve/degrade if we only do each attention once, or three times?  Is the RNN actually important?  What if we just predict with an MLP and interpolate?  Are the patches actually important or just going from 10Hz to 1Hz?\n\n## 6. Output introspection\nIt is typical for papers in this space to provide analysis beyond aggregate metrics.  There should be a visual analysis of the output.  Examples should be provided of instances where the model produced excellent output (specifically in comparison to other methods).  Examples should also be provided of instances where the model produced poor outputs.  Are there any connecting themes?  I.e. does the model struggle with certain road geometries or interactions?\n\n## 7. Alternative validation of value\nThe real claim of this paper is that they have produced a model which will produce more realistic simulations which in turn lead to improved development of autonomous vehicles.  Benchmarks are proxy evidence for this claim.  Can this be supported in some other way as well?  If we were to train a planner or motion forecasting method on output from this sim vs. another sim, would it perform better?\n\n## 8. Inference performance\nAutonomous vehicle companies need to run simulations at massive scales. (Tens of millions of scenarios).  To that end it would be nice to see the paper report the inference performance of their model (on reported hardware).\n\n## Conclusion\nAs noted in the beginning, not all of these areas need to be addressed in the limited rebuttal window. 1-3 should definitely be addressed along with some subset of 4-8 (7 is a long shot for sure).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive suggestions, which greatly help improve the quality of our manuscript. In the following, we attempt to address some critical concerns you raised.\n\n**1. Scope down claims slightly**\n\n(1) **Sample efficiency**: We evaluate our models trained with different proportions of training data. Our model is able to achieve very decent performance when trained on merely 20% of data, which is attributed to the high sample efficiency of our approach.\n| Training Data | minADE $\\downarrow$ | Realism $\\uparrow$ | Offroad $\\uparrow$  |\n| :---: | :---: | :---: | :---: | \n| 20% | 1.4881 | 0.7396 | 0.9207 |\n| 50% | 1.4060 | 0.7427 | 0.9250 |\n| 100% | 1.3804 | 0.7438 | 0.9268 |\n\n(2) **Decoder-only**: We agree that the notion of \"decoder-only\" is somewhat semantic. We will revise the introduction of our paper to emphasize our approach's capability of parallel training instead of claiming \"the first decoder-only agent simulator.\"\n\n(3) **Next-patch prediction**: In our approach, each token in the decoder has fused the information of multi-step states and takes charge of generating multiple subsequent states; since we define multiple consecutive states as a patch, we think it appropriate to use the term \"next-patch prediction.\"\n***\n**2. Reproducibility**\n\nMore details on implementation will be supplemented in the revised version. Below are the ones that you are concerned about:\n\n(1) **Map tokenization**: We sample points along the polylines every 5 meters and tokenize the semantic category of each 5-meter segment via learnable embeddings. The shape of the embedding table is [17, 128], indicating 17 categories and a hidden size of 128.\n\n(2) **RNN head**: A single RNN+MLP outputs all the mixture components.\n\n(3) **Hidden size**: All hidden sizes are 128.\n***\n**3. Manuscript quality**\n\n(1) **Figure**: Thank you for the suggestions. We will adjust the figures (e.g., removing the redundant cartoon in Figure 3 and deleting Figure 4) to leave some space for qualitative results. We will also expand the captions to describe more details.\n\n(2) **Reference**: Thank you for spotting the issues. We will clean up the references carefully.\n\n(3) **Notation**: Indeed, the notion of scene-level patches is not that important in our model. We will change Equation 2 into an inline equation.\n***\n**4. Forecasting results**\n\nBehaviorGPT considers the **joint distribution of ALL agents in the scene**, which is misaligned with the objective of existing motion prediction benchmarks. For example, the Waymo Motion Prediction Benchmark and the Argo 2 Single-Agent Motion Forecasting Benchmark are for marginal prediction; the Waymo Interaction Prediction Benchmark and the Argo 2 Multi-Agent Motion Forecasting Benchmark concern the joint distribution of a subset of agents in the scene. Indeed, we can customize our model architecture and training objective to obtain high scores on motion forecasting benchmarks, but that is out of the scope of this work. Thus, it is more reasonable to compare our approach with joint multi-agent motion prediction models that consider the joint distribution of all agents. We also noted that QCNeXt, a joint multi-agent prediction model that has won 1st place in the CVPR 2023 Argo 2 multi-agent motion forecasting challenge, is also on the WOSAC leaderboard. We found that QCNeXt performs much better on minADE (1.08 vs our 1.54), but its closed-loop performance lags far behind most simulation-oriented models. It seems to be a trade-off between open-loop and closed-loop performance.\n***\n**5. Model introspection**\n\n(1) **A2A attention**: Without A2A attention, the model cannot capture agent interactions.\n| A2A | minADE $\\downarrow$ | Realism $\\uparrow$ | Collision $\\uparrow$  |\n| :---: | :---: | :---: | :---: |\n| &check; | 1.6247 | 0.7349 | 0.9409 |\n| &cross; | 2.1489 | 0.6659 | 0.6987 |\n\n\n(2) **#Layers**: We try to fix the hidden size as 128 and vary the number of attention layers. We found that increasing the depth of the models can benefit the performance.\n| #Layer | minADE $\\downarrow$ | Realism $\\uparrow$ | Offroad $\\uparrow$  |\n| :---: | :---: | :---: | :---: |\n| 1 | 1.7318 | 0.7319 | 0.9149 |\n| 2 | 1.6247 | 0.7349 | 0.9163 |\n| 3 | 1.5381 | 0.7387 | 0.9199 |\n| 4 | 1.4881 | 0.7396 | 0.9207 |   \n\n(3) **Patching**: We try to increase the replan frequency by discarding a portion of the predicted states at each simulation step. The test set results produced by the model with a patch size of 10 are shown as follows. From the table, we can see that increasing the replan frequency from 1 Hz to 2 Hz can even improve the overall performance, which may benefit from the enhanced reactivity. This phenomenon demonstrates that the performance gain is not merely due to the lower replan frequency, as the model with a patch size of 10 beats that with a patch size of 5 even harder if using the same replan frequency (i.e., 2Hz). However, we found that an overly high replan frequency harms the performance, as indicated by the third row of the table. Overall, we conclude that using a larger patch indeed helps long-term reasoning, but a moderate replan frequency is important for temporal stability.\n| Patch Size | Replan Frequency | minADE $\\downarrow$ | Realism $\\uparrow$ | Offroad $\\uparrow$  |\n| :---: | :---: | :---: | :---: | :---: |\n| 10 | 1 Hz | 1.5405 | 0.7414 | 0.9308 |\n|10 | 2 Hz | **1.4147** | **0.7473** | **0.9349** |\n| 10 | 5 Hz | 1.5693 | 0.7342 | 0.9089 |\n***\n**6. Output introspection**\n\nPlease refer to the PDF file in the general response.\n***\n**7. Alternative validation**\n\nValidating the usefulness of data-driven simulators for motion planning/prediction is definitely something worthy of doing. This will be our next step!\n***\n**8. Inference**\n\nThe average latency per simulation step is less than 10 ms on an NVIDIA L20 GPU (seq length: 9 secs).\n***\n**9. Number of patches**\n\n9-sec sequences are used for training, while 1-sec trajectories are used as initial history during inference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

Please find the qualitative results and the loss curves in the attached PDF file.

Best,

Authors

Meta-review: The paper proposes a decoder-only multi-agent traffic simulation called "BehaviorGPT", which is based on a transformer model (with only 3M parameters) that predicts patches of trajectories (instead of predicting next states/tokens), which are on a coarser time granularity. A recurrent neural network is used as a patch generator that then outputs a mixture of Gaussians/Laplacians for each agent's x, y, and heading, that generates a fine-grained sequence of states. 

BehaviorGPT achieves state-of-the-art performance on the Waymo Open Sim Agents Challenge while being considerably smaller than competitive methods. 

Decision: All reviewers voted for clear acceptance. I therefore recommend to accept the paper and encourage the authors to use the feedback provided to improve the paper for its final version.
Decision: Accept (poster)