Title: BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction
Venue: NeurIPS 2024
Accepted: yes
Keywords: Multi-Agent Systems

Transformers

Generative Models

Autonomous Driving
Abstract: Simulating realistic behaviors of traffic agents is pivotal for efficiently validating the safety of autonomous driving systems. Existing data-driven simulators primarily use an encoder-decoder architecture to encode the historical trajectories before decoding the future. However, the heterogeneity between encoders and decoders complicates the models, and the manual separation of historical and future trajectories leads to low data utilization. Given these limitations, we propose BehaviorGPT, a homogeneous and fully autoregressive Transformer designed to simulate the sequential behavior of multiple agents. Crucially, our approach discards the traditional separation between "history" and "future" by modeling each time step as the "current" one for motion generation, leading to a simpler, more parameter- and data-efficient agent simulator. We further introduce the Next-Patch Prediction Paradigm (NP3) to mitigate the negative effects of autoregressive modeling, in which models are trained to reason at the patch level of trajectories and capture long-range spatial-temporal interactions. Despite having merely 3M model parameters, BehaviorGPT won first place in the 2024 Waymo Open Sim Agents Challenge with a realism score of 0.7473 and a minADE score of 1.4147, demonstrating its exceptional performance in traffic agent simulation.

BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction

Abstract

Simulating realistic behaviors of traffic agents is pivotal for efficiently validating the safety of autonomous driving systems. Existing data-driven simulators primarily use an encoder-decoder architecture to encode the historical trajectories before decoding the future. However, the heterogeneity between encoders and decoders complicates the models, and the manual separation of historical and future trajectories leads to low data utilization. Given these limitations, we propose BehaviorGPT, a homogeneous and fully autoregressive Transformer designed to simulate the sequential behavior of multiple agents. Crucially, our approach discards the traditional separation between "history" and "future" by modeling each time step as the "current" one for motion generation, leading to a simpler, more parameter- and data-efficient agent simulator. We further introduce the NextPatch Prediction Paradigm (NP3) to mitigate the negative effects of autoregressive modeling, in which models are trained to reason at the patch level of trajectories and capture long-range spatial-temporal interactions. Despite having merely 3M model parameters, BehaviorGPT won first place in the 2024 Waymo Open Sim Agents Challenge with a realism score of 0.7473 and a minADE score of 1.4147, demonstrating its exceptional performance in traffic agent simulation.

Keywords: Multi-Agent Systems, Transformers, Generative Models, Autonomous Driving

1 Introduction

Autonomous driving has emerged as an unstoppable trend, with its rapid development increasing the demand for faithful evaluation of autonomy systems' reliability [32]. While on-road testing can measure driving performance by allowing autonomous vehicles (AVs) to interact with the physical world directly, the high testing cost and the scarcity of safety-critical scenarios in the real world have hindered large-scale and comprehensive evaluation. As an alternative, validating system safety via simulation has become increasingly attractive [14, 48, 53, 44] as it enables rapid testing in diverse driving scenarios simulated at a low cost. This work focuses on smart agent simulation, i.e., simulating the behavior of traffic participants such as vehicles, pedestrians, and cyclists in the digital world, which is critical for efficiently validating and iterating behavioral policies for A Vs.

A good simulator should be realistic, matching the real-world distribution of multi-agent behaviors to support the assessment of AVs' ability to coexist with humans safely. To this end, researchers started

by designing naive simulators that mainly replay the driving logs collected in the real world [27, 29]. When testing new driving policies that deviate from the ones during data collection, agents in such simulators often exhibit unrealistic interactions with A Vs, owing to the lack of reactivity to A Vs' behavior changes. To simulate reactive agents, traditional approaches [14, 28] apply traffic rules to control agents heuristically [45, 26], which may struggle to capture real-world complexity. Recently, the availability of large-scale driving data [6, 15, 50], the emergence of powerful deep learning tools [19, 47, 20], and the prosperity of related fields such as motion forecasting [16, 46, 59, 42, 58], have spurred the development of data-driven agent simulation [44, 4, 24, 52, 56] towards more precise matching of behavioral distribution. With the establishment of standard benchmarks like the Waymo Open Sim Agents Challenge (WOSAC) [32], which systematically evaluates the realism of agent simulation in terms of kinematics, map compliance, and multi-agent interaction, the research on data-driven simulation approaches has been further advanced [49, 35].

Existing learning-based agent simulators [44, 4, 24, 52, 56, 49, 35] mainly mirror the techniques from motion forecasting [16, 46, 59, 42, 58, 41, 18] and opt for an encoder-decoder architecture, presumably due to the similarity between the two fields. Typically, these models use an encoder to extract historical information and a decoder to predict agents' future states leveraging the encoded features. This paradigm requires manually splitting the multi-agent time series into a historical and a future segment, with the two segments being processed by separate encoders and decoders with heterogeneous architecture. For example, MVTA [49] constructs training samples by randomly selecting a 'current' timestamp to divide sequences into historical and future components. Others [52, 35] use fixed-length agent trajectories as historical scene context, conditioned on which the multi-agent future is sampled from the decoder. Nonetheless, the benefit of employing heterogeneous modules to separately encode the history and decode the future, at the cost of significantly complicating the architecture, is unclear. Moreover, the manual separation of history and future leads to low utilization of data and computation: as every point in the sequence can be used for the separation, we believe a sample-efficient framework should be able to learn from every possible history-future pair from the sequence in parallel, which cannot be easily achieved by encoder-decoder solutions owing to their heterogeneous processing for the historical and the future time steps.

Inspired by the success of decoder-only Large Language Models (LLMs) [37, 38, 5], we introduce a fully autoregressive Transformer architecture, dubbed BehaviorGPT, into the field of smart agent simulation to overcome the limitations of previous works. By applying homogeneous Transformer blocks [47] to the complete trajectory snippets without differentiating history and future, we arrive at a simpler, more parameter-efficient, and more sample-efficient solution for agent simulation. Utilizing relative spacetime representations [58], BehaviorGPT symmetrically models each agent state in the sequence as if it were the 'current' one and tasks each state with modeling subsequent states' distribution during training. As a result, our framework maximizes the utilization of traffic data for autoregressive modeling, avoiding wasting any learning signals available in the time series.

Autoregressive modeling with imitation learning, however, suffers from compounding errors [39] and causal confusion [11]. Concerning the behavior simulation task, we observed that blindly mimicking LLMs' training paradigm of next-token prediction [35], regardless of the difference in tokens' semantics across tasks, will make these issues more prominent. For a next-token prediction model embedding tokens at 10 Hz, a low training loss can be achieved by simply copying and pasting the current token as the next one without performing any long-range interaction reasoning in space or time. To mitigate this issue, we introduce the Next-Patch Prediction Paradigm (NP3) that enables models to reason at the patch level of trajectories, as illustrated in Figure 1. By enforcing models to autoregressively generate the next trajectory patch containing multiple time steps, which requires understanding the high-level semantics of agent behaviors and capturing long-range spatialtemporal interactions, we prevent models from leveraging trivial shortcuts during training. We equip BehaviorGPT with NP3 and attain superior performance on WOSAC [32] with merely 3M model parameters, demonstrating the effectiveness of our modeling framework for smart agent simulation.

Our main contributions are three-fold. First, we propose a fully autoregressive architecture for smart agent simulation, which consists of homogeneous Transformer blocks that process multi-agent long sequences with high parameter and sample efficiency. Second, we develop the Next-Patch Prediction scheme to enhance long-range interaction reasoning, leading to more realistic multi-agent simulation over a long horizon. Third, we achieve remarkable performance on the Waymo Open Motion Dataset, winning first place in the 2024 Waymo Open Sim Agents Challenge.

patch size =1

Mask

Mask

Mask

1 step (0.1s) 1 step( 0.1s)

patchsize=5

Mask

o

Mask

oooooo

5 steps (0.5s)

5 steps (0.5s)

patch size=10

Mask

Mask

oo

10 steps (1s)

10 steps (1s)

2 Related Work

2.1 Multi-Agent Traffic Simulation

Multi-agent traffic simulation is essential for developing and testing autonomous driving systems. From early systems like ALVINN [36] to contemporary simulators such as CARLA [14] and SUMO [28], these platforms have used heuristic driving policies to simulate agents' reactive behaviors [8, 7, 10]. However, they struggle to capture real-world complexity since policies based on simple heuristics are not robust enough to handle all sorts of scenarios. With the availability of large-scale data and deep learning approaches, generative models like VAEs [44], GANs [24], Diffusion [56], and autoregressive models [49, 41, 35] have gained success in generating multi-agent motions, which greatly enhance the realism of simulations. Given the temporal dependency of agent trajectories, autoregressive models naturally fit the simulation task, while others require extra designs to capture such dependencies. Among the existing autoregressive models, two representatives are MotionLM [41] and Trajeglish [35]. Both of them adopt an encoder-decoder paradigm, designing complicated scene context encoders to extract historical information before autoregressive decoding. In contrast, our approach is fully autoregressive similar to decoder-only LLMs [37, 38, 5], which eliminates the need for using heterogeneous modules to process the historical and future time steps and achieves higher efficiency in terms of data and parameters via simpler architectural design.

2.2 Patching Operations in Transformers

The application of patches in Transformer models has demonstrated significant potential across various data modalities. For instance, BERT [12] employs subword tokenization [40] for natural language processing, while ViT [13] segments images into 2D patches for visual understanding. The patching design has also found applications in time-series forecasting [51, 57, 34], aiming at retaining local semantics and reducing computational complexity [34]. Moreover, it has shown the effectiveness in self-supervised learning, which has significantly facilitated representation learning and contributed to excellent fine-tuning results on large datasets [2, 21, 3]. Since the task of agent simulation also involves time-series data, we expect the patching mechanism to help models effectively capture the spatial-temporal interactions in driving scenarios and enhance the realism of the generated motion. Our proposed Next-Patch Prediction Paradigm (NP3) utilizes patch-level tokens in autoregressive modeling and trains each token to generate the next patch that comprises multi-step motions, which shares some similarities to multi-token prediction in LLMs [17].

3 Methodology

This section presents the proposed BehaviorGPT for multi-agent behavior simulation, with Figure 2 illustrating the overall framework. To begin with, we provide the formulation of our map-conditioned, multi-agent autoregressive modeling. Then, we detail the architecture of BehaviorGPT, which adopts a Transformer decoder with a triple-attention mechanism to operate sequences at the patch level. Finally, we present the objective for model training.

Agent Data

Map Data

Traffic Scenario

Embedding

Decoder

Tokenization

Temporal

Self-Attention

Agent-Map

Cross-Attention

Agent-Agent

Self-Attention

Head

Pos

0.6

0.3

0.1

0.1: 40km/h

0.3: 50km/h

0.6:

40km/h

0.6

0.3

0.1

Current

Agent State

Predicted Position

Past Agent State

0.3

Predicted Yaw and

Probability

Predicted Velocity

and Probability

0.3: 50km/h

...

...

...

...

...

...

...

...

...

...

...

..

...

Patching

3.1 Problem Formulation

In multi-agent traffic simulation, we aim to simulate agents' future behavior in dynamic and complex environments. Specifically, we define a scenario as the composite of a vector map M and the states of N agent agents over T time steps. At each time step, the state of the i -th agent S i includes the agent's position, velocity, yaw angle, and bounding box size. The semantic type of agents (e.g., vehicles, pedestrians, and cyclists) are also available. Given the sequential nature of agent trajectories, we formulate the problem as sequential predictions over trajectory patches, where the prediction of each patch will affect the subsequent patches. We define an agent-level trajectory patch as

where ℓ is the number of time steps covered by a patch, N patch = T/ℓ indicates the number of patches, and P τ i represents the τ -th trajectory patch of the i -th agent, with S (( τ -1) × ℓ +1):( τ × ℓ ) i denoting the states within the patch. On top of P τ i , we use P τ = S (( τ -1) × ℓ +1):( τ × ℓ ) 1: N agent to denote the τ -th multi-agent patch, where P τ incorporates all agents' states at the τ -th patch. Next, we factorize the multi-agent joint distribution over patches along the time axis according to the chain rule:

where Pr( S 1: T 1: N agent | M ) is the joint distribution of all agents' states over all time steps conditioned on the map M . Further, we factorize over agents the conditional distribution of multi-agent patches based on the assumption that agents plan their motions independently within the horizon of a patch:

Considering the multimodality of agents' behavior within the horizon of a patch, we assume Pr( P τ i | P 1:( τ -1) , M ) to be a mixture model consisting of N mode modes:

where π τ i,k is the probability of the k -th mode. Given the sequential nature of the states within a patch, we further conduct factorization over the states per mode using the chain rule:

Such an autoregressive formulation can be interpreted as planning the patch-level behavior of each agent independently (Eq. (3)), freezing agents' behavior mode per ℓ time steps (Eq. (4)),

and autoregressively unrolling the next state under a specific behavior mode (Eq. (5)). Under this formulation, we can flexibly adjust the replan frequency during inference to control the reactivity of agents. For example, we can let agents execute α ∈ { 1 , . . . , ℓ } steps of the planned motions and choose a new behavior mode after α steps to react to the change in environments.

3.2 Relative Spacetime Representation

In our autoregressive formulation, we treat each trajectory patch as the 'current' patch that is responsible for estimating the next-patch distribution during training, contrasting many existing approaches that designate one current time step per sequence [52, 49, 23]. As a result, it is inefficient to employ the well-established agent- or polyline-centric representation from the field of motion forecasting [46, 59, 33, 42, 25, 54, 43], given that these representations are computed under the reference frames determined by one current time step per sequence. For this reason, we adopt the relative spacetime representation introduced in QCNet [58] to model the patches symmetrically in space and time, achieving simultaneous multi-agent prediction when implementing Eq. (3) and allowing parallel next-patch prediction for the modeling of Eq. (2). Under this representation, the features of each map element and agent state are derived from coordinate-independent attributes, e.g., the semantic category of a map element and the speed of an agent state. On top of this, we effectively maintain the spatial-temporal relationships between input elements via relative positional embeddings. Specifically, we use i and j to index two different input elements and compute the relative spatial-temporal embedding by

where R j → i is the relational embedding from j to i , ∥ d j → i ∥ is the Euclidean distance between them, ∠ ( n i , d j → i ) is the angle between n i (i.e., the orientation of i ) and d j → i (i.e., the displacement vector from j to i ), ∆ θ j → i / ∆ z j → i is the relative yaw/height from j to i , and ∆ τ j → i is the time difference.

3.3 Map Tokenization and Agent Patching

Before performing spatial-temporal relational reasoning among the input elements of a traffic scenario, we must convert the raw information into high-dimensional embeddings. We first embed map information by sampling points along map polylines every 5 meters and tokenizing the semantic category of each 5 -meter segment (e.g., lane centerlines, road edges, and crosswalks) via learnable embeddings. The i -th polyline segment's embedding is denoted by ˆ M i , which does not include any information about coordinates. On the other hand, we process agent states using attention-based patching to obtain patch-level embeddings of trajectories. For the i -th agent's state S t i at time step t , we employ an MLP to transform the speed, the velocity vector's angle relative to the bounding box's heading, the size of the bounding box, and the semantic type of the agent, into a feature vector ˆ S t i . To further acquire patch embeddings, we collect the feature vectors of ℓ consecutive agent states and apply the attention mechanism with relative positional embeddings to them:

where ˆ P τ i is the patch embedding of the i -th agent at the τ -th patch, MHSA ( · ) denotes the multi-head self-attention [47], [: , :] denotes concatenation, and R t → ( τ × ℓ ) i indicates the positional embedding of S t i relative to S τ × ℓ i computed according to Eq. (6). Such an operation can be viewed as aggregating the features of S (( τ -1) × ℓ +1):( τ × ℓ -1) i into ˆ S τ × ℓ i and using the embeddings fused with high-level semantics as the agent tokens in the subsequent modules.

3.4 Triple-Attention Transformer Decoder

After obtaining map tokens and the patch embeddings of agents, we employ a Transformer decoder [47] with the triple-attention mechanism to model the spatial-temporal interactions among scene elements. As illustrated in Figure 3, the triple-attention mechanism considers three distinct sources of relations in the scene, including the temporal dependencies over the trajectory patches per agent, the regulations of the map elements on the agents, and the social interactions among agents.

Temporal Self-Attention. This module captures the relationships among the trajectory patches of each individual agent. Similar to decoder-only LLMs [37, 38, 5], it leverages the multi-head

Attention

Attention

Attention

Tensor

Tensor

Tensor

Tensor

Tensor

Tensor

Tensor

Tensor

Tensor

Tensor

Tensor

Tensor

91 steps (9.1s)

Vehicle01

Vehicle01

Vehicle02

(a)TemporalSelf-Attention

(b)Agent-MapCross-Attention

(c)Agent-AgentSelf-Attention

self-attention (MHSA) with a causal mask to enforce each trajectory patch to only attend to the preceding patches of the same agent, accommodating our autoregressive formulation. The temporal MHSA is equipped with relative positional embeddings:

where F τ a 2 t,i and ˆ P τ i are the temporal-aware feature vector and the patch embedding of the i -th agent at the τ -th patch, respectively, and R t × ℓ → τ × ℓ i embeds the relative position from S t × ℓ i to S τ × ℓ i , which represents the spatial-temporal relationship between the patches P t i and P τ i .

Agent-Map Cross-Attention. Unlike natural language which only has a sequence dimension, we must also conduct spatial reasoning to consider the environmental influence on agents' behavior. To facilitate the modeling of agent-map interactions, we apply the multi-head cross-attention (MHCA) to each trajectory patch in the scenario. Considering that a scenario may comprise an explosive number of map polyline segments and that an agent would not be influenced by map elements far away, we filter the key/value map elements in MHCA using the k-nearest neighbors algorithm [42, 54]. The agent-map cross-attention is formulated as

where F τ a 2 m,i is the map-aware feature vector for the i -th agent at the τ -th patch, ˆ M j is the embedding of the j -th map polyline segment, R τ × ℓ j → i is the relative positional embedding between the agent state S τ × ℓ i and the j -th map polyline segment, and N ( i, τ ) denotes the k-nearest map neighbors of S τ × ℓ i .

Agent-Agent Self-Attention. We further capture the social interactions among agents by applying the MHSA to the space dimension of the trajectory patches. In this module, we also utilize the locality assumption induced by the k-nearest neighbor selection for better computational and memory efficiency. Specifically, the map-aware features of trajectory patches are refined by

where F τ a 2 a,i is the feature vector enriched with spatial interaction information among agents for the i -th agent at the τ -th patch, R τ × ℓ j → i contains the relative information between the i -th and the j -th agent at the τ -th patch, and N ( i, τ ) filters the k-nearest agent neighbors of S τ × ℓ i .

Overall Decoder Architecture. Each of the attention layers above is enhanced by commonly used components in Transformers [47], including feed-forward networks, residual connections [19], and Layer Normalization [1] in a pre-norm fashion. To enable higher-order relational reasoning, we stack multiple triple-attention blocks by interleaving the three Transformer layers. We denote the ultimate feature of the i -th agent at the τ -th patch as F τ i , which will serve as the input of the prediction head for next-patch prediction modeling.

3.5 Next-Patch Prediction Head

Given the interaction-aware patch features output by the Transformer decoder, we develop a nextpatch prediction head to model the marginal multimodal distribution of agent trajectories, which estimates the distributional parameters of each patch's successor.

The following describes the process of next-patch prediction regarding the τ -th patch of the i -th agent. Based on the attention output F τ i , we intend to estimate the parameters of the next patch's mixture model pre-defined with N mode modes. First, we use an MLP to transform F τ i into π τ +1 i ∈ R N mode , the mixing coefficient of the modes. In each mode, the conditional distribution of the next agent state, as depicted in Eq. (5), is considered a multivariate marginal distribution that parameterizes the position and velocity components as Laplace distributions and the yaw angle as a von Mises distribution. Based on this formulation, we employ a GRU-based autoregressive RNN [9] to unroll the states within the next patch step by step, with each step being conditioned on the previously predicted states. Specifically, The hidden state h τ,t i,k of the RNN is initialized with F τ i at t = 1 for ∀ k ∈ { 1 , . . . , N modes } . At each step of the rollout, we use an MLP to estimate the location and scale parameters of the next agent state's position and velocity based on the hidden state. On the other hand, the MLP also estimates the location and concentration parameters of the next yaw angle. The location parameters of the newly predicted state, including the 3D positions, the 2D velocities, and the yaw angle, are used to update the RNN's hidden state directly without relying on the predicted scale/concentration parameters for sampling. The whole process is summarized as follows:

where { µ τ × ℓ + t i,k ∈ R 6 } t ∈{ 1 ,...,ℓ } , { b τ × ℓ + t i,k ∈ R 5 } t ∈{ 1 ,...,ℓ } , and { κ τ × ℓ + t i,k ∈ R } t ∈{ 1 ,...,ℓ } are the location, scale, and concentration parameters in the k -th mode, and Z k is the k -th learnable mode embedding.

3.6 Training Objective

To train BehaviorGPT, we apply the negative log-likelihood loss L NLL to the factorized distribution of Pr( S 1: T 1: N agent | M ) as formulated previously:

Note that each ground-truth trajectory patch is transformed into the viewpoint of its previous patch. During training, we utilize teacher forcing to parallelize the modeling of next-patch prediction and ease the learning difficulty, but we do not use the ground-truth agent states when updating the RNN's hidden states, intending to train the model to recover from its mistakes made in next-state prediction.

4 Experiments

This section first introduces the dataset and the evaluation metrics used in our experiments, followed by presenting the implementation details and the rollout results obtained by BehaviorGPT on the Waymo Open Sim Agents Benchmark [32]. Finally, we conduct ablation studies to further compare and analyze the performance of BehaviorGPT under various settings.

4.1 Dataset and Metrics

Our experiments are conducted on the Waymo Open Motion Dataset (WOMD) [15]. The dataset comprises 486,995/44,097/44,920 training/validation/testing scenarios. Each scenario includes 91 -step observations sampled at 10 Hz, totaling 9 . 1 seconds. Given 11 -step initial states of the scenarios, we simulate up to 128 agents and generate 80 simulation steps per agent at 0 . 1 -second intervals in an autoregressive and reactive manner. Each agent requires 32 simulations comprising x/y/z centroid coordinates and a heading value. The results on the test set are obtained by utilizing the full training set, while the performance on the validation set is based on 20% of training data unless specified.

We use various metrics for evaluation. The minADE measures the minimum average displacement error over multiple simulated trajectories, assessing trajectory accuracy. REALISM is the meta-metric

Original Scenario

Predicted Scenario 1

Predicted Scenario 2

Predicted Scenario 3

1250

1250

1250

1250

1275

1275

1275

1275

1300

1300

-1300

1300

1325

1325

1325

1325

1350

1350

1350

1350

1375

1375

1375

1375

1400

1400

1400

1400

1425

1425

1425

1425

2575-25502525-25002475-2450-24252400

2575 2550 2525 2500 2475 2450 2425 2400

25752550-252525002475-24502425-2400

2575-2550-2525-2500-2475-2450-2425-2400

that expects the simulations to match the real-world distribution. LINEAR SPEED and LINEAR ACCEL evaluate the realism regarding speed and acceleration. Similarly, ANG SPEED and ANG ACCEL measure the realism of angular speed and acceleration. DIST TO OBJ considers the distances to objects, while COLLISION and TTC assess the simulation performance in terms of collision and time to collision. Finally, DIST TO ROAD EDGE and OFFROAD focus on map compliance.

4.2 Implementation Details

The optimal patch size we experimented with is 10 , corresponding to 1 second. All hidden sizes are set to 128 . Each attention layer has 8 attention heads with 16 dimensions per head. To save training resources, we limit the maximum number of agents per scenario to 128 and restrict the maximum number of neighbors in kNN attention layers to 32 . The prediction head produces 16 modes per agent and time step. We train the models for 30 epochs on 8 NVIDIA RTX 4090 GPUs with a batch size of 24 , utilizing the AdamW optimizer [31]. The weight decay rate and dropout rate are both set to 0 . 1 . The learning rate is initially set to 5 × 10 -4 and decayed to 0 following a cosine annealing schedule [30]. Our results in the 2024 WOSAC are obtained using a single model with 2 decoding blocks and a total of 3M parameters. To produce 32 replicas of rollouts, we randomly sample behavior modes from agents' next-patch distributions until completing the 8 -second multi-agent trajectories, and we repeat this process with different random seeds. The final results on the leaderboard are based on a replan rate of 2 Hz, while the ablation studies are based on a 1 -Hz replan rate unless specified.

4.3 Quantitative Results

We report the test set results in Table 1. Notably, BehaviorGPT achieves the lowest minADE and the best REALISM, underscoring the model's ability to match the real-world distribution. Its excellent performance on COLLISION and OFFROAD also indicates that the model has successfully captured the agent-agent and agent-map interactions in driving scenarios. Besides the benchmarking results, we also compare the number of model parameters in BehaviorGPT and other baselines. Table 1 demonstrates that BehaviorGPT, with only 3M parameters, achieves more realistic simulation than significantly larger models like MVTE [49] and GUMP [22], which demonstrates the parameter efficiency of our approach. Without employing tricks like data augmentation, model ensemble, or post-processing steps, BehaviorGPT won first place in the 2024 Waymo Open Sim Agents Challenge.

Original Scenario

Predicted Scenario 1

Predicted Scenario 2

Predicted Scenario 3

5175

5175

5175

5175

5200

5200

5200

5200

5225

5225

5225

5225

5250

5250

5250

5250

5275

5275

5275

5275

5300

5300

5300

5300

5325

5325

5325

5325

5350

5350

5350

5350

4.4 Qualitative Results

Figure 4 visualizes some qualitative results of the rollouts produced by our model. In this scenario, BehaviorGPT can generate multiple plausible futures given the same initial states of agents, which demonstrates its capability of simulating diverse yet realistic agent behavior. However, we also note that autoregressive models still suffer from accumulated errors in some cases. As shown in Figure 5, the vehicle in orange gradually goes out of the road as time goes by, which indicates the inherent limitations of autoregressive generation.

4.5 Ablation Studies

We conduct some ablation studies to gain a more in-depth understanding of our approach.

Impact of patch size. Table 2 presents the results of BehaviorGPT with varying patch sizes. According to the results, it is evident that using a patch size of 5 , i.e., training and predicting with 2 -Hz tokens, significantly outperforms the baseline without patching. Moreover, increasing the patch size to 10 further enhances the overall performance. These results demonstrate the benefits of incorporating the NP3 into agent simulation. However, changing the patch size also leads to a variation in replan frequency, which also has an influence on simulation. Next, we investigate the impact of replan frequency on the test set using the model submitted to the 2024 WOSAC.

Impact of replan frequency. During inference, we vary the replan frequency of the model with a patch size of 10 by discarding a portion of the predicted states at each simulation step. As shown in Table 3, increasing the replan frequency from 1 Hz to 2 Hz can even improve the overall performance, which may benefit from the enhanced reactivity. This phenomenon demonstrates that the performance gain is not merely due to the lower replan frequency, as the model with a patch size of 10 beats that with a patch size of 5 even harder if using the same replan frequency of 2 Hz. However, using an overly high replan frequency harms the performance, as indicated by the third row of Table 3. Overall, we conclude that using a larger patch indeed helps long-term reasoning, but a moderate replan frequency is important for temporal stability, which may be neglected by prior works.

Impact of multi-agent interaction modeling.

We remove all agent-agent self-attention layers in the first row of Table 4 to show that modeling the interactions among agents can boost minADE and REALISM. In particular, the realism in terms of collision is improved by 34 . 66% when employing agent-agent self-attention.

Scaling with data. We train our models with different proportions of training data. All the models have 4 decoding blocks and a hidden size of 128 , totaling 5M parameters. As shown in Table 5, BehaviorGPT is able to achieve remarkable performance with merely 20% of training data, which is attributed to the high data efficiency of our approach. Increasing the proportion of training data from 20% to 50% further improves the performance on minADE and REALISM, and training on 100% of the data continues to gain enhancement. Judging from the trend in Table 5, we believe that feeding more data for model training will continuously achieve better simulation performance.

Scaling with model size. We investigate the effects of scaling up the model size based on some preliminary experiments with 20% of training data. In Table 6, we vary the number of decoding blocks while fixing the hidden size as 128 . On the other hand, we fix the number of decoding blocks as 2 and vary the hidden size, as depicted in Table 7. Based on the experimental results, we can summarize that enlarging the model consistently leads to more realistic simulation, which showcases the potential of BehaviorGPT for scaling up.

Extrapolation ability. We tried training a model on 5 -second sequences and generating 9 . 1 -second sequences during inference. The results in Table 8 show that this model achieves similar performance compared with the baseline trained with 9 . 1 -second sequences, demonstrating our approach's extrapolation ability to generate longer sequences.

5 Conclusion

This work introduced BehaviorGPT, a fully autoregressive architecture designed to enhance smart agent simulation for autonomous driving. By applying homogeneous Transformer blocks to entire trajectory snippets and utilizing relative spacetime representations, BehaviorGPT simplifies the modeling process and maximizes data utilization. To enable high-level understanding and long-range interaction reasoning in space and time, we developed the Next-Patch Prediction Paradigm, which tasks models with generating trajectory patches instead of single-step states. Experimental results on the Waymo Open Sim Agents Challenge demonstrate that BehaviorGPT achieves outstanding performance with merely 3M model parameters, highlighting its potential to further improve the realism of agent simulation with more data and computation.

Limitations. First, BehaviorGPT is currently inferior in kinematics-related performance, which can be enhanced by incorporating a kinematic model, e.g., the bicycle model. Second, the current version of BehaviorGPT does not support controlling agent behavior with specific prompts such as language and goal points. However, achieving controllable generation should be trivial given a powerful base model. Finally, we have not verified whether BehaviorGPT will facilitate the development of motion planning, which we leave as future work.

References

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems , 2020.

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations , 2022.

Luca Bergamini, Yawei Ye, Oliver Scheel, Long Chen, Chih Hu, Luca Del Pero, Bła˙ zej Osi´ nski, Hugo Grimmett, and Peter Ondruska. Simnet: Learning reactive self-driving simulations from real-world observations. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 5119-5125. IEEE, 2021.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems , 2020.

Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8748-8757, 2019.

Dian Chen, Vladlen Koltun, and Philipp Krähenbühl. Learning to drive from a world on rails. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15590-15599, 2021.

Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Conference on Robot Learning , pages 66-75. PMLR, 2020.

Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing , 2014.

Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End-to-end driving via conditional imitation learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 4693-4700. IEEE, 2018.

Pim De Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. In Advances in Neural Information Processing Systems , 2019.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics , 2019.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on Robot Learning , pages 1-16. PMLR, 2017.

Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9710-9719, 2021.

Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11525-11533, 2020.

Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. arXiv preprint arXiv:2404.19737 , 2024.

Junru Gu, Chen Sun, and Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15303-15312, 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems , 2020.

Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 29:3451-3460, 2021.

Yihan Hu, Siqi Chai, Zhening Yang, Jingyu Qian, Kun Li, Wenxin Shao, Haichao Zhang, Wei Xu, and Qiang Liu. Solving motion planning tasks with a scalable generative model. In European Conference on Computer Vision , 2024.

Zhiyu Huang, Zixu Zhang, Ameya Vaidya, Yuxiao Chen, Chen Lv, and Jaime Fernández Fisac. Versatile scene-consistent traffic scenario generation as optimization with diffusion. arXiv preprint arXiv:2404.02524 , 2024.

Maximilian Igl, Daewoo Kim, Alex Kuefler, Paul Mougin, Punit Shah, Kyriacos Shiarlis, Dragomir Anguelov, Mark Palatucci, Brandyn White, and Shimon Whiteson. Symphony: Learning realistic and diverse agents for autonomous driving simulation. In 2022 International Conference on Robotics and Automation (ICRA) , pages 2445-2451. IEEE, 2022.

Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. Hdgt: Heterogeneous driving graph transformer for multi-agent trajectory prediction via scene encoding. IEEE transactions on pattern analysis and machine intelligence , 2023.

Arne Kesting, Martin Treiber, and Dirk Helbing. General lane-changing model mobil for car-following models. Transportation Research Record , 1999(1):86-94, 2007.

Parth Kothari, Christian Perone, Luca Bergamini, Alexandre Alahi, and Peter Ondruska. Drivergym: Democratising reinforcement learning for autonomous driving. arXiv preprint arXiv:2111.06889 , 2021.

Daniel Krajzewicz, Georg Hertkorn, Christian Rössel, and Peter Wagner. Sumo (simulation of urban mobility)-an open-source traffic simulation. In Proceedings of the 4th middle East Symposium on Simulation and Modelling (MESM20002) , pages 183-187, 2002.

Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(3):3461-3475, 2022.

Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations , 2017.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.

Nico Montali, John Lambert, Paul Mougin, Alex Kuefler, Nicholas Rhinehart, Michelle Li, Cole Gulino, Tristan Emrich, Zoey Zeyu Yang, Shimon Whiteson, Brandyn White, and Dragomir Anguelov. The waymo open sim agents challenge. In Advances in Neural Information Processing Systems Track on Datasets and Benchmarks , 2023.

Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 2980-2987. IEEE, 2023.

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations , 2023.

Jonah Philion, Xue Bin Peng, and Sanja Fidler. Trajeglish: Traffic modeling as next-token prediction. In The Twelfth International Conference on Learning Representations , 2024.

Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In Advances in Neural Information Processing Systems , 1988.

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. OpenAI blog , 2018.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 2019.

Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 627-635, 2011.

Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages 51495152. IEEE, 2012.

Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S Refaat, Rami Al-Rfou, and Benjamin Sapp. Motionlm: Multi-agent motion forecasting as language modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8579-8590, 2023.

Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. In Advances in Neural Information Processing Systems , 2022.

Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Mtr++: Multi-agent motion prediction with symmetric scene modeling and guided intention querying. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024.

Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. Trafficsim: Learning to simulate realistic multi-agent behaviors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10400-10409, 2021.

Martin Treiber, Ansgar Hennecke, and Dirk Helbing. Congested traffic states in empirical observations and microscopic simulations. Physical Review E , 62(2):1805, 2000.

Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, et al. Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction. In 2022 International Conference on Robotics and Automation (ICRA) , pages 7814-7821. IEEE, 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , 2017.

Eugene Vinitsky, Nathan Lichtlé, Xiaomeng Yang, Brandon Amos, and Jakob Nicolaus Foerster. Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world. In Advances in Neural Information Processing Systems Track on Datasets and Benchmarks , 2022.

Yu Wang, Tiebiao Zhao, and Fan Yi. Multiverse transformer: 1st place solution for waymo open sim agents challenge 2023. arXiv preprint arXiv:2306.11868 , 2023.

Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In Advances in Neural Information Processing Systems Track on Datasets and Benchmarks , 2021.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Advances in Neural Information Processing Systems , 2021.

Danfei Xu, Yuxiao Chen, Boris Ivanovic, and Marco Pavone. Bits: Bi-level imitation for traffic simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 2929-2936. IEEE, 2023.

Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1389-1399, 2023.

Zhejun Zhang, Alexander Liniger, Christos Sakaridis, Fisher Yu, and Luc Van Gool. Real-time motion prediction via heterogeneous polyline transformer with relative pose encoding. In Advances in Neural Information Processing Systems , 2023.

Zhejun Zhang, Christos Sakaridis, and Luc Van Gool. Trafficbots v1. 5: Traffic simulation via conditional vaes and transformers with relative pose encoding. arXiv preprint arXiv:2406.10898 , 2024.

Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone. Guided conditional diffusion for controllable traffic simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 3560-3566. IEEE, 2023.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence , 2021.

Zikang Zhou, Jianping Wang, Yung-Hui Li, and Yu-Kai Huang. Query-centric trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17863-17873, 2023.

Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, and Kejie Lu. Hivt: Hierarchical vector transformer for multi-agent motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8823-8833, 2022.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have clearly outlined the scope and the targeted task of this work, and presented a concise introduction of our approach. The important contributions are described in detail, and the reported results obtained by our model are tested in the Waymo Open Sim Agents Challenge. This open challenge allows for fair comparisons with other baselines.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations of this work in the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: BehaviorGPT is an application-oriented approach for autonomous driving. No new theory has been proposed in this work.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We've described our approach in detail, including the model architecture, loss function, datasets, and metrics used for training and testing. We will also make our code publicly available.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The data used in this work is the Waymo Open Motion Dataset, which is publicly available. Our code will also be made public after the paper is published.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: As the data used in this work is the Waymo Open Motion Dataset, the training and testing details are available on the website of Waymo Open Dataset Challenge. We have also clarified other details in Section 3, Section 4, and the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: While not conducting significance tests over results, our experiments are conducted on the Waymo Open Motion Dataset, which has a large data scale. Thus, the experimental results are stable across multiple trials, and the reported results are reliable and authentic.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The resources used for model training have been introduced clearly in the implementation details.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: There are no violations against the Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The proposed BehaviorGPT is particularly designed for smart agent simulation, which can benefit the verification of autonomous driving systems and is expected to promote the development of autonomous driving. While autonomous driving itself may be a doublesided sword, its negative side is out of the scope of this work and the developed BehaviorGPT will not directly bring negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper has cited the original paper that produced the Waymo Open Motion

Dataset, and the information about the dataset is available online.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction introduces a model architecture for multi-agent simulation of dynamic traffic actors.  Improved simulation capabilities are essential to the safe and rapid development of autonomous vehicles.  BehaviorGPT structures multi-agent simulation a next-patch-predition problem.  Given a history of trajectory patches (embeddings of trajectory subsequences of fixed length), the model predicts the next patch's set of states for each actor independently.  These predictions can then be treated as a patch and the process can be repeated in an autoregressive rollout. Through use of relative encodings (in both time and space) and a \"decoder-only\" architecture, BehaviorGPT can be trained in parallel any continuous subset of patches.  BehaviorGPT is state-of-the-art on the Waymo Sim Agents Benchmark with impressive scores for both trajectory accuracy (minADE) as well as realism.  Furthermore, BehaviorGPT accomplishes these SOTA results with only 3M parameters (an order of magnitude lower than other competing approaches).\n\nSTRENGTHS:\n- The paper has several empirical strengths.  Most notably it is SOTA on a challenging benchmark (the Waymo Sim Agents Benchmark) despite being an surprisingly small model (3M parameters).  Improvement on this benchmark is meaningful as simulation (and the existing sim2real gap) is a major challenge for development and deployment of autonomous vehicles.\n\n- The manuscript does an excellent job of contextualizing this work within the growing cannon of motion forecasting and simulation literature.\n\n- The manuscript does a nice job providing intuitive motivations for their architectural choices.  For example, the RNN in the prediction head is clearly motivated by the sequential nature of trajectory simulation.\n\nWEAKNESSES:\nAt a very high level, I think this paper presents a model which produces impressive results on an important benchmark (which represents an important problem).  However, I think that the current manuscript falls short in multiple dimensions.  I do not expect or think that the authors must make substantive changes with respect to all of the points below.  However if the authors were able to expand or address a couple of the weaknesses described below it would significantly improve the strength of the submission.\n\n## 1. Scope down claims slightly\nThe results are super impressive, but I would scope down some of the adjacent claims.  For example, there is the assertion that there is a big sample efficiency gain over MTVA and Trajeglish but there is no evidence to support that.  I also think that the claim of being the first \"decoder only\" model is a bit overstated; this is largely semantic.  Other authors writing the same paper may have called your decoder the encoder and the prediction head the decoder.  Additionally the paper Query-Centric Trajectory Prediction had an awesome ablation in their paper where the reduced the encoder to zero layers and still had excellent results).  Instead of focusing on being decoder only, focus on the ability for further parallelize training.  I also find the statements around \"next patch prediction\" to be a bit strange because you predict states not a patch ... it only becomes a patch when it is embedded by the decoder.\n\n## 2. Reproducibility\nI don't find the current manuscript to be sufficiently detailed to reproduce the model.  (Code release will fix a lot of this).  Some specific examples include:\n- More details are need around map tokenization - I would not know how to translate your text into code here.\n- In the prediction head it is not clear to me if there is a single RNN and a single MLP which is outputing all the mixture components or there are multiple copies (I assume the former ... but its not clear).\n- Embedding sizes?\n\n## 3. Manuscript quality\n- The figures and captions could use work.  In general the captions are far to brief and do not provide enough information to a reader skimming the paper.  The figures themselves, with the exception of figure 2 provide little explanatory value.  Figure 3 is a poor use of space as most of the figure is an identical cartoon repeated three times.  It would be great to replace some of the non-useful figures with visual representations of the model output (i.e. real examples).\n- References need to be cleaned up ... I only skimmed but a couple examples include [18] where Densetnt -> DenseTNT and [48] should cite the NeurIPS 2023 manuscript rather than arxiv.\n- I would steer away from introducing the concept of a scene level patch $P^{\\tau}$ as it is never actually constructed.\n\n## 4. Forecasting metrics/leaderboards\nAs noted in the paper, the simulation problem and the forecasting problem are quite similar.  The minADE numbers suggest that this method would also perform well on motion forecasting benchmarks.  I would love to see numbers there.  I do believe the Waymo Sim Benchmark is an important one, but it is also new and has not had as many competitive submissions as the Waymo or Argoverse forecasting benchmarks.  How does this model compare with QCNet on those benchmarks?\n\n## 5. Model introspection\nIt is not clear from this analysis, which elements of the architecture are actually important for its performance.  I would like to see more ablations.  What happens when we drop out one of the factorized attention modules?  Does the model really get any worse if we don't to  the a2a attention?  Does the model improve/degrade if we only do each attention once, or three times?  Is the RNN actually important?  What if we just predict with an MLP and interpolate?  Are the patches actually important or just going from 10Hz to 1Hz?\n\n## 6. Output introspection\nIt is typical for papers in this space to provide analysis beyond aggregate metrics.  There should be a visual analysis of the output.  Examples should be provided of instances where the model produced excellent output (specifically in comparison to other methods).  Examples should also be provided of instances where the model produced poor outputs.  Are there any connecting themes?  I.e. does the model struggle with certain road geometries or interactions?\n\n## 7. Alternative validation of value\nThe real claim of this paper is that they have produced a model which will produce more realistic simulations which in turn lead to improved development of autonomous vehicles.  Benchmarks are proxy evidence for this claim.  Can this be supported in some other way as well?  If we were to train a planner or motion forecasting method on output from this sim vs. another sim, would it perform better?\n\n## 8. Inference performance\nAutonomous vehicle companies need to run simulations at massive scales. (Tens of millions of scenarios).  To that end it would be nice to see the paper report the inference performance of their model (on reported hardware).\n\n## Conclusion\nAs noted in the beginning, not all of these areas need to be addressed in the limited rebuttal window. 1-3 should definitely be addressed along with some subset of 4-8 (7 is a long shot for sure).\n\nQUESTIONS:\nIt wasn't clear how many patches were available as history during training or inference?\n\nLIMITATIONS:\nNo concerns.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

comments: SUMMARY:
This work focuses on multi-agent simulation for autonomous driving. Instead of commonly used encoder-decoder structure, the authors propose a decoder-only autoregressive architecture for better data utilization, and achieve SOTA on Waymo SimAgents Benchmark.

STRENGTHS:
* The low data utilization problem using common encoder-decoder structure is identified, which requires a sequence be split into history and future. In proposed autoregressive architecture, each time step is treated as current, resulting in higher data utilization.
* Next-Patch Prediction Paradigm is introduced to force the model perform long-range interaction, preventing the shortcut learning in next-token prediction.

WEAKNESSES:
* It seems that no solid evidence is provided to prove the model performance "scales seamlessly with data and computation". I do not think Table 4 can support this assertion.
* Though parameter-efficient, it may not be superior in terms of computation or inference latency, providing more details about this would be better.

QUESTIONS:
See weaknesses.

LIMITATIONS:
See weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new decoder-only learning scheme for autonomous driving dynamics. Rather than using an encoder-decoder type architecture, the model uses spatial, temporal, and “social” attention between map-agent, time-agent, and agent-agent respectively in a time-autoregressive manner. Further, the work explores the usage of “next patch prediction”, where multiple timesteps are bundled, transformed, and unrolled using an RNN. This allows for easier modelling of long-range dependencies and more efficient processing.\n\nSTRENGTHS:\n**Originality**: To our knowledge, this is the first work in Traffic Simulation utilizing decoder-only autoregressive prediction, so there is a significant degree of novelty. \n\n**Quality** The different “arbitrary” selections, such as the number of neighbors to consider, the top-p to sample and patch size are explicitly ablated, which is nice to see. The modelling of the multiple domains present in autonomous driving seems to be sound.\n\n**Clarity**: The paper is generally structured well and readable.\n\n**Significance**: To our knowledge this is the first fully autoregressive model for autonomous driving simulation. The work is also a clear step up from the previous solutions in the WASAC 2023 challenge.\n\nWEAKNESSES:\n**Originality**:  The comparison against previous timeseries-transformers is rather lacking. I would like to see Autoformer and informer mentioned, and, possibly the most relevant here the Space-Time transformer (https://arxiv.org/pdf/2109.12218) which models local interactions within a timestep using graphs (similar to your k-nearest neighbor all2all attention).\n\n**Quality**: Something I would like to see though is a scaling comparison: Transformers tend to perform exponentially better as they scale larger, so seeing something like a 30M model to match the “Trajeglish” work would be nice. It is unclear whether, for instance, the patching is actually necessary to the degree described, or whether this is just an artifact of the model being rather small and shallow. Prior work on the expressivity of transformers, such as https://arxiv.org/abs/2205.11502 has shown that one can map K reasoning steps onto a K-deep model, so it might be that a patching of 10 items is only necessary because the model is too shallow to learn the combination itself. I'm aware that scaling a method might not be possible due to the expense of training a larger model, but without it one cannot be sure of the contributions of patching. I will not see it as a big negative if this is impossible because the performance is good enough to argue for the efficacy of the entire system as a whole, but I'm unsure about the importance of each individual component.\n\n**Clarity**: The notation of the patches is a little hard to parse. I would write \n$$S_i^{((\\tau-1)\\times (l+1))\\ :\\ (\\tau \\times l)}$$\n to make the grouping more clear (It took me way too long to mentally group the terms together…). Specifically, it’s hard to visually group the two sides of the “:” together. Figure 1 is slightly confusing: I understand what you want to show, but maybe put boxes to group the timesteps into tokens such that it becomes clear the transformer predicts the next token which is composed of multiple timesteps.\nSomething I did not quite understand was the tokenization of the map: You say that you sample every 5m and then assign a class to that sample. Does that mean you assign a class to the span from the last sample to the next one (i.e. you assign a class for [0m, 5m], [5m,10m],...) or just at that sample point (i.e. you assign for 0m, 5m, 10m). In the latter case I would assume you can easily jump over crucial information like center lines since those are less than 5m wide.\n\n**Significance**: The small size of BehaviorGPT is nice, but not too interesting for many real world problems: Generally, I would rather have a more accurate 30M model (that e.g. closes the gap to MVTE in acceleration), than a less accurate 3M variant. There are obviously advantages to a model being small, but the runtime difference between a 3M and a 30M parameter model is rarely big enough to make the former worth it (you can run a ResNet 50 with 25M parameters on a raspberry pi in real time…).\n\nQUESTIONS:\nWhat happens when scaling the model up? Does the need for patching vanish?\n\nWhat are the results after 20% of training? (particularly interesting because you criticize the data-inefficiency of prior art)\n\nWhy are mixing coefficients only predicted once per patch?\n\nWhy does training take so much time? (I expect a 3M timeseries model to train a lot more quickly)\n\nDoes training time change significantly with larger patches? (due to RNNs needing to be unrolled)\n\nWhy only autoregressive across time and not also across e.g. space?\n\nYou claim that \n> Developed the Next-Patch Prediction scheme to enhance models’ capability of long-range interaction reasoning, leading to more realistic multi-agent simulation over a long horizon;\n\nCan you support this by plotting e.g. accuracy vs. prediction horizon?\nAs is, it is unclear whether the higher performance is actually due to long-horizon performance or just an improvement in the short term and then matching the long-horizon performance.\n\nMore generally: How is the generalisation to longer sequence lengths? Generalisation to longer sequences is a known problem for transformers.\n\nLIMITATIONS:\nAll models considered are really small, making it hard to judge whether the individual components of the model are necessary or just an artifact of the expressivity of small models being low.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents BehaviorGPT, a model for trajectory prediction which is decoder only and respects temporal causality by employing a autoregressive sequence model.  They opt for a coarser time resolution of the sequence they call \"patching\" for reasons of efficiency and larger context, in analogy to word-level instead of byte-level representations for language sequence models. After predicting a coarser time segment (\"patch\") embedding, they then decode this into the finer time resolution sequence of states also respecting temporal causality with a RNN decoder within the \"patch\".\n\nThe model is decoder only, in the sense that they can process an infinite temporal stream where agent input representation is the same as agent output representation.  For the static map information, this is encoded once and cross-attended to in the rest of the decoder-only model.  The inner architecture is interleaved attention between map, agents, and time dimension.\n\nThey report exceptional performance results on the Waymo Open Sim Agents Challenge (WOSAC), which requires the models obey temporal causality to respect a sim agents use case. At the same time, their model is 92% smaller than competitive methods, which is substantially more parameter efficient.\n\nSTRENGTHS:\nThis is a significantly novel decoder-only model, with reasonable complexity, with very strong results.\n\nThe ideas like \"triple attention\", QCNet's positional encoding are present in other papers.  Predicting coarser state subsequences is a straightforward idea.  But piecing these ideas together into the full decoder only model results in a significant new model (in the specific area of multi-agent trajectory prediction).\n\nWith caveats, the paper is mostly very clearly explained, and will definitely have impact for other practitioners to replicate/extend.\n\nWEAKNESSES:\nThe biggest doubt I have about the impact of this paper is: is the impact of the paper \nA. the decoder only architecture or \nB. \"patching\", aka lower resolution modeling for efficiency reasons\n\nWithout patching, the model performs significantly worse than other top methods (compare minADE with patchsize=1 in Table 3 to minADEs in Table 1).  Thus,\n - does decoder-only really matter?\n -  if one added the patching idea to Trajeglish or MVTE or MTR++, would those also perform much better?\n\nMy hunch is patching is doing the heavy lifting here, but that is not clear in the story.\n\nIf this hunch is true, one story of this paper is \"other methods faithfully adhere to the overly-fine 10hz native processing, but BehaviorGPT found a way to bypass that inefficiency\"\n\nQUESTIONS:\nQ1. I have a big pet peeve with the terminology \"patching\", which is why I mostly keep it in quotes throughout this writeup.  To me (as a native english speaker), the notion of a \"patch\" only makes sense to describe a 2D or 3D region.  Calling a subsequence or segment of a 1D sequence a \"patch\" is very unintuitive.  In language models and in computer science when referring to 1D sequences in general, more common terms are subsequence, (sub)segment, or chunk.  In my opinion, \"chunk\" is nice, and gives me intuition that your process is analogous to this: https://chunkviz.up.railway.app/\n\nSo, not a question, but please consider this!\n\nQ2. How does this model do on the Waymo Open Motion Dataset's Motion Forecasting benchmark? Many papers in Table 1 also report results there, so I am curious.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a decoder only model for traffic simulation. The paper proposes to not predict the next state/token but to predict the next patch a small trajectory segment. The network can select several possible patch generators which are based on RNNs. The resulting architecture achieves strong results with a small transformer based architecture only using about 3M parameters.\n\nSTRENGTHS:\nProposed approach is interesting and works well. The method combines several ideas that have been shown to be efficient and combines them in a network for traffic simulation. The results of the model given its size are impressive and it would be interesting how it performed if the size was scaled up.\n\nWEAKNESSES:\n-The ablation study is not fully clear to me, I can not see a base model, and I also do not know what parameters are used in the final model. \n\n-Most of the ideas in the paper are known in other related fields. Not just predicting the action/next state is known in imitation learning to help with compounding errors. It is also getting used in LMMs [1]. The transformer architecture design is similar to HPTR. \n\n-A lot of details to reproduce the results are not clear (see questions)\n\n[1] Better & Faster Large Language Models via Multi-token Prediction, Gloeckle. et al.\n\nQUESTIONS:\n-Is it correct that you have as many RNNs as modes, with N_mode different parameters?\n\n-How do you select the 32 trajectories for the submission? \n\n-If you keep all modes with p>=0.9 do you not have issues with an excessive amount of trajectories? Or do you just keep the top-1?\n\n-Are there more tricks in the post processing, this is normally important to achieve good results for the sim agent challenge.\n\n-Is it correct that your input tokens are continuous?\n\n-Do you run the patch generation in a receding horizon fashion, where the model would run at 10Hz even though you generate 1s outputs. If not, would it be possible to do so?\n\n-Why is the training so slow? I believed that one of the advantages of a decoder only model compared to more policy learning based methods such as TraffiBot would be training speed. Since there is not autoregressive rollout during the training\n\n-Why do you use a RNN for the patch generation, could a simple MLP not do the same job and be faster?\n\nLIMITATIONS:\nLimitations could be discussed in more details.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents BehaviorGPT, a model for multi-agent traffic simulation. BehaviorGPT's architecture is based on a decoder-only transformer that autoregressively predicts patches of trajectories. The key insight is that predicting patches forces the model to learn longer-horizon reasoning. Then, to predict states from patches, BehaviorGPT uses an RNN that outputs a mixture of Gaussians/Laplacians for each agent's x, y, and heading. BehaviorGPT achieves state-of-the-art performance on the Waymo Open Sim Agents Challenge.\n\nSTRENGTHS:\n- The proposed architecture is simple yet incorporates well-reasoned inductive biases like relative positional encodings and factorized self-attention. Despite its simplicity, BehaviorGPT achieves state-of-the-art performance on the Waymo Open Sim Agents Challenge. Moreover, BehaviorGPT does this with 10x fewer parameters than the previous state-of-the-art, Trajeglish.\n- The paper ablates the efficacy of patch-based tokens, convincingly demonstrating that it is critical to having strong performance. To my knowledge, this design choice is novel within multi-agent traffic simulation and its adjacent fields. \n- Code will be made available after publication.\n- The paper is generally well-written and easy-to-follow.\n\nWEAKNESSES:\n- The paper could be strengthened with stronger ablation studies to highlight the design choices that make BehaviorGPT outperform prior work, like Trajeglish. A number of design choices in BehaviorGPT set it apart from Trajeglish, but it is not clear which ones actually contribute to its ability to outperform Trajeglish even with significantly less parameters; e.g., is it the patch-based tokens? is it the factorized attention with relative positional encodings? is it the specific mixture of Gaussians/Laplacians output distribution? Such analysis would give the reader more insight into what design choices are important for multi-agent traffic simulation.\n- The paper does not provide sufficient detail to reproduce the model; e.g., basic hyperparameters like the number of transformer layers. That said, this weakness is mitigated by the authors' intention to publish code upon acceptance.\n\nQUESTIONS:\n1. During inference, how do you generate 32 rollouts? Do you randomly sample with a different seed 32 times?\n2. What is the significance of not using ground truth state when unrolling the RNN? Do you have experiments to illustrate why it's necessary?\n3. Can the authors discuss limitations of their method in a more meaningful manner than \"performance can be better\"?\n\nLIMITATIONS:\nLimitations are mentioned (in the checklist section only) but limited to \"performance could be better\". The paper can be improved with a more thorough discussion of limitations; e.g., what are the reasons why BehaviorGPT still underperforms in certain metrics on Table 1?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}