BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction

Authors: zikang  zhou
haibo  hu
xinhong  chen
jianping  wang
nan  guan
kui  wu
yung-hui  li
yu-kai  huang
chun jason xue
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 10:35:29

Abstract: Simulating realistic behaviors of traffic agents is pivotal for efficiently validating the safety of autonomous driving systems. Existing data-driven simulators primarily use an encoder-decoder architecture to encode the historical trajectories before decoding the future. However, the heterogeneity between encoders and decoders complicates the models, and the manual separation of historical and future trajectories leads to low data utilization. Given these limitations, we propose BehaviorGPT, a homogeneous and fully autoregressive Transformer designed to simulate the sequential behavior of multiple agents. Crucially, our approach discards the traditional separation between "history" and "future" by modeling each time step as the "current" one for motion generation, leading to a simpler, more parameter- and data-efficient agent simulator. We further introduce the Next-Patch Prediction Paradigm (NP3) to mitigate the negative effects of autoregressive modeling, in which models are trained to reason at the patch level of trajectories and capture long-range spatial-temporal interactions. Despite having merely 3M model parameters, BehaviorGPT won first place in the 2024 Waymo Open Sim Agents Challenge with a realism score of 0.7473 and a minADE score of 1.4147, demonstrating its exceptional performance in traffic agent simulation.

Main paper: 
 introduction: 
 Autonomous driving has emerged as an unstoppable trend, with its rapid development increasing the demand for faithful evaluation of autonomy systems' reliability [32] . While on-road testing can measure driving performance by allowing autonomous vehicles (AVs) to interact with the physical world directly, the high testing cost and the scarcity of safety-critical scenarios in the real world have hindered large-scale and comprehensive evaluation. As an alternative, validating system safety via simulation has become increasingly attractive [14, 48, 53, 44] as it enables rapid testing in diverse driving scenarios simulated at a low cost. This work focuses on smart agent simulation, i.e., simulating the behavior of traffic participants such as vehicles, pedestrians, and cyclists in the digital world, which is critical for efficiently validating and iterating behavioral policies for AVs.A good simulator should be realistic, matching the real-world distribution of multi-agent behaviors to support the assessment of AVs' ability to coexist with humans safely. To this end, researchers started *Equal contribution. by designing naive simulators that mainly replay the driving logs collected in the real world [27, 29] . When testing new driving policies that deviate from the ones during data collection, agents in such simulators often exhibit unrealistic interactions with AVs, owing to the lack of reactivity to AVs' behavior changes. To simulate reactive agents, traditional approaches [14, 28] apply traffic rules to control agents heuristically [45, 26] , which may struggle to capture real-world complexity. Recently, the availability of large-scale driving data [6, 15, 50] , the emergence of powerful deep learning tools [19, 47, 20] , and the prosperity of related fields such as motion forecasting [16, 46, 59, 42, 58] , have spurred the development of data-driven agent simulation [44, 4, 24, 52, 56] towards more precise matching of behavioral distribution. With the establishment of standard benchmarks like the Waymo Open Sim Agents Challenge (WOSAC) [32] , which systematically evaluates the realism of agent simulation in terms of kinematics, map compliance, and multi-agent interaction, the research on data-driven simulation approaches has been further advanced [49, 35] .Existing learning-based agent simulators [44, 4, 24, 52, 56, 49, 35] mainly mirror the techniques from motion forecasting [16, 46, 59, 42, 58, 41, 18] and opt for an encoder-decoder architecture, presumably due to the similarity between the two fields. Typically, these models use an encoder to extract historical information and a decoder to predict agents' future states leveraging the encoded features. This paradigm requires manually splitting the multi-agent time series into a historical and a future segment, with the two segments being processed by separate encoders and decoders with heterogeneous architecture. For example, MVTA [49] constructs training samples by randomly selecting a "current" timestamp to divide sequences into historical and future components. Others [52, 35] use fixed-length agent trajectories as historical scene context, conditioned on which the multi-agent future is sampled from the decoder. Nonetheless, the benefit of employing heterogeneous modules to separately encode the history and decode the future, at the cost of significantly complicating the architecture, is unclear. Moreover, the manual separation of history and future leads to low utilization of data and computation: as every point in the sequence can be used for the separation, we believe a sample-efficient framework should be able to learn from every possible history-future pair from the sequence in parallel, which cannot be easily achieved by encoder-decoder solutions owing to their heterogeneous processing for the historical and the future time steps.Inspired by the success of decoder-only Large Language Models (LLMs) [37, 38, 5] , we introduce a fully autoregressive Transformer architecture, dubbed BehaviorGPT, into the field of smart agent simulation to overcome the limitations of previous works. By applying homogeneous Transformer blocks [47] to the complete trajectory snippets without differentiating history and future, we arrive at a simpler, more parameter-efficient, and more sample-efficient solution for agent simulation. Utilizing relative spacetime representations [58] , BehaviorGPT symmetrically models each agent state in the sequence as if it were the "current" one and tasks each state with modeling subsequent states' distribution during training. As a result, our framework maximizes the utilization of traffic data for autoregressive modeling, avoiding wasting any learning signals available in the time series.Autoregressive modeling with imitation learning, however, suffers from compounding errors [39] and causal confusion [11] . Concerning the behavior simulation task, we observed that blindly mimicking LLMs' training paradigm of next-token prediction [35] , regardless of the difference in tokens' semantics across tasks, will make these issues more prominent. For a next-token prediction model embedding tokens at 10 Hz, a low training loss can be achieved by simply copying and pasting the current token as the next one without performing any long-range interaction reasoning in space or time. To mitigate this issue, we introduce the Next-Patch Prediction Paradigm (NP3) that enables models to reason at the patch level of trajectories, as illustrated in Figure 1 . By enforcing models to autoregressively generate the next trajectory patch containing multiple time steps, which requires understanding the high-level semantics of agent behaviors and capturing long-range spatialtemporal interactions, we prevent models from leveraging trivial shortcuts during training. We equip BehaviorGPT with NP3 and attain superior performance on WOSAC [32] with merely 3M model parameters, demonstrating the effectiveness of our modeling framework for smart agent simulation.Our main contributions are three-fold. First, we propose a fully autoregressive architecture for smart agent simulation, which consists of homogeneous Transformer blocks that process multi-agent long sequences with high parameter and sample efficiency. Second, we develop the Next-Patch Prediction scheme to enhance long-range interaction reasoning, leading to more realistic multi-agent simulation over a long horizon. Third, we achieve remarkable performance on the Waymo Open Motion Dataset, winning first place in the 2024 Waymo Open Sim Agents Challenge. 2 Related Work 
multi-agent traffic simulation: 
 Multi-agent traffic simulation is essential for developing and testing autonomous driving systems. From early systems like ALVINN [36] to contemporary simulators such as CARLA [14] and SUMO [28] , these platforms have used heuristic driving policies to simulate agents' reactive behaviors [8, 7, 10] . However, they struggle to capture real-world complexity since policies based on simple heuristics are not robust enough to handle all sorts of scenarios. With the availability of large-scale data and deep learning approaches, generative models like VAEs [44] , GANs [24] , Diffusion [56] , and autoregressive models [49, 41, 35] have gained success in generating multi-agent motions, which greatly enhance the realism of simulations. Given the temporal dependency of agent trajectories, autoregressive models naturally fit the simulation task, while others require extra designs to capture such dependencies. Among the existing autoregressive models, two representatives are MotionLM [41] and Trajeglish [35] . Both of them adopt an encoder-decoder paradigm, designing complicated scene context encoders to extract historical information before autoregressive decoding. In contrast, our approach is fully autoregressive similar to decoder-only LLMs [37, 38, 5] , which eliminates the need for using heterogeneous modules to process the historical and future time steps and achieves higher efficiency in terms of data and parameters via simpler architectural design.The application of patches in Transformer models has demonstrated significant potential across various data modalities. For instance, BERT [12] employs subword tokenization [40] for natural language processing, while ViT [13] segments images into 2D patches for visual understanding. The patching design has also found applications in time-series forecasting [51, 57, 34] , aiming at retaining local semantics and reducing computational complexity [34] . Moreover, it has shown the effectiveness in self-supervised learning, which has significantly facilitated representation learning and contributed to excellent fine-tuning results on large datasets [2, 21, 3] . Since the task of agent simulation also involves time-series data, we expect the patching mechanism to help models effectively capture the spatial-temporal interactions in driving scenarios and enhance the realism of the generated motion. Our proposed Next-Patch Prediction Paradigm (NP3) utilizes patch-level tokens in autoregressive modeling and trains each token to generate the next patch that comprises multi-step motions, which shares some similarities to multi-token prediction in LLMs [17] .Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We have discussed the limitations of this work in the paper. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 This section presents the proposed BehaviorGPT for multi-agent behavior simulation, with Figure 2 illustrating the overall framework. To begin with, we provide the formulation of our map-conditioned, multi-agent autoregressive modeling. Then, we detail the architecture of BehaviorGPT, which adopts a Transformer decoder with a triple-attention mechanism to operate sequences at the patch level. Finally, we present the objective for model training. In multi-agent traffic simulation, we aim to simulate agents' future behavior in dynamic and complex environments. Specifically, we define a scenario as the composite of a vector map M and the states of N agent agents over T time steps. At each time step, the state of the i-th agent S i includes the agent's position, velocity, yaw angle, and bounding box size. The semantic type of agents (e.g., vehicles, pedestrians, and cyclists) are also available. Given the sequential nature of agent trajectories, we formulate the problem as sequential predictions over trajectory patches, where the prediction of each patch will affect the subsequent patches. We define an agent-level trajectory patch asEQUATIONwhere ℓ is the number of time steps covered by a patch, N patch = T /ℓ indicates the number of patches, and P τ i represents the τ -th trajectory patch of the i-th agent, with S ((τ -1)×ℓ+1):(τ ×ℓ) i denoting the states within the patch. On top of P τ i , we use P τ = S ((τ -1)×ℓ+1):(τ ×ℓ) 1:Nagent to denote the τ -th multi-agent patch, where P τ incorporates all agents' states at the τ -th patch. Next, we factorize the multi-agent joint distribution over patches along the time axis according to the chain rule:EQUATIONwhere Pr(S 1:T 1:Nagent | M ) is the joint distribution of all agents' states over all time steps conditioned on the map M . Further, we factorize over agents the conditional distribution of multi-agent patches based on the assumption that agents plan their motions independently within the horizon of a patch:EQUATIONConsidering the multimodality of agents' behavior within the horizon of a patch, we assume Pr(P τ i | P 1:(τ -1) , M ) to be a mixture model consisting of N mode modes:EQUATIONwhere π τ i,k is the probability of the k-th mode. Given the sequential nature of the states within a patch, we further conduct factorization over the states per mode using the chain rule:EQUATIONSuch an autoregressive formulation can be interpreted as planning the patch-level behavior of each agent independently (Eq. ( 3)), freezing agents' behavior mode per ℓ time steps (Eq. ( 4)), and autoregressively unrolling the next state under a specific behavior mode (Eq. ( 5)). Under this formulation, we can flexibly adjust the replan frequency during inference to control the reactivity of agents. For example, we can let agents execute α ∈ {1, . . . , ℓ} steps of the planned motions and choose a new behavior mode after α steps to react to the change in environments.In our autoregressive formulation, we treat each trajectory patch as the "current" patch that is responsible for estimating the next-patch distribution during training, contrasting many existing approaches that designate one current time step per sequence [52, 49, 23] . As a result, it is inefficient to employ the well-established agent-or polyline-centric representation from the field of motion forecasting [46, 59, 33, 42, 25, 54, 43] , given that these representations are computed under the reference frames determined by one current time step per sequence. For this reason, we adopt the relative spacetime representation introduced in QCNet [58] to model the patches symmetrically in space and time, achieving simultaneous multi-agent prediction when implementing Eq. ( 3) and allowing parallel next-patch prediction for the modeling of Eq. ( 2). Under this representation, the features of each map element and agent state are derived from coordinate-independent attributes, e.g., the semantic category of a map element and the speed of an agent state. On top of this, we effectively maintain the spatial-temporal relationships between input elements via relative positional embeddings. Specifically, we use i and j to index two different input elements and compute the relative spatial-temporal embedding byEQUATIONwhere R j→i is the relational embedding from j to i, ∥d j→i ∥ is the Euclidean distance between them, ∠(n i , d j→i ) is the angle between n i (i.e., the orientation of i) and d j→i (i.e., the displacement vector from j to i), ∆θ j→i /∆z j→i is the relative yaw/height from j to i, and ∆τ j→i is the time difference.Before performing spatial-temporal relational reasoning among the input elements of a traffic scenario, we must convert the raw information into high-dimensional embeddings. We first embed map information by sampling points along map polylines every 5 meters and tokenizing the semantic category of each 5-meter segment (e.g., lane centerlines, road edges, and crosswalks) via learnable embeddings. The i-th polyline segment's embedding is denoted by Mi , which does not include any information about coordinates. On the other hand, we process agent states using attention-based patching to obtain patch-level embeddings of trajectories. For the i-th agent's state S t i at time step t, we employ an MLP to transform the speed, the velocity vector's angle relative to the bounding box's heading, the size of the bounding box, and the semantic type of the agent, into a feature vector Ŝt i . To further acquire patch embeddings, we collect the feature vectors of ℓ consecutive agent states and apply the attention mechanism with relative positional embeddings to them:EQUATIONwhere P τ i is the patch embedding of the i-th agent at the τ -th patch, MHSA(•) denotes the multi-head self-attention [47] , [:, :] denotes concatenation, and R t→(τ ×ℓ) i indicates the positional embedding of S t i relative to S τ ×ℓ i computed according to Eq. ( 6). Such an operation can be viewed as aggregating the features of S ((τ -1)×ℓ+1):(τ ×ℓ-1) i into Ŝτ×ℓ i and using the embeddings fused with high-level semantics as the agent tokens in the subsequent modules.After obtaining map tokens and the patch embeddings of agents, we employ a Transformer decoder [47] with the triple-attention mechanism to model the spatial-temporal interactions among scene elements. As illustrated in Figure 3 , the triple-attention mechanism considers three distinct sources of relations in the scene, including the temporal dependencies over the trajectory patches per agent, the regulations of the map elements on the agents, and the social interactions among agents.Temporal Self-Attention. This module captures the relationships among the trajectory patches of each individual agent. Similar to decoder-only LLMs [37, 38, 5] , it leverages the multi-head self-attention (MHSA) with a causal mask to enforce each trajectory patch to only attend to the preceding patches of the same agent, accommodating our autoregressive formulation. The temporal MHSA is equipped with relative positional embeddings:EQUATIONwhere F τ a2t,i and P τ i are the temporal-aware feature vector and the patch embedding of the i-th agent at the τ -th patch, respectively, and R t×ℓ→τ ×ℓ i embeds the relative position from S t×ℓ i to S τ ×ℓ i , which represents the spatial-temporal relationship between the patches P t i and P τ i . Agent-Map Cross-Attention. Unlike natural language which only has a sequence dimension, we must also conduct spatial reasoning to consider the environmental influence on agents' behavior. To facilitate the modeling of agent-map interactions, we apply the multi-head cross-attention (MHCA) to each trajectory patch in the scenario. Considering that a scenario may comprise an explosive number of map polyline segments and that an agent would not be influenced by map elements far away, we filter the key/value map elements in MHCA using the k-nearest neighbors algorithm [42, 54] . The agent-map cross-attention is formulated asEQUATIONwhere F τ a2m,i is the map-aware feature vector for the i-th agent at the τ -th patch, Mj is the embedding of the j-th map polyline segment, R τ ×ℓ j→i is the relative positional embedding between the agent state S τ ×ℓ i and the j-th map polyline segment, and N (i, τ ) denotes the k-nearest map neighbors of S τ ×ℓ i .Agent-Agent Self-Attention. We further capture the social interactions among agents by applying the MHSA to the space dimension of the trajectory patches. In this module, we also utilize the locality assumption induced by the k-nearest neighbor selection for better computational and memory efficiency. Specifically, the map-aware features of trajectory patches are refined byEQUATIONwhere F τ a2a,i is the feature vector enriched with spatial interaction information among agents for the i-th agent at the τ -th patch, R τ ×ℓ j→i contains the relative information between the i-th and the j-th agent at the τ -th patch, and N (i, τ ) filters the k-nearest agent neighbors of S τ ×ℓ i .Overall Decoder Architecture. Each of the attention layers above is enhanced by commonly used components in Transformers [47] , including feed-forward networks, residual connections [19] , and Layer Normalization [1] in a pre-norm fashion. To enable higher-order relational reasoning, we stack multiple triple-attention blocks by interleaving the three Transformer layers. We denote the ultimate feature of the i-th agent at the τ -th patch as F τ i , which will serve as the input of the prediction head for next-patch prediction modeling.Given the interaction-aware patch features output by the Transformer decoder, we develop a nextpatch prediction head to model the marginal multimodal distribution of agent trajectories, which estimates the distributional parameters of each patch's successor.The following describes the process of next-patch prediction regarding the τ -th patch of the i-th agent. Based on the attention output F τ i , we intend to estimate the parameters of the next patch's mixture model pre-defined with N mode modes. First, we use an MLP to transform F τ i into π τ +1 i ∈ R Nmode , the mixing coefficient of the modes. In each mode, the conditional distribution of the next agent state, as depicted in Eq. ( 5), is considered a multivariate marginal distribution that parameterizes the position and velocity components as Laplace distributions and the yaw angle as a von Mises distribution. Based on this formulation, we employ a GRU-based autoregressive RNN [9] to unroll the states within the next patch step by step, with each step being conditioned on the previously predicted states. Specifically, The hidden state h τ,t i,k of the RNN is initialized with F τ i at t = 1 for ∀k ∈ {1, . . . , N modes }. At each step of the rollout, we use an MLP to estimate the location and scale parameters of the next agent state's position and velocity based on the hidden state. On the other hand, the MLP also estimates the location and concentration parameters of the next yaw angle. The location parameters of the newly predicted state, including the 3D positions, the 2D velocities, and the yaw angle, are used to update the RNN's hidden state directly without relying on the predicted scale/concentration parameters for sampling. The whole process is summarized as follows:EQUATIONwhere{µ τ ×ℓ+t i,k ∈ R 6 } t∈{1,...,ℓ} , {b τ ×ℓ+t i,k∈ R 5 } t∈{1,...,ℓ} , and {κ τ ×ℓ+t i,k∈ R} t∈{1,...,ℓ} are the location, scale, and concentration parameters in the k-th mode, and Z k is the k-th learnable mode embedding.To train BehaviorGPT, we apply the negative log-likelihood loss L NLL to the factorized distribution of Pr(S 1:T 1:Nagent | M ) as formulated previously:L NLL = Npatch τ =1 Nagent i=1 -log Nmode k=1 π τ i,k τ ×ℓ t=(τ -1)×ℓ+1Pr S t i,k | S ((τ -1)×ℓ+1):(t-1) i,k, P 1:(τ -1) , M .(12) Note that each ground-truth trajectory patch is transformed into the viewpoint of its previous patch.During training, we utilize teacher forcing to parallelize the modeling of next-patch prediction and ease the learning difficulty, but we do not use the ground-truth agent states when updating the RNN's hidden states, intending to train the model to recover from its mistakes made in next-state prediction.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 
experiments: 
 This section first introduces the dataset and the evaluation metrics used in our experiments, followed by presenting the implementation details and the rollout results obtained by BehaviorGPT on the Waymo Open Sim Agents Benchmark [32] . Finally, we conduct ablation studies to further compare and analyze the performance of BehaviorGPT under various settings.Our experiments are conducted on the Waymo Open Motion Dataset (WOMD) [15] . The dataset comprises 486,995/44,097/44,920 training/validation/testing scenarios. Each scenario includes 91step observations sampled at 10 Hz, totaling 9.1 seconds. Given 11-step initial states of the scenarios, we simulate up to 128 agents and generate 80 simulation steps per agent at 0.1-second intervals in an autoregressive and reactive manner. Each agent requires 32 simulations comprising x/y/z centroid coordinates and a heading value. The results on the test set are obtained by utilizing the full training set, while the performance on the validation set is based on 20% of training data unless specified.We use various metrics for evaluation. The minADE measures the minimum average displacement error over multiple simulated trajectories, assessing trajectory accuracy. REALISM is the meta-metric The optimal patch size we experimented with is 10, corresponding to 1 second. All hidden sizes are set to 128. Each attention layer has 8 attention heads with 16 dimensions per head. To save training resources, we limit the maximum number of agents per scenario to 128 and restrict the maximum number of neighbors in kNN attention layers to 32. The prediction head produces 16 modes per agent and time step. We train the models for 30 epochs on 8 NVIDIA RTX 4090 GPUs with a batch size of 24, utilizing the AdamW optimizer [31] . The weight decay rate and dropout rate are both set to 0.1. The learning rate is initially set to 5 × 10 -4 and decayed to 0 following a cosine annealing schedule [30] . Our results in the 2024 WOSAC are obtained using a single model with 2 decoding blocks and a total of 3M parameters. To produce 32 replicas of rollouts, we randomly sample behavior modes from agents' next-patch distributions until completing the 8-second multi-agent trajectories, and we repeat this process with different random seeds. The final results on the leaderboard are based on a replan rate of 2 Hz, while the ablation studies are based on a 1-Hz replan rate unless specified.We report the test set results in Table 1 . Notably, BehaviorGPT achieves the lowest minADE and the best REALISM, underscoring the model's ability to match the real-world distribution. Its excellent performance on COLLISION and OFFROAD also indicates that the model has successfully captured the agent-agent and agent-map interactions in driving scenarios. Besides the benchmarking results, we also compare the number of model parameters in BehaviorGPT and other baselines. Table 1 demonstrates that BehaviorGPT, with only 3M parameters, achieves more realistic simulation than significantly larger models like MVTE [49] and GUMP [22] , which demonstrates the parameter efficiency of our approach. Without employing tricks like data augmentation, model ensemble, or post-processing steps, BehaviorGPT won first place in the 2024 Waymo Open Sim Agents Challenge. Figure 4 visualizes some qualitative results of the rollouts produced by our model. In this scenario, BehaviorGPT can generate multiple plausible futures given the same initial states of agents, which demonstrates its capability of simulating diverse yet realistic agent behavior. However, we also note that autoregressive models still suffer from accumulated errors in some cases. As shown in Figure 5 , the vehicle in orange gradually goes out of the road as time goes by, which indicates the inherent limitations of autoregressive generation.We conduct some ablation studies to gain a more in-depth understanding of our approach.Impact of patch size. Table 2 presents the results of BehaviorGPT with varying patch sizes. According to the results, it is evident that using a patch size of 5, i.e., training and predicting with 2-Hz tokens, significantly outperforms the baseline without patching. Moreover, increasing the patch size to 10 further enhances the overall performance. These results demonstrate the benefits of incorporating the NP3 into agent simulation. However, changing the patch size also leads to a variation in replan frequency, which also has an influence on simulation. Next, we investigate the impact of replan frequency on the test set using the model submitted to the 2024 WOSAC. Impact of replan frequency. During inference, we vary the replan frequency of the model with a patch size of 10 by discarding a portion of the predicted states at each simulation step. As shown in Table 3 , increasing the replan frequency from 1 Hz to 2 Hz can even improve the overall performance, which may benefit from the enhanced reactivity. This phenomenon demonstrates that the performance gain is not merely due to the lower replan frequency, as the model with a patch size of 10 beats that with a patch size of 5 even harder if using the same replan frequency of 2 Hz. However, using an overly high replan frequency harms the performance, as indicated by the third row of Table 3 . Overall, we conclude that using a larger patch indeed helps long-term reasoning, but a moderate replan frequency is important for temporal stability, which may be neglected by prior works. Impact of multi-agent interaction modeling. We remove all agent-agent self-attention layers in the first row of Table 4 to show that modeling the interactions among agents can boost minADE and REALISM. In particular, the realism in terms of collision is improved by 34.66% when employing agent-agent self-attention. Scaling with model size. We investigate the effects of scaling up the model size based on some preliminary experiments with 20% of training data. In Table 6 , we vary the number of decoding blocks while fixing the hidden size as 128. On the other hand, we fix the number of decoding blocks as 2 and vary the hidden size, as depicted in Table 7 . Based on the experimental results, we can summarize that enlarging the model consistently leads to more realistic simulation, which showcases the potential of BehaviorGPT for scaling up.Extrapolation ability. We tried training a model on 5-second sequences and generating 9.1-second sequences during inference. The results in Table 8 show that this model achieves similar performance compared with the baseline trained with 9.1-second sequences, demonstrating our approach's extrapolation ability to generate longer sequences. 
conclusion: 
 This work introduced BehaviorGPT, a fully autoregressive architecture designed to enhance smart agent simulation for autonomous driving. By applying homogeneous Transformer blocks to entire trajectory snippets and utilizing relative spacetime representations, BehaviorGPT simplifies the modeling process and maximizes data utilization. To enable high-level understanding and long-range interaction reasoning in space and time, we developed the Next-Patch Prediction Paradigm, which tasks models with generating trajectory patches instead of single-step states. Experimental results on the Waymo Open Sim Agents Challenge demonstrate that BehaviorGPT achieves outstanding performance with merely 3M model parameters, highlighting its potential to further improve the realism of agent simulation with more data and computation.Limitations. First, BehaviorGPT is currently inferior in kinematics-related performance, which can be enhanced by incorporating a kinematic model, e.g., the bicycle model. Second, the current version of BehaviorGPT does not support controlling agent behavior with specific prompts such as language and goal points. However, achieving controllable generation should be trivial given a powerful base model. Finally, we have not verified whether BehaviorGPT will facilitate the development of motion planning, which we leave as future work.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The data used in this work is the Waymo Open Motion Dataset, which is publicly available. Our code will also be made public after the paper is published.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: As the data used in this work is the Waymo Open Motion Dataset, the training and testing details are available on the website of Waymo Open Dataset Challenge. We have also clarified other details in Section 3, Section 4, and the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: While not conducting significance tests over results, our experiments are conducted on the Waymo Open Motion Dataset, which has a large data scale. Thus, the experimental results are stable across multiple trials, and the reported results are reliable and authentic.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The proposed BehaviorGPT is particularly designed for smart agent simulation, which can benefit the verification of autonomous driving systems and is expected to promote the development of autonomous driving. While autonomous driving itself may be a doublesided sword, its negative side is out of the scope of this work and the developed BehaviorGPT will not directly bring negative societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The paper has cited the original paper that produced the Waymo Open Motion Dataset, and the information about the dataset is available online.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators.  
 
 Appendix: 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We have clearly outlined the scope and the targeted task of this work, and presented a concise introduction of our approach. The important contributions are described in detail, and the reported results obtained by our model are tested in the Waymo Open Sim Agents Challenge. This open challenge allows for fair comparisons with other baselines. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: There are no violations against the Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper proposes a decoder-only multi-agent traffic simulation called "BehaviorGPT", which is based on a transformer model (with only 3M parameters) that predicts patches of trajectories (instead of predicting next states/tokens), which are on a coarser time granularity. A recurrent neural network is used as a patch generator that then outputs a mixture of Gaussians/Laplacians for each agent's x, y, and heading, that generates a fine-grained sequence of states. 

BehaviorGPT achieves state-of-the-art performance on the Waymo Open Sim Agents Challenge while being considerably smaller than competitive methods. 

Decision: All reviewers voted for clear acceptance. I therefore recommend to accept the paper and encourage the authors to use the feedback provided to improve the paper for its final version.

Reviews: review: paper_summary: paper_summary: This paper presents BehaviorGPT, a model for multi-agent traffic simulation. BehaviorGPT's architecture is based on a decoder-only transformer that autoregressively predicts patches of trajectories. The key insight is that predicting patches forces the model to learn longer-horizon reasoning. Then, to predict states from patches, BehaviorGPT uses an RNN that outputs a mixture of Gaussians/Laplacians for each agent's x, y, and heading. BehaviorGPT achieves state-of-the-art performance on the Waymo Open Sim Agents Challenge.
strength_weakness: strength_weakness: - The paper could be strengthened with stronger ablation studies to highlight the design choices that make BehaviorGPT outperform prior work, like Trajeglish. A number of design choices in BehaviorGPT set it apart from Trajeglish, but it is not clear which ones actually contribute to its ability to outperform Trajeglish even with significantly less parameters; e.g., is it the patch-based tokens? is it the factorized attention with relative positional encodings? is it the specific mixture of Gaussians/Laplacians output distribution? Such analysis would give the reader more insight into what design choices are important for multi-agent traffic simulation.
- The paper does not provide sufficient detail to reproduce the model; e.g., basic hyperparameters like the number of transformer layers. That said, this weakness is mitigated by the authors' intention to publish code upon acceptance.
questions: questions: 1. During inference, how do you generate 32 rollouts? Do you randomly sample with a different seed 32 times?
2. What is the significance of not using ground truth state when unrolling the RNN? Do you have experiments to illustrate why it's necessary?
3. Can the authors discuss limitations of their method in a more meaningful manner than "performance can be better"?
limitations: limitations: Limitations are mentioned (in the checklist section only) but limited to "performance could be better". The paper can be improved with a more thorough discussion of limitations; e.g., what are the reasons why BehaviorGPT still underperforms in certain metrics on Table 1?
paper_summary: paper_summary: The paper proposes a decoder only model for traffic simulation. The paper proposes to not predict the next state/token but to predict the next patch a small trajectory segment. The network can select several possible patch generators which are based on RNNs. The resulting architecture achieves strong results with a small transformer based architecture only using about 3M parameters.
strength_weakness: strength_weakness: -The ablation study is not fully clear to me, I can not see a base model, and I also do not know what parameters are used in the final model. 

-Most of the ideas in the paper are known in other related fields. Not just predicting the action/next state is known in imitation learning to help with compounding errors. It is also getting used in LMMs [1]. The transformer architecture design is similar to HPTR. 

-A lot of details to reproduce the results are not clear (see questions)

[1] Better & Faster Large Language Models via Multi-token Prediction, Gloeckle. et al.
questions: questions: -Is it correct that you have as many RNNs as modes, with N_mode different parameters?

-How do you select the 32 trajectories for the submission? 

-If you keep all modes with p>=0.9 do you not have issues with an excessive amount of trajectories? Or do you just keep the top-1?

-Are there more tricks in the post processing, this is normally important to achieve good results for the sim agent challenge.

-Is it correct that your input tokens are continuous?

-Do you run the patch generation in a receding horizon fashion, where the model would run at 10Hz even though you generate 1s outputs. If not, would it be possible to do so?

-Why is the training so slow? I believed that one of the advantages of a decoder only model compared to more policy learning based methods such as TraffiBot would be training speed. Since there is not autoregressive rollout during the training

-Why do you use a RNN for the patch generation, could a simple MLP not do the same job and be faster?
limitations: limitations: Limitations could be discussed in more details.
paper_summary: paper_summary: This work presents BehaviorGPT, a model for trajectory prediction which is decoder only and respects temporal causality by employing a autoregressive sequence model.  They opt for a coarser time resolution of the sequence they call "patching" for reasons of efficiency and larger context, in analogy to word-level instead of byte-level representations for language sequence models. After predicting a coarser time segment ("patch") embedding, they then decode this into the finer time resolution sequence of states also respecting temporal causality with a RNN decoder within the "patch".

The model is decoder only, in the sense that they can process an infinite temporal stream where agent input representation is the same as agent output representation.  For the static map information, this is encoded once and cross-attended to in the rest of the decoder-only model.  The inner architecture is interleaved attention between map, agents, and time dimension.

They report exceptional performance results on the Waymo Open Sim Agents Challenge (WOSAC), which requires the models obey temporal causality to respect a sim agents use case. At the same time, their model is 92% smaller than competitive methods, which is substantially more parameter efficient.
strength_weakness: strength_weakness: The biggest doubt I have about the impact of this paper is: is the impact of the paper 
A. the decoder only architecture or 
B. "patching", aka lower resolution modeling for efficiency reasons

Without patching, the model performs significantly worse than other top methods (compare minADE with patchsize=1 in Table 3 to minADEs in Table 1).  Thus,
 - does decoder-only really matter?
 -  if one added the patching idea to Trajeglish or MVTE or MTR++, would those also perform much better?

My hunch is patching is doing the heavy lifting here, but that is not clear in the story.

If this hunch is true, one story of this paper is "other methods faithfully adhere to the overly-fine 10hz native processing, but BehaviorGPT found a way to bypass that inefficiency"
questions: questions: Q1. I have a big pet peeve with the terminology "patching", which is why I mostly keep it in quotes throughout this writeup.  To me (as a native english speaker), the notion of a "patch" only makes sense to describe a 2D or 3D region.  Calling a subsequence or segment of a 1D sequence a "patch" is very unintuitive.  In language models and in computer science when referring to 1D sequences in general, more common terms are subsequence, (sub)segment, or chunk.  In my opinion, "chunk" is nice, and gives me intuition that your process is analogous to this: https://chunkviz.up.railway.app/

So, not a question, but please consider this!

Q2. How does this model do on the Waymo Open Motion Dataset's Motion Forecasting benchmark? Many papers in Table 1 also report results there, so I am curious.
limitations: limitations: None
paper_summary: paper_summary: This paper proposes a new decoder-only learning scheme for autonomous driving dynamics. Rather than using an encoder-decoder type architecture, the model uses spatial, temporal, and “social” attention between map-agent, time-agent, and agent-agent respectively in a time-autoregressive manner. Further, the work explores the usage of “next patch prediction”, where multiple timesteps are bundled, transformed, and unrolled using an RNN. This allows for easier modelling of long-range dependencies and more efficient processing.
strength_weakness: strength_weakness: **Originality**:  The comparison against previous timeseries-transformers is rather lacking. I would like to see Autoformer and informer mentioned, and, possibly the most relevant here the Space-Time transformer (https://arxiv.org/pdf/2109.12218) which models local interactions within a timestep using graphs (similar to your k-nearest neighbor all2all attention).

**Quality**: Something I would like to see though is a scaling comparison: Transformers tend to perform exponentially better as they scale larger, so seeing something like a 30M model to match the “Trajeglish” work would be nice. It is unclear whether, for instance, the patching is actually necessary to the degree described, or whether this is just an artifact of the model being rather small and shallow. Prior work on the expressivity of transformers, such as https://arxiv.org/abs/2205.11502 has shown that one can map K reasoning steps onto a K-deep model, so it might be that a patching of 10 items is only necessary because the model is too shallow to learn the combination itself. I'm aware that scaling a method might not be possible due to the expense of training a larger model, but without it one cannot be sure of the contributions of patching. I will not see it as a big negative if this is impossible because the performance is good enough to argue for the efficacy of the entire system as a whole, but I'm unsure about the importance of each individual component.

**Clarity**: The notation of the patches is a little hard to parse. I would write 
$$S_i^{((\tau-1)\times (l+1))\ :\ (\tau \times l)}$$
 to make the grouping more clear (It took me way too long to mentally group the terms together…). Specifically, it’s hard to visually group the two sides of the “:” together. Figure 1 is slightly confusing: I understand what you want to show, but maybe put boxes to group the timesteps into tokens such that it becomes clear the transformer predicts the next token which is composed of multiple timesteps.
Something I did not quite understand was the tokenization of the map: You say that you sample every 5m and then assign a class to that sample. Does that mean you assign a class to the span from the last sample to the next one (i.e. you assign a class for [0m, 5m], [5m,10m],...) or just at that sample point (i.e. you assign for 0m, 5m, 10m). In the latter case I would assume you can easily jump over crucial information like center lines since those are less than 5m wide.

**Significance**: The small size of BehaviorGPT is nice, but not too interesting for many real world problems: Generally, I would rather have a more accurate 30M model (that e.g. closes the gap to MVTE in acceleration), than a less accurate 3M variant. There are obviously advantages to a model being small, but the runtime difference between a 3M and a 30M parameter model is rarely big enough to make the former worth it (you can run a ResNet 50 with 25M parameters on a raspberry pi in real time…).
questions: questions: What happens when scaling the model up? Does the need for patching vanish?

What are the results after 20% of training? (particularly interesting because you criticize the data-inefficiency of prior art)

Why are mixing coefficients only predicted once per patch?

Why does training take so much time? (I expect a 3M timeseries model to train a lot more quickly)

Does training time change significantly with larger patches? (due to RNNs needing to be unrolled)

Why only autoregressive across time and not also across e.g. space?

You claim that 
> Developed the Next-Patch Prediction scheme to enhance models’ capability of long-range interaction reasoning, leading to more realistic multi-agent simulation over a long horizon;

Can you support this by plotting e.g. accuracy vs. prediction horizon?
As is, it is unclear whether the higher performance is actually due to long-horizon performance or just an improvement in the short term and then matching the long-horizon performance.

More generally: How is the generalisation to longer sequence lengths? Generalisation to longer sequences is a known problem for transformers.
limitations: limitations: All models considered are really small, making it hard to judge whether the individual components of the model are necessary or just an artifact of the expressivity of small models being low.
paper_summary: paper_summary: This work focuses on multi-agent simulation for autonomous driving. Instead of commonly used encoder-decoder structure, the authors propose a decoder-only autoregressive architecture for better data utilization, and achieve SOTA on Waymo SimAgents Benchmark.
strength_weakness: strength_weakness: * It seems that no solid evidence is provided to prove the model performance "scales seamlessly with data and computation". I do not think Table 4 can support this assertion.
* Though parameter-efficient, it may not be superior in terms of computation or inference latency, providing more details about this would be better.
questions: questions: See weaknesses.
limitations: limitations: See weaknesses.
paper_summary: paper_summary: BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction introduces a model architecture for multi-agent simulation of dynamic traffic actors.  Improved simulation capabilities are essential to the safe and rapid development of autonomous vehicles.  BehaviorGPT structures multi-agent simulation a next-patch-predition problem.  Given a history of trajectory patches (embeddings of trajectory subsequences of fixed length), the model predicts the next patch's set of states for each actor independently.  These predictions can then be treated as a patch and the process can be repeated in an autoregressive rollout. Through use of relative encodings (in both time and space) and a "decoder-only" architecture, BehaviorGPT can be trained in parallel any continuous subset of patches.  BehaviorGPT is state-of-the-art on the Waymo Sim Agents Benchmark with impressive scores for both trajectory accuracy (minADE) as well as realism.  Furthermore, BehaviorGPT accomplishes these SOTA results with only 3M parameters (an order of magnitude lower than other competing approaches).
strength_weakness: strength_weakness: At a very high level, I think this paper presents a model which produces impressive results on an important benchmark (which represents an important problem).  However, I think that the current manuscript falls short in multiple dimensions.  I do not expect or think that the authors must make substantive changes with respect to all of the points below.  However if the authors were able to expand or address a couple of the weaknesses described below it would significantly improve the strength of the submission.

## 1. Scope down claims slightly
The results are super impressive, but I would scope down some of the adjacent claims.  For example, there is the assertion that there is a big sample efficiency gain over MTVA and Trajeglish but there is no evidence to support that.  I also think that the claim of being the first "decoder only" model is a bit overstated; this is largely semantic.  Other authors writing the same paper may have called your decoder the encoder and the prediction head the decoder.  Additionally the paper Query-Centric Trajectory Prediction had an awesome ablation in their paper where the reduced the encoder to zero layers and still had excellent results).  Instead of focusing on being decoder only, focus on the ability for further parallelize training.  I also find the statements around "next patch prediction" to be a bit strange because you predict states not a patch ... it only becomes a patch when it is embedded by the decoder.

## 2. Reproducibility
I don't find the current manuscript to be sufficiently detailed to reproduce the model.  (Code release will fix a lot of this).  Some specific examples include:
- More details are need around map tokenization - I would not know how to translate your text into code here.
- In the prediction head it is not clear to me if there is a single RNN and a single MLP which is outputing all the mixture components or there are multiple copies (I assume the former ... but its not clear).
- Embedding sizes?

## 3. Manuscript quality
- The figures and captions could use work.  In general the captions are far to brief and do not provide enough information to a reader skimming the paper.  The figures themselves, with the exception of figure 2 provide little explanatory value.  Figure 3 is a poor use of space as most of the figure is an identical cartoon repeated three times.  It would be great to replace some of the non-useful figures with visual representations of the model output (i.e. real examples).
- References need to be cleaned up ... I only skimmed but a couple examples include [18] where Densetnt -> DenseTNT and [48] should cite the NeurIPS 2023 manuscript rather than arxiv.
- I would steer away from introducing the concept of a scene level patch $P^{\tau}$ as it is never actually constructed.

## 4. Forecasting metrics/leaderboards
As noted in the paper, the simulation problem and the forecasting problem are quite similar.  The minADE numbers suggest that this method would also perform well on motion forecasting benchmarks.  I would love to see numbers there.  I do believe the Waymo Sim Benchmark is an important one, but it is also new and has not had as many competitive submissions as the Waymo or Argoverse forecasting benchmarks.  How does this model compare with QCNet on those benchmarks?

## 5. Model introspection
It is not clear from this analysis, which elements of the architecture are actually important for its performance.  I would like to see more ablations.  What happens when we drop out one of the factorized attention modules?  Does the model really get any worse if we don't to  the a2a attention?  Does the model improve/degrade if we only do each attention once, or three times?  Is the RNN actually important?  What if we just predict with an MLP and interpolate?  Are the patches actually important or just going from 10Hz to 1Hz?

## 6. Output introspection
It is typical for papers in this space to provide analysis beyond aggregate metrics.  There should be a visual analysis of the output.  Examples should be provided of instances where the model produced excellent output (specifically in comparison to other methods).  Examples should also be provided of instances where the model produced poor outputs.  Are there any connecting themes?  I.e. does the model struggle with certain road geometries or interactions?

## 7. Alternative validation of value
The real claim of this paper is that they have produced a model which will produce more realistic simulations which in turn lead to improved development of autonomous vehicles.  Benchmarks are proxy evidence for this claim.  Can this be supported in some other way as well?  If we were to train a planner or motion forecasting method on output from this sim vs. another sim, would it perform better?

## 8. Inference performance
Autonomous vehicle companies need to run simulations at massive scales. (Tens of millions of scenarios).  To that end it would be nice to see the paper report the inference performance of their model (on reported hardware).

## Conclusion
As noted in the beginning, not all of these areas need to be addressed in the limited rebuttal window. 1-3 should definitely be addressed along with some subset of 4-8 (7 is a long shot for sure).
questions: questions: It wasn't clear how many patches were available as history during training or inference?
limitations: limitations: No concerns.

Comments: title: Thank you
Thank you
Rebuttal followup
Thank you
Thanks for your positive feedback!
Thanks for your positive feedback!
Rebuttal followup
comment: Thank you again for the constructive suggestions!
Thanks a lot for addressing my comments, my main concerns are resolved and given that the material of the rebuttal is added I would like to increase my rating to a 7.
Thank you for the feedback. We will add more discussions about this work's limitations as you suggested.
Thank you for your response. The authors have addressed most of my questions about the paper, particularly as it relates to my question regarding why BehaviorGPT outperforms the state-of-the-art. My only remaining suggestion is that the authors discuss the limitations of this paper in more depth in the camera ready; the current limitations remain surface-level discussions that do not contribute to the reader's understanding of BehaviorGPT's efficacy.

Considering this, I would like to maintain my rating and recommend this paper's acceptance.
Thank you again for the valuable suggestions!
Thank you again to the authors for the followup.  Based on rebuttals to all reviewers and commitments for the final manuscript I would like to change my score to a 7 (accept).
Thank you again for the feedback!
Thank you very much for the positive feedback! We are committed to improving the quality of our manuscript based on the discussion during the rebuttal, though we are unable to upload the revised paper at this stage according to the policy of NeurIPS 2024. 

Regarding your follow-up questions:

**1. Sample efficiency.**

Currently, few or no methods on the benchmark have been open-source, which brings difficulties in replicating different models and evaluating their sample efficiency directly. Perhaps a piece of indirect evidence supporting our claim is Figure 11 in the Trajeglish paper, which indicates that the Trajeglish model is data-hungry. More direct evidence is that the 5M BehaviorGPT trained on 20% data can achieve lower minADE than the Trajeglish trained on 100%.

**2. Inference efficiency.**

The number is averaged over all scenarios on the validation set, with the standard variance of 3ms. The number of agents on the validation set is 65$\pm$36 (max: 128, min: 2).
Thank you very much for the positive feedback! We will supplement these important experimental results and continue to improve the quality of our paper as you suggested.

Below is a bit more discussion regarding your comments: 

**1. Assumption of unimodal within a path.**

The motivation behind this assumption is that a path represents a high-level intention of agents, with the corresponding mixing coefficient capturing the likelihood of the intention (which we call "intention uncertainty"). Each step of a path is modeled as a unimodal distribution, with the variance of the distribution capturing the "control uncertainty."

**2. The notion of "long-range."**

We are aware that the meaning of "long-range" depends on the specific context. For decision making and motion planning in autonomous driving, 9-second sequences are fairly long (imagine yourself as a driver or a pedestrian:  is it trivial to anticipate what will happen on the road 9 seconds later?). Since we have confined our paper to the domain of autonomous driving, we think the notion of long-range interaction would not be that confusing, but we will still clarify the sequence length produced by our model in the revised paper.

Thank you again for the insightful comment!
Thank you to the authors for their detailed response, i think most of my concerns are resolved. I would like to have my rating unchanged.
Thank you for your thorough followup!

> Scaling experiments / Significance of small models

The scaling experiments are very interesting.
 The reason I'm so hung up on model size is that up-to a certain size (depending on e.g. the cache size of the GPU) there is only a small impact when increasing the model size significantly, so increasing the model size by a factor X would not affect the runtime by that same factor X. This is especially true during inference where models need much less memory due to the lack of gradients.
 I do think that especially the second table still shows improvements as the model's depth is increased (plotting minADE is an almost linear improvement per added layer), but as I mentioned I do not hold the lack of training ressources against this work. It is still interesting that when increasing the depth we could expect further improvements.

> Does an extremely large model need the patching design?
> if autonomous driving companies can attain satisfying simulation quality with a small model, there is no reason for them to spend way more money on training and deploying an extremely large model.

Fair argument.

> Data efficiency

That's a pretty good result even after 20% of training. I recommend to add this into at least the appendix to support your claim of higher data efficiency (line 86 in your paper).

>Why are mixing coefficients only predicted once per patch?
> A7: Because we desire the mixing coefficients to represent the likelihood of multi-step behavior

Let me check whether I get this right: you effectively treat every chain of RNN calls as one possible "future" and then mix the different paths with your mode weights? I.e. you assume the distribution of paths is multi-modal, while each path in isolation remains unimodal. 
In that case your modelling makes sense (though I'm not sure how realistic that "unimodal within path" assumption is).

> Q8: Why does training take so much time?

Oh, I did not consider that the dataset is 1.4TB worth of motion data. In that case it makes sense that training through 100% of the data takes as much time as it does.

> Q9: Does training time change significantly with larger patches?

Good to know: My concern was that the recursive evaluation of the RNN itself made the training time as long as it took, but if that isn't a problem than that's great.

> Q10: Why only model the space dimension autoregressively?

Makes sense

> Q11: Long-term vs short-term. / How about the extrapolation ability to longer sequences?

The model being able to extraplolate to ~2 times its training length is sufficient for me to support your claim in line 87 (maybe also add this into the appendix).
It's worth noting that "long range interaction" can mean very different things: In your case "long range" is 90 steps (9s@10Hz) while other transformer papers talk about "long ranges" in the order of 1000 to 16000 steps (e.g. https://arxiv.org/abs/2011.04006). However, I'm also not sure how you can easily clarify this since what people think of "long range" heavily depends on their background.


From my end, the authors have adressed all my concerns to the best of their ability and I would increase my score to an accept (7).
Thank you to the authors for their thorough rebuttal.  I look forward to reading the revised manuscript.  I have a couple of small follow up points:

*On sample efficiency* - The claim in the paper is a comparative claim.  I.e. this method is "more efficient" than other methods.  The table provided demonstrates that this method performs well with a fraction of the training data.  Is there any reason to believe that other methods wouldn't scale similarly?

*On inference latency* - I would love more information here.  The 10ms, how many agents is that for?  Is that a mean?  What is the variance of inference times?

In short, the authors have addressed most of my concern and pending a revised manuscript I would significantly raise my score.