DriveGPT: Scaling Autoregressive Behavior Models for Driving
Authors: Xin Huang
Eric M Wolff
Paul Vernaza
Tung Phan-Minh
Hongge Chen
David S Hayden
Mark Edmonds
Brian Pierce
Xinxin Chen
Pratik Elias Jacob
Xiaobai Chen
Chingiz Tairbekov
Pratik Agarwal
Tianshi Gao
Yuning Chai
Siddhartha Srinivasa
Venue: ICML 2025 poster (2025)
Keywords: Autonomous Driving
Foundation Models
Behavior Modeling
Primary area: applications->robotics
TL;DR: Our paper studies the scaling properties in driving, across multiple orders of magnitude in data, model size, and compute.

Abstract: We present DriveGPT, a scalable behavior model for autonomous driving. We model driving as a sequential decision-making task, and learn a transformer model to predict future agent states as tokens in an autoregressive fashion. We scale up our model parameters and training data by multiple orders of magnitude, enabling us to explore the scaling properties in terms of dataset size, model parameters, and compute. We evaluate DriveGPT across different scales in a planning task, through both quantitative metrics and qualitative examples, including closed-loop driving in complex real-world scenarios. In a separate prediction task, DriveGPT outperforms state-of-the-art baselines and exhibits improved performance by pretraining on a large-scale dataset, further validating the benefits of data scaling.

DriveGPT: Scaling Autoregressive Behavior Models for Driving
Xin Huang'
Eric M. '
Paul Vernaza'
Tung Phan-Minh'
Hongge Chen' David S. Hayden
Mark Edmonds'
Brian Pierce'
Xinxin Chen
Pratik Elias Jacob'
Xiaobai Chen
Chingiz Tairbekov
Pratik Agarwal*
Tianshi Gao
Yuning Chai 2
Siddhartha S. Srinivasa
Abstract
We present DriveGPT, a scalable behavior model for autonomous driving. We model driving as sequential decision-making task, and learn a transformer model predict future agent states tokens in autoregressive fashion. We scale up model parameters and training data by multiple orders of magnitude, enabling us to ex- plore the scaling properties in terms of dataset size, model parameters, and compute. We evalu- ate DriveGPT across different scales a planning task, through both quantitative metries and quali- tative examples, including closed-loop driving in complex real-world scenarios. In separate pre- diction task, DriveGPT outperforms state-of-the- art baselines and exhibits improved performance by pretraining on a large-scale dataset, further validating the benefits of data scaling.
DO
Figure I. Through data and model scaling, DriveGPT (red) handles complex real-world driving scenarios, such lane changing in heavy traffic and yielding to a cyclist in the opposite lanc, com- pared to smaller bascline trained on less data (pink).
Model
Parameters
Training Segments
VectorNet
72K
211K
PlanTF
2.1M
1M
1. Introduction
QCNet-Argo2
7.3M
200K
OCNet-WOMD
7.5M
2.2M
Transformer-based foundation models have become increas- ingly prevalent in sequential modeling tasks across various machine learning domains. These models are highly ef- fective in handling sequential data by capturing long-range dependencies and temporal relationships. Their success has been evident in natural language processing (Mann et al., 2020; Kaplan et al., 2020; Hoffmann et al., 2022), time- series forecasting (Zhou et al., 2021), and speech recogni- tion (Kim et al., 2022), where sequential patterns play a crucial role. One the key strengths transformer-basee models their capacity to learn from large datasets includ- ing millions of training examples, enabling them address complex tasks with increased model sizes, up to billions of model parameters.
Wayformer
20M
2.2M
MotionLM
27M
1.IM
MTR
66M
2.2M
GUMP
523M
2.6M
DriveGPT (Ours) 1.4B
120M
Table 1. DriveGPT is 3x larger and is trained on ~500 more data sequences than existing published behavior models.
While scaling up model and dataset sizes has been critical for recent advances in sequential modeling for text predic- tion (Kaplan et al., 2020; Hoffmann et al., 2022), it remains unclear whether these scaling trends can be directly ex- tended behavior modeling, particularly in driving tasks, due to several unique challenges. First, driving tasks in- volve wider range of input modalities, including agent trajectories and map information, unlike language tasks that rely solely on textual inputs. Second, behavior modeling demands spatial reasoning and an understanding of physical kinematics. Such capabilities are typically beyond the scope of language models. Finally, the collection of large-scale
'Cruise LLC, San Francisco, CA MMeaa, Menlo Park, CA. Correspondence to: Xin Huang, Eric M. Wolff (cyruu.huung cric. cric.wolff' @ getcruise.com>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DriveGPT: Scaling Autoregressive Behavior Models for Driving
driving datasets requires substantial effort and resources, making it far more challenging than gathering textual data. As result, existing work is often constrained by the avail- ability of training data or the scalability of the models, as summarized in Table
2. Related Work
2.1. Behavior Modeling
Behavior modeling is a critical task in autonomous driving, which covers a broad spectrum of tasks including planning, prediction, and simulation. Taking multimodal inputs in- cluding agent history states and map information, behavior models predict the future states of these traffic agents by rea- soning about agent dynamics (Cui et al., 2020; Song et al., 2022), interactions (Sun et al., 2022; Jiang et al., 2023), human intent (Shi et al., 2022; Huang et al., 2022; Sun et al., 2024), and driving environments (Liang et al., 2020; Kim et al., 2021).
In our work, we present a comprehensive study of scaling up data sizes and model parameters in the context of be- havior modeling for autonomous driving, which predicts future actions of traffic agents to support critical tasks such as planning and motion prediction. Specifically, we train transformer-basee autoregressive behavior model on over 100 million high-quality human driving examples, ~50 times more than existing open-source datasets, and scale the model over billion parameters, outsizing existing pub- lished behavior models.
Among learning- carning-based models, transformers have gained popularity due to their ability to fuse multimodal inputs as encoders (Nayakanti et al., 2023; Zhang et al., 2023; Zhou et al., 2023; Jia et al., 2023; Gan et al., 2024) and model long-range temporal relationships as decoders (Seff et al., 2023; Shi al., 2022; 2024). Despite the success of trans- formers in behavior modeling. existing literature is often restricted by the size of model parameters due to limited training data, which fails to capture the full scaling poten- tial of wansformer-basee models. In our work, we scale up our transformer models to include billions of parameters, by training on a large-scale dataset including more than 100M driving demonstrations, and validate the scalability of transformer- models in the context of autonomous driving.
As we scale up the volume of training data and the num- ber of model parameters, we observe improvements in both quantitative metrics and qualitative behaviors. More impor- tantly, large models trained on extensive, diverse datasets can better handle rare or edge-case scenarios, which of- ten pose significant challenges for autonomous vehicles, as shown in Fig. 1 As result, we see great potential in scal- ing up behavior models through data and model parameters to improve the safety and robustness of autonomous driving systems.
Our main contributions are as follows:
1. We present DriveGPT, large autoregressive behavior model for driving, by scaling up both model parameters and real-world training data samples.
2.2. Large Transformer Models
Large transformers have demonstrated great success in se- quential modeling tasks, by scaling up model parameters and data sizes (Kaplan et al., 2020; Hoffmann al., 2022: Zhai et al., 2022; Muennighoff et al., 2023). These scaling laws have pushed the boundary of many sequential mod- eling tasks including natural language processing (Mann et al., 2020), time-series forecasting (Zhou et al., 2021), and speech recognition (Kim et al., 2022).
2. We determine empirical driving scaling laws for an autoregressive behavior model in terms of data size, model parameters, and compute. We validate the value of scaling up training data and compute, and observe better model scalability as training data increases, con- sistent with the language scaling literature.
Recent work has studied the scalability of behavior mod- els in the context of motion prediction (Seff et al., 2023; Ettinger et al., 2024), planning (Sun et al., 2023), and sim- ulation (Hu et al., 2024b), yet these studies are either con- strained by limited data size (up to a couple of million training examples), or focused on a few orders of magnitude in terms of data and model scaling, limiting the potential to draw statistically significant conclusions over a large scaling range.
3.
We quantitatively and qualitatively compare models from our scaling experiments to validate their effec- tiveness in real-world driving scenarios. We present real-world deployment of our model through closed- loop driving in challenging conditions.
4.
We demonstrate the generalizability of our model on the Waymo Open Motion Dataset, which outperforms prior state-of-the-art on the motion prediction task and achieves improved performance through large-scale pretraining.
In this paper, we study the scaling properties (in terms of data samples, model parameters, and compute) across a much larger range compared to prior work. More specifi- cally, we target an autoregressive decoder architecture that has been proven to be both scalable (as in the L.M liter-
'The baseline model trained on ~50 times less data and uses 50 times fewer parameters.
2

---

DriveGPT: Scaling Autoregressive Behavior Models for Driving
$1
32
ST
S
Transformer Encoder
Transformer Decoder
Scene Embedding
Tokens
So
$1
ST
Sy
Figure 2. DriveGPT architecture, including a transformer encoder and transformer decoder. The transformer encoder summarizes relevant scene context, such as target agent history, nearby agent history, and map information, into set of scene embedding tokens. The transformer decoder follows an LLM-style architecture that takes a sequence of agent states as input and predicts discrete distribution of actions the next tick, conditioned on previous states.
ature) and effective in generating accurate trajectories for traffic agents (Seff et al., 2023; Chen et al., 2024; Hu al., 2024a).
3.2. Scene Encoder
The encoder follows a standard transformer encoder archi- tecture (Shi et al., 2022) that fuses all input modalities into set of scene embedding tokens. It consumes raw input features, including target agent history states, nearby agent history states, and map states set of vectors, and nor- malizes all inputs to an agent-centric view. Each vector is mapped to token embedding through a PointNet-like encoder as (Gao et al., 2020). At the end of the encoder, we apply a self-attention transformer (Vaswani al., 2017) 7)) to fuse all input context into set of encoder embeddings. paxd where n is the number of vectors and the token dimensions, that summarize the driving scene.
Beyond autonomous driving, there is limited relevant lit- erature on building large transformer models for robotic tasks (O'Neill et al., 2024; Octo Model Team et al., 2024), which share transformer architecture similar to our work. Robotics models often share different input features and dynamic models, and operate in different environments, making them difficult apply directly driving tasks.
3. Behavior Model
We use standard encoder-decoder architecture as our be- havior model, as shown in Fig. 2. We use seaeffomer-bbsee models as our encoder and decoder backbones due to their scalability related sequential modeling tasks.
3.3. LLM-Style Trajectory Decoder
Inspired by the LLM literature (Radford al., 2019), we follow (Seff et al., 2023) to transformer decoder archi- tecture to predict the distribution of agent positions each step in the future.
3.1. Problem Formulation
We model the problem as a sequential prediction task over the future positions of the target agent up to horizon T,, by applying the chain rule each step, conditioning on driving context information and historical agent positions s:
The decoder first tokenizes agent positions at all steps into embeddings with dimension d through a linear layer, fol- lowed by a LayerNorm layer and a ReLU layer. At each step the decoder takes agent embeddings up to f, and cross attends them with the encoder embeddings distribution of agent positions at the next step / + 1.
predict the
P(surlC) P(sa ((
The output is a set of discrete actions a represented as the Verlet action (Rhinehart et al., 2018; Seff et al., 2023), as the second derivative of positions. We can apply the following equation to map Verlet actions to positions:
The context information includes target agent history states Ourget, nearby agent history states Cnemby, and map states Cmap- The historical agent information includes agent positions from previous historical steps, i.e. S0:t-1 if want to predict agent positions step
81+1
(sc 8t-1) ) + dz
(2)
We define "state" as complete kinematic state including position, orientation, velocity, and acceleration, which is commonly available in agent history observations, and "po- sition" as 2-D (x, coordinates simplify the output space.
where a, is the predicted Verlet action, and (st s s,-1) assumes a constant velocity step. This representation helps predict smooth trajectories using small set of actions.
3

---

DriveGPT: Scaling Autoregressive Behavior Models for Driving
3.4. Training
Model Size
Hidden Dimension (dmodel) Max LR
To train a DriveGPT model, we follow teacher forcing by applying ground truth future positions as imput to the trajec- tory decoder. This allows us to predict all future steps in parallel.
1.5M
64
0.0050
4M
128
0.0020
8M
192
0.0014
12M
256
0.0010
26M
384
0.0010
We use a single cross-entropy classification loss over the action space, where the target action is selected as the one that closest to the ground truth future trajectories. We refer the reader to Appendix A for more training details.
94M
768
0.0007
163M
1024
0.0004
358M
1536
0.0002
629M
2048
0.0001
1.4B
3072
0.0001
3.5. Inference
At inference time, we follow a standard LLM setup and roll out trajectory over horizon T' autoregressively, by repeating the process of predicting the action distribution at the next step, sampling action, and adding it back to the input sequence.
Table 2. We vary the model size over three orders of magnitude through hidden dimensions. For each model size, report the optimal learning rate, which decreases as the model size increases, matching the observations the LLM scaling literature (Kaplan et al., 2020; Hoffmaan et al., 2022).
We sample multiple trajectories batch to approximate the distribution and then subsample to the desired number of modes using K-Means, as in (Seff et al., 2023).
multiple learning rate schedules using a cosine decay over the full training steps and select the maximum learning rate that yields the best performance. Table. 2 summarizes the optimal learning rate for each model size. Consistent with practices in large language model scaling (Kaplan et al., 2020), each model is trained for single epoch.
4. Scaling Experiments
The goal of our scaling experiments is to determine the effect of data and model size on behavior prediction perfor- mance. Quantifying scaling laws similar to those seen in LLMs (Kaplan et al., 2020) can help prioritize the value of data and compute for future research directions in behavior modeling. We focus our effort on exploring the next frontier of data and model size over order of magnitude beyond previously published work.
We evaluate model performance using validation loss, com- puted on a comprehensive validation set of 10 million sam- ples drawn from the same distribution the training data, with no overlap. This set remains fixed across all scaling experiments to ensure consistency. We use validation loss as proxy to measure model performance, following standard practices in scaling studies (Kaplan et al., 2020; Hoffmann et al., 2022; Muennighoff et al., 2023). This loss, calculated as cross-entropy on next-action prediction, serves as our primary performance metric. Additional driving-specific metrics are reported in Sec. 5.1.
Large-scale driving dataset From millions of miles of high-quality real-world human driving demonstrations, we curate small subset of 120M segments for an internal research dataset. The dataset is carefully curated and bal- anced to represent diverse geographic regions across multi- ple cities and countries, including the United States, Japan, and the UAE Data collection is evenly distributed between daytime and nighttime and is conducted primarily in ur- ban environments. The dataset captures a wide range of challenging driving scenarios, such as lane changes, inter- sections, double-parked vehicles, construction zones, and close interactions with pedestrians and cyclists.
4.1. Data Scaling
Data scaling results are summarized in Fig. 3. The smallest dataset of 2.2M samples mimics the size of Waymo Open Motion Dataset (WOMD) (Ettinger et al., 2021), large open-source dataset for behavior modeling (~444 scenarios with multiple target agents per scenario). We select a few subsets of our internal research dataset to study data scaling across different orders of dataset sizes. Our experiments use 500 more data than WOMD, exploring new region of the design space.
We extracted map information, target agent states, and nearby agent states into vectorized representation, as cus- tomary in behavior modeling literature (Gao et al., 2020).
The results indicate that as the model is trained on more unique data samples, the performance improves, regardless of model size. Extrapolating from the scaling law in Fig. 3, to improve the best loss by another 10%, we would need to include 350M more training examples. 20% improvement
Scaling overview We scale the model size across three orders magnitude, from 1.5 million to 1.4 billion parame- ters, by increasing the embedding dimension in both encoder and decoder transformers. For each model size, we explore
4

---

DriveGPT: Scaling Autoregressive Behavior Models for Driving
3.0
perameters
1.3M
2.8
Unique training samples
8.0M
Validation loss
Validelion loss
26M
2.6
5M
3.0
94M
358M
163M
2.4
10.5M
21M
629M
42M
1.46
120M
10a
10a
10"
Trainable parameters
10a
Unique training samples
Figure 4. Model scaling more effective as training data increases. The validation loss improves up to ~100M parameters when trained on the full dataset.
Figure 3. Performance increases with dataset size across range of model parameters, indicating that data is limiting factor. Both log scale. An exponential fit applied to all data points except for the 1.5M curve, resulting in the following relationship: log(L) -0.102 log D) 2.663 with R value of 0.986, where L is the validation loss and D is the number of unique training samples.
Trainable
3.0
1.5M
Training loss
8M
2.0
26M
would require about 1.4B more examples. As result, we find that data remains the bottleneck for further improving driving performance.
2.6
163M
358M
629M
1.48
Lastly, the scaling results remain relatively consistent across model sizes. This consistency indicates that data scaling comparisons be done reasonably small model sizes beyond 10M parameters.
Min bound
10
FLOPS
4.2. Model Scaling
Figure 5. Relationship between (smoothed) training loss and FLOPs. Each curve represents experiment corresponding to specific model size, and "min bound" indicates the best perfor- mance possible for given FLOP budget.
We study model sizes across three orders of magni- tude (1.5M 1.4B parameters), listed in Table 2. We increase model size by increasing the hidden dimensions of transformers for simplicity. We notice that modifying other parameters such as number attention heads and hidden dimensions per head does not lead noticeable changes in the results, studied in Appendix B.
plateau and eventually overfitting. These findings fur- ther reinforce that data is the primary bottleneck for scaling models, aligning with observations in the LLM scaling liter- ature (Kaplan et al., 2020). We leave further exploration of scaling behavior with larger datasets future work.
Training larger models is sensitive learning rates, as ob- served in other scaling studies (Kaplan et al., 2020; Hoff- mann et al., 2022). For each model size, we run multiple experiments at different learning rates to select the one with the optimal performance, as summarized in Table. 2.
4.3. Compute Scaling
In Fig. 5, we examine how compute affects training loss, where compute is measured by Floating Point Operations (FLOPs). We identify monotonically decreasing "min- bound" boundary, which shows the lowest training loss observed up to the current compute value. As we increase compute, training loss generally decreases. Initially, this decrease quite steep, but it gradually slows down at higher FLOPs values. This trend consistent with observations in the LLM scaling literature, such as those reported by (Hoff-
Results Fig. 4 demonstrate that increasing the amount of training data enhances the effectiveness of model scal- ing. Specifically, when the dataset size is up to 21M, the impact on the validation loss is barely noticeable across large range of model sizes. Beyond 21M samples, validation losses improve with larger models- up to 12M parameters when trained on the 42M dataset and up to 94M parame- ters when trained on the 120M dataset before reaching
5

---

DriveGPT: Scaling Autoregressive Behavior Models for Driving
3.75
DriveGPT (ours)
FLOPs
minFDE 6s [m]
One-shot decoder
3.50
1.0e15
4.4e15
Validation loss
3.25
8.9e15
1.1e16
0.9
3.00
2.5e16
3.1e16
2.75
4.8e16
1.0c17
107
109
2.50
1.9e17
Trainable parameters
3,0e1/
2.25
Best
Figure 7. DriveGPT achieves better scalability in terms of minFDE, by adopting autoregressive architecture compared to one-shot architecture.
Trainable parameters
parameters. While find it harder scale the one-shot decoder, confirm that our autoregressive decoder scales up to 100M parameters in terms of prediction accuracy, and defer further scalability study on the one-shot decoder as future work.
Figure 6. Performance as a function of model size for fixed com- pute hudgets. The solid gray line connects the best result from each compute budget.
mann et al., 2022), covering a subset of the full FLOP range explored in these studies.
5. Planning and Prediction Experiments
In this section, show detailed results of DriveGPT in planning task using our internal research dataset and motion prediction task using an external dataset. The results here further explore the impact of scaling from Section 4 and help ground those results in driving tasks and metrics.
Next, investigate whether there is an optimal combina- tion of data size and model parameters for fixed compute budget. Given the large computational expense for train- ing models at the scales we are exploring, it is important to make the best use of our data. In this study, we fixed the compute budget in different FLOP groups. For a fixed compute budget, we can allocate resources either to model parameters or data samples, keeping their product constant.
5.1. Internal Evaluation: AV Planning
For the planning task, we train our model using our internal research dataset, composed of millions of high-quality hu- man driving demonstrations, and generate AV trajectories by autoregressively predicting the next action, as described in Sec. 3.5.
Fig. 6 plots the performance with different compute budgets. The trend clearly shows that larger compute budget leads to better performance, with optimal model size increasing accordingly, as indicated by the "best" gray line. The results further reveal that data is the main bottleneck, as the smallest model outperforms others in the three largest FLOP groups.
We approximate the distribution by oversampling trajecto- ries in batch and subsampling to 6 trajectories as in (Seff et al., 2023). While the AV must ultimately select a single trajectory for planning, returning multiple samples helps bet- ter understand multimodal behavior and aligns with motion prediction metrics.
4.4. Ablation Study on Decoder Architecture
We scale up two different model architectures by two orders of magnitude: our autoregressive decoder and a one-shot decoder. For the one-shot decoder, follow (Nayakanti et al., 2023) to a transformer decoder that takes set of learned queries and cross attends them with scene em- beddings to produce trajectory samples. This decoder is referred to as "one-shot" because generates the full tra- jectory rollout at once, where an autoregressive decoder follows an LLM-style to produce trajectories one step at a time.
We measure the planning performance comprehensive test set through set of standard geometric metrics including minADE (mADE), minFDE (mFDE), and miss rate (MR). Additionally, we use semantic- based metrics including of- froad rate (Offroad) that measures the ratio of trajectories that leave the road and collision rate (Collision) that mea- sures the ratio of trajectories overlapping with traffic agents. We normalize these metrics across experiments to highlight relative performance changes.
The results are summarized in Fig. 7, where we minFDE at 6 seconds as proxy to measure the model performance because of different loss definitions between two decoder architectures. Despite worse performance at small-scale parameters, our autoregressive decoder achieves better scal- ability and outperforms the one-shot baseline beyond 8M
1.1.1. DATA SCALING RESULTS
We compare a DriveGPT model at 26M parameters trained on datasets of different sizes. The baseline dataset (2.2M) is selected to mimic the size of a typical behavior modeling
6

---

DriveGPT: Scaling Autoregressive Behavior Models for Driving
Data
mADE 4
mFDE 4
MR
Offroad
4 Collision +
Model
mADE + mFDE +
MR +
Offroad 4 Collision
2.2M
1.000
1.000
1.000
1.000
1,000
8M
1.000
1.000
1.000
1.000
1.000
21M
0.561
0.496
0.420
0.326
0.269
26M
0.954
0.950
0.902
0.858
0.915
85M
0,496
0,441
0.332
0.238
0.217
94M
0.937
0.925
0.866
0.815
0,890
120M
0.489
0.433
0.317
0.198
0.196
163M
0.943
0.925
0.875
0.815
0.817
Table As we scale up more training data, DriveGPT produces better AV trajectories as measured across metries. Metries are normalized to highlight relative performance.
Table 4, As we scale more model parameters, all metrics im- prove up 94M. Collision rate continues improve at 163M. Metries are normalized to highlight relative performance.
Figure 9. By training with 12x more parameters, DriveGPT (red) produces realistic trajectories that obey lane boundaries when performing right turn into traffic.
Figure 8. By training on 50x more data, DriveGPT (red) is able produce high-quality trajectories red that keep a safe lateral distance from jaywalker (top) and go around two double-parked vehicles (bottom).
with further gains in the collision metric at 163M parameters (see Sec. D.1 for qualitative examples). Although valida- tion loss shows diminishing returns beyond 94M in Fig. 4, driving metrics continue to improve with increased model capacities, highlighting the potential benefits of introducing more parameters for enhanced driving performance.
dataset such as WOMD.
The results are presented in Table 3, where we see that train- ing on more data samples significantly improves the quality of the predicted AV trajectories, in terms of critical seman- ties metries in driving including offroad rate and collision rate, as well as geometric metrics. These improvements are consistent with Sec. 4.1.
We present a qualitative example in Fig. 9, where larger DriveGPT including 94M parameters produces better trajectory samples in red that stay within the road boundary, compared to a smaller version including 8M parameters, in right turn scenario.
We further present two qualitative examples in Fig. to illustrate the value of training on more data. In these exam- ples, red trajectories represent DriveGPT trained on 120M samples, and pink trajectories are from the same model trained 2.2M samples. The examples show that our method produces map-compliant and collision-free free trajec- tories when trained on more data, successfully handling complicated interactions involving a jaywalking pedestrian and two double-parked vehicles.
5.1.3. CLOSED-LOOD DRIVING
We demonstrate the effectiveness of DriveGPT as real- time motion planner deployed in closed-loop setting. The model takes input features from an industry-level perception system that outputs agent states and map information. We used the 8M DriveGPT model, trained on the full dataset, to drive the car, achieving a latency of under 50ms on single onboard GPU.
5.1.2. MODEL SCALING RESULTS
We train four models using 120M internal research dataset, and select the 8M model as the baseline. The base- line represents a reasonable size at which our model starts to outperform one-shot decoders, as shown in Sec. 4.4.
In Fig. 10, we present challenging example in dense urban traffic where there are two double-parked vehicles blocking the path forward along with other oncoming ve- hicles. DriveGPT generates smooth and safe trajecto- ries, bypassing the blocking vehicles and moving back to the original lane afterward. More examples are pre-
The results are presented in Table 4, where all metrics im- prove as the model size increases up to 94M parameters,

---

DriveGPT: Scaling Autoregressive Behavior Models for Driving
matic features for traffic agents in WOMD.
We pretrain DriveGPT by training our internal research dataset for one epoch (as in Sec. 4), We load the pretrained checkpoint and finetune the model using the same training setup as in the MTR codebase, where we train the model for epochs using a weighted decay learning rate scheduler.
5.2.3. RESULTS
We measure model performance via of standard WOMD metrics, including minADE, minFDE, miss rate, and soft mAP. Each metric is measured on the test set and computed over three different time horizons.
108
We present two variants of our method to validate its effec- tiveness on the motion prediction task, including DriveGPT- WOMD that is trained on WOMD, and DriveGPT- 1 120M internal research
Finetune that is pretrained dataset and finetuned on WOMD. For baselines, we use set of representative state-of-the -hrt models.
We report results on the WOMD test set in Table 5. The results demonstrate that our method outperforms existing state-of- the-art non-ennebblee models in terms of geometric metries. Compared to Wayformer (Nayakanti al., 2023) and MotionLM (Seff al., 2023) that ensembles of to 8 replicas, our model achieves the best minADE and minFDE metrics and the second-best miss rate metric with- out any ensembling.
Figure Example of DriveGPT running as closed-loop planner in real time in dense urban traffic. It first produces a forward tra- jectory (top), updates it to bypass houble-parked vehicles (middle), and drives back the original lane at the end (hottom).
sented in the supplementary video at https: / youtu. be/-hLi4 4P EYBg, where DriveGPT alone is responsible for driving in real time.
While we prioritize geometric metries that emphasize the recall of predicted trajectory samples (i.e. not missing the critical trajectory), our model shows lower soft mAP scores due to suboptimal probability estimates. These estimates suffer from accumulated noises over time, as they are com- puted by compounding the action probabilities over a long sequence (as described in Eq. (1)), resulting in less accurate sample probabilities and reduced soft mAP, which relies on accurate probability assignments across predicted samples. Consequently, we notice that the gap in soft mAP grows as the prediction horizon increases. This reveals a limitation of using autoregressive decoder for accurate probabil- ity estimation, as also noted in the LLM literature (Jiang et al., 2021; Geng et al., 2024). We defer improving the probability estimates of autoregressive models for behav- ior modeling as future work. One potential direction is to train an additional probability prediction head for each sam- ple, which could enhance probability estimates and lead to improved soft mAP scores.
5.2. External Evaluation: Motion Prediction
To directly compare with published results, we evalu- ate DriveGPT on the WOMD motion prediction task. Addi- tionally, we explore the benefits of scale by pretraining on our internal research dataset and finetuning on the signifi- cantly smaller WOMD dataset.
5.2. 1. OPEN-SOURGE ENCODER
For our external evaluation, we use the open-source MTR (Shi et al., 2022) encoder. This encoder is similar to the one described in Sec. 3.2. We make this change to improve the reproducibility of our results and take advan- tage of MTR's 's open- open-source dataloading code for WOMD. We use the same autoregressive decoder described in Sec. 3.3.
We observe up to 3% additional gains by pretraining on our internal dataset, despite a large distribution shift between our internal dataset and the public WOMD dataset, terms
5.2.2. PRETRAINING SETUP
We made a couple of minor modifications to DriveGPT to be compatible with the WOMD dataset. First, we modify our map data to include the same semantics as in WOMD. Second, we modify our agent data to include the same kine-
Metrics are sourced from (Zhang et al., 2023) and WOMD leaderboard.
8

---

DriveGPT: Scaling Autoregressive Behavior Models for Driving
minADE
minFDE4
Miss Rate]
Soft mAP
MTR (Shi et al., 2022)
0.6050
1.2207
0.1351
0.4216
HDGT (Jia et al., 2023)
0.7676
1.1077
0.1325
0.3709
HPTR (Zhang et al., 2023)
0.5565
1.1393
0.1434
0.3968
ControlMTR (Sun et al., 2024)
0.5897
1.1916
0.1282
0.4572
MTR++ (Shi et al., 2024)
0,5906
1.1939
0.1298
0,4414
Wayformer! (Nayakanti et al., 2023)
0.5454
1.1280
0. 0.1228
0,4335
MotionLM (Seff et al., 2023)
0.5509
1.1199
0.1058
0.4507
DriveGPT-WOMD
0.5279
1.0609
0. 0.1236
0.3795
DriveGPT-Finetune
0.5240
1.0538
0.1202
0.3857
Table On the WOMD test set, DriveGPT achieves better results in geometric metrics without any ensembling, and improved performance after pretraining. Results averaged over three agent types (vehicle, pedestrian, eyclist) and three prediction horizons (3s, 5s, 8s). The best metric is highlighted bold and the second best is underlined. t denotes ensemble.
Vehicle
minADEJ
minFDE4
Miss Rate
MTR
DriveGPT
MTR
0.7642
1.5257
0.1514
DrOM
0.6396
1.2823
0.1213
DriveGeT-Finetune
0.6326
1.2636
0.1181
Cyclist
minADE
minFDEJ
Miss Ratel
MTR
0.7022
1.4093
0.1786
Drve
0.6317
1.2617
0.1756
DriveGPT-Finetune
0.6294
1.2641
0.1691
DriveGPT
Pedestrian
minADEJ
minFDE,
Miss Rate.
MTR
0.3486
0.7270
0.0753
DriVEGPT-WOMD
0.3122
0.6388
0.0738
DriveGPT-Finetune
0.3100
0.6338
0.0733
Table DriveGPT achieves better geometric metrics across diverse set of agent types. Results are averaged over 3 prediction horizons.
Figure 11. Compared MTR (left), DriveGPT (right) produces more accurate and diverse trajectories in complex intersections. Blue represents ground truth future trajectories.
of trajectory distributions, feature noises, and differences in semantic definitions.
6. Conclusion
Diving deeper into agent-specific results in Table 6, we see consistent improvements across all agent types, compared to MTR that shares the same encoder as ours. This further validates the generalizability of our method, in addition to vehicle behavior modeling results described in Sec. 5.1.
We introduced DriveGPT, LLM-style autoregressive be- havior model, to better understand the effects of model parameters and dataset size for autonomous driving. We sy systematically examined model performance as a function of both dataset size and model capacity, revealing LLM- like scaling laws for data and compute, well as dimin- ishing returns with increased model size. We showed the quantitative and qualitative benefits of scaling for planning in real-world driving scenarios. Additionally, we demon- strated our method on public motion prediction benchmark, where DriveGPT outperformed state-of-the --thh-a art baselines and achieved improved performance through pretraining on large-scale dataset.
5.2.4. QUALITATIVE COMPARISON
We present two qualitative comparisons Fig. 11, where DriveGPT produces better trajectories terms of diversity (covering more distinct outcomes) and accuracy (matching with the ground truth future) compared to MTR. This improvement is evident in challenging scenarios with limited agent history information (top row) and multiple future modalities (bottom row).

---

DriveGPT: Scaling Autoregressive Behavior Models for Driving
Acknowledgments
Geng, J., Cai, F., Wang, Y., Koeppl, H., Nakov, P., and Gurevych, I. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 6577-6595, 2024.
We thank, in alphabetical order, Abbas Shikari, Akshay Rangesh, Allan Lazarovici, Andy Zhuang, Ann Yan, Anton Spiridonov, Burkay Donderici, David Fang, Dennis Park, Florent Bekerman, Greg Meyer, Hans Zhang, Haytham Mo- hamed, Joe Liu, Jonathan Sorg, Junyuan Shi, Ling Li, Mao Ye, Matthew Creme, Peng Tang, Poonam Suryanarayan, Rizwan Chaudhry, Rosalie Zhu, Siva Karthik Mustikovela, Srinidhi Myadaboyina, Tanmay Agarwal, Xinlei Pan, Yi Hao, Yunlong He, Zaiwei Zhang, Zisu Dong, and many oth- ers from Cruise data and infra teams, for their contributions to the paper.
Hoffmann, J., Borgeaud, S., Mensch, A.. Buchatskaya, E., Cai, T., Rutherford, E.,, Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.1555. 2022.
Hu, X., Yin, W., Jia, M.. Deng, J., Guo, X., Zhang, Q., Long, X., and Tan, P. DrivingWorld: Constructing world model for autonomous driving via video GPT. arXiv preprint arXiv: 24/2. 19505, 2024a.
Impact Statement
This paper aims to advance research in behavior modeling for autonomous vehicles. Since AVs interact with the public, our work has the potential to enhance their behavior around other road users, ultimately contributing to safer roads.
Hu, Y., Chai, S., Yang, Z., Qian, J., Li, K.,, Shao, W., Zhang, H., Xu, W., and Liu, Q. Solving motion planning tasks with scalable generative model. In European Confer- ence on Computer Vision, pp. 386-404. Springer, 2024b.
References
Huang, X., Rosman, G., Gilitschenski, I,, Jasour, A., McGill, S. G., Leonard, J. J., and Williams, B. C. HYPER: Learned hybrid trajectory prediction via factored infer- ence and adaptive sampling. In 2022 International Con- ference on Robotics and Automation (ICRA), pp. 2906- 2912. IEEE, 2022.
Chen, Y., Wang, Y., and Zhang, Z. DrivingGPT: Unify- ing driving world modeling and planning with multi- modal autoregressive transformers. arXiv preprint arXiv:2412.18677, 2024.
Cui, H., Nguyen, T., Chou, F.-C., Lin, T.H.., Schneider, J., Bradley, D., and Djuric, N. Deep kinematic models for kinematically feasible vehicle trajectory predictions. In 2020 IEEE International Conference on Robotics and Automation (ICRA). pp. 10563-10560 IEEE, 2020.
Jia, X., Wu, P., Chen, L., Liu, Y., Li, H., and Yan, J. HDGT: Heterogeneous driving graph transformer for multi-agent trajectory prediction via scene encoding. IEEE transac- tions on pattern analysis and machine intelligence, 2023.
Ettinger, S., Cheng. S., Caine, B., Liu, C., Zhao, H., Prad- han, S., Chai, Y. Sapp, B., Qi C. R., Zhou, Y., et al. Large scale interactive motion forecasting for autonomous driv- ing: The Waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9710-9719, 2021.
Jiang, C., Cormman, A.. Park, C., Sapp, B., Zhou, Y., Anguelov, D., et al. MotionDiffuser: Controllable multi- agent motion prediction using diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9644-9653, 2023.
Jiang, 7.,, Araki, J., Ding, H., and Neubig, G. How can we know when language models know? On the calibration of language models for question answering. Transactions of the Association Computational Linguistics, 9:962- 977, 2021.
Ettinger, S., Goel, K., Srivastava, A., and AI-Rfou, R. Scal- ing motion forecasting models with ensemble distillation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 4812 4818. IEEE, 2024.
Gan, Y., Xiao, H., Zhao, Y., Zhang, F.,, Huang, 7.,, Ye, X., and Ge, L. Multi-granular transformer for motion predic- tion with lidar. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 15092- 15098. IEEE, 2024.
Kaplan, J., McCandlish, S., Henighan. T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001 0006001010031.1.11000 2020.
Kim, B., Park, S. H., Lee, S., Khoshimjonov, E.,, Kum, D., Kim, J., Kim, J. S., and Choi, J. W. LaPred: Lane-aware prediction of multi-modal future trajectories of dynamic agents. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition. pp. 14636 14645, 2021.
Gao, J.. Sun, C., Zhao, H., Shen, Y., Anguelov, D., Li, C.,, and Schmid, C. VectorNet: Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11525-11533. 2020.
10

---

DriveGPT: Scaling Autoregressive Behavior Models for Driving
Kim, S., Gholami, A., Shaw, A., Lee, N., Mangalam, K., Malik, J., Mahoney, M. W., and Keutzer, K. Squeeze- former: An efficient transformer for automatic speech recognition. Advances in Neural Information Processing Systems, 0199361-9375 2022.
Shi, S., Jiang, L.,, Dai, D., and Schiele, B. Motion trans- former with global intention localization and local move- ment refinement. Advances in Neural Information Pro- cessing Systems, 35:6531 1-543, 2022.
Shi, S., Jiang, L., Dai, D., and Schiele, B. MTR++ Multi- agent motion prediction with symmetric scene modeling and guided intention querying. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5):3955- 3971, 2024.
Liang, M., Yang, B., Hu, R., Chen, Y.. Liao, R., Feng, S., and Urtasun, R. Learning lane graph representations for motion forecasting. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 541-556. Springer, 2020.
Song, H., Luan, D., Ding, W., Wang, M. Y., and Chen, Q. Learning to predict vehicle trajectories with model- based planning. In Conference on Robot Learning, PP. 1035-1045. PMLR, 2022.
Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., et al. Language models are few-shot learners. arXiv preprint arXiv:2005, 14165,, 1, 2020.
Sun, J., Yuan, C., Sun, S., Wang, S., Han, Y., Ma. S., Huang, Z., Wong, A., Tee, K. P., and Ang, M. H. ControlMTR: -ontrol-guided motion transformer with scepe-compliant intention points for feasible motion prediction. In 2024 IEEE 27th International Conference Intelligent Trans- portation Systems (ITSC), pp. 1514- IEEE, 2024.
Muennighoff, N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. A. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358-50333
2023. Nayakanti, N., Al-Rfou, R., Zhou, A., Goel, K., Refaat, K. S., and Sapp, B. Wayformer: Motion forecasting via simple & efficient attention networks. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2980-2987. IEEE, 2023.
Sun, Q.. Huang, X., Gu, J., Williams, B. C., and Zhao, H. M21: From factored marginal trajectory prediction to interactive prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6643-6552 2022.
Sun, Q., Zhang, S., Ma, D., Shi, J., Li, D., Luo, S., Wang. Y., Xu, N., Cao, G., and Zhao, H. Large trajectory models are scalable motion predictors and planners. arXiv preprint arXiv:2310. 2023.
Octo Model Team, Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Xu, C., Luo, J., Kreiman, T., Tan, Y., Chen, Y., Sanketi, P., Vuong, Q., Xiao, T., Sadigh, D., Finn, C., and Levine, S. Octo: An open-source generalist robot policy. In Proceedings of Science and Systems, Delft, Netherlands, 2024.
Vaswani. A., Shazeer, N., Parmar, N., Uszkoreit. J., Jones, L., Gomez, A. N., Kaiser, t.., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30(2017), 2017.
Robotics:
O' O'Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embbdiment collab- oration O. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 2024.
Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 12104-12113, 2022.
IEEE,
Zhang, 7.,, Liniger, A., Sakaridis, C., Yu, F., and Van Gool, L. Real-time motion prediction via heterogeneous polyline transformer with relative pose encoding. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Radford, A., Wu, J.. Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference artificial intelligence, volume 35, pp. 11106- 11115, 2021.
Rhinehart, N., Kitani, K. M., and Vernaza, P. R2P2: A reparameterized pushforward policy for diverse, precise generative path forecasting. In ECCV, pp. 772-788, 2018
Seff, A., Cera, B., Chen, D., Ng, M., Zhou, A., Nayakanti, N., Refaat, K. S., Al-Rfou, R., and Sapp. B. MotionLM: Multi-agent motion forecasting as language modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 2599, 2023.
Zhou, Z., Wang, J., Li, Y.-H., and Huang, Y.-K Query- centric trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17863-17873, 2023.

---

DriveGPT: Scaling Autoregressive Behavior Models for Driving
A. Training Detail
A.1. Internal Evaluation: AV Planning
384
768
We train our models on the internal research dataset for epoch, as eustomary in the LLM scaling literature (Kaplan al., 2020; Hoffmann al., 2022).
Vallsation
Our models follow the implementations described in (Nayakanti et al., 2023; Seff et al., 2023), and are trained using a batch size of 2048 and standard Adam optimizer adopted in (Kaplan et al., 2020). We follow the optimal learning rate schedule discovered in (Hoffmann et al., 2022), which applies cosine decay with a eycle length equiva- lent the total number of training steps. All models were trained on 16 NVIDIA H100 GPUs.
encoder attention heads decoder attention heads
Figure 12. Adjusting the number of attention heads has smaller effect the trend of validation losses, compared changing the hidden dimension.
A.2. External Evaluation: Motion Prediction
We train our models on the WOMD data following the same setup in (Shi al., 2022). More specifically, use batch size of 80 and an AdamW optimizer with a learning rate of 0.0001. The models are trained for 30 epochs, where the learning rate is decayed by factor of 0.5 every 2 epochs, starting from epoch 20.
3.2
24.5
Uatted scall enooter city scaling decoder Solich scaling amooddr decocer
Unique training samples
3.0
20.1
10.5M
2.9
18.2
21M
Validetion
2.8
16.4
42M
10.5M
2.7
14.9
21M
B. Additional Ablation Studies
2.6
13.5
42M
B.1. Ablation Study on Attention Heads
2.5
12.2
21M
42M
In Fig. 12, present an ablation study examining the impact of the number of heads in the encoder and decoder transformers, while keeping the hidden dimension the same, as specified in Table. 4. For each hidden dimension (dmosta), we conduct experiments across multiple configurations of attention heads and present their validation losses using distinct color. While we notice a pattern where more decoder attention heads lead to better performance for small models, the scaling trend is more influenced by the hidden dimension size than by the number of attention heads when the hidden dimension remains fixed.
Trainable parameters
Figure 13. Scaling only the decoder exhibits a similar trend but worse results compared scaling both encoder and decoder. Scal- ing only the encoder has little impact beyond 10M parameters.
B.3. Ablation Study on Encoder Sealing
In Fig. 13, we present model scaling results (in dotted lines) where we only scale up the encoder, while keeping the decoder fixed size. The results indicate that encoder scaling is not as effective decoder scaling.
B.2. Ablation Study on Decoder Scaling
In Fig. 13, we present model scaling results (in dashed lines) where we only scale up the autoregressive decoder, while keeping the encoder at a fixed size. Compared to scaling both encoder and decoder (in solid lines), scaling only the decoder exhibits similar trend but yields worse perfor- mance, especially for models exceeding 10M parameters in two larger datasets. Therefore, our main results focus scaling both encoder and decoder.
C. Scaling From Perplexity View
Following the LLM scaling literature, we present our scaling results as function of validation loss. Another key aspect to highlight is the perplexity scale, Perplexity, defined the exponentiation of entropy, quantifies how well probability model predicts sample. It reflects the effective number of choices the model considers: perplexity of l: implies the model is k-way perplexed.
For the last group, we omit a few models duc to memory constraints.
In Fig. 14, we re-plot the scaling results, adding a secondary y-axis to represent perplexity. In Fig. 15, we plot the per-
12

---

DriveGPT: Scaling Autoregressive Behavior Models for Driving
3.0
20.1
2.8
16.4
training samples
Validation loc.
2.6
13.5
Peplexity
5M
10.5M
2.4
11.0
21M
42M
120M
10'
10a
10a
Trainable parameters
Figure 14. Fig. 4 replotted with a perplexity scale on the right y-- axis for of reference.
Figure 16. A larger DriveGPT model at 163M parameters gener- ates trajectories (red) that maintain safer distance from other traffic agents and road debris compared to a smaller model at 94M parameters (pink).
acrocs model
DriveGPT
10
DriveGPT
Unique training examples
Figure 15. We fit the best perplexity values from all models in Fig. 3 as function of dataset size. A simple lincar regression yields log(P") = 0.28 log(D) (D+ + 7.22 with an R2 value of 0.994, where / is the validation perplexity and D is the number of unique training examples.
Figure 17. examples where DriveGPT the largest error re- gressions compared to MTR, it predicts less accurate trajectories due to the presence of lane boundary (top row) or missing map information (bottom row). Blue represents ground truth future.
plexity of our best models function of unique training examples. Extrapolating from the perplexity fit in the figure, a perplexity of 7 would require about 70M more training examples and a perplexity of 5 would require 500M more training examples.
setting. The video covers representative challenging scenar- ios in driving, including a) unprotected left turn, b) double parked vehicle, c) construction zone, d) blow-through cy- clist, and e) lane change in heavy traffic.
D. Additional Qualitative Examples
D.1. Qualitative Comparison of Large Models
D.3. Failure Cases on WOMD
In Fig. 16, we present two qualitative examples demon- strating that a larger DriveGPT model (163M parameters) exhibits better collision avoidance capabilities, compared to its smaller -aM-aaaaeeeer variant.
We show two representative failure examples of DriveGPT in Fig. 17, selected based on the largest minFDE regressions of our method compared to the MTR baseline on WOMD, In the first example (top row), trajectories that make a left turn, likely due to the pres- ence of a lane boundary. In the second example (bottom row), DriveGPT predicts an undershooting trajectory that results in a larger longitudinal error due to missing map information in the data.
method fails to predict
D.2. Additional Closed-Looop Driving Examples
In the supplementary video at https: / /t///youuuuu bee chLi44P fYBg, we present additional examples of deploy- ing DriveGPT as a real-time motion planner in a closed-loop
13

---

DriveGPT: Scaling Autoregressive Behavior Models for Driving
Vehicle
Data
MADE
mFDE]
MR.
DromeGPT-WOMD
N/A
0.6381
1.2846
0.1233
DriveGPT-Finetune
40M
0.6368
1.2788
0.1217
DriveGPTT-Finetue
80M
0.6341
1.2736
0.1207
DriveGPT-Finetune
120M
0.6310
1.2655
0.1195
Table Using more training samples during pretraining gives bet- ter results (averaged over 3 horizons) in the finetuned model. This verifies the benefit of data scaling as shown Sec. 4.1.
Model Parameters mADE
mFDE]
MR.
2M
0.6953
1.4297
0.1489
4M
0.6534
1.3202
0.1298
9M
0.6423
1.2955
0.1249
14M
0.6381
1.2846
0.1233
21M
0.6396
1.2871
0.1225
Table Scaling model parameters DriveGPT-WOMI improves vehicle prediction metrics up 14M model parameters. Results are averaged over prediction horizons.
E. Additional WOMD Validation Results
In this section, we present additional ablation studies on the WOMD validation set, due to the submission limit of the WOMD test server.
E.1. WOMD Data Scaling in Pretraining
We validate the effectiveness of data scaling, by pretrain- ing DriveGPT on various sizes of our internal dataset. The results in Table 7 indicate that pretraining on more unique samples leads to better results in the finetuned model, align- ing with our findings in Sec. 4.1 that data scaling improves model performance.
E.2. WOMD Model Scaling
We present an ablation study on model scaling ing DriMeGPT-WOMD in Table 8. The results show that the model performance continues to improve to 14M parameters, with no further gains beyond this point due to the limited sample size in WOMD. Therefore, we choose to report the results from the 14M model Sec. 5.2.
14


Reviews:
[{"review_id": "EgaVPVKZIB", "reviewer": "Reviewer_mYMk", "summary": "The paper presents a large transformer model predicting future ego agent states in a Birds Eye View for autonomous driving. The focus lies on an investigation of the scaling properties of transformers for behavior modeling by increasing the model and dataset size significantly. The method beats some baselines on the Waymo Open Prediction test Dataset and can be transformed into a prediction planning method to drive in real life.\n\n## update after rebuttal\nGiven the rebuttal of the authors and comments of other reviewers I do not see controversy in leaning towards acceptance. Therefore I leave my original rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your detailed and thoughtful feedback. We appreciate the positive assessment of our work as a valuable contribution to the autonomous driving literature through our large-scale scaling experiments, with simple straightforward experimental design and good performance in WOMD and real driving. \n\nWe address your comments and questions below.\n\n---\n**Newer WOMD approaches**\n\nWe thank the reviewer for pointing this out. We will include more recent methods in the related works section and additional results including MTR++ in Table 4 in the final version.\n\nWhile MTR++ introduces a set of novel techniques to improve upon MTR, our method outperforms MTR++ in terms of minADE, minFDE, and MR, as indicated by Table 4 of our paper and Table 1 in the MTR++ paper.\n\n---\n**Diminishing scaling returns**\n\nWe agree with the reviewer’s observations on diminishing returns, which have also been noted in the LLM literature, and we appreciate the positive feedback on this insight. These findings are an important part of the story we want to share with the community to encourage further research in this direction. In the final version, we will clarify this discussion and defer exploring additional scaling trends using more data samples as future work.\n\n---\n**Dataset details**\n\nOur dataset is collected from multiple countries and cities across North America and Asia, primarily through urban driving, with data evenly distributed across day and night. To maintain anonymity for the double-blind review, we will provide more details on the countries and cities in the final version.\n\nAs noted on Page 4, Line 171, we process camera, LiDAR, and HD map data into vectorized representations. While the training data includes only vehicle driving, each scene is captured in dense urban environments containing numerous cyclists and pedestrians exhibiting diverse behaviors, such as jaywalking, blowthrough, and riding in the opposite lane, as shown in our qualitative examples.\n\nWe will include all requested dataset details in the final version and are happy to provide any additional information if the reviewer identifies any gaps.\n\n---\n**Figure 1 baseline**\n\nThe baseline is an 8M model using the same transformer architecture, trained on 2.2M data. We will clarify this in the final version.\n\n---\n**Closed-loop driving information**\n\nWe use DriveGPT to drive a car in real time, taking input features from a perception system that provides agent states and map information. While the full system incorporates additional components to handle long-tail events, we showcase challenging scenarios in the supplementary video where DriveGPT alone is responsible for driving, demonstrating its effectiveness.\n\nWe used the 8M model, trained on the full dataset, to drive the car, achieving a latency of under 50ms. While training the model requires 16 H100 GPUs with a batch size of 2048 (see Page 11, Line 559), real-time inference only requires a batch size of 1 and can run on a single onboard GPU.\n\nWe will include all requested details in the final version and are happy to provide any additional information if the reviewer identifies any gaps.\n\n---\n**Wayformer metrics**\n\nThe Wayformer results reported in Table 4 of our paper are sourced from Table 1 in “Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose Encoding” (Zhang et al., NeurIPS 2023). In that paper, the authors attributed the Wayformer results to [Nayakanti et al., 2023] and explicitly labeled them as ensemble in the table caption.\n\nIn [Nayakanti et al., 2023], the authors stated in Section 5.4: “We further apply ensembling, a standard practice for producing SOTA results for leaderboard submissions.” The numbers reported in Table 1 (last but second row, *LQ + Multi-Axis*) of [Nayakanti et al., 2023] match those in Table 1 of [Zhang et al., 2023], confirming consistency.\n\nTo prevent confusion with [Ettinger et al., 2024], we will rename \"Wayformer Ensemble\" to \"Wayformer\" and clarify the source of the baseline metrics in the final version. We will also include additional discussion on [Ettinger et al., 2024] in the related works section.\n\n---\nThank you once again for your constructive comments. We hope that we have addressed your questions and that you will consider supporting the acceptance of our manuscript.", "claims_and_evidence": "- Present DriveGPT (fulfilled with the paper)\n- Determine empirical driving scaling laws for auto-regressive behavior models. \n\nDetailed investigations with fixed or variable FLOP budgets, parameter and dataset sizes and comparison with baselines are given, e.g. in Figure 5 and 6.\n\n- Validate in real-world scenarios and closed-loop driving\n\nThis is shown in one video and described in one small paragraph. While interesting this seems under-reported.\n\n- Outperform SOTA on WOMD\n\nThe model seems to be among the best, with very good minADE and minFDE values. Miss rate and Soft mAP are better for other approaches which the authors report themselves.", "methods_and_evaluation_criteria": "The WOMD test set is used widely in the domain and is suitable for comparing marginal prediction as done with this model.", "theoretical_claims": "Claims are empirical and not theoretical.", "experimental_designs_or_analyses": "The method has a very simple straightforward design which is not hard to understand. The experimental design is common in the domain where approaches are compared on BEV views in terms of standardized metrics. There are no general issue apart from missing clarity of the closed-loop setting (more below).", "relation_to_broader_scientific_literature": "The paper beats the state of the art but does not really try to improve on a methodological level a specific architectural trick or additional input modalities. The work positions itself well within conservative standards regarding the method but then investigates scalability using resources which are not easily available to others. The contribution to the literature is therefore probably biggest in terms of the study on scalability.", "essential_references_not_discussed": "Some baselines are succeeded by newer approaches, e.g. MTR now competes in the Waymo challenge as MTR++. \"MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Querying\", Shi et al. (2024 arXiv). However, while the approaches are compared on the Waymo leaderboard the relevant paper is not yet peer reviewed and published. So in this fast moving field there are some references not discussed but for this study it is probably justified to compare to the field at a decently recent point in time as the authors did.", "other_strengths_and_weaknesses": "Strengths:\n\nThe approach seems to assign a significant probability to alternative paths which is a good thing. Comparable approaches can suffer from mode collapse, predict only one solution and then struggle when the driving situation changes quickly or in an unexpected way. This is well illustrated in Figure 11.\n\nThe architecture is actually very simple. For an investigation into scaling laws this is positive to make it easier to attribute any performance gains.\n\nThe range of investigations, including those in the appendix, are interesting for the domain because they are hard to reproduce. Given the large training set and amount of GPUs used the contribution here lies also in providing a study that a group with less resources could not do.\n\nWeaknesses:\n\nResults on the WOMD test set do show a better minADE and minFDE rate but do not outperform MotionLM Ensemble on the Miss Rate or Soft mAP. Given the high amount of parameters and training examples, this would suggest the approach leads to diminishing returns. This is further suggested by Table 8 in the appendix. \n\nThe performance can not be reproduced or built upon from the very limited information about the \"Large-scale driving dataset\". The least amount of information which would be needed would be countries in which the recording took place, percentage of rural vs. city driving, percentage of daytime vs. nighttime driving and ideally some information about the data recording. It was shown in \"A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving\" Feng et al. (ITSC 2020) that performance across datasets can vary a lot even if they were recorded in the same country. Without camera input that effect will be smaller. However, to maximize the usefulness of the insights of this study more information should be given to be able to judge the amount of outliers, proportion of cyclists and humans, their customs of crossing the street and other regional features that can impact some methods more than others.\n\nIn Figure 1, it is not clear what the smaller baseline is. Information should be given about the proportion of training data used and architecture. Is it the same transformer simply trained on less data? If yes, on how much less?\n\nThere should be more information about the closed-loop driving. Presumably the system drove on real streets with a safety driver but is DriveGPT really used to drive the car? Detection of traffic lights, complex local rules which are potentially only in place during certain hours and ad-hoc instructions by traffic guiding police officers would be surely an issue. It is not clear which of the models was used on the street as well. Training happened on 16 H100 GPUs, how was driving realized? If inference needs the same amount of compute, was that realized in a car or remote? If in a car, was there some distillation involved? The information about this contribution are lacking a lot of details.\n\nIn summary, the paper seems to shows a moderately better performance, beating the state of the art incrementally. While there is no large methodological novelty the study is a valuable investigation. In parallel to scaling large language models, scaling driving performance is hard to estimate without large computational resources and datasets, which the authors both provided. Even though it seems the conclusion must be that magnitudes of more data and FLOPs leads soon to diminishing returns, that is an insight that is potentially very interesting for the community.", "other_comments_or_suggestions": "The Wayformer Ensemble metrics do not seem to fit to the Nayakanti et al. 2023 paper (which does not seem to be the Ensemble paper), the WOMD leaderboard of the challenge or the newer paper which should be the Wayformer Ensemble paper, \"Scaling Motion Forecasting Models with Ensemble Distillation\", Ettinger et al. (ICRA 2024). The results seem closest to the Nayakanti paper but this refers to older results from 2021. The authors should check the metrics (which are approximately correct not but 100%) and either correct or explain more specifically where they are coming from. There may be some confusion of what the Ensemble paper is.", "questions_for_authors": "The authors should remark on comments where something in the paper is lacking or missing.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZDwFLt0LoR", "reviewer": "Reviewer_Uoxf", "summary": "This paper presents DriveGPT, a scalable behavior model for autonomous driving. The model has 1.4B parameters and 120M data are trained. DriveGPT is ∼3x larger and is trained on ∼50x more data sequences than existing published behavior models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your time and feedback. We appreciate the positive assessment of our well-written paper, good experiment results, and contribution to exploring the scaling law of the motion prediction large models.\n\nWe address your comments and questions below.\n\n---\n**WOMD top-performing methods**\n\nWe appreciate the reviewer for highlighting these methods that achieve top performance on WOMD.\n\nMGTR leverages an augmented WOMD-LiDAR dataset that incorporates additional LiDAR inputs, which are not available in the standard WOMD dataset used by our method and most existing baselines in the literature (e.g., MTR, Wayformer, MotionLM). While we adhere to the standard WOMD dataset and use the open-sourced MTR encoder implementations for better reproducibility, our approach does not make any specific assumptions about encoder design. We believe our insights could be extended to additional input modalities and encoders in future work.\n\nControlMTR introduces a set of novel techniques to improve upon MTR. As shown in Table 4 of our paper and Table II in ControlMTR [B], our method outperforms ControlMTR in terms of minADE, minFDE, and MR.\n\nWe will ensure that discussions on these two methods, as well as additional relevant papers from the WOMD leaderboard, are included in the final version.\n\n---\n**Table 4 baselines**\n\nWe appreciate the reviewer’s comments on our WOMD baselines. While most of our baselines were published in 2023, Wayformer remains a high-ranked method on the WOMD 2024 leaderboard among all published and preprint papers.\n\nAmong all 30 methods on the WOMD 2024 leaderboard (https://waymo.com/open/challenges/2024/motion-prediction/), our minADE of 0.5240 and minFDE of 1.0538 are the second best, trailing only IMPACT_e2e, which was submitted in March 2025 (after our paper submission) and has no publication or preprint record as of today.\n\nIn the final version, we will include results from recently published papers and preprints, including [A] and [B] as referenced by the reviewer.\n\n---\n**Soft mAP clarification**\n\nWe discussed our soft mAP results in Section 5.2.3 (Page 8, Line 416). This limitation arises from using an autoregressive decoder to estimate sample weights, which are computed by accumulating probability estimates at each prediction step, as shown in Eq. (1).\n\nMore specifically, in WOMD, the model predicts 80 steps into the future to generate 8-second trajectory samples. Summing log probabilities over these steps can introduce compounding noise, leading to suboptimal probability estimates for each sample and lower soft mAP scores, which depend on accurate probability estimates across predicted samples.\n\nWhile we follow the standard approach from the LLM literature for computing autoregressive sample weights, a potential venue for future work is to train an additional probability prediction head for each sample, which could enhance probability estimates and lead to improved soft mAP scores. We will further clarify this limitation with additional details in the final version.\n\n---\n**Clarification of contribution**\n\nAs we intend to demonstrate the generalizability of our method through WOMD experiments, the primary contribution of this work lies in providing a unique perspective to the community through our empirical scaling results, as acknowledged by the reviewer (“This paper explores the scaling law of the motion prediction large models”) and other reviewers (Reviewer *mYMk*: “The architecture is actually very simple. For an investigation into scaling laws this is positive to make it easier to attribute any performance gains”; “scaling driving performance is hard to estimate without large computational resources and datasets, which the authors both provided... that is an insight that is potentially very interesting for the community”. Reviewer *i1gv*: “this paper explores the effects of scaling in behavior modeling, which is highly valuable for the future development of autonomous driving”).\n\nWe hope our findings offer insightful contributions to the community and inspire further research in this direction.\n\n---\nThank you again for your valuable feedback. We hope our responses sufficiently address your concerns and clarify the contributions and impact of our work. We would greatly appreciate your support in allowing us to share what we believe are meaningful insights with a broader audience and contribute to the advancement of next-generation foundation models for autonomous driving.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "NA.", "experimental_designs_or_analyses": "Yes. Good experiment results of scaling law of motion prediction models.", "relation_to_broader_scientific_literature": "Scaled up the model and dataset.", "essential_references_not_discussed": "Yes. I browsed the WOMD leaderboard and noticed that some of the top-performing methods haven't been discussed, like [A] and [B].\n\n[A] MGTR: Multi-Granular Transformer for Motion Prediction with LiDAR\n\n[B] ControlMTR: Control-Guided Motion Transformer with Scene-Compliant Intention Points for Feasible Motion Prediction", "other_strengths_and_weaknesses": "Strengths:\n- The paper is well-written and easy to follow.\n- This paper explores the scaling law of the motion prediction large models.\n\nWeaknesses:\n- The main concerns are in the experiments. (1) The motion prediction results on WOMD (Table 4) only include works before 2023. The shown methods are not leading-edge enough at present. (2) The results are much lower considering the Soft mAP metric. The authors claim this is \"due to suboptimal probability estimates\". However, I'm not familiar enough with this task to fully understand their explanation.", "other_comments_or_suggestions": "NA.", "questions_for_authors": "Please refer to the concerns listed above, especilly for the comparisons with other methods.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P8RcwUnx4T", "reviewer": "Reviewer_i1gv", "summary": "This paper explores behavior modeling for autonomous driving and investigates the scaling properties from data to model parameters. The proposed method, DriveGPT, validates the benefits of scaling up both training data and compute, demonstrating improved model scalability as data increases—consistent with findings in language model scaling. To assess effectiveness, quantitative and qualitative comparisons are conducted across models from the scaling experiments. Furthermore, real-world deployment is showcased through closed-loop driving in challenging conditions, demonstrating the model's generalizability on the Waymo Open Motion Dataset, where it outperforms previous state-of-the-art methods in motion prediction.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your review and positive comments about our work. We are encouraged by your recognition of the value of our work for the future development of autonomous driving.\n\nWe address your comments and questions below.\n\n---\n**Larger models beyond 94M**\n\nWe agree with the reviewer’s observation that geometric metrics stabilize beyond 94M in Table 3, yet we have found that semantic metrics, such as collision rate, continue to improve as the model scales to 163M, as shown in the last column of Table 3 and further discussed in Section 5.1.2 (Page 7, Line 355). These results suggest that increasing model parameters could yield additional benefits in actual driving performance. In the final version, we will provide more qualitative comparisons between the 94M and 163M models in terms of semantic driving performance.\n\n---\n**Scaling gains compared to LLMs**\n\nWe appreciate the reviewer’s observation regarding the scale of our models compared to LLMs, which are typically trained on trillions of tokens with billions of parameters. A key challenge in scaling driving models is the collection of large-scale driving datasets. Unlike text data, which is abundant and easily accessible, high-quality driving data is expensive to acquire, requiring extensive real-world deployment across diverse scenarios. As discussed in Section 1 (Page 1, Line 043), this makes scaling driving models inherently more challenging than scaling LLMs.\n\nDespite being relatively smaller in scale compared to LLMs, our work represents the largest effort in scaling driving behavior models to date. As reviewer *mYMk* noted, “Even though it seems the conclusion must be that magnitudes of more data and FLOPs leads soon to diminishing returns, that is an insight that is potentially very interesting for the community.” We hope our findings provide meaningful insights to the community and contribute to the advancement of the next generation of foundation models for autonomous driving.\n\n---\n**Image-based behavior modeling literature**\n\nThanks for highlighting these methods. We will include discussions on DrivingGPT and DrivingWorld in the related works section of the final version. As discussed in the paper, our approach focuses on a simple and scalable autoregressive model architecture, leveraging commonly adopted vector representations in the field. We believe our insights could be extended to additional input representations and encoders in future work.\n\n---\n**Data release**\n\nWe are highly interested in releasing a subset of our internal driving data and providing additional metadata information to facilitate further exploration in the field, which is currently pending internal review.\n\n---\n**Data quality**\n\nWe carefully curated our dataset to encompass a diverse range of urban driving scenarios with balanced distributions, including lane changes, intersections, double-parked vehicles, construction zones, and close interactions with pedestrians and cyclists. We believe that further improving data quality and sample diversity could enhance scaling results and defer a more comprehensive study as future work. We will add more details in the final version.\n\n---\n**Validation set**\n\nThe validation set was curated to include 10M samples sharing the same distribution as the training set but with no overlap. We used the same validation set for all scaling experiments for consistency. We will add clarifications in the final version.\n\n---\nThank you once again for your time and thoughtful feedback. We hope that we have addressed your questions and that you will consider supporting the acceptance of our manuscript.", "claims_and_evidence": "This paper claims similar scaling properties in behavior modeling, supported by experiments. However, although scaling effects are observed in validation loss, the model's scaling effect is not significant. From Table 3, it appears that there is little improvement in performance beyond 94M.", "methods_and_evaluation_criteria": "This paper explores scaling properties using the same paradigm as LLMs. From the validation loss and the internal and WOMD evaluation metrics, the approach appears generally reasonable.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "The overall analysis of the experiments is fairly thorough, with testing conducted on both internal experiments and WOMD. However, the results suggest that models around 100M perform the best. How do larger models perform? The scaling gains of the model appear to be not very significant, especially when compared to LLMs, where models with hundreds of millions of parameters are still considered relatively small.", "relation_to_broader_scientific_literature": "This paper aims to advance research in behavior modeling for autonomous vehicles. Previous work has mainly explored small datasets and models, while this paper explores the effects of scaling in behavior modeling, which is highly valuable for the future development of autonomous driving.", "essential_references_not_discussed": "There is another line of methods, such as [1] and [2], that approach behavior modeling from an image modeling perspective and also use transformers for autoregressive action prediction. These methods can be briefly discussed in the related works section.\n\n[1] DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers\n\n[2] DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT", "other_strengths_and_weaknesses": "1. Is it possible to release some of the internal driving data in the future to facilitate further exploration?", "other_comments_or_suggestions": "1. Is the quality of the data also important, such as trajectory mining in complex scenarios, rather than just scaling the data volume?", "questions_for_authors": "1. How is the validation set divided?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received three Weak Accept recommendations, with all reviewers expressing support for its publication. After carefully considering their assessments, the AC agrees with the positive consensus and recommends acceptance.

While the paper does not introduce major methodological innovations, all reviewers agree that the paper presents a valuable empirical investigation. The proposed approach shows moderate but consistent improvements over the state of the art. Importantly, the authors back their claims with substantial computational resources and large-scale datasets—critical components for evaluating performance in the context of autonomous driving. Although the findings suggest that increasing data and compute leads to diminishing returns, this is a meaningful insight for the community, especially in an era where scaling laws are often assumed to yield proportional gains. Overall, the study offers a relevant contribution and is likely to be of interest to the ICML audience.
Decision: Accept (poster)