TrajTok: What makes for a good trajectory tokenizer in behavior generation?
Authors: Zhiyuan Zhang
Xiaosong Jia
Guanyu Chen
Qifeng Li
Zuxuan Wu
Yu-Gang Jiang
Junchi Yan
Venue: ICLR 2026 Poster (2026)
Keywords: behavior generation
tokenizer
autonomous driving
Primary area: applications to robotics, autonomy, planning

Abstract: Behavior generation in autonomous driving aims to simulate dynamic driving scenarios from recorded driving logs. A popular approach is to apply next-token-prediction with discrete trajectory tokenization. In this work, we explore what makes a good trajectory tokenizer from the perspective of logged data usage. We first analyze the four properties (coverage, utilization, symmetry and robustness) of vocabularies of data-driven and rule-based trajectory tokenizers and their impact on performance and generalization. Data-driven tokenizers often build vocabularies with better utilization but suffer from insufficient coverage and sensitivity to noise, while rule-based methods have better coverage but contain too many useless tokens. With these insights, we propose TrajTok, a trajectory tokenizer that combines the two methods with rule-based vocabulary candidate setup and data-driven filtering and selection processes. The tokenizer has balanced coverage and utilization as well as good symmetry and robustness. Furthermore, we propose a spatial-aware label smoothing method for the cross-entropy loss to better model the similarities between the trajectory tokens. Our method wins first place in the 2025 Waymo Open Sim Agents Challenge.

Published as a conference paper at ICLR 2026
TRAJTOK: WHAT MAKES FOR A GOOD TRAJECTORY TOKENIZER IN BEHAVIOR GENERATION?
Zhiyuan Zhang', Xiaosong Jia Guanyu Chen', Qifeng Lii,
Zuxuan W Yu-Gang Jiang?., Junchi Yan
Sch. of Artificial Intelligence & Sch. of Computer Science, Shanghai Jiao Tong University
2. Institute of Trustworthy Embodied AI, Fudan University
Shanghai Key Laboratory of Multimodal Embodied Al
*Correspondence authors
ABSTRACT
Behavior generation in autonomous driving aims to simulate dynamic driving sce- narios from recorded driving logs. A popular approach is apply next-token- prediction with discrete trajectory tokenization. In this work, we explore what makes good trajectory tokenizer from the perspective of logged data usage. We first analyze the four properties (coverage, utilization, symmetry and robustness) of vocabularies of data-driven and rule-based trajectory tokenizers and their im- pact on performance and generalization. Data-driven tokenizers often build vocab- ularies with better utilization but suffer from insufficient coverage and sensitivity to noise, while mulebbssed methods have better coverage but contain too many use- less tokens. With these insights, we propose TrajTok, a trajectory tokenizer that combines the two methods with rule-based vocabulary candidate setup and data- driven filtering and selection processes. The tokenizer has balanced coverage and utilization well good symmetry and robustness. Furthermore, spatial-aware label smoothing method for the cross-entropy loss to better model the similarities between the trajectory tokens. Our method wins first place in the 2025 Waymo Open Sim Agents Challenge.
propose
INTRODUCTION
NTP Behavior Generation
Tokenization Methods
logged trajectory
(a) Data Driven
(b) Rule Based
Rule Based Data Driven
Properties
Coverage
Utilization
Symmetry
Robustness
Figure 1: Comparison of tokenizers with different logged data usage in NTP behavior genera- tion model. Combining data-driven and rule-based method, the proposed TrajTok balances coverage and utilization while has well symmetry and robustness.
Behavior generation aims to simulate realistic and dynamic chains of actions using logged data. In autonomous driving, behavior generation is key component in simulators for data collection and evaluation. It typically takes the historical trajectories of agents and environmental information as inputs and generates future multi-agent trajectories auto-regressively (Chen al. 2024).
This work was supported by Scientific Research Innovation Capability Support Project for Young Faculty (U40) of the Ministry of Education of China, SRCCSPYF-ZY202500) This work also was in part supported by the Science and Technology Commission of Shanghai Municipality (No. 24511 1 103 1100) and the New Cor- nerstone Science Foundation through the XPLORER PRIZE.

---

Published as a conference paper at ICLR 2026
2020 Touvron et al. 2023 Yang et al.] 2023), series of behavior generation models adopt the next- next-token sokt-token-pree wowee-peedccttinnn (NTP) paradigm (Philion et al. 2023 Seff et al. 2023 Wu et al. 2024 Zhao et al. 2024). as shown in Fig. (upper left). These models build a vocabulary. a finite set of trajectories that tries to model the continuous distribution space with trajectory tokenizers. With the trajectory vocabulary, these models transform the complex trajectory distribution in continuous space into simpler discrete space for easier multi-modal behavior handling and model fitting.
Recently, inspired by large language models (LLMs) (Floridi & Chiriatti
Due to the low-dimensional nature of trajectories, various methods can be applied for trajectory tokenization, including data-driven methods such VQ-VAE (Van Den Oord al. (017). K- means (Arthur & Vassilvitskii (006)) and K-disks (Philion et al.] (002)), and rule-based methods such as gridding (Sef' et al. 2023). However, which method and what properties lead to a good trajectory tokenizer remain unclear to the community.
In this paper, we study what makes good trajectory tokenizer and how to use the logged data in the tokenization process. First, we analyze the properties of trajectory tokens, including coverage, utilization, symmetry, and robustness, as well as their impact on the performance and generalization of NTP behavior generation models. As shown in Fig- (a), we find that data-driven tokenizers tend to produce vocabularies within the recorded distribution, resulting in high utilization but low cover- age. Additionally, their sensitivity noisy data results in erroneous vocabulary and the asymmetry affects generalization. In contrast, rule-based methods cover a wider region with good symmetry and robustness, which leads to better generalization. However, they rely solely human priors without adapting to the real data distribution, resulting in vocabularies that contain many redundant trajectories that do appear in reality and low utilization, as shown in Fig. (b). Under the same vocabulary size, their performance is far inferior to data-driven methods.
Based on these analyses, we propose TrajTok, tokenizer specifically designed for trajectories that combines data-driven and rule-based methods, as shown in Fig. (c). It first sets up primary vo- cabulary candidate with rules, then applies data-driven filtering and expansion. The former ensures the stability and symmetry of the vocabulary, while the latter balances coverage and utilization.
Furthermore, we study loss designs related vocabularies. few NTP models use cross-entropy loss with label smoothing (Szegedy et al.] 2016) to reduce overfitting and improve generalization. The standard label smoothing mechanism assigns the same weight to -on-ground-truth tokens, ignoring their similarity to ground-truth ones. As the similarity of trajectories is closely related to spatial distance, we propose a spatial-award -watre label smoothing method that assigns different weights to tokens conditioned on their distances to the ground-truth one. This smoothing method improves the performance of NTP behavior generation models effectively.
Our method wins first place in the Waymo Open Sim Agent Challenge (WOSAC) (Montali al. 2023) 2025 with a Realism Meta of 0.7852, with good generalization across datasets.
Our contributions are threefold:
We conduct a detailed investigation of trajectory tokenization in NTP behavior generation from the perspective of logged data usage and analyze the coverage, utilization, symmetry, and robustness of data-driven and rule-based methods.
We propose TrajTok, a plug-and-play trajectory tokenizer that combines rule-based and data-driven methods, which achieves state-of- of-te--o--he-rrtt performance on the Waymo Open Sim Agent Challenge.
We explore token-related loss design in NTP behavior generation models and propose spatial-aware label smoothing method for the cross-entropy loss.
2
RELATED WORK
2. BEHAVIOR GENERATION MODELS
Behavior generation modules key components in simulators and benchmarks (Jia al. 2024b
You et al. 2024 Zhou et al. 2025 Jia et al.) 202T) to evaluate the ability of autonomous driving
models (Hu et al. 2023 Lu et al. 2024 Jia et al. 2025a 2023d:
Wu et al.] 2022) and serve as
2

---

Published as a conference paper at ICLR 2026
environments for reinforcement learning (Li et al. 2025 Jia et al. 2023b Yang et al. 2025c). The task proposed by Waymo Open Sim Agent Challenge (WOSAC) (Montali et al. 2023 aims to gen- erate next states (including position and heading) of agents step by step realistically conditioned on historical steps and environmental information. The traditional encoder-decoder architectures (Shi et al. 2022 (Zhou al. 2023 Jia et al. 2023c (202)) in motion prediction can be used in this task, but they suffer from low data utilization and significant out-of-distribution (OOD) problems in the auto-regressive generation process (Zhou al. 2024 Jía al. (202aa). Inspired by large language models (LLM) and vision- langun--anuugee-ea language-action (VLA) Models Yang et al. al.]| 2025b Fan et al. 2025 most recent works adapt the next-token- poeeen-pediition (NTP) paradigm (Jia et al., 2024a). Trajeglish (Philion et al. 2023) first models the behavior generation task as discrete NTP with data-driven tokenizer K-disks. SMART (Wu et al., 2024) introduces a map discrete tokenizer and further analyzes the scalability of the architecture, while KiGRAS (Zhao et al. 2024) factorizes the driving scene in action space with kinematic transformations. Furthermore, CATK (Zhang et al. 2025) introduces a closed-loop fine-tuning strategy to further address the OOD problem.
Despite NTP-based models, diffusion models are another approach for behavior generation. These models focus on generating controllable (Zhong et al.) 2022]] Huang al. 2024; Jiang et al. [024
Pronovost et al. 2023] Yang et al. 2025a) or adversarial (Xie et al. [2024] Yin et al.) [2024] Chang
al. 2024 Xu al. 2025) behaviors with guidance, while their realism is much lower than NTP-
based models on WOSAC.
2.2 TRAJECTORY TOKENIZERS FOR BEHAVIOR GENERATION
Tokenizers are widely used in deep-learning, such as getting image features in perception tasks Li
al. 2023 Jia al. 2020 Wu et al. 2023 Gao al. 2020 Zhu al., 2025 Jia al. 2025b)
In behavior generation, trajectory tokenizers transform the states of agents from continuous space to discrete space. Various tokenizers are adopted in discrete NTP behavior generation models, in- cluding VQ-VAE, K-means, K-disks and naive grid-based methods. Most of them are long-existing general approaches and only the K-disk proposed by Trajeglish (Philion al. 2023) is specifically designed for trajectory tokens. K-disks randomly select a state in logged data trajectory token and exclude all logged states within a certain distance from the selected states, then randomly select the next token in the remaining data. Therefore, it is more like a sampling algorithm rather than clustering algorithm. Grid-based methods such as that used in MotionLM (Seff et al. 2023) simply select the token with uniform distribution in a pre-defined region of cach state component.
Varying from deep-learning methods, clustering or sampling algorithms and rule-based methods, these tokenizers rely on logged trajectory data at different levels. However, there is a lack of compre- hensive analysis of these tokenizers and the data usage preference behind them. Trajeglish (Philion) et al. 2023) compares some tokenizers only in terms of discretization error, without studying their impacts on the performance of behavior generation models.
3 METHODS
3.1 PROBLEM FORMULATION
The behavior generation task can be defined as follows: Given an initial scene, including the HD) map M and the past Tn states of all agents {S-Th, So), the goal is generate the states of all agents at each future time step within T' steps, i.e., (SI, (S,,,,.,TT,hh
We focus on the discrete NTP behavior generation models that output trajectories as actions, such as SMART (Wu et al. 2024) and Trajeglish (Philion et al. 2023). We denote the model as Na. with trainable parameters / and trajectory vocabulary V C, C2 (..
(p, C Each trajectory token G R Lx3 consists of L points with (x, y. yaw) in the agent-centric coordinate. We denote the number of all agents N and the model outputs the trajectory tokens a, E RR-xXLx3 for each agent in the interval L as:
= (A,x,,
(1)
where E (0, 1, (Tr//))} is the end timestamp for each interval. The output trajectory token for each agent is selected from the vocabulary, i.c. at E V. Then, a coordinate transform f is applied to calculate states in the next interval:
3

---

Published as a conference paper at ICLR 2026
Raw trajectory endpoints in dataset
Step 1. Data preparation and flipping
Step 2. Vocabulary candidate preliminary selection
Step 3. Filtering and expanding
Step 4. Final generation
Generated vocabulary
Figure 2: The vocabulary generation process of TrajTok. The blue points are endpoints of logged trajectories and the yellow grids selected in each process.
(Lx1)) S(),,
The tokenizer Tok() generates the vocabulary V. Pure rule-based tokenizers, only use predefined hyper-parameters Ozak. as the vocabulary size and sample range as input, i.e.
based Tok(f
(3)
Data-driven tokenizers generate the vocabulary with dataset DD i.e.
Vastadriven Tok(Orok: D)
3.2 TRAJTOK
As Fig. shows, TrajTok generates the trajectory vocabulary through the following steps:
Step 1: Data preparation and flipping. First, all valid trajectories of length L in the dataset are extracted and normalized the agent-centric coordinate system. We denote the normalized trajectories as D E RN where Np is the number of trajectories. Then, the trajectories are flipped along the x-axis and concatenated with original trajectories, i.e.
D = Concatenate D, Flip(())
(5)
The flipping operation ensures the symmetry of trajectories, which results in symmetric tokens along the x-axis.
Step 2: Rule-based vocabulary candidate setup and preliminary selection. The vocabulary
candidate is set up with a grid. Given ranges Tmins Tmax. Ymin- Vmux and intervals Tinteryal- Vintevall- the size of the grid is (H, W) = (/mm [9mms-Bhe -Shmn Shmn -um==miaa Each trajectory in flipped data D is associated Miueral Finecal
with a cell if its endpoint falls within the cell. We denote Dey M Lx3 as all trajectories associated with cell (f,j) and Ni as the number of these trajectories for each grid. For preliminary selection, a grid is marked as valid if N is larger than threshold 8p. The validity binary map B is set through
By 1 (N 2+0]
(6)
Step 3: Data-driven filtering and expanding. The preliminary selected grids are within the dis- tribution of logged data, and are sensitive noise, shown in Fig To cover more possible trajectories in reality and improve robustness, filtering and expanding are then applied. For grid (i,j) the map, compute the number of its selected neighbors within the distance of k, i.e.
Nob DI İitk En Bun
4

---

Published as a conference paper at ICLR 2026
Then we add an unselected grid from step 2 if the number of its selected neighbors N is larger than the threshold Sa, and remove selected grid if Nij is less than the threshold 8, through
Big = =100 if
111
Bu = and Bi and
N 80 NYb S 84
(8)
Step 4: Final generation. Finally, the trajectory vocabulary is generated for each selected grid. The average of logged data trajectories is used if there are trajectories associated with the cell.
m= Dism N
===
| iE [E 1,
B.j = 1 No 0}
(9)
If not, the vocabulary is generated with curve interpolation from the origin point to the center of the grid Pij. This happens the cell valid in the expanding process. The yaw of the endpoint rij is estimated from the trajectories in nearby grids.
222 = [Curve_Interp[] Pij, Piy,
I E (E B. = =1 I = 0} (10)
The final vocabulary is:
VirafTok = V1V
(11)
3.3 SPATIAL-AWARE LABEL SMOOTHING
NTP models often use cross-entropy loss with label smoothing (Szegedy et al,, 2016) to alleviate overfitting and improve generalization. Standard label smoothing assigns the same probability to each non-ground-trutt label. Denoting the index of the ground-truth label as j and the parameter of label smoothing as the target probability y for each label is calculated
1- if i= M
y:
if itj
(12)
Under this mechanism, the loss is the same for predicting trajectory token from the ground truth and one next to the ground truth. However, the former impacts the performance of behavior gener- ation more significantly. Thus, we hope the model can be more tolerant of tokens that are spatially close to the ground-truth, while rejecting tokens that are away. We calculate the average er- between each token trajectory and the fround-truth trajectory, and assign the target probability inversely proportional to the square of the error:
1
h;
(13)
|c: 12
if i=
y/
itj
(14)
0. km
This spatial-aware label smoothing can both reduce errors and improve generalization.
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
Dataset. We conduct experiments the Waymo Open Motion Dataset (WOMD) (Ettinger et al. 2021). There are 486, 008.999//44097744999999 scenarios in the training/vllidtiionntess set. Each scenario contains 9 seconds of agent states with an interval of 0.1s and high- high-definition map. The behavior generation task is to generate the states in the future seconds of up to 128 agents with states in the past second in autoregressive manner.
Metrics. We use the metrics in the Waymo Open Sim Agents Challenge (Montali et al. 2023 ) The main metric is the Realism Meta, which is composed of the Kinematic, Interactive, and Map-based
5

---

Published as a conference paper at ICLR 2026
Table Waymo Open Sim Agents Challenge leaderboard 2025. Top 15 entries in the Submission highlights the best metric and
Period are presented.
highlights the second best.
Method
Realism Meta
Kinematicf
Interactive
Map-based >
minADE 1
SMART-RI
0.7855
0.4940
0.8109
0.9194
1,2990
TrajTok (Ours)
0.7852
0.4887
0.8116
0,9207
1.3179
unimotion
0,7851
0,4943
0.8105
0.9187
1,3036
SMART-dny-CLSPF (Zhang et al. 2025
0,7846
0.4931
0.8106
0.9177
1,3065
AMART-iny-RLFTSS
0.7844
0,4893
0.8128
0,9164
1,3470
comBOT
0.7837
0,4899
0.8 0.8102
0.9175
1,3687
AgentFormer
0,7836
0.4906
0.8103
0,9167
1.3422
UniMM [Lin et 2025
0,7829
0,4914
0,8089
0,9161
1,2949
R1Sim
0.7827
0.4894
0.8
0.9147
1,3593
SimFormer
0.7820
0.4920
0.8060
0,9167
1,3221
SMART-iny-RLFT
0.7815
0.8107
0,9133
1opR33
0,7814
0.4854
0,8089
0,9153
1,3931
SMART-tnn-RLFT
0.7780
0.4799
0.8070
0.9109
1.6388
11m2ad
0.7779
0.4846
0.8048
0.9109
1.2827
UniTrormer
0.7776
0.4892
0.7997
0.9140
1.3592
Table Performance SMART model with different tokenizers on validation split of WOMD.
The models trained a random 20% subset of the training set and the metrics of WOSAC 2024 version. The tokenizer of original SMART model K-disks.
Tokenizer
Realism Meta
Kinematict
Interactive
Map-based
minADE |
VQ-VAE
0.7596
0,4629
0.8101
0.8642
1,3982
K-means
0.7476
0,4375
0,7903
0.8635
1.4797
K-disks
0.7584
0,4602
0.8004
0.8748
1,3532
Grid
0.7527
0,4121
0.8099
0.8737
1.4137
TrajTok
0.7702
0.4867
0.8132
0.8769
1.3428
Table 3: Performance of K-disks and TrajTok Table 4: Performance of K-disks and TrajTok with different sizes of logged data.
across datasets.
Tokenizer
Logged Dataset
Realism Meta f
minADE
Tokenizer
Logged Data Size
Realism Meta f
minADE
K-disks K-disks
Waymo nuScenes
0.7584
1.3537
K-disks
107
0,7584
1.3537
0.7350 (-0.0234)
1 1.0744
K-disks
10
0.7539 (-0.0045)
1.3628
TrojTok
Waymo
0.7702
1.3428
K-disks
10
0,7442 (-0.0142)
1,3696
TraTok
nuScenes
0.7641 (-0.0061)
1.3681
Truj Tok
107
0,7702
1,3428
Trj
10
0.7691 (-0.0011)
1.3445
Toy
10a
0.7675 (-0.0027)
1.3511
Notably, it takes several days to compute the metries on the whole validation set with the metries.
official code. Previous works (Zhang et al. (225) conduct ablation studies with evaluation on computation
small subset, which may lead biased results. We improve the code and decrease time to less than 2 hours while strictly aligning the protecol and results with the official version. With the efficient evaluation tool, all our ablation studies are evaluated on the whole validation set.
4.2 RESULTS
WOSAC leaderboard. Table lists the results of the top 15 entries in the 2025 Waymo Open Sim Agents Challenge. TrajTok wins the first place in the Waymo Open Sim Agent Challenge 2025 with a Realism Meta metric of 0.7852. It also reaches state-of-the-art performance on Map-based metrics of 0.9207 and competitive performance on other metrics. Without any fine-tuning processes, our approach has comparable or superior performance compared other methods that are also developed on the SMART-tiny model but need fine-tuning, such as EMART-tiny-LLSH (Zhang al. 2025).
Comparison with other tokenizers. We compare TrajTok with other tokenizers, including VQ- VAF, K-means, K-disks and grid method on the SMART (Wu et al., (204) model, as shown in Table 2 Results show that our method reaches the best performance among all tokenizers that are adopted behavior generation models.
The generalization of the tokenizer is reflected in its ability to handle diverse real-world data with vocabularies derived from any data record. We investigate generalization of ToajTok through two experimental setups: 1 Building vocabularies with nuScenes (Caesar al.
Generalization.

---

Published as a conference paper at ICLR 2026
Gridding
5
5
10
15
K-means
-1
-2
10
15
K-disks
0-
-1
-25
TrajTok
1
-1
-25
10
15
Figure 3: Trajectory vocabularies from Tok and other tokenizers. Each colored line repre- sents trajectory token within 0.5 seconds in agent-centric coordinate system. The size of all four vocabularies is 2000.
Figure 4: Generated behavior in two scenarios.
2020) data and training and evaluating on WOMD to evaluate the tokenizer's generalization across datasets. The results are shown in Table. 3 2 Building vocabularies with small subset of WOMD evaluate the tokenizer's ability infer the overall distribution from few sampled data. The results
are shown in Table. 4
Qualitative results. Fig. shows the vehicle trajectory vocabularies generated by different tok- enizers. The data-driven methods K-means and K-disks produce trajectory tokens that are asym- metric and contain kinematically implausible noisy trajectories. The purely rule-based grid method generates many invalid trajectories (e.g., lateral shifts of over 1 meter within 0.5 seconds) and has lower density at the same vocabulary size. In contrast, TrajTok demonstrates good symmetry and stability, with generated trajectories that largely adhere kinematic principles. Fig. 4 illlssraates

---

Published as a conference paper at ICLR 2026
0.76
Realism Mera
0.75
0.74
K-disks
0.73
Kmeans
0.77
Gridding
TrajTok
500
1000
2000
4000
8000
16000
Vocabulary Size
Figure 5: Ablation of vocabulary size of vehicles.
Table 5: Ablation of Spatial-Aware label smoothing.
Tokenizer
Label Smoothing Type
Realism Metaf
minADE $
K-disks
default
0.7443
1.4230
K-disks
spatial-aware
0.7584
1.3537
TrajTok
default
0.7597
1.3797
TrajTok
spatial-aware
0.7702
1.3428
the behavior generated by TrajTok in two scenarios. The first is a busy intersection scene where traffic in direction A is queued, maintaining appropriate distances between vehicles, while vehicles in directions and C make left turns. This demonstrates TrajTok's capability to handle interactions among large number of vehicles precisely. The second scenario features atypical intersection where vehicles direction E need to turn left and backward, alternating passage with vehicles from directions D) and F. Benefiting from its superior coverage, TrajTok effectively manages vehicle in- teractions in uncommon scenarios.
4.3 ABLATION
Vocabulary Size. We ablate the vocabulary size in Fig. Increasing the vocabulary size improves the ability to represent complex distributions but may lead to model underfitting, which is also observed in LLMs (Tao al. 2024 Rule-based gridding methods require larger vocabulary size to reach optimal performance, while data-driven methods need a smaller size. The optimal vocabulary size for TrajTok lies between data-driven and rule-based methods. Additionally, TrajTok achieves the best performance across different vocabulary sizes.
Spatial-Aware label smoothing. Table presents the ablation study for Spatial-Awar Label Smoothing. Whether using K-disks or Traj Tok as tokenizers, Spatial-Aware Label Smoothing im- proves performance compared to standard label smoothing.
K-disks
K-means
0.5
0.5
-1.0
-1.0
4.5
-1.5
2
8
12
14
10
14
16
Gridding
TrajTok
0.5
0.5
0
0.5
0.5
-10
-1.5
-1.5
2
10
12
14
30
12
14
16
Figure 6: The distribution of logged trajectories and vocabularies of vehicles. Orange dots indicate the endpoints of logged trajectories, while blue dots indicate those of vocabularies.
8

---

Published as a conference paper at ICLR 2026
100
0.10
Missing Rate (%)
0.08
1113
Proportion (9g
0.06
Fos nos 9330 cons oos 536 1553 ouss 3000
0.04
1887 1607
0.02
8.00
0.02
0.04
0.06
0.08
0.10
0.00
750 1250 2000 Sorted Token ID
Distance (m)
Figure 7: The missing rate of tokenization for cyclists.
Figure 8: The frequency of vocabularies for cyclists.
Table 6: Symmetric tokenizers have better performance.
Table 7: The average discretization error of tokenizers for vehicles. The vocabulary size 2000.
Tokenizer
Realism Metaf
minADE
K-disks w/ symmetry
0.7610
1.3541
Tokenizer Realism Meta
average discretization ervor (m)
K-disks w/o symmetry
0.7584
1.3532
K-means
0.7476
0.0224
K-means w/ symmetry K-means w/o symmetry
0.7526
1.3519
K-disks
0.7584
0.0204
0.7476
1.3597
Grid
0.7527
0.0813
TrajTok
0,7702
0,0520
TrajTok w/ symmetry
0.7702 0.7670
1.3428
ToajTok w/o symmetry
1.3611
4.4 ANALYSIS AND DISCUSSIONS
Coverage. Qualitatively, Fig. plots the endpoints of real trajectories and vocabulary trajectories. Data-driven methods generate vocabularies within the distribution of real trajectories and leave sev- eral insufficiently covered regions. The vocabulary generated by rule-based methods extends far beyond the real distribution range. In contrast, the vocabulary of TrajTok covers most of the val- ued trajectories. Quantitatively, Fig. 7 shows the missing rate of tokenization at different distances, which is defined as the proportion of trajectories with discretization errors larger than a given dis- tance. TrajTok has a better missing rate at larger distances. shows that TrajTok has appropriate coverage and is better dealing with the long tail effect in tokenization.
Utilization. Fig. 8 shows the frequency of each token in the vocabulary. Rule-based gridding methods have large number tokens with zero frequency. Although the frequency of uncommon tokens of TrajTok is lower than that of data-driven methods, most of them match with several logged trajectories. With the increase of tokens for rare trajectories, there trade-off between coverage and utilization. TrajTok balances these two properties.
Symmetry. The symmetry of the possible distribution boundary of trajectories in reality comes from that of vehicle kinematic models. If a trajectory recorded, flipped counterpart is also physically drivable. Considering complex driving scenarios and road structures in reality, the flipped trajectory may possibly driven scenario that was not logged in the dataset. In the experiments, remove the flipping operation in the TrajTok generation process, resulting in performance de- cline. Data-driven methods, due to the randomness of data distribution and algorithms, can not guarantee the generation of symmetric vocabulary. We flip the trajectories below the x-axis to above the x-axis, generate a vocabulary of half size on the flipped trajectories. and then flip it back obtain the complete vocabulary. The vocabularies generated in this manner show improved per- formance, as shown in Table Both aspects of the experiments demonstrate the importance of symmetry.
Robustness. Fig. 3 and Fig. show that TrajTok is less sensitive noise. In contrast, K-disks may directly add noisy trajectories to the vocabulary during random sampling. K-means may also pick outliers since they have large distances to other samples.
Does lower average discretization error lead to a better tokenizer? Average discretization error used to evaluate the performance of a tokenizer in previous work (Philion et al., 2023). However, shown in Table 7 the average discretization error of K-disks is lower than that of TrajTok, yet

---

Published as a conference paper at ICLR 2026
its performance is inferior to TrajTok. The indicator cannot entirely reflect the distance between the vocabulary and distribution in the real world. For example, it cannot well indicate the long-tail effect in Fig. 7 When the average discretization error is already low, the performance of the tokenizer is influenced mostly by other factors.
5 CONCLUSION
In this paper, wC analyze data-driven and rule-based tokenizers from four perspectives including coverage, utilization, symmetry, and robustness. Based the analysis, we combine data-driven and rule-based methods and introduce TrajTok. Additionally, we propose a spatial-aware label smooth- ing method to better model the similarities between the trajectory tokens. Experiments demonstrate the effectiveness of our methods.
6 ETHICS STATEMENT
The research conducted in the paper conforms with the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
We describe the proposed tokenizer and loss in Sec. 3 and implementation details in Appendix A The datasets are public accessible. The code will be open-sourced for reproduction.
REFERENCES
David Arthur and Sergei Vassilvitskii. k-means++: The advantages of careful seeding. Technical report, Stanford, 2006.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 1621--11631, 2020.
Wei-Jer Chang, Francesco Pittaluga, Masayoshi Tomizuka, Wei Zhan, and Manmohan Chandraker. Safe-sim: Safety-criticall closed-loop traffic simulation with diffusion cnffusion-comtrollabl adversaries. In European Conference on Computer Vision, pp. 242-258. Springer, 2024.
Di Chen, Meixin Zhu, Hao Yang, Xuesong Wang, and Yinhai Wang. Data-driven traffic simulation:
A comprehensive review. IEEE Transactions on Intelligent Vehicles, 444):4333447488 2024.
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings the IEEE/CVF Intern-- tional Conference on Computer Vision, pp. 9710-9719, 2021.
Cunxin Fan, Xiaosong Jia, Yihang Sun, Yixiao Wang, Jianglan Wei, Ziyang Gong, Xiangyu Zhao, Masayoshi Tomizuka, Xue Yang, Junchi Yan, et al. Interleave-vla: Enhancing robot manipulation with interleaved image-text instructions, arXiv preprint arXiv:2505.02155 2025.
Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds
and machines, 30(4):681 -694, 2020.
Xiaofeng Gao, Xiaosong Jia, Chaoqi Yang, and Guihai Chen. Using survival theory in early pattern detection for viral cascades. IEEE Transactions Knowledge and Data Engineering, 34(5): 2497-2511, 2020.
Yihan Hu, Jiazhi Yang. Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tian- wei Lin, Wenhai Wang. Lewei Lu, Xiaosong Jia. Qiang Liu, Jifeng Dai. Yu Qiao, and Hongyang Li. Planning-orieenn oriented autonomous driving. Proceedings of the IEEE/CVF conference on com- puler vision and pattern recognition, pp. 17853-17866 17862,, 2023.
10

---

Published as a conference paper at ICLR 2026
Zhiyu Huang, Zixu Zhang, Ameya Vaidya, Yuxiao Chen, Chen Lv, and Jaime Fernández Fisac. Ver- satile cennisconnittent traffic scenario generation as optimization with diffusion. arXiv preprint 0220022404.02522, 3, 2024.
Xiaosong Jia, Qitian Wu, Xiaofeng Gao, and Guihai Chen. Sentimem: Attentive memory networks for sentiment classification user review. In International Conference on Database Systems for Advanced Applications, pp. 736-751. Springer, 2020.
Xiaosong Jia, Liting Sun, Masayoshi Tomizuka, and Wei Zhan. Ide-net: Interactive driving event and pattern extraction from human data. IEEE robotics and automation letters, 602):3065-3072, 2021.
Xiaosong Jia, Liting Sun, Hang Zhao, Masayoshi Tomizuka, and Wei Zhan. Multi-agent trajectory prediction by combining egocentric and allocentric views. In Conference on Robot Learning, pp- 1434-1443, PMLR, 2022.
Xiaosong Jia, Li Chen, Penghao Wu, Jia Zeng, Junchi Yan, Hongyang Li, and Yu Qiao. Towards capturing the temporal dynamics for trajectory prediction: a coarse-to-fine approach. In Confer- ence on Robot Learning, pp. 910-920. PMLR, 2023a.
Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langechuan Liu, and Hongyang Li. Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 7953-7963, 2023b.
Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. Hdgt: Heteroge- neous driving graph transformer for multi-agent trajectory prediction via scene encoding. IEEE transactions on pattern analysis and machine intelligence, 45(1 1):1 18860-13755 2023c.
Xiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie, Conghui He. Junchi Yan, and Hongyang Li. Think twice before driving: Towards scalable decoders for end-to-end autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 21983-21994, 2023d.
Xiaosong Jia, Shaoshuai Shi, Zijun Chen, Li Jiang, Wenlong Liao, Tao He, and Junchi Yan. Amp: Autoregressive motion prediction revisited with next token prediction for autonomous driving. arXiv preprint arXiv:2403.1333 2024a.
Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. In NeurIPS 2024
Datasets and Benchmarks Track, 2024b.
Xiaosong Jia, Junqi You, Zhiyuan Zhang, and Junchi Yan. Drivetransformer: Unified transformer for scalable end-to-end autonomous driving. In International Conference on Learning Represen- tations (ICLR), 2025a.
Xiaosong Jia, Chenhe Zhang, Yule Jiang, Songbur Wong, Zhiyuan Zhang, Chen Chen, Shaofeng Zhang, Xuanhe Zhou, Xue Yang. Junchi Yan, al. Spatial retrieval augmented autonomous driving. arXiv preprint arXiv:25/2.0688 2025b.
Max Jiang, Yijing Bai, Andre Commaan, Christopher Davis, Xiukun Huang, Hong Jeon, Sakshum Kulshrestha, John Lambert, Shuangyu Li, Xuanyu Zhou, et al. Scenediffuser: Efficient and con- trollable driving simulation initialization and rollout. Advances Neural Information Processing Systems, 37:55729 2024.
Hongyang Li, Chonghao Sima, Jifeng Dai, Wenhai Wang, Lewei Lu, Huijie Wang, Jia Zeng. Zhiqi Li, Jiazhi Yang, Hanming Deng, Hao Tian, Enze Xie, Jiangwei Xie, Li Chen, Tianyu Li, Yang Li, Yulu Gao, Xiaosong Jia, Si Liu, Jianping Shi, Dahua Lin, and Yu Qiao. Delving into the devils of bird's eeey'-ewe perception: A review, evaluation and recipe. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(4):2151-2170, 2023.
Qifeng Li. Xiaosong Jia, Shaobo Wang, and Junchi Yan. Think2drive: Efficient reinforcement learning by thinking with latent world model for autonomous driving (in carla-v2). In European Conference Computer Vision, pp. 142-158. Springer, 2025.
11

---

Published as a conference paper at ICLR 2026
Longzhong Lin, Xuewu Lin, Kechun Xu, Haojian Lu, Lichao Huang, Rong Xiong, and Yue Wang- Revisit mixture models for agget simulation: Experimental study within a unified frame- work, 2025. httLhhtp://arrrte wttp://g/////..11110000
Han Lu, Xiaosong Jia, Yichen Xie, Wenlong Liao, Xiaokang Yang, and Junchi Yan. Ac- tivead: Planning-oriente active learning for end-to-end autonomous driving. arXiv preprint arXiv:2403.02877, 2024.
Nico Montali, John Lambert, Paul Mougin, Alex Kuefler, Nicholas Rhinehart, Michelle Li, Cole Gulino, Tristan Emrich, Zoey Yang, Shimon Whiteson, et al. The waymo open sim agents chal- lenge. Advances in Neural Information Processing Systems, 36:59151-59171, 2023.
Jonah Philion, Xue Bin Peng, and Sanja Fidler. Trajeglish: Traffic modeling as next-token predic- tion. arXiv preprint arXiv:2312.04535, 2023.
Ethan Pronovost, Meghana Reddy Ganesina, Noureldin Hendy, Zeyu Wang. Andres Morales, Kai Wang, and Nick Roy. Scenario diffusion: Controllable driving scenario generation with diffusion. Advances in Neural Information Processing Systems, 36:68873-68894 2023.
Ari Seff, Brian Cera, Dian Chen, Mason Ng. Aurick Zhou, Nigamaa Nayakanti, Khaled S Re- faat, Rami Al-Rfou, and Benjamin Sapp. Motionlm: Multi-agent motion forecasting as language modeling. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp- 2023.
Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. Advances in Neural Information Processing Systems, 35:6531-6543, 2022.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2818-2826, 2016.
Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. Scaling laws with vocabulary: Larger models deserve larger vocabularies. Advances in Neural Information Processing Systems, 37:114147-1141791 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 1rXiv:2302.13977 2023.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory-guider control prediction for end-to-end autonomous driving: A simple yet strong baseline. Advances in Neural Information Processing Systems, 35:6119 15.6119-613,, 2022.
Penghao WU, Li Chen, Hongyang Li, Xiaosong Jia, Junchi Yan, and Yu Qiao. Policy pre-training for autonomous driving via self-supervised geometric modeling. In International Conference Learning Representations, 2023.
Wei Wu, Xiaoxin Feng, Ziyan Gao, and Yuheng Kan. Smart: Scalable multi-agent real-time motion generation via next-token prediction. Advances in Neural Information Processing Systems, 37: 144048-114077 2024.
Yuting Xie, Xianda Guo, Cong Wang. Kunhua Liu, and Long Chen. Advdiffuser: Generating adversarial safety-critical driving scenarios via guided diffusion. In 2024 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pp. 9983-9989. IEEE, 2024.
Chejian Xu, Aleksandr Petiushko, Ding Zhao, and Bo Li. Diffscene: Diffusion iifuueion-based safety-critice scenario generation for autonomous vehicles. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 8797-8805, 2025.
12

---

Published as a conference paper at ICLR 2026
Xiuyu Yang. Shuhan Tan, and Philipp Krähenbühl. Long-term traffic simulation with interleaved autoregressive motion and scenario generation. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15305-25314, 2025a.
Zhenjie Yang, Xiaosong Jia, Hongyang Li, and Junchi Yan. LIm4drive: A survey of large language models for autonomous driving, 2023.
Zhenjie Yang. Yilin Chai, Xiaosong Jia, Qifeng Li, Yuqian Shao, Xuekai Zhu, Haisheng Su, and Junchi Yan. Drivemoe: Mixture-of-experts for lasion--agguage-action model in end-to-end au- tonomous driving. arXiv preprint 21X1: 2025b.
Zhenjie Yang, Xiaosong Jía, Qifeng Li, Xue Yang, Maoqing Yao, and Junchi Yan. Raw2drive: Reinforcement learning with aligned world models for end-to-end autonomous driving (in carla v2). Advances in Neural Information Processing Systems, 2025c.
Yuan Yin, Pegah Khayatan, Éloi Zablocki, Alexandre Boulch, and Matthieu Cord. Regents: Real- world safety-criticn critical driving scenario generation made stable, In European Conference on Com- puter Vision, pp. 262-222222 276. Springer, 2024.
Junqi You, Xiaosong Jia, Zhiyuan Zhang, Yutao Zhu, and Junchi Yan. Bench2drive-r: Turning real world data into reactive closed-loop autonomous driving benchmark by generative model. arXiv preprint ar arXhv:24/2.09647 2024.
Zhejun Zhang, Peter Karkus, Maximilian Igl, Wenhao Ding, Yuxiao Chen, Boris Ivanovic, and Marco Pavone. Closed-loop supervised fine-tuning of tokenized traffic models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
Jianbo Zhao, Jiaheng Zhuang, Qibin Zhou, Taiyu Ban, Ziyao Xu, Hangning Zhou, Junhe Wang, Guoan Wang, Zhiheng Li, and Bin Li. Kigras: Kinematic-driven generative model for realistic agent simulation. IEEE Robotics and Automation Letters, 2024.
Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone. Guided conditional diffusion for controllable traffic simulation. arXiv preprint arXiv:2210.17366 2022.
Sizhuo Zhou, Xiaosong Jia, Fanrui Zhang, Junjie Li, Juyong Zhang, Yukang Feng, Jianwen Sun, Songbur Wong, Junqi You, and Junchi Yan. Lagen: Towards autoregressive lidar scene generation, 2025. URLhttpp
wat.aaxivvvv.......aaaaaaaaaa org abs 2b11/2211.21256
Zikang Zhou, Zihao Wen, Jianping Wang, Yung-Hui Li, and Yu-Kai Huang. Qenext: A next-generation framework for joint multi-agent trajectory prediction. arXiv preprint arXiv:2306. 10508, 2023.
Zikang Zhou, HU Haibo, Xinhong Chen, Jianping Wang, Nan Guan, Kui Wu, Yung-Hui Li, Yu-Kai Huang, and Chun Jason Xue. Behaviorgpt: Smart agent simulation for autonomous driving with next-patch prediction. Advances in Neural Information Processing Systems, 37:79597-79617, 2024.
Yutao Zhu, Xiaosong Jia, Xinyu Yang, and Junchi Yan. Flatfusion: Delving into details of sparse transformer-base camera-lidar fusion for autonomous driving. IEEE International Conference on Robotics and Automation (ICRA), 2025.
A IMPLEMENTATION DETAILS
Model Settings. We use the SMART-tiny (Wu et al. 2024) model as base NTP behavior generation model for experiments with TrajTok. Follow original settings, the the number of decoder layers (including Map-Agent Cross-Attention Layer, Agent Interaction Self-Attention Layer and Temporal Self-Attention Layer) is 6 and the hidden dim is 128. The interval L is set to and the re-plan frequency is 2 Hz, which means the model predict the 0.5-second trajectory at 10Hz every 0.5s. The original model build different trajectory vocabularies for each type of agents (vehicle, bicycle and pedestrian) but the same head to predict classification logits for all types. We use separate
13

---

Published as a conference paper at ICLR 2026
prediction heads instead and set their output dim the the vocabulary size each. For spatial- aware label smoothing, the total target probability of all non-ground-tre truth labels 0.1, which is the same as standard label smoothing used in original model.
Table 8:
Detailed hyper-paramedn of the submit version of TrajTok. The units for mmin+ m Finterval Vmins Umax Vinerval are all meters.
Agent Type
Tmin
a max
Tinterval
Ymin
Ymax
Vinterval
k
Sp
8
8T
Vchicle Bicycle
-5
20
0.1
-1.5
1.5
0.05
4
1
20
20
-1
8
0.05
-1
0.05
4
1
20
20
Pedestrian
-1.5
4.5
0.05
-2
2
0.05
4
1
20
20
Tokenizer. For TrajTok, we set the grid range and interval for each type of agents as Table 8 We extract trajectories that last 0.5s from the Waymo Open Motion Dataset (WOMD). In the submit version, the sizes of vocabularies for vehicle, bicycle and pedestrian are 8040, 2798, 3001 separately.
Training Details. We train the model with 8xA100 80GB GPUs for 32 epochs on the training split of the WOMD with the AdamW optimizer. The total batch size 48. The initial learning rate to 5 x 10-4 and is decayed to 5 x 10-6 based on the cosine annealing schedule.
B SENSITIVITY ANALYSIS
The sensitivity analysis on grid range, grid resolution and filtering and expansion parameters of TrajTok are shown in Table Table and Table
Table 9: Sensitivity analysis on grid range.
Vehicle
Bicycle
Pedestrain
RealismMeta f
Zmin
Jmax
Umin
Umax
Zmin
Tmax
Umin
Umux
Tmin
Tmax
Ymin
Umax
-5
20
-1.5
1.5
-1
8
-1
-1.5
4.5
-2
2
0.7702
-10
25
-2
-3
12
-2
-3
6
-3
0.7701
15
-1
-1
5
-0.75
0.75
-0.5
3
-1.5
1.5
0.7689
Table 10: Sensitivity analysis on grid resolution.
Vehicle
Bicycle
Pedestrain
RealismMeta
Pinterval
Vinterval
Tinterval
Vinterval
Tinterval
Vinervaal
0.1
0.05
0.05
0.05
0.05
0.05
0.7702
0.05
0.05
0.025
0.025
0.025
0.025
0.7681
0.1
0.1
0.1
0.1
0.1
0.1
0.7687
C
COMPUTATIONAL COST
We report the average inference time with different vocabulary sizes in Table 12 and training time with spatially aware label smoothing Table T3, Larger vocabularies increase the inference time, but the cost remains relatively acceptable. Spatially aware label smoothing does significantly increase the total computational cost.
D DISCUSS ON FAILURE CASES
There are two types of failure cases:
(a) Unrealistic motion patterns may be introduced if there too many noisy logged trajectories in a certain region. With advances in data collection and annotation, as well as more precise sensors and improved annotation workflows, the occurrence of such cases can be reduced.
14

---

Published as a conference paper at ICLR 2026
Table 11: Sensitivity analysis on filtering and expansion parameters. These parameters are the same for all agent types.
k
5p
Sa
Sr
RealismMeta t
4
1
20
20
0.7702
3
1
10
0.7693
4
4
20
20
0.7695
Table 12: Average inference time on a single NVIDIA A100 GPU for a single rollout of 16 steps on scenario with different vocabulary sizes.
Vocabulary size for all agent types
inference time
2000
399 ms
4000
421 ms
8000
441 ms
Table 13: Training time for a single epoch on 8 NVIDIA A100 GPUs with a 20% training split.
Label smoothing type
training time
Standard Spatially Aware
18m 25s 18m 34s
(b) Long-tail behaviors may not be captured if the logged dataset is too small or lacks sufficient diversity. As larger and more diverse datasets become available and data collection costs decrease, these cases are expected be reduced.
E MORE VISUALIZATIONS
Two extra figures are presented illustrate how tokenization affects behavior generation. The red dots in figures represent endpoints of trajectories in the vocabulary, and the red arrows show the selected next actions.
Fig. shows a vehicle navigating dedicated right-turn lane. The K-disk vocabulary has insufficient coverage and offers low diversity turning trajectories, causing the vehicle lack sufficient turning and collide with the road edge (black lines in figure). In contrast, TrajTok provides a richer set of turning trajectory tokens for the model to learn and choose from, allowing the vehicle to stay within the lane successfully.
Fig. I0 shows two vehicles moving towards each other in parking lot. Similarly, TrajTok's vo- cabulary offers more diverse and drivable choices for the model handle interactions, successfully modeling yield behavior between the two vehicles, whereas k-means leads to a collision.
F
LLM USAGE
LLMs are used in writing for improving grammar and correcting typos.
15

---

Published as a conference paper at ICLR 2026
Frame 11
Frame 21
Frame 31
Frame 41
Frame 51
K-disks
Frame 11
Frame 21
Frame 31
Frame 41
Frame 51
TrajTok
Figure 9: Visualization in right turn.
Frame 21
Frame 31
Frame 41
Frame 51
Frame 61
K-means
Frame 21
Frame 31
Frame 41
Frame 51
Frame 61
TrajTok
Figure 10:
Visualization at parking lot.
16


Reviews:
[{"review_id": "xu83GAXxLJ", "reviewer": "Reviewer_kf6b", "summary": "The paper presents TrajTok, a hybrid trajectory tokenizer designed for behavior generation in autonomous driving. It investigates what constitutes an effective trajectory tokenizer under the next-token prediction (NTP) paradigm, analyzing four key properties—coverage, utilization, symmetry, and robustness—of existing data-driven and rule-based approaches. TrajTok integrates the advantages of both: it first constructs a rule-based grid of trajectory candidates and then applies data-driven filtering and expansion to balance vocabulary coverage and data efficiency. Additionally, the authors propose a spatial-aware label smoothing technique that weights token similarity by spatial distance, improving model generalization. Experiments on the Waymo Open Motion Dataset demonstrate that TrajTok achieves state-of-the-art performance, ranking first in the 2025 Waymo Open Sim Agents Challenge with superior realism and robustness across datasets and data scales.", "questions": "Please refer to the weaknesses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) This paper makes a clear and timely contribution by looking closely at what makes a good trajectory tokenizer in the next-token prediction (NTP) setup. The four proposed criteria—coverage, utilization, symmetry, and robustness—give a simple but useful way to understand and compare different tokenizers, which hasn’t really been discussed in earlier work like Trajeglish or MotionLM.\n\n(2) The proposed TrajTok method is simple but well thought out. It combines a rule-based start with data-driven filtering and expansion, which makes sense and nicely balances coverage and efficiency. This hybrid idea helps fix problems that appear in purely data-driven (too noisy) or rule-based (too redundant) tokenizers.\n\n(3) The paper also adds a spatial-aware label smoothing technique that slightly changes the standard cross-entropy loss. It’s an intuitive idea that takes spatial similarity between tokens into account, helping the model generalize better without depending on any specific architecture.", "weaknesses": "(1) The paper only validates TrajTok within the SMART model [1]. Since TrajTok is designed as a general tokenizer, applying it to other NTP-based architectures (such as Trajeglish [2] or MotionLM [3]) would further support its claimed generality.\n\n(2) The paper defines several thresholds for the filtering and expansion process, but the actual parameter values and tuning details are not provided. It is unclear how sensitive the model is to these choices or whether small changes in these thresholds would affect the final vocabulary and performance. Including the specific values or a short sensitivity analysis would help improve reproducibility and confidence in the results.\n\n[1] Wei Wu, et al. “Smart: Scalable multi-agent real-time motion generation via next-token prediction.” Advances in Neural Information Processing Systems, 37:\n114048–114071, 2024.\n[2] Jonah Philion, et al. “Trajeglish: Traffic modeling as next-token prediction.” arXiv preprint arXiv:2312.04535, 2023\n[3] Ari Seff, et al. “Motionlm: Multi-agent motion forecasting as language modeling.” In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\n8579–8590, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uJq5RTJyap", "reviewer": "Reviewer_9sjc", "summary": "The paper studies trajectory tokenization for next-token-prediction (NTP) behavior generation in autonomous driving simulators. \nIt argues that existing data-driven tokenizers such as VQ-VAE or K-disks have good utilization but poor coverage and symmetry, while rule-based grid tokenizers have wide coverage but many redundant or unrealistic tokens.\nTo balance these properties, the authors propose TrajTok, a hybrid tokenizer that first builds a rule-based grid vocabulary and then filters and expands it using logged trajectory data. They also introduce spatial-aware label smoothing for cross-entropy loss, where non-ground-truth tokens are weighted according to spatial distance.\nExperiments on the Waymo Open Motion Dataset and the 2025 Waymo Open Sim Agents Challenge show that TrajTok ranks first on the leaderboard and yields slightly higher realism metrics than baselines.", "questions": "What tokenizer was used in the SMART-tiny baseline that appears in Table 1? Without knowing this, it is difficult to measure the real gain from TrajTok.\n\nHave you tested TrajTok on prediction tasks that use continuous outputs rather than discrete NTP models to verify that the benefit comes from tokenization rather than training heuristics?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tClear motivation from data analysis. The authors systematically examine four tokenizer properties (coverage, utilization, symmetry, robustness) and relate them to logged data usage. This diagnostic perspective is useful for understanding tokenization quality.\n\n2.\tSimple yet general method. TrajTok is a lightweight combination of rule-based and data-driven principles that can be plugged into existing NTP architectures without retraining structural components.\n\n3.\tPractical evaluation. The paper reports results on the Waymo Open Sim Agents Challenge with official metrics, including the Realism Meta score, and provides ablations for label smoothing and vocabulary size.", "weaknesses": "# Major\n\n- Unclear evidence of improvement: In Table 1, TrajTok achieves 0.7852 while SMART-R1 reaches 0.7855, and SMART-tiny-CLSFT gives 0.7846. These gaps are within noise. It is difficult to identify a clear gain attributable to the tokenizer.\n\n- Ambiguous reference tokenizer: Table 2 compares TrajTok with VQ-VAE, K-means, K-disks, and grid tokenizers, but it is not stated which tokenizer SMART-tiny originally used on the leaderboard. The reader cannot determine whether the new tokenizer outperforms the baseline used in the challenge submission.\n\n- Vague qualitative evidence: Figure 4 does not convincingly demonstrate that the proposed tokenizer contributes to better behavior generation. The visualization focuses on scene outcomes rather than showing how tokenization differences influence the trajectories. Without comparisons using the same scenario and seeds, the figure offers little empirical value.\n\n- Questionable justification of symmetry: The paper claims symmetry is critical for vehicle kinematics and real-world diversity, yet real traffic is not necessarily symmetric. For example, in right-hand traffic countries, turning behaviors are directionally biased. The necessity of symmetric flipping should be justified with more empirical or theoretical evidence. Table 6 shows a small gain from symmetry, but the physical rationale is not convincing.\n\n# Minor\n\n- Limited exploration of failure cases: The discussion does not examine cases where the tokenizer introduces unrealistic motion patterns or under-represents long-tail behaviors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mDikdGp9qI", "reviewer": "Reviewer_scBZ", "summary": "This paper studies what makes for a good trajectory tokenizer for next token prediction behavior generation and proposes TrajTok, a plug and play tokenizer that combines rule based vocabulary construction with data driven filtering and expansion. The method explicitly targets four properties of a vocabulary, namely coverage, utilization, symmetry, and robustness, and introduces a spatial aware label smoothing scheme that assigns higher probability to tokens that are closer in trajectory space to the ground truth. The authors evaluate on the Waymo Open Motion Dataset and the Waymo Open Sim Agents Challenge and report first place on the 2025 leaderboard with consistent gains when TrajTok is used with a SMART backbone on validation metrics.", "questions": "1. The appendix switches to separate prediction heads per agent type. How much of the improvement in validation metrics is due to this change rather than the tokenizer? A brief ablation on common heads versus separate heads would clarify attributions.\n2. How sensitive are the reported gains to the grid range and resolution per agent type and to the neighborhood thresholds for filtering and expansion? A small sweep around the appendix settings would help readers tune the method.\n3. In L377, “Increasing the vocabulary size improves the ability to represent complex distributions but may lead to model underfitting”, should this be the opposite (i.e. a smaller size causes underfitting)? Intuitively a larger vocabulary size should need more data, is under-training a better word?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- There is a clear problem framing and taxonomy of desirable tokenizer properties. The paper analyzes how data driven methods often achieve high utilization but limited coverage and how rule based methods tend to overcover with many unused tokens. This motivates a hybrid design that TrajTok implements.\n- The construction pipeline is simple and reproducible. The four step procedure is easy to follow, from agent centric normalization and symmetry by flipping, to grid based candidate selection, to neighborhood based filtering and expansion, to final token generation including curve interpolation when a selected cell has no examples.\n- The paper reports strong results on both the public leaderboard and controlled comparisons, showing first place on WOSAC 2025 and better validation performance than VQ VAE, K means, K disks, and a pure grid, using a SMART based model under a common vocabulary size. Cross dataset and low data experiments further support generalization.", "weaknesses": "- Scope is limited to trajectory tokenization. The method relies on a base next token prediction model, most experiments use SMART tiny, and the paper makes a few implementation choices that deviate from the original backbone, such as separate heads per agent type. It would help to isolate how much gain comes purely from the tokenizer versus modest architecture changes.\n- Computational cost and model scale trade offs are not discussed. The approach can yield large vocabularies across agent types, and spatially aware label smoothing appears to require computing distances from each target to many tokens. A brief cost analysis and a description of any approximations would improve clarity. The paper notes overall training settings but not the incremental overhead from TrajTok and the smoothing.\n- Minor presentation issue - I could not find what the highlight colors mean in Tables 1 and 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5YU1Dxe9im", "reviewer": "Reviewer_edLu", "summary": "This paper proposes TrajTok, a trajectory tokenizer for discrete next-token-prediction (NTP) behavior generation in autonomous driving. It combines data-driven and rule-based trajectory tokenizers, and has balanced coverage and utilization as well as good symmetry and robustness. It builds on top of the SMART NTP behavior generation backbone. It shows decent results in the 2025 Waymo Open Sim Agents Challenge.", "questions": "The results on Table 2 use 20% of the training set. Do other tokenizers (VQ-VAE, K-means, K-disks, Grid) close the gap using a larger training set?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The coverage, utilization, symmetry, and robustness perspectives are intuitive and useful.\n2. The algorithm is simple and the resulting vocabulary is nice and clean. \n3. The algorithm achieves good empirical results and high ranking on Waymo Open Sim Agents Challenge.", "weaknesses": "1. The authors claim that TrajTok wins first place in the Waymo Open Sim Agent Challenge 2025. This contradicts Table 1, which shows SMART-R1 wins the first place, while TrajTok places 2nd. Looking at the official leaderboard (https://waymo.com/open/challenges/2025/sim-agents/) we see that TrajTok placed the 5th. \n2. TrajTok changes the tokenizer on top of the SMART backbone. Since the top 5 models on the official leaderboard are all variants of SMART, it seems TrajTok’s contribution is incremental or marginal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)