UniMotion: A Unified Motion Framework for Simulation, Prediction and Planning
Authors: Nan Song
Junzhe Jiang
jingyu li
Xiatian Zhu
Li Zhang
Venue: NeurIPS 2025 poster ()
Keywords: autonomous driving; motion simulation; trajectory prediction; planning
Primary area: applications

Abstract: Motion simulation, prediction and planning are foundational tasks in autonomous driving, each essential for modeling and reasoning about dynamic traffic scenarios. While often addressed in isolation due to their differing objectives, such as generating diverse motion states or estimating optimal trajectories, these tasks inherently depend on shared capabilities: understanding multi-agent interactions, modeling motion behaviors, and reasoning over temporal and spatial dynamics.
Despite this underlying commonality, existing approaches typically adopt specialized model designs, which hinders cross-task generalization and system scalability. More critically, this separation overlooks the potential mutual benefits among tasks. Motivated by these observations, we propose **UniMotion**, a unified motion framework that captures shared structures across motion tasks while accommodating their individual requirements. Built on a decoder-only Transformer architecture, UniMotion employs dedicated interaction modes and tailored training strategies to simultaneously support these motion tasks. This unified design not only enables joint optimization and representation sharing but also allows for targeted fine-tuning to specialize in individual tasks when needed. Extensive experiments on the Waymo Open Motion Dataset (WOMD) demonstrate that joint training leads to robust generalization and effective task integration. With further fine-tuning, UniMotion achieves state-of-the-art performance across a range of motion tasks, establishing it as a versatile and scalable solution for autonomous driving.

UniMotion: A Unified Motion Framework for Simulation, Prediction and Planning
Nan Song
Junzhe Jiang'
Jingyu Li
Xiatian Zhu
Li Zhang 1.2
Schooo of Data Science, Fudan University
Shanghai Innovation Institute
University of Surrey
Abstract
Motion simulation, prediction and planning are foundational tasks in autonomous driving, each essential for modeling and reasoning about dynamic traffic scenar- ios. While often addressed in isolation due to their differing objectives, such generating diverse motion states or estimating optimal trajectories, these tasks inherently depend shared capabilities: understanding multi-agent interactions, modeling motion behaviors, and reasoning over temporal and spatial dynamics. Despite this underlying commonality, existing approaches typically adopt special- ized model designs, which hinders cross-task generalization and system scalability. More critically, this separation overlooks the potential mutual benefits among tasks. Motivated by these observations, we propose UniMotion, a unified motion frame- work that captures shared structures across motion tasks while accommodating their individual requirements. Built on a decoder-on)) Transformer architecture, UniMotion employs dedicated interaction modes and tailored training strategies to simultaneously support these motion tasks. This unified design not only en- ables joint optimization and representation sharing but also allows for targeted fine-tuning to specialize in individual tasks when needed. Extensive experiments on the Waymo Open Motion Dataset demonstrate that joint training leads to robust generalization and effective task integration. With further fine-tuning, UniMotion achieves state-of-the-ar performance across range of motion tasks, establishing it as versatile and scalable solution for autonomous driving.
Introduction
Motion understanding a cornerstone of autonomous driving, supporting essential capabilities such as interactive behavior modeling, motion analysis, and decision-making for the ego vehicle. Core motion tasks include motion simulation, trajectory prediction, and ego planning. Among them, trajectory prediction [1, 2, 3] and ego planning [4] are critical intermediaries between perception modules and control systems in the driving pipeline, while motion simulation [5] generates rich and diverse agent behaviors, serving both development and evaluation for motion models. The complexity of real-world driving, characterized by uncertain behaviors and densely interactive environments, makes these tasks challenging.
Accordingly, recent research has produced a wide range task-specific models to optimize per- formance for its designated objective. Trajectory prediction and planning models have primarily focused on comprehensive representation learning [6, 7, 8, 9]. along with growing emphasis on precise waypoint estimation [10, 11, 12, 13]. In contrast. motion simulation has leaned heavily
"Li Zhang (lizhangfd@ fudan.edu.cn) is the corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Joint training
(b) Task fine-tuning
Tokerization
NTP
for sim
UniMotion
UniMotion
Multi-modal for pred
agent
LFR
Pred2Gen for plan
traffic scenario
Figure
Overview of our UniMotion pipeline. (a) We train jointly our model with combined
generative and forecasting supervision, resulting in multi-task model that can simultaneously address all kinds of motion tasks autonomous driving. (b) We further adopt dedicated fine-tuning strategies for each task, producing task-specific models promote specialization.
generative modeling. Inspired by the success of Large Language Models (LLMs), recent GPT-style architectures [14, 15] have been applied to simulate agent behaviors, Furthermore, this paradigm has also been introduced into prediction [16, 17] and planning [13] tasks to achieve progressive and more accurate regression.
While these task-customized models have proven effective for different motion tasks, it is challenging to transfer models and learned knowledge across tasks. Considering that these tasks are essentially similar due their reliance on interaction modeling and motion reasoning, our goal is build a general model that encourages knowledge sharing among motion tasks and enables cross-task generalization. To achieve this aim, we revisit the formulation of motion tasks and propose unified motion framework for autonomous driving. We begin by abstracting existing motion tasks into two fundamental categories: diverse motion generation and long- long-range trajectory forecasting. which serve as conceptual foundation for unification. From this, we develop UniMotion, unified framework based on decoder-only Transformer, chosen for its simplicity, scalability, and strong generative capacity across tasks. By incorporating task-aware interaction modes and training strategies (see Figure (a)), UniMotion enables joint training across simulation, prediction and planning, fostering shared representations and efficient multi-task learning. To further enhance task proficiency, we also introduce dedicated fine-tuning strategies (see Figure I(b)) that adapt the unified model to specific objectives, improving performance and practical deployment.
Our contributions are summarized as follows: (i) We abstract motion tasks into two core categories and propose UniMotion, a unified Transformer-basee framework that jointly models simulation, prediction and planning to promote generality and cross-task knowledge sharing. (ii) We design effective fine-tuning strategies that specialize the jointly trained model for individual tasks, allowing the model to master task specific expertise. (iii) Extensive experiments on the Waymo Open Motion Dataset (WOMD) show that UniMotion achieves competitive performance under joint training and sets new state-of-the-ard results when fine-tuned for specific tasks.
2
Related work
2.1 Motion tasks in autonomous driving
Existing motion tasks in autonomous driving can be roughly categorized into three types according to their objectives: motion simulation, trajectory prediction, and ego planning. We provide detailed discussion for each type in the following.
Motion simulation focuses on simulating diverse and complex motion patterns, augmenting training samples and offering board assessment for autonomous driving algorithms. To effectively measure simulation quality, Sim Agents [5] first proposes a benchmark with comprehensive metrics, expecting models to progressively evolve the motion states of all traffic agents. Motivated by the performance of L.LMs, existing methods [14, 15] tokenize the agent trajectories and map elements, and employ scalable decoder-only model to perform auto-regressive trajectory generation. Besides, CATK [18] utilizes closed-loop supervised finetuning with Closest Among Top-K rollouts, which mitigates covariate shift in open-loop behavior cloning and further improves these GPT-style methods. In contrast, UniMM [19] introduces unified mixture model with both continuous mixture models and discrete GPT-style models to address simulation.

---

Trajectory prediction anticipates models to rapidly estimate the future trajectories of traffic par- ticipants, providing prior knowledge for subsequent rrcision-making As a foundation task in autonomous driving, trajectory prediction has been widely explored [6, 8, 11] with Transformer frameworks. Recent methods (20, 21, 22] uncover detailed temporal information and achieve more accurate prediction. Moreover, there are methods utilizing particular strategies for impressive perfor- mance improvements, such as model pre-training [23, 24] and trajectory post-refinement [10, 12].
Ego planning is responsible for giving final driving trajectories for ego vehicle according to the driving environment and upstream results. Although rule-based models [25, 26] still hold a crucial position, learning-base based methods have emerged and exceeded. PlanTF [27] and PLUTO [28] effectively alleviate the limit of imitation-basee planners through designed model architecture and training strategies. Besides, BeTopNet [9] further improves planning performance by explicitly representing behavioral topology among multi-agent future states.
2.2 GPT-style motion models
GPT-style LLMs have demonstrated strong capabilities in language tasks, which stimulates the emergence of similar architectures in autonomous driving [29, 30J. Given that both trajectories and text share a sequential structure, it is natural to introduce analogous modules and designs into motion models. In particular, the aforementioned simulation methods directly utilize decoder-only models with next-token prediction for simulation tasks, generating diverse multi-agent trajectory sets. There are also some prediction and planning methods adopting modified auto-regressive decoding. For example, MotionLM [17] and Trajeglish [16] introduce a causal decoder for iterative trajectory prediction, and similarly, CarPlanner [13] adopts consistent auto-regressive planning with reinforcement learning. In contrast, we unify all motion tasks under a GPT-style model, fully exploiting the capability of this framework.
3 Methodology
3.1 Preliminary
Task definition. Motion understanding in autonomous driving focuses on three tasks: simulation, prediction and planning. All these tasks share the same traffic scenario input, including historical agent states An and a high-definition (HD) map M of driving scenes. Motion simulation is typically designed to support other motion tasks. Given A,, as initial motion states, it aims to iteratively and concurrently simulate multi-agent states { Af] where K represents the number of generated rollouts. Simulation aims to exhibit high short-term generation diversity. In contrast, trajectory prediction provides distant future trajectories Ay of traffic agents over span of time Ty. In real- world scenarios, we expect the predictors to generate the results directly and efficiently, rather than relying on
auto-regressive generation [16, 17]. Accordingly. this task evaluates the ability of models to perform accurate long-range predictions. Ego planning aims to determine feasible trajectories for the ego vehicle by referencing surrounding agents. It can be viewed as first predicting the behaviors of other traffic participants, followed by the progressive generation of the ego vehicle's trajectory. In light of the above observations, summarize motion tasks into two fundamental components: diverse motion generation and long-range trajectory forecasting. As motivated earlier, we strive to integrate these tasks into a unified framework and jointly develop both capabilities in a single model, thereby enhancing overall motion understanding and fostering mutual complementarity between tasks.
Input representation. To ensure the generalization of representations across tasks, we standardize the input format via tokenization following [14, 15]. Concretely, the trajectories of all agents are segmented at fixed time intervals and normalized to form trajectory set, which is then clustered
to construct the agent token vocabulary S.. Similarly, map tokens S,, are generated by dividing original map polylines into fixed-length segments followed by clustering. The corresponding agent token embeddings Esa E RR KC and map token embeddings Esm E RRR. xC obtained by applying MLP modules to S, and Sm, respectively, where K()) and C are the number of tokens and the dim of feature embeddings. Subsequently, the agent states A and HD map M are tokenized based on geometric similarity of segments, forming the input agent embeddings E. XC and map
3

---

ego vehicle features
Agent decoder
X
Motion tasks
agent features
map festures
Agent-agent self-attention
simulation
attended
not attended
causal mask
NTP
Agent-map cross-attention
Map encoder
prediction
history future
map tokenization
Temporal self-attention
seq mask
LFR
planning
predictions
agent
Tokenization
OQQO
LFR
NTP ego vehicle
agents and map polylines
agents features
agents
Figure 2: Overview of UniMotion architecture. It takes input tokenized agent trajectories and map polylines, and adopts a decoder-only structure equipped with an additional map encoder to yield motion states. By employing task-specific attention masks and training objectives, UniMotion is empowered to flexibly and effectively address multiple motion tasks simultancously.
embeddings En E NR. xC, where Na and No. denote the number of agents and map segments, and denotes the time frames.
3.2 UniMotion for multiple motion tasks
As depicted in Figure 2, the overall design of our UniMotion follows a decoder-only architecture for agent trajectory generation and prediction, with an additional map context encoder. For each specific motion task, we introduce dedicated interactive patterns and training objectives for specialization, but enable weight sharing learn general representations.
3.2.1 Scene context encoding and motion decoding
To ensure the robustness and generalizability of the model, we employ the Transformer architecture with relative positional embedding following [8, 14, 15], which remains invariant under coordinate transformations, Specifically, we first stack several self-attention modules to encode map embeddings, yielding map features Fm. Then, agent embeddings E., along with Fm, are further delivered to agent Transformer decoder constructed with factorized attentions, which are composed of temporal self-attention, agent-map cross-attention and agent-agent self-attention. This can formulated as:
F. MHSA, Fa, K = V = Fa + Re(Pa)),
F. MHCA, F,, K V Fm Ra-m (Pa; Pmn));
(1)
F, = MHSA,-.(Q = F., K - = F, (Pa))
where the relative positional embedding for spatial and temporal position information Pt.j- and the initial Fg is directly derived from Eg. This decoder module enables the thorough interaction among agents and between agents and road elements, finally giving step-wise motion states.
3.2.2 Joint training across motion tasks
Following the designs of multi-task language models, we apply customized interactive attention masks for different motion tasks and introduce tailored training objectives to improve both step- wise generation and long-range prediction. At inference time, we adaptively leverage the model's capabilities according to the specific task.
Motion simulation. The motion simulation task closely resembles language generation, which aims iteratively select suitable tokens from motion vocabulary. Accordingly, we adopt a causal attention mask and employ Next- Token Prediction (NTP) as the learning strategy, modeling the generation of

---

predicted trajectories
traffic scenario
(x, y)
UniMotion
Multi-modal Transformer decoder
Filter & Tokenization
R₂
R2
R
Reward
UniMotion
Suup
Lr
intention embedding
anchor-based
ft ego generation
predictions
(a) RL-based fine-tuning
(b) Multi-modal fine-tuning
(c) P2G fine-tuning
Figure 3: Illustration of task-specific fine-tuning strategies. We adopt (a) RL-based fine-tuning to improve simulation likelihood while maintaining closed-loop consistency. (b) Multi-modal fine- tuning is introduced to refine the multi-modal behavior of predictions. With reference to the planning inference, we utilize (c) Pred2Gen fine-tuning address distribution mismatch.
the next agent token Ae step from the conditional probability P(A, Acr). During inference, motion processes for all agents are simultancously simulated through auto-regressive generation.
Trajectory prediction. We regard trajectory prediction as pouueecceto-sequene task, which maps historical segments to future trajectories. Tokens within the historical segments are allowed attend only to each other, while future tokens can attend both to preceding tokens within the future segment and all tokens in the historical segments. Moreover, we propose Long-range Future Regression (LFR) to meet the strict requirements of prediction task on accuracy. Specifically, we replace token classification with normalized trajectory regression, and it predicts complete long-range future trajectory instead of short segments with token-level length, in which case future trajectories A, can be derived from:
A = MLP((aa, A, E NoXTXXxXxx
(2)
Notably, all these tokens utilized for dense supervision in the training phase, while only need to perform single-pace pass reasoning at current step for inference.
Ego planning. As mentioned in Section 3.1, we consider planning task to be the combination of generation and prediction. Hence, the training objectives for simulation and prediction naturally benefit the planning task, without requiring any additional training strategies. After training, we adopt two-stage inference approach for ego trajectory generation. Concretely, we first jointly prediet future trajectories for all traffic participants, and then tokenize the future parts these predictions to form the future tokens. Given the predicted future tokens of surrounding agents, we then progressively generate the planning trajectory of ego vehicle.
3.2.3 Task-specific fine-tuning strategies
Despite the capability of solving multiple tasks simultaneously, models are typically expected to exhibit higher specialization for individual tasks real-world applications. To this end, we introduce task-specific fine-tuning strategies to further enhance the specialization of UniMotion. Overall, we prefer not to introduce additional parameters during fine-tuning stage. According to this principle, utilize reinforcement learning-base -bsed fine-tuning for simulation and two-stage pred2gen finetuning for planning. Additionally, trajectory prediction requires to produce multi-modal outputs, which will significantly increase the computational burden under dense supervision. To alleviate this, employ a multi-modd modal decoder to support the prediction process as compromise.
In line with the purpose of Reinforcement Learning from Human Feedback (RLHF) in LLMs, we enhance the model to generate trajectories that align with human driving behavior and traffic rules, which cannot be constrained through direct supervision alone. Hence, we introduce RL-baaed fine-tuning for simulation. As shown in Figure 3(a), we first generate n rollouts driving scenes group following GRPO [31], with activating the gradient for only one rollout. Subsequently,
5

---

rewards R are computed using the same algorithms as the metrics in Sim Agents [5], which consist of kinematic reward and collision reward for simplicity. We formulate this as:
R expl
log g + 1 not collision),
where the kinematic reward is defined as the exponentiated log-likelihood of ground truth samples under the distribution induced by corresponding generated trajectories, whereas the collision reward is an indicator function. Then, the advantage A is derived from the normalized rewards within the group of each agent. Notably, we only supervise one rollout to reduce computational overhead. More details are provided in Section 3.3 and supplementary materials.
To improve
of predicted trajectories, we introduce a lightweight transformer decoder similar to [11,32] as illustrated in Figure 3(b). At this stage, we only focus on the agents of interest that are utilized for evaluation, and perform prediction within perspective local system. To this end, the porddinat-iivariian features are transformed into the local agent features 1 and map features Fm through adding the local normalized position content encoded by MLP layer. For each interested agent, the estimated multi-modal trajectories aj E IR X'yy x2 for Nmo modes can be derived by:
af Transformer (Q Fino: V= (Fa
1)
where Fmo denotes the mode features encoded from intention points, and F* and Fm are with respect to current interested agent. Benefiting from distinct intention points, our model can provide more diverse trajectory choices after fine-tuning.
During planning inference, the predicted trajectory tokens for surrounding agents will participate in and significantly affect ego generation, inconsistent with the independent training process. To alleviate potential distribution mismatch,
further fine-tune ego generation using the predicted results. As shown in Figure 3(c), the incorrectly predicted trajectories are first filtered out and replaced by the ground truth. Then, we tokenize all predictions and feed them into our network for ego generation, which is fine-tuned with end-to-end supervision.
3.3 Model training
During the joint training process, mainly adopt NTP classification loss Lntp between next tokens A, and corresponding ground truth A, for generation supervision, and LFR regression loss Liee between As and ground truth A for trajectory prediction supervision. We combine these two individual losses with equal weights to form the overall loss L, formulated as follows:
L = Lup As) Cin (A),
(5)
where we employ the cross-entropy loss and the smooth-L1 loss for Lutp and Litr. respectively.
During the fine-tuning stage, we employ different supervision strategies for each motion task. For simulation fine-tuning, we follow [33] to simplify the GRPO but still preserve the consistency constraint through closed-loop supervision on the gradient-activated rollout A, RR.,.T XT Besides, light of efficiency concerns, we only use A, rather than rollouts to perform a single policy update per iteration. The simulation fine-tuning loss is:
La-sim = A,) (,C,,Ar,
(6)
where Lce is the cross-entropy loss between A, and ground truth A,, Cp is the policy utilized in GRPO with taking A, and advantage A as inputs, and w, is balancing factor. The prediction loss consists of Gaussian NLL loss Cun to supervise multi-modal trajectories o and cross-entropy loss for mode classification. Meanwhile, the model also learns to regress the local future trajectories A for other agents as auxiliary supervision. The loss can be formulated as:
La.ped = Wn (A,, Aÿ) + Lan(uj, (ā) + Lee-
In addition, simply supervise the generation dego for ego vehicle and the predictions A for surrounding agents in the planning fine-tuning stage as:
Ctpplan up1Lur ,ATIIT) + Cutp(aegu Aego).
(8)

---

Table 1: Performance comparison 2025 Waymo Open Sim Agents Challenge.
Realism Meta Metric 1
Kinematic
Interactive Metrics
Map-based Metrics ?
Method
minADE 4
Metrics f
UniTAM
0.7698
0.4869
0.7876
0.9085
1.3940
UniTFormer
0.7776
0.4892
0.7997
0.9140
1.3592
LLM2AD
0.7779
0,4846
0.8048
0.9109
1.2827
UniMM [34]
0.7829
0,4914
0.8089
0.9161
1.2949
CATK [18]
0.7846
0.4931
0.8106
0.9177
1.3065
UniMotion
0.7851
0.4943
0.8105
0.9187
1,3036
Table 2:
Performance comparison on WOMD Prediction Leaderboard. For each metric, the best and second best results are highlighted in bold and underline, respectively.
Method
minADE |
minFDE
MR
mAP1 1
Soft mAP
HDGT [35]
0.5933
1.2055
0.1854
0.3577
0.3709
MTR [11]
0.6050
1.2207
0.1351
0.4129
0.4216
MTR++ [36]
0.5906
1.1939
0.1298
0.4329
0.4414
MGTR [37]
0.5918
1.2135
0.1298
0.4505
0.4599
EDA [32]
0.5718
1.1702
0.1169
0.4487
0,4596
ControlMTR [38]
0.5897
1.1916
0.1282
0.4414
0.4572
RMP-YOLE [39]
0,5737
1.1697
0.1160
0.4523
0.4673
UniMotion
0.5718
1.1643
0.1162
0.4534
0.4642
Experiments
4.1 Experimental settings
Datasets and metrics. To evaluate the performance our method, we conduct extensive experi- ments on the Waymo Open Motion Dataset, comprising 486,995 training scenarios, 44,097 validation scenarios and 44.920 testing scenarios. Each scenario has an observation window of 9.1 seconds, consisting of 91 frames sampled at 10 Hz. Given 1 11-frame historical agent states and HD map, the motion models are tasked with generating predicting the future states of 80 frames. Additionally, prediction and planning tasks only perform one-step inference at the current time, while simulation task requires 32 simulations per scenario.
We employ respective benchmark metrics to assess our model. For simulation, the core metric the Realism Meta Metric, which is utilized to measure the similarity between the simulations and the real-world distribution. It is calculated by the weighted sum of Kinematic Metrics for motion features, Interactive Metries capture the interaction among agents and Map-based Metrics test the agent behavior on the map. For prediction, 6 predicted trajectories for each interested agent are expected for evaluation, with metrics including minADE (minimum Average Displacement Error), minFDE (minimum Final Displacement Error), Miss Rate, mAP and Soft MAP. In addition, due to the absence of official benchmark for planning, we adopt the metries proposed by [40] as measures.
Implementation details. We tokenize the trajectories and map elements following [15, 18), with 2,048 tokens for agents and 1,024 tokens for map. During the training phase, We train our models using the AdamW W [41] optimizer with total batch size of 48 on NVIDIA A6000 GPUs for 30 epochs, with the weight decay set to 0.01. We initialize the learning rate of x 10 which is then decayed following a cosine annealing schedule. Regarding the fine-tuning stage, we employ the same learning rate for prediction, while lower learning rate of 5 x 10 for simulation and planning. Beside, we also fine-tune these tasks with the same epochs.
Each agent token covers 0.5 seconds, and each map token covers about 0.5 meters. After tokenization, original 91-step trajectories in Waymo are compressed into frames. For the training phase, training time for full data is approximately 3.5 days, while using 20% of the data takes around day in our experimental settings. In addition, we set the balance factor w.s Upr and wpl to 0. 1, 0.1 and 0.5, respectively.

---

Table 3: Performance comparison for Waymo planning.
Planning error
Method
Collision
Red light
Off route
@ ls
@3s
@5s
5.469
2.772
4.816
0.175
1.416
4.194
IL+Prediction
3,930
1.670
4.542
0.127
0.892
2.901
Sep. Plan+Pred.
1.813
1.327
0.527
0.238
1.251
3.466
DIPP [40]
1.802
1.235
0.506
0.227
1.187
3.335
UniMotion 1.565
1.309
0.477
0.083
0.591
2.246
4.2 Comparison with state of the art
We compare the performance of our UniMotion with top-ranked competitors on Sim Agents [5]. WOMD Prediction and Waymo Planning [40] benchmarks. The leaderboards for Sim Agents and WOMD Prediction are presented in Table and Table 2, respectively. For simulation, achieve the best score of 78.51 on Realism Meta Metric. Benefiting from the kinematic reward of fine-tuning, the model is encouraged generate trajectories with higher motion similarity to real-world situations, driving better Kinematic Metrics than other methods. However, the minADE metric exhibits relatively poorer performance, which can be attributed to the tokenization baseed design's ss stronger emphasis on generation diversity at the cost of slightly reduced prediction accuracy. The leaderboard of the prediction task demonstrate that method achieves promising results across almost all metrics, except for Soft mAP. Regarding the planning task, UniMotion has far outperformed most of previous approaches as depicted in Table 3, showing that our method stands distinctly ahead of others in terms of planning error. Moreover, to ensure generality and consistency, we supervise model training solely using logged trajectories. This mechanism leads to suboptimal performance in reacting to red lights, which can be mitigated by incorporating traffic rule constraints, as in [40].
method
4.3 General ablation study
We conduct ablation studies the WOMD validation split examine the effectiveness of each component in UniMotion. For efficiency, we use only 20% of the training data for ablation studies, while adopting the default experimental settings following Section 4.1 for all other aspects. We focus on the core modules here, while leaving task-specific ablation to the supplementary materials.
Effects of NTP and LFR. As shown in Table 4, simply assess the effectiveness of NTP and LFR in our network on simulation and prediction tasks. The first and second rows demonstrate the results of training UniMotion using only NTP LFR, respectively. Despite the absence of dedicated supervision for another one of tasks, we report all relevant metrics to compare with joint training and showease the model's ability handle multiple tasks. It can be observed that the model benefits from both NTP and LFR in generation and prediction, while it performs poorly on another task. In the third row, we implement a joint training strategy with these two types of supervision, simultaneously endowing the model with diverse generative and long-range predictive capabilities. Besides, we find that joint training further improves prediction accuracy compared to using LFR alone, which we attribute to the diverse generation directions serving targets to guide the prediction process.
Table 4: Ablation on joint training with NTP and LFR. "Kin."/"Inter." Kinematic//nteratiive metrics.
Simulation
Prediction minFDE
NTP
LFR
Kin.
Inter.
Map
minADE
minADE
mAP
0.4884
0.7961
0.9148
1.4074
0.7697
1.5547
0.2629
0.4401
0.7742
0.8949
1.2965
0.6668
1.3613
0.2935
0.4892
0.7968
0.9144
1.4011
0.6508
1.3296
0.3147
Effects of task-specific fine-tuning. After training a preliminary model with a joint strategy, we further tailor it with designs specific to the target task. As demonstrated in Table 5, fine-tuning methods can intuitively bring consistent improvements across motion tasks. For simulation, all metrics exhibit varying degrees of gains after fine-tuning, especially in the reward items. The 3rd and

---

4th rows of prediction task show that the mAP metric has been remarkably enhanced by introducing an extra anchor-based decoder, which can cover more potential future trajectories. In addition. noticeable improvements are also observed on the minADE and MR metrics, achieving the expected goals fine-tuning. Regrading the planning task, the 5th row shows limited performance due the distribution mismatch between predicted tokens and original trajectory tokens. After fine-tuning, the model adapted to the two-stage process, and there are considerable improvements for all metrics.
Table 5: Ablation task task-specific fine-tuning for simulation, prediction and planning. Notably, selected a set of metrics with substantial variation to facilitate comparison. Additionally, we fine-tune planning model with the data following [40] instead of 20% data.
Task
ft.
Metrics
Kinematic
Interactive
Map-based
minADE +
Simulation
0.4892
0.7968
0.9144
1.4011
0.4939
0.8032
0.9159
1.3904
minADE +
minFDE
MR +
mÄP 1
Prediction
0.6508
1.3296
0.1626
0.3147
0.6413
1.3368
0.1493
0,3856
Collision +
Err @ 4
Err @33s
@@55 4
Planning
X
1.7347
0.1326
0.7943
2.7809
1.5650
0.0834
0.5908
2.2455
4.4 Task-specific ablation study
In this section, we adopt the same experimental settings and conduct task-specific ablation studies on the simulation and prediction tasks investigate the effects different module designs.
Effects of RL-based fine-tuning for simulation. We present the effeets of different constraints in Table 6. As shown in the 2nd and 3rd rows, closed-loop supervised fine-tuning leads notable improvements, while relying solely on RL-based supervision, like DAPO [33], fails to produce satisfactory results. We attribute this to two primary factors: insufficient reward signals and limited model capacity.
Table 6: Ablation on RL-based fine-tuning. "con.", consistency constraint. "Kin."/"Inter" Kine- matic/Interactive metrics.
con. rl
Kin.
Inter.
Map
minADE
0.4892
0.7968
0.9144
1.4011
0.4921
0.8019
0.9156
1.3933
0.4913
0.7974
0.9120
1.3967
0.4939
0.8032
0.9159
1.3904
Effects fine-tuning for prediction. We present the effects of different fine-tuning strategies in Table 7. The comparison between the 2nd and 3rd rows demonstrates that although the end-to-end fine-tuning strategy (without intention anchors) can significantly reduces prediction errors, severely compromises prediction diversity and confidence, leading degraded mAP performance. Hence, retain the intention anchors as part of the fine-tuning process.
Table 7: Ablation on prediction fine-tuning.
Strategy
minADE
minFDE
MR
mAP
w/o ft.
0,6508
1.3296
0.1626
0.3147
ft. w/o intention
0.6174
1.2639
0.1313
0.3389
ft. w/ intention
0.6413
1.3368
0.1493
0.3856

---

Rollout
0s
46
88
Rollout
Simulation
46
B5
Prediction
Planning
Figure Qualitative results on WOMD validation split. The top and bottom rows present two rollouts of simulation and the results of prediction and planning, respectively. The red and green arrowed curves are ground truth and predicted trajectories.
4.5 Qualitative results
In Figure 4, we present qualitative results of our network on three motion tasks. The first two panels illustrate the diverse simulation results of all traffic agents at Os, 4s, and 8s, respectively. The bottom-left figure depicts the multi-modal predictions for interested agents, while the bottom-right figure presents the ego trajectory generation process in token-by-toker manner. These qualitative results demonstrate our network's capability handling multiple motion tasks.
Conclusion
In this work, abstract mainstream motion tasks into core types and aim address all motion tasks within unified framework, thereby promoting knowledge sharing and enhancing model generalization. Achieving this. we present UniMotion, an integrated model designed particularly to support joint training for these tasks. The critical components of our framework are distinct interactive modes and training strategies. Besides, we further perform task-specific fine-tuning for each task, fulfilling the requirement for expert-level models in real-world applications. Our extensive experiments on several task benchmarks comprehensively demonstrate that UniMotion is capable of unifying motion tasks, and the fine-tuned versions surpass the current state-of-the-a performance in the respective tasks.
I Limitations and future work. Although we employ decode-only framework to ensure flexibility, the coupled spatial and temporal interaction and relative position embedding rely on detailed posi- tional information of traffic elements, constraining the model from leveraging more L.LM-related technologies. Moreover, after applying tokenization and conducting joint multi-task training, is worthwhile to explore cross-dataset learning, which could lead to substantial improvements in joint training performance. Limited by computational resources and training time, we have not explored this technique in the present work.
A Acknowledgments
This work was supported in part by National Natural Science Foundation of China (Grant No. 62376060).
10

---

References
[1] Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi. Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In ICCV, 2021. 1, 13
[2] Ming-Famd Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In CVPR, 2019. 1
[3] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagject Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next generation datasets for self-driving perception and forecasting, In NewrIPS, 2021.
[4] Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong. Eric Wolff, Alex Lang. Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprins, 2021.
[5] Nico Montali, John Lambert, Paul Mougin, Alex Kuefter, Nicholas Rhinchart, Michellc Li, Cole Gulino, Tristan Emrich, Zoey Yang, Shimon Whiteson, et al. The waymo open sim agents challenge. In NeurlPS, 2023. 1, 2, 6, 8,
[6] Jiyang Gao. Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In CVPR, 2020. 1,
[7] Zikang Zhou, Luyao Yc, Jianping Wang. Kui Wu, and Kejie Lu. Hivt: Hicrarchical vector transformer for multi-agent motion prediction. In CVPR, 2022. 1,3
[8] Zikang Zhou, Jianping Wang, Yung-Hui Mui Li, and Yu-Kai Huang. Query-centric trajectory prediction. In CVPR, 2023. 3, 4
[9] Haochen Liu, Li Chen, Yu Qiao, Chen Lv, and Hongyang Li. Reasoning multi-agent behavioral topology for interactive autonomous driving. NeurIPS, 2024. 1,3
[10] Sehwan Choi, Jungho Kim, Junyong Yun, and Jun Won Choi. R-pred: Two-stage motion prediction via tube-query attention- trajectory refinement. In ICCV, 2023. 1,3
[11] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. In NeurlPS, 2022. 1, 1,3,6 6,
[12] Yang Zhou, Hao Shao, Letian Wang. Steven L Waslander, Hongsheng Li, and Yu Liu. Smartrefine: An scenario-udaptive refinement framework for efficient motion prediction. In CVPR, 2024. 1,
[13] Dongkun Zhang, Jiaming Liang, Ke Guo, Sha Lu, Qi Wang, Rong Xiong, Zhenwei Miao, and Yue Wang. Carplanner: Consistent auto-regressive trajectory planning for large-scale reinforcement learning in autonomous driving. CVPR, 2025. 1, 3
[14] Zikang Zhou, HU Haibo, Xinhong Chen, Jianping Wang, Nan Guan, Kui Wu, Yung-Hui Li, Yu-Kai Huang, and Chun Jason Xue, Behaviorgpt: Smart agent simulation for autonomous driving with next-patch prediction. NeurlPS, 2024. 2, 3.4
[15] Wei Wu, Xiaoxin Feng, Ziyan Gao, and Yuheng Kan. Smart: Scalable aueti-agent real-time motion generation via next-token prediction. NewrlPS, 2024. 3.4.
[16] Jonah Philion, Xue Bin Peng, and Sanja Fidler. Trajeglish: Traffic modeling as next-token prediction. ICLR, 2024.77 2,
[17] Ari Seff, Brian Cera, Dian Chen, Mason Ng. Aurick Zhou, Nigamaa Nayakanti, Khaled S Refaat, Rami Al-Rfou, and Benjamin Sapp. Motionlm: Multi-agent motion forecasting as language modeling. In ICCV, 2023. 2, 3
[18] Zhejun Zhang, Peter Karkus, Maximilian Igl. Wenhao Ding, Yuxiao Chen, Boris Ivanovic, and Marco Pavone. Closed-loop supervised fine-tuning of tokenized traffic models. CVPR, 2025. 2, 7
[19] Longzhong Lin, Xuewu Lin, Kechun Xu, Haojian Lu, Lichao Huang, Rong Xiong, and Yue Wang. Revisit mixture models for multi-agent simulation: Experimental study within a unified framework. arXiv preprint, 2025. 2

---

[20] Xiaolong Tang, Meina Kan, Shiguang Shan, Zhilong Ji. Jinfeng Bai, and Xilin Chen. Hpnet: Dynamic trajectory forecasting with historical prediction attention. In CVPR, 2024. 3
[21] Nan Song. Bozhou Zhang, Xiatian Zhu, and Li Zhang. Motion forecasting in continuous driving. NeurlPS, 2024. 3
[22] Bozhou Zhang, Nan Song, and Li Zhang. Decoupling motion forecasting into directional intentions and dynamic states. NewrIPS, 2024.
[23] Jie Cheng. Xiaodong Mei, and Ming Liu. Forecast-mae: Self-supervised pre-training for motion forecasting with masked autocncoders. In ICCV, 2023. 3
[24] Zhiqian Lan, Yuxuan Jiang, Yao Mu, Chen Chen, and Shengbo Eben Li. Sept: Towards efficient scene representation learning for motion prediction. In ICLR, 2024. 3
[25] Martin Treiber, Ansgar Hennecke, and Dirk Helbing, Congested traffic states in empirical observations and microscopic simulations. Physical review E, 2000. 3
[26] Daniel Dauner, Marcel Hallgarten, Andreas Geiger, and Kashyap Chitta, Parting with misconceptions about lcarning-hased vehicle motion planning. In CoRT, 2023. 3
[27] Jie Cheng, Yingbing Chen, Xiaodong Mei, Bowen Yang, Bo Li, and Ming Liu. Rethinking imitation-based planners for autonomous driving. In ICRA, 2024. 3
[28] Jie Cheng, Yingbing Chen, and Qifeng Chen. Pluto: Pushing the limit of imitation learning-bass -based planning for autonomous driving. arXiv preprint, 2024. 3
[29] Yuntao Chen, Yuqi Wang, and Zhaoxiang Zhang. Drivinggpt: Unifying driving world modeling and planning with multi-modal autoregressive transformers. arXiv preprint, 2024. 3
[30] Xiaosong Jia, Junqi You, Zhiyuan Zhang, and Junchi Yan. Drivetransformer: Unified transformer for scalable end-to-end autonomous driving. ICLR, 2025. 3
[31] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang. Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint, 2024. 5
[32] Longzhong Lin, Xuewu Lin, Tianwei Lin, Lichao Huang, Rong Xiong, and Yue Wang. Eda: Evolving and distinct anchors for multimodal motion prediction. In AAAI, 2024. 6.7
[33] Qiying Yu, Zheng Zhang, Ruofci Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale, arXiv preprint, 2025. 6,9 9
[34] Longzhong Lin, Xuewu Lin, Kechun Xu, Haojian Lu, Lichao Huang, Rong Xiong, and Yue Wang. Revisit mixture models for multi-agent simulation: Experimental study within a unified framework. In arXiv preprint, 2025. 7
[35] Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. Hdgt: Heterogenenus driving graph transformer for multi-agent trajectory prediction via scene encoding. IEEE TPAMI, 2023.
[36] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Mir+:: Multi-agent motion prediction with symmetric scene modeling and guided intention querying. IEEE TPAMI, 2024.
[37] Yigian Gan, Hao Xiao, Yizhe Zhao, Ethan Zhang, Zhe Huang, Xin Ye, and Lingting Ge. Multi-granular transformer for motion prediction with lidar. ICRA, 2024.
[38] Jiawei Sun, Chengran Yuan, Shuo Sun, Shanze Wang. Yuhang Han, Shuailei Ma, Zefan Huang, Anthony Wong, Keng Peng Tee, and Marcelo H Ang. Controlmtr: guudro motion transformer with scene-compliant intention points for feasible motion prediction. In IEEE ITSC, 2024.
[39] Jiawei Sun, Jiahui Li, Tingchen Liu, Chengran Yuan, Shuo Sun, Zefan Huang, Anthony Wong, Keng Peng Tee, and Marcelo H Ang Jr. Rmp-yolo: A robust motion predictor for partially observable scenarios even if you only look once. arXiv preprint, 2024." 7
[40] Zhiyu Huang, Haochen Liu, Jingda Wu, and Chen Lv. Differentiable integrated motion prediction and planning with leamaalle cost function for autonomous driving. TNNLS, 2023. 7, 8, 9
[41] Tlya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2018.
12

---

A Details for simulation fine-tuning
After obtaining the generated rollout A, and advantage A, we update policy gradient using simplified objective to ensure stable training of lightweight models, the gradient of which for token can be estimated as:
Volog (A,,()) A(A.)
(9)
where Fo denotes our trainable models. Besides, considering the relatively uniform rewards and the large number of iterations, we perform policy update for only one generation per iteration to promote efficient training.
B
Discussions
Why we employ decoder-only Transformers as the backbone? We opt for decoder-only Transformer structure because it better suited for multi-step generative tasks like simulation, where input and output lengths can vary. Conversely, the encoder-decoder structure, while excellent for sequence-to-sqquence prediction tasks, cannot generalize as well across the diverse demands of our multi-task framework. Adopting a decoder-only architecture during joint training ensures stronger generalization capabilities across all our tasks.
Why the joint training UniMotion works well? As detailed in Section 3.1, our primary objectives are NTP and LFR. Intuitively, this joint training paradigm fosters the learning of shared, rich intermediate representations across multiple tasks. This enables tokens within the model to attend simultancously to both their immediate surroundings and more distant scene contexts, leading to more holistic understanding of the environment.
Further, despite their apparent differences, the two task objectives possess crucial commonalities. Specifically, while LFR provides direct supervision on long-range logged trajectories, the initial motion direction embedded within these trajectories concurrently serves the maximization objec- tive for NTP. This underlying consistency and synergistic relationship between LFR and NTP are fundamental to ensuring stable joint training and preventing mode collapse, allowing the model to learn effectively from both detailed local cues and broad long-range dependencies.
C More qualitative results
We provide more qualitative results of our framework in Figure 5 and Figure 6 on the validation set of Waymo Open Motion Dataset [1, 5].
D Failure cases
Although our UniMotion achieves strong performance across multiple motion tasks, still encounters certain failure cases. We provide qualitative results and analysis to offer a more comprehensive understanding, which we hope will support future efforts toward developing more robust and capable algorithms, as shown in Figure 7 and Figure 8.
In simulation task, there is still possibility of collisions in complex scenario illustrated in Figure 7 (a). Besides, we have also observed some unrealistic simulations in intersection scenarios as shown in Figure (b), where agents remain stationary despite the green light. We provide the failure cases of prediction task in Figure 8 (a), which demonstrate that it is challenging to predict complex motion patterns, especially sudden turns. Similar issues are also observed in planning, particularly when dealing with sudden U-turns. Additionally, the model might yield reasonable trajectories that fail to reflect the driver's subjective intention.
13

---

S0
4s
8s
R1
R2
(a) Parking lot scene
R1
R2
(b) Intersection scene
Figure 5: Qualitative results on the Sim Agents. We present several complex scenarios in autonomous driving, each accompanied by two rollouts. The circled areas highlight the key differences between the rollouts.
14

---

BLL
(a) Prediction
Lo
(b) Planning
Figure 6: Qualitative results the WOMD for prediction and planning task.
(a) Collision
(b) Non-red-light stop
Figure 7: Failure cases in simulation. In the first row, the model causes collision in crowded parking lot. In the second row, incorrectly incorrectly stop without a red light at an intersection.
15

---

(a) Prediction
(b) Planning
Figure Failure cases in prediction and planning. The model fails to accurately predict the trajectories of vehicles and pedestrians. In terms of planning, it struggles with scenarios where the targets involve drivers' subjective intentions.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see the experiment section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer:
[No]
Justification:
The details are provided the main paper and the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.o cc/ Btalc//guieess/ooeeee
oniioiicpyc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececccccccccct /ttpa//gguiieeeeeeeuuuuiiiiiiiii /CodeSubmic ssionnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see the experiment section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
19

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: : /ttp///eerrp//ee/ee//ee//e cc hc/pppblitt/thicccsueeeeeee..eel
didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our main idea is specifically designed for applications in autonomous driving.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
20

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please see the experiment section.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
21

---

Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
22

---

Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: : http///rpps..ccoooofenenneeeeeee for what should or should not be described.
23


Reviews:
[{"review_id": "c9SvdNS9ac", "reviewer": "Reviewer_8f42", "summary": "Motion simulation, prediction, and planning are core components of autonomous driving, each critical for understanding and navigating dynamic traffic environments. These tasks are often treated separately due to their distinct goals, leading to the development of specialized models for each. However, this separation limits cross-task generalization and hinders system scalability, while also missing opportunities for shared learning. To address this, the paper introduces UniMotion, a unified framework that captures common structures among motion-related tasks while respecting their individual needs. Experiments on the Waymo Open Motion Dataset show that joint training with UniMotion improves generalization and enables more effective integration across tasks.", "questions": "1. Author could conduct ablations using a more substantial portion of the dataset.\n\n2. How is the model's run-time analysis?\n\n3. Further explanation on suboptimal metrics would helpful.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n1. Integrates simulation, prediction, and planning within a unified framework to encourage generalization and facilitate knowledge sharing across tasks.\n\n2. Introduces tailored fine-tuning strategies that adapt the jointly trained model to individual tasks, enabling it to develop specialized, task-specific capabilities.\n\nWeaknesses:\n\n1. In the ablation experiments, the authors only train on 20% of the training dataset due to efficiency considerations. However, this limited training setup raises concerns about the fairness and reliability of the reported results. Training on such a small subset may fail to capture the full diversity and complexity of the data, potentially skewing the outcomes of the ablation study. The reduced data volume can also lead to high variance in model performance, making it harder to draw meaningful conclusions about the contribution of specific components. To accurately assess each model component, it is important to conduct ablations using a more substantial portion of the dataset, ensuring that the results reflect realistic and generalizable behavior.\n\n2. The run-time performance during testing is a critical metric that should be reported in the paper, particularly for the planning module. In the proposed approach, the planning module relies on a multi-step process. This multi-stage pipeline introduces additional computational overhead compared to standard planning models. Therefore, reporting the run-time of the planning module is essential to assess the practical feasibility of the proposed method and to validate that the added modeling complexity results in meaningful improvements without compromising real-time applicability.\n\n3. In the section on comparison with state-of-the-art methods, the paper would benefit from a more detailed analysis of Tables 1, 2, and 3 to better support its claims. Table 1: While the overall results appear competitive, the performance on minADE is relatively weak compared to other methods. The relatively poor performance here may suggest that the model struggles to generate a highly accurate trajectory among its top candidates. Table 3: The results in Table 3 show suboptimal behavior when reacting to red lights, a critical safety-related scenario. Discussing these metrics in greater depth would lead to a more comprehensive and transparent evaluation of the model's performance.\n\nMinor: Equation 1 is missing closing parentheses.", "was_revised": true, "final_justification": "The authors’ rebuttal addresses some of my concerns regarding the ablation studies and the run-time performance analysis. I encourage the authors to include these additional results and clarifications in the final version. I would like to raise my score to borderline accept.", "rebuttal": "We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:\n\n### **1. Concerns about the fairness and reliability of performing ablation on 20% data.**\n\n**R:** Thanks for the reviewer's rigorous concern. This is mainly due to the constraint of resource available. Ablation on 20% data is acceptable and reasonable as exemplifed in previous perception models [1] and recent motion-related models [2, 3]. As the Waymo dataset offers a much larger volume of data than other datasets (such as Argoverse or nuScenes), even 20% of the data can reflect the overall distribution well. We thus followed this practice.\n\n\nBesides, we have now conducted the ablation for Tab. 5 on the full data. The results below show consistent observation as that with 20% data.\n\n\n|simulation|Kinematic|Interactive|Map-based|minADE|\n| ---- | ---- | ---- | ---- | ---- |\n|w/o ft.|0.4915|0.8047|0.9162|1.3138|\n|w/ ft.|**0.4948**|**0.8098**|**0.9181**|**1.3055**|\n\n|prediction|minADE|minFDE|MR|mAP|\n| ---- | ---- | ---- | ---- | ---- |\n|w/o ft.|0.5752|1.1688|0.1245|0.4019|\n|w/ ft.|**0.5671**|**1.1562**|**0.1163**|**0.4635**|\n\n|planning|Collision|Err @1s|Err @3s|Err @5s|\n| ---- | ---- | ---- | ---- | ---- |\n|w/o ft.|1.7347|0.1326|0.7943|2.7809|\n|w/ ft.|**1.5650**|**0.0834**|**0.5908**|**2.2455**|\n\n\n\n### **2. Run-time performance during testing should be reported, particularly for the planning module.**\n\n\n**R:** Apoligies for lack of run-time analysis. We have now provided the inference time on RTX 3090 for all tasks as below. Please note, for the planning latency, we perform auto-regressive generation to yield complete trajectories for offline evaluation, while in real-world scenarios, per-step planning (online) happens. \n\nWe will add this to the revised version.\n\nSimulation one-step inference latency\n| Method | Latency |\n| ---- | ---- |\n|UniMM|33ms|\n|CATK|~20ms|\n|UniMotion|24ms|\n\nPrediction inference latency\n| Method | Latency |\n| ---- | ---- |\n|MTR|62ms|\n|EDA|62ms|\n|RMP-YOLO|100ms|\n|UniMotion|89ms|\n\nPlanning inference latency\n| Method | Latency |\n| ---- | ---- |\n|DIPP|1.78s|\n|UniMotion(online)|76ms|\n|UniMotion(offline)|468ms|\n\n\n### **3. The paper would benefit from a more detailed analysis of comparison with state-of-the-art methods to better support its claims.**\n\n\n**R:** Thanks, and we will further elaborate: \nFor Tab. 1, different from prediction task, simulation task focuses more on generative ability of motion models, and is not particularly concerned with accurately predicting the logged trajectories (not involved in Realism Meta Metric). Hence, we adopt the tokenization-based NTP designs, which select trajectory segments from clustered trajectory token vocabulary, hence showing stronger generative performance but inferior in accurate prediction over the regression-based methods.\n\nFor Tab. 3, DIPP adopts a tailored planning cost function that involves  many rules, such as velocity, comfort and red light. To ensure generality and consistency, we just supervise training with logged trajectories, and this metric can be further improved by involving traffic rules, like [4, 5].\n\nThanks. We have revised the formatting errors in Equation 1.\n\n\n>[1] Embracing Single Stride 3D Object Detector with Sparse Transformer. Lue Fan et al. CVPR 2022\n\n>[2] BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction. Zikang Zhou et al. NeurIPS 2024\n\n>[3] Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Models. Zhejun Zhang et al. CVPR 2025\n\n>[4] Rethinking Imitation-based Planners for Autonomous Driving. Jie Cheng et al. ICRA 2024 \n\n>[5] PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Driving. Jie Cheng et al. arXiv preprint"}, {"review_id": "8CSBnw2dQj", "reviewer": "Reviewer_Rodt", "summary": "This paper presents UniMotion, a unified framework for three major motion tasks in autonomous driving: simulation, prediction, and planning. These tasks are typically addressed in isolation due to their differing objectives, but this work observes that they share underlying modeling requirements, such as multi-agent interactions and temporal-spatial reasoning.\n\nUniMotion leverages a decoder-only Transformer architecture, inspired by GPT-style models, to model all three tasks within a single framework. It introduces task-specific attention masks and training objectives (Next Token Prediction for simulation, Long-range Future Regression for prediction) and supports dedicated fine-tuning for individual task specialization. Experiments on the Waymo Open Motion Dataset show that UniMotion achieves strong joint performance and even surpasses state-of-the-art methods in each task after fine-tuning.", "questions": "1. Why use decoder-only Transformers for all tasks? Would encoder-decoder structures (e.g., for prediction or planning) offer benefits in terms of stability or sample efficiency? The choice is reasonable but could benefit from comparative justification.\n\n2. How is the method's generalization capability to out-of-domain data (e.g., a new dataset)?\n\n3. Could the authors provide some theoretical discussion on why joint training works well?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n1. Unified Viewpoint and Framework. A major conceptual contribution is the abstraction of motion tasks into two categories—diverse motion generation and long-range trajectory forecasting—enabling a coherent formulation. Unifying simulation, prediction, and planning within a single model reduces redundancy and potentially boosts cross-task generalization.\n\n2. Technical Soundness and Design. The adoption of decoder-only Transformers with task-aware masking is principled and well-motivated by analogies to language modeling. Fine-tuning strategies, especially the reinforcement learning–based simulation refinement and multi-modal prediction decoder, are thoughtfully designed and justified.\n\n3. Strong Empirical Results. Extensive benchmarks on the WOMD and Sim Agents show that UniMotion is competitive or state-of-the-art in all three tasks. Ablation studies convincingly show the value of joint training and fine-tuning, with meaningful gains reported in metrics like planning error and simulation realism.\n\n**Weaknesses**:\n\n1. The experiments only involve a single dataset, Waymo Open Motion Dataset, which could raise questions on the generalization capability of this method to new, unseen data.\n\n2. In all 3 benchmarks, the method delivers sub-optimal results in at least one of the metrics.\n\n3. Considering the sub-optimal results, the paper could benefit from some quantitative or qualitative analysis on some of the failure cases, which could provide more insight than average result numbers.\n\n4. The qualitative results in Fig. 4 only present the results of the proposed method. A qualitative comparison with other methods should be provided in order to demonstrate the strength of the proposed method.", "was_revised": true, "final_justification": "The authors have addressed all my concerns by providing new experiments and new analysis. Therefore, I'm maintaining my positive score.", "rebuttal": "We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:\n\n\n### **1. Weakness 1 and Q2: The experiments only involve a single dataset, which could raise questions on the generalization capability of this method.**\n\n\n**R:** Excellent suggestion! We have now conducted experiments on transferability from Waymo to nuPlan (Val14), and the results are presented below.\n\nOur findings indicate two key points:\n- As expected, zero-shot performance is less than satisfactory due to inherent data distribution discrepancies (especially for motion token vocabulary) between the two datasets.\n- Crucially, our model still achieves decent performance after fine-tuning, even if it is slightly inferior to methods specifically tailored for the nuPlan dataset.\n\nThese results verify the generality and adaptability of our solution across different autonomous driving datasets. We will include this transferability test in the revised paper.\n\n\n| Method | NR-CLS  | R-CLS  |\n| ---- | ---- | ---- |\n|PlanTF|0.85|0.77|\n|UniMotion(zero-shot)|0.66|0.62|\n|UniMotion(fine-tuned)|0.81|0.75|\n\n### **2. Weakness 2 and 3: The method delivers sub-optimal results in at least one of the metrics, and the paper could benefit from some quantitative or qualitative analysis on some of the failure cases.**\n\n**R:** Thanks. UniMotion consistently ranks among the top two across various metricc; For some results with relatively large differences, we provide more analysis below.\n\nTab. 1 minADE: Different from prediction task, simulation task focuses more on generative ability of motion models. Hence, we adopt the tokenization-based NTP designs, which select trajectory segments from clustered trajectory token vocabulary, hence showing stronger generative performance but inferior in accurate prediction over the regression-based methods.\n\nTab. 3 red light: DIPP adopts a tailored planning cost function that involves many rules, such as velocity, comfort and red light. To ensure generality and consistency, we just supervise training with logged trajectories, and this metric can be further improved by involving traffic rules.\n\nBesides, we have already provided more analysis of specific failure cases in the supplementary material, which the reviewer could refer to.\n\n### **3. Weakness 4: A qualitative comparison with other methods should be provided in order to demonstrate the strength of the proposed method.**\n \n**R:** Great suggestion. Our intention in Figure 4 was to showcase the ability of our method in handling various motion tasks, while overlooking the comparison with other methods. We will supplement this part accordingly.\n\n### **4. Q1: Why use decoder-only Transformers for all tasks？**\n\n**R:** Great point, we will clarify:\nWe opted for a decoder-only Transformer structure because it's better suited for multi-step generative tasks like simulation, where input and output lengths can vary. Conversely, the encoder-decoder structure, while excellent for sequence-to-sequence prediction tasks, doesn't generalize as well across the diverse demands of our multi-task framework. Adopting a decoder-only architecture during joint training ensures stronger generalization capabilities across all our tasks.\n\nFor the prediction fine-tuning stage, we do introduce an additional multi-modal decoder to handle richer multi-modal outputs, which can be seen as a encoder-decoder structure.\n\n### **5. Q3: Could the authors provide some theoretical discussion on why joint training works well?**\n\n\n**R:** Thanks, we will elaborate on the theoretical underpinnings of our joint training approach: \nAs detailed in Section 3.1, our primary objectives are NTP and LFR. Intuitively, this joint training paradigm fosters the learning of shared, rich intermediate representations across multiple tasks. This enables tokens within the model to attend simultaneously to both their immediate surroundings and more distant scene contexts, leading to a more holistic understanding of the environment.\n\nFurther, despite their apparent differences, the two task objectives possess crucial commonalities. Specifically, while LFR provides direct supervision on long-range logged trajectories, the initial motion direction embedded within these trajectories concurrently serves as the maximization objective for NTP. This underlying consistency and synergistic relationship between LFR and NTP are fundamental to ensuring stable joint training and preventing mode collapse, allowing the model to learn effectively from both detailed local cues and broad long-range dependencies."}, {"review_id": "AdfsvklVIv", "reviewer": "Reviewer_Bhcg", "summary": "This paper abstracts motion tasks into two core categories: diverse motion generation and long-range trajectory forecasting, based on which it proposes UniMotion, a unified Transformer-based framework that jointly models simulation, prediction and planning for better generality and cross-task knowledge sharing. It designs effective fine-tuning strategies that specialize the jointly trained model for individual tasks. Experiments on the Waymo Open Motion Dataset (WOMD) show that UniMotion achieves competitive performance under joint training and fine-tuning. The primary insight of UniMotion is that a unified framework enables knowledge sharing across various motion tasks. This insight aligns with the advantages of multi-task learning.", "questions": "1. In ADS, trajectory prediction targets the surrounding agents, and motion planning focuses on the ego vehicle. While traditionally they are trained separately due to different input/output requirements, recent works increasingly merge prediction and planning. For instance, Systems like DIPP and PRIME unify prediction and planning modules, enabling the planner to react to probabilistic forecasts while learning cost functions from data. Another example is interaction-aware planning: Treats prediction and planning as interdependent to avoid conservative behaviors caused by assuming static predictions.  What are the methodological differences between UniMotion and these existing works? \n\n2. The paper claims that the fine-tuned models outperform the corresponding Waymo open motion challenge leaderboard models. Nevertheless, the models compared in this paper differ from the top models published on the 2025 Waymo Open Dataset Challenges website (https://waymo.com/open/challenges/). It is thus unclear whether the claimed advantages in the paper remain valid.  \n\n3. While intuitively, multitask learning enables joint optimization and representation sharing, more discussion is needed on how different motion tasks in the context of ADS, which have quite different targets and input/output requirements, share \"knowledge.\" Please be more specific and provide explicit details.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength: The paper presents a unified framework to train motion tasks in autonomous driving systems. Such a framework can leverage the underlying commonality of different motion tasks for better knowledge sharing. When the jointly trained model is fine-tuned for a specific motion task, the fine-tuned model has been shown to outperform state-of-the-art models trained independently and specifically for the given motion task. \n\nWeakness: \n(1) UniMotion must be fine-tuned to work well for a given motion task in ADS. This two-phase procedure, involving joint training and fine-tuning, significantly increases the model's complexity and computational cost for a specific motion task.  Nevertheless, this critical issue has not been evaluated in the paper. \n(2) The paper claims that the fine-tuned models outperform the corresponding Waymo open motion challenge leaderboard models. Nevertheless, the models compared in this paper differ from the top models published on the 2025 Waymo Open Dataset Challenges website (https://waymo.com/open/challenges/). It is thus unclear whether or not the claimed advantages in the paper are valid.", "was_revised": true, "final_justification": "The author's rebuttal has addressed my concerns and I have raised my score to 4.", "rebuttal": "We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:\n\n\n### **1. Weakness 1: This two-phase procedure significantly increases the model's complexity and computational cost for a specific motion task.**\n**R:** A great comment regarding our two-phase procedure! However, all our fine-tuned models originate from the same jointly trained model, and we introduce no additional parameters during the fine-tuning stage (except for the prediction task). This means it barely affects overall model complexity. For overall training cost, this approach can often be more efficient because the joint training, typically the most expensive part, is shared across all tasks, and subsequent fine-tuning is comparatively cheaper.\n\nFor full transparency, we've provided a statistics table detailing our model size and computational time below.\n| Task | size |  training time (A6000)|inference time (RTX3090)|\n| ---- | ---- |  ---- |  ---- |\n|simulation|7M|23h+13h|24ms|\n|prediction|7M+6M|23h+8h|89ms|\n|planning|7M|23h+10h|76ms|\n\n### **2. Weakness 2 and Q2: The models compared in this paper differ from the top models published on the 2025 Waymo Open Dataset Challenges website.**\n\n**R:** We regret that we couldn't participate in the award selection for the 2025 Waymo Challenges. However, for a direct comparison, the reviewer can find the relevant leaderboard by appending \"2025/sim-agents\" to the Waymo Challenges website (we're unable to provide external links directly in this response).\n\nAt the time of our NeurIPS submission, our method achieved an RMM of **0.7851**, which is directly comparable to the first-place approach TrajTok (RMM **0.7852**) in the 2025 Waymo Challenges.\n\nAdditionally, it's worth noting that there is no official motion prediction track in the 2025 challenges. Therefore, our motion prediction comparisons were conducted against the 2024 leaderboard to ensure a relevant benchmark.\n\n### **3. Q1: What are the methodological differences between UniMotion and these existing works?**\n\n**R:** Thanks, we will clarify:\nThe key methodological difference lies in our scope: the metioned works primarily concentrate on optimising the planning task. In contrast, UniMotion proposes a unified motion framework designed to handle all motion tasks (prediction, planning, and simulation) within a single architecture. Therefore, our overall methodologies are largely orthogonal.\n\nRegarding the specific design for planning fine-tuning, our method adopts an agent trajectory prediction stage followed by ego planning generation, which can be regarded as an extension of existing approaches where the planner reacts to predictions.\n\n### **4. Q3: More discussion is needed on how different motion tasks in the context of ADS, which have quite different targets and input/output requirements, share \"knowledge\".**\n\n**R:** Thanks, we will further elaborate as below.\n\nIn fact, these tasks share significant commonalities in their underlying input and output information. For instance, all these tasks typically process agent trajectories and high-definition (HD) maps as input, and ultimately aim to output future trajectories for scene agents. This shared informational foundation is precisely what enables our unified model. The main differences, as discussed in Section 3.1, lie in the varying demands placed on these future trajectories — whether it's for diverse motion generation or accurate long-horizon prediction.\n\nFurthermore, while the specific objectives of tasks like LFR and NTP appear distinct, they actually share crucial underlying commonalities. LFR directly supervises the model with long-range logged trajectories, and critically, the initial motion direction embedded within these trajectories simultaneously serves as the maximization objective for NTP. This inherent consistency and synergistic relationship ensures stable joint training, allowing the model to effectively leverage shared knowledge across different motion challenges."}, {"review_id": "Bise7dl31e", "reviewer": "Reviewer_uSmG", "summary": "This work allows a transformer-based model to unify the skills needed for three common self-driving tasks, including trajectory prediction, agent simulation, and ego-car planning. This base model then follows three distinct post-training/fine-tuning methods to tailor itself for the three tasks individually. The SOTA benchmark results on Waymo prediction, Sim Agent Challenge, and Waymo planning show the effectiveness of the recipe. Ablation study further highlights the effectiveness of each training design and the three post-training strategies.", "questions": "What is the size of the model? If it is too large, then we would still stick to individual solutions for specific tasks.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strength\n1. The nature of the three common self-driving tasks is the same: trajectory prediction. Thus, they can be addressed together, which is a good motivation for this work.\n2. The method itself follows the standard LLM training paradigm with transformer backbone, next-token prediction, and RL-based post-training. It makes it easy to reuse the mature infrastructure of LLM development for training a UniMotion model.\n3. The experiment is thorough and shows SOTA performance, and the ablation study clearly shows why a certain module/process works.\n\n### Weakness\n1. The main weakness is the evaluation of planning ability. It is tested on a benchmark that is not commonly used and verified by the community. Also, the benchmark has some drawbacks, like log-replayed traffic agents and a lack of support for long-horizon closed-loop planning. In other words, this benchmark is only closed-loop for the ego-car itself in a short time window. Thus, the results on this benchmark can not fairly reflect the ability of the UniMotion model. I understand that it is hard to do evaluations for the three tasks within one domain, like nuPlan is tailored for planning, and Waymo only officially supports the sim agent and prediction. To be honest, I can not suggest a better way to evaluate the model, especially considering that training and deploying on nuPlan is quite time-consuming. Thus, I view this weakness as an impossible-to-address one.", "was_revised": true, "final_justification": "With the new results provided, the paper's claim is well-supported. I tend to accept this paper.", "rebuttal": "We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:\n\n\n### **1. Authors could provide a better way to evaluate the model.**\n\n**R:** We appreciate the reviewer's understanding of this challenge. Jointly training a model across multiple datasets and tasks for motion prediction incurs substantial computational costs. For this foundational work, we focused on successfully addressing multiple tasks within a single dataset. Expanding our evaluation across diverse datasets is a valuable direction we've left for future work.\n\nFurther, we have now conducted experiments on transferability from Waymo to nuPlan (Val14), and the results are presented below. Our findings indicate two key points:\n- As expected, zero-shot performance is less than satisfactory due to inherent data distribution discrepancies (especially for motion token vocabulary) between the two datasets.\n- Crucially, our model still achieves decent performance after fine-tuning, even if it is slightly inferior to methods specifically tailored for the nuPlan dataset.\n\nThese results verify the generality and adaptability of our solution across different autonomous driving scenarios. We will include this transferability test in the revised paper.\n\n| Method | NR-CLS  | R-CLS  |\n| ---- | ---- | ---- |\n|PlanTF|0.85|0.77|\n|UniMotion(zero-shot)|0.66|0.62|\n|UniMotion(fine-tuned)|0.81|0.75|\n\n\n\n### **2. What is the size of the model?**\n\n**R:** Thanks, we will clarify: We follow the lightweight Transformer design with factorized attentions and the model size is approximately 7M."}]

Author rebuttal:

Meta-review: After a thorough review of all reviewers' comments, the authors' detailed rebuttal, and subsequent discussions, we are pleased to recommend accepting this manuscript for publication.  

The reviewers raised several critical concerns that are central to the rigor and impact of the work, including the reliability of ablation studies (Reviewer 8f42), generalization capability and qualitative analysis (Reviewer Rodt), model complexity and computational costs (Reviewer Bhcg), and the evaluation of planning ability (Reviewer uSmG). These concerns directly address core aspects of experimental robustness, method generalizability, and technical clarity—standards that are essential for validating contributions in this field.  

Notably, the authors have responded to each concern with targeted and actionable improvements: They supplemented full-data ablation experiments to strengthen the reliability of their findings, conducted transfer experiments to explicitly demonstrate generalization across scenarios, provided statistical tables detailing model size and runtime to clarify computational feasibility, and enhanced the evaluation framework with additional datasets to validate planning ability. These revisions are not merely superficial responses but substantive additions that address the root of the reviewers’ questions, thereby elevating the work’s methodological rigor and interpretability.  

The reviewers’ recognition of the effectiveness of these responses further confirms that the manuscript has been significantly strengthened. The work now stands on a more solid foundation, with clearer experimental logic, more comprehensive validation, and improved alignment with the field’s standards for reproducibility and generalizability.  

We encourage the authors to integrate the promised revisions (e.g., supplementary qualitative comparisons) into the final manuscript to ensure completeness. This work, with its enhanced rigor, makes a valuable contribution to the field, and we are confident in its suitability for publication.
Decision: Accept (poster)