Title: UniMotion: A Unified Motion Framework for Simulation, Prediction and Planning
Venue: NeurIPS 2025
Accepted: yes
Keywords: autonomous driving; motion simulation; trajectory prediction; planning
Abstract: Motion simulation, prediction and planning are foundational tasks in autonomous driving, each essential for modeling and reasoning about dynamic traffic scenarios. While often addressed in isolation due to their differing objectives, such as generating diverse motion states or estimating optimal trajectories, these tasks inherently depend on shared capabilities: understanding multi-agent interactions, modeling motion behaviors, and reasoning over temporal and spatial dynamics.
Despite this underlying commonality, existing approaches typically adopt specialized model designs, which hinders cross-task generalization and system scalability. More critically, this separation overlooks the potential mutual benefits among tasks. Motivated by these observations, we propose **UniMotion**, a unified motion framework that captures shared structures across motion tasks while accommodating their individual requirements. Built on a decoder-only Transformer architecture, UniMotion employs dedicated interaction modes and tailored training strategies to simultaneously support these motion tasks. This unified design not only enables joint optimization and representation sharing but also allows for targeted fine-tuning to specialize in individual tasks when needed. Extensive experiments on the Waymo Open Motion Dataset (WOMD) demonstrate that joint training leads to robust generalization and effective task integration. With further fine-tuning, UniMotion achieves state-of-the-art performance across a range of motion tasks, establishing it as a versatile and scalable solution for autonomous driving.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work allows a transformer-based model to unify the skills needed for three common self-driving tasks, including trajectory prediction, agent simulation, and ego-car planning. This base model then follows three distinct post-training/fine-tuning methods to tailor itself for the three tasks individually. The SOTA benchmark results on Waymo prediction, Sim Agent Challenge, and Waymo planning show the effectiveness of the recipe. Ablation study further highlights the effectiveness of each training design and the three post-training strategies.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n1. The nature of the three common self-driving tasks is the same: trajectory prediction. Thus, they can be addressed together, which is a good motivation for this work.\n2. The method itself follows the standard LLM training paradigm with transformer backbone, next-token prediction, and RL-based post-training. It makes it easy to reuse the mature infrastructure of LLM development for training a UniMotion model.\n3. The experiment is thorough and shows SOTA performance, and the ablation study clearly shows why a certain module/process works.\n\n### Weakness\n1. The main weakness is the evaluation of planning ability. It is tested on a benchmark that is not commonly used and verified by the community. Also, the benchmark has some drawbacks, like log-replayed traffic agents and a lack of support for long-horizon closed-loop planning. In other words, this benchmark is only closed-loop for the ego-car itself in a short time window. Thus, the results on this benchmark can not fairly reflect the ability of the UniMotion model. I understand that it is hard to do evaluations for the three tasks within one domain, like nuPlan is tailored for planning, and Waymo only officially supports the sim agent and prediction. To be honest, I can not suggest a better way to evaluate the model, especially considering that training and deploying on nuPlan is quite time-consuming. Thus, I view this weakness as an impossible-to-address one.\n\nQUESTIONS:\nWhat is the size of the model? If it is too large, then we would still stick to individual solutions for specific tasks.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper abstracts motion tasks into two core categories: diverse motion generation and long-range trajectory forecasting, based on which it proposes UniMotion, a unified Transformer-based framework that jointly models simulation, prediction and planning for better generality and cross-task knowledge sharing. It designs effective fine-tuning strategies that specialize the jointly trained model for individual tasks. Experiments on the Waymo Open Motion Dataset (WOMD) show that UniMotion achieves competitive performance under joint training and fine-tuning. The primary insight of UniMotion is that a unified framework enables knowledge sharing across various motion tasks. This insight aligns with the advantages of multi-task learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: The paper presents a unified framework to train motion tasks in autonomous driving systems. Such a framework can leverage the underlying commonality of different motion tasks for better knowledge sharing. When the jointly trained model is fine-tuned for a specific motion task, the fine-tuned model has been shown to outperform state-of-the-art models trained independently and specifically for the given motion task. \n\nWeakness: \n(1) UniMotion must be fine-tuned to work well for a given motion task in ADS. This two-phase procedure, involving joint training and fine-tuning, significantly increases the model's complexity and computational cost for a specific motion task.  Nevertheless, this critical issue has not been evaluated in the paper. \n(2) The paper claims that the fine-tuned models outperform the corresponding Waymo open motion challenge leaderboard models. Nevertheless, the models compared in this paper differ from the top models published on the 2025 Waymo Open Dataset Challenges website (https://waymo.com/open/challenges/). It is thus unclear whether or not the claimed advantages in the paper are valid.\n\nQUESTIONS:\n1. In ADS, trajectory prediction targets the surrounding agents, and motion planning focuses on the ego vehicle. While traditionally they are trained separately due to different input/output requirements, recent works increasingly merge prediction and planning. For instance, Systems like DIPP and PRIME unify prediction and planning modules, enabling the planner to react to probabilistic forecasts while learning cost functions from data. Another example is interaction-aware planning: Treats prediction and planning as interdependent to avoid conservative behaviors caused by assuming static predictions.  What are the methodological differences between UniMotion and these existing works? \n\n2. The paper claims that the fine-tuned models outperform the corresponding Waymo open motion challenge leaderboard models. Nevertheless, the models compared in this paper differ from the top models published on the 2025 Waymo Open Dataset Challenges website (https://waymo.com/open/challenges/). It is thus unclear whether the claimed advantages in the paper remain valid.  \n\n3. While intuitively, multitask learning enables joint optimization and representation sharing, more discussion is needed on how different motion tasks in the context of ADS, which have quite different targets and input/output requirements, share \"knowledge.\" Please be more specific and provide explicit details.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents UniMotion, a unified framework for three major motion tasks in autonomous driving: simulation, prediction, and planning. These tasks are typically addressed in isolation due to their differing objectives, but this work observes that they share underlying modeling requirements, such as multi-agent interactions and temporal-spatial reasoning.\n\nUniMotion leverages a decoder-only Transformer architecture, inspired by GPT-style models, to model all three tasks within a single framework. It introduces task-specific attention masks and training objectives (Next Token Prediction for simulation, Long-range Future Regression for prediction) and supports dedicated fine-tuning for individual task specialization. Experiments on the Waymo Open Motion Dataset show that UniMotion achieves strong joint performance and even surpasses state-of-the-art methods in each task after fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. Unified Viewpoint and Framework. A major conceptual contribution is the abstraction of motion tasks into two categories—diverse motion generation and long-range trajectory forecasting—enabling a coherent formulation. Unifying simulation, prediction, and planning within a single model reduces redundancy and potentially boosts cross-task generalization.\n\n2. Technical Soundness and Design. The adoption of decoder-only Transformers with task-aware masking is principled and well-motivated by analogies to language modeling. Fine-tuning strategies, especially the reinforcement learning–based simulation refinement and multi-modal prediction decoder, are thoughtfully designed and justified.\n\n3. Strong Empirical Results. Extensive benchmarks on the WOMD and Sim Agents show that UniMotion is competitive or state-of-the-art in all three tasks. Ablation studies convincingly show the value of joint training and fine-tuning, with meaningful gains reported in metrics like planning error and simulation realism.\n\n**Weaknesses**:\n\n1. The experiments only involve a single dataset, Waymo Open Motion Dataset, which could raise questions on the generalization capability of this method to new, unseen data.\n\n2. In all 3 benchmarks, the method delivers sub-optimal results in at least one of the metrics.\n\n3. Considering the sub-optimal results, the paper could benefit from some quantitative or qualitative analysis on some of the failure cases, which could provide more insight than average result numbers.\n\n4. The qualitative results in Fig. 4 only present the results of the proposed method. A qualitative comparison with other methods should be provided in order to demonstrate the strength of the proposed method.\n\nQUESTIONS:\n1. Why use decoder-only Transformers for all tasks? Would encoder-decoder structures (e.g., for prediction or planning) offer benefits in terms of stability or sample efficiency? The choice is reasonable but could benefit from comparative justification.\n\n2. How is the method's generalization capability to out-of-domain data (e.g., a new dataset)?\n\n3. Could the authors provide some theoretical discussion on why joint training works well?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMotion simulation, prediction, and planning are core components of autonomous driving, each critical for understanding and navigating dynamic traffic environments. These tasks are often treated separately due to their distinct goals, leading to the development of specialized models for each. However, this separation limits cross-task generalization and hinders system scalability, while also missing opportunities for shared learning. To address this, the paper introduces UniMotion, a unified framework that captures common structures among motion-related tasks while respecting their individual needs. Experiments on the Waymo Open Motion Dataset show that joint training with UniMotion improves generalization and enables more effective integration across tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1. Integrates simulation, prediction, and planning within a unified framework to encourage generalization and facilitate knowledge sharing across tasks.\n\n2. Introduces tailored fine-tuning strategies that adapt the jointly trained model to individual tasks, enabling it to develop specialized, task-specific capabilities.\n\nWeaknesses:\n\n1. In the ablation experiments, the authors only train on 20% of the training dataset due to efficiency considerations. However, this limited training setup raises concerns about the fairness and reliability of the reported results. Training on such a small subset may fail to capture the full diversity and complexity of the data, potentially skewing the outcomes of the ablation study. The reduced data volume can also lead to high variance in model performance, making it harder to draw meaningful conclusions about the contribution of specific components. To accurately assess each model component, it is important to conduct ablations using a more substantial portion of the dataset, ensuring that the results reflect realistic and generalizable behavior.\n\n2. The run-time performance during testing is a critical metric that should be reported in the paper, particularly for the planning module. In the proposed approach, the planning module relies on a multi-step process. This multi-stage pipeline introduces additional computational overhead compared to standard planning models. Therefore, reporting the run-time of the planning module is essential to assess the practical feasibility of the proposed method and to validate that the added modeling complexity results in meaningful improvements without compromising real-time applicability.\n\n3. In the section on comparison with state-of-the-art methods, the paper would benefit from a more detailed analysis of Tables 1, 2, and 3 to better support its claims. Table 1: While the overall results appear competitive, the performance on minADE is relatively weak compared to other methods. The relatively poor performance here may suggest that the model struggles to generate a highly accurate trajectory among its top candidates. Table 3: The results in Table 3 show suboptimal behavior when reacting to red lights, a critical safety-related scenario. Discussing these metrics in greater depth would lead to a more comprehensive and transparent evaluation of the model's performance.\n\nMinor: Equation 1 is missing closing parentheses.\n\nQUESTIONS:\n1. Author could conduct ablations using a more substantial portion of the dataset.\n\n2. How is the model's run-time analysis?\n\n3. Further explanation on suboptimal metrics would helpful.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}