DeMo: Decoupling Motion Forecasting into  Directional Intentions and Dynamic States
Authors: Bozhou Zhang
Nan Song
Li Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Motion Forecasting
Autonomous Driving
Mamba
Attention
Primary area: robotics
TL;DR: A framework that decouples multi-modal trajectory queries into mode queries for directional intentions and state queries for dynamic states, utilizing Attention and Mamba.

Abstract: Accurate motion forecasting for traffic agents is crucial for ensuring the safety and efficiency of autonomous driving systems in dynamically changing environments. Mainstream methods adopt a one-query-one-trajectory paradigm, where each query corresponds to a unique trajectory for predicting multi-modal trajectories. While straightforward and effective, the absence of detailed representation of future trajectories may yield suboptimal outcomes, given that the agent states dynamically evolve over time. To address this problem, we introduce DeMo, a framework that decouples multi-modal trajectory queries into two types: mode queries capturing distinct directional intentions and state queries tracking the agent's dynamic states over time. By leveraging this format, we separately optimize the multi-modality and dynamic evolutionary properties of trajectories. Subsequently, the mode and state queries are integrated to obtain a comprehensive and detailed representation of the trajectories. To achieve these operations, we additionally introduce combined Attention and Mamba techniques for global information aggregation and state sequence modeling, leveraging their respective strengths. Extensive experiments on both the Argoverse 2 and nuScenes benchmarks demonstrate that our DeMo achieves state-of-the-art performance in motion forecasting. In addition, we will make our code and models publicly available.

DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic States
Bozhou Zhang
Nan Song
Li Zhang"
School of Data Science, Fudan University
https://github.
DeMo
Abstract
Accurate motion forecasting for traffic agents crucial for ensuring the safety and efficiency of autonomous driving systems in dynamically changing environments. Mainstream methods adopt a corresponds to unique trajectory for predicting multi-modal trajectories. While straightforward and effective, the absence of detailed representation of future tra- jectories may yield suboptimal outcomes, given that the agent states dynamically evolve over time. To address this problem, we introduce DeMo, a framework that decouples multi-modal trajectory queries into two types: mode queries capturing distinct directional intentions and state queries tracking the agent's ' dynamic states over time. By leveraging this format, separately optimize the multi-modalityy and dynamic evolutionary properties of trajectories. Subsequently, the mode and state queries are integrated obtain comprehensive and detailed representa- tion of the trajectories. To achieve these operations, we additionally introduce combined Attention and Mamba techniques for global information aggregation and state sequence modeling, leveraging their respective strengths. Extensive ex- periments both the Argoverse 2 and nuScenes benchmarks demonstrate that our DeMo achieves state-of-the performance in motion forecasting.
paradigm, where each query
Introduction
Motion forecasting [29, 58, 67] empowers self-driving vehicles to anticipate how surrounding agents will move and affect the ego car, providing references and conditions for the ego-action. It critical for maintaining safety and dependability, enabling vehicles to comprehend the dynamics of driving environments and make calculated decisions. The challenges and complexities of this task arise from various factors, including unpredictable road conditions, varied movement patterns of traffic participants, and the necessity simultaneously analyze the states of observed agents along with the road maps.
The research community has witnessed significant progress in the representation of driving scenes [18, 36, 44] and the paradigm of trajectory decoding [26, 38, 54, 77]. These methods have achieved substantial adyancements in prediction accuracy, primarily following certain pattern inspired from detection [4, 40], i.e., the one-query-one -e--uure--o---rrjccttryy paradigm [38, 54, 59, 77J. This paradigm utilizes several queries represent different estimated trajectories, as shown in Figure I (a), covering the possibilities of distinct motion intentions. Although effective, these approaches can only approximately provide a direction and collect surroundings to generate various trajectory waypoints in a she-shot fashion, overlooking the detailed relationships with scenes. The lack of concrete representation for trajectories and comprehensive spatiotemporal interactions with the surrounding
Zhang (lizhangfd @ fudan. ceddan.edu.cn) the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

A
(a) Previous Methods
(b) State Queries
(c) Our DeMo
Figure 1: The primary distinction between previous methods and ours lies in the representation of future trajectories. Previous methods, as depicted in (a), use only one mode query for each trajectory. Our approach, illustrated in (c), utilizes decoupled mode queries, as shown in (a), and state queries, as shown (b), to represent the multi-modal trajectories.
environment and among each other might lead decline in accuracy and consistency across varying time steps.
To solve this problem, we propose a novel framework dubbed DeMo, which provides a detailed representation of multi-modal trajectories. Specifically, we decouple forecasting queries into two types: besides the original motion mode queries to capture different directional intentions as shown Figure (a), we introduce the dynamic state queries for future trajectories to track the agent's dynamic states across various time steps, shown in Figure 1 (b). This approach allows achieve a comprehensive query representation within our framework. as illustrated in Figure I (c). Mode queries and state queries processed using the Mode Localization Module and the State Consistency Module, respectively. These modules enable the explicit interactions of queries with the surrounding environments and among each other, by which the directional accuracy and temporal consistency of future trajectories are significantly optimized. Subsequently, two types of queries integrated by our Hybrid Coupling Module to achieve comprehensive representation of future trajectories. Due to the sequential nature of trajectory states, Mamba is particularly well-suited for modeling the temporal consistency of dynamic states. Therefore, we utilize combination of Attention and Mamba our modules to effectively and efficiently aggregate global information and model state sequences, leveraging the strengths of both techniques.
to
Our contributions are summarized as follows: (i) We propose a motion forecasting framework that decouples multi-modal trajectory queries into mode queries and state queries represent directional intentions and dynamic states, respectively. (ii) We design three modules based on integrated Attention and Mamba process decoupled mode queries, state queries, and coupled mode and state queries. (iii) Extensive experiments on both the Argoverse and nuScenes benchmarks demonstrate that DeMo achieves state-of-the-art performance.
2 Related work
Motion forecasting. In recent advancements in autonomous driving, it is critical to effectively predict the movements of relevant agents by accurately representing scene components. Tradi- tional methods [5, 20, 48] transformed driving scenarios into image formats and used conventional convolutional networks for scene context encoding. However, these techniques often failed to suffi- ciently capture intricate structural details. This challenge has led to the adoption of vectorized scene representations [26, 60, 75, 79], exemplified by the introduction of VectorNet [18]. Additionally, graph-based structures are also widely utilized to represent the relationships between agents and their environments [14, 21, 30, 31, 36, 50, 69].
Existing methodologies have delved into a variety of frameworks to predict multi-modaa future trajectories given the scene features. Initially, prediction techniques were centered on goal-oriented methods [26, 71] or employed probability heatmaps to sample trajectories [20, 21]. However, contemporary strategies, such as MTR [54] and QCNet [77], among others [41, 43, 44, 73], utilize Transformer [61] models to analyze relationships within the scene. Additionally, the introduction

---

Scene context encoding
Trijectory decoding with decoupled quaries
aun
CrossAtte
Settkiz
Hybrid GrossAtin
(R.RT,
Made
(B.K ()
Hybrid
Queriiss
Mode Setfamn K.
18.
State SiMamba
(OK, T
HD Map
(C Hybnd Coupling Module
Prababilcties
Figure 2: Overview of our DeMo framework: The HD maps and agents are first processed by the encoder to obtain the scene context. The decoding pipeline includes: (a) the Mode Localization Module, which processes mode queries by interacting with the scene context from the encoder and among themselves; (b) the State Consistency Module, which processes state queries; and (c) the Hybrid Coupling Module, which combines these queries to generate the final output. The feature dimension is illustrated using single-agent setting, where B represents the batch size.
of novel paradigms such pre-training [7, 8, 34), historical prediction design [46, 59], GPT--styl next-token prediction [49, 53), and post-refinmment [9, 76] in some techniques has led to remarkable advancements in performance.
Furthermore, the advancements in multi-agent forecasting aim to enhance the applicability pre- dicted trajectories for various agents in real-world scenarios. Several approaches [22, 55, 79] follow an agent-centric model, where trajectories are forecasted individually for each agent, process that might be slow. On the other hand, alternative approaches [44, 78] utilize scene-centric model that allows for simultancous forecasting across all agents, introducing an innovative approach to trajectory prediction.
Inspired by the progress in object detection and motivated by its significant success [4, 40], main- stream methods [38, 54, 59, 77] have adopted a ree--uery-one-traeette
paradigm to achieve high performance in motion forecasting benchmarks [6, 58, 67]. These methods use transformers to model the relationship between each trajectory query and its environment but lack detailed representa- tion. We propose decoupled mode queries and state queries for more detailed and comprehensive representation of multi-modal trajectories.
State space models. Originally developed for modeling dynamic systems with state variables in fields such as control theory, state space models (SSMs) have emerged as promising alternatives to Transformers [61] in sequence modeling, particularly due to their effectiveness in addressing attention complexity and capturing long-term dependencies. As SSMs have evolved [17, 25, 56), a new class termed Mamba [24], which incorporates selection mechanisms and hardware-aware architectures, has recently demonstrated significant promise in long-sequence modeling. Several studies have explored Mamba's 's substantial potential across a range of fields, including natural language processing [27, 37] and computer vision [28, 35, 74, 80]. Notably, in the vision domain, Mamba has demonstrated superior GPU efficiency and effectiveness compared to Transformers in tasks such visual representation learning [80], video understanding [35], and human motion generation [74]. Building on these achievements, to the best of our knowledge, this is the first method combine the strengths of Mamba with mainstream Transformer-basee architecture achieve impressive performance.
Methodology
In this section, present DeMo, which utilizes decoupled mode queries and state queries for directional intentions and dynamic states to predict future trajectories. We also employ a hybrid architecture combining Attention and Mamba, along with two auxiliary losses for feature modeling.

---

3.1 Problem formulation
Given HD map and agents in the driving scenario, motion forecasting aims to predict the future trajectories for the interested agents. The HD map comprises several polylines of lanes or crossings, while agents are traffic participants like vehicles and pedestrians. To transform these elements into easily processable and learnable inputs, utilize popular vectorized representation following [18]. Specifically, the map M E N is generated by dividing each line into several shorter segments, where Nm. L, and Cm denote the number map polylines, divided segments and feature channels, respectively. We represent the historical information of agents as A E Nu. Th. and C. are the number of agents, historical timestamps, and motion states (e.g., position, heading angle, velocity). Additionally, the future trajectories A E R x x for agents of interest are estimation objectives. with Naoi, Ti indicating the number of selected agents and the future timestamps, respectively.
xCs, where
3.2 Scene context encoding
Given the vectorized representations for agents and M for HD map. we first employ individual encoders to process them separately. Specifically, we use a PointNet-based polyline encoder, as described in [8, 54, 55], to process the map representation M, generating the map features F., E RNN,x For the agents A, we replace generic Transformer [61] or RNN with several Unidirectional Mamba [24] blocks, which are more efficient and effective for sequence encoding, to aggregate the historical trajectory features F, NR up to the current time. Subsequently, the scene context features F3 E RNN N,. are formed by concatenating them and further propagated to Transformer encoder for intra-interaction learning. The overall process can be formulated as:
F. PointNee(MM),
F. UniMamba(A)),
F. Transformer(Concatff. Fm)). (I)
3.3 Trajectory decoding with decoupled queries
After obtaining the scene context features, we aim to decode multi-modal future trajectories for each interested agent based on our proposed decoupled queries. As illustrated in Figure 2, the decoder network comprises a State Consistency Module that enhances the consistency and accuracy of dynamic future state queries, Mode Localization Module for learning distinct motion modes, and a Hybrid Coupling Module to integrate the decoupled queries and generate the final output. The detailed description of these components are provided in the following.
Dynamic state consistency. Considering the recurrence and causality of the future trajectories Ar. we propose to represent them as a series of dynamic states across various time steps, distinct yet interconnected. To preserve precise time information, the state queries Q2 E Nan xT, xC are initialized with MLP module for real-time differences. is notable that the steps T. can differ from Ti to balance the effectiveness and efficiency, especially when predicting long-term future trajectories or a higher frequency of future trajectories. The State Consistency Module is then employed to enhance the consistency of the state queries and aggregate the specific scene context, which can be formulated as follows:
Q, MLP((11 12, ,tr:))
Q. MultiHeadAAttu(Q = Q. Vv
Q BiMamba(Q.).
Specifically, cross-attention is first applied to enable state queries interact with the scene context, followed by a Mamba block to model sequence relationships with tinear-time complexity. Simultane- ously, to account for the influences of rear state queries on the front ones, we adopt the bidirectional Mamba [35, 80] for both forward and backward scanning. Additionally, simple MLP module is utilized to decode the state queries Q2 into a single future trajectory for explicit supervision of time consistency.

---

Directional intention localization. Mode queries Qm E RRNwxiXX represent different motion modes, with each query responsible for decoding one of the K trajectories. We utilize the Mode Localization Module to localize the potential directional intentions, as shown below:
Qm MultiHcadAttn(Q = Qm: K F., Fs),
Qm MultiHoadAttt(( Qu: K Qm: V Qm)
For spatial motion learning, two Multi-Head Attention blocks are employed to enable interactions among mode queries and with the scene context. Additionally, we also employ simple MLPs to decode the future trajectories and probabilities. Similarly, we introduce another auxiliary supervision to endow mode queries with distinet motion intentions.
Hybrid query coupling. To incorporate dynamic states and directional intentions, we simply add Qm and Q. together to form the hybrid spatiotemporal queries Qn E R R.N x xT, Then, the Hybrid Coupling Module is utilized further process Qn and yield comprehensive representation for future trajectories, as formulated below:
Qn MultiHeedAAtuu( = Qn, = Fs, V = Ft),
Qn AybriddMultiHeeddAttt- Qn, Qu. V =Q).
(4)
Qn ModeMMiltHHeadAAttule = K, = Qn. - VQQu.
Qn BiMamba(Qh).
Besides the Attention and Mamba modules for interaction with the scene context, among modes, and across time states, we additionally introduce hybrid self-attention layer, which connects queries across both time and modes, boosting the diversity of predicted trajectories. The change in feature dimensions in this module is shown in Figure 2. The final predictions are generated by decoding the output Qu into trajectory positions and probabilities with MLPs.
3.4 Training losses
DeMo is trained with three component losses in end-to-end manner. Primarily, the regression loss freg and the classification loss Las are employed to supervise the accuracy predicted trajectories and their associated probability scores. Additionally, we introduce two auxiliary losses, Las and L. for intermediate features of time states and motion modes, respectively. The former enhances the coherence and causality of dynamic states across various time steps, while the latter endows mode with distinct directional intentions. The overall loss a combination these individual losses with equal weights, formulated follows:
L = Creg + Lels + Lts + L...
(5)
We adopt the cross-entropy loss for probability score classification and the smooth-L1 loss for trajectory regression tasks. The winner-take takeerl strategy is employed, optimizing only the best prediction with minimal average prediction error the ground truth.
4 Experiments
4.1
Experimental settings
We evaluate our method's performance using the Argoverse 2 [67] and nuScenes [3] motion forecasting datasets. The Argoverse dataset comprises 250,000 scenarios with a sampling frequency of 10 Hz, each featuring 5s historical trajectory length and predicting 6s future ones. The nuScenes dataset contains 1,000 scenes Hz, predicting the next trajectories with the past. 2s history.
Datasets.
5

---

Table I: Performance comparison Argoverse single-agent in the official leaderboard. For each metric, the best result is in bold and the second best result is underlined. The upper part features single model, while the lower part employs model ensembling trick.
Method
minFDE1
minADE
minFDEs
minADEs
MRs
b-minFDEo
FRM [47]
5.93
2.37
1.81
0.89
0.29
2.47
HDGT [3]]
5.37
2.08
1.60
0.84
0.21
2.24
SIMPL [72]
5.50
2.03
1.43
0.72
0.19
2.05
THOMAS [22]
4.71
1.95
1.51
0.88
0.20
2.16
GoRela [11]
4.62
1.82
1.48
0.76
0.22
2.01
MTR[54]
4.39
1.74
1.44
0.73
0.15
1.98
HPTR [73]
4.61
1.84
1.43
0.73
0.19
2.03
GANet [64]
4.48
1.77
1.34
0.72
0.17
1.96
ProphNet [65]
4.74
1.80
1.33
0.68
0.18
1.88
QCNet [77]
4.30
1.69
1.29
0.65
0.16
1.91
SmartRefine [76]
4.17
1.65
1.23
0.63
0.15
1.86
DeMo (Ours)
3.74
1.49
1.17
0.61
0.13
1.84
QML [57]
4.98
1.84
1.39
0.69
0.19
1.95
TENET [66]
4.69
1.84
1.38
0.70
0.19
1.90
MacFormer [15]
4.69
1.84
1.38
0.70
0.19
1.90
BANet [70]
4.61
1.79
1.36
0.71
0.19
1.92
Gnet [19]
4.40
1.72
1.34
0.69
0.18
1.90
Forecast-MAP MMAE [8]
4.15
1.66
1.34
0.69
0.17
1.91
QCNet [77]
3.96
1.56
1.19
0.62
0.14
1.78
DeMo (Ours)
3.70
1.49
1.11
0.60
0.12
1.73
Evaluation metrics. We adopt the common metries: minADE, minF DE, M R, and b-minFDE For multi-agent scenarios, we use avg inggMnnDDE AD avg dataset is evaluated across six prediction modes, while nuScenes is evaluated across ten prediction modes. We typically follow the evaluation metrics from the official leaderboard, setting to and for the Argoverse dataset, and K to 5 and 10 for the nuScenes dataset.
DE, and actor M R. The Argoverse
Implementation details. Our models trained for 60 epochs using the AdamW [42] optimizer, with batch size of 16 per GPU. The training is conducted end-to-end with a learning rate of 0.003 and weight decay of 0.01. We adopt an agent-centric coordinate system and sample scene elements within a 150-meter radius of the agents of interest. All experiments are conducted on 8 NVIDIA GeForce RTX 3090 GPUs. Additional details and further experiments are provided in Appendix and Appendix B.
4.2 Comparison with state of the art
We first compare our method, DeMo with several models on the Argoverse 2 [67] motion forecasting benchmark for the single-agent setting demonstrated in Table To ensure a comprehensive and fair comparison, we separately evaluate the performance of different methods with and without the model ensembling technique. It shown that DeMo has significantly outperformed all previous approaches, including the state-of- of-toe-aattee model QCNet [77] and its post-refinemet enhancement SmartRefine [76]. Concretely, our method distinctly surpasses other methods across all metrics, par- ticularly in terms of minFF DE and min where it demonstrates performance improvements of 13.02% and 11.83% relative to QCNet, respectively. After using ensembling techniques similar to other entries, DeMo surpasses all methods on all metrics by large margin. Then, we compare the performance of DeMo on the nuScenes [3] motion forecasting benchmark, with the results of the test split presented in Table 2. Our method is also superior to others over all metries except ADD..

---

Table 2:
Performance comparison muScenes test set in the official leaderboard. The "L" symbol means the corresponding metric unknown.
Method
minFDE,
minADEs
minADE10
MR;
MR10
Trajectron++ [51]
9.52
1.88
1.51
0.70
0.57
LaPred [33]
8.37
1.47
1.12
0.53
0.46
P2T [13]
10.50
1.45
1.16
0.64
0.46
GOHOME [21]
6.99
1.42
1.15
0.57
0.47
CASPNet [52]
1.41
1.19
0.60
0.43
Autobot [23]
8.19
1.37
1.03
0.62
0.44
THOMAS [22]
6.71
1.33
1.04
0.55
0.42
PGP [14]
7.17
1.27
0.94
0.52
0.34
LAformer [39]
6.95
1.19
1.19
0.48
0,48
DeMo (Ours)
6.60
1.22
0.89
0.43
0.34
Table Performance comparison on Argoverse multi-agent test set the official leaderboard.
Method
avgMinFDE, avgMinADE, avgMinFDE,
avgMinADE,
actorMRe
FIMP [50]
4.00
1.52
1.89
0.81
0.23
Forecast-MAE [8]
3,33
1.30
1.55
0.69
0.19
FFINet [32]
3.18
1.24
1.77
0.77
0.24
Gnet [19]
3.05
1.23
1.46
0.69
0.19
DeMo (Ours)
2.78
1.12
1.24
0.58
0.16
4.3 Multi-agent quantitative results
In multi-agent environments, is essential for predictors simultaneously forecast the future paths of all relevant agents to comprehensively understand the driving situation. To validate the efficacy of our model, DeMo, we conduct tests on the Argoverse multi-agent dataset [67]. The results, presented in Table 3, show that despite lacking the specialized multi-agent forecasting features found models such [44, 55, 78], our model surpasses recent advancements across all evaluated metries due to our novel designs.
4.4 Ablation study
Effects of components. Table 4 demonstrates the effectiveness of each component in method. We show the baseline the first row, which similar to previous methods [54, 77] and utilizes mode queries to generate multi-modal future trajectories. Then, we directly adopt state queries in the second row (ID-2) to decode the trajectories. performance decline is observed due to the surplus queries, which impose burden the model and make it difficult distinguish the meanings of different types. In the third row (ID-3), we introduce two auxiliary losses, resulting improvement compared the first row. Although the model can identify what each query represents, it demonstrates only moderate performance due to the limited information. In the fourth row (ID-4), incorporate the three aggregation modules in Figure but remove auxiliary losses, leading to significant performance enhancements. Finally, in the fifth row (ID-5), our DeMo integrates all these techniques and achieves outstanding performance.
slight
Effects of state sequence modeling with Mamba. Mamba excels at sequence modeling, so we utilize Bidirectional Mamba [35, 80] enhance the consistency of states across different time steps. To demonstrate its effectiveness, we compare Bidirectional Mamba with several other modules, including Unidirectional Mamba [24], Attention, Conv 1d, and GRU [10]. As illustrated in the left part of Table 5, our Bidirectional Mamba configuration outperforms the others due to its specialized design for sequence modeling, compared Attention, and its capability perform both forward and backward scans, unlike Unidirectional Mamba.

---

Table 4: Ablation study the components of DeMo the Argoverse single-agent validation set. "Deepl. Query" indicates decoupled query paradigm. "Agg. Module" indicates three aggregation modules. "Aux. Loss" indicates two auxiliary losses.
State
Decpl. Query
Agg. Module
Aux. Loss
minFDE;
minADE;
minFDEa
minADEs
MRG
b-minFDEa
Query
4,489
1.792
1,414
0,750
0.184
2.067
4,494
1.800
1.505
0.777
0.208
2.138
V
4.385
1.746
1.405
0.761
0.180
2.051
4.247
1.695
1.319
0.687
0.166
1.961
3.917
1.609
1.268
0.674
0.152
1.918
Table 5: Ablation study on (a) (left) the sequence modeling choices and (b) (right) the effects of aggregation modules and auxiliary losses. For (a), "Uni-MB" and "Bi-MB" represent Unidirectional Mamba and Bidirectional Mamba. For (b), "H.C." indicates Hybrid Coupling Module. "S.C." indicates State Consistency Module. "M.L."" indicates Mode Localization Module.
minFDEs
minADEs
MRg
minFDEs
minADE,
MRg
None GRU
1.307
0.692
0.161
w/o Lts
1.290
0.715
0.161
1.842
0.923
0.274
w/o Li
1.289
0.687
0.159
Convl6 l
1.304
0.693
0.161
w/o H.C.
1.324
0.704
0.164
Attn
1.289
0.687
0.159
w/o S.C.
1.317
0.697
0.162
Uni-MB Bi-MB 1.268
1.288
0.690
0,156
w/o M.L.
1.297
0.693
0.158
0.674
0.152
All
1.268
0.674
0.152
Effects of auxiliary losses and aggregation modules. We conduct ablation study to assess the impacts of auxiliary losses and aggregation modules. As illustrated in the right part of Table 5, removing any of these losses modules leads performance decline in the model. Notably, the aggregation modules have greater impact than the auxiliary losses. This is attributed to the critical role of learning information from the scene context and from each other, which is essential for decoupling queries to represent distinct meanings.
Effects of state queries. We conduct an ablation study on the number of state queries, as shown in the left part of Table 6. In our default setting, we use 60 state queries to represent the future states 60 timestamps.
we gradually reduce number of state queries, we observe a performance decline due to the increasing ambiguity of the state query meanings.
Effects of the depth of Attention and Mamba blocks. A suitable depth configuration Attention and Mamba units is crucial for achieving optimal balance between efficiency and performance. As depicted the right part Table 6, we conduct an ablation study focusing on the layer depth. is observed that the best results obtained with Attention units at depth of three and Mamba units at a depth of two.
Table Ablation study on (a) (left) state queries and (b) (right) the depth of Attention and Mamba layers.
Queries
minFDEs
minADEs
MRg
Attn
M.B.
minFDEs
minADEs
MRs
10
1,312
0.704
0.160
1
1
1.309
0.708
0.160
20
1.294
0.688
0.157
1.288
0.691
0.157
30
1.290
0.692
0.155
1.268
0.674
0.152
60
1.268
0.674
0.152
1.276
0.675
0.154
Effects of the depth of Mamba blocks in the encoder. We add ablation studies on the Mamba for encoding agent historical information in the encoder of our DeMo. As shown in Table 7, the left part shows different modules for encoding the historical information of agents. Our goal to aggregate historical information up to the present time, making Unidirectional Mamba the most suitable choice.

---

The right part presents an ablation study concerning the number of Mamba blocks, indicating that three layers yield the optimal performance.
Table 7:
Ablation study on (a) (left) sequence modeling choices and (b) (right) depth of Mamba blocks in agent historical information encoding.
minFDEs
minADEo
MRg
Num
minFDEs
minADEs
MRg
GRU 1.344
0.726
0.170
1.312
0.701
0.162
Bi-MB Uni-MB
1.280
0.684
0.154
2
1.283
0.681
0.155
1.268
0.674
0.152
3
1.268
0.674
0.152
4.5 An analysis to improve the measurement of query decoupling
We measure the outputs of state queries and mode queries with minADE and minFDE, as shown in Table We can see that the Ain and minF DE of the trajectories from state query outputs are better than those from mode query outputs. This means state dynamics are encoded in state queries. Additionally, there are six output trajectories from mode queries, indicating that directional information is predominantly stored in mode queries. The final outputs take advantage of the strengths of both.
Table 8: An analysis improve the measurement of query decoupling.
minFDE;
minADE;
minFDEg
minADEg
state query outputs
3.84
1.52
mode query outputs
4.12
1.63
1.31
0.67
final outputs
3.93
1.54
1.24
0.64
4.6 Efficiency analysis and qualitative results
Balancing performance, inference speed, and model size is important for the model deployment. We compare DeMo with recent, representative models: the state-of-the-arr QCNet [77] and its enhancement through post-refinemet, SmartRefine [76]. The size of our model is 5.9M, in contrast to 7.7M for QCNet and 8.0M for SmartRefine. Despite its smaller size, demonstrates superior performance by a significant margin, as detailed in Table 1. As for inference speed, we compare DeMo with QCNet, both end-to-end methods. Measurements are conducted on the Argoverse 2 single-agent validation set using an NVIDIA GeForce RTX 3090 GPU, with maintained batch size of one. The average inference speed of DeMo is only 38ms, which is approximately 2.5 times faster than QCNet's 94ms. This demonstrates that our method is not only superior to QCNet but also more efficient.
model
In Figure 3, we present qualitative results of our network. The results the bascline model, which lacks the decoupled query paradigm, are shown in panel (a), while the results of our DeMo shown in panel (b). From the first two rows, is evident that by explicitly optimizing the dynamic states of future trajectories, ground truth. From the third row, it is apparent that our model can better capture potential directional intentions. Additional qualitative results and failure cases are detailed in Appendix D and E.
model predicts trajectories that are more accurate and closer the
4.7 Computational cost compared to other methods
provide a comparison computational cost with resent representative methods in Table 9. The experiments conducted on Argoverse 2 [67] dataset using 8 NVIDIA GeForce RTX 3090 GPUs.

---

Table 9: Computational cost compared to other methods.
Method
FLOPs
Training time
Memory
Parameter
Batch size
SIMPL [72]
19.7 GFLOPs
8h
14G
1.9M
16
QCNet [77]
53.4 GFLOPs
45h
16G
7.7M
4
DeMo (Ours)
22.8 GFLOPs
9h
12G
5.9M
16
(a) w/o Decoupled Queries
(b) DeMo w Decoupled Queries
(c) GT
1
Predicted Trajectury
GT Trajectory
Agent of Interest
Surrounding Agent
Figure 3: Qualitative results on the Argoverse single-agent validation set. Panel (a) illustrates the results of the baseline model without decoupled queries; Panel (b) illustrates the results of our DeMo, which employs decoupled queries; and Panel (c) represents the ground truth.
Conclusion
In this paper, we introduce DeMo, which redefines the motion forecasting task by decoupling into expressions of directional intentions and dynamic states. We utilize state queries to model various states across different time, and mode queries capture the agent's motion intentions. Our approach incorporates three aggregation modules, combining Attention and Mamba for effective modeling. Comprehensive experiments, covering both single-agent and multi-agent scenarios, indi- cate that DeMo outperforms the current state-of-the-an methods. This highlights its potential as promising approach for achieving safe and reliable motion forecasting in the rapidly advancing field of autonomous driving.
Limitations and future work. The proposed framework adopts decoupled query paradigm, which may lead to heavier models due to the need predict longer trajectories. Our current model design does not sufficiently take model efficiency into account. In the future, we plan to use sparse states for modeling trajectories, thereby making the framework more deployment-friemte
10

---

Acknowledgments
This work was supported in part by National Natural Science Foundation of China (Grant No. 62106050 and 62376060), Natural Science Foundation of Shanghai (Grant No. 222R 1407500))
References
[1] G. Aydemir, A. K. Akan, and F. Güney. Adapt: Efficient multi-agent trajectory prediction with adaptation.
In ICCV, 2023. 15
[2] P. Bhattacharyya, C. Huang, and K. Czamecki. Ssl-lanes: Self-supervised learning for motion forecasting
in autonomous driving. In CoRL, 2023. 15
[3] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and
O. Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR, 2020. 6
[4] N. Carion, F. Massa, G. Synnaeve. N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection
with transformers. In ECCV, 2020. 3,
[5] Y. Chai, B. Sapp, M. Bansal, and D. Anguelov. Multipath: Multiple probabilistic anchor trajectory
hypotheses for behavior prediction. In CoRI., 2020.
[6] M.-F. Chang. J. Lambert, P. Sangkloy. J. Singh, S. Bak, A. Hartnett, D. Wang. P. Carr, S. Lucey, D. Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In CVPR, 2019. 3, 15
[7] H. Chen, J. Wang. K. Shao, F. Liu, J. Hao, C. Guan, G. Chen, and P.-A. Heng. Traj-mae: Masked
autoencoders for trajectory prediction. In ICCV, 2023. 3
[8] J. Cheng, X. Mei, and M. Liu. Forecast-mae: Self-supervised pre-training for motion forecasting with masked autoencoders. In ICCV, 2023. 3, 4, 6,
191 S. Choi, Kim, J. Yun, and J. W. Choi. R-pred: Two-stage motion prediction via tube-query attention-
trajectory refinement. In ICCV, 2023. 3, 15
[10] Chung, C. Gulcehre, K. Cho, and Bengio. Empirical evaluation of gated recurrent neural networks on
sequence modeling. arXiv preprint, 2014. 7
[11] A. Cui, S. Casas, K. Wong, S. Suo, and R. Urtasun. Gorela: Go relative for cesvpoint-invariall motion forecasting. In ICRA, 2023. 6
[12] F. Da and Y. Zhang. Path-aware graph attention for hd maps in motion prediction. In ICRA, 2022. 15
[13] N. Deo and M. M. Trivedi. Trajectory forecasts in unknown environments conditioned on grid-hased plans.
arXiv preprint, 2020. 7
[14] N. Deo, E. Wolff, and O. Beijbom. Multimodal trajectory prediction conditioned on lane-graph traversals.
In CoRL, 2022. 2,
[15] C. Feng, H. Zhou, H. Lin, Z. Zhang, Z. Xu, C. Zhang, B. Zhou, and S. Shen. Macformer: Map-agent
coupled transformer for real-time and robust trajectory prediction. IEEE RA-L, 2023. 6
[16] L. Feng, M. Bahari, K. M. B. Amor, É. Zablocki. M. Cord, and A. Alahi. Unitraj: A unified framework for scalable vehicle trajectory prediction. In ECCV, 2024. 15
[17] D. Y. Fu, T. Dao, K. K. Saab, A. W. Thomas, A. Rudra, and C. Re. Hungry hungry hippos: Towards language modeling with state space models. In ICLR, 2023. 3
[18] J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. L.i, and C. Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In CVPR, 2020. 2,
[19] X. Gao, X. Jia, Y. Li, and H. Xiong. Dynamic scenario representation learning for motion forecasting with heterogeneous graph convolutional recurrent networks. IEEE RA-L, 2023. 6,
[20] T. Gilles, S. Sabatini, D. Tsishkou, B. Stanciulescu, and F. Moutarde. Home: Heatmap output for future motion estimation. In IEEE ITSC, 2021.
[21] T. Gilles, Sabatini, D. Tsishkou, B. Stanciulescu, and Moutarde. Gohome: Graph-oriented heatmap
output for future motion estimation. In ICRA, 2022. 2,7
[22] T. Gilles, S. Sabatini, D. Tsishkou, B. Stanciulescu, and Moutarde. THOMAS: Trajectory heatmap
output with Iearned multi-agent sampling. In ICLR, 2022. 3, 6,7
[23] R. Girgis, F. Golemo, F. Codevilla, M. Weiss, variable sequential set transformers for joint multi-agent motion prediction. ICLR, 2022.
D'Souza, S. E. Kahou, Heide, and C. Pal. Latent
[24] A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprini.
2023. 3.4.7
[25] A. Gu, K. Goel, and C. Re. Efficiently modeling long sequences with structured state spaces. In ICLR,
2022. 3
[26] J. Gu, C. Sun, and H. Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In ICCV, 2021. 2, 15
[27] W. He, K. Han, Y. Tang, C. Wang. Y. Yang. T. Guo, and Y. Wang. Densemamba: State space models with dense hidden connection for efficient large language models. arXiv preprint, 2024.
11

---

[28] V. T. Hu, S. A. Baumann, M. Gui, O. Grebenkova, P. Ma, J. Fischer, and B. Ommer. Zigma: A dit-style zigzag mamba diffusion model. In ECCV, 2024. 3
[29] Y. Huang, J. Du, Z. Yang. Z. Zhou, L. Zhang, and H. Chen. A survey on trajectory-prodiction methods for autonomous driving. IEEE T-/V, 2022.
[30] x. Jia, L. Sun, H. Zhao, M. Tomizuka, and W. Zhan. Multi-agent trajectory prediction by combining egocentric and allocentric views. In CoRL, 2022. 2
[31] X. Jia, Wu, I.. Chen, Y. Liu, H. Ii, and Yan. Hdgt: Heterogeneous driving graph transformer for
multi-agent trajectory prediction via scene encoding. IEEE TPAMI, 2023. 2, 6
[32] Kang, Wang. Zhou, K. Ye, Jiang, and N. Zheng. Ffinet: Future feedback interaction network for motion forecasting. JEEE Transactions on Intelligent Transportation Systems, 2024. 7
[33] B. Kim, S. H. Park, S. Lee, E. Khoshimjonov,
Kum, Kim, Kim, and J. W. Choi Lapred: Lane-aware prediction of multi-modal future trajectories of dynamic agents. In CVPR, 2021. 7
[34] Z. Lan, Y. Jiang. Y. Mu, C. Chen, and S. E. Li. Sept: Towards efficient scene representation learning for
motion prediction. ICLR, 2024. 3, 16
[35] K. Li, X. Li, Y. Wang. Y. Hc, Y. Wang. L. Wang, and Y. Qiao. Vidcomamba: State space model for efficient
video understanding. In ECCV, 2024. 3, 4,7,
[36] M. Liang. B. Yang. R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun. Learning lane graph representations
for motion forecasting. In ECCV, 2020. 2, 15
[37] O. Lieher, B. Lenz, H. Bata, G. Cohen, J. Osin, I. Dalmedigos, E. Safahi, S. Mcirom, Y. Belinkov, S. Shalev-Shwartz, et al. Jamba: A hybrid transformer-mant language model. arXiv preprint, 2024. 3
[38] L. Lin, X. Lin. T. Lin, L. Huang. R. Xiong. and Y. Wang. Eda: Evolving and distinct anchors for multimodal
motion prediction. In AAAI, 2024. 1,3
[39] M. Liu, H. Cheng. I.. Chen, H. Broszio, J. Li, R. Zhao, M. Sester, and M. Y. Yang. Laformer: Trajectory
prediction for autonomous driving with lane-aware scene constraints. arXiv preprint, 2023. 7
[40] S. Liu, F. Li. H. Zhang, X. Yang. X. Qi, H. Su, Zhu, and L. Zhang. Dab-detr: Dynamic anchor boxes are
better queries for detr. In ICLR, 2022. 1,3
[41] Y. Liu, J. Zhang, I.. Fang. Q. Jiang, and B. Zhou. Multimodal motion prediction with stacked transformers.
In CVPR, 2021. 2, 15
[42] 1. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019. 6
[43] N. Nayakanti, R. Al-Rfou, A. Zhou, K. Goel, K. S. Refaat, and B. Sapp. Wayformer: Motion forecasting
via simple & efficient attention networks. In ICRA, 2023, 2
[44] J. Ngiam, V. Vasudevan, B. Caine, Z. Zhang, H.-T. L. Chiang, J. Ling, R. Roelofs, A. Bewley, C. Liu,
A. Venugopal, D. J. Weiss, B. Sapp, Z. Chen, and J. Shlens. Scene transformer: A unified architecture for predicting future trajectories of multiple agents. In ICLR, 2022. 1, 3,
[45] Z. Pang, D. Ramanan, M. Li, and Y.X. Wang. Streaming motion forecasting for autonomous driving. In IROS, 2023. 14
[46] D. Park. J. Jeong, S.-H. Yoon, J. Jeong, and K.-J. Yoon. 14p: Test-time training of trajectory prediction via masked autoencoder and actor-spocific token memory. In CVPR, 2024. 3
[47] D. Park, H. Ryu, Y. Yang, J. Cho, J. Kim, and K.-J. Yoon. Leveraging future relationship reasoning for vehicle trajectory prediction. In ICLR, 2023. 6, 15
[48] T. Phan-Mine Minh, E. Grigore, F. A. Boulton, O. Beijbom, and E. M. Wolff. Covernet: Multimodal behavior prediction using trajectory sets. In CVPR, 2020.
[49] J. Philion, X. B. Peng, and S. Fidler. Trajeglish: Learning the language of driving scenarios. In ICLR,
2024. 3
[50] L. Rowe, M. Ethier, E.-H. Dykhne, and K. Czameekk. Fjmp: Factorized joint multi- agent motion prediction over learned directed acyelic interaction graphs. CVPR, 2023. 2,7
[51] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++: Dynamically-feasible trajectory
forecasting with heterogeneous data. In ECCV, 2020. 7
[52] M. Schäfer, K. Zhao, M. Bühren, and A. Kummert. Context- scene prediction network (caspnet). In IEEE ITSC, 2022.
[53] A. Seff, B. Cera, D. Chen, M. Ng. A. Zhou, N. Nayakanti, K. S. Refaat, R. Al-Rfou, and B. Sapp. Motionlm: Multi-agent motion forecasting as language modeling. In ICCV, 2023. 3
[54] S. Shi, L. Jiang, D. Dai, and B. Schiele. Motion transformer with global intention localization and local movement refinement. NeurIPS, 2022. 1, 2, 3, 4, 6,
[55] S. Shi. L. Jiang, D. Dai, and B. Schiele. Mir++: Multi-agent motion prediction with symmetric scene
modeling and guided intention querying. JEEE TPAMI, 2024. 3,4,7
[56] J. T. Smith, Warrington, and S. Linderman. Simplified state space layers for sequence modeling. In ICLR, 2023. 3
[57] T. Su, X. Wang, and X. Yang. Qml for argoverse 2 motion forecasting challenge. arXiv preprint, 2022. 6 [58] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020. 1,3, 15
12

---

[59] X. Tang, M. Kan, S. Shan, Z. Ji, J. Bai, and X. Chen. Hpnet: Dynamic trajectory forecasting with historical
prediction attention. In CVPR, 2024. 1, 14, 15
[60] B. Varadarajan, A. Hefny, A. Srivastava, K. S. Refaat, N. Nayakanti, A, Cormman, K. Chen, B. Douillard,
C. P. Lam, D. Anguelov, et al. Multipath+++: Efficient information fusion and trajectory aggregation for behavior prediction. In ICRA, 2022.
[61] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.
Attention is all you need. NeurlPS, 2017. 3,4
[62] Wang. T. Ye, Z. Gu, and Chen. Lip: Lane-based trajectory prediction for autonomous driving. In
CVPR, 2022. 15
[63] M. Wang, X. Ren, R. Jin, M. Li, X. Zhang, C. Yu, M. Wang, and W. Yang. Futurenet-lof: Joint trajectory
prediction and lane occupancy field prediction with future context encoding. arXiv preprint, 2024.
[64] M. Wang. X. Zhu, C. Yu. W. Li, Y. Ma. R. Jin, X. Ren, D. Ren, M. Wang, and W. Yang. Ganet: Goal area
network for motion forecasting. In ICRA, 2023. 6
[65] X. Wang, T. Su, F. Da, and X. Yang. Prophnet: Efficient agent-centric motion forecasting with anchor-
informed proposals. In CVPR, 2023. 6
[66] Y. Wang. H. Zhou, 7. Zhang. C. Feng, H. Lin, C. Gao, Y. Tang, Z. 7hao, S. Zhang. J. Guo, et al. Tenet: Transformer encoding network for effective temporal flow on motion prediction. arXiv preprint, 2022. [67] B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, et al. Argoverse Next generation datasets for self-driving perception and forecasting. NeurlPS,
2021. 1, 3, 5, 6, 7,
[68] M. Ye, T. Cao, and Q Chen. Tpen: Temporal point cloud networks for motion forecasting. In CVPR, 2021.
15
[69] W. Zeng, M. Liang, R. Liao, and R. Urtasun. Lanerenn: Distributed representations for graph-centric motion forecasting. In IROS, 2021. 2, 15
[70] C. Zhang, H. Sun, C. Chen, and Y. Guo. Banet: Motion forecasting with houndary aware network. arXiv
preprint, 2022. 6
[71] I.. Zhang, P. Li, J. Chen, and Shen. Trajectory prediction with graph-based dual-scale context fusion. In IROS, 2022. 2, 15
[72] L. Zhang. P. Li, S. Liu, and S. Shen. Simpl: A simple and efficient multi-agent motion prediction baseline
for autonomous driving. IEFE RA-I., 2024. 6, 10, 15
[73] Z. Zhang, A. Liniger, C. Sakaridis, F. Yu, and L.. V. Gool. Real-time motion prediction via heterogencous
polyline transformer with relative pose encoding. NeurlPS, 2023. 2,
[74] Z. Zhang. A. Liu, I. Reid, R. Hartley, B. Zhuang, and H. Tang. Motion mamba: Efficient and long sequence
motion generation with hierarchical and bidirectional selective ssm. In ECCV, 2024. 3
[75] H. Zhao, Gao, Lan, Sun, B. Sapp. B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid, al. Tnt:
Target-driven trajectory prediction. In CoRL., 2021. 2, 15
[76] Y. Zhou, H. Shao, L.. Wang, S. L. Waslander, H. Li, and Y. Liu. Smartrefine: A scenario-adaptive refinement
framework for efficient motion prediction. In CVPR, 2024. 3, 6, 9
[77] Z. Zhou, J. Wang, Y.H.. Li, and Y.-K. Huang. Query-centric trajectory prediction. In CVPR, 2023. 1, 2, 3,
6, 6,7,9 9, 10
[78] Z. Zhou, Z. Wen, J. Wang, Y.-.. Li, and Y.KK. Huang. Qenext: A next-generation framework for joint
multi-agent trajectory prediction. arXiv preprint, 2023. 3, 7, 16
[79] Z. Zhou, L. Yc, J. Wang. K. Wu, and K. H. Lu. Hicrarchical vector transformer for multi-agent motion prediction. In CVPR, 2022. 2, 3, 15
[80] L. Zhu, Liao, Q Zhang, X. Wang, W. Liu, and Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. In ICML, 2024, 3, 4,
13

---

Appendix
A Experimental settings
A.1 Baseline model
Based the streaming nature of driving data, our baseline model employs lightweight transformer- based architecture capable of historical prediction [45, 59]. This model predicts future trajectories at intervals of 3s, 4s, and 5s for the Argoverse 2 dataset, and at 1s, 1.5s, and 2s for the Argoverse 1 dataset. The final prediction is the ultimate outcome these trajectories. Features from the encoder and decoder in the historical prediction phase aggregated with those from the encoder and decoder in the subsequent prediction phase. The aggregation module comprises two cross- cross-attention layers. The historical prediction component is not used for the nuScenes dataset due the limited history steps. In ablation studies, we remove the historical prediction component from the baseline model to make the experiments more efficient.
A.2 Single-agent evaluation metrics
We employ established metries to evaluate our models, including minimum Average Displacement Error (minAD Ex), minimum Final Displacement Error (minFDE,) Miss Rate (MRR), and Brier minimum Final Displacement Error (b-minF FE)). The minAD Eg metric calculates the Le distance between the ground-truth trajectory and the best K predicted trajectories, averaged over all future time steps. The minF D metric measures the discrepancy between the endpoints of the predicted trajectories and the ground truth. The RR, metric represents the proportion of scenes where FiinDDE, exceeds meters. To provide more nuanced evaluation of uncertainty, b-minFFD Eg incorporates (1 )²² into the final displacement error, where indicates the probability score assigned by the model to the best-predicted trajectory.
A.3 Multi-agent evaluation metrics
Following the official settings for the Argoverse 2 multi-agent test set, we use metrics like Average Minimum Final Displacement Error (avg.MnnFD DE), Average Minimum Average Displacement Error in and Actor Miss Rate (actor R). The arg metric calculates the average of the lowest Final Displacement Errors (FDEs) for all scored actors in scenario, reflecting prediction accuracy. Similarly, Errors (ADEs) for all scored actors, showing overall movement accuracy. The actor M R measures the proportion of actors missed across the evaluation set, as previously described.
is the average of the lowest Average Displacement
A.4 More implementation details
In addition to the details in Section 4, we set the dropout rates at 0.2 for single-agent settings and 0.1 for multi-agent settings. We employ a cosine learning rate schedule with a warm-up phase of 10 epochs. For normalization, we use mn.LayerNorm, and for activation, we use nn. GELU.. No data augmentation techniques are used. For details on the number of layers in each component, please refer to Table 10.
A.5 precise formulation the auxiliary losses
We use an MIP to decode state queries into single future trajectory Y, and calculate the loss with ground truth Yft to obtain Lis:
Cs SmoothLi(Y, Ygt).
(6)
We use MLPs to decode the future trajectories Y, and probabilities Py. So L. is shown below:
Ybett Poest SelectBest (Y,, Ygr), L. SmoothL. Yor) CrossEntropy (Pj, Poest)-
14

---

Table 10: Number of layers in each component.
Ene/Dec
Name
Num- Num-AVi&&e 1en--VI&&AVV
Num-nuScenee
Agent Encoding Mamba
4
2
Enc
Scene Context Transformer
5
State Consistency Module Attention
2
2
State Consistency Module Mamba
2
Dec
Mode Localization Module Attention
3
2
Hybrid Coupling Module Attention
3
Hybrid Coupling Module Mamba
2
2
B
More experiments
B.1 Performance comparison on the Argoverse dataset
To fully demonstrate the effectiveness of our DeMo, compare it with several recent models on the Argoverse [6] dataset. The results from the validation split are shown in Table 11, indicating that model achieves impressive performance.
Table 11: Performance comparison on Argoverse I validation set.
Method
minADEg
minFDEs
MRs
LTP [62]
0.78
1.07
LaneRCNN [69]
0.77
1.19
0.08
TPCN [68]
0.73
1.15
0.11
DenseT'NT [26]
0.73
1.05
0.10
TNT [75]
0.73
1.29
0.09
mmTransformer [41]
0.71
1.15
0.11
LaneGCN [36]
0.71
1.08
SSL-Lanes [2]
0.70
1.01
0.09
PAGA [12]
0.69
1.02
DSP [71]
0.69
0.98
0.09
FRM [47]
0.68
0.99
ADAPT [1]
0.67
0.95
0.08
SIMPL [72]
0.66
0.95
0.08
HiVT [79]
0.66
0.96
0.09
R-Pred [9]
0.66
0.95
0.09
HPNet [59]
0.64
0.87
0.07
DeMo (Ours)
0.59
0.90
0.07
B.2 Performance the Argoverse 2 leaderboard
We provide a performance comparison of the top methods on the Argoverse leaderboard as of September 2024. The results for the single-agent setting shown in Table 12, and the results for the multi-agent setting are shown in Table 13.
B.3 Results on Waymo open motion dataset
We provide results on WOMD [58] Table 14 using the settings in UniTraj [16], as shown below. The results of other methods are also from UniTraj.
B.4
Model ensembling
In our approach, we use model ensembling, an essential technique enhance the accuracy final predictions. We train six sub-models with various random seeds and training epochs, resulting in 36
15

---

Table 12:
Performance comparison of the single-agent setting the Argoverse 2 dataset in the official leaderboard. Unreleased works are marked with the symbol "*"
Method
b-minFDEg
Rank
LOF [63]
1.63
1
iDLab-SEPT+ (SEPT++)
1.65
2
EACON (JMaC)
1.67
3
PolarMotion (PolarMotion) *
1.71
4
DeMo (Ours)
1.73
5
iDLab-SEPT (SEPT) [34]
1.74
6
xPnC (X-MotionFormer)
1.74
7
GACRND-XLAB (XPredFormer) *
1.76
8
Table 13: Performance comparison of the multi-agent setting on the Argoverse 2 dataset in the official leaderboard. Unreleased works are marked with the symbol "**
Method
argBrierMinFDE, Rank
EACON (JMaC) *
1.62
QCNet-AV2 (QCNeXt) [78]
1.65
2
Lite-QCNet
1.67
LOF [63]
1.68
4
iDLab-SEPT [34]
1.80
DeMo (Ours)
1.93
6
FAW-Prediction *
1.93
berste (OGD_testt2)
1.95
8
Table 14: Results on WOMD.
Method
minFDEs
minADEg
MTR
1.78
0.78
Wayformer
1.46
0.65
AutoBot
1.65
0.73
DeMo (Ours)
1.59
0.75
predicted future trajectories for each agent. We then apply k-means clustering with six cluster centers to process these trajectories. For each cluster group, we compute the average trajectory within the group to determine the final trajectories. We present the results, both with and without the model ensembling technique, on the Argoverse 2 single-agent test set in Table 1.
C Mamba introduction
Mamba is inspired by a continuous system that maps 1-D function or sequence x(t) E to v(t) E IR, utilizing hidden state h(t) R. In this system, A RNxN serves as the evolution parameter, while B E Nx and C E Ix act as the projection parameters.
(1(I) Ah(() + Br((), vt) Ch(t).
(8)
Mamba represents the discrete version of a continuous system and includes timescale parameter A convert the continuous parameters and B into discrete counterparts A and B. The most commonly used method for this transformation is zero-order hold (ZOH). After discretizing A and B, the discretized form of Equation (8) using step size of A can be reformulated as:
16

---

A exp (AA)
B (AA) (exp (AA) I) AB.
h A 1 + B
U/2 Ch+-
Finally, the models compute the output through global convolution that utilizes a structured convolutional kernel K R&,, where M represents the length of the input sequence x.
K = (CB, CAB,
CA M LB.
B).
(10)
y =
D
More qualitative results
We provide more qualitative results of our DeMo in Figure 5.
E Failure cases
Although our DeMo has demonstrated exceptional performance on motion forecasting benchmarks, it is not without its failures. We analyze these typical cases and present qualitative results give readers insight into the scenarios where our model might underperform. This analysis is intended to support future efforts in developing algorithm that both more robust and powerful, illustrated in Figure 4. In the first row, the vehicle will turn into an alley, reflecting a kind of subjective driving behavior. However, the model predicts that the vehicle will just keep going straight. To improve predictions in cases like this, we could enhance how the model interacts with additional information about what the vehicle intends do, such adding visual cues, such as turn signals. the second row, the agent needs to navigate through a complex intersection to reach one of the roads; however, the model fails accurately predict this driving behavior. This inaccuracy may be caused by lack of comprehensive understanding of the complex map topology and the unbalanced distribution of driving data, addressing the issue data balance necessary to solve this problem.
17

---

(a) Our Model Output
(b)
Predicted Trajectory
GT Trajectory
Agent Interest
Surrounding Agent
Figure 4: Failure cases.
18

---

(a) w/o Decoupled Querles
(b) DeMo Decoupled Queries
(c) GT
6
Predicted Trajectory
Trajectory
Agent Interest
Surrounding Agent
Figure 5: More qualitative results.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
20

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification:
The details of the model and experiments are provided in the main paper and the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. ec/
Policy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: cenollc/guidess..
/ /nips...
sssisssin ssissssliccyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: pettps//neerippln wwt.rriipp.cppppppee http::pp.......leeccccccccceeeee
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our main idea is specifically designed for applications in real-world au- tonomous driving.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Section 4.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "REM4cHslRL", "reviewer": "Reviewer_pCB4", "summary": "The paper presents DeMo, a novel framework for motion forecasting in autonomous driving systems. DeMo decouples the motion forecasting task into two distinct components: mode queries for capturing directional intentions and state queries for modeling dynamic states over time. This separation allows DeMo to separately optimize for multi-modality and dynamic state evolution, leading to a more comprehensive representation of future trajectories. The framework employs a combination of Attention and Mamba techniques for global information aggregation and state sequence modeling. Extensive experiments on the Argoverse 2 and nuScenes benchmarks demonstrate that DeMo achieves state-of-the-art performance in motion forecasting.", "questions": "1. In the first line of Eq. (2), what do the symbols {$\\{ t_1, t_2, ...t_{T_s} \\}$} represent, and how do $T_s$ and $T_f$ differ from or relate to each other? How are these time values obtained and utilized in the model?\n\n2. For the motion model $Q_m$, what specific features are considered for different motion models? The paper does not clearly specify the features used in these motion models.\n\n3. In Section 3.4, the loss function $L_{ts}$ is described as related to \"intermediate features of time state.\" What exactly are these intermediate features, and what is their role in the model? This concept is not clearly defined in the paper.\n\n4. Table 1 does not include baseline methods such as SEPT and SEPT++ that have better performance and earlier submission dates than DeMo. Similarly, Table 3 omits models like QCNet, which outperform DeMo. Why are these models not included in the comparisons?\n\n5. The paper lacks detailed information about the model’s parameters and settings. Can you provide more specific details on the parameters used for DeMo?", "limitations": "The technical descriptions lack clarity and the overall contribution is limited.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The overall idea behind DeMo is reasonable and technical soundness. Additionally, the experiments are comprehensive, demonstrating the results of DeMo across two different datasets, Argoverse 2 and nuScenes.", "weaknesses": "There are two major weaknesses: 1. Technical Descriptions Lack Clarity: The technical explanations of the methods and algorithms used in DeMo are not sufficiently clear, making it challenging for readers to fully understand the proposed techniques and their implementations. 2. Unconvincing Contributions: The claimed contributions of the paper are not entirely convincing. For instance, the paper doesn’t compare DeMo to models that have superior performance on existing leaderboards, questioning the novelty of its improvements. Additionally, there are previous works (e.g. Motion Mamba by Zhang et.al.) that have already combined Mamba and Transformer techniques for time series motion data, which undermines the claim that DeMo introduces a novel approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:\n\n## _Response to technical descriptions._\n\n**Q1: About Eq. (2) and related symbols meaning.**\n\nFor the meaning of {$t_1,…,t_{T_s}$}: as in Line 135, these represent real-time differences of temporal states. For example, in Argoverse 2, we aim to predict future trajectories over 60 time steps. Therefore, if we use 60 state queries, $T_s=60$, and {$t_1,…,t_{T_s}$} corresponds to {$1,2,…,60$}.\n\nFor $T_f$ and $T_s$: As stated in Line 114, $T_f$ refers to the future time steps. As stated in Line 135, $T_s$ represents the state steps used for initializing state queries. As stated in Line 233, in our default setting, $T_s=T_f$. We utilize one state query to represent several time steps in the left part of Table 6. In these cases, $T_s$ differs from $T_f$.\n\n**Q2: For the motion mode $Q_m$.**\n\nAs stated in Line 146 and Line 151, we have $K$ mode queries to represent $K$ trajectories. We decode the $K$ future trajectories and their corresponding probabilities, using losses to optimize specific features for different motion modes, as done in QCNet [1], MTR [2], and other works.\n\n**Q3: About loss function $L_{ts}$ and intermediate features.**\n\nAs stated in Line 143 and shown in Figure 2, we use an MLP to decode state queries into a single future trajectory $Y_f$ and calculate the loss with ground truth $Y_{gt}$ to obtain $L_{ts}$. \n\nThus, the state queries $Q_s$ serve as intermediate features, similar to the mode queries $Q_m$. The role of these two types of intermediate features is to form the hybrid queries $Q_h$ to decode the final outputs.\n\n**Q4: About detailed information about the model’s parameters and settings.**\n- Training settings: Line 184, Line 459 (Appendix). \n- Dataset and metric settings: Line 175, Line 180.\n- Model size (5.9M) and inference speed (38ms): Line 242.\n- The number of layers in each component: Table 7 (Appendix).\n\n## _Response to contributions._\n\n**Q5: Table 1 and Table 3 do not include baseline methods such as SEPT, SEPT++, QCNeXt.**\n\nFor Table 1, our method (ranked 3rd) is better than SEPT [3] (ranked 4th) on the official leaderboard of Argoverse 2. SEPT is a self-supervised method utilizing all sets (including the test set) for pretraining. It is orthogonal to ours, and we believe that SEPT can also be integrated into our method for further improvements. Additionally, SEPT++ has not been released yet.\n\nFor Table 3, our model is not specifically designed for the multi-agent setting, unlike QCNeXt [4], which is a method specifically designed for competition. We apologize for the insufficient comparison, which we have addressed in the revised paper.\n\n**Q6: Compare with previous works (e.g. Motion Mamba) for time series data.**\n\nAlthough both Motion Mamba [5] and our method utilize Mamba modules, the motivation and tasks are completely different. Our critical idea is decoupling motion forecasting, and Mamba is an effective tool to implement this aim. Additionally, the structures are also completely different. We will add a discussion to highlight the distinctions and contributions of our work compared to Motion Mamba in the revised manuscript.\n\n> [1] Query-centric trajectory prediction. CVPR, 2023.\n\n> [2] Motion transformer with global intention localization and local movement refinement. NeurIPS, 2022.\n\n> [3] Sept: Towards efficient scene representation learning for motion prediction. ICLR, 2024.\n\n> [4] Qcnext: A next-generation framework for joint multi-agent trajectory prediction. arXiv preprint:2306.10508, 2023.\n\n> [5] Motion mamba: Efficient and long sequence motion generation with hierarchical and bidirectional selective ssm. arXiv preprint:2403.07487, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S531u6NOdV", "reviewer": "Reviewer_ZMh7", "summary": "DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic States introduces a state of the art model architecture for motion forecasting (predicting the future trajectories of road actors for the purpose of autonomous driving).  The authors make two notable contributions.  First they provide an alternative to the typical one-query one-trajectory paradigm of current high performing models.  Instead they decompose queries into \"mode queries\" which attempt to capture directional intentions and \"state queries\" which attempt to capture the dynamic properties of a trajectory.  The authors also replace the sequence processing portion of the encoder (typically cross attention layers in current models or RNNs in older models) with Mamba blocks.  The authors include numerous ablations demonstrating that each of these contributions plays a substantive role in the performance of their model.  The authors present results on two significant motion forecasting benchmarks.", "questions": "- Can you provide a precise formulation of the auxilary losses?\n- Can you report training time?\n- How does the model in ID3 (table 4) work?  Are the decouple queries just concatenated and fed directly to the MLP decoder?\n- How does the model size vary across your ablations (table 4)?  I.e. why should I ascribe the performance improvements to architectural choices and not just growing model capacity?\n- For inference speed can you report 99th percentile (or similar) rather than mean?  The AV industry is primarily concerned with worst-case performance.\n- How does the multi-agent set up work?  Do I re-encode the scene in each agent-centric frame?\n- Line 488 -- the text in this paragraph makes it sound like ZOH is a continuous->discrete transformation.  I might be wrong here, but I typically think of it as a digital->analog transformation.  Are we really doing something like the inverse of ZOH?", "limitations": "No concerns.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is a very strong and well-presented manuscript.  Some strengths include:\n\n### Presentation\n- The copy is excellent.  The paper is well written and largely quite clear.\n- Tables are excellent.\n- Visual examples (figures 3-5) are clear and compelling.\n\n### Results\nThe results are presented on two (of the three) most important motion forecasting benchmarks for self-driving.  In both cases DeMo is clearly SOTA.  They additionally present ensemble results.\n\n### Analysis\nThe authors conduct an extensive set of ablation experiments.  They conduct the expected ablations (disabling various parts of their architecture).  Additionally, they explore the impact of: layer types (RNN vs Uni MB vs Bi MB), the number of layers, the number of state queries, the auxiliary losses.", "weaknesses": "This is already quite a solid manuscript, however I hope that the authors can selectively address a few of the weaknesses and questions below to arrive at an even better paper.\n\n### Results\nWhile AV2 and NuScenes are both important benchmarks, WOMD (Waymo) is probably the most important (or co-most-important) benchmark in this space right now.  It would have been nice to see results on WOMD.  There are tools like https://github.com/vita-epfl/UniTraj now to make this easier.\n\n### Analysis\n- While tables 4 and 5 clearly demonstrate that the query decomposition and auxilary losses contribute significantly to the model performance, there is little evidence to support the motivating intuitions.  For example, the introduction of state queries and the associated auxiliary loss is supposed to produce better dynamics.  Can we come up for a metric to measure that?  Can we find ways to demonstrate that our decomposition works as we expect?  Directional information is predominantly stored in mode queries while dynamics are encoded in state queries?\n- I would have liked to see the authors get an RNN based model to successfully converge.  (The GRU result is disappointing).\n\n\n### Reproducibility\nThe authors promise to release code (which would entirely alleviate my concern here), however I do not feel that the model is reproducible from the manuscript alone.  There is very little information about the auxilary losses.  Additionally there is no supplementary diagram or table with an explicit architectural description (layer sizes, normalization, etc).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:\n\n## _Response to Weaknesses._\n\n**1. Results: Results on WOMD (Waymo).**\n\nWe provide results on WOMD using the settings in UniTraj [1], as shown below. The results of other methods are also from UniTraj.\n\n|Method|$minFDE_6$|$minADE_6$|\n|:---:|:---:|:---:|\n|MTR|1.78|0.78|\n|Wayformer|1.65|0.73|\n|AutoBot|1.65|0.73|\n|DeMo (Ours)|1.59|0.75|\n\n**2. Analysis: Better measure query decomposition.**\n\nThank you for this valuable suggestion. We measure the outputs of state queries and mode queries with $minADE$ and $minFDE$, as shown below. We can see that the $minADE_1$ and $minFDE_1$ of the trajectories from state query outputs are better than those from mode query outputs. This means state dynamics are encoded in state queries. Additionally, there are six output trajectories from mode queries, indicating that directional information is predominantly stored in mode queries. The final outputs take advantage of the strengths of both. We will add this analysis to the revised paper.\n\n|Method|$minFDE_1$|$minADE_1$|$minFDE_6$|$minADE_6$|\n|:---:|:---:|:---:|:---:|:---:|\n|state query outputs|3.84|1.52|-|-|\n|mode query outputs|4.12|1.63|1.31|0.67|\n|final outputs|3.93|1.54|1.24|0.64|\n\n**3. Analysis: About RNN-based model and the GRU result.**\n\nAs shown in the left part of Table 5, we conducted an ablation study on an RNN-based model to compare GRU with Mamba. If we directly replace Mamba with GRU to process state queries, it is difficult to achieve convergence, leading to rather poor results, as shown below. Perhaps RNN-based models can process state queries in some other way; this could be a research problem on its own. \n\n||$minFDE_6$|$minADE_6$|$MR_6$|\n|:---:|:---:|:---:|:---:|\n|GRU|1.842|0.923|0.274|\n\n**4. Reproducibility: Architectural description.**\n\nThank you for your question. The layer sizes are provided in Table 7 in the Appendix. For normalization, we use nn.LayerNorm, and for activation, we use nn.GELU. Additional details can be found in the implementation section at Lines 184 and 459.\n\n\n\n## _Response to Questions._\n\n**1. A precise formulation of the auxiliary losses.**\n\nAs in Line 143, we use an MLP to decode state queries into a single future trajectory $Y_f$ and calculate the loss with ground truth $Y_{gt}$ to obtain $L_{ts}$. \n\n$L_{ts} = {\\rm SmoothL1}(Y_f, Y_{gt})$\n\nAs in Line 150, we use MLPs to decode the future trajectories $Y_f$ and probabilities $P_f$. So $L_m$ is shown below: \n\n$Y_{best}, P_{best} = {\\rm SelectBest}(Y_f, Y_{gt})$\n\n$L_{m} = {\\rm SmoothL1}(Y_{best}, Y_{gt}) + {\\rm CrossEntropy}(P_f, P_{best})$\n\n**2. Training time.**\n\nAbout 9 hours in total. Our settings are as mentioned in Line 184 and Line 463.\n\n**3. The model in ID3 (Table 4).**\n\nYes, you are right. Decoupled queries are just concatenated and fed directly to the MLP decoder, and we use two auxiliary losses to optimize the two types of queries.\n\n**4. Model size vary across your ablations (Table 4).**\n\nThe table below shows the model size variations across our ablations in Table 4. Additionally, we perform an ablation on the depth of Attention and Mamba layers, as shown in the right part of Table 6. We can see that even a single layer can achieve decent performance. This indicates that the performance improvements are due to architectural choices rather than merely increasing model capacity.\n\n|ID|model size|\n|:---:|:---:|\n|1|2.0M|\n|2|2.0M|\n|3|2.1M|\n|4|5.8M|\n|5|5.9M|\n\n**5. For inference speed.**\n\nThank you for this valuable suggestion. In most scenarios ($>$ 90\\%), the inference speed is between 25 to 45 ms. In some complex scenarios, it can go up to 70 ms, while in some easy scenarios, it is only 20 ms.\n\n**6. Multi-agent setting.**\n\nWe use queries for each agent to predict their trajectories in the ego agent's coordinate system, so we do not re-encode the scene in each agent-centric frame. This approach avoids making the model costly.\n\n**7. For Line 488 about ZOH in Mamba.**\n\nZOH can discretize continuous signals; the specific formula can be found in the Mamba [2] paper.\n\n> [1] UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction. arXiv preprint:2403.15098, 2024.\n\n> [2] Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv preprint:2312.00752, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4T17EnL8SK", "reviewer": "Reviewer_V1LD", "summary": "The manuscript presents a novel decoupling method for motion forecasting tasks, where the directional intentions are predicted first and the dynamic states following the predicted direction are predicted accordingly. The proposed solution is easy-to-follow, the model size is small, and the experimental results are convincing, achieving the first place of the Argoverse 2 dataset challenge assumed by the reviewer.", "questions": "1. The loss function is simply a sum of all loss terms. Is it possible to have imbalanced losses, or in other words, one loss dominating the training?\n\n2. Is there a Typo at Line 154? $Q_a$ or $Q_s$? \n\n3. Between Lines 135-136, the authors said that $T_s$ and $T_f$ can differ. In this case, is the prediction conducted multiple times to fill the $T_f$ steps? If so, can the directional intention change in this process? If this is possible, can the model handle it?", "limitations": "See above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Simple model design with good performance\n2. Extensive experiment and convincing results.", "weaknesses": "1. Some unclear details about loss function and model design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:\n\n**Q1: About loss function.**\n\nThank you for your careful consideration of our work. As indicated in the right part of Table 5 in our paper, each loss component in our model contributes effectively to the training process, ensuring that no single loss term dominates the optimization. This approach aligns with recent works, such as QCNet [1] and MTR [2], which also employ a simple sum of all loss terms.\n\n**Q2: A Typo at Line 154.**\n\nYes, we apologize for the typo. It should be $Q_s$. We have revised it in the manuscript.\n\n**Q3: About $T_s$ and $T_f$ between Lines 135-136.**\n\nIf $T_s$ and $T_f$ differ, it results in the cases mentioned in the left part of Table 6, where we reduce the number of state queries to make the model more efficient. For example, in Argoverse 2, with $T_f=60$ and $T_s=30$, this corresponds to the case in the third row of the left part of Table 6, where one state query represents two contiguous time steps. There is no need for multiple predictions, and the directional intentions remain unchanged.\n\n> [1] Query-centric trajectory prediction. CVPR, 2023.\n\n> [2] Motion transformer with global intention localization and local movement refinement. NeurIPS, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vw7DfCp6I7", "reviewer": "Reviewer_Rxa7", "summary": "In this paper, a decoupled decoding process integrating intention and future state querying is proposed for motion prediction task. Through hybrid design with attention and Mamba, the proposed DeMo framework achieved strong performance on AV2 and NuScenes benchmarks.", "questions": "1. Whats is the computational cost such as FLOPs and training time of DeMo compared to other methods?", "limitations": "N/A.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A novel decoupled query design for intentions and future states.\n2. A good trial in leveraging Mamba structure for efficient statewise decoding.\n3. Solid performance on prediction benchmarks.", "weaknesses": "1. An over claim for SOTA performance: For instance, the LOF [1] method showcases much better performance in AV2 compared with proposed DeMo. I think a thorough comparison and corrections should be conducted.\n\n2. Heavy computational load for long-horizon state queries.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:\n\n**Q1: Claim for SOTA performance.**\n\nThank you for your attention. Regarding the Contemporaneous Work LOF [1], it was released on arXiv on June 20, 2024, after we submitted our paper to NeurIPS in May 2024. We will add a discussion on LOF to provide a thorough comparison in the revised paper.\n\n**Q2: Heavy computational load for long-horizon state queries.**\n\nTo address this issue, we propose balancing efficiency and performance by utilizing one state query to represent several time steps, as shown in the left part of Table 6. This can effectively reduce the computational load.\n\n**Q3: Computational cost compared to other methods.**\n\nThank you for this valuable suggestion. We have added a table below to further compare computational cost. The experiments are conducted on Argoverse 2 using 8 NVIDIA GeForce RTX 3090 GPUs.\n\n|Method|FLOPs|Training time|Memory|Parameter|Batch size|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|SIMPL [2]|19.7 GFLOPs|8h|14G|1.9M|16|\n|QCNet [3]|53.4 GFLOPs|45h|16G|7.7M|4|\n|DeMo (Ours)|22.8 GFLOPs|9h|12G|5.9M|16|\n\n> [1] FutureNet-LOF: Joint Trajectory Prediction and Lane Occupancy Field Prediction with Future Context Encoding. arXiv preprint: 2406.14422, 2024.\n\n> [2] SIMPL: A Simple and Efficient Multi-agent Motion Prediction Baseline for Autonomous Driving. IEEE Robotics and Automation Letters, 2024.\n\n> [3] Query-centric trajectory prediction. CVPR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces DeMo, a novel framework for motion forecasting in autonomous driving systems. DeMo decouples the motion forecasting task into two components: mode queries for capturing directional intentions and state queries for modeling dynamic states over time. The framework employs a combination of Attention and Mamba techniques for global information aggregation and state sequence modeling.

The reviewers generally agree that the paper presents an innovative approach with solid technical foundations. Reviewer ZMh7 notes that "This is a very strong and well-presented manuscript" and highlights the "excellent" presentation and comprehensive ablation studies. Reviewer V1LD praises the "Simple model design with good performance" and "Extensive experiment and convincing results."

The experimental results on Argoverse 2 and nuScenes benchmarks are seen as a strength by most reviewers. Reviewer ZMh7 states that "In both cases DeMo is clearly SOTA," while Reviewer V1LD mentions that the paper achieves "the first place of the Argoverse 2 dataset challenge."

However, there are some concerns and areas for improvement:
-  Clarity of technical descriptions: Reviewer pCB4 points out that "The technical explanations of the methods and algorithms used in DeMo are not sufficiently clear." This sentiment is echoed by Reviewer ZMh7, who notes that the model is not reproducible from the manuscript alone.
- Comparison with state-of-the-art models: Reviewer pCB4 and Reviewer Rxa7 both mention that the paper doesn't compare DeMo to some models with superior performance on existing leaderboards. Reviewer Rxa7 specifically mentions the LOF method as having "much better performance in AV2 compared with proposed DeMo."
- Computational cost: Reviewer Rxa7 raises concerns about the "Heavy computational load for long-horizon state queries" and asks for more information on the computational cost compared to other methods.
- Additional benchmarks: Reviewer ZMh7 suggests including results on the WOMD (Waymo) benchmark, noting it as "probably the most important (or co-most-important) benchmark in this space right now."
- Further analysis: Reviewer ZMh7 requests more evidence to support the motivating intuitions behind the query decomposition and auxiliary losses.

Recommendation: Despite these concerns, the overall sentiment from the reviewers is positive. Three out of four reviewers recommend acceptance (one "Accept" and two "Weak Accept"), while one reviewer gives a "Borderline Accept."

Based on these reviews, the recommendation is to accept the paper, contingent on the authors addressing the following key points in the camera-ready version:
- Improve the clarity of technical descriptions, especially regarding the auxiliary losses and architectural details.
Provide a more comprehensive comparison with state-of-the-art models, including those mentioned by the reviewers (e.g., LOF, SEPT, SEPT++, QCNet).
- Include information on computational costs and training time.
- If possible, include results on the WOMD benchmark or explain why it was not included.
- Provide additional analysis to support the intuitions behind the query decomposition and auxiliary losses.
- Address the specific questions raised by the reviewers, particularly those related to loss functions, model design, and handling of changing directional intentions.
Decision: Accept (poster)