DeMo: Decoupling Motion Forecasting into  Directional Intentions and Dynamic States

Authors: bozhou  zhang
nan  song
li  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-27 07:01:07

Abstract: Accurate motion forecasting for traffic agents is crucial for ensuring the safety and efficiency of autonomous driving systems in dynamically changing environments. Mainstream methods adopt a one-query-one-trajectory paradigm, where each query corresponds to a unique trajectory for predicting multi-modal trajectories. While straightforward and effective, the absence of detailed representation of future trajectories may yield suboptimal outcomes, given that the agent states dynamically evolve over time. To address this problem, we introduce DeMo, a framework that decouples multi-modal trajectory queries into two types: mode queries capturing distinct directional intentions and state queries tracking the agent's dynamic states over time. By leveraging this format, we separately optimize the multi-modality and dynamic evolutionary properties of trajectories. Subsequently, the mode and state queries are integrated to obtain a comprehensive and detailed representation of the trajectories. To achieve these operations, we additionally introduce combined Attention and Mamba techniques for global information aggregation and state sequence modeling, leveraging their respective strengths. Extensive experiments on both the Argoverse 2 and nuScenes benchmarks demonstrate that our DeMo achieves state-of-the-art performance in motion forecasting. In addition, we will make our code and models publicly available.

Main paper: 
 introduction: 
 Motion forecasting [29, 58, 67] empowers self-driving vehicles to anticipate how surrounding agents will move and affect the ego car, providing references and conditions for the ego-action. It is critical for maintaining safety and dependability, enabling vehicles to comprehend the dynamics of driving environments and make calculated decisions. The challenges and complexities of this task arise from various factors, including unpredictable road conditions, varied movement patterns of traffic participants, and the necessity to simultaneously analyze the states of observed agents along with the road maps.The research community has witnessed significant progress in the representation of driving scenes [18, 36, 44] and the paradigm of trajectory decoding [26, 38, 54, 77] . These methods have achieved substantial advancements in prediction accuracy, primarily following a certain pattern inspired from detection [4, 40] , i.e., the one-query-one-trajectory paradigm [38, 54, 59, 77] . This paradigm utilizes several queries to represent different estimated trajectories, as shown in Figure 1 (a) , covering the possibilities of distinct motion intentions. Although effective, these approaches can only approximately provide a direction and collect surroundings to generate various trajectory waypoints in a one-shot fashion, overlooking the detailed relationships with scenes. The lack of concrete representation for trajectories and comprehensive spatiotemporal interactions with the surrounding To solve this problem, we propose a novel framework dubbed DeMo, which provides a detailed representation of multi-modal trajectories. Specifically, we decouple forecasting queries into two types: besides the original motion mode queries to capture different directional intentions as shown in Figure 1 (a), we introduce the dynamic state queries for future trajectories to track the agent's dynamic states across various time steps, as shown in Figure 1 (b) . This approach allows us to achieve a comprehensive query representation within our framework, as illustrated in Figure 1 (c) . Mode queries and state queries are processed using the Mode Localization Module and the State Consistency Module, respectively. These modules enable the explicit interactions of queries with the surrounding environments and among each other, by which the directional accuracy and temporal consistency of future trajectories are significantly optimized. Subsequently, two types of queries are integrated by our Hybrid Coupling Module to achieve a comprehensive representation of future trajectories. Due to the sequential nature of trajectory states, Mamba is particularly well-suited for modeling the temporal consistency of dynamic states. Therefore, we utilize a combination of Attention and Mamba in our modules to effectively and efficiently aggregate global information and model state sequences, leveraging the strengths of both techniques.Our contributions are summarized as follows: (i) We propose a motion forecasting framework that decouples multi-modal trajectory queries into mode queries and state queries to represent directional intentions and dynamic states, respectively. (ii) We design three modules based on integrated Attention and Mamba to process decoupled mode queries, state queries, and coupled mode and state queries. (iii) Extensive experiments on both the Argoverse 2 and nuScenes benchmarks demonstrate that DeMo achieves state-of-the-art performance. 
related work: 
 Motion forecasting. In recent advancements in autonomous driving, it is critical to effectively predict the movements of relevant agents by accurately representing scene components. Traditional methods [5, 20, 48] transformed driving scenarios into image formats and used conventional convolutional networks for scene context encoding. However, these techniques often failed to sufficiently capture intricate structural details. This challenge has led to the adoption of vectorized scene representations [26, 60, 75, 79] , exemplified by the introduction of VectorNet [18] . Additionally, graph-based structures are also widely utilized to represent the relationships between agents and their environments [14, 21, 30, 31, 36, 50, 69] .Existing methodologies have delved into a variety of frameworks to predict multi-modal future trajectories given the scene features. Initially, prediction techniques were centered on goal-oriented methods [26, 71] or employed probability heatmaps to sample trajectories [20, 21] . However, contemporary strategies, such as MTR [54] and QCNet [77] , among others [41, 43, 44, 73] , utilize Transformer [61] models to analyze relationships within the scene. Additionally, the introduction of novel paradigms such as pre-training [7, 8, 34] , historical prediction design [46, 59] , GPT-style next-token prediction [49, 53] , and post-refinement [9, 76] in some techniques has led to remarkable advancements in performance.Furthermore, the advancements in multi-agent forecasting aim to enhance the applicability of predicted trajectories for various agents in real-world scenarios. Several approaches [22, 55, 79] follow an agent-centric model, where trajectories are forecasted individually for each agent, a process that might be slow. On the other hand, alternative approaches [44, 78] utilize a scene-centric model that allows for simultaneous forecasting across all agents, introducing an innovative approach to trajectory prediction.Inspired by the progress in object detection and motivated by its significant success [4, 40] , mainstream methods [38, 54, 59, 77 ] have adopted a one-query-one-trajectory paradigm to achieve high performance in motion forecasting benchmarks [6, 58, 67] . These methods use transformers to model the relationship between each trajectory query and its environment but lack detailed representation. We propose decoupled mode queries and state queries for a more detailed and comprehensive representation of multi-modal trajectories.State space models. Originally developed for modeling dynamic systems with state variables in fields such as control theory, state space models (SSMs) have emerged as promising alternatives to Transformers [61] in sequence modeling, particularly due to their effectiveness in addressing attention complexity and capturing long-term dependencies. As SSMs have evolved [17, 25, 56] , a new class termed Mamba [24] , which incorporates selection mechanisms and hardware-aware architectures, has recently demonstrated significant promise in long-sequence modeling. Several studies have explored Mamba's substantial potential across a range of fields, including natural language processing [27, 37] and computer vision [28, 35, 74, 80] . Notably, in the vision domain, Mamba has demonstrated superior GPU efficiency and effectiveness compared to Transformers in tasks such as visual representation learning [80] , video understanding [35] , and human motion generation [74] . Building on these achievements, to the best of our knowledge, this is the first method to combine the strengths of Mamba with mainstream Transformer-based architecture to achieve impressive performance.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 In this section, we present DeMo, which utilizes decoupled mode queries and state queries for directional intentions and dynamic states to predict future trajectories. We also employ a hybrid architecture combining Attention and Mamba, along with two auxiliary losses for feature modeling.Given HD map and agents in the driving scenario, motion forecasting aims to predict the future trajectories for the interested agents. The HD map comprises several polylines of lanes or crossings, while agents are traffic participants like vehicles and pedestrians. To transform these elements into easily processable and learnable inputs, we utilize a popular vectorized representation following [18] . Specifically, the map M ∈ R Nm×L×Cm is generated by dividing each line into several shorter segments, where N m , L, and C m denote the number of map polylines, divided segments and feature channels, respectively. We represent the historical information of agents as A ∈ R Na×T h ×Ca , where N a , T h , and C a are the number of agents, historical timestamps, and motion states (e.g., position, heading angle, velocity). Additionally, the future trajectories A f ∈ R Naoi×T f ×2 for agents of interest are estimation objectives, with N aoi , T f indicating the number of selected agents and the future timestamps, respectively.Given the vectorized representations A for agents and M for HD map, we first employ individual encoders to process them separately. Specifically, we use a PointNet-based polyline encoder, as described in [8, 54, 55] , to process the map representation M, generating the map features F m ∈ R Nm×C . For the agents A, we replace generic Transformer [61] or RNN with several Unidirectional Mamba [24] blocks, which are more efficient and effective for sequence encoding, to aggregate the historical trajectory features F a ∈ R Na×C up to the current time. Subsequently, the scene context features F s ∈ R (Na+Nm)×C are formed by concatenating them and further propagated to a Transformer encoder for intra-interaction learning. The overall process can be formulated as:F m = PointNet(M), F a = UniMamba(A), F s = Transformer(Concat(F a , F m )). (1)After obtaining the scene context features, we aim to decode multi-modal future trajectories for each interested agent based on our proposed decoupled queries. As illustrated in Figure 2 , the decoder network comprises a State Consistency Module that enhances the consistency and accuracy of dynamic future state queries, a Mode Localization Module for learning distinct motion modes, and a Hybrid Coupling Module to integrate the decoupled queries and generate the final output. The detailed description of these components are provided in the following. Dynamic state consistency. Considering the recurrence and causality of the future trajectories A f , we propose to represent them as a series of dynamic states across various time steps, distinct yet interconnected. To preserve precise time information, the state queries Q s ∈ R Naoi×Ts×C are initialized with an MLP module for real-time differences. It is notable that the steps T s can differ from T f to balance the effectiveness and efficiency, especially when predicting long-term future trajectories or a higher frequency of future trajectories. The State Consistency Module is then employed to enhance the consistency of the state queries and aggregate the specific scene context, which can be formulated as follows:Q s = MLP([t 1 , t 2 , • • • , t Ts ]), Q s = MultiHeadAttn(Q = Q s , K = F s , V = F s ), Q s = BiMamba(Q s ).(2) Specifically, cross-attention is first applied to enable state queries to interact with the scene context, followed by a Mamba block to model sequence relationships with linear-time complexity. Simultaneously, to account for the influences of rear state queries on the front ones, we adopt the bidirectional Mamba [35, 80] for both forward and backward scanning. Additionally, a simple MLP module is utilized to decode the state queries Q s into a single future trajectory for explicit supervision of time consistency.Mode queries Q m ∈ R Naoi×K×C represent different motion modes, with each query responsible for decoding one of the K trajectories. We utilize the Mode Localization Module to localize the potential directional intentions, as shown below:EQUATIONFor spatial motion learning, two Multi-Head Attention blocks are employed to enable interactions among mode queries and with the scene context. Additionally, we also employ simple MLPs to decode the future trajectories and probabilities. Similarly, we introduce another auxiliary supervision to endow mode queries with distinct motion intentions.Hybrid query coupling. To incorporate dynamic states and directional intentions, we simply add Q m and Q s together to form the hybrid spatiotemporal queries Q h ∈ R Naoi×K×Ts×C . Then, the Hybrid Coupling Module is utilized to further process Q h and yield a comprehensive representation for future trajectories, as formulated below:EQUATIONBesides the Attention and Mamba modules for interaction with the scene context, among modes, and across time states, we additionally introduce a hybrid self-attention layer, which connects queries across both time and modes, boosting the diversity of predicted trajectories. The change in feature dimensions in this module is shown in Figure 2 . The final predictions are generated by decoding the output Q h into trajectory positions and probabilities with MLPs.DeMo is trained with three component losses in an end-to-end manner. Primarily, the regression loss L reg and the classification loss L cls are employed to supervise the accuracy of predicted trajectories and their associated probability scores. Additionally, we introduce two auxiliary losses, L ts and L m , for intermediate features of time states and motion modes, respectively. The former enhances the coherence and causality of dynamic states across various time steps, while the latter endows mode with distinct directional intentions. The overall loss L is a combination of these individual losses with equal weights, formulated as follows:EQUATIONWe adopt the cross-entropy loss for probability score classification and the smooth-L1 loss for trajectory regression tasks. The winner-take-all strategy is employed, optimizing only the best prediction with minimal average prediction error to the ground truth.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification: The details of the model and experiments are provided in the main paper and the appendix.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental settings: 
 Datasets. We evaluate our method's performance using the Argoverse 2 [67] and nuScenes [3] motion forecasting datasets. The Argoverse 2 dataset comprises 250,000 scenarios with a sampling frequency of 10 Hz, each featuring a 5s historical trajectory length and predicting a 6s future ones. The nuScenes dataset contains 1,000 scenes at 2 Hz, predicting the next 6s trajectories with the past 2s history. For multi-agent scenarios, we use avgM inADE, avgM inF DE, and actorM R. The Argoverse 2 dataset is evaluated across six prediction modes, while nuScenes is evaluated across ten prediction modes. We typically follow the evaluation metrics from the official leaderboard, setting K to 1 and 6 for the Argoverse 2 dataset, and K to 5 and 10 for the nuScenes dataset.Implementation details. Our models are trained for 60 epochs using the AdamW [42] optimizer, with a batch size of 16 per GPU. The training is conducted end-to-end with a learning rate of 0.003 and a weight decay of 0.01. We adopt an agent-centric coordinate system and sample scene elements within a 150-meter radius of the agents of interest. All experiments are conducted on 8 NVIDIA GeForce RTX 3090 GPUs. Additional details and further experiments are provided in Appendix A and Appendix B.We first compare our method, DeMo with several models on the Argoverse 2 [67] motion forecasting benchmark for the single-agent setting as demonstrated in Table 1 . To ensure a comprehensive and fair comparison, we separately evaluate the performance of different methods with and without the model ensembling technique. It is shown that DeMo has significantly outperformed all previous approaches, including the state-of-the-art model QCNet [77] and its post-refinement enhancement SmartRefine [76] . Concretely, our method distinctly surpasses other methods across all metrics, particularly in terms of minF DE 1 and minADE 1 , where it demonstrates performance improvements of 13.02% and 11.83% relative to QCNet, respectively. After using ensembling techniques similar to other entries, DeMo surpasses all methods on all metrics by a large margin. Then, we compare the performance of DeMo on the nuScenes [3] motion forecasting benchmark, with the results of the test split presented in Table 2 . Our method is also superior to others over all metrics except minADE 5 . In multi-agent environments, it is essential for predictors to simultaneously forecast the future paths of all relevant agents to comprehensively understand the driving situation. To validate the efficacy of our model, DeMo, we conduct tests on the Argoverse 2 multi-agent dataset [67] . The results, presented in Table 3 , show that despite lacking the specialized multi-agent forecasting features found in models such as [44, 55, 78] , our model surpasses recent advancements across all evaluated metrics due to our novel designs.Effects of components. Table 4 demonstrates the effectiveness of each component in our method. We show the baseline in the first row, which is similar to previous methods [54, 77] and utilizes mode queries to generate multi-modal future trajectories. Then, we directly adopt state queries in the second row (ID-2) to decode the trajectories. A performance decline is observed due to the surplus queries, which impose a burden on the model and make it difficult to distinguish the meanings of different types. In the third row (ID-3), we introduce two auxiliary losses, resulting in a slight improvement compared to the first row. Although the model can identify what each query represents, it demonstrates only moderate performance due to the limited information. In the fourth row (ID-4), we incorporate the three aggregation modules in Figure 2 but remove auxiliary losses, leading to significant performance enhancements. Finally, in the fifth row (ID-5), our DeMo integrates all these techniques and achieves outstanding performance.Effects of state sequence modeling with Mamba. Mamba excels at sequence modeling, so we utilize Bidirectional Mamba [35, 80] to enhance the consistency of states across different time steps.To demonstrate its effectiveness, we compare Bidirectional Mamba with several other modules, including Unidirectional Mamba [24] , Attention, Conv1d, and GRU [10] . As illustrated in the left part of Table 5 , our Bidirectional Mamba configuration outperforms the others due to its specialized design for sequence modeling, compared to Attention, and its capability to perform both forward and backward scans, unlike Unidirectional Mamba. Effects of auxiliary losses and aggregation modules. We conduct an ablation study to assess the impacts of auxiliary losses and aggregation modules. As illustrated in the right part of Table 5 , removing any of these losses or modules leads to a performance decline in the model. Notably, the aggregation modules have a greater impact than the auxiliary losses. This is attributed to the critical role of learning information from the scene context and from each other, which is essential for decoupling queries to represent distinct meanings.Effects of state queries. We conduct an ablation study on the number of state queries, as shown in the left part of Table 6 . In our default setting, we use 60 state queries to represent the future states at 60 timestamps. As we gradually reduce the number of state queries, we observe a performance decline due to the increasing ambiguity of the state query meanings.Effects of the depth of Attention and Mamba blocks. A suitable depth configuration of Attention and Mamba units is crucial for achieving an optimal balance between efficiency and performance. As depicted in the right part of Table 6 , we conduct an ablation study focusing on the layer depth. It is observed that the best results are obtained with Attention units at a depth of three and Mamba units at a depth of two. Effects of the depth of Mamba blocks in the encoder. We add ablation studies on the Mamba for encoding agent historical information in the encoder of our DeMo. As shown in Table 7 , the left part shows different modules for encoding the historical information of agents. Our goal is to aggregate historical information up to the present time, making Unidirectional Mamba the most suitable choice.The right part presents an ablation study concerning the number of Mamba blocks, indicating that three layers yield the optimal performance. Balancing performance, inference speed, and model size is important for the model deployment. We compare our DeMo with two recent, representative models: the state-of-the-art QCNet [77] and its enhancement through post-refinement, SmartRefine [76] . The size of our model is 5.9M, in contrast to 7.7M for QCNet and 8.0M for SmartRefine. Despite its smaller size, our model demonstrates superior performance by a significant margin, as detailed in Table 1 . As for inference speed, we compare DeMo with QCNet, both end-to-end methods. Measurements are conducted on the Argoverse 2 single-agent validation set using an NVIDIA GeForce RTX 3090 GPU, with a maintained batch size of one. The average inference speed of DeMo is only 38ms, which is approximately 2.5 times faster than QCNet's 94ms. This demonstrates that our method is not only superior to QCNet but also more efficient.In Figure 3 , we present qualitative results of our network. The results of the baseline model, which lacks the decoupled query paradigm, are shown in panel (a), while the results of our DeMo are shown in panel (b). From the first two rows, it is evident that by explicitly optimizing the dynamic states of future trajectories, our model predicts trajectories that are more accurate and closer to the ground truth. From the third row, it is apparent that our model can better capture potential directional intentions. Additional qualitative results and failure cases are detailed in Appendix D and E.We provide a comparison of computational cost with resent representative methods in Table 9 . The experiments are conducted on Argoverse 2 [67] dataset using 8 NVIDIA GeForce RTX 3090 GPUs.  
conclusion: 
 In this paper, we introduce DeMo, which redefines the motion forecasting task by decoupling it into expressions of directional intentions and dynamic states. We utilize state queries to model various states across different time, and mode queries to capture the agent's motion intentions. Our approach incorporates three aggregation modules, combining Attention and Mamba for effective modeling. Comprehensive experiments, covering both single-agent and multi-agent scenarios, indicate that DeMo outperforms the current state-of-the-art methods. This highlights its potential as a promising approach for achieving safe and reliable motion forecasting in the rapidly advancing field of autonomous driving.Limitations and future work. The proposed framework adopts a decoupled query paradigm, which may lead to heavier models due to the need to predict longer trajectories. Our current model design does not sufficiently take model efficiency into account. In the future, we plan to use sparse states for modeling trajectories, thereby making the framework more deployment-friendly.  
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: [NA] Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Our main idea is specifically designed for applications in real-world autonomous driving.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: See Section 4.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: To fully demonstrate the effectiveness of our DeMo, we compare it with several recent models on the Argoverse 1 [6] dataset. The results from the validation split are shown in Table 11 , indicating that our model achieves impressive performance. [41] 0.71 1.15 0.11 LaneGCN [36] 0.71 1.08 -SSL-Lanes [2] 0.70 1.01 0.09 PAGA [12] 0.69 1.02 -DSP [71] 0.69 0.98 0.09 FRM [47] 0.68 0.99 -ADAPT [1] 0.67 0.95 0.08 SIMPL [72] 0.66 0.95 0.08 HiVT [79] 0.66 0.96 0.09 R-Pred [9] 0.66 0.95 0.09 HPNet [59] 0.64 0.87 0.07 DeMo (Ours) 0.59 0.90 0.07We provide a performance comparison of the top methods on the Argoverse 2 leaderboard as of September 2024. The results for the single-agent setting are shown in Table 12 , and the results for the multi-agent setting are shown in Table 13 .We provide results on WOMD [58] in Table 14 using the settings in UniTraj [16] , as shown below.The results of other methods are also from UniTraj.In our approach, we use model ensembling, an essential technique to enhance the accuracy of final predictions. We train six sub-models with various random seeds and training epochs, resulting in 36 predicted future trajectories for each agent. We then apply k-means clustering with six cluster centers to process these trajectories. For each cluster group, we compute the average trajectory within the group to determine the final trajectories. We present the results, both with and without the model ensembling technique, on the Argoverse 2 single-agent test set in Table 1 .Mamba is inspired by a continuous system that maps a 1-D function or sequence x(t) ∈ R to y(t) ∈ R, utilizing a hidden state h(t) ∈ R N . In this system, A ∈ R N ×N serves as the evolution parameter, while B ∈ R N ×1 and C ∈ R 1×N act as the projection parameters.EQUATIONMamba represents the discrete version of a continuous system and includes a timescale parameter ∆ to convert the continuous parameters A and B into discrete counterparts A and B. The most commonly used method for this transformation is zero-order hold (ZOH). After discretizing A and B, the discretized form of Equation ( 8) using a step size of ∆ can be reformulated as:EQUATIONFinally, the models compute the output through a global convolution that utilizes a structured convolutional kernel K ∈ R M , where M represents the length of the input sequence x.EQUATIONWe provide more qualitative results of our DeMo in Figure 5 .Although our DeMo has demonstrated exceptional performance on motion forecasting benchmarks, it is not without its failures. We analyze these typical cases and present qualitative results to give readers insight into the scenarios where our model might underperform. This analysis is intended to support future efforts in developing an algorithm that is both more robust and powerful, as illustrated in Figure 4 . In the first row, the vehicle will turn into an alley, reflecting a kind of subjective driving behavior. However, the model predicts that the vehicle will just keep going straight. To improve predictions in cases like this, we could enhance how the model interacts with additional information about what the vehicle intends to do, such as adding visual cues, such as turn signals. In the second row, the agent needs to navigate through a complex intersection to reach one of the roads; however, the model fails to accurately predict this driving behavior. This inaccuracy may be caused by a lack of comprehensive understanding of the complex map topology and the unbalanced distribution of driving data, addressing the issue of data balance is necessary to solve this problem. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.* Li Zhang (lizhangfd@fudan.edu.cn) is the corresponding author. 


Decision: This paper introduces DeMo, a novel framework for motion forecasting in autonomous driving systems. DeMo decouples the motion forecasting task into two components: mode queries for capturing directional intentions and state queries for modeling dynamic states over time. The framework employs a combination of Attention and Mamba techniques for global information aggregation and state sequence modeling.

The reviewers generally agree that the paper presents an innovative approach with solid technical foundations. Reviewer ZMh7 notes that "This is a very strong and well-presented manuscript" and highlights the "excellent" presentation and comprehensive ablation studies. Reviewer V1LD praises the "Simple model design with good performance" and "Extensive experiment and convincing results."

The experimental results on Argoverse 2 and nuScenes benchmarks are seen as a strength by most reviewers. Reviewer ZMh7 states that "In both cases DeMo is clearly SOTA," while Reviewer V1LD mentions that the paper achieves "the first place of the Argoverse 2 dataset challenge."

However, there are some concerns and areas for improvement:
-  Clarity of technical descriptions: Reviewer pCB4 points out that "The technical explanations of the methods and algorithms used in DeMo are not sufficiently clear." This sentiment is echoed by Reviewer ZMh7, who notes that the model is not reproducible from the manuscript alone.
- Comparison with state-of-the-art models: Reviewer pCB4 and Reviewer Rxa7 both mention that the paper doesn't compare DeMo to some models with superior performance on existing leaderboards. Reviewer Rxa7 specifically mentions the LOF method as having "much better performance in AV2 compared with proposed DeMo."
- Computational cost: Reviewer Rxa7 raises concerns about the "Heavy computational load for long-horizon state queries" and asks for more information on the computational cost compared to other methods.
- Additional benchmarks: Reviewer ZMh7 suggests including results on the WOMD (Waymo) benchmark, noting it as "probably the most important (or co-most-important) benchmark in this space right now."
- Further analysis: Reviewer ZMh7 requests more evidence to support the motivating intuitions behind the query decomposition and auxiliary losses.

Recommendation: Despite these concerns, the overall sentiment from the reviewers is positive. Three out of four reviewers recommend acceptance (one "Accept" and two "Weak Accept"), while one reviewer gives a "Borderline Accept."

Based on these reviews, the recommendation is to accept the paper, contingent on the authors addressing the following key points in the camera-ready version:
- Improve the clarity of technical descriptions, especially regarding the auxiliary losses and architectural details.
Provide a more comprehensive comparison with state-of-the-art models, including those mentioned by the reviewers (e.g., LOF, SEPT, SEPT++, QCNet).
- Include information on computational costs and training time.
- If possible, include results on the WOMD benchmark or explain why it was not included.
- Provide additional analysis to support the intuitions behind the query decomposition and auxiliary losses.
- Address the specific questions raised by the reviewers, particularly those related to loss functions, model design, and handling of changing directional intentions.

Reviews: review: paper_summary: paper_summary: The paper presents DeMo, a novel framework for motion forecasting in autonomous driving systems. DeMo decouples the motion forecasting task into two distinct components: mode queries for capturing directional intentions and state queries for modeling dynamic states over time. This separation allows DeMo to separately optimize for multi-modality and dynamic state evolution, leading to a more comprehensive representation of future trajectories. The framework employs a combination of Attention and Mamba techniques for global information aggregation and state sequence modeling. Extensive experiments on the Argoverse 2 and nuScenes benchmarks demonstrate that DeMo achieves state-of-the-art performance in motion forecasting.
strength_weakness: strength_weakness: There are two major weaknesses: 1. Technical Descriptions Lack Clarity: The technical explanations of the methods and algorithms used in DeMo are not sufficiently clear, making it challenging for readers to fully understand the proposed techniques and their implementations. 2. Unconvincing Contributions: The claimed contributions of the paper are not entirely convincing. For instance, the paper doesn’t compare DeMo to models that have superior performance on existing leaderboards, questioning the novelty of its improvements. Additionally, there are previous works (e.g. Motion Mamba by Zhang et.al.) that have already combined Mamba and Transformer techniques for time series motion data, which undermines the claim that DeMo introduces a novel approach.
questions: questions: 1. In the first line of Eq. (2), what do the symbols {$\{ t_1, t_2, ...t_{T_s} \}$} represent, and how do $T_s$ and $T_f$ differ from or relate to each other? How are these time values obtained and utilized in the model?

2. For the motion model $Q_m$, what specific features are considered for different motion models? The paper does not clearly specify the features used in these motion models.

3. In Section 3.4, the loss function $L_{ts}$ is described as related to "intermediate features of time state." What exactly are these intermediate features, and what is their role in the model? This concept is not clearly defined in the paper.

4. Table 1 does not include baseline methods such as SEPT and SEPT++ that have better performance and earlier submission dates than DeMo. Similarly, Table 3 omits models like QCNet, which outperform DeMo. Why are these models not included in the comparisons?

5. The paper lacks detailed information about the model’s parameters and settings. Can you provide more specific details on the parameters used for DeMo?
limitations: limitations: The technical descriptions lack clarity and the overall contribution is limited.
paper_summary: paper_summary: DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic States introduces a state of the art model architecture for motion forecasting (predicting the future trajectories of road actors for the purpose of autonomous driving).  The authors make two notable contributions.  First they provide an alternative to the typical one-query one-trajectory paradigm of current high performing models.  Instead they decompose queries into "mode queries" which attempt to capture directional intentions and "state queries" which attempt to capture the dynamic properties of a trajectory.  The authors also replace the sequence processing portion of the encoder (typically cross attention layers in current models or RNNs in older models) with Mamba blocks.  The authors include numerous ablations demonstrating that each of these contributions plays a substantive role in the performance of their model.  The authors present results on two significant motion forecasting benchmarks.
strength_weakness: strength_weakness: This is already quite a solid manuscript, however I hope that the authors can selectively address a few of the weaknesses and questions below to arrive at an even better paper.

### Results
While AV2 and NuScenes are both important benchmarks, WOMD (Waymo) is probably the most important (or co-most-important) benchmark in this space right now.  It would have been nice to see results on WOMD.  There are tools like https://github.com/vita-epfl/UniTraj now to make this easier.

### Analysis
- While tables 4 and 5 clearly demonstrate that the query decomposition and auxilary losses contribute significantly to the model performance, there is little evidence to support the motivating intuitions.  For example, the introduction of state queries and the associated auxiliary loss is supposed to produce better dynamics.  Can we come up for a metric to measure that?  Can we find ways to demonstrate that our decomposition works as we expect?  Directional information is predominantly stored in mode queries while dynamics are encoded in state queries?
- I would have liked to see the authors get an RNN based model to successfully converge.  (The GRU result is disappointing).


### Reproducibility
The authors promise to release code (which would entirely alleviate my concern here), however I do not feel that the model is reproducible from the manuscript alone.  There is very little information about the auxilary losses.  Additionally there is no supplementary diagram or table with an explicit architectural description (layer sizes, normalization, etc).
questions: questions: - Can you provide a precise formulation of the auxilary losses?
- Can you report training time?
- How does the model in ID3 (table 4) work?  Are the decouple queries just concatenated and fed directly to the MLP decoder?
- How does the model size vary across your ablations (table 4)?  I.e. why should I ascribe the performance improvements to architectural choices and not just growing model capacity?
- For inference speed can you report 99th percentile (or similar) rather than mean?  The AV industry is primarily concerned with worst-case performance.
- How does the multi-agent set up work?  Do I re-encode the scene in each agent-centric frame?
- Line 488 -- the text in this paragraph makes it sound like ZOH is a continuous->discrete transformation.  I might be wrong here, but I typically think of it as a digital->analog transformation.  Are we really doing something like the inverse of ZOH?
limitations: limitations: No concerns.
paper_summary: paper_summary: The manuscript presents a novel decoupling method for motion forecasting tasks, where the directional intentions are predicted first and the dynamic states following the predicted direction are predicted accordingly. The proposed solution is easy-to-follow, the model size is small, and the experimental results are convincing, achieving the first place of the Argoverse 2 dataset challenge assumed by the reviewer.
strength_weakness: strength_weakness: 1. Some unclear details about loss function and model design.
questions: questions: 1. The loss function is simply a sum of all loss terms. Is it possible to have imbalanced losses, or in other words, one loss dominating the training?

2. Is there a Typo at Line 154? $Q_a$ or $Q_s$? 

3. Between Lines 135-136, the authors said that $T_s$ and $T_f$ can differ. In this case, is the prediction conducted multiple times to fill the $T_f$ steps? If so, can the directional intention change in this process? If this is possible, can the model handle it?
limitations: limitations: See above.
paper_summary: paper_summary: In this paper, a decoupled decoding process integrating intention and future state querying is proposed for motion prediction task. Through hybrid design with attention and Mamba, the proposed DeMo framework achieved strong performance on AV2 and NuScenes benchmarks.
strength_weakness: strength_weakness: 1. An over claim for SOTA performance: For instance, the LOF [1] method showcases much better performance in AV2 compared with proposed DeMo. I think a thorough comparison and corrections should be conducted.

2. Heavy computational load for long-horizon state queries.
questions: questions: 1. Whats is the computational cost such as FLOPs and training time of DeMo compared to other methods?
limitations: limitations: N/A.

Comments: comment: Dear Reviewer Rxa7

We appreciate the reviewer's time for reviewing and thanks again for the valuable comments and the positive score! 

Best wishes

Authors
Dear Reviewer pCB4

We appreciate the reviewer's time for reviewing and thanks again for the valuable comments and the improved score! We will revise and refine the paper as suggested in the revision.

Best wishes

Authors
Thank you for the reply, and I appreciate the additional information, which does help clarify some points for the reader. Regarding the methods on the leaderboard, I suggest including all relevant methods rather than selectively comparing with those that perform worse than the proposed model. Explaining why certain methods outperform the proposed model would not necessarily diminish the overall contribution of the paper. Additionally, it would be beneficial to address why one model performs better than another under different circumstances, as well as the limitations of the proposed method compared to others. Given the added details and improved clarity, I am inclined to raise my score.
Thanks very much for perfectly addressing my problems. I would like to raise the Soundness, Contribution and maintaining the general rating.
Dear Reviewer pCB4,

We appreciate your time for reviewing, and we really want to have a further discussion with you to see if our response solves the concerns. We have addressed all the thoughtful questions raised by the reviewer (eg, technical descriptions and contributions), and we hope that our work’s impact and results are better highlighted with our responses. It would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you!

Best wishes,

Authors
Dear Reviewer V1LD

We appreciate the reviewer's time for reviewing and thanks again for the valuable comments.

Best wishes

Authors
The authors have addressed most of my concerns. I will maintain my current score.