Title: Motion Forecasting in Continuous Driving
Venue: NeurIPS 2024
Accepted: yes
Keywords: Motion forecasting; Autonomous Driving;
Abstract: Motion forecasting for agents in autonomous driving is highly challenging due to the numerous possibilities for each agent's next action and their complex interactions in space and time. 
In real applications, motion forecasting takes place repeatedly and continuously as the self-driving car moves. However, existing forecasting methods typically process each driving scene within a certain range independently, totally ignoring the situational and contextual relationships between successive driving scenes. This significantly simplifies the forecasting task, making the solutions suboptimal and inefficient to use in practice. To address this fundamental limitation, we propose a novel motion forecasting framework for continuous driving, named RealMotion.
It comprises two integral streams both at the scene level:
(1) The scene context stream progressively accumulates historical scene information until the present moment, capturing temporal interactive relationships among scene elements.
(2) The agent trajectory stream optimizes current forecasting by sequentially relaying past predictions.
Besides, a data reorganization strategy is introduced to narrow the gap between existing benchmarks and real-world applications, consistent with our network. These approaches enable exploiting more broadly the situational and progressive insights of dynamic motion across space and time. 
Extensive experiments on Argoverse series with different settings demonstrate that our RealMotion achieves state-of-the-art performance, along with the advantage of efficient real-world inference.

Motion Forecasting in Continuous Driving

Abstract

Motion forecasting for agents in autonomous driving is highly challenging due to the numerous possibilities for each agent's next action and their complex interactions in space and time. In real applications, motion forecasting takes place repeatedly and continuously as the self-driving car moves. However, existing forecasting methods typically process each driving scene within a certain range independently , totally ignoring the situational and contextual relationships between successive driving scenes. This significantly simplifies the forecasting task, making the solutions suboptimal and inefficient to use in practice. To address this fundamental limitation, we propose a novel motion forecasting framework for continuous driving, named RealMotion . It comprises two integral streams both at the scene level : (1) The scene context stream progressively accumulates historical scene information until the present moment, capturing temporal interactive relationships among scene elements. (2) The agent trajectory stream optimizes current forecasting by sequentially relaying past predictions. Besides, a data reorganization strategy is introduced to narrow the gap between existing benchmarks and real-world applications, consistent with our network. These approaches enable exploiting more broadly the situational and progressive insights of dynamic motion across space and time. Extensive experiments on Argoverse series with different settings demonstrate that our RealMotion achieves state-of-the-art performance, along with the advantage of efficient real-world inference.

1 Introduction

Motion forecasting is a crucial element in contemporary autonomous driving systems, enabling self-driving vehicles to predict the movement patterns of surrounding agents [43, 17]. This prediction is vital for ensuring the safety and reliability of driving. However, numerous complex factors, including stochastic road conditions and the diverse motion modes of traffic participants, make resolving this task challenging. Recent developments have focused on the study of representation and modeling [10, 52, 51], in tandem with a growing emphasis on precise trajectory predictions [6, 32, 49, 15, 50, 35]. Furthermore, the field has witnessed an increased focus on multi-agent forecasting, a more challenging yet valuable subtask [26, 1, 14, 31]. These advancements have collectively contributed to substantial progress in motion forecasting in recent years.

However, we realize that existing methods tackle motion forecasting tasks in an isolated manner, i.e., they treat every individual driving scene within a limited range independently, overlooking that in reality motion forecasting is inherently temporally interrelated while any ego-car drives on. That means previous methods ignore the driving context across successive scenes, as well as the corresponding potentially useful information from previous driving periods (Fig. 1).

t

t-1

t-2

(b) RealMotion

(a) Existing methods

t-3

invisible

history

current

visible

(c) Comparison of prediction

Independent and our predictions

Visible field

scene and stream features

Under the above insight and consideration, we propose an efficient in-context motion forecasting framework for continuous driving, named RealMotion . It comprises two streams to transit states of scenes: (1) A scene context stream that accumulates historical scene context progressively, capturing temporal interactions among scene elements and addressing complex driving situations. (2) An agent trajectory stream that continually optimizes the predictions for dynamic agents like vehicles, considering temporal consistency constraints and capturing precise motion intention. Each stream utilizes a specially designed cross-attention mechanism to transit scene states and fulfill its function.

Our contributions are summarized as follows: (i) We solve the motion forecasting problem from a perspective of the real-world applications, which enables the extraction and utilization of valuable situational and progressive knowledge. (ii) We introduce RealMotion, a novel motion forecasting approach that sequentially leverages both scene context and predicted agent motion status over time, meanwhile maintaining a lower real-world inference latency. (iii) To support the continuous driving setting on existing benchmarks, we implement a data reorganization strategy to generate scene sequences, closely simulating the real-world driving scenarios. Extensive experiments on Argoverse series with different settings demonstrate that RealMotion achieves state-of-the-art performance.

2 Related work

In autonomous driving, accurately predicting future trajectories of agents of interest relies on an appropriate representation of scene elements. Early methods [29, 12, 2] rasterize driving scenarios into images and utilize off-the-shelf convolutional networks for scene context encoding. However, due to their limited ability to capture intricate structural information, recent studies [49, 15, 52, 36] have shifted towards vectorized representations, as exemplified by the emergence of VectorNet [10]. Additionally, graph-based structures are widely adopted to model dynamics, interactions, and relationships among agents and maps [22, 13, 44, 19, 18, 9].

With the encoded scene features, various approaches are explored for estimating multi-modal future trajectories. Early methods focus on goal-based prediction [49, 15] or use probability distribution heatmaps for trajectory sampling [12, 13]. Recent approaches like HDGT [19], Wayformer [25], and others [23, 26, 48, 32, 51] leverage Transformer architectures [37] to model detailed relationships within the overall scene. Moreover, there are methods introducing novel paradigms (e.g. pretraining [5, 4, 21, 28], post-refinement [6, 50] or Diffusion [20]) to achieve impressive performance.

To address the relevance of predicted trajectories for different agents in real-life scenarios, recent efforts have focused on multi-agent forecasting. Some methods [14, 38, 52, 33] adopt an agentcentric design, iteratively predicting trajectories for each agent, which can be inefficient and hinder exploration of relationships among agents. Conversely, SceneTransformer [26] introduces a scenecentric framework for joint forecasting of all agents, presenting a novel design. Moreover, recent methods explore a query-centric design [51] or consider adaptations [1] to address this task.

Nevertheless, the methods mentioned above independently perform forecasting for each scene sample, which conflicts with practical settings where driving scenarios are interconnected over time. To overcome this limitation, pioneering work [27] first introduce a temporal motion benchmark based

(a) A given

independent

scen

e

(c)

sub-

s

cen

es

future and historical trajectory

scene elements

𝑻

i-1

𝑻

i

𝑻

(b) Chunking and aggregating

𝑻

i-1

𝑻

i

𝑻

on tracking dataset. Instead, we design a transformed data structure to mimic real-world conditions and propose RealMotion to effectively model the temporal relationships.

3 Methodology

3.1 Preliminary

Data reorganization. Considering the discrepancy between existing benchmarks and practical applications, our first step is to reorganize these datasets by transforming each sample scene into a continuous sequence, mimicking the successive real-world driving scenarios. Specifically, we retrospectively examine each independent scene by evenly splitting agent trajectories into shorter segments and sampling local map elements (refer to Fig. 2). Specifically, we first select several split points T i along historical frame steps. Next, we generate trajectory segments of identical length by extending from these points both into the past and the future. The number of historical and future steps is determined by the minimum split point and the length of ground-truth trajectory, respectively. Also, surrounding agents within a certain range and a local map are aggregated for interested agents at each split point, forming a sequence of sub-scenes. This reorganization allows freely capitalizing on the original elements to offer valuable situational and progressive insights at the scene level for model optimization. Hence, existing methods can also involve and benefit from the novel data structure. We have implemented this approach within the state-of-the-art method QCNet [51], which is further discussed in the appendix, highlighting the generality of our data structure.

Input representation. In the context of motion forecasting, the trajectories of agents and a highdefinition road map are provided in the driving scenario. Following [10], we transform these scene elements into vectorized representations as input. The historical trajectories are denoted as A ∈ R N a × T × C a , where N a , T , and C a represent the number of agents, the number of historical frames, and the motion states (e.g., position, angle, velocity, and acceleration), respectively. The road map is divided into several lane segments, denoted as M ∈ R N m × P × C m , where N m , P , and C m indicate the number of lane segments, the points of each segment, and the lane features (typically represented as coordinates). All these states are normalized relative to the current position of the agent of interest.

3.2 RealMotion for continuous forecasting

As depicted in Fig. 3, our RealMotion approach comprises an encoder, a decoder, a scene context stream, and an agent trajectory stream. Following the encoder-decoder structure, the two streams are designed to execute temporal modeling, focusing on context information and trajectory prediction along the time dimension.

Input agents

and HD map

∙∙∙

Output

trajectories

Context

referencing

Trajectory

relaying

Context

referencing

Trajectory

relaying

Context

referencing

Trajectory

relaying

∙∙∙

∙∙∙

∙∙∙

TE

Context Referencing

Decode

t-1

t

t+1

Trajectory Relaying

cross-attn

memory update

Output

(a)

(b)

(c)

(d)

TE

𝑭𝐦𝐨

𝒀𝐦𝐨

෩

𝑭𝐛

෩

𝒀𝐛

map-map

cross-attn

𝑭𝐬

𝒕

agent-context

cross-attn

෩

𝑭𝐬

𝒕-𝟏

align

𝑭𝐦

𝒕

𝑭𝐚

𝒕

Encoder

Decoder

Agent trajectory stream

Scene context stream

Agent and map features

3.2.1 Multimodal feature encoding and trajectory decoding

For scalability and simplicity, we adopt the plain encoder and decoder design following [5, 32]. Specifically, we encode the map features F m ∈ R N m × D by a PointNet-like encoder [30], where D refers to the embedding dimension. As [5], we extract the agent features F a ∈ R N a × D by stacked neighborhood attention blocks [16]. Given the agent and map features, we concatenate them to derive the scene features F s ∈ R ( N a + N m ) × D . We then employ a Transformer [37] encoder to learn the interrelationships of these features.

For trajectory prediction along with the probability, we utilize two Multilayer Perceptron (MLP) modules. Additionally, we forecast a singular trajectory for auxiliary training purposes, focusing on the movement patterns of other agents.

3.2.2 Scene context stream

The scene context stream is meticulously designed to progressively gather information about the surrounding environment, thereby enhancing trajectory prediction. Positioned after the encoder, this component plays a crucial role as the historical scene profoundly influences the understanding of temporal motion behaviors exhibited by agents. For instance, the ability to estimate the trajectory in complex scenes or evaluate currently occluded agents can be greatly improved by taking into account the previous situation and context.

At the current time t , we denote the current and historical scene features as F t s and F t -1 s , respectively, extracted from the respective local coordinate systems. The process begins by projecting F t -1 s onto the current system. To achieve this, the two features must be aligned, considering the gap of motion between them. Motion-aware Layer Normalization (MLN) [40] is employed for this purpose:

where ∆ θ denotes the heading angles, ∆ t refers to the timestamps, and (∆ x, ∆ y ) represents the difference between their positions. PE indicates the position encoding function. Subsequently, we repartition the scene features ˜ F t -1 s and F t s into the agent and map parts for following process. To incorporate the historical information into the current, we employ map-map and agent-scene cross-attention modules with Multi-Head Attention (MHA) blocks for map and agent features, respectively:

where map features only interact with each other across time to enrich map elements, while agent features aggregate the overall historical scene context for comprehensive scene understanding.

Given the sequential nature, this past context-referenced feature will be further propagated to future timestamps. Simultaneously, it serves as the input to the decoder for trajectory prediction at the current timestamp.

3.2.3 Agent trajectory stream

In addition to referencing scene context, we enhance trajectory forecasting by establishing temporal relationships to achieve further improvement. This involves leveraging the inherent temporal continuity and smoothing nature of trajectories. This is accomplished through the agent trajectory stream, which is equipped with a memory bank for storing historical trajectories, enabling temporal relaying.

To maintain a set of n historical trajectories for each agent of interest, we design the trajectory memory bank as M ( a ) = { ( y 1 , f 1 ) , ( y 2 , f 2 ) , ..., ( y n , f n ) } , where y i denotes predicted trajectories projected onto the global coordinate system, and f i represents corresponding mode features recording historical motion information. Unlike the detection or tracking tasks, where position changes of objects typically remain relatively consistent, the future motion patterns (under local system) involves significant changes due to variations in road conditions. Therefore, directly decoding with historical memory query features is inappropriate. Considering that the simple decoder can provide satisfactory predictions in practice, modifying initial predictions using the memory bank proves more effective.

Before refinement, we align the saved trajectories and mode features with the current coordinate system. While handling features consistently with the above, it is crucial to align trajectories in Euclidean space for more precise comparisons. Concerning the trajectory y i ∈ R K × 2 (with K being the frame steps of the trajectories), we compute the transformed trajectory ˜ y i as

Here, R ( θ ) is the rotation matrix for the current heading angle, and y ori i denotes the trajectory origin chosen based on the time difference ∆ t and sampling frequency q . Recognizing that memory trajectories may originate from different historical times, their origins also differ.

After transformation, trajectories whose latter part resembles the former part of current predictions contribute more to the refinement process. To aggregate memory information accordingly, we update the current mode features with a lightweight Transformer Decoder utilizing Trajectory Embedding (TE) to measure spatial similarity and replace the original positional embedding. Then, we further propagate updated modes into a MLP module to generate offsets and update initial predictions. This procedure is defined as follows:

where F mo and Y mo are mode features and initial predicted trajectories in the current context, ˜ F b and ˜ Y b represent aligned features and trajectories retrieved from the memory bank. Besides, we employ a single layer MLP to embed the flattened trajectories as the TE.

Ultimately, we save the refined trajectories (projected onto the global system) and features while removing outdated ones (first in first out). This updated trajectory memory will be further passed down for future timestamps. Importantly, due to the generation of only a small number of motion modes, we do not apply any filtering to ensure the multimodality and diversity of the bank.

3.3 Model training

During the training process, we supervise the estimated trajectories using the regression loss L reg and the associated probabilities through the classification loss L cls . Additionally, we introduce the

refinement loss L refine to guide the learning of predicted trajectory offsets within our agent trajectory stream. The overall loss L combines these individual losses with equal weights, formulated as follows:

For L reg and L refine , we employ the smooth-L1 loss, while the cross-entropy loss is utilized for L cls .

4 Experiments

4.1 Experimental settings

Datasets and metrics We assess the performance of our method using the Argoverse 1 [3] and Argoverse 2 [43] motion forecasting datasets in both single-agent and multi-agent settings. The Argoverse 1 dataset comprises 323,557 sequences from Miami and Pittsburgh, while the Argoverse 2 dataset contains 250,000 scenes spanning six cities. In the Argoverse 1 dataset, predictors are tasked with forecasting 3 seconds of future trajectories for agents based on 2 seconds of historical observations. In contrast, the Argoverse 2 dataset offers improved data diversity, higher data quality, a larger observation window of 5 seconds, and an extended prediction horizon of 6 seconds. Additionally, both datasets have a sampling frequency of 10 Hz.

We employ standard benchmark metrics, encompassing minimum Average Displacement Error ( minADE k ), minimum Final Displacement Error ( minFDE k ), Miss Rate ( MR k ), and brier minimum Final Displacement Error ( b -minFDE k ), across six prediction modes designed for the single-agent setting. Further details can be found in the appendix.

Implementation details We train our models using the AdamW [24] Optimizer with a batch size of 32 per GPU for 60 epochs. Our model is trained end-to-end with a learning rate of 0.001 and a weight decay of 0.01. The latent feature dimension is set to 128. Following [5, 32], we consider only agents and lane segments within a 150-meter radius of the focal agent. For the samples in Argoverse 2, we split the whole scene into 3 segments, each with a historical observation window of 3s and a same prediction horizon of 6s as the original. As for the Argoverse 1, we set the historical window of 1s. To fully utilize historical information, we compute gradients and perform back propagation for all segments. Moreover, the RealMotion-I is trained and evaluated with a common configuration without dataset reorganization and stream modules.

4.2 Comparison with state of the art

We first compare the performance of our RealMotion with several top-ranked models on the Argoverse 2 [43] motion forecasting benchmark. The results on the test split are presented in Tab. 1. RealMotion has far outperformed most of previous approaches. Concretely, our method stands distinctly ahead of other methods in terms of minFDE 1 and minADE 1 , showing performance enhancements of 8.60% and 5.91% relative to QCNet, respectively. We also get the almost top 2 place for other metrics. Compared to our independent variant RealMotion-I, the proposed method exhibits significant performance improvements across all metrics, which conclusively demonstrates the effectiveness of our designs. Then, we compare the performance of RealMotion on the Argoverse 1 benchmark, with the results of the validation split presented in Tab. 2. It is shown that our method also achieves a decent performance, especially for minADE 6 . We also provide our ensemble and multi-agent results in Appendix B.

4.3 Multi-agent quantitative results

In the multi-agent setting, predictors are required to jointly estimate the future trajectories of all interested agents, which is crucial for the comprehensive perception of the driving scenario. Therefore, we also evaluate our RealMotion on the Argoverse 2 Multi-agent test set to prove the effectiveness, and provide simple results as shown in Tab. 3. Although not integrated with specialized designs for multi-agent forecasting like [26, 1], our model also demonstrates superior performance compared to recent works owing to our sequential techniques.

4.4 Ablation study

We conduct ablation studies on the Argoverse 2 validation split for the single-agent setting to examine the effectiveness of each component in RealMotion. We adopt the default experiment settings following Sec. 4.1 to perform ablation in this section.

Effects of components. As shown in Tab. 4, we assess the effectiveness of each component in our network. The first row (ID-1) represents the independent framework RealMotion-I, which is the same as reported in Tab. 1. First, our data reorganization approach extends dataset at no extra cost and enables the exhaustive utilization of temporally continuous motion, resulting in a noticeable improvement as shown in the second row. Then, our proposed two streams can better learn temporal

relationships at the scene level, thereby both bringing additional improvements, represented as ID-3 and ID-4, respectively. Considering that these two streams are complementary to each other, therefore, RealMotion that involves all these sequential techniques achieves remarkable performance gains, as indicated in the final row. Additionally, in contrast to the consistent improvements observed with the data processing approach, it is worth noting that our two streams demonstrate more significant advantages in single-mode metrics compared to the six-mode metrics. As shown in Fig. 4, we attribute this phenomenon to the enhanced capability of our streams to modify less accurate trajectories.

Effects of feature alignment and TE. The misalignment of features might have an adverse effect on the performance when implementing feature interaction across scenes. Hence, we evaluate the impact of the alignment modules in both the Context Referencing and the Trajectory Relaying blocks in Tab. 5(a). As depicted in the first four rows, the removal of the alignment modules clearly results in a performance decline. By incorporating these modules, We have, to some extent, alleviated the negative impact of misalignment. However, executing it twice only yields marginal gains, which might be caused by the function overlap. Besides, we also analyze the performance of our proposed Trajectory Embedding in the 4th and 5th rows. The performance gains indicate that the Trajectory Embedding can facilitate the selection of similar historical trajectories from the memory bank, thereby easily constraining and refining current predictions. Despite the simplicity in the design of these modules, they both contribute to additional benefits for our network.

Effects of split points and gradient steps. In Tab. 5(b), we investigate the performance variations with respect to the number of gradient steps taken and the split points along historical steps. From the 1st to the 3rd row, as we progressively increase the number of steps used for gradient computation, there is a noticeable improvement in performance. This enhancement can be attributed to the fact that computing the gradient for specific steps allows us to better capture the trajectory distribution for model training. From the 3rd to the 5th row, we change the interval of split points from 5 frames to 15 frames. Accordingly, the length of historical trajectory also changes, ranging from 40 frames to 20 frames. As observed, similar trajectories occur in the sequence when using a short interval, which can significantly hinder the model to learn distinct motion patterns. Conversely, using a longer interval results in fewer historical trajectory frames available in each segment, which contradicts the optimization of one-shot forecasting. It is evident that our choices for the gradient steps and the split points are well-suited for the Argoverse 2 benchmark. Moreover, we attempt to divide the trajectory into 5 segments in the final row. An excessively long trajectory sequence imposes a

(a)

Step1

(T=30)

(b)

(T=40)

(c)

Step3

(T=50)

(d)

RealMotion

-

I

(T=50)

①

②

Predicted Trajectory

GT Trajectory

Agent of Interest

Surrounding Agent

③

learning burden on our framework, preventing it from focusing on temporal relationships and yielding limited improvements.

Effects of the depth of cross-attention blocks. As shown in Tab. 7, we explore the influence of depth variations of cross-attention unit in the Context Referencing and the Trajectory Relaying blocks. Primarily our temporal blocks are lightweight regardless of depth to ensure the efficiency and universality. a relatively deep cross-attention unit in the Context Referencing and the Trajectory Relaying blocks is necessary for processing history information and current information. We use a depth of 2 as our default setting considering its better efficiency-performance balance.

4.5 Efficiency analysis

Balancing performance, inference speed, and model size is important for the model deployment. We compare our RealMotion with recent representative works, which include the real-time forecasting approach HPTR [48] and the state-of-the-art approach QCNet [51]. We measure these approaches

and RealMotion on the Argoverse 2 test set using an NVIDIA GeForce RTX 3090 GPU, maintaining a batch size of 1 and following an end-to-end manner. As shown in Tab. 6, RealMotion has the competitive inference time and a competitive model size while achieving the best performance. It is worth noting that 'online' indicates the latency for practical autonomous driving systems, which is optimized compared to the 'offline' version by some efficient designs (e.g. the caching technique in [48]). As for RealMotion, we must consider three times latency for 'offline' dataset, where only the final prediction is utilized for evaluation. In contrast, the forecasting results of each iteration is meaningful in 'online' application.

4.6 Qualitative results

In Fig. 4, we present qualitative results of our network compared to the independent version on the Argoverse 2 validation set. Panels (a), (b), and (c) illustrate the forecasting results at 3s, 4s and 5s, respectively. Panel (c) displays the final results used for evaluation, which are more accurate and closer to the ground truth. By comparing panel (c) and (d), it can be seen that RealMotion remarkably outperforms the independent version. As demonstrated in the panels from (a) to (c), our RealMotion can progressively refine the estimated trajectories from coarse to fine as the motion progresses and accurately capture the possible motion intention. However, the one-shot forecasting of RealMotion-I shown in the panel (d) leads to significant estimation errors.

5 Conclusion

In this work, we anticipate to address the motion forecasting task from a more practical continuous driving perspective. This in essence places the motion forecasting function in a wider scene context compared to the previous setting. We further present RealMotion, a generic framework designed particularly for supporting the successive forecasting actions over space and time. The critical components of our framework are the scene context stream and the agent trajectory stream, both of which function in a sequential manner and progressively capture the temporal relationships. Our extensive experiments under several settings comprehensively demonstrate that RealMotion surpasses the current state-of-the-art performance, thereby offering a promising direction for safe and reliable motion forecasting in the rapidly evolving field of autonomous driving.

Limitations. A clear constraint of our data processing approach is its requirement for a sufficient number of historical frames for serialization. Consequently, it is not applicable to short-term benchmarks such as the Waymo Open Dataset, which provides only 10 frames of historical trajectory. Moreover, existing datasets typically provide limited historical information distinct from real-world settings, which inhibits our sequential designs from fully leveraging their advantages. Hence, we anticipate to integrate our framework into a sequential autonomous driving system to maximize the benefits of streaming designs in our future work.

Acknowledgments

This work was supported in part by National Natural Science Foundation of China (Grant No. 62106050 and 62376060), Natural Science Foundation of Shanghai (Grant No. 22ZR1407500).

References

G. Aydemir, A. K. Akan, and F. Güney. Adapt: Efficient multi-agent trajectory prediction with adaptation. In ICCV , 2023. 1, 2, 7

Y. Chai, B. Sapp, M. Bansal, and D. Anguelov. Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction. In CoRL , 2020. 2

M.-F. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In CVPR , 2019. 6

H. Chen, J. Wang, K. Shao, F. Liu, J. Hao, C. Guan, G. Chen, and P.-A. Heng. Traj-mae: Masked autoencoders for trajectory prediction. In ICCV , 2023. 2

J. Cheng, X. Mei, and M. Liu. Forecast-mae: Self-supervised pre-training for motion forecasting with masked autoencoders. In ICCV , 2023. 2, 4, 6, 7

S. Choi, J. Kim, J. Yun, and J. W. Choi. R-pred: Two-stage motion prediction via tube-query attention-based trajectory refinement. In ICCV , 2023. 1, 2, 7

A. Cui, S. Casas, K. Wong, S. Suo, and R. Urtasun. Gorela: Go relative for viewpoint-invariant motion forecasting. In ICRA , 2023. 6

F. Da and Y. Zhang. Path-aware graph attention for hd maps in motion prediction. In ICRA , 2022. 7

N. Deo, E. Wolff, and O. Beijbom. Multimodal trajectory prediction conditioned on lane-graph traversals. In CoRL , 2022. 2

J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In CVPR , 2020. 1, 2, 3

X. Gao, X. Jia, Y. Li, and H. Xiong. Dynamic scenario representation learning for motion forecasting with heterogeneous graph convolutional recurrent networks. IEEE RA-L , 2023. 6, 7

T. Gilles, S. Sabatini, D. Tsishkou, B. Stanciulescu, and F. Moutarde. Home: Heatmap output for future motion estimation. In IEEE ITSC , 2021. 2

T. Gilles, S. Sabatini, D. Tsishkou, B. Stanciulescu, and F. Moutarde. Gohome: Graph-oriented heatmap output for future motion estimation. In ICRA , 2022. 2

T. Gilles, S. Sabatini, D. Tsishkou, B. Stanciulescu, and F. Moutarde. Thomas: Trajectory heatmap output with learned multi-agent sampling. In ICLR , 2022. 1, 2, 6

J. Gu, C. Sun, and H. Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In CVPR , 2021. 1, 2, 7

A. Hassani, S. Walton, J. Li, S. Li, and H. Shi. Neighborhood attention transformer. In CVPR , 2023. 4

Y. Huang, J. Du, Z. Yang, Z. Zhou, L. Zhang, and H. Chen. A survey on trajectory-prediction methods for autonomous driving. IV , 2022. 1

X. Jia, L. Sun, H. Zhao, M. Tomizuka, and W. Zhan. Multi-agent trajectory prediction by combining egocentric and allocentric views. In CoRL , 2022. 2

X. Jia, P. Wu, L. Chen, Y. Liu, H. Li, and J. Yan. Hdgt: Heterogeneous driving graph transformer for multi-agent trajectory prediction via scene encoding. IEEE TPAMI , 2023. 2, 6

C. Jiang, A. Cornman, C. Park, B. Sapp, Y. Zhou, D. Anguelov, et al. Motiondiffuser: Controllable multi-agent motion prediction using diffusion. In CVPR , 2023. 2

Z. Lan, Y. Jiang, Y. Mu, C. Chen, and S. E. Li. Sept: Towards efficient scene representation learning for motion prediction. In ICLR , 2024. 2

M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun. Learning lane graph representations for motion forecasting. In ECCV , 2020. 2, 7

Y. Liu, J. Zhang, L. Fang, Q. Jiang, and B. Zhou. Multimodal motion prediction with stacked transformers. In CVPR , 2021. 2, 7

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR , 2018. 6

N. Nayakanti, R. Al-Rfou, A. Zhou, K. Goel, K. S. Refaat, and B. Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. In ICRA , 2023. 2

J. Ngiam, V. Vasudevan, B. Caine, Z. Zhang, H.-T. L. Chiang, J. Ling, R. Roelofs, A. Bewley, C. Liu, A. Venugopal, et al. Scene transformer: A unified architecture for predicting future trajectories of multiple agents. In ICLR , 2021. 1, 2, 7

Z. Pang, D. Ramanan, M. Li, and Y.-X. Wang. Streaming motion forecasting for autonomous driving. In IROS , 2023. 2

D. Park, J. Jeong, S.-H. Yoon, J. Jeong, and K.-J. Yoon. T4p: Test-time training of trajectory prediction via masked autoencoder and actor-specific token memory. In CVPR , 2024. 2

T. Phan-Minh, E. C. Grigore, F. A. Boulton, O. Beijbom, and E. M. Wolff. Covernet: Multimodal behavior prediction using trajectory sets. In CVPR , 2020. 2

C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR , 2017. 4

L. Rowe, M. Ethier, E.-H. Dykhne, and K. Czarnecki. Fjmp: Factorized joint multi-agent motion prediction over learned directed acyclic interaction graphs. In CVPR , 2023. 1, 7

S. Shi, L. Jiang, D. Dai, and B. Schiele. Motion transformer with global intention localization and local movement refinement. In NeurIPS , 2022. 1, 2, 4, 6

S. Shi, L. Jiang, D. Dai, and B. Schiele. Mtr++: Multi-agent motion prediction with symmetric scene modeling and guided intention querying. IEEE TPAMI , 2024. 2

T. Su, X. Wang, and X. Yang. Qml for argoverse 2 motion forecasting challenge. arXiv preprint , 2022. 6

X. Tang, M. Kan, S. Shan, Z. Ji, J. Bai, and X. Chen. Hpnet: Dynamic trajectory forecasting with historical prediction attention. In CVPR , 2024. 1, 7

B. Varadarajan, A. Hefny, A. Srivastava, K. S. Refaat, N. Nayakanti, A. Cornman, K. Chen, B. Douillard, C. P. Lam, D. Anguelov, et al. Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction. In ICRA , 2022. 2

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In NeurIPS , 2017. 2, 4

J. Wang, T. Ye, Z. Gu, and J. Chen. Ltp: Lane-based trajectory prediction for autonomous driving. In CVPR , 2022. 2

M. Wang, X. Zhu, C. Yu, W. Li, Y. Ma, R. Jin, X. Ren, D. Ren, M. Wang, and W. Yang. Ganet: Goal area network for motion forecasting. In ICRA , 2023. 6

S. Wang, Y. Liu, T. Wang, Y. Li, and X. Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. In ICCV , 2023. 4

X. Wang, T. Su, F. Da, and X. Yang. Prophnet: Efficient agent-centric motion forecasting with anchorinformed proposals. In CVPR , 2023. 6

Y. Wang, H. Zhou, Z. Zhang, C. Feng, H. Lin, C. Gao, Y. Tang, Z. Zhao, S. Zhang, J. Guo, et al. Tenet: Transformer encoding network for effective temporal flow on motion prediction. arXiv preprint , 2022. 6

B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, D. Ramanan, P. Carr, and J. Hays. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In NeurIPS , 2021. 1, 6, 7

W. Zeng, M. Liang, R. Liao, and R. Urtasun. Lanercnn: Distributed representations for graph-centric motion forecasting. In IROS , 2021. 2, 7

C. Zhang, H. Sun, C. Chen, and Y. Guo. Banet: Motion forecasting with boundary aware network. arXiv preprint , 2022. 6

L. Zhang, P. Li, J. Chen, and S. Shen. Trajectory prediction with graph-based dual-scale context fusion. In IROS , 2022. 7

L. Zhang, P. Li, S. Liu, and S. Shen. Simpl: A simple and efficient multi-agent motion prediction baseline for autonomous driving. IEEE RA-L , 2024. 6

Z. Zhang, A. Liniger, C. Sakaridis, F. Yu, and L. Van Gool. Real-time motion prediction via heterogeneous polyline transformer with relative pose encoding. In NeurIPS , 2023. 2, 6, 9, 10

H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp, B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid, et al. Tnt: Target-driven trajectory prediction. In CoRL , 2021. 1, 2

Y. Zhou, H. Shao, L. Wang, S. L. Waslander, H. Li, and Y. Liu. Smartrefine: An scenario-adaptive refinement framework for efficient motion prediction. In CVPR , 2024. 1, 2

Z. Zhou, J. Wang, Y.-H. Li, and Y.-K. Huang. Query-centric trajectory prediction. In CVPR , 2023. 1, 2, 3, 6, 9, 13

Z. Zhou, L. Ye, J. Wang, K. Wu, and K. Lu. Hivt: Hierarchical vector transformer for multi-agent motion prediction. In CVPR , 2022. 1, 2, 7

Appendix

A More experimental settings

A.1 Evaluation metrics

For single-agent evaluation, we employ standard metrics for evaluation, including minimum Average Displacement Error ( minADE k ), minimum Final Displacement Error ( minFDE k ), Miss Rate ( MR k ), and brier minimum Final Displacement Error ( b -minFDE k ). minADE K calculates the L 2 distance between the ground-truth trajectory of the best K predicted trajectories, averaged across all future time steps. While minFDE k measures the difference between the predicted endpoints and the ground truth. MR k is the ratio of scenes where minFDE k exceeds 2 meters. To further assess uncertainty estimation performance, the metric b -minFDE k adds (1 -π ) 2 to the final-step error, where π denotes the best-predicted trajectory's probability score that the model assigns. As a common practice, K is selected as 1 and 6.

For multi-agent evaluation, We use standard metrics including Average Minimum Final Displacement Error ( avgMinFDE ), Average Minimum Average Displacement Error ( avgMinADE ) and Actor Miss Rate ( actorMR ). avgMinFDE is the average of the lowest FDEs for all scored actors within a scene, reflecting the prediction accuracy of a scene outcome. avgMinADE represents the average of the lowest ADEs for all scored actors within a scene, indicating the general accuracy of the predicted movements. Across the evaluation set, the actorMR is the proportion of missed actor (same as above).

B More experiments

B.1 Model ensemble

Model ensemble, a crucial technique for enhancing the accuracy of final predictions, is employed in our approach. We utilize six sub-models trained with various random seeds and split points. Consequently, we generate 36 predicted future trajectories for each agent, and then apply k-means clustering to process them with 6 cluster centers. For each cluster group, we calculate the average of all trajectories within the group to produce the final trajectories. The results with and without model ensemble trick on the Argoverse 2 test set are shown in Tab. 8.

Table 8: Performance comparison between results with and without ensemble on Argoverse 2 test set in the official leaderboard. For each metric, the better result is in bold .

B.2 Generality evaluation with integrating RealMotion

For generality test, we integrate our RealMotion data with the state-of-the-art method QCNet [51]. Due to the big size of QCNet, the scene context and agent trajectory streams have to be excluded for memory constraint. With minimal alterations applied to this prior model, we observe a noticeable improvement. The results are shown in Tab. 9.

C More qualitative results

We provide more qualitative results of our framework in Fig. 6 and Fig. 7 on the Argoverse 2 validation set.

D Failure cases

Although our RealMotion has achieved outstanding performance on motion forecasting benchmarks, it still has some failure cases. We analyze these typical cases and provide qualitative results to help readers understand the circumstances under which our model may fail. This analysis will aid future work in developing a more powerful and robust algorithm, as shown in Fig. 5.

D.1 Case 1: Complex map topology

In the first row of Fig. 5, the agent requires to navigate through a complex intersection to one of the roads, but the model fails to predict this possible driving behavior and just anticipates the agent to drive straight ahead. This may be caused by the lack of a comprehensive understanding of the complex map topology and the unbalanced distribution of driving data. In most scenarios, agents tend to exhibit only trivial behaviors, such as moving straight ahead at a nearly constant velocity. Consequently, this raises issues regarding data balance, and the figures indicate that our model is more likely to make mistakes when it comes to turning.

D.2 Case 2: Subjective driving behavior

In the second row of Fig. 5, the vehicle is expected to park on the side of the road, which is a kind of subjective driving behaviors. However, the model only predicts that the vehicle will keep going ahead. To improve the forecasting of such cases, we could enhance the interaction between the model with additional intentions of human and incorporate more information, such as visual cues like turn signals and parking spaces.

(a) Step1 (T=30)

(c) Step3 (T=50)

(d)

RealMotion

-

I

(T=50)

②

①

Predicted Trajectory

GT Trajectory

Agent of Interest

Surrounding Agent

①

②

③

④

⑤

Predicted Trajectory

GT Trajectory

Agent of Interest

Surrounding Agent

(a) Step1 (T=30)

(c) Step3 (T=50)

(d)

RealMotion

-

I

(T=50)

①

(a) Step1 (T=30)

(c) Step3 (T=50)

(d)

RealMotion

-

I

(T=50)

②

③

④

⑤

Predicted Trajectory

GT Trajectory

Agent of Interest

Surrounding Agent

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Experiments in Sec. 4

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: see Conclusion in 5

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Experiments in Sec. 4 and Appendix

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please see experimental settings in Sec. 4.1

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No] .

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes] .

Justification: See Experiments in Sec. 4

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: It is our main idea designed for real-world autonomous driving applications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes] .

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a model for motion forecasting that models the continuous stream of world state over sequential timesteps.  This is in contrast to most/all other works which do an indepedent prediction of the future for each timestep based on a fixed window of history, without context of previous model belief states or outputs from previous time steps.  One stream of data is scene context, represented as embeddings, where context is fused with current accounting for transformation of the scene due to ego motion.  The second stream is agent trajectories, where a history of n previous predicted trajectories for each agent is maintained explicitly in a memory bank, and fused into the current time steps prediction as additional signal.\n\nThey do thorough ablations on Argoverse benchmark dataset, showing compelling evidence that the streaming architecture is beneficial.  Furthermore, they exceed SOTA on most metrics in the benchmark. When I checked today (mid July), they are ranked 7th overall in the leaderboard - https://eval.ai/web/challenges/challenge-page/1719/leaderboard/4098\n\nSTRENGTHS:\nThis paper addresses an important problem often ignored in these sanitized datasets: that these models in real world  application in an AV system run in a sequential streaming fashion, and (to my knowledge) all other works operate on each time frame independently (given a fixed window of historical state).  \n\nThe proposed solution itself is a good one, transforming old belief state into current reference frames in both embedding space and with spatial transformations where appropriate.  At the same time, the complexity of the architecture is kept reasonably low, with each contribution justified. \n\nResults are very strong and well justified with thorough ablations.\n\nWEAKNESSES:\nMinor, but I had some difficulty understanding some of the explanations:\n\n - Data reorganization / Figure 2: \"we select several split points Ti along historical frame steps\" --> how are they selected? what are \"historical frame steps\"? Can you more simply say this as, for example, \"We split sequences evenly into sub-sequences\"?\n\n - Around L158: you started using the term \"modes\" for the first time, where I think you should just say trajectories, or be more careful defining the tensors F_{mo} and Y_{mo} making clear their dimensionality.  \n\n - You never clearly state the output representation of your model (which I assumed to be a weighted trajectory set). Without being more clear here, Section 3.3. Model training is sort of meaningless - for example, what is the classification loss classifying? (I'm 99% sure it's a softmax distribution over trajectory modes, but this is not said anywhere.)\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission tackles the task of trajectory forecasting in autonomous driving. \nIn particular, it proposes improvements in two aspects:\n\n1/ Data reorganization: Current datasets are artificially split into non-overlapping segments. The authors propose to reorganize the data to have overlapping windows.\n\n2/ Relaying context and predictions: since the sub-scenes now overlap, they can use context and predictions from the previous timesteps (corresponding to a different sub-scene) to improve predictions at the current timestep.\nRelaying context serves to provide more information about the surroundings (an agent could have been missed in the current frame for instance), and relaying the predictions would smooth out the predictions across time.\n\nThey demonstrate the validity of their approach on Argoverse benchmarks, achieving state-of-the-art performances.\n\nSTRENGTHS:\n- The proposed improvements are interesting, sound, and motivated.\n- The data reorganization, I believe, is a simple idea that would that should be pushed in the motion forecasting community to build a new standard that better leverage the data. It can also immediately be applied to other models and hopefully improve performances.\n- The relaying through time helps achieve strong results either with their architecture or on top of QCNet.\n\nWEAKNESSES:\n- The results are not very conclusive:\n  - Compared to  QCNet, except for $ADE_1$/$FDE_1$ (which are not the main metrics usually considered for motion forecasting) the improvements are small. It would be useful to give more insight into why only these metrics improve. We also don't have the $ADE_1$/$FDE_1$ values of QCNet + RealMotion so we don't know if these findings are consistent across models.\n  - The comparisons are not fair in terms of available information. Because of the propagation, RealMotion has access to more past information both in the past trajectories and agents, and the past map. A naïve way to integrate more past trajectories would be to just use longer past history. This should be a minimal baseline or ablation in the paper. If we also wanted to add agents seen in the past but not at the current timestep, it could be done with some masking or extrapolation scheme for missing frames. \nRemains the past map information, for which it's indeed more complicated to have a naïve baseline for a fair comparison, as simple aggregation could drastically raise the computation footprint so I can understand why the authors wouldn't want to test this baseline.\n- The loss functions, and in particular $\\mathcal{L}_{refine}$ are not detailed enough. I can guess where they apply but it would be good if that was stated explicitly.\n- Figure 2 is unclear to me. Perhaps it could need more captions or a rework.\n- I think it would be easier to interpret the results if the architecture of RealMotion would be more clearly delivered, perhaps in a Figure  (in Appendix?) that represent the full architecture detailing in particular how the different elements interact in the encoder and the decoder that are not given in Figure 3.\n\nQUESTIONS:\n- Why do we see drastic improvement on ADE/FDE_1 but the other improvements are very tame? Does that imply most of the improvements are to the scoring function so that RealMotion gets a better top 1? If that's the case, what could be the mechanisms that explain this behavior?\n\nOtherwise, I think this paper rightfully questions some practices of motion forecasting in autonomous driving. Generalizing something along the lines of the proposed data reorganization would be very helpful for the community. However, I think the experiments on relaying might not be fair enough and need to be compared to simple baselines with more/better past history. We would also need a few more numbers to see if most of the findings and recommendations have a good chance at being consistent accross models.\n\nLIMITATIONS:\n- Limitations are discussed in the Appendix.\n\n- Broader societal impacts do not seem explicitly addressed despite what has been reported in the checklist. I'm not sure there are obvious uncontroversially negative societal impacts to be discussed though.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework named \"RealMotion\" highlighting the importance of continuous motion forecasting in autonomous driving. From the formulation perspective, RealMotion investigates predicting trajectories in a continuous sequence of timestamps instead of previous independent predictions. From the methodology perspective, RealMotion proposes the \"relaying\" modules that propagate historical scene and agent contexts into the current frame to enhance prediction. Finally, RealMotion achieves top performance on Argoverse benchmarks and proves the importance of continuous contexts.\n\nSTRENGTHS:\n1. This paper investigates a foundational challenge in autonomous driving that is overlooked by previous people -- how to model the continuous contexts in motion forecasting. This problem is reasonable and is also something I would love to follow and work on.\n\n2. The methods proposed by RealMotion are intuitive and reasonable. Relaying the historical context to current frames to enhance motion forecasting is indeed a novel contribution to autonomous driving.\n\n3. The experiments prove the effectiveness of continuous contexts. In addition to the state-of-the-art performance, the comparison between RealMotion-I and RealMotion in Table 1 makes it a strong argument.\n\nWEAKNESSES:\n1. The authors have missed some closely related works, such as streaming motion forecasting (e.g., Pang et al.) and other end-to-end autonomous driving (e.g., Gu et al.), where motion forecasting is continuous. For instance, Pang et al. also discover the setbacks of predicting trajectories on independent timestamps, formulate the \"streaming\" forecasting task, and propose a differentiable filter to refine the predicted trajectories. I acknowledge the newly designed modules by RealMotion, but I would expect the authors to have a better discussion of these relevant efforts.\n\nPang et al. Streaming Motion Forecasting for Autonomous Driving. IROS 2023.\nGu et al. ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries. CVPR 2023.\n\n2. RealMotion is also closely related to QCNet and based on its code. QCNet already has the interface of continuous motion forecasting and leverages a GRU to deal with the temporal contexts in queries. First of all, I would like the authors to clarify or discuss in the paper about the difference/improvement to QCNet more explicitly. In addition, I am also curious if it is possible to support that the \"relaying\" modules in RealMotion is better than QCNet's end-to-end mechanism. For instance, is it possible to plug the relaying module to QCNet and make a comparison?\n\n3. I would like the authors to develop some metrics to analyze the improvement in the trajectories. More specifically, we wish to analyze \"where does RealMotion improve the trajectory quality?\" Since the main argument of this paper is \"continuity,\" I am curious if the author can use a \"fluctuation\" measure like Pang et al. to prove that the trajectory smoothness is indeed improved. It is also encouraged to show some visual comparisons if space is allowed.\n\nQUESTIONS:\nSee weakness above.\n\nLIMITATIONS:\nThe authors have discussed limitations. No additional limitations from my side.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an approach to iteratively process temporal scene context for the purposes of agent motion prediction. This differs from traditional approaches that ingest the whole context directly. \n\nThe scene is processed in temporal chunks, where in each chunk contains agent-centric context consisting of other agents and map within 150m. Information is aggregated across chunks (i.e. over time) using two novel components: \n- \"Scene context stream\", where agent/map features from the past frame are transformed using Motion-aware Layer Normalization [40] and cross-attended by current frame features. \n- \"Agent context stream\", where agent predictions from the current frame are modified by attending predictions from past frames (stored in a memory bank and properly aligned). \n\nThis 'streaming-type' method obtains very strong results on the Argoverse 1 and 2 datasets, and is shown to improve two different baseline  model architectures based on [5] and [51] that consume the whole historical context in a single step. Some helpful ablations and latency studies of the proposed components is provided.\n\nSTRENGTHS:\n- Strong idea to stream scene context into the model, shown to be effective even with short 2-3 'chunks' of context over 2-3 seconds. \n- Intuitive and well designed scene and agent context stream components that aggregate information across frames (or 'chunks'). The components are designed to model local information in agent-centric coordinates and aggregate it in a geometry-aware manner. \n- Strong overall results on Argoverse 1 and 2, and when comparing the baseline models the new technique is extending.  \n- Mostly comprehensive related work section.\n\nWEAKNESSES:\n- In the abstract/intro, I find some of the framing to be a bit misleading. Examples: \n  - \"7:This significantly simplifies the forecasting task, making the solutions unrealistic to use in practice\". I do not find this claim to be well supported. The current methods still ingest a significant amount of relevant context. To me the contribution of the paper is to follow a streaming paradigm and design more domain-appropriate fusion mechanisms across time than the vanilla MLP or transformer ops that are used traditionally. \n  - \"31 - 36: existing methods all tackle an artificially narrowed and unrealistic motion forecasting task\".... The methods still ingest 2-3 seconds of context, but perhaps not as effectively as the proposed work. That does not confirm this overly strong claim however. Also the paper only demonstrates good performance with up to 3s of context, that does not prove yet that much longer context is helpful or practical. \n  - To a similar point, Fig 1 is also a bit misleading: current methods would directly ingest the available scene history in the current datasets, and fuse the information, even if the information fusion is not done with the same geometric and streaming intuitions. \n\n- In the method description, some details are unclear or could be explained better:\n  - 83: \"generate trajectory segments of identical length\" Length can be misconstrued to be the length of the trajectories in meters, not that they are the same across chunks. This whole paragraph could be rewritten for clarity and a more helpful diagram could be shown -- illustrating what exactly is done for Argoverse 2 (one now needs to read all the way to Sec 4 to understand how this works). \n  - \"114: Additionally, we forecast a singular trajectory for auxiliary training purposes, focusing on the movement patterns of other agents. \" \nUnclear what this means exactly. Do you produce only a single trajectory prediction in past steps?\n  - \"“Equal length into past and future” → confusing, assume fixed time interval in both directions. \n  - The L_refine loss in Eq 5 is not explained, unclear what is being refined. My guess is that this is predictions in earlier chunks\n  - What is exactly RealMotion without the context stream components? This is present both in the Ablation and in Appendix B.3. How do you fuse information across time then? \n\n- In the experimental results, some SOTA methods from the leaderboard seem to omit SEPT (ICLR 2023), which has higher numbers than RealMotion, even though it is cited. Why did you you omit it? \n\n- A couple more ablations or experiments would be helpful. \n  - It seems that the method is similar to a filter, where the current chunk only attends to previous chunk (but not to the ones before). Is my reading of the paper correct? If so, an experiment where the method can attend longer context memory with multiple frames would be interesting. \n  - It would also be interesting to try pretraining this method on more data examples (those can be produced by predicting shorter futures, then you can use more of the segments). And then fine-tune on 6s futures.\n\nQUESTIONS:\nMain question is, do you agree with my reading that \"the contribution of the paper is to follow a streaming paradigm and design more domain-appropriate fusion mechanisms across time than the vanilla MLP or transformer ops that are used traditionally. \" As opposed to the strong claims in the current abstract / intro. \n\nWhy did you omit SEPT from the leaderboard results (Table 1)? \n\nPlease see my clarification or ablation/more experiment comments in the Weaknesses. I am particularly interested to understand why you did not try to attend to longer temporal context in the last chunk (if I read the paper correctly).\n\nLIMITATIONS:\nThe same RealMotion methods tried here could be tried on the Waymo Open Dataset which also has 9 second segments, but one could take the first 3 seconds as history and predict 6 seconds, as opposed to the standard 1s / 8s split that is mentioned in the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}