Motion Forecasting in Continuous Driving
Authors: Nan Song
Bozhou Zhang
Xiatian Zhu
Li Zhang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Motion forecasting; Autonomous Driving;
Primary area: robotics

Abstract: Motion forecasting for agents in autonomous driving is highly challenging due to the numerous possibilities for each agent's next action and their complex interactions in space and time. 
In real applications, motion forecasting takes place repeatedly and continuously as the self-driving car moves. However, existing forecasting methods typically process each driving scene within a certain range independently, totally ignoring the situational and contextual relationships between successive driving scenes. This significantly simplifies the forecasting task, making the solutions suboptimal and inefficient to use in practice. To address this fundamental limitation, we propose a novel motion forecasting framework for continuous driving, named RealMotion.
It comprises two integral streams both at the scene level:
(1) The scene context stream progressively accumulates historical scene information until the present moment, capturing temporal interactive relationships among scene elements.
(2) The agent trajectory stream optimizes current forecasting by sequentially relaying past predictions.
Besides, a data reorganization strategy is introduced to narrow the gap between existing benchmarks and real-world applications, consistent with our network. These approaches enable exploiting more broadly the situational and progressive insights of dynamic motion across space and time. 
Extensive experiments on Argoverse series with different settings demonstrate that our RealMotion achieves state-of-the-art performance, along with the advantage of efficient real-world inference.

Motion Forecasting in Continuous Driving
Nan Song'
Bozhou Zhang'
Xiatian Zhu?
Li Zhang'
School of Data Science Fudan University
University of Surrey
https://githe com/fudate http://udannngg///aaee 2vg/RealMotion
Abstract
Motion forecasting for agents in autonomous driving is highly challenging due to the numerous possibilities for each agent's next action and their complex in- teractions in space and time. real applications, motion forecasting takes place repeatedly and continuously as the self- self-driving car moves. However, existing forecasting methods typically process each driving scene within a certain range independently, totally ignoring the situational and contextual relationships between successive driving scenes. This significantly simplifies the forecasting task, mak- ing the solutions suboptimal and inefficient to use in practice. To address this fundamental limitation, we propose a novel motion forecasting framework for continuous driving, named RealMotion. It comprises two integral streams both at the scene level: (1) The scene context stream progressively accumulates histor- ical scene information until the present moment, capturing temporal interactive relationships among scene elements. (2) The agent trajectory stream optimizes current forecasting by sequentially relaying past predictions. Besides, data reor- ganization strategy is introduced to narrow the gap between existing benchmarks and real-world applications, consistent with network. These approaches enable exploiting more broadly the situational and progressive insights of dynamic motion across space and time. Extensive experiments on Argoverse series with different settings demonstrate that our RealMotion achieves state-of-the-ar performance, along with the advantage of efficient real-world inference.
Introduction
Motion forecasting is a crucial element in contemporary autonomous driving systems, enabling self-driving vehicles to predict the movement patterns surrounding agents [43, 17]. This prediction is vital for ensuring the safety and reliability of driving. However, numerous complex factors, including stochastic road conditions and the diverse motion modes of traffic participants, make resolving this task challenging. Recent developments have focused on the study of representation and modeling [10, 52, 51], tandem with growing emphasis on precise trajectory predictions [6, 32, 49, 15, 50, 35]. Furthermore, the field has witnessed increased focus multi-agent forecasting, a more challenging yet valuable subtask [26, 1, 14, 31]. These advancements have collectively contributed to substantial progress in motion forecasting in recent years.
However, we realize that existing methods tackle motion forecasting tasks in an isolated manner, i.e. they treat every individual driving scene within a limited range independently, overlooking that in reality motion forecasting inherently temporally interrelated while any ego-car drives on. That means previous methods ignore the driving context across successive scenes, as well as the corresponding potentially useful information from previous driving periods (Fig. 1).
"Li Zhang (lizhangfd@ fudan.edu.cn) is the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

history
visible
current
invisible
2-
1-3
Existing methods
(b) RealMotion
Comparison of prediction
Independent and our predictions
scene and stream features tere Visible field
Figure I: Comparison of (a) existing methods independently processing each scene and (b) our Real- Motion recurrently collecting historical information. (c) For example, RealMotion can perceive the currently invisible pedestrian and predict the giving way for the interested agent.
Under the above insight and consideration, we propose efficient in-context motion forecasting framework for continuous driving, named RealMotion. It comprises two streams to transit states of scenes: (1) A scene context stream that accumulates historical scene context progressively, capturing temporal interactions among scene elements and addressing complex driving situations. (2) An agent trajectory stream that continually optimizes the predictions for dynamic agents like vehicles, considering temporal consistency constraints and capturing precise motion intention. Each stream utilizes specially designed cross-altention mechanism transit scene states and fulfill its function.
Our contributions are summarized as follows: (i) We solve the motion forecasting problem from a perspective of the real-world applications, which enables the extraction and utilization of valuable situational and progressive knowledge. (ii) We introduce RealMotion, a novel motion forecasting approach that sequentially leverages both scene context and predicted agent motion status time, meanwhile maintaining a lower real-world inference latency. (iii) To support the continuous driving setting on existing benchmarks, implement data reorganization strategy to generate scene sequences, closely simulating the real-world driving scenarios. Extensive experiments on Argoverse series with different settings demonstrate that RealMotion achieves state-of-the-ar performance.
2 Related work
In autonomous driving, accurately predicting future trajectories of agents of interest relies on an appropriate representation of scene elements. Early methods [29, 12, 2] rasterize driving scenarios into images and utilize off-the-shelf convolutional networks for scene context encoding. However, due their limited ability capture intricate structural information, recent studies [49, 15, 52, 36] have shifted towards vectorized representations, exemplified by the emergence of VectorNet [10]. Addi- tionally, graph-based structures are widely adopted model dynamics, interactions, and relationships among agents and maps [22, 13, 44, 19, 18, 9].
With the encoded scene features, various approaches explored for estimating multi-modas future trajectories. Early methods focus on goal- prediction [49, 15] or use probability distribution heatmaps for trajectory sampling [12, 13]. Recent approaches like HDGT [19], Wayformer [25], and others [23, 26, 48, 32, 51] leverage Transformer architectures [37] to model detailed relationships within the overall scene. Moreover, there methods introducing novel paradigms (e.g. pre- training [5, 4, 21, 28), post-refinement [6, 50) or Diffusion [20]) to achieve impressive performance.
To address the relevance of predicted trajectories for different agents in real-life scenarios, recent efforts have focused on multi-agent forecasting. Some methods [14, 38, 52, 33] adopt an agent- centric design, iteratively predicting trajectories for each agent, which can be inefffccent and hinder exploration of relationships among agents. Conversely, Scene Transformer [26] introduces a scene- centric framework for joint forecasting of all agents, presenting novel design. Moreover, recent methods explore a query-centric design [51] or consider adaptations [1] to address this task.
Nevertheless, the methods mentioned above independently perform forecasting for each scene sample, which conflicts with practical settings where driving scenarios are interconnected over time. To overcome this limitation, pioneering work [27] first introduce temporal motion benchmark based

---

(a) A given independent sceme
(b) Chunking and aggregating
fubure Nistoricall trjectoyy
61
T₁
T+1
sconoo elemmnts
TH
Ti
TH1
Continuous sub-scenes
Illustration of our data reorganization strategy. processing (a) a given independent scene by (b) chunking the trajectories into segments and aggregating surrounding elements, generating the (c) continuous sub-scenes.
Figure 2:
on tracking dataset. Instead, we design a transformed data structure mimic worl-world conditions and propose RealMotion to effectively model the temporal relationships.
3 Methodology
3.1 Preliminary
Data reorganization. Considering the discrepancy between existing benchmarks and practical applications, our first step is to reorganize these datasets by transforming each sample scene into continuous sequence, mimicking the successive real-world driving scenarios. Specifically, retrospectively examine each independent scene by evenly splitting agent trajectories into shorter segments and sampling local map elements (refer to Fig. 2). Specifically, first select several split points Ti along historical frame steps. Next, we generate trajectory segments of identical length by extending from these points both into the past and the future. The number of historical and future steps is determined by the minimum split point and the length of ground truth trajectory, respectively. Also, surrounding agents within a certain range and local map are aggregated for interested agents at each split point, forming sequence of sub-scenes. This reorganization allows freely capitalizing on the original elements to offer valuable situational and progressive insights at the scene level for model optimization. Hence, existing methods can also involve and benefit from the novel data structure. We have implemented this approach within the state-of-the-an method QCNet (51] J, which is further discussed in the appendix, highlighting the generality of our data structure.
Input representation. In the context of motion forecasting, the trajectories of agents and high- definition road map are provided in the driving scenario. Following [10], we transform these scene clements into vectorized representations input. The historical trajectories denoted as A E N. where Na. T. and C2 represent the number of agents, the number of historical frames, and the motion states (e.g., position, angle, velocity, and acceleration), respectively. The road map is divided into several lane segments, denoted as M E RN x PxCm, where N... P. and C.n indicate the number of lane segments, the points of each segment, and the lane features (typically represented as coordinates). All these states are normalized relative to the current position of the agent of interest.
3.2 RealMotion for continuous forecasting
As depicted in Fig. 3, our RealMotion approach comprises an encoder, a decoder, a scene context stream, and an agent trajectory stream. Following the encoder-decoder structure, the two streams are designed to execute temporal modeling, focusing on context information and trajectory prediction along the time dimension.
3

---

t-1
t+1
Trajectory Relaying
memory update
Output trajectories
Trajectory relaying
Trajectory relaying
Trajectory
relaying
TE
(b)
Output
Context referencing
Context referencing
Context referencing
Contect Referencing
P
(c)
Decode
agent-conteat cross-attn
(a)
map-map cross-attm
Input agents and
Encoder
Decoder
Agent trajectory stream
Scene context stream
Agent and map features
Figure 3: Overview of our RealMotion architecture. RealMotion adopts mncoder-decoder structure with two intermediate streams designed to capture interactive relationships within each scene and across the continuous scenes. The (a) Scene context stream and (b) Agent trajectory stream iteratively accumulate information for the scene context and rectify the prediction, respectively. The (c) context referencing and (d) trajectory relaying modules are specially-designed cross-attention mechanism for each stream.
3.2.1 Multimodal feature encoding and trajectory decoding
For scalability and simplicity, we adopt the plain encoder and decoder design following [5, 32]. Specifically, we encode the map features Fm E N.. xD by a PointNet-like encoder [30], where D refers to the embedding dimension. As [5], we extract the agent features Fa R N.ND by stacked neighborhood attention blocks [16]. Given the agent and map features, we concatenate them to derive the scene features Fg E (NN We then employ a Transformer [37] encoder to learn the interrelationships of these features.
For trajectory prediction along with the probability, we utilize two Multilayer Perceptron (MLP) modules. Additionally, we forecast a singular trajectory for auxiliary training purposes, focusing the movement patterns of other agents.
3.2.2
Scene context stream
The scene context stream is meticulously designed to progressively gather information about the surrounding environment, thereby enhancing trajectory prediction. Positioned after the encoder, this component plays crucial role the historical scene profoundly influences the understanding of temporal motion behaviors exhibited by agents. For instance, the ability to estimate the trajectory in complex scenes evaluate currently occluded agents can be greatly improved by taking into account the previous situation and context.
At the current time t, denote the current and historical scene features F and Ft respectively. extracted from the respective local coordinate systems. The process begins by projecting F Ft onto the current system. To achieve this, the two features must be aligned, considering the gap of motion between them. Motion-awart Layer Normalization (MLN) [40] is employed for this purpose:
7- MLN(EE-L PE(AA, Ay, A0, At)),
where AO denotes the heading angles, At refers to the timestamps, and (Ar, Ay) represents the difference between their positions. PE indicates the position encoding function. Subsequently, we repartition the scene features - and F into the agent and map parts for following process. To incorporate the historical information into the current, we employ map-map and agent-scene cross-attention modules with Multi-Head Attention (MHA) blocks for map and agent features, respectively:

---

F MHA(Q F K = F#-1 V F }
F MHA(Q Ftt F1-1 V Ft-1)
F Concatenate Fa Ft
where map features only interact with each other across time to enrich map elements, while agent features aggregate the overall historical scene context for comprehensive scene understanding.
Given the sequential nature, this past context-referenced feature will be further propagated to future timestamps. Simultaneously, it serves as the input the decoder for trajectory prediction at the current timestamp.
3.2.3 Agent trajectory stream
In addition referencing scene context, we enhance trajectory forecasting by establishing temporal relationships to achieve further improvement. This involves leveraging the inherent temporal continu- ity and smoothing nature of trajectories. This accomplished through the agent trajectory stream, which is equipped with memory bank for storing historical trajectories, enabling temporal relaying.
To maintain set of historical trajectories for each agent of interest, design the trajectory memory bank as M(a)) = { (2), f2), (Jn fn)]. where yi denotes predicted trajectories projected onto the global coordinate system, and fi represents corresponding mode features recording historical motion information. Unlike the detection or tracking tasks, where position changes of objects typically remain relatively consistent, the future motion patterss (under local system) involves sigmificant changes due to variations in road conditions. Therefore. directly decoding with historical memory query features inappropriate. Considering that the simple decoder can provide satisfactory predictions in practice, modifying initial predictions using the memory bank proves more effective.
Before refinement, we align the saved trajectories and mode features with the current coordinate system. While handling features consistently with the above, it is crucial to align trajectories in Euclidean space for more precise comparisons. Concerning the trajectory J/i R (with K being the frame steps of the trajectories), we compute the transformed trajectory Øt as
J R(0)= (y: " where y = y: ql.
(3)
Here, R(0) is the rotation matrix for the current heading angle, and yf"i denotes the trajectory origin chosen based on the time difference At and sampling frequency 9. Recognizing that memory trajectories may originate from different historical times, their origins also differ.
After transformation, trajectories whose latter part resembles the former part of current predictions contribute more to the refinement process. To aggregate memory information accordingly, we update the current mode features with lightweight Transformer Decoder utilizing Trajectory Embedding (TE) to measure spatial similarity and replace the original positional embedding. Then, we further propagate updated modes into MLP module to generate offsets and update initial predictions. This procedure is defined as follows:
Fmo = MHA(Q Fmo TEF(Y K, K= F. TE(V). P.),
(4)
Ymn = MLP((mo)) Ymo:
where Fmo and Ymo are mode features and initial predicted trajectories in the current context, Fb and Y,, represent aligned features and trajectories retrieved from the memory bank. Besides, we employ a single layer MLP to embed the flattened trajectories as the TE.
Ultimately, we save the refined trajectories (projected onto the global system) and features while removing outdated ones (first in first out). This updated trajectory memory will be further passed down for future timestamps. Importantly, due to the generation of only a small number of motion modes, we do not apply any filtering ensure the multimodality and diversity of the bank.
3.3 Model training
During the training process, we supervise the estimated trajectories using the regression loss Lreg and the associated probabilities through the classification loss Lele- Additionally, introduce the
5

---

Table Performance comparison on Argoverse set in the official leaderboard. For each metric. the best result is in bold and the second best result is underlined. ".": Unreported results; "f". Methods that use model ensemble trick. RealMotion- refers the independent variant of our model without data reorganization and stream modules, simply taking the original trajectory as input to forecast the motion like previous methods.
Method
minFDE,
minADE
minFDEg
minADEs
MRg
b-minFDE.
HDGT [19]
5.37
2.08
1.60
0.84
0.21
2.24
THOMAS [14]
4.71
1.95
1.51
0.88
0.20
2.16
GoRela [7]
4.62
1.82
1,48
0.76
0.22
2.01
HPTR [48]
4.61
1.84
1,43
0.73
0.19
2.03
QMLi [34]
4.98
1.84
1.39
0.69
0.19
1.95
Forecast-MAE [5]
4.36
1.74
1.39
0.71
0.17
2.03
TENET [42]
4.69
1.84
1.38
0.70
0.19
1.90
BANeti [45]
4.61
1.79
1.36
0.71
0.19
1.92
GANet [39]
4.48
1.78
1.35
0.73
0.17
1.97
SIMPL [47]
1,43
0.72
0.19
2.05
Gnetf [11]
4.40
1.72
1.34
0.69
0.18
1.90
ProphNet [41]
4.74
1.80
1.33
0.68
0.18
1.88
QCNet [51]
4.30
1.69
1.29
0.65
0.16
1.91
RealMotion-1
4.42
1.73
1.38
0.70
0.18
2.01
RealMotton
3.93
1.59
1.24
0.66
0.15
1.89
refinement loss Lrofine to guide the learning of predicted trajectory offsets within our agent trajectory stream. The overall loss L combines these individual losses with equal weights, formulated as follows:
L= L + Less + Lrefine,
(5)
For Creg and Crefine, employ the smooth-Ll loss, while the cross-entropy loss is utilized for Lele-
4 Experiments
4.1 Experimental settings
Datasets and metrics We assess the performance of our method using the Argoverse [3] and Argoverse [43] motion forecasting datasets both single-agent and multi-agent settings. The Argoverse 1 dataset comprises 323,557 sequences from Miami and Pittsburgh, while the Argoverse 2 dataset contains 250,000 scenes spanning six cities. the Argoverse dataset, predictors are tasked with forecasting 3 seconds of future trajectories for agents based on 2 seconds of historical observa- tions. In contrast, the Argoverse 2 dataset offers improved data diversity, higher data quality, larger observation window of 5 seconds, and an extended prediction horizon of 6 seconds. Additionally, both datasets have sampling frequency
Hz.
We employ standard benchmark metrics, encompassing minimum Average Displacement Error (minADEg) Ex), minimum Final Displacement Error (minFDEg). Miss Rate (MR)). and brier min- imum Final Displacement Error (b min F D D&.), across six prediction modes designed for the single-agent setting. Further details can found the appendix.
Implementation details We train our models using the AdamW W [24] Optimizer with a batch size of 32 per GPU for 60 epochs. Our model trained end-to-end with learning rate of 0.001 and weight decay of 0.01. The latent feature dimension is set to 128. Following [5, 32], we consider only agents and lane segments within a 150-meter radius the focal agent. For the samples in Argoverse 2, we split the whole scene into 3 segments, each with historical observation window of 3s and same prediction horizon of as the original. As the Argoverse 1, we set the historical window of 1s. To fully utilize historical information, we compute gradients and perform back propagation for all segments. Moreover, the RealMotion- trained and evaluated with a common configuration without dataset reorganization and stream modules.

---

Table 2: Performance comparison on Argoverse I validation set.
Method
minADEs
minFDEs
MRg
LaneRCNN [44]
0.77
1.19
0.08
TNT [15]
0.73
1.05
0.10
mmTransformer [23]
0.71
1.15
0.11
LaneGCN [22]
0.71
1.08
PAGA [8]
0.69
1.02
DSP [46]
0.69
0.98
0.09
ADAPT [1]
0.67
0.95
0.08
HiVT [52]
0.66
0.96
0.09
R-Pred [6]
0.66
0.95
0.09
HPNet [35]
0.64
0.87
0.07
RealMotion
0.61
0.91
0.07
Table 3: Performance comparison on Argoverse Multi--gent test set in the official leaderboard.
Method
avgMinFDE,
avgMinADE,
avgMinFDEg
avgMinADE,
actorMR;
FJMP [3]]
4.00
1.52
1.89
0.81
0.23
Forecast-MAB [5]
3.33
1.30
1.55
0.69
0.19
Gnet [11]
3.05
1.23
1.46
0.69
0.19
RealMotion
2.87
1.14
1.32
0.62
0.18
4.2
Comparison with state of the art
We first compare the performance of RealMotion with several top-ranked models the Ar- goverse 2 [43] motion forecasting benchmark. The results on the test split are presented in Tab. 1. RealMotion has far outperformed most of previous approaches. Concretely, our method stands distinctly ahead of other methods in terms of minFDE, and minADE,, showing performance enhancements of 8.60% and 5.91% relative to QCNet, respectively. We also get the almost top place for other metrics. Compared to our independent variant RealMotion-l, the proposed method exhibits significant performance improvements across all metrics, which conclusively demonstrates the effectiveness of our designs. Then, we compare the performance of RealMotion on the Argoverse I benchmark, with the results of the validation split presented Tab, 2. It is shown that our method also achieves a decent performance, especially for minAD Eg. We also provide our ensemble and multi-agent results Appendix
4.3 Multi-agent quantitative results
In the multi-agent setting, predictors are required to jointly estimate the future trajectories of all interested agents, which is crucial for the comprehensive perception of the driving scenario. Therefore, we also evaluate our RealMotion on the Argoverse Multi-agent test to prove the effectiveness, and provide simple results shown in Tab. 3. Although not integrated with specialized designs for multi-agent forecasting like [26, 1], our model also demonstrates superior performance compared to recent works owing to our sequential techniques.
4.4 Ablation study
We conduct ablation studies on the Argoverse 2 validation split for the single-agent setting to examine the effectiveness of each component in RealMotion. We adopt the default experiment settings following Sec. 4.1 to perform ablation in this section.
Effects of components. As shown in Tab, 4, we assess the effectiveness of each component in our network. The first row (ID-1) represents independent framework RealMotion-I,, which the same as reported in Tab. 1. First, our data reorganization approach extends dataset at extra cost and enables the exhaustive utilization of temporally continuous motion, resulting in noticeable improvement shown in the second row. Then, our proposed two streams can better learn temporal

---

Table 4: Ablation study on the core components of RealMotion on the Argoverse validation set. "Con. Data" indicates the processed continuous scenes. "SC Strm" and "AT Strm" indicate our proposed scene context stream and agent trajectory stream, respectively.
Con. SC
AT
ID
minFDE,
minFDEa minADE
minADE
MR.
b-minFDEg
Data
Strm
Strm
4.499
1,793
1.423
0.721
0.185
2.054
2
4.397
1.722
1.357
0.687
0.169
2,001
3
V
4.129
1.648
1.344
0.678
0.160
1.987
4
V 4.194
1.667
1.331
0.673
0.164
1.976
5
V
V
4.091
1.620
1.312
0,664
0.156
1.961
Table 5: Ablation study (a) (left) the Feature Alignment and Trajectory Embedding and (b) (right) the gradient steps and split points. For (a), "C.A." and "T.A." represenn the feature alignment modules used in the Context Referencing and the Trajectory Relaying blocks. "T.E." represents the Trajectory Embedding. For (b), "Grad Steps" indicates the number of steps we take to compute the gradient.
"Split Pts" indicates the split points used divide the trajectory.
C.A.
T.A.
T.F. minFDFs
minADEg
MRg
Steps
Sphi Pis
minFDEg
minADE5
MRg
1.334
0.681
0.163
(30, 40, 50)
1.420
0.716
0.175
1.328
0.674
0.160
2
(30, 40, 50)
1.341
0.681
0.162
4
1.326
0.673
0,158
(30, 40, 50)
1.312
0.664
0.136
v
1.324
0.670
0.158
(20, 50)
1.331
0.674
0.158
1.312
0.664
0.156
(40, 45, 50)
1.365
0.692
0.163
5
(30, 35, 40, 50)
1.323
0.668
0.158
relationships at the scene level, thereby both bringing additional improvements, represented as ID-3 and II)-4, respectively. Considering that these two streams are complementary each other, therefore, RealMotion that involves all these sequential techniques achieves remarkable performance gains, as indicated in the final row. Additionally, in contrast to the consistent improvements observed with the data processing approach, it is worth noting that our two streams demonstrate more significant advantages in single-mode metrics compared to the six-mode metries. As shown in Fig. 4, attribute this phenomenon to the enhanced capability of our streams to modify less accurate trajectories.
Effects of feature alignment and TE. The misalignment of features might have adverse effect on the performance when implementing feature interaction across scenes. Hence, we evaluate the impact of the alignment modules in both the Context Referencing and the Trajectory Relaying blocks in Tab. 5(a). As depicted in the first four rows, the removal of the alignment modules clearly results in performance decline. By incorporating these modules, We have, some extent, alleviated the negative impact of misalignment. However, executing it twice only yields marginal gains, which might be caused by the function overlap. Besides, we also analyze the performance our proposed Trajectory Embedding in the 4th and 5th rows. The performance gains indicate that the Trajectory Embedding can facilitate the selection of similar historical trajectories from the memory bank, thereby easily constraining and refining current predictions. Despite the simplicity in the design of these modules, they both contribute additional benefits for our network.
Effects of split points and gradient steps. In Tab. 5(b), we investigate the performance variations with respect the number of gradient steps taken and the split points along historical steps. From the 1st to the 3rd row, as we progressively increase the number of steps used for gradient computation, there a noticeable improvement in performance. This enhancement can be attributed the fact that computing the gradient for specific steps allows us to better capture the trajectory distribution for model training. From the 3rd the 5th row, we change the interval of split points from frames to 15 frames. Accordingly, the length of historical trajectory also changes, ranging from 40 frames 20 frames. As observed, similar trajectories occur in the sequence when using a short interval, which can significantly hinder the model to learn distinct motion patterns. Conversely, using a longer interval results in fewer historical trajectory frames available in each segment, which contradicts the optimization of one-shot forecasting. It is evident that our choices for the gradient steps and the split points are well-suited for the Argoverse benchmark. Moreover, we attempt to divide the trajectory into 5 segments in the final row. An excessively long trajectory sequence imposes

---

(4) Step1 (1=30)
(b) Step2 (T=40)
(4) Step3 (T=50)
(d) RealMation- (T=50)
Predicted Trajectory
GT Trajectory
Agout Interest
Surrounding Agent
Figure 4: Qualitative results on the Argoverse validation set. The panel (a)-(c) demonstrate the progressive forecasting results of our RealMotion, where the panel (c) is the final predictions for evaluation. The panel (d) shows the one-shot forecasting RealMotion-L.
Table 6: Comparison of model performance, inference speed, and memory size. "Latency": Inference
speed. "Params": : The number of parameters.
Method
Latency
Params
minF
minFDE,
HPTR (online)[48)
13ms
15.1M
1.43
4.61
HPTR (offline)
28ms
[1Ne
94ms
7.7M
1.29
4.30
RealMotion-I
16ms
2.0M
1.38
4.42
RealMotion (online)
20ms
2.9M
1.24
3.93
RealMotion (offline)
62ms
learning burden on our framework, preventing it from focusing on temporal relationships and yielding limited improvements.
Effects of the depth of cross-attention blocks. As shown in Tab. 7, we explore the influence of depth variations of cross-attenss cosssattentionn unit in the Context Referencing and the Trajectory Relaying blocks. Primarily our temporal blocks are lightweight regardless of depth to ensure the efficiency and universality. relatively deep cross-attention unit in the Context Referencing and the Trajectory Relaying blocks is necessary for processing history information and current information. We use depth of our default setting considering its better
balance.
4.5 Efficiency analysis
Balancing performance, inference speed, and model size is important for the model deployment. We compare our RealMotion with recent representative works, which include the real-time forecasting approach HPTR [48] and the state-of-the-an approach QCNet [5]]. We measure these approaches

---

Table Ablation on the cross-attention block depth.
"Params": The number of parameters.
depth
Params
minFDEs
minADE;
MRa
2.5M
1.348
0.679
0.165
2
2.9M
1.312
0.664
0.156
3
3.3M
1.328
0.677
0.159
and RealMotion on the Argoverse 2 test set using an NVIDIA GeForce RTX 3090 GPU, maintaining batch size of 1 and following end-to-end manner. As shown in Tab. 6, RealMotion has the competitive inference time and a competitive model size while achieving the best performance. worth noting that "online" indicates the latency for practical autonomous driving systems, which is optimized compared to the "offline" version by some efficient designs (e.g. the caching technique in [48]). As for RealMotion,
must consider three times latency for "offline" dataset, where only the final prediction is utilized for evaluation. In contrast, the forecasting results of each iteration is meaningful in "online" application.
4.6 Qualitative results
In Fig. 4, we present qualitative results of our network compared the independent version on the Argoverse 2 validation set. Panels (a), (b), and (c) illustrate the forecasting results at 3s, 4s and 5s, respectively. Panel (c) displays the final results used for evaluation, which are more accurate and closer to the ground truth. By comparing panel (c) and (d), it can be seen that RealMotion remarkably outperforms the independent version. As demonstrated in the panels from (a) to (c), our RealMotion can progressively refine the estimated trajectories from coarse to fine as the motion progresses and accurately capture the possible motion intention. However, the one-shot forecasting of RealMotion-1 shown in the panel (d) leads to significant estimation errors.
5
Conclusion
In this work, we anticipate to address the motion forecasting task from more practical continuous driving perspective. This essence places the motion forecasting function wider scene context compared to the previous setting. We further present RealMotion, a generic framework designed particularly for supporting the successive forecasting actions over space and time. The critical components of our framework are the scene context stream and the agent trajectory stream, both of which function sequential manner and progressively capture the temporal relationships. Our extensive experiments under several settings comprehensively demonstrate that RealMotion surpasses current state-of-theaar performance, thereby offering a promising direction for safe and reliable motion forecasting in the rapidly evolving field of autonomous driving.
Limitations. clear constraint of our data processing approach its requirement for a sufficient number of historical frames for serialization. Consequently, it is not applicable to short-term benchmarks such as the Waymo Open Dataset, which provides only 10 frames of historical trajectory, Moreover, existing datasets typically provide limited historical information distinct from real-world settings, which inhibits our sequential designs from fully leveraging their advantages. Hence, we anticipate to integrate framework into sequential autonomous driving system to maximize the benefits of streaming designs in our future work.
Acknowledgments
This work was supported in part by National Natural Science Foundation of China (Grant No. 62106050 and 62376060), Natural Science Foundation of Shanghai (Grant No. 227R 2 1405500).
10

---

References
[1] G. Aydemir, A. K. Akan, and F. Güney, Adapt: Efficient multi- trajectory prediction with adaptation. In ICCV, 2023. 2,
[2] Y. Chai, B. Sapp, M. Bansal, and D. Anguelov. Multipath: Multiple probabilistic anchor trajectory
hypotheses for behavior prediction. In CoRL, 2020. 2
[3] M.-F. Chang. Lambert, Sangkloy, J. Singh, Bak, Hartnett, Wang. P. Carr, S. Lucey, D. Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In CVPR, 2019. 6
H. Chen, J. Wang. K. Shao, F. Liu, J. Hao, C. Guan, G. Chen, and P.-A. Heng. Traj-mae: Masked
autoencoders for trajectory prediction. ICCV, 2023. 2
[5] J. Cheng. X. Mei, and M. Liu. Forecast-mae: Self-supervised pre-training for motion forecasting with
masked autoencoders. In ICCV, 2023. 2, 4, 7
[6] S. Choi, J. Kim, J. Yun, and J. W. Choi. R-pred: Two-stage motion prediction via tube-query attention-based
trajectory refinement. In ICCV, 2023. 1, 2,
[7] A. Cui, S. Casas, K. Wong. S. Suo, and R. Urtasun. Gorela: Go relative for viewpoint-invariant invariant motion
forecasting. In ICRA, 2023. 6
[8] F. Da and Y. Zhang. Path-aware graph attention for hd maps motion prediction. In ICRA, 2022. 7
[9] N. Deo, E. Wolff, and O. Beijbom. Multimodal trajectory prediction conditioned on lane-graph traversals. In CoRL, 2022.
[10] J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid. Vectornet: Encoding hd maps and agent dynamies from vectorized representation. In CVPR, 2020. 1,2,
[11] X. Gao, X. Jia, Y. Li, and H. Xiong. Dynamic scenario representation learning for motion forecasting with heterogeneous graph convolutional recurrent networks. IEEE RA-L, 2023. 6,
[12] T. Gilles, S. Sabatini, D. Tsishkou, B. Stanciuleseu, and F. Moutarde. Home: Heatmap output for future motion estimation. In JEEE ITSC, 2021.
[13] T. Gilles, S. Sahatini, D. Tsishkou, B. Stanciulescu, and F. Moutarde. Gohome: Graph-oriented heatmap
output for future motion estimation. In ICRA, 2022. 2
[14] T. Gilles, S. Sahatini, D. Tsishkou, B. Stanciulescu, and F. Moutarde. Thomas: Trajectory heatmap output
with learned multi-agent agget sampling. In ICLR, 2022. 1, 2, 6
[15] Gu, C. Sun, and H. Zhao, Densetnt: End-to-end trajectory prediction from dense goal sets. In CVPR,
2021. 2,
[16] A. Hassani, S. Walton, Li, Li, and H. Shi. Neighborhood attention transformer. CVPR, 2023, 4 [17] Y. Huang. J. Du. Z. Yang. Z. Zhou, L. Zhang, and H. Chen. A survey on trajectory prediction methods for autonomous driving. IV, 2022.
trajectory-prediciinn
[18] X. Jia, L. Sun, H. Zhao, M. Tomizuka, and W. Zhan. Multi-agent trajectory prediction by combining egocentric and allocentric views. In CoRL, 2022. 2
[19] x. Jia, Wu, L. Chen, Y. Liu, H. Li, and Yan. Hdgt: Heterogeneous driving graph transformer for multi-agent trajectory prediction via scene encoding. IEEE TPAMI, 2023. 2, 6
[20] C. Jiang, A. Cormman,
Park, B. Sapp, Y. Zhou, D. Anguelov, et al. Motiondiffuser: Controllable multi-agent motion prediction using diffussio. In CVPR, 2023.
[21] Z. Lan, Y. Jiang, Y. Mu, C. Chen, and S. E. Li. Sept: Towards efficient seene representation learning for motion prediction. In ICLR, 2024.
[22] M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun. Learning lane graph representations for motion forccasting, In ECCV, 2020. 2,7
[23] Y. Liu, J. Zhang, L. Fang, Q. Jiang, and B. Zhou. Multimodal motion prediction with stacked transformers.
In CVPR, 2021. 2,7
[24]
Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2018. 6
[25] N. Nayakanti, R. Al-Rfou, A. Zhou, K. Goel. K. S. Refaat, and B. Sapp. Wayformer: Motion forecasting
via simple & efficient attention networks. In ICRA, 2023. 2
[26] J. Ngiam, V. Vasudevan, B. Caine, 7. Zhang, H.-T. I., Chiang, J. Ling, R. Roelofs, A. Bewley, C. Liu,
A. Venugopal, et al. Scene transformer: A unified architecture for predicting future trajectories of multiple agents. In /CLR, 2021. 1, 2.7
[27] 7. Pang, D. Ramanan, M. Li, and Y.-X. Wang. Streaming motion forecasting for autonomous driving. In IROS, 2023. 2
[28] D. Park, J. Jeong, S.-H. Yoon, J. Jeong, and K.-. Yoon. T4p:
Test-time training of trajectory prediction via masked autoencoder and actor-specific token memory. In CVPR, 2024. 2
[29] T. Phan-Minh, F. C. Grigore, F. A. Boulton, O. Beijhom, and E. M. Wolff. Covernet: Multimodal hehavior prediction using trajectory sets. CVPR, 2020.
[30] C. R. Qi. H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017. 4
[31] L. Rowe, M. Ethier, E.-H. Dykhne, and K. Czamecki, Fjmp: Factorized joint multi-agent motion prediction over learned directed acyclic interaction graphs. In CVPR, 2023. 1.7
11

---

[32] S. Shi, L. Jiang, D. Dai, and B. Schiele. Motion transformer with global intention localization and local movement refinement. In NeurlPS. 2022. 1, 2, 4, 6
[33] S. Shi, L. Jiang. D. Dai, and B. Schiele. Mir+:: Multi-agent motion prediction with symmetric scene
modeling and guided intention querying. IEEE TPAMI, 2024. 2
[34] T. Su, X. Wang. and X. Yang. Qml for argoverse motion forecasting challenge. arXiv preprint, 2022.
[35] X. TTngg M. Kan, S. Shan, 7. Ji, J. Bai, and X. Chen. Hpnet: Dynamic trajectory forecasting with historical prediction attention.
CVPR, 2024. 7,7
[36] B. Varadarajan, A. Hefny, A. Srivastava, K. S. Refaat, N. Nayakanti, A, Cormman, K. Chen, B. Douillard,
C. P. Lam, D. Anguelov, al. Multipath++:: Efficient information fusion and trajectory aggregation for
behavior prediction. In ICRA, 2022.
[37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, I., Jones, A. N. Gomez, t.,. Kaiser, and I. Polosukhin. Attention is all you need. In NeurIPS, 2017. 2,4
[38] Wang. T. Ye, 7.. Gu, and Chen. Ltp: Lane-based trajectory prediction for autonomous driving. In CVPR, 2022. 2
[39] M. Wang, X. Zhu, C. Yu, W. Li, Y. Ma, R. Jin, X. Ren, D. Ren, M. Wang, and W. Yang. Ganet: Goal area
network for motion forecasting. In ICRA, 2023. 6
[40] S. Wang. Y. Liu, T. Wang. Y. Li, and X. Zhang. Exploring object-centric temporal modeling for efficient
multi-view 3d object detection, In ICCV, 2023.
[41] X. Wang. T. Su, F. Da, and X. Yang. Prophnet: Efficient agent-centric motion forecasting with anchor-
informed proposals. In CVPR, 2023. 6
[42] Y. Wang. H. Zhou, 7. Zhang. C. Feng. H. Lin, C. Gao, Y. Tang, Z. Zhao, Zhang, J. Guo, et al. Tenet: Transformer encoding network for effective temporal flow on motion prediction. arXiv preprint, 2022. 6 [43] B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, D. Ramanan, P. Carr, and Hays. Argoverse 2: Next generation datasets for self-driving perception
and forecasting. In NeurIPS, 2021. 1, 7,
[44] W. Zeng, M. Liang, R. Liao, and Urtasun. Lanerenn: Distributed representations for graph-centric motion forecasting.
IROS, 2021. 7,7
[45] C. Zhang, H. Sun, C. Chen, and Y. Guo. Banet: Motion forecasting with boundary aware network. arXiv
preprint, 2022. 6
[46] I. Zhang, P. Li, J. Chen, and S. Shen. Trajectory prediction with graph-based dual-scale context fusion. In IROS, 2022. 7
[47] L. Zhang, P. Li, S. Liu, and S. Shen. Simpl: A simple and efficient multi-an multi-agent motion prediction bascline for autonomous driving. IEEE RA-I., 2024. 6
[48] Z. Zhang, A. Liniger, C. Sakaridis, F. Yu, and Van Gool. Real-time motion prediction via heterogeneous polyline transformer with relative pose encoding. In NeurlPS, 2023. 2, 6, 9, 10
[49] H. Zhao, Gao, T. Lan, C. Sun, B. Sapp. B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid, al. Tnt: Target-driven trajectory prediction. In CoRL., 2021. 1,2
[50] Y. Zhou, H. Shao, L. Wang, S. L. Waslander, H. Li, and Y. Liu. Smartrefine: An scenario-adaptive
refinement framework for efficient motion prediction. In CVPR, 2024. 1.2
[51] Z. Zhou, J. Wang, Y.H. Li, and Y.-K. Huang. Query-centric trajectory prediction. In CVPR, 2023. 1, 2, 3,
6, 6,9,1 13
[52] Z. Zhou, L. Ye, Wang, Wu, and K. Lu. Hivt: Hierarchical vector transformer for multi aglti-agent motion prediction. In CVPR, 2022. 1, 2,
12

---

Appendix
A More experimental settings
A.1 Evaluation metrics
For single-agent evaluation, we employ standard metrics for evaluation, including minimum Average Displacement Error (minADDR Fx.), minimum Final Displacement Error (min F&g), Miss Rate (M Rx), and brier minimum Final Displacement Error min D2)) minADEx calculates the L2 distance between the ground-truth trajectory of the best K predicted trajectories, averaged across all future time steps. While min Es measures the difference between the predicted endpoints and the ground truth. M R& is the ratio of scenes where min DEg exceeds 2 meters. To further assess uncertainty estimation performance, the metric minn D DE, adds (1 x)² to the final-step error, where denotes the best-predee best-predicted trajectory's 's probability score that the model assigns. As common practice, K is selected as and 6.
For multi-agent evaluation, We use standard metrics including Average Minimum Final Displacement Error (avg.innF FDF), Average Minimum Average Displacement Error (aug F) and Actor Miss Rate (actor M R). ing.MinFDD.. is the average of the lowest FDEs for all scored actors within scene, reflecting the prediction accuracy of the lowest ADEs for all scored actors within scene, indicating the general accuracy of the predicted movements. Across the evaluation set, the actor M R the proportion of missed actor (same as above).
scene outcome. angMinnAA
represents the average
B
More experiments
B.1
Model ensemble
Model ensemble, crucial technique for enhancing the accuracy of final predictions, is employed our approach. We utilize six sub-models trained with various random seeds and split points. Consequently, we generate 36 predicted future trajectories for each agent, and then apply k-means clustering to process them with 6 cluster centers. For each cluster group, we calculate the average of all trajectories within the group to produce the final trajectories. The results with and without model ensemble trick on the Argoverse test set are shown in Tab.
Table Performance comparison between results with and without ensemble on Argoverse 2 test set in the official leaderboard. For each metric, the better result is in bold.
Motion
minFDE
minADE
minFDEg
minADEs
MRg
b-minFDEs
w/o ensemble
3.93
1.59
1.24
0.66
0.15
1.89
w/ ensemble
3.87
1.55
1.18
0.63
0.13
1.78
B.2 Generality evaluation with integrating RealMotion
For generality test, we integrate our RealMotion data with the state-of-the-art method QCNet [51]. Due to the big size of QCNet, the scene context and agent trajectory streams have to be excluded for memory constraint. With minimal alterations applied to this prior model, we observe noticeable improvement. The results are shown in Tab. 9.
Table 9: Generality evaluation with integrating RealMotion.
Method
min minFDEe
minADE,
MRs
QCNet [51]
1.27
0.73
0.16
CNet w/ RealMotion
1.24
0.71
0.15
13

---

C More qualitative results
We provide more qualitative results of our framework in Fig. and Fig. on the Argoverse validation set.
D Failure cases
Although our RealMotion has achieved outstanding performance on motion forecasting benchmarks, it still has some failure cases. We analyze these typical cases and provide qualitative results to help readers understand the circumstances under which our model may fail. This analysis will aid future work in developing
powerful and robust algorithm, shown in Fig. 5.
D.1 Case 1: Complex map topology
In the first row Fig. 5, the agent requires to navigate through complex intersection to one of the roads, but the model fails to predict this possible driving behavior and just anticipates the agent to drive straight ahead. This may be caused by the lack of a comprehensive understanding of the complex map topology and the unbalanced distribution of driving data. In most scenarios, agents tend to exhibit only trivial behaviors, such as moving straight ahead a nearly constant velocity. Consequently, this raises issues regarding data balance, and the figures indicate that model is more likely to make mistakes when it comes to turning.
D.2 Case 2: Subjective driving behavior
In the second row of Fig. 5, the vehicle is expected to park on the side of the road, which is a kind of subjective driving behaviors. However, the model only predicts that the vehicle will keep going ahead. To improve the forecasting of such cases, we could enhance the interaction between the model with additional intentions human incorporate more information, such as visual cues like turn signals and parking spaces.
(a) Step1 (T=30)
(b) Step2 (T=40)
(c) Step3 (T=50)
(d) RealMotionI (T=50)
Predicted Trajectory
GT Trajectory
Agent of Interest
Surroundigg Agent
Figure 5: Failure cases. the first row, the model fails predict the turning behavior at complex intersections, while in the second row, it fails to predict the parking behavior.
14

---

(a) Step1 (T-30)
(b) Step2 (T-40)
Step3 (I-50)
(d) RealMotion-(1-50)
Predicted Trajectory
GT Trajectory
Agent of Interest
Surrounding Agent
Figure 6: More qualitative results.
15

---

(a) Step1 (T=30)
(b) Step2 (T=40)
(c) Step3 (T=50)
RealMotion-I (T=50)
Predicted Trajectory
GT Trajectory
Agent of Interest
Surrounding Agent
Figure 7: More qualitative results.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Experiments in Sec. 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: see Conclusion in 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
17

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Experiments in Sec. 4 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
Seerides/CooSSubmiise wwwiisssssoolioioooooo icy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececceccccccct /ttp://///iieeeeeeeuuuubiiiiiiii /CodeSubmic ssisnnonllll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see experimental settings in Sec. 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No].
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
19

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification: See Experiments in Sec. 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: : /ttp///eerrpseee/ee//ee//e cc hc/pppblicttthicccsueeeeeee..eee
didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: It is main idea designed for real-world autonomous driving applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
20

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
21

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "KTrk9UzsFC", "reviewer": "Reviewer_fgWb", "summary": "The paper introduces an approach to iteratively process temporal scene context for the purposes of agent motion prediction. This differs from traditional approaches that ingest the whole context directly. \n\nThe scene is processed in temporal chunks, where in each chunk contains agent-centric context consisting of other agents and map within 150m. Information is aggregated across chunks (i.e. over time) using two novel components: \n- \"Scene context stream\", where agent/map features from the past frame are transformed using Motion-aware Layer Normalization [40] and cross-attended by current frame features. \n- \"Agent context stream\", where agent predictions from the current frame are modified by attending predictions from past frames (stored in a memory bank and properly aligned). \n\nThis 'streaming-type' method obtains very strong results on the Argoverse 1 and 2 datasets, and is shown to improve two different baseline  model architectures based on [5] and [51] that consume the whole historical context in a single step. Some helpful ablations and latency studies of the proposed components is provided.", "questions": "Main question is, do you agree with my reading that \"the contribution of the paper is to follow a streaming paradigm and design more domain-appropriate fusion mechanisms across time than the vanilla MLP or transformer ops that are used traditionally. \" As opposed to the strong claims in the current abstract / intro. \n\nWhy did you omit SEPT from the leaderboard results (Table 1)? \n\nPlease see my clarification or ablation/more experiment comments in the Weaknesses. I am particularly interested to understand why you did not try to attend to longer temporal context in the last chunk (if I read the paper correctly).", "limitations": "The same RealMotion methods tried here could be tried on the Waymo Open Dataset which also has 9 second segments, but one could take the first 3 seconds as history and predict 6 seconds, as opposed to the standard 1s / 8s split that is mentioned in the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- Strong idea to stream scene context into the model, shown to be effective even with short 2-3 'chunks' of context over 2-3 seconds. \n- Intuitive and well designed scene and agent context stream components that aggregate information across frames (or 'chunks'). The components are designed to model local information in agent-centric coordinates and aggregate it in a geometry-aware manner. \n- Strong overall results on Argoverse 1 and 2, and when comparing the baseline models the new technique is extending.  \n- Mostly comprehensive related work section.", "weaknesses": "- In the abstract/intro, I find some of the framing to be a bit misleading. Examples: \n  - \"7:This significantly simplifies the forecasting task, making the solutions unrealistic to use in practice\". I do not find this claim to be well supported. The current methods still ingest a significant amount of relevant context. To me the contribution of the paper is to follow a streaming paradigm and design more domain-appropriate fusion mechanisms across time than the vanilla MLP or transformer ops that are used traditionally. \n  - \"31 - 36: existing methods all tackle an artificially narrowed and unrealistic motion forecasting task\".... The methods still ingest 2-3 seconds of context, but perhaps not as effectively as the proposed work. That does not confirm this overly strong claim however. Also the paper only demonstrates good performance with up to 3s of context, that does not prove yet that much longer context is helpful or practical. \n  - To a similar point, Fig 1 is also a bit misleading: current methods would directly ingest the available scene history in the current datasets, and fuse the information, even if the information fusion is not done with the same geometric and streaming intuitions. \n\n- In the method description, some details are unclear or could be explained better:\n  - 83: \"generate trajectory segments of identical length\" Length can be misconstrued to be the length of the trajectories in meters, not that they are the same across chunks. This whole paragraph could be rewritten for clarity and a more helpful diagram could be shown -- illustrating what exactly is done for Argoverse 2 (one now needs to read all the way to Sec 4 to understand how this works). \n  - \"114: Additionally, we forecast a singular trajectory for auxiliary training purposes, focusing on the movement patterns of other agents. \" \nUnclear what this means exactly. Do you produce only a single trajectory prediction in past steps?\n  - \"“Equal length into past and future” → confusing, assume fixed time interval in both directions. \n  - The L_refine loss in Eq 5 is not explained, unclear what is being refined. My guess is that this is predictions in earlier chunks\n  - What is exactly RealMotion without the context stream components? This is present both in the Ablation and in Appendix B.3. How do you fuse information across time then? \n\n- In the experimental results, some SOTA methods from the leaderboard seem to omit SEPT (ICLR 2023), which has higher numbers than RealMotion, even though it is cited. Why did you you omit it? \n\n- A couple more ablations or experiments would be helpful. \n  - It seems that the method is similar to a filter, where the current chunk only attends to previous chunk (but not to the ones before). Is my reading of the paper correct? If so, an experiment where the method can attend longer context memory with multiple frames would be interesting. \n  - It would also be interesting to try pretraining this method on more data examples (those can be produced by predicting shorter futures, then you can use more of the segments). And then fine-tune on 6s futures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review as well as the suggestions for improvement. Our response to the reviewer’s comments is below:\n\n**Q1: Some of the framing to be a bit misleading.**\n\nApologies for clarity issue in abstract and introduction. We intent to stress the significance of streaming design in real-world settings. We will improve the clarity.\n\nAs for Fig. 1, we discuss in real-world setting where scenes should be continuous, going beyond the datasets. In this case, current methods can only process independent scenes with limited history as set up in current datasets, while we advocate to utilize more abundant historical information. \n\n**Q2: Method description.**\n\nApologies for unclear expression.\n\n+ **83:** Indeed, the $length$ refers to time steps instead of meters. We will clarify this part.\n\n+ **114:** It is not for past steps but for other agents. In addition to predicting the trajectories of agents of interest for evaluation, we also predict a single trajectory of other surrounding agents (not involved in evaluation) for auxiliary supervision. That is typical and beneficial for learning better motion pattern.\n\n+ **Equal length:** We assume this refers to Line 83 -- that means the newly generated sub-trajectories have equal total steps, but the number of steps in the history and future can differ. \n\n+ **refine loss:** $L_{\\rm refine}$ is utilized to supervise the refined trajectories $Y_{\\rm mo}$ mentioned in Eq(4), which can be formulated as $L_{\\rm refine} = {\\rm smoothL1}(Y_{\\rm mo}, Y_{\\rm gt})$, where $Y_{\\rm gt}$ refers to the ground truth trajectories.\n\n+ **RealMotion:** In ablation, RealMotion without stream modules can be considered as a one-shot forecasting models similar to current methods. This variant cannot fuse temporal information, but utilizing our reorganized data as a data augmentation method is still possible. In Appendix B.3, it should refer to only using our reorganized data, which we will clarify.\n\n**Q3: Comparison with SEPT.**\n\nWe consider that SEPT is a pretrained self-supervised method orthogonal to ours. For example, it utilizes all sets (including test set) to address the disparity between train set and test set, leading to more data used than current methods and making the direct comparison unfair. A property way of using SEPT as model pretraining is likely to improve general methods like ours, and we could include them for further improvements.\n\n**Q4: More ablations**\n\n+ Actually, the current chunk has attended to several preceding chunks. For the trajectory stream, we explicitly maintain a memory bank for more history predictions from a longer history. For the scene stream, since previous chunk involves earlier context, the current can also utilize a longer context to some extent through implicit propagation. Besides, We have tried to propagate two frames of scene context as shown below:\n\n    ||Latency|Memory|$minFDE_6$|$minADE_6$|\n    |:---:|:---:|:---:|:---:|:---:|\n    |2 frames|23ms|1.8G|1.28|0.64|\n    |1 frames|20ms|1.4G|1.31|0.66|\n\n    This strategy can indeed bring slight improvements, but we finally did not adopt more frames due to considerations of simplicity, efficiency and memory overhead, especially in complex scenarios such as intersections.\n\n+ It is an excellent insight! To make the data more aligned with the forecasting task, we did not process future trajectories (such as Waymo mentioned below) in this paper, but it is indeed worth integrating and exploring distinct motion tasks (simulation, prediction and planning) and datasets. We intend to further extend our model in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0fC9BZ5jAS", "reviewer": "Reviewer_fqqC", "summary": "This paper proposes a framework named \"RealMotion\" highlighting the importance of continuous motion forecasting in autonomous driving. From the formulation perspective, RealMotion investigates predicting trajectories in a continuous sequence of timestamps instead of previous independent predictions. From the methodology perspective, RealMotion proposes the \"relaying\" modules that propagate historical scene and agent contexts into the current frame to enhance prediction. Finally, RealMotion achieves top performance on Argoverse benchmarks and proves the importance of continuous contexts.", "questions": "See weakness above.", "limitations": "The authors have discussed limitations. No additional limitations from my side.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper investigates a foundational challenge in autonomous driving that is overlooked by previous people -- how to model the continuous contexts in motion forecasting. This problem is reasonable and is also something I would love to follow and work on.\n\n2. The methods proposed by RealMotion are intuitive and reasonable. Relaying the historical context to current frames to enhance motion forecasting is indeed a novel contribution to autonomous driving.\n\n3. The experiments prove the effectiveness of continuous contexts. In addition to the state-of-the-art performance, the comparison between RealMotion-I and RealMotion in Table 1 makes it a strong argument.", "weaknesses": "1. The authors have missed some closely related works, such as streaming motion forecasting (e.g., Pang et al.) and other end-to-end autonomous driving (e.g., Gu et al.), where motion forecasting is continuous. For instance, Pang et al. also discover the setbacks of predicting trajectories on independent timestamps, formulate the \"streaming\" forecasting task, and propose a differentiable filter to refine the predicted trajectories. I acknowledge the newly designed modules by RealMotion, but I would expect the authors to have a better discussion of these relevant efforts.\n\nPang et al. Streaming Motion Forecasting for Autonomous Driving. IROS 2023.\nGu et al. ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries. CVPR 2023.\n\n2. RealMotion is also closely related to QCNet and based on its code. QCNet already has the interface of continuous motion forecasting and leverages a GRU to deal with the temporal contexts in queries. First of all, I would like the authors to clarify or discuss in the paper about the difference/improvement to QCNet more explicitly. In addition, I am also curious if it is possible to support that the \"relaying\" modules in RealMotion is better than QCNet's end-to-end mechanism. For instance, is it possible to plug the relaying module to QCNet and make a comparison?\n\n3. I would like the authors to develop some metrics to analyze the improvement in the trajectories. More specifically, we wish to analyze \"where does RealMotion improve the trajectory quality?\" Since the main argument of this paper is \"continuity,\" I am curious if the author can use a \"fluctuation\" measure like Pang et al. to prove that the trajectory smoothness is indeed improved. It is also encouraged to show some visual comparisons if space is allowed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review as well as the suggestions for improvement. Our response to the reviewer’s comments is below:\n\n**Q1: Missing related work.**\n\nThanks. We will add them.\n\n**Q2: Comparison with QCNet.**\n\nTo ensure efficiency, our method adopts an agent-centric design different from QCNet. As mentioned with QCNet only the query-centric design is good at continuous motion forecasting and GRU is employed for fusing continuous predictions in the current scene. Hence, it should still be considered an independent method.\n\nWe have tried to integrate our stream modules with QCNet, but unfortunately, it cannot be trained in our environment due to huge memory overhead. We hence just provided the results using our reorganized data in Appendix B.3.\n\n**Q3: Continuity metrics.**\n\nFollowing Pang et al., we evaluate the $fluctuation$ for RealMotion and baseline using our reorganized data as shown below. It can be observed that our method certainly improves the $fluctuation$ metric, with the Trajectory Stream playing a particularly important role. \n\n||baseline|baseline+scene stream|baseline+traj stream|RealMotion|\n|:---:|:---:|:---:|:---:|:---:|\n|$fluctuation$|0.385|0.371|0.354|0.347|\n\nWe also provide some qualitative results in PDF file Fig. 2, which demonstrates that RealMotion can better maintain temporal consistency of trajectories.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kScocGcx8b", "reviewer": "Reviewer_GAo1", "summary": "This submission tackles the task of trajectory forecasting in autonomous driving. \nIn particular, it proposes improvements in two aspects:\n\n1/ Data reorganization: Current datasets are artificially split into non-overlapping segments. The authors propose to reorganize the data to have overlapping windows.\n\n2/ Relaying context and predictions: since the sub-scenes now overlap, they can use context and predictions from the previous timesteps (corresponding to a different sub-scene) to improve predictions at the current timestep.\nRelaying context serves to provide more information about the surroundings (an agent could have been missed in the current frame for instance), and relaying the predictions would smooth out the predictions across time.\n\nThey demonstrate the validity of their approach on Argoverse benchmarks, achieving state-of-the-art performances.", "questions": "- Why do we see drastic improvement on ADE/FDE_1 but the other improvements are very tame? Does that imply most of the improvements are to the scoring function so that RealMotion gets a better top 1? If that's the case, what could be the mechanisms that explain this behavior?\n\nOtherwise, I think this paper rightfully questions some practices of motion forecasting in autonomous driving. Generalizing something along the lines of the proposed data reorganization would be very helpful for the community. However, I think the experiments on relaying might not be fair enough and need to be compared to simple baselines with more/better past history. We would also need a few more numbers to see if most of the findings and recommendations have a good chance at being consistent accross models.", "limitations": "- Limitations are discussed in the Appendix.\n\n- Broader societal impacts do not seem explicitly addressed despite what has been reported in the checklist. I'm not sure there are obvious uncontroversially negative societal impacts to be discussed though.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed improvements are interesting, sound, and motivated.\n- The data reorganization, I believe, is a simple idea that would that should be pushed in the motion forecasting community to build a new standard that better leverage the data. It can also immediately be applied to other models and hopefully improve performances.\n- The relaying through time helps achieve strong results either with their architecture or on top of QCNet.", "weaknesses": "- The results are not very conclusive:\n  - Compared to  QCNet, except for $ADE_1$/$FDE_1$ (which are not the main metrics usually considered for motion forecasting) the improvements are small. It would be useful to give more insight into why only these metrics improve. We also don't have the $ADE_1$/$FDE_1$ values of QCNet + RealMotion so we don't know if these findings are consistent across models.\n  - The comparisons are not fair in terms of available information. Because of the propagation, RealMotion has access to more past information both in the past trajectories and agents, and the past map. A naïve way to integrate more past trajectories would be to just use longer past history. This should be a minimal baseline or ablation in the paper. If we also wanted to add agents seen in the past but not at the current timestep, it could be done with some masking or extrapolation scheme for missing frames. \nRemains the past map information, for which it's indeed more complicated to have a naïve baseline for a fair comparison, as simple aggregation could drastically raise the computation footprint so I can understand why the authors wouldn't want to test this baseline.\n- The loss functions, and in particular $\\mathcal{L}_{refine}$ are not detailed enough. I can guess where they apply but it would be good if that was stated explicitly.\n- Figure 2 is unclear to me. Perhaps it could need more captions or a rework.\n- I think it would be easier to interpret the results if the architecture of RealMotion would be more clearly delivered, perhaps in a Figure  (in Appendix?) that represent the full architecture detailing in particular how the different elements interact in the encoder and the decoder that are not given in Figure 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review as well as the suggestions for improvement. Our response to the reviewer’s comments is below:\n\n**Q1: Drastic improvement on $ADE_1/FDE_1$.**\n\nBy comparing QCNet and other methods (such as ProphNet), it can be seen this phenomenon is common. This is due to different calculation of metrics: the metrics@1 is the error for the one selected trajectory according to the score, while the metrics@6 is the minimum error among 6 predicted trajectories. Hence, there is more room for improvement for the metrics@1.\n\nRegarding the comparison between QCNet and ours, the improvement in $FDE_{6}$ is still salient. Besides, we provide the complete comparisons below. Since we can only integrate our restructured data, please refer to Tab.3 row 1 and 2 for comparison.\n\n|Method|$minFDE_1$|$minADE_1$|$minFDE_6$|$minADE_6$|\n|:---:|:---:|:---:|:---:|:---:|\n|QCNet|4.34 | 1.69 | 1.27 | 0.73|\n|QCNet w/ data|4.26 | 1.64 | 1.24|0.71|\n\n\n**Q2: The comparisons are not fair.**\n\nWe do NOT use more past information compared to existing methods, but just process past information in a different way with the same datasets. Taking existing methods on Argoverse 2 as an example, they directly process each independent scenes with 50 historical frames, while we split each scene into several continuous small scenes with less past information (30 historical frames), then processing them in a streaming fashion. Hence, existing methods and RealMotion all use the same past information, making a fair comparison.\n\n**Q3: Loss functions.**\n\nApologies for unclear expression. We will add the formulation for these losses. Especially, $L_{\\rm refine}$ is utilized to supervise the refined trajectories $Y_{\\rm mo}$ mentioned in Eq(4), which can be formulated as\n$L_{\\rm refine} = {\\rm smoothL1}(Y_{\\rm mo}, Y_{\\rm gt})$, where $Y_{\\rm gt}$ refers to the ground truth.\n\n**Q4: Fig.2.**\n\nFig.2 depicts how we process existing datasets with independent scenes into a continuous format to simulate real-world situations. We will remove some redundant elements to make it clearer.\n\n**Q5: The architecture of RealMotion**\n\nTo ensure the generalizability of our proposed modules, as most current methods we adopt a standard forecasting framework, which consists of an agent encoder and a map encoder to respectively encode agent and map features, a Transformer encoder to implement interaction, and a decoder to generate trajectories and corresponding probabilities. The figure is shown in PDF file Fig.1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tB95M5ct6g", "reviewer": "Reviewer_u4R4", "summary": "This paper proposes a model for motion forecasting that models the continuous stream of world state over sequential timesteps.  This is in contrast to most/all other works which do an indepedent prediction of the future for each timestep based on a fixed window of history, without context of previous model belief states or outputs from previous time steps.  One stream of data is scene context, represented as embeddings, where context is fused with current accounting for transformation of the scene due to ego motion.  The second stream is agent trajectories, where a history of n previous predicted trajectories for each agent is maintained explicitly in a memory bank, and fused into the current time steps prediction as additional signal.\n\nThey do thorough ablations on Argoverse benchmark dataset, showing compelling evidence that the streaming architecture is beneficial.  Furthermore, they exceed SOTA on most metrics in the benchmark. When I checked today (mid July), they are ranked 7th overall in the leaderboard - https://eval.ai/web/challenges/challenge-page/1719/leaderboard/4098", "questions": "See weaknesses", "limitations": "None", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "This paper addresses an important problem often ignored in these sanitized datasets: that these models in real world  application in an AV system run in a sequential streaming fashion, and (to my knowledge) all other works operate on each time frame independently (given a fixed window of historical state).  \n\nThe proposed solution itself is a good one, transforming old belief state into current reference frames in both embedding space and with spatial transformations where appropriate.  At the same time, the complexity of the architecture is kept reasonably low, with each contribution justified. \n\nResults are very strong and well justified with thorough ablations.", "weaknesses": "Minor, but I had some difficulty understanding some of the explanations:\n\n - Data reorganization / Figure 2: \"we select several split points Ti along historical frame steps\" --> how are they selected? what are \"historical frame steps\"? Can you more simply say this as, for example, \"We split sequences evenly into sub-sequences\"?\n\n - Around L158: you started using the term \"modes\" for the first time, where I think you should just say trajectories, or be more careful defining the tensors F_{mo} and Y_{mo} making clear their dimensionality.  \n\n - You never clearly state the output representation of your model (which I assumed to be a weighted trajectory set). Without being more clear here, Section 3.3. Model training is sort of meaningless - for example, what is the classification loss classifying? (I'm 99% sure it's a softmax distribution over trajectory modes, but this is not said anywhere.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review as well as the suggestions for improvement. Our response to the reviewer’s comments is below:\n\nApologies for these unclear explanations.\n\n+ **Data reorganization:**\nAs shown in Fig. 2(b), the dashed part refers to historical time steps (50 historical steps in AV2), and we select several split points on this part. We consider an original trajectory (the gray arrow) as a sequence, and our processing approach generates several shorter trajectories (sub-sequences). Besides, the split points are manually specified in our method and discussed in ablation Tab.4(b).\n\n+ **Mode:**\nAll correct! Multiple trajectories are predicted for each agent to enable comprehensive motion possibilities, in which case $mode$ is commonly used to represent each prediction. We will make it clear. Besides, the dimension of $Y_{mo}$ is $[B, N_a, N_{mode}, 2]$.\n\n+ **Model output:**\nThe output is several predicted trajectories with corresponding probabilities, as the reviewer understood. While providing the types of loss functions in Line 177, we indeed overlooked the concrete input items. We will clarify this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We provide additional figures in the PDF files for more intuitive demonstration.

Meta-review: The paper received unanimous acceptance recommendations, with reviewers praising the novelty and importance of the problem studied, as well as the strong experimental results of the proposed approach. However, serious concerns were raised regarding overclaiming in the paper's positioning and the omission of related work. In their rebuttal and subsequent discussion, the authors committed to comprehensively updating the abstract and introduction to accurately reflect the paper’s contributions—specifically, by framing the contribution as "following a streaming paradigm and designing more domain-appropriate fusion mechanisms across time than the traditional vanilla MLP or transformer operations," as suggested by Reviewer fgWb. Additionally, the authors agreed to discuss and include the missing related works in the experimental evaluation.

After considering the paper, reviews, rebuttal, and further discussion, the area chair decided to accept the paper, conditional on the revisions detailed above. The area chair will review the camera-ready version to ensure that these conditions have been met.
Decision: Accept (spotlight)