Motion Forecasting in Continuous Driving

Authors: nan  song
bozhou  zhang
xiatian  zhu
li  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 12:36:51

Abstract: Motion forecasting for agents in autonomous driving is highly challenging due to the numerous possibilities for each agent's next action and their complex interactions in space and time. 
In real applications, motion forecasting takes place repeatedly and continuously as the self-driving car moves. However, existing forecasting methods typically process each driving scene within a certain range independently, totally ignoring the situational and contextual relationships between successive driving scenes. This significantly simplifies the forecasting task, making the solutions suboptimal and inefficient to use in practice. To address this fundamental limitation, we propose a novel motion forecasting framework for continuous driving, named RealMotion.
It comprises two integral streams both at the scene level:
(1) The scene context stream progressively accumulates historical scene information until the present moment, capturing temporal interactive relationships among scene elements.
(2) The agent trajectory stream optimizes current forecasting by sequentially relaying past predictions.
Besides, a data reorganization strategy is introduced to narrow the gap between existing benchmarks and real-world applications, consistent with our network. These approaches enable exploiting more broadly the situational and progressive insights of dynamic motion across space and time. 
Extensive experiments on Argoverse series with different settings demonstrate that our RealMotion achieves state-of-the-art performance, along with the advantage of efficient real-world inference.

Main paper: 
 introduction: 
 Motion forecasting is a crucial element in contemporary autonomous driving systems, enabling self-driving vehicles to predict the movement patterns of surrounding agents [43, 17] . This prediction is vital for ensuring the safety and reliability of driving. However, numerous complex factors, including stochastic road conditions and the diverse motion modes of traffic participants, make resolving this task challenging. Recent developments have focused on the study of representation and modeling [10, 52, 51] , in tandem with a growing emphasis on precise trajectory predictions [6, 32, 49, 15, 50, 35] . Furthermore, the field has witnessed an increased focus on multi-agent forecasting, a more challenging yet valuable subtask [26, 1, 14, 31] . These advancements have collectively contributed to substantial progress in motion forecasting in recent years.However, we realize that existing methods tackle motion forecasting tasks in an isolated manner, i.e., they treat every individual driving scene within a limited range independently, overlooking that in reality motion forecasting is inherently temporally interrelated while any ego-car drives on. That means previous methods ignore the driving context across successive scenes, as well as the corresponding potentially useful information from previous driving periods (Fig. 1 ). Under the above insight and consideration, we propose an efficient in-context motion forecasting framework for continuous driving, named RealMotion. It comprises two streams to transit states of scenes: (1) A scene context stream that accumulates historical scene context progressively, capturing temporal interactions among scene elements and addressing complex driving situations. (2) An agent trajectory stream that continually optimizes the predictions for dynamic agents like vehicles, considering temporal consistency constraints and capturing precise motion intention. Each stream utilizes a specially designed cross-attention mechanism to transit scene states and fulfill its function.Our contributions are summarized as follows: (i) We solve the motion forecasting problem from a perspective of the real-world applications, which enables the extraction and utilization of valuable situational and progressive knowledge. (ii) We introduce RealMotion, a novel motion forecasting approach that sequentially leverages both scene context and predicted agent motion status over time, meanwhile maintaining a lower real-world inference latency. (iii) To support the continuous driving setting on existing benchmarks, we implement a data reorganization strategy to generate scene sequences, closely simulating the real-world driving scenarios. Extensive experiments on Argoverse series with different settings demonstrate that RealMotion achieves state-of-the-art performance. 
related work: 
 In autonomous driving, accurately predicting future trajectories of agents of interest relies on an appropriate representation of scene elements. Early methods [29, 12, 2] rasterize driving scenarios into images and utilize off-the-shelf convolutional networks for scene context encoding. However, due to their limited ability to capture intricate structural information, recent studies [49, 15, 52, 36] have shifted towards vectorized representations, as exemplified by the emergence of VectorNet [10] . Additionally, graph-based structures are widely adopted to model dynamics, interactions, and relationships among agents and maps [22, 13, 44, 19, 18, 9] .With the encoded scene features, various approaches are explored for estimating multi-modal future trajectories. Early methods focus on goal-based prediction [49, 15] or use probability distribution heatmaps for trajectory sampling [12, 13] . Recent approaches like HDGT [19] , Wayformer [25] , and others [23, 26, 48, 32, 51] leverage Transformer architectures [37] to model detailed relationships within the overall scene. Moreover, there are methods introducing novel paradigms (e.g. pretraining [5, 4, 21, 28] , post-refinement [6, 50] or Diffusion [20] ) to achieve impressive performance.To address the relevance of predicted trajectories for different agents in real-life scenarios, recent efforts have focused on multi-agent forecasting. Some methods [14, 38, 52, 33] adopt an agentcentric design, iteratively predicting trajectories for each agent, which can be inefficient and hinder exploration of relationships among agents. Conversely, SceneTransformer [26] introduces a scenecentric framework for joint forecasting of all agents, presenting a novel design. Moreover, recent methods explore a query-centric design [51] or consider adaptations [1] to address this task.Nevertheless, the methods mentioned above independently perform forecasting for each scene sample, which conflicts with practical settings where driving scenarios are interconnected over time. To overcome this limitation, pioneering work [27] first introduce a temporal motion benchmark based 3 MethodologyQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: see Conclusion in 5Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary: 
 Data reorganization. Considering the discrepancy between existing benchmarks and practical applications, our first step is to reorganize these datasets by transforming each sample scene into a continuous sequence, mimicking the successive real-world driving scenarios. Specifically, we retrospectively examine each independent scene by evenly splitting agent trajectories into shorter segments and sampling local map elements (refer to Fig. 2 ). Specifically, we first select several split points T i along historical frame steps. Next, we generate trajectory segments of identical length by extending from these points both into the past and the future. The number of historical and future steps is determined by the minimum split point and the length of ground-truth trajectory, respectively. Also, surrounding agents within a certain range and a local map are aggregated for interested agents at each split point, forming a sequence of sub-scenes. This reorganization allows freely capitalizing on the original elements to offer valuable situational and progressive insights at the scene level for model optimization. Hence, existing methods can also involve and benefit from the novel data structure. We have implemented this approach within the state-of-the-art method QCNet [51] , which is further discussed in the appendix, highlighting the generality of our data structure.Input representation. In the context of motion forecasting, the trajectories of agents and a highdefinition road map are provided in the driving scenario. Following [10] , we transform these scene elements into vectorized representations as input. The historical trajectories are denoted as A ∈ R Na×T ×Ca , where N a , T , and C a represent the number of agents, the number of historical frames, and the motion states (e.g., position, angle, velocity, and acceleration), respectively. The road map is divided into several lane segments, denoted as M ∈ R Nm×P ×Cm , where N m , P , and C m indicate the number of lane segments, the points of each segment, and the lane features (typically represented as coordinates). All these states are normalized relative to the current position of the agent of interest.As depicted in Fig. 3 , our RealMotion approach comprises an encoder, a decoder, a scene context stream, and an agent trajectory stream. Following the encoder-decoder structure, the two streams are designed to execute temporal modeling, focusing on context information and trajectory prediction along the time dimension. For scalability and simplicity, we adopt the plain encoder and decoder design following [5, 32] . Specifically, we encode the map features F m ∈ R Nm×D by a PointNet-like encoder [30] , where D refers to the embedding dimension. As [5] , we extract the agent features F a ∈ R Na×D by stacked neighborhood attention blocks [16] . Given the agent and map features, we concatenate them to derive the scene features F s ∈ R (Na+Nm)×D . We then employ a Transformer [37] encoder to learn the interrelationships of these features.For trajectory prediction along with the probability, we utilize two Multilayer Perceptron (MLP) modules. Additionally, we forecast a singular trajectory for auxiliary training purposes, focusing on the movement patterns of other agents.The scene context stream is meticulously designed to progressively gather information about the surrounding environment, thereby enhancing trajectory prediction. Positioned after the encoder, this component plays a crucial role as the historical scene profoundly influences the understanding of temporal motion behaviors exhibited by agents. For instance, the ability to estimate the trajectory in complex scenes or evaluate currently occluded agents can be greatly improved by taking into account the previous situation and context.At the current time t, we denote the current and historical scene features as F t s and F t-1 s , respectively, extracted from the respective local coordinate systems. The process begins by projecting F t-1 s onto the current system. To achieve this, the two features must be aligned, considering the gap of motion between them. Motion-aware Layer Normalization (MLN) [40] is employed for this purpose:EQUATIONwhere ∆θ denotes the heading angles, ∆t refers to the timestamps, and (∆x, ∆y) represents the difference between their positions. PE indicates the position encoding function. Subsequently, we repartition the scene features F t-1 s and F t s into the agent and map parts for following process. To incorporate the historical information into the current, we employ map-map and agent-scene cross-attention modules with Multi-Head Attention (MHA) blocks for map and agent features, respectively:EQUATIONwhere map features only interact with each other across time to enrich map elements, while agent features aggregate the overall historical scene context for comprehensive scene understanding.Given the sequential nature, this past context-referenced feature will be further propagated to future timestamps. Simultaneously, it serves as the input to the decoder for trajectory prediction at the current timestamp.In addition to referencing scene context, we enhance trajectory forecasting by establishing temporal relationships to achieve further improvement. This involves leveraging the inherent temporal continuity and smoothing nature of trajectories. This is accomplished through the agent trajectory stream, which is equipped with a memory bank for storing historical trajectories, enabling temporal relaying.To maintain a set of n historical trajectories for each agent of interest, we design the trajectory memory bank as M(a) = {(y 1 , f 1 ), (y 2 , f 2 ), ..., (y n , f n )}, where y i denotes predicted trajectories projected onto the global coordinate system, and f i represents corresponding mode features recording historical motion information. Unlike the detection or tracking tasks, where position changes of objects typically remain relatively consistent, the future motion patterns (under local system) involves significant changes due to variations in road conditions. Therefore, directly decoding with historical memory query features is inappropriate. Considering that the simple decoder can provide satisfactory predictions in practice, modifying initial predictions using the memory bank proves more effective.Before refinement, we align the saved trajectories and mode features with the current coordinate system. While handling features consistently with the above, it is crucial to align trajectories in Euclidean space for more precise comparisons. Concerning the trajectory y i ∈ R K×2 (with K being the frame steps of the trajectories), we compute the transformed trajectory ỹi asEQUATIONHere, R(θ) is the rotation matrix for the current heading angle, and y ori i denotes the trajectory origin chosen based on the time difference ∆t and sampling frequency q. Recognizing that memory trajectories may originate from different historical times, their origins also differ.After transformation, trajectories whose latter part resembles the former part of current predictions contribute more to the refinement process. To aggregate memory information accordingly, we update the current mode features with a lightweight Transformer Decoder utilizing Trajectory Embedding (TE) to measure spatial similarity and replace the original positional embedding. Then, we further propagate updated modes into a MLP module to generate offsets and update initial predictions. This procedure is defined as follows:EQUATIONwhere F mo and Y mo are mode features and initial predicted trajectories in the current context, Fb and Ỹb represent aligned features and trajectories retrieved from the memory bank. Besides, we employ a single layer MLP to embed the flattened trajectories as the TE.Ultimately, we save the refined trajectories (projected onto the global system) and features while removing outdated ones (first in first out). This updated trajectory memory will be further passed down for future timestamps. Importantly, due to the generation of only a small number of motion modes, we do not apply any filtering to ensure the multimodality and diversity of the bank.During the training process, we supervise the estimated trajectories using the regression loss L reg and the associated probabilities through the classification loss L cls . Additionally, we introduce the EQUATIONFor L reg and L refine , we employ the smooth-L1 loss, while the cross-entropy loss is utilized for L cls .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: It is our main idea designed for real-world autonomous driving applications. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to Answer: [NA] Justification: Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
experimental settings: 
 Datasets and metrics We assess the performance of our method using the Argoverse 1 [3] and Argoverse 2 [43] motion forecasting datasets in both single-agent and multi-agent settings. The Argoverse 1 dataset comprises 323,557 sequences from Miami and Pittsburgh, while the Argoverse 2 dataset contains 250,000 scenes spanning six cities. In the Argoverse 1 dataset, predictors are tasked with forecasting 3 seconds of future trajectories for agents based on 2 seconds of historical observations. In contrast, the Argoverse 2 dataset offers improved data diversity, higher data quality, a larger observation window of 5 seconds, and an extended prediction horizon of 6 seconds. Additionally, both datasets have a sampling frequency of 10 Hz.We employ standard benchmark metrics, encompassing minimum Average Displacement Error (minADE k ), minimum Final Displacement Error (minF DE k ), Miss Rate (M R k ), and brier minimum Final Displacement Error (b -minF DE k ), across six prediction modes designed for the single-agent setting. Further details can be found in the appendix.Implementation details We train our models using the AdamW [24] Optimizer with a batch size of 32 per GPU for 60 epochs. Our model is trained end-to-end with a learning rate of 0.001 and a weight decay of 0.01. The latent feature dimension is set to 128. Following [5, 32] , we consider only agents and lane segments within a 150-meter radius of the focal agent. For the samples in Argoverse 2, we split the whole scene into 3 segments, each with a historical observation window of 3s and a same prediction horizon of 6s as the original. As for the Argoverse 1, we set the historical window of 1s. To fully utilize historical information, we compute gradients and perform back propagation for all segments. Moreover, the RealMotion-I is trained and evaluated with a common configuration without dataset reorganization and stream modules. [15] 0.73 1.05 0.10 mmTransformer [23] 0.71 1.15 0.11 LaneGCN [22] 0.71 1.08 -PAGA [8] 0.69 1.02 -DSP [46] 0.69 0.98 0.09 ADAPT [1] 0.67 0.95 0.08 HiVT [52] 0.66 0.96 0.09 R-Pred [6] 0.66 0.95 0.09 HPNet [35] 0.64 0.87 0.07 RealMotion 0.61 0.91 0.07 We first compare the performance of our RealMotion with several top-ranked models on the Argoverse 2 [43] motion forecasting benchmark. The results on the test split are presented in Tab. 1.RealMotion has far outperformed most of previous approaches. Concretely, our method stands distinctly ahead of other methods in terms of minF DE 1 and minADE 1 , showing performance enhancements of 8.60% and 5.91% relative to QCNet, respectively. We also get the almost top 2 place for other metrics. Compared to our independent variant RealMotion-I, the proposed method exhibits significant performance improvements across all metrics, which conclusively demonstrates the effectiveness of our designs. Then, we compare the performance of RealMotion on the Argoverse 1 benchmark, with the results of the validation split presented in Tab. 2. It is shown that our method also achieves a decent performance, especially for minADE 6 . We also provide our ensemble and multi-agent results in Appendix B.In the multi-agent setting, predictors are required to jointly estimate the future trajectories of all interested agents, which is crucial for the comprehensive perception of the driving scenario. Therefore, we also evaluate our RealMotion on the Argoverse 2 Multi-agent test set to prove the effectiveness, and provide simple results as shown in Tab. 3. Although not integrated with specialized designs for multi-agent forecasting like [26, 1] , our model also demonstrates superior performance compared to recent works owing to our sequential techniques.We conduct ablation studies on the Argoverse 2 validation split for the single-agent setting to examine the effectiveness of each component in RealMotion. We adopt the default experiment settings following Sec. 4.1 to perform ablation in this section.As shown in Tab. 4, we assess the effectiveness of each component in our network. The first row (ID-1) represents the independent framework RealMotion-I, which is the same as reported in Tab. 1. First, our data reorganization approach extends dataset at no extra cost and enables the exhaustive utilization of temporally continuous motion, resulting in a noticeable improvement as shown in the second row. Then, our proposed two streams can better learn temporal relationships at the scene level, thereby both bringing additional improvements, represented as ID-3 and ID-4, respectively. Considering that these two streams are complementary to each other, therefore, RealMotion that involves all these sequential techniques achieves remarkable performance gains, as indicated in the final row. Additionally, in contrast to the consistent improvements observed with the data processing approach, it is worth noting that our two streams demonstrate more significant advantages in single-mode metrics compared to the six-mode metrics. As shown in Fig. 4 , we attribute this phenomenon to the enhanced capability of our streams to modify less accurate trajectories.Effects of feature alignment and TE. The misalignment of features might have an adverse effect on the performance when implementing feature interaction across scenes. Hence, we evaluate the impact of the alignment modules in both the Context Referencing and the Trajectory Relaying blocks in Tab. 5(a). As depicted in the first four rows, the removal of the alignment modules clearly results in a performance decline. By incorporating these modules, We have, to some extent, alleviated the negative impact of misalignment. However, executing it twice only yields marginal gains, which might be caused by the function overlap. Besides, we also analyze the performance of our proposed Trajectory Embedding in the 4th and 5th rows. The performance gains indicate that the Trajectory Embedding can facilitate the selection of similar historical trajectories from the memory bank, thereby easily constraining and refining current predictions. Despite the simplicity in the design of these modules, they both contribute to additional benefits for our network.Effects of split points and gradient steps. In Tab. 5(b), we investigate the performance variations with respect to the number of gradient steps taken and the split points along historical steps. From the 1st to the 3rd row, as we progressively increase the number of steps used for gradient computation, there is a noticeable improvement in performance. This enhancement can be attributed to the fact that computing the gradient for specific steps allows us to better capture the trajectory distribution for model training. From the 3rd to the 5th row, we change the interval of split points from 5 frames to 15 frames. Accordingly, the length of historical trajectory also changes, ranging from 40 frames to 20 frames. As observed, similar trajectories occur in the sequence when using a short interval, which can significantly hinder the model to learn distinct motion patterns. Conversely, using a longer interval results in fewer historical trajectory frames available in each segment, which contradicts the optimization of one-shot forecasting. It is evident that our choices for the gradient steps and the split points are well-suited for the Argoverse 2 benchmark. Moreover, we attempt to divide the trajectory into 5 segments in the final row. An excessively long trajectory sequence imposes a learning burden on our framework, preventing it from focusing on temporal relationships and yielding limited improvements.Effects of the depth of cross-attention blocks. As shown in Tab. 7, we explore the influence of depth variations of cross-attention unit in the Context Referencing and the Trajectory Relaying blocks. Primarily our temporal blocks are lightweight regardless of depth to ensure the efficiency and universality. a relatively deep cross-attention unit in the Context Referencing and the Trajectory Relaying blocks is necessary for processing history information and current information. We use a depth of 2 as our default setting considering its better efficiency-performance balance.Balancing performance, inference speed, and model size is important for the model deployment. We compare our RealMotion with recent representative works, which include the real-time forecasting approach HPTR [48] and the state-of-the-art approach QCNet [51] . We measure these approaches and RealMotion on the Argoverse 2 test set using an NVIDIA GeForce RTX 3090 GPU, maintaining a batch size of 1 and following an end-to-end manner. As shown in Tab. 6, RealMotion has the competitive inference time and a competitive model size while achieving the best performance. It is worth noting that "online" indicates the latency for practical autonomous driving systems, which is optimized compared to the "offline" version by some efficient designs (e.g. the caching technique in [48] ). As for RealMotion, we must consider three times latency for "offline" dataset, where only the final prediction is utilized for evaluation. In contrast, the forecasting results of each iteration is meaningful in "online" application.In Fig. 4 , we present qualitative results of our network compared to the independent version on the Argoverse 2 validation set. Panels (a), (b), and (c) illustrate the forecasting results at 3s, 4s and 5s, respectively. Panel (c) displays the final results used for evaluation, which are more accurate and closer to the ground truth. By comparing panel (c) and (d), it can be seen that RealMotion remarkably outperforms the independent version. As demonstrated in the panels from (a) to (c), our RealMotion can progressively refine the estimated trajectories from coarse to fine as the motion progresses and accurately capture the possible motion intention. However, the one-shot forecasting of RealMotion-I shown in the panel (d) leads to significant estimation errors. 
conclusion: 
 In this work, we anticipate to address the motion forecasting task from a more practical continuous driving perspective. This in essence places the motion forecasting function in a wider scene context compared to the previous setting. We further present RealMotion, a generic framework designed particularly for supporting the successive forecasting actions over space and time. The critical components of our framework are the scene context stream and the agent trajectory stream, both of which function in a sequential manner and progressively capture the temporal relationships. Our extensive experiments under several settings comprehensively demonstrate that RealMotion surpasses the current state-of-the-art performance, thereby offering a promising direction for safe and reliable motion forecasting in the rapidly evolving field of autonomous driving.Limitations. A clear constraint of our data processing approach is its requirement for a sufficient number of historical frames for serialization. Consequently, it is not applicable to short-term benchmarks such as the Waymo Open Dataset, which provides only 10 frames of historical trajectory. Moreover, existing datasets typically provide limited historical information distinct from real-world settings, which inhibits our sequential designs from fully leveraging their advantages. Hence, we anticipate to integrate our framework into a sequential autonomous driving system to maximize the benefits of streaming designs in our future work. For generality test, we integrate our RealMotion data with the state-of-the-art method QCNet [51] . Due to the big size of QCNet, the scene context and agent trajectory streams have to be excluded for memory constraint. With minimal alterations applied to this prior model, we observe a noticeable improvement. The results are shown in Tab. 9.  
 
 Appendix: We provide more qualitative results of our framework in Fig. 6 and Fig. 7 on the Argoverse 2 validation set.Although our RealMotion has achieved outstanding performance on motion forecasting benchmarks, it still has some failure cases. We analyze these typical cases and provide qualitative results to help readers understand the circumstances under which our model may fail. This analysis will aid future work in developing a more powerful and robust algorithm, as shown in Fig. 5 .In the first row of Fig. 5 , the agent requires to navigate through a complex intersection to one of the roads, but the model fails to predict this possible driving behavior and just anticipates the agent to drive straight ahead. This may be caused by the lack of a comprehensive understanding of the complex map topology and the unbalanced distribution of driving data. In most scenarios, agents tend to exhibit only trivial behaviors, such as moving straight ahead at a nearly constant velocity. Consequently, this raises issues regarding data balance, and the figures indicate that our model is more likely to make mistakes when it comes to turning.In the second row of Fig. 5 , the vehicle is expected to park on the side of the road, which is a kind of subjective driving behaviors. However, the model only predicts that the vehicle will keep going ahead. To improve the forecasting of such cases, we could enhance the interaction between the model with additional intentions of human and incorporate more information, such as visual cues like turn signals and parking spaces. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper received unanimous acceptance recommendations, with reviewers praising the novelty and importance of the problem studied, as well as the strong experimental results of the proposed approach. However, serious concerns were raised regarding overclaiming in the paper's positioning and the omission of related work. In their rebuttal and subsequent discussion, the authors committed to comprehensively updating the abstract and introduction to accurately reflect the paper’s contributions—specifically, by framing the contribution as "following a streaming paradigm and designing more domain-appropriate fusion mechanisms across time than the traditional vanilla MLP or transformer operations," as suggested by Reviewer fgWb. Additionally, the authors agreed to discuss and include the missing related works in the experimental evaluation.

After considering the paper, reviews, rebuttal, and further discussion, the area chair decided to accept the paper, conditional on the revisions detailed above. The area chair will review the camera-ready version to ensure that these conditions have been met.

Reviews: review: paper_summary: paper_summary: The paper introduces an approach to iteratively process temporal scene context for the purposes of agent motion prediction. This differs from traditional approaches that ingest the whole context directly. 

The scene is processed in temporal chunks, where in each chunk contains agent-centric context consisting of other agents and map within 150m. Information is aggregated across chunks (i.e. over time) using two novel components: 
- "Scene context stream", where agent/map features from the past frame are transformed using Motion-aware Layer Normalization [40] and cross-attended by current frame features. 
- "Agent context stream", where agent predictions from the current frame are modified by attending predictions from past frames (stored in a memory bank and properly aligned). 

This 'streaming-type' method obtains very strong results on the Argoverse 1 and 2 datasets, and is shown to improve two different baseline  model architectures based on [5] and [51] that consume the whole historical context in a single step. Some helpful ablations and latency studies of the proposed components is provided.
strength_weakness: strength_weakness: - In the abstract/intro, I find some of the framing to be a bit misleading. Examples: 
  - "7:This significantly simplifies the forecasting task, making the solutions unrealistic to use in practice". I do not find this claim to be well supported. The current methods still ingest a significant amount of relevant context. To me the contribution of the paper is to follow a streaming paradigm and design more domain-appropriate fusion mechanisms across time than the vanilla MLP or transformer ops that are used traditionally. 
  - "31 - 36: existing methods all tackle an artificially narrowed and unrealistic motion forecasting task".... The methods still ingest 2-3 seconds of context, but perhaps not as effectively as the proposed work. That does not confirm this overly strong claim however. Also the paper only demonstrates good performance with up to 3s of context, that does not prove yet that much longer context is helpful or practical. 
  - To a similar point, Fig 1 is also a bit misleading: current methods would directly ingest the available scene history in the current datasets, and fuse the information, even if the information fusion is not done with the same geometric and streaming intuitions. 

- In the method description, some details are unclear or could be explained better:
  - 83: "generate trajectory segments of identical length" Length can be misconstrued to be the length of the trajectories in meters, not that they are the same across chunks. This whole paragraph could be rewritten for clarity and a more helpful diagram could be shown -- illustrating what exactly is done for Argoverse 2 (one now needs to read all the way to Sec 4 to understand how this works). 
  - "114: Additionally, we forecast a singular trajectory for auxiliary training purposes, focusing on the movement patterns of other agents. " 
Unclear what this means exactly. Do you produce only a single trajectory prediction in past steps?
  - "“Equal length into past and future” → confusing, assume fixed time interval in both directions. 
  - The L_refine loss in Eq 5 is not explained, unclear what is being refined. My guess is that this is predictions in earlier chunks
  - What is exactly RealMotion without the context stream components? This is present both in the Ablation and in Appendix B.3. How do you fuse information across time then? 

- In the experimental results, some SOTA methods from the leaderboard seem to omit SEPT (ICLR 2023), which has higher numbers than RealMotion, even though it is cited. Why did you you omit it? 

- A couple more ablations or experiments would be helpful. 
  - It seems that the method is similar to a filter, where the current chunk only attends to previous chunk (but not to the ones before). Is my reading of the paper correct? If so, an experiment where the method can attend longer context memory with multiple frames would be interesting. 
  - It would also be interesting to try pretraining this method on more data examples (those can be produced by predicting shorter futures, then you can use more of the segments). And then fine-tune on 6s futures.
questions: questions: Main question is, do you agree with my reading that "the contribution of the paper is to follow a streaming paradigm and design more domain-appropriate fusion mechanisms across time than the vanilla MLP or transformer ops that are used traditionally. " As opposed to the strong claims in the current abstract / intro. 

Why did you omit SEPT from the leaderboard results (Table 1)? 

Please see my clarification or ablation/more experiment comments in the Weaknesses. I am particularly interested to understand why you did not try to attend to longer temporal context in the last chunk (if I read the paper correctly).
limitations: limitations: The same RealMotion methods tried here could be tried on the Waymo Open Dataset which also has 9 second segments, but one could take the first 3 seconds as history and predict 6 seconds, as opposed to the standard 1s / 8s split that is mentioned in the limitations.
paper_summary: paper_summary: This paper proposes a framework named "RealMotion" highlighting the importance of continuous motion forecasting in autonomous driving. From the formulation perspective, RealMotion investigates predicting trajectories in a continuous sequence of timestamps instead of previous independent predictions. From the methodology perspective, RealMotion proposes the "relaying" modules that propagate historical scene and agent contexts into the current frame to enhance prediction. Finally, RealMotion achieves top performance on Argoverse benchmarks and proves the importance of continuous contexts.
strength_weakness: strength_weakness: 1. The authors have missed some closely related works, such as streaming motion forecasting (e.g., Pang et al.) and other end-to-end autonomous driving (e.g., Gu et al.), where motion forecasting is continuous. For instance, Pang et al. also discover the setbacks of predicting trajectories on independent timestamps, formulate the "streaming" forecasting task, and propose a differentiable filter to refine the predicted trajectories. I acknowledge the newly designed modules by RealMotion, but I would expect the authors to have a better discussion of these relevant efforts.

Pang et al. Streaming Motion Forecasting for Autonomous Driving. IROS 2023.
Gu et al. ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries. CVPR 2023.

2. RealMotion is also closely related to QCNet and based on its code. QCNet already has the interface of continuous motion forecasting and leverages a GRU to deal with the temporal contexts in queries. First of all, I would like the authors to clarify or discuss in the paper about the difference/improvement to QCNet more explicitly. In addition, I am also curious if it is possible to support that the "relaying" modules in RealMotion is better than QCNet's end-to-end mechanism. For instance, is it possible to plug the relaying module to QCNet and make a comparison?

3. I would like the authors to develop some metrics to analyze the improvement in the trajectories. More specifically, we wish to analyze "where does RealMotion improve the trajectory quality?" Since the main argument of this paper is "continuity," I am curious if the author can use a "fluctuation" measure like Pang et al. to prove that the trajectory smoothness is indeed improved. It is also encouraged to show some visual comparisons if space is allowed.
questions: questions: See weakness above.
limitations: limitations: The authors have discussed limitations. No additional limitations from my side.
paper_summary: paper_summary: This submission tackles the task of trajectory forecasting in autonomous driving. 
In particular, it proposes improvements in two aspects:

1/ Data reorganization: Current datasets are artificially split into non-overlapping segments. The authors propose to reorganize the data to have overlapping windows.

2/ Relaying context and predictions: since the sub-scenes now overlap, they can use context and predictions from the previous timesteps (corresponding to a different sub-scene) to improve predictions at the current timestep.
Relaying context serves to provide more information about the surroundings (an agent could have been missed in the current frame for instance), and relaying the predictions would smooth out the predictions across time.

They demonstrate the validity of their approach on Argoverse benchmarks, achieving state-of-the-art performances.
strength_weakness: strength_weakness: - The results are not very conclusive:
  - Compared to  QCNet, except for $ADE_1$/$FDE_1$ (which are not the main metrics usually considered for motion forecasting) the improvements are small. It would be useful to give more insight into why only these metrics improve. We also don't have the $ADE_1$/$FDE_1$ values of QCNet + RealMotion so we don't know if these findings are consistent across models.
  - The comparisons are not fair in terms of available information. Because of the propagation, RealMotion has access to more past information both in the past trajectories and agents, and the past map. A naïve way to integrate more past trajectories would be to just use longer past history. This should be a minimal baseline or ablation in the paper. If we also wanted to add agents seen in the past but not at the current timestep, it could be done with some masking or extrapolation scheme for missing frames. 
Remains the past map information, for which it's indeed more complicated to have a naïve baseline for a fair comparison, as simple aggregation could drastically raise the computation footprint so I can understand why the authors wouldn't want to test this baseline.
- The loss functions, and in particular $\mathcal{L}_{refine}$ are not detailed enough. I can guess where they apply but it would be good if that was stated explicitly.
- Figure 2 is unclear to me. Perhaps it could need more captions or a rework.
- I think it would be easier to interpret the results if the architecture of RealMotion would be more clearly delivered, perhaps in a Figure  (in Appendix?) that represent the full architecture detailing in particular how the different elements interact in the encoder and the decoder that are not given in Figure 3.
questions: questions: - Why do we see drastic improvement on ADE/FDE_1 but the other improvements are very tame? Does that imply most of the improvements are to the scoring function so that RealMotion gets a better top 1? If that's the case, what could be the mechanisms that explain this behavior?

Otherwise, I think this paper rightfully questions some practices of motion forecasting in autonomous driving. Generalizing something along the lines of the proposed data reorganization would be very helpful for the community. However, I think the experiments on relaying might not be fair enough and need to be compared to simple baselines with more/better past history. We would also need a few more numbers to see if most of the findings and recommendations have a good chance at being consistent accross models.
limitations: limitations: - Limitations are discussed in the Appendix.

- Broader societal impacts do not seem explicitly addressed despite what has been reported in the checklist. I'm not sure there are obvious uncontroversially negative societal impacts to be discussed though.
paper_summary: paper_summary: This paper proposes a model for motion forecasting that models the continuous stream of world state over sequential timesteps.  This is in contrast to most/all other works which do an indepedent prediction of the future for each timestep based on a fixed window of history, without context of previous model belief states or outputs from previous time steps.  One stream of data is scene context, represented as embeddings, where context is fused with current accounting for transformation of the scene due to ego motion.  The second stream is agent trajectories, where a history of n previous predicted trajectories for each agent is maintained explicitly in a memory bank, and fused into the current time steps prediction as additional signal.

They do thorough ablations on Argoverse benchmark dataset, showing compelling evidence that the streaming architecture is beneficial.  Furthermore, they exceed SOTA on most metrics in the benchmark. When I checked today (mid July), they are ranked 7th overall in the leaderboard - https://eval.ai/web/challenges/challenge-page/1719/leaderboard/4098
strength_weakness: strength_weakness: Minor, but I had some difficulty understanding some of the explanations:

 - Data reorganization / Figure 2: "we select several split points Ti along historical frame steps" --> how are they selected? what are "historical frame steps"? Can you more simply say this as, for example, "We split sequences evenly into sub-sequences"?

 - Around L158: you started using the term "modes" for the first time, where I think you should just say trajectories, or be more careful defining the tensors F_{mo} and Y_{mo} making clear their dimensionality.  

 - You never clearly state the output representation of your model (which I assumed to be a weighted trajectory set). Without being more clear here, Section 3.3. Model training is sort of meaningless - for example, what is the classification loss classifying? (I'm 99% sure it's a softmax distribution over trajectory modes, but this is not said anywhere.)
questions: questions: See weaknesses
limitations: limitations: None

Comments: comment: Thanks. We will make sure to update abstract and introduction section to accurately reflect the contributions, report SEPT results, and thoroughly discuss the similarities and differences between missing related work and ours.
I thank the reviewer for providing a thorough feedback and fully support the importance of appropriate positioning for this work. For a positive decision on the submission the authors will have to explicitly commit to comprehensively updating the abstract and introduction of the paper to accurately reflect the contributions (not just changing a few words here and there), as well as reporting SEPT results in the main paper (with a disclaimer that the setting are not fully comparable) and thoroughly discussing the missing related work (not just adding citations).

Reviewer fgWb, please comment if the title of the paper is adequate or requires updating as well.
We appreciate the reviewer's time for reviewing and thanks for the recognition.
Indeed, after going again through the submission with the new information from the rebuttal and the reviews, I find my concerns has been adressed properly in the rebuttal.
Moreover, I believe the reasons for my initial wrong impression have also been mostly adressed thanks to the discussions between the authors and Reviewer fgWb. Since the authors have commited to make the necessary change, I'm happy to increase my rating.
We appreciate the reviewer's time for reviewing and thanks again for the valuable comments. We will revise and refine the paper as suggested in the revision.
Thank you for all the clarifications. Overall I am leaning to accept, as the ideas and results generally warrant it, and will maintain my score.

In doing this, I assume you will temper claims that existing solutions are "unrealistic to use in practice" even if you seem to be able to improve on them, and will explicitly call out the fact that you have not actually validated benefits from being able to process longer histories, per se, which seems left for future work.
Dear Reviewer GAo1

Thanks again for the valuable comments. We believe our responses addressed all the questions/concerns. It would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you!

Best wishes,

Authors
> please explain what your updated intro contribution statement relative to existing methods is, in 2 sentences" to see how it would address my overclaiming concerns.

Thanks. Existing methods tackle motion forecasting task in a scene-independent manner, which could limit the model performance in real-world settings where motion is typically continuous while ego-car drives on. Upon this observation, we propose temporal relationship modeling across scenes and introduce an effective approach, RealMotion, to realize this idea in a two stream framework.

> I suggest at the very least you augment the figure caption to clarify that each blue scene circle contains X seconds of past history, or some such.

Thanks for the suggestion and we will revise as suggested.
> We cannot provide the evaluation of longer history due to there is no temporal relationships between different scenes in existing benchmarks. Once such longer history data becomes available, it is definitely insightful to test our approach then. 

This is understandable. But this is an important point/disclaimer, and it is not actually mentioned in the intro etc. 
I also had asked: "please explain what your updated intro contribution statement relative to existing methods is, in 2 sentences" to see how it would address my overclaiming concerns. I note that your response does not address this ask. 

> Regarding Fig.1, please note each blue circle represents a specific scene rather than a single frame. 

I suggest at the very least you augment the figure caption to clarify that each blue scene circle contains X seconds of past history, or some such.
Thanks for the recognition. 

We cannot provide the evaluation of longer history due to there is no temporal relationships between different scenes in existing benchmarks. Once such longer history data becomes available, it is definitely insightful to test our approach then. That being said, we have now tested the longest possible history with the existing benchmark sampled from the real-world driving data.

Regarding Fig.1, please note each blue circle represents a specific scene rather than a single frame, which consists of historical trajectories and HP map within a certain range. In this case, current methods independently process every individual scene, while RealMotion is designed to model the temporal relationships across successive scenes. We will further clarify to minimize similar misunderstanding.
Authors have answered my technical and clarification questions in a satisfactory manner (Q2 - Q4). Please try to address them in a future manuscript version, some of these details are key to the method (e.g. trajectory stream attending to multiple past frames and L_refine definition etc) but are not sufficiently well described. 

Wrt my Q1 I am not particularly reassured yet. Please explain what your new pitch is in 2 sentences. Also, while in theory your method can benefit from longer historical information, in practice you have not tried it, so it is not proven. What seems proven to me is that presenting the information in a streaming fashion may be superior. 
Fig 1 is confusing because it shows that for standard methods, each frame attending only its own timestep independently. This is not correct since it attends X frames of previous data but it is not shown (yet the past frames are shown in yellow for RealMotion).
Thanks. We will further highlight the crucial parts.
Thanks for the quick response.

Exactly. This is more than just simply considering longer historical frames like previous methods, otherwise, our method would not exceed previous alternatives. Our formulation allows to capitalize the sequential data more effectively in real world scenarios, not only emphasizing a longer historical scene context but also leveraging historical predictions, which cannot be achieved by simply increasing historical frames. We will further clarify in the revised version.
I have checked the rebuttal and the reviews from other reviewers. The added analytical experiment makes sense to me, and the added results of improving QCNet with RealMotion are also reasonable.

I think the added visualization is good. For the revisions, I suggest the authors add some highlights pointing to the frames/branches where RealMotion has better continuity than the baseline.

I maintain my original rating of weak accept for now.
Thank you for your answers and your clarifications. After going through the paper in light of the provided answers, it seems I indeed misunderstood an important part of the submission.

To make sure I'm getting it right this time, would you agree your working hypothesis could be sumarized as follows?
- Given a large history (of 50 frames), it is better to have a model that operate on a smaller sequence (30 frames) + an embedding of the past that you can obtain using the same model on earlier frames, than directly on the full (50 frames) history like QCNet does.