Title: Learning Cooperative Trajectory Representations for Motion Forecasting
Venue: NeurIPS 2024
Accepted: yes
Keywords: Cooperative Autonomous Driving

Motion Forecasting
Abstract: Motion forecasting is an essential task for autonomous driving, and utilizing information from infrastructure and other vehicles can enhance forecasting capabilities.
Existing research mainly focuses on leveraging single-frame cooperative information to enhance the limited perception capability of the ego vehicle, while underutilizing the motion and interaction context of traffic participants observed from cooperative devices. 
In this paper, we propose a forecasting-oriented representation paradigm to utilize motion and interaction features from cooperative information. 
Specifically, we present V2X-Graph, a representative framework to achieve interpretable and end-to-end trajectory feature fusion for cooperative motion forecasting. 
V2X-Graph is evaluated on V2X-Seq in vehicle-to-infrastructure (V2I) scenarios.
To further evaluate on vehicle-to-everything (V2X) scenario, we construct the first real-world V2X motion forecasting dataset V2X-Traj, which contains multiple autonomous vehicles and infrastructure in every scenario.
Experimental results on both V2X-Seq and V2X-Traj show the advantage of our method. 
We hope both V2X-Graph and V2X-Traj will benefit the further development of cooperative motion forecasting.
Find the project at https://github.com/AIR-THU/V2X-Graph.

Learning Cooperative Trajectory Representations for Motion Forecasting

Abstract

Motion forecasting is an essential task for autonomous driving, and utilizing information from infrastructure and other vehicles can enhance forecasting capabilities. Existing research mainly focuses on leveraging single-frame cooperative information to enhance the limited perception capability of the ego vehicle, while underutilizing the motion and interaction context of traffic participants observed from cooperative devices. In this paper, we propose a forecasting-oriented representation paradigm to utilize motion and interaction features from cooperative information. Specifically, we present V2X-Graph, a representative framework to achieve interpretable and end-to-end trajectory feature fusion for cooperative motion forecasting. V2X-Graph is evaluated on V2X-Seq in vehicle-to-infrastructure (V2I) scenarios. To further evaluate on vehicle-to-everything (V2X) scenario, we construct the first real-world V2X motion forecasting dataset V2X-Traj, which contains multiple autonomous vehicles and infrastructure in every scenario. Experimental results on both V2X-Seq and V2X-Traj show the advantage of our method. We hope both V2X-Graph and V2X-Traj will benefit the further development of cooperative motion forecasting. Find the project at https://github.com/AIR-THU/V2X-Graph.

1 Introduction

In recent years, autonomous driving has made significant progress. However, single-vehicle autonomous driving still faces substantial safety challenges due to its limited perception ability. Utilizing external information, such as data from other connected autonomous vehicles and infrastructure sensors through vehicle-to-everything (V2X), has shown great potential to enhance autonomous driving capabilities. In this paper, we focus on motion forecasting, a fundamental task for autonomous driving that has received significant attention in recent years [40, 33, 31, 19]. Specifically, considering currently practical communication conditions, we transmit perception results and input trajectories from the ego vehicle and external sources for cooperative motion forecasting.

Cooperative motion forecasting involves the ego vehicle aggregating its own data with data transmitted from other connected vehicles or infrastructure devices to predict future waypoints for each agent in traffic scenarios. To accommodate limited communication conditions, we consider data in the form of perception results. These perception results form historical trajectories of agents from respective views, termed cooperative trajectories. The autonomous vehicle utilizes these trajectories to enhance its motion forecasting capabilities. High-definition (HD) maps are also used in this task.

To leverage cooperative trajectories for improving motion forecasting performance, two critical issues must be addressed: (1) Observations of the agents from different views may different due to various sensor perspectives and configurations; (2) In the cooperative scenario, there are multi-

(a) Existing Cooperative Method

Prediction

(b) Cooperative Trajectory Representations Paradigm

Motion

Prediction

Interaction

Other View

Interpretable Graph

Other View

Ego

Agents

view observations of multi-agents, the redundant data need to be leveraged interpretably. Existing research mainly focuses on single-frame feature fusion to support real-world applications and enhance detection performance [50, 44, 46, 4, 42]. A recent method [51] attempts to complement perception over the historical horizon to improve forecasting performance. These methods are depicted in fig. 1(a). However, the above single-frame methods obtain the agent state at each frame individually, which may lead to a trade-off between the states observed from distinct views, and cannot utilize motion and interaction context sufficiently, thus failing to sufficiently model the historical behavior of agents. Instead, considering historical observations from each view holistically could address these shortcomings. Compared to previous approaches, this paper explores a novel forecasting-oriented trajectory feature fusion method, which aims to enhance the historical representation of agents, including their historical motion and surrounding interactions, for motion forecasting.

To address the challenges and effectively utilize the cooperative information, we propose V2X-Graph, a graph-based framework to achieve cooperative trajectory feature fusion for motion forecasting. Theoretically, V2X-Graph offers two advantages for enhancing motion forecasting performance. (1) Forecasting-oriented cooperative representation. For accurate motion forecasting, it is common practice to represent motion features from an agent's historical trajectory and interaction features from other agents [10, 25, 58, 19]. Instead of perception complement, our method is the first to consider cooperative perception information from the typical motion forecasting perspective, which independently represents motion and interaction representations of cooperative trajectories, customized relative spatial-temporal encodings are designed to support trajectories feature fusion of each agent over historical horizon. (2) Graph-guided heterogeneous feature fusion. To support multi-agent motion forecasting in a cooperative scenario, it is essential to interpretably integrate heterogeneous motion and interaction features for each specific agent from cooperative trajectories. Drawing inspiration from graph link prediction [12, 55], a classical task that analyzes the relationship of nodes in a graph, this may further support downstream applications like node feature propagation. To achieve end-to-end optimization in V2X-Graph, we constructed a graph that represents the cooperative scenario, an interpretable association is established to guide heterogeneous feature fusion based on agent identification across views. The framework is represented in fig. 1(b).

V2X-Graph is evaluated on V2X-Seq [51], which contains vehicle-to-infrastructure (V2I) cooperative scenarios. To evaluate its effectiveness in vehicle-to-vehicle (V2V) and further more cooperative devices scenarios, we construct the first public and real-world vehicle-to-everything (V2X) motion forecasting dataset V2X-Traj. This dataset is the first to include multiple autonomous vehicles and infrastructure in every scenario, broadening the research devoted to V2X motion forecasting task. Extensive experiments conducted on V2X-Seq and V2X-Traj show the advantages of V2X-Graph in utilizing additional cooperative information to enhance the motion forecasting capability.

Our contributions are four fold: (1) We propose a forecasting-oriented representation paradigm to utilize motion and interaction features from cooperative information. (2) We design V2X-Graph, a representative framework to achieve interpretable and end-to-end trajectory feature fusion for cooperative motion forecasting. (3) We construct V2X-Traj, which is the first public and real-world dataset for V2X motion forecasting. It includes not only V2I but also V2V cooperation in every cooperative scenario. (4) Our approach achieves state-of-the-art on both V2X-Seq and V2X-Traj.

2 Related Work

Cooperative Autonomous Driving. In recent years, more and more researchers pay attention to cooperative autonomous driving, which leverages additional information from infrastructure-side devices and other vehicles to achieve system-wide performance improvement. As several public cooperative perception datasets [45, 49, 15] have been released, most of them focus on cooperative perception. Different from the single-side object detection [56, 24, 8, 47, 48], cooperative detection methods aim to promote performance and transmission latency trade-offs to support real-world applications [46, 22, 44, 41, 50, 9, 4]. Some works also dive into cooperative segmentation task [38, 43]. However, as the downstream task of cooperative perception and directly influences the actions of autonomous vehicles, cooperative motion forecasting has not been well studied. A recent endeavor supplies historical observations with perception information from infrastructure devices to improve motion forecasting performance [51]. Instead of perception completion then forecasting, this paper presents an end-to-end cooperative motion forecasting framework for cooperative trajectory feature fusion, to achieve comprehensive utilization of motion and interaction contexts from cooperative information. To further broaden the research into V2X cooperative motion forecasting, we construct the first real-world and public motion forecasting dataset for general V2X scenario, termed V2X-Traj, including not only V2I but also V2V cooperations in every scenario.

Motion Forecasting. Motion forecasting is an indispensable task in autonomous driving systems, which takes sequential perception results of agents and map elements into account to predict future trajectories of agents. Early works rasterize scenarios as images and deploy convolution neural networks to extract information [34, 21]. The research community turns to vectorize the representations of agents and maps for motion and interaction contexts [10]. Some works consider pooling mechanism for feature fusion [1, 14, 10, 35]. Others utilize the convolution technique to extract local features [25, 52, 5]. Inspired by the effectiveness and widespread usage of the Transformer model [36], recent works adopted attention mechanism for learning representations in motion forecasting task [26, 39, 31, 58, 33, 19, 40]. Instead of sequential perception supplementation then forecasting, V2X-Graph explores novel trajectory feature fusion to comprehensively utilize information. It represents and decouples trajectory features into motion and interaction features for each view independently, and employs customized Transformer modules for aggregating interpretable features based on agent identification, which facilitates cooperative motion forecasting.

GNNs for Motion Forecasting. Graph neural network (GNN) [20, 37] is a common structure for motion forecasting. A graph consists of nodes and edges, with each node typically representing information related to an agent or a map element. While edges represent the relative information between pairs of nodes. The message-passing mechanism aggregates and updates node features from their neighbors. Previous methods adopted homogeneous GNN for unified but coarse scene representation [30, 10, 23, 57, 25, 13, 11]. While recent research introduced heterogeneous GNN [54, 16] to distinguish and further extracting features based on various settings of agents [32, 29, 18, 19]. Compared to previous approaches, V2X-Graph explores leveraging heterogeneous edge encodings and interpretable graph link prediction for trajectory-based motion and interaction features fusion.

3 Preliminary

Cooperative motion forecasting can play an important role for autonomous driving as it better reasons the future movements of surrounding agents. The ego vehicle receives sequential perception results from cooperative devices, including infrastructures and other vehicles, to enhance the capability of motion forecasting. The contextual information in the vector map is also taken into account.

Problem Formulation. The inputs of cooperative motion forecasting include multiple-source trajectories and vector maps. The cooperative motion forecasting scenario is represented as S = { T , L } , where T and L are described as follows. (1) Trajectory. In a typical V2X cooperation scenario, each cooperative device independently captures the historical status of agents as trajectories. The multi-source trajectories are denoted as T = { T ego , T other } , here T other can include received multi-source cooperative trajectories such as T inf and T veh from the views of infrastructure and cooperative vehicles. The total number of trajectories is N t = N ego + N other . Trajectory information is summarized as T ∈ R N t × T × C t , where T is the historical horizon and C t is the attributes of each trajectory to depict corresponding agent ( e.g. , tracking id, location, heading angle, detection bounding box and agent type). Specifically, the historical spatial status of each trajectory is formulated as { p t i , r t i } T t =1 , where p t i ∈ R 2 is the trajectory i 's location, and r t i represents the heading theta vector at time step t . (2) Vector Map. Vectorized representation [10] is usually adopted for representing the

map elements in motion forecasting task, which leverages the sample points of the centerline within each lane and enables an efficient vectorized representation of spatial information. In this paper, we further consider vectorized lane segment, i.e. , the vector between each two neighboring sample points. The set of vectorized lane segments is denoted as: L ∈ R N l × 2 × C l , where N l is the number of lane segments and C l is the attributes of each lane segment ( e.g. , location and road type). The start point and the end point of the lane segment are formulated as { p start l , p end l } , here l ∈ [1 , 2 , ..., N l ] .

Evaluation. The output is K future trajectories of the specified target agent in each scenario, the best one is chosen for evaluation, here K = 6 . Evaluation metrics are minADE, minFDE and MR, standard metrics for motion forecasting. Lower number is better .

Challenges. To enhance the capability of motion forecasting considering abundant cooperative trajectories, it is essential to: (1) effectively represent the cooperative scenario, (2) efficiently utilize valuable information from redundant cooperative trajectories.

4 Methodology

This section presents V2X-Graph, a graph-based framework designed to achieve interpretable trajectory feature fusion for cooperative motion forecasting. To represent the cooperative scenario, it constructs a graph with node and edge encodings. To enhance cooperative trajectory feature fusion, an interpretable graph consisting of three subgraphs is designed for the aggregation of heterogeneous motion and interaction features. The overall architecture is depicted in fig. 2.

4.1 Scene Representation with Graph

In the graph that represents the cooperative scenario, trajectories from each view and their corresponding lane segments are independently encoded as nodes, while the relative spatial and temporal features between these nodes are encoded as edges.

Graph Node Encodings. We encode the node features from three perspectives: trajectory motion features, trajectory spatial-temporal features, and lane segment spatial features.

Compared to previous methods that leverage cooperative information at each frame individually, we encode differential information from each view then fuse it with correlation to mitigate the deviation caused by direct single-frame fusion. The embeddings of differential coordinates { p t i -p t -1 i } T t =1 of the trajectory are considered as the motion features at each timestep. The self-attention mechanism [6] is adopted to incorporate temporal dependency. Here, missing frames are padded with learnable tokens, and the attention mechanism is enforced to only attend to the preceding time steps.

For the purposes of trajectory identification for interpretable association and motion correlation measurement, we also encode spatial-temporal features for trajectories from each view. The spatialtemporal features are encoded by incorporating the temporal dependency between the ego-centric normalized coordinates of the trajectory. Missing frames are masked in the attention module.

Overall, node encodings of trajectory from each view are formulated as:

where PE t signifies the learnable positional embedding at timestep t , SelfAttn ( · ) is the multi-head self-attention module, and MLP ( · ) represents a multi-layer perceptron.

To enhance the representation of trajectories for future intention reasoning, the feature of the vector map structure is incorporated. Specifically, the spatial feature of lane segments are represented by the relative coordinates (from the start point to the end point of each lane segment) in an agent-centric frame [35], and further encoded as nodes for feature aggregation. The formulation is:

where r T i,l is the relative heading vector between trajectory i with lane segment l in its current frame.

Graph Edge Encodings. For effective trajectory feature fusion, we design heterogeneous edge encodings, including spatial-temporal encoding and relative spatial encoding.

We use an attention module to aggregate the spatial-temporal encodings between each pair of crossview trajectories. The spatial-temporal encoding captures the spatial-temporal correlations between two trajectories at each timestep, which facilitates motion feature fusion.

Trajectory

Decoder

Multi-agent Prediction

𝐓

𝒆𝒈𝒐

Other View

𝐓

𝒐𝒕𝒉𝒆𝒓

Interpretable Association

Graph Link Prediction

MFG

𝓖

'()

CIG

𝓖

𝒂𝒍)

ALG

𝓖

𝒂𝒍)

Motion Feature

Interaction Feature

Other View

ST-Edges

RS-Edges

Graph

Encoder

Input

Output

Trajectory Feature Fusion

Interpretable Graph

𝐀𝐝𝐣𝐚𝐜𝐞𝐧𝐭 𝐌𝐚𝐭𝐫𝐢𝐱 𝐀

Spatial-temporal

Attention

Spatial

Relativity

To capture the interaction features, we also introduce relative spatial encodings as edges.

Edge encodings can be formulated as follows:

where e rs i → j represents both agent-agent and agent-lane interaction relations. For agent-agent interaction, the coordinates of the current frame for trajectories are denoted by p i and p j . While for agent-lane interaction, the encoding represents the feature between current coordinate p i of trajectory i and the starting point coordinates p start l of the lane segment.

4.2 Feature Fusion with Interpretable Graph

To achieve comprehensive historical representations of agents in a cooperative scenario, an interpretable graph is designed for multi-view trajectory feature fusion. Serving as a guidance for heterogeneous feature representations, the Interpretable Association component (IA) establishes explicit associations between trajectories of the same agent across views. The Motion Fusion subGraph (MFG) represents cooperative motion features by considering both explicit associations and implicit spatial-temporal encodings. The Agent-Lane subGraph (ALG) fuses the features from each view with lane segment features. The Cooperative Interaction subGraph (CIG) represents dense interaction representations by leveraging the spatial encodings between different agents from all views.

Algorithm 1: Pseudo Labels Generator

Interpretable Association. To achieve end-to-end optimization of heterogeneous feature fusion, we formulate the association process as a graph link prediction problem. An interpretable association component is introduced to establish interpretable associations of cross-view trajectories of agents, providing explicit guidance for the fusion of motion and interaction features. Additionally, we propose

a pseudo label generator, described in alg. 1, to collect trajectory matching labels for trajectories from the ego and another view over the historical horizon within the training set for knowledge distillation.

We denote an adjacency matrix for the two sets of trajectories A = { a i,j | i ∈ N ego , j ∈ N other } , where elements referring to associations are supervised by ˜ A which can be predicted as:

here Φ Classifier refers to a MLP for binary classification to determine whether there exists an association between two trajectories, thereby instructing the perception information belonging to the same agent. The classification is based on relative spatial-temporal encodings of trajectories across views.

Motion Fusion SubGraph. To comprehensively represent the historical motion of agents, we employ the Motion Fusion subGraph (MFG) to aggregate cooperative motion feature representations from cross-view associated trajectories. MFG models cooperative motion representations by incorporating both interpretable associations and temporal-spatial correlations.

MFG is defined as: G mfg = ( V , E ) and A mfg = A , where the node set V = { v i | i ∈ N t } , and the edge set E = { e i,j } denotes the edges captured by A mfg . Feature fusion and update process is:

where v ( k +1) i represents the updated feature of node v ( k ) i , FFN ( · ) is a feed-forward network.

Agent-Lane SubGraph. We employ the Agent-Lane subGraph (ALG) to incorporate map information for cooperative motion forecasting. ALG is a bipartile graph that enables agent motion features to query relevant lane segment interaction features using relative spatial encodings. We consider all lane segments within the observation range of the current frame of agents from each view.

ALG is defined as: G alg = ( V , E ) , where V = { v i , v l | i ∈ N t , l ∈ N l } and E = { e i,l } . The process of interaction feature aggregation and update from agents to lane segments can be formulated as:

where a l represents learnable tokens of semantic attributes associated with the corresponding lane segment, such as turn direction and road type.

Cooperative Interaction SubGraph. The Cooperative Interaction subGraph (CIG) is employed for cooperative interaction features representation between trajectories of distinguished agents in all views. Incorporating both interpretable associations and relative spatial correlations, CIG models denser interaction representations in a cooperative scenario.

CIG is defined as: G cig = ( V , E ) and A cig = ∼ A , where V = { v i | i ∈ N t } , E = { e i,j } denotes the set of not associated cross-view trajectories, which is captured by the adjacency matrix A cig , and all intra-view edges. The process of interaction feature fusion and update in CIG formulated as:

where a i,j represents the learnable features of interaction attributes associated between two trajectories, such as relative headings at the current timestep. Note that CIG only considers the interaction among trajectories observed in the current frame.

4.3 Multimodal Future Decoder

The future motion of traffic agents is inherently multi-modal. We parameterize the distribution of future trajectories as Laplacian Mixture Model (LMM) following [58]. Every agent from each view is predicted and supervised during training phase for representations of trajectory feature fusion. While during inference phase, the decoder predicts the future trajectory of distinct agents in the cooperative scenario from the ego-view, guided by interpretable associations. Technically, we employ a MLP as a prediction head to aggregate all the intermediate features, which can be formulated as:

where Z 1: T i includes K Laplacian components N 1: K with multi-modal probability distributions p 1: K . The formulation for predicting the future coordinate distribution of agent i at time t is as:

the future positions are generated by the center of distributions. The distribution N ( µ x , µ y ) and corresponding probability p k are generated by two MLPs separately.

(a) Statistic

(b) Visualizations of Ego-Infra-Veh Views

1E+8

100

1E+7

Number  Length

Number of Trajectory

1E+6

80

Average Length

1E+5

60

1E+4

1E+3

40

1E+2

20

1E+1

1E+0

CAR

CYCUST

MOTORCYCLST

PEDESTRAN

BUS

TRICYCLST

TRUCK

4.4 Training Losses

To achieve interpretable trajectory feature fusion, the framework is trained in an end-to-end manner with two components of optimization objectives. The first part is the cross-entropy loss to optimize the graph link prediction. The second part includes the regression loss and classification loss to optimize the motion forecasting. Please refer to appendix for more loss details.

5 Experiment

In this section, we evaluate the proposed V2X-Graph framework not only on V2I cooperative scenarios but also on V2V and broader V2X cooperative scenarios.

5.1 Experimental Setup

Dataset. V2X-Graph is evaluated on both V2I and broader V2X scenarios. (1) V2X-Seq [51]. A public large-scale and real-world V2I dataset. V2X-Seq consists of 51,146 V2I scenarios, and each scenario is 10 seconds long with the sample rate of 10 Hz. The task is to predict the motion of agents for the next 5 seconds, given the initial 5-second observation from both infrastructure and ego-view. (2) V2X-Traj (Ours) . To study the effectiveness of V2X-Graph in V2V and broader V2X scenarios, especially its ability to handle more than two views of trajectories, including both V2I and V2V cooperation, we construct the first real-world and public V2X cooperative motion forecasting dataset, termed V2X-Traj. It comprises 10,102 scenarios in challenging intersections. Each scenario includes two intelligent vehicles and an infrastructure perception device. The statistics and visualization of V2X-Traj are presented in fig. 3. Each scenario lasts for 8 seconds with a sample rate of 10 Hz. The 4-second observations from each view are used to predict the future motion in the next 4 seconds. We hope the V2X-Traj dataset can facilitate the development of cooperative motion forecasting for general V2X scenarios. More details are described in the Appendix.

Implementation Details. For scene graph representation, V2X-Graph employs a 4-layer temporal self-attention Transformer to encode motion features, a 2-layer temporal self-attention Transformer and a 2-layer self-attention module for relative temporal-spatial feature encoding. In the interpretable graph, there are 3 layers of MFG, 1 layer of ALG and 3 layers of CIG. The dimensions of the hidden feature is set as 128, and the number of heads in all multi-head attention blocks is 16. The lane segments corresponding to agents within a observation range of 50 meters are taken into consideration. For training, the initial learning rate is set to 1 × 10 -3 and is scheduled according to cosine annealing [27]. The AdamW optimizer [28] is adopted with a weight decay of 1 × 10 -4 . The model is trained for 64 epochs with batch size of 64 on a server with 8 NVIDIA RTX 4090s.

5.2 Main Results

Cooperative method comparison. We compare our method with other cooperative methods on V2X-Seq. To the best of our knowledge, PP-VIC [51] is the only existing method of the same type. Typically, PP-VIC provides the ego vehicle with infra-side perception information in a frame-byframe manner within the historical horizon. After supplementation, the perception information is provided to popular and competitive vanilla forecasting methods DenseTNT [13] and HiVT [58]. For comparison, we also provide the output of PP-VIC to V2X-Graph in a similar way to the egoview perception. As shown in table 1, perception completion enhances the downstream forecasting performance of HiVT and V2X-Graph. However, the trade-off in cross-view perception also leads to error propagation, resulting in performance degradation of DenseTNT. Instead, V2X-Graph enhances the historical representation of agents through trajectory feature fusion, leading to performance improvements, as evidenced by -0 . 07 in minADE, -0 . 19 in minFDE, and -5% in MR.

Graph-based methods comparison. To reflect the unique advantages of V2X-Graph, we conduct evaluations on V2X-Traj and compared it with representative graph-based methods. Similar to V2XGraph, cooperative trajectories are encoded as vanilla nodes of agents in each compared methods, for fair comparison. Experimental results in four typical settings are reported in table 2. As shown in the table, HDGT [19] achieves superior performance through precise heterogeneous design to represent relationship of agents, compared with DenseTNT [13], which employs a homogeneous graph to represent the scenario. V2X-Graph is also highly competitive in vehicle-only task, without sophisticated feature engineering and decoder design. However, our method outperforms compared methods by large margins in all cooperative settings and achieves the best performance in V2V&I cooperation, with -0 . 22 in minADE, -0 . 43 in minFDE, and -6% in MR, illustrating the effectiveness of aggregation heterogeneous motion and interaction features to enhance cooperative forecasting.

5.3 Ablation Study

To further illustrate the effectiveness of the method for trajectory feature fusion and the final result, we conduct ablation studies on the V2X-Traj validation set. Considering extensive ablation studies, experiments are conducted based on our small model with a hidden-size of 64.

Effectiveness of Major Components. Firstly, we alternately removing one of the components to illustrate the contribution of each component to the cooperative motion forecasting performance. As shown in table 3, the components within the interpretable graph separately represent the typical motion feature of historical states of agents and the interaction features with surroundings, demonstrating a marked performance enhancement of motion forecasting.

Effectiveness of Cooperative Representations.

We further evaluate the effectiveness of the model in trajectory feature fusion by alternately masking the features of cooperative trajectories within each component. As demonstrated in table 4, each customized component benefits tra-

jectory feature fusion and results in performance improvements to a certain degree. Specifically, the MFG effectively integrates the motion features of associated trajectories, leading to -0 . 08 in minADE. Relatively, the ALG and CIG components fuse the interaction features of lane segments and cooperative trajectories. These component primarily enhance the performance of long-term intention prediction, as indicated -0 . 59 in minFDE and -7% in MR.

Effectiveness of Interpretable Feature Fusion. Moreover, we evaluate the effectiveness of the proposed interpretable graph in aggregating heterogeneous motion and interaction features within cooperative trajectories. In table 5, the first line presents the result of the vehicle-only setting, which has no cross-view motion and interaction features fusion. We simply employ fully connections in our graph to aggregate motion and interaction features, compared with the ego-setting, there is no obvious positive effect with a large amount of features fusion. The last line shows the result of interpretable

0.9

1

1.1

1.2

1.3

0.00

0.05

0.10

0.15

ADE

1.6

1.8

2

2.2

2.4

2.6

0.00

0.05

0.10

0.15

FDE

0.20

0.22

0.24

0.26

0.28

0.30

0.00

0.05

0.10

0.15

MR

features fusion, it achieves improved performance with a low computational cost, demonstrating the effectiveness to interpretably aggregate heterogeneous cooperative features.

Effectiveness of Pseudo Label Supervision. To demonstrate the quality of pseudo labels and influence on the final motion forecasting performance, we disturb labels randomly for supervision and evaluate the motion forecasting results. As shown in fig. 4, as the proportion of disturbed pseudo labels increases, corresponding forecasting performance decreases, which illustrates the effectiveness of both interpretable feature fusion and pseudo label supervision. This also suggests that there is potential for further improvement in forecasting performance as the quality of labels increases.

6 Conclusion and Limitation

In this paper, we introduce a forecasting-oriented representation paradigm to utilize motion and interaction features from cooperative information, and present V2X-Graph, a graph-based framework to achieve interpretable and end-to-end trajectory feature fusion for cooperative motion forecasting. Comparing to existing methods that rely on single-frame perception information cooperation, our approach enhances the historical representation of agents from lightweight cooperative trajectories, and achieve improved performance in the downstream task, namely cooperative motion forecasting. Moreover, we construct V2X-Traj, the first real-world and public V2X cooperative motion forecasting dataset, expanding the research from V2I to broader V2X motion forecasting task. Experiments on both two datasets demonstrate the effectiveness of our method.

Limitation and future work. Compared to the single-frame method, the proposed V2X-Graph explores trajectory feature fusion, which mitigates errors from single-frame perception completion and achieves better motion and interaction representation of agents. Despite these advantages, the performance still relies on the tracking quality from each view. Jointly optimizing the performance from perception to forecasting is significant to explore in the future.

Acknowledgment

The paper is supported by funding from Wuxi Research Institute of Applied Technologies, Tsinghua University under Grant 20242001120.

References

Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, and Silvio Savarese. Social lstm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , 2016.

Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving, 2020.

Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , 2019.

Ziming Chen, Yifeng Shi, and Jinrang Jia. Transiff: An instance-level feature fusion framework for vehicle-infrastructure cooperative 3d detection with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023.

Fang Da and Yu Zhang. Path-aware graph attention for HD maps in motion prediction. In Proceedings of the International Conference on Robotics and Automation (ICRA) , 2022.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the North American Association for Computational Linguistics (NAACL) , 2019.

Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles Qi, Yin Zhou, Zoey Yang, Aurelien Chouard, Pei Sun, Jiquan Ngiam, Vijay Vasudevan, Alexander McCauley, Jonathon Shlens, and Dragomir Anguelov. Large scale interactive motion forecasting for autonomous driving : The waymo open motion dataset, 2021.

Siqi Fan, Zhe Wang, Xiaoliang Huo, Yan Wang, and Jingjing Liu. Calibration-free bev representation for infrastructure perception. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2023.

Siqi Fan, Haibao Yu, Wenxian Yang, Jirui Yuan, and Zaiqing Nie. Quest: Query stream for vehicle-infrastructure cooperative perception. In Proceedings of the International Conference on Robotics and Automation (ICRA) , 2024.

Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

Harshayu Girase, Haiming Gang, Srikanth Malla, Jiachen Li, Akira Kanehara, Karttikeya Mangalam, and Chiho Choi. Loki: Long term and key intentions for trajectory prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the ACM SIGKDD international conference on knowledge discovery & data mining (KDD) , 2016.

Junru Gu, Chen Sun, and Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) , 2018.

Yue Hu, Yifan Lu, Runsheng Xu, Weidi Xie, Siheng Chen, and Yanfeng Wang. Collaboration helps camera overtake lidar in 3d detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In Proceedings of the ACM Web Conference (WWW) , 2020.

Xinyu Huang, Peng Wang, Xinjing Cheng, Dingfu Zhou, Qichuan Geng, and Ruigang Yang. The apolloscape open dataset for autonomous driving and its application. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 2020.

Xiaosong Jia, Liting Sun, Hang Zhao, Masayoshi Tomizuka, and Wei Zhan. Multi-agent trajectory prediction by combining egocentric and allocentric views. In Proceedings of the Conference on Robot Learning (CoRL) , 2022.

Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. Hdgt: Heterogeneous driving graph transformer for multi-agent trajectory prediction via scene encoding. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 2023.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In Proceedings of the International Conference on Learning Representations (ICLR) , 2016.

Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.

Zixing Lei, Shunli Ren, Yue Hu, Wenjun Zhang, and Siheng Chen. Latency-aware collaborative perception. In Proceedings of the European Conference on Computer Vision (ECCV) , 2022.

Jiachen Li, Fan Yang, Masayoshi Tomizuka, and Chiho Choi. Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers. In Proceedings of the European Conference on Computer Vision (ECCV) , 2022.

Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. In Proceedings of the European Conference on Computer Vision (ECCV) , 2020.

Yicheng Liu, Jinghuai Zhang, Liangji Fang, Qinhong Jiang, and Bolei Zhou. Multimodal motion prediction with stacked transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In Proceedings of the International Conference on Learning Representations (ICLR) , 2016.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Representations (ICLR) , 2018.

Xiaoyu Mo, Zhiyu Huang, Yang Xing, and Chen Lv. Multi-agent trajectory prediction with heterogeneous edge-enhanced graph attention network. IEEE Transactions on Intelligent Transportation Systems (TITS) , 2022.

Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , 2020.

Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene transformer: A unified architecture for predicting future trajectories of multiple agents. In Proceedings of the International Conference on Learning Representations (ICLR) , 2021.

Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data. In Proceedings of the European Conference on Computer Vision (ECCV) , 2020.

Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Charlie Tang and Russ R Salakhutdinov. Multiple futures prediction. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, et al. Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction. In Proceedings of the International Conference on Robotics and Automation (ICRA) , 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems (NeurIPS) , 2017.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In Proceedings of the International Conference on Learning Representations (ICLR) , 2018.

Binglu Wang, Lei Zhang, Zhaozhong Wang, Yongqiang Zhao, and Tianfei Zhou. Core: Cooperative reconstruction for multi-agent perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023.

Jingke Wang, Tengju Ye, Ziqing Gu, and Junbo Chen. Ltp: Lane-based trajectory prediction for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , 2022.

Xishun Wang, Tong Su, Fang Da, and Xiaodong Yang. Prophnet: Efficient agent-centric motion forecasting with anchor-informed proposals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Zhe Wang, Siqi Fan, Xiaoliang Huo, Tongda Xu, Yan Wang, Jingjing Liu, Yilun Chen, and Ya-Qin Zhang. Vimi: Vehicle-infrastructure multi-view intermediate fusion for camera-based 3d object detection. In Proceedings of the International Conference on Robotics and Automation (ICRA) , 2024.

Sizhe Wei, Yuxi Wei, Yue Hu, Yifan Lu, Yiqi Zhong, Siheng Chen, and Ya Zhang. Robust asynchronous collaborative 3d detection via bird's eye view flow. In Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Runsheng Xu, Zhengzhong Tu, Hao Xiang, Wei Shao, Bolei Zhou, and Jiaqi Ma. Cobevt: Cooperative bird's eye view semantic segmentation with sparse transformers. In Proceedings of the Conference on Robot Learning (CoRL) , 2023.

Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang, and Jiaqi Ma. V2x-vit: Vehicle-to-everything cooperative perception with vision transformer. In Proceedings of the European Conference on Computer Vision (ECCV) , 2022.

Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma. Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. In Proceedings of the International Conference on Robotics and Automation (ICRA) , 2022.

Dingkang Yang, Kun Yang, Yuzheng Wang, Jing Liu, Zhi Xu, Rongbin Yin, Peng Zhai, and Lihua Zhang. How2comm: Communication-efficient and collaboration-pragmatic multi-agent perception. In Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Lei Yang, Kaicheng Yu, Tao Tang, Jun Li, Kun Yuan, Li Wang, Xinyu Zhang, and Peng Chen. Bevheight: A robust framework for vision-based roadside 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Lei Yang, Xinyu Zhang, Jun Li, Li Wang, Minghan Zhu, Chuang Zhang, and Huaping Liu. Mixteaching: A simple, unified and effective semi-supervised learning framework for monocular 3d object detection. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT) , 2023.

Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, and Zaiqing Nie. Dair-v2x: A large-scale dataset for vehicleinfrastructure cooperative 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Haibao Yu, Yingjuan Tang, Enze Xie, Jilei Mao, Ping Luo, and Zaiqing Nie. Flow-based feature fusion for vehicle-infrastructure cooperative 3d object detection. In Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Haibao Yu, Wenxian Yang, Hongzhi Ruan, Zhenwei Yang, Yingjuan Tang, Xu Gao, Xin Hao, Yifeng Shi, Yifeng Pan, Ning Sun, et al. V2x-seq: A large-scale sequential dataset for vehicle-infrastructure cooperative perception and forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Wenyuan Zeng, Ming Liang, Renjie Liao, and Raquel Urtasun. Lanercnn: Distributed representations for graph-centric motion forecasting. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2021.

Wei Zhan, Liting Sun, Di Wang, Haojie Shi, Aubrey Clausse, Maximilian Naumann, Julius Kummerle, Hendrik Konigshof, Christoph Stiller, Arnaud de La Fortelle, and Masayoshi Tomizuka. Interaction dataset: An international, adversarial and cooperative motion dataset in interactive driving scenarios with semantic maps. arXiv preprint arXiv:1910.03088 , 2019.

Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. Heterogeneous graph neural network. In Proceedings of the ACM SIGKDD international conference on knowledge discovery & data mining (KDD) , 2019.

Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. In Advances in Neural Information Processing Systems (NeurIPS) , 2018.

Renrui Zhang, Han Qiu, Tai Wang, Ziyu Guo, Xuanzhuo Xu, Yu Qiao, Peng Gao, and Hongsheng Li. Monodetr: depth-guided transformer for monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023.

Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Ben Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, et al. Tnt: Target-driven trajectory prediction. In Proceedings of the Conference on Robot Learning (CoRL) , 2020.

Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, and Kejie Lu. Hivt: Hierarchical vector transformer for multi-agent motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Appendix

In this supplementary material, detailed information about the training loss is provided in appendix A. Comprehensive details regarding the V2X-Traj dataset are presented in appendix B. Additional experimental results are presented in appendix C. The implementation details of the compared methods on the V2X-Traj and V2X-Seq datasets are presented in appendix D. Additional qualitative results of the proposed V2X-Graph framework on the V2X-Seq dataset are provided in appendix E.

A Details of Training Loss

To achieve interpretable trajectory feature fusion in cooperative scenarios, the framework is trained in an end-to-end manner with two components of optimization objectives.

For the first part of optimization objectives, we utilize the cross-entropy as the knowledge distillation loss L dis to optimize the prediction of the adjacency matrix A = { a i,j } , which represents the associations between trajectories of agents across views and is supervised by the pseudo labels α ( ˜ A ) , here α ( · ) indicates a pre-pruning strategy to mitigate the class imbalance issue. This strategy is utilized when there is no intersection between the minimum bounding rectangles of two trajectories, or when the agent types are different.

The second part includes the regression loss L reg and classification loss L cls to optimize the motion forecasting. For optimization, we select the trajectory with the minimum average L2 distance from the ground truth among K modalities. We utilize the cross-entropy loss as the selection loss to optimize p i,k . The negative log-likelihood loss is employed as the regression loss and can be formulated as:

where P ( ·|· ) represents the probability density function of the Laplace distribution, and { ˆ µ t i } T + H t = T +1 , { ˆ b t i } T + H t = T +1 denote the coordinates and the corresponding uncertainties of the best-predicted trajectory for the agent. Overall, the final training loss can be formulated as follow:

B Details of V2X-Traj Dataset

V2X-Traj is the first and real-world V2X cooperative motion forecasting dataset. In this section, we provide a more comprehensive description of our V2X-Traj dataset. Dataset comparison is presented in table 6. Other details includes dataset composition (appendix B.1), collection and annotation process (appendix B.2) and additional visualizations (appendix B.3).

B.1 Dataset Composition

V2X-Traj dataset contains a total of 10,102 scenarios, which are randomly split into the training, validation, and test set, consisting of 6,062, 2,020, and 2,020 scenarios, respectively. Each scenario comprises three independent sets of trajectories from two autonomous driving vehicles and an infrastructure-side perception device. Additionally, considering the agents are moved with the constraints of traffic rules, we also provide the static vector map and real-time traffic light signals.

Trajectory. Each trajectory represents the information of an agent detected and tracked independently by a single perception device. The trajectory information includes the timestamp, unique ID, agent type, location, 7-dimensional detection bounding box, heading, and velocity.

Vector Map. Following [3], we collect map information in the form of vectorized representations to provide valuable hints for motion forecasting. Vector maps contain lane, crosswalk, stopline, and junction elements. For each lane, we provide sample points of centerline and boundary, and semantic attributes such as turning direction, lane topology and traffic control.

Traffic Light. We provide real-time traffic light signals as they have a substantial impact on the behavior of traffic participants. During the data collection and storage process, we simultaneously

record traffic light data at a frequency of 10 Hz. The traffic light signal information includes the timestamp, location, direction, corresponding lane ID, color status, and remaining time.

We provide detailed statistics in table 7. As the table shows, the V2X-Traj dataset contains abundant trajectories of 8 classes of agents to depict real-world V2X cooperative scenarios.

Dataset schema is represented in fig. 6. In each cooperative scenario, three sets of trajectories are independently collected by the ego vehicle, the infrastructure perception device, and another autonomous vehicle; simultaneously, data on traffic light synchronization is gathered. Additionally, we offer a comprehensive vector map covering intersections.

B.2 Data Collection and Annotation

This subsection details the process to construct the V2X-Traj dataset.

We choose 28 urban traffic intersections in Beijing and deploy 4-6 pairs of 300-beam LiDAR and highresolution cameras for each intersection. These infrastructure sensors can fully cover the intersection areas. We deploy one 40-beam LiDAR and six high-quality cameras for the autonomous vehicle. We provide the configuration of sensor deployment of autonomous vehicles and infrastructure in fig. 7. The perception devices of autonomous driving vehicles and infrastucture devices have trained 3D object detection and tracking models. These models are used to generate trajectory sequences.

To collect the trajectory data, the two autonomous driving vehicles were driven simultaneously and randomly through areas equipped with sensors. The V2X cooperative scenarios are collected when there is a certain overlap in the perception range of the two vehicles and the infrastructure device, resulting in the V2X cooperative trajectory sequences repository. The trajectories from each view were stored independently.

Finally, we mined interesting segments from the repository to create 10,102 cooperative scenarios. The trajectory mining process consisted of several steps, including scenario fragmentation, trajectory scoring and scenario selection. In the first step, we divided the sequences from each view into 8-second segments. In the second step, a score was assigned to each trajectory from ego-view based on the interesting behaviors of agents such as turning, speeding up, slowing down and lane changing. In the third step, we retained a total of 10,102 sequences with high-score trajectories. Within each segment, one trajectory was designated as the target agent for prediction, while the remaining trajectories were assigned as others.

V2X-Traj Dataset

Ego/Infra/Veh-Trajectories

city,

timestamp,

id,

type,

sub_type,

agent_tag,

x,

y,

z,

length,

width,

height,

heading_theta,

intersection_id

Vector Map

LANE

├─

lane_id,

├─

has_traffic_control,

├─

lane_type,

├─

turn_direction,

├─

is_intersection,

├─

l_neighbor_id,

├─

r_neighbor_id,

├─

predecessors,

├─

successors,

└─

centerline

STOPLINE

├─

stoplane_id,

└─

centerline

CROSSWALK

├─

crosswalk_id,

└─

polygon

JUNCTION

├─

junction_id,

└─

polygon

Traffic Light

city,

timestamp,

x,

y,

direction,

lane_id,

color_1,

remain_time_1,

remain_time_2,

color_3,

remain_time_3,

intersection_id

Camera

LiDAR

Camera

GPSAntenna

Camera

Insideof theTrunk

Industrial PC

TrunkLiner

RADAR

GPSReveiver

IMU

x8Camera

x4 Fisheye Camera

x4 LiDAR

×1 RSU

x1 PC

x1 Signal Cabinet

B.3 Additional Dataset Visualization

fig. 9 presents four interesting scenarios from our V2X-Traj dataset. As the figure describes, each scenario includes trajectories from the ego-vehicle, the cooperative infrastructure and the cooperative autonomous driving vehicle. The inclusion of cooperative trajectories from both the autonomous vehicle and the infrastructure device enhances the information available to the ego-vehicle.

C Additional Experimental Results

C.1 Effectiveness of Major Components

In this experiment, we conduct an additional ablation study on the V2X-Seq dataset to demonstrate the contribution of each key component to the cooperative motion forecasting performance. Similar to before, we evaluate the effectiveness by removing one of the components alternately. As shown in Table 8, each component within the interpretable graph demonstrates a marked enhancement in performance. The MFG efficiently integrates motion features from associated trajectories, leading to a significant decrease in minADE by 0 . 15 . The ALG incorporates interaction feature from lane segments and the CIG merges dense interaction features from cooperative trajectories, resulting in a maximum reduction of 0 . 51 in minFDE and 7% in MR. In summary, these components contribute significantly to performance improvements by learning heterogenerous cooperative feature representations.

C.2 Effectiveness of Cooperative Representation

We further evaluate the effectiveness of the model in cooperative feature representations on V2X-Seq by alternately masking out the infrastructure-view trajectories in each component. As demonstrated in Table 9, the customized cooperative representation learning of each component results in performance improvements to a certain degree. Concretely, the MFG effectively integrates the motion features of associated trajectories through explicit associations and implicit temporal-spatial encoding. This component leads to improved performance in motion forecasting for each future frame, as evidenced by the decrease of 0.15 in the minADE metric. Relatively, the ALG and CIG components fuse the interaction features of lane segments and cooperative trajectories through compact spatial encoding. These component primarily enhances the performance of long-term intention prediction, as indicated by a maximum reduction of 0 . 20 in the minFDE and 3% in the MR metrics.

1.1

1.2

1.3

1.4

1.5

1.6

5k

10k

15k

20k

25k

minADE

1.9

2.1

2.3

2.5

2.7

5k

10k

15k

20k

25k

minFDE

Vehicle-only

Cooperation

0.28

0.30

0.32

0.34

0.36

0.38

0.40

0.42

5k

10k

15k

20k

25k

MR

C.3 Scalability Study

Compared with vanilla motion forecasting, collecting large-scale data in cooperative scenarios for cooperative motion forecasting poses some realistic limitations. Therefore, it is essential to study how the performance of the method scales with dataset size.

In this experiment, we randomly split data of different sizes from the training set of V2X-Seq to train the V2X-Graph framework and evaluate the model on the full validation set. We compare the proposed V2X-Graph framework in both vehicle-only and cooperative settings to assess how the scalability of the proposed framework for cooperative motion forecasting varies with dataset size.

As shown in fig. 8, as the amount of training data increases, both vehicle-only and cooperative settings achieve improved performance. Notably, V2X-Graph achieves greater advantages in cooperation, which illustrates the scalability of the framework in cooperative motion forecasting.

C.4 Robustness of Latency and Data Loss

We conduct robustness experiments on V2X-Seq dataset, taking both data synchronization problem and communication latency into consideration.

Specifically, we simulate latency by dropping the latest one or two frames of infra-view data during transmission. And we address the latency issue with a simple interpolation to obtain synchronized trajectory data. Experiment results in table 10 shows that there is little performance degradation to communication latency.

Data loss and sensor failures are common practical challenges. The performance advantage on real-world datasets demonstrates the robustness of our method. We further evaluate V2X-Graph under different data loss ratios on V2X-Seq dataset. Specifically, we randomly drop perception results data in each frame from the infra-view in transmission with various dropping ratios. As shown in

table 11, the forecasting performance decreases as the ratio increases, but it is worth mentioning that our method outperforms the compared methods (table 1 on page 8) even under extreme conditions with a 50% loss rate.

C.5 Parameter Size and Inference Cost Comparison

As shown in table 12, the parameter size of V2X-Graph is comparable with other forecasting methods.

Then we conduct the inference experiment on single NVIDIA GTX 4090 and compare the inference cost. As for the experiment results shown in table 13, the proposed V2X-Graph is even faster than the compared vanilla motion forecasting methods, benefiting from the synchronous temporal state modeling and integration.

D Additional Implementation Details

In this section, we provide implementation details of compared methods in our experiemnts.

Implementation Details of HiVT. We compare our proposed V2X-Graph with the official evaluation of the V2X-Seq dataset [51] from https://github.com/AIR-THU/DAIR-V2X-Seq. For fair comparison, we re-implemented a larger model of HiVT [58] with a hidden size of 128 for improved performance as reported in the corresponding paper.

Implementation Details of DenseTNT. For comparison, we re-implemented the classical homogeneous graph method DenseTNT [13] on our V2X-Traj dataset using their official code package, from https://github.com/Tsinghua-MARS-Lab/DenseTNT. The model is trained using the default settings of two stages on the V2X-Traj training set, utilizing a server with 8 NVIDIA RTX 3090s. During the first stage, all modules are trained, except for the goal set predictor, for 16 epochs. In the second stage, the goal set predictor is trained for 6 epochs. The batch size is set to 64, the initial learning rate is 0.001, and it decays by 30% every epoch. The hidden size of the feature vectors is set to 128, and the head number of our goal set predictor is 12.

Implementation Details of HDGT. We further re-implemented the advanced heterogeneous graph method HDGT [19] based on their official code from https://github.com/OpenDriveLab/HDGT. We

re-implemented a larger model of HDGT with a hidden size of 256, and the number of heads in all multi-head attention blocks is 64, for better performance for comparison. The size of the kernel of the AgentTemporalEncoder in HDGT is set to 40-13-3 to accommodate the specific observation horizon of 40 time steps in V2X-Traj. During training phase, we follow the default settings, which uses the AdamW optimizer with an initial learning rate of 5 × 10 -4 , weight decay of 1 × 10 -4 , and batch size of 64. For the V2X-Traj dataset, the number of training epochs is set to 30, with a 1-epoch warmup and linear decay to 0. The type-specific agent distance buffer hyperparameters are empirically set to 30 meters for vehicles, 10 meters for pedestrians, and 20 meters for cyclists.

E Additional Qualitative Results

In this section, we present qualitative results on V2X-Seq dataset, including the visualizations of interpretable association (appendix E.1) and the visualizations of our proposed V2X-Graph and compared method (appendix E.2).

E.1 Visualizations of Interpretable Association

Here we present the visualizations of the interpretable association on V2X-Seq. For clarity, we only visualize the associations of the historical trajectories of the target agent from the ego-view (fig. 10 (a)) and the infrastructure-side (fig. 10 (b)). Dashed circles indicate a part of the additional information from infrastructure-side trajectories that can be utilized. As shown in the figure, our method enables the interpretable association of trajectories across views, serving as guidance for the end-to-end learning of cooperative trajectory representations.

E.2 Qualitative Results on V2X-Seq

We present additional qualitative results on the V2X-Seq dataset; three challenge scenarios are selected for methods comparison. In particular, the motion forecasting results of HiVT [58], which employs PP-VIC [51] to utilize cooperative information, are shown in fig. 10 (c), and the results of our proposed V2X-Graph are shown in fig. 10 (d). In the methods comparison, our method exhibits exceptional performance in motion forecasting, in particular of predicting long-range intentions, which demonstrates the ability of proposed V2X-Graph for further utilization of information within cooperative trajectories.

Ego-View

Infra-View

Veh-View

Scenario 1

Scenario 2

Scenario 3

Scenario 4

Scenario 1

Scenario 2

Scenario 3

(a) Ego-View

Trajectory

(b) Infra-View

Trajectory

(c) Compared

Method

(d) V2X-Graph

(Ours)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and introduction, we claim the contributions made in cooperative motion forecasting in this paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In section 6, we discuss the contributions of cooperative motion forecasting in this paper and the relationship of the proposed method to the vanilla forecasting task. In appendix C.3, we discuss how the performance of the method scales with dataset size.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide details of the proposed method in section 4 and describe the experimental settings in section 5. The code is provided in the supplemental material and will be released along with the dataset once accepted.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide details of the experimental settings in section 5. The code is provided in the supplemental material, the code and the dataset will be released along with sufficient instructions once accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide details of the experimental settings in section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars are not formally reported because it would be too computationally expensive. However, we conduct the experiment multiple times and record universally representative results to support the main claims of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide details of the computational resources in section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research in this paper conforms to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We hope both the V2X-Graph framework and V2X-Traj dataset can facilitate the further development of cooperative motion forecasting.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: The proposed real-world dataset does not contain personal information and has been encrypted to mitigate related risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The related code and data in this paper have proper licenses and have been cited appropriately.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide details of the new real-world dataset in appendix B, and the local government has granted permission for the release of this dataset.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces V2X-Graph, a novel framework using trajectories of agents (include ego and others) and vector map as inputs for cooperative motion forecasting that fuses trajectory features in an interpretable, end-to-end manner. The authors evaluate V2X-Graph using V2X-Seq, a vehicle-to-infrastructure (V2I) motion forecasting dataset. Additionally, they create V2X-Traj, the first real-world dataset for vehicle-to-everything (V2X) motion forecasting, including scenarios with multiple autonomous vehicles and infrastructure. Author shows the V2X-Graph is achieving state-of-the-art results on both V2X-Seq and V2X-Traj datasets.\n\nSTRENGTHS:\n1) The paper is well-written, with a clear introduction of methods and detailed explanations of the experiments conducted.\n2) This paper proposes a novel framework to address motion forecasting in V2X scenarios that not only encodes multiple trajectories with map information but also designs additional graphs for heterogeneous feature representations. Moreover, the paper discusses the effectiveness of different components.\n3) This paper introduces the V2X-Traj dataset, which includes rich information such as trajectories, traffic lights, and maps, making it a valuable resource for the community to study V2X scenarios.\n\nWEAKNESSES:\nThe proposed method emphasizes V2X; however, the inputs do not include important factors such as traffic lights, lane markings, and pedestrians, which may significantly impact the future motion of the vehicle. There is concern that V2X-Graph oversimplifies the problem.\n\nQUESTIONS:\n1. There are more works of GNNs for motion forecasting such as [1][2][3].\n2. What’s the efficiency or latency of this framework given it has quite complicated network components?\n3. Please check the weakness.\n\n[1] Mohamed, Abduallah, et al. \"Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n\n[2] Li, Jiachen, et al. \"Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning.\" Advances in neural information processing systems 33 (2020): 19783-19794.\n\n[3] Girase, Harshayu, et al. \"Loki: Long term and key intentions for trajectory prediction.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\nLIMITATIONS:\nThe paper discusses the limitations of the work in section 6.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the V2X-Graph method for cooperative motion forecasting. In a cooperative autonomous driving setting, an autonomous driving vehicle receives sensor data from surrounding vehicles and infrastructure-side devices.\n\nExisting cooperative autonomous driving approaches focus on perception completion, where the vehicles receive only perception data, and motion forecasting is performed with completed perception data without considering the cooperative setting.\n\nThe proposed V2X-Graph method is a graph-based framework that performs cooperative motion forecasting with trajectory feature fusion. It fuses history trajectories from different views with an Interpretable Graph. The Interpretable Graph consistents of three subgraphs: Motion Fusion subGraph (MFG), Agent-Lane subGraph (ALG), and Cooperative Interaction subGraph (CIG).\n\nThe authors also provide a new dataset V2X-Traj. Compared to the existing V2X-Seq dataset which contains only vehicle-to-infrastructure (V2I) cooperative scenarios, V2X-Traj also includes vehicle-to-vehicle (V2I) cooperative scenarios, making it vehicle-to-everything (V2X).\n\nThe authors evaluated their V2X-Graph method on the existing V2X-Seq dataset as well as their new V2X-Traj dataset. They compared against the PP-VIC baseline. The result shows that adding trajectory feature fusion improves the performance over the PP-VIC baseline.\n\nSTRENGTHS:\n* The proposed V2X-Graph method improves the cooperative motion forecasting performance over the PP-VIC baseline.\n\n* The new V2X-Traj dataset will be very useful to the community.\n\nWEAKNESSES:\n* I am not fully convinced by the motivation of why perception completion is not enough for cooperative motion forecasting. It will be useful to give some examples to explain how feature fusion helps.\n\n* The writing can be improved. For example, it will be useful to give an overview of the architecture either in the caption of Figure 2 or at the beginning of Section 4. The module names MFG, ALG, CIG, etc in the architecture diagram are not explained until Section 4.2. The evaluation setup of Table 2 (Graph-based methods comparison) is not clearly explained.\n\nQUESTIONS:\n* Table 1: Why are the PP-VIC + HiVT results different from those in the original PP-VIC paper?\n\n* Line 284: How are the observations of the same agent from different views aggregated?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-05"}

comments: SUMMARY:
This paper tackles the cooperative motion forecasting problem for vehicles. This paper introduces additional information other than the ego view agent from other view to expand the perception field of the prediction. The authors propose a graph network to extract information for multimodal trajectory prediction. The authors introduce a new dataset, namely V2X-Traj, for this problem setting. Experiments show the efficacy of the proposed method.

STRENGTHS:
1. The problem setting is new. With more and more autonomous vehicles on the road, it is possible to include cooperative information.
2. The experimental results on both datasets show that V2X-Graph outperforms existing methods.

WEAKNESSES:
1. The cooperative setting is limited to two vehicles. 
2. The scalability of the V2X-Graph framework in terms of computational resources and its performance with larger datasets or in more complex environments is not extensively discussed. The training cost and inference cost should be compared with the baselines.
3. There is not error analysis. When does the method/cooperative setting fail and why?
4. Potential cooperative forecasting problems are not considered. How does the V2X-Graph framework handle communication delays or data loss in real-time applications?

QUESTIONS:
Please address the weaknesses.

---Post rebuttal
Thank you for the author's response. The reply addressed most of my concerns and the added experiments on inference efficiency are important. I will increase my score

LIMITATIONS:
The authors have discussed the potential limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel graph-based framework called V2X-Graph for learning cooperative trajectory representations in motion forecasting for autonomous vehicles. V2X-Graph aims to enhance the motion prediction capabilities of autonomous vehicles by leveraging cooperative information from vehicles and traffic infrastructure. The framework represents trajectories as nodes in a graph and fuses motion and interaction features from multiple views to achieve interpretable end-to-end trajectory feature fusion. The authors also constructed the first real-world V2X motion forecasting dataset, V2X-Traj, which includes cooperative scenarios with multiple autonomous vehicles and traffic infrastructure. Experiments conducted on both the V2X-Seq and V2X-Traj datasets demonstrate the advantages of V2X-Graph in utilizing additional cooperative information to enhance motion forecasting capabilities.The creation of the V2X-Traj dataset, a real-world, public dataset for V2X motion forecasting, adds significant value to the field, promoting further research in this domain.\n\nSTRENGTHS:\n- The paper introduces a novel forecasting-oriented representation paradigm, V2X-Graph, which leverages cooperative perception information from multiple sources (e.g., infrastructure and other vehicles) to enhance motion forecasting for autonomous driving. This approach is unique in focusing on the holistic historical motion and interaction features rather than single-frame perception.\n- The methodology section (pages 4-6) is detailed and provides a comprehensive description of the V2X-Graph framework, including scene representation with graphs and trajectory feature fusion. The use of graph neural networks (GNNs) to encode cooperative scenarios and perform motion forecasting is well-justified.\n- The paper includes extensive experimental results (pages 8-10) on both the V2X-Seq and V2X-Traj datasets, demonstrating the effectiveness of the proposed approach with clear performance improvements over baseline methods.\n- The paper is well-structured, with a clear flow from the introduction of the problem (pages 1-2), through the related work (pages 2-3), to the proposed methodology and experimental results. Each section builds logically on the previous one, making the paper easy to follow. Figures and tables (e.g., Figure 1 on page 3 and Table 1 on page 8) are effectively used to illustrate key concepts and results, enhancing the reader’s understanding.\n- By addressing the underutilization of cooperative perception information in motion forecasting, the paper tackles a critical challenge in autonomous driving. The proposed V2X-Graph framework has the potential to significantly improve the safety and efficiency of autonomous vehicles by leveraging a richer set of perception data.\n\nWEAKNESSES:\n- While the application of graph neural networks (GNNs) to encode cooperative scenarios is well-executed, the approach may lack significant novelty. GNNs have been widely applied in various fields, including autonomous driving. The paper would benefit from a more detailed comparison with existing GNN-based methods for motion forecasting to highlight the unique contributions and advancements of the proposed V2X-Graph framework (page 6, Section 3.2).\n- The experimental evaluation compares the proposed method primarily against a few baselines. Including a broader range of state-of-the-art methods for motion forecasting, especially those utilizing cooperative perception, would strengthen the validation of the proposed approach. This would provide a more comprehensive view of where V2X-Graph stands in the context of current research.\n- Limited Discussion on Real-World Implementation: lacking a thorough discussion on the practical challenges and considerations of implementing the V2X-Graph framework in real-world autonomous driving systems. Issues such as communication latency, data synchronization, and robustness to sensor failures are crucial for practical deployment but are not sufficiently addressed (page 10, Section 5, Discussion).\n- Some parts of the methodology section are dense and might be challenging for readers to follow, particularly those less familiar with GNNs. Simplifying the mathematical notation or providing additional explanatory text and illustrative examples would improve clarity and accessibility\n\nQUESTIONS:\nWhile the experiments demonstrate the effectiveness of V2X-Graph on the V2X-Seq and V2X-Traj datasets, these datasets may not capture the full range of scenarios encountered in real-world driving. Expanding the evaluation to include more diverse and challenging driving environments would enhance the generalizability and robustness of the proposed method.\nV2X-Traj dataset, while a valuable contribution, might have inherent biases that are not discussed in the paper. Providing a more detailed analysis of the dataset's composition, including the diversity of scenarios and the representativeness of different driving conditions, would be beneficial (page 7, Section 4.1).\n\nLIMITATIONS:\n- The paper briefly mentions some limitations in the discussion section, but it does not provide an in-depth analysis. The authors should more thoroughly acknowledge the limitations of their approach, including potential biases in the V2X-Traj dataset and the constraints of their methodology. Specific limitations such as the scalability of the V2X-Graph framework in highly dynamic and dense traffic environments, potential issues with data communication latency, and the synchronization of information from multiple sources are not adequately discussed. Including a more detailed assessment of these factors would be beneficial.\n\nOthers:\n- The paper should address practical challenges such as the computational overhead required for real-time processing, the robustness of the system in adverse weather conditions, and the reliability of V2X communication in various urban and rural environments.\nDiscussing potential solutions or future work aimed at overcoming these challenges would strengthen the practical applicability of the proposed method.\n\n- While the paper makes a significant contribution to the field of cooperative motion forecasting, it would benefit from a more thorough discussion of its limitations and potential negative societal impacts. By addressing these aspects in detail and providing constructive suggestions for mitigating these issues, the authors can enhance the robustness, and applicability of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}