Learning Cooperative Trajectory Representations for Motion Forecasting
Authors: Hongzhi Ruan
Haibao Yu
Wenxian Yang
Siqi Fan
Zaiqing Nie
Venue: NeurIPS 2024 poster (2024)
Keywords: Cooperative Autonomous Driving
Motion Forecasting
Primary area: robotics

Abstract: Motion forecasting is an essential task for autonomous driving, and utilizing information from infrastructure and other vehicles can enhance forecasting capabilities.
Existing research mainly focuses on leveraging single-frame cooperative information to enhance the limited perception capability of the ego vehicle, while underutilizing the motion and interaction context of traffic participants observed from cooperative devices. 
In this paper, we propose a forecasting-oriented representation paradigm to utilize motion and interaction features from cooperative information. 
Specifically, we present V2X-Graph, a representative framework to achieve interpretable and end-to-end trajectory feature fusion for cooperative motion forecasting. 
V2X-Graph is evaluated on V2X-Seq in vehicle-to-infrastructure (V2I) scenarios.
To further evaluate on vehicle-to-everything (V2X) scenario, we construct the first real-world V2X motion forecasting dataset V2X-Traj, which contains multiple autonomous vehicles and infrastructure in every scenario.
Experimental results on both V2X-Seq and V2X-Traj show the advantage of our method. 
We hope both V2X-Graph and V2X-Traj will benefit the further development of cooperative motion forecasting.
Find the project at https://github.com/AIR-THU/V2X-Graph.

Learning Cooperative Trajectory Representations for Motion Forecasting
Hongzhi Ruan
Haíbao Yu 1,3+ Wenxian Yang
Siqi Fan
Zaiqing Nie
Institute for Al Industry Research (AIR), Tsinghua University
University of Chinese Academy of Science
The University of Hong Kong
hongzhi www.nyynnnnn....ooommmmm com, yuhaibaof40gaail.com zaiqingfair. tsinghua, edu,
Abstract
Motion forecasting is an essential task for autonomous driving, and utilizing infor- mation from infrastructure and other vehicles can enhance forecasting capabilities. Existing research mainly focuses on leveraging fingle-frame cooperative informa- tion to enhance the limited perception capability of the ego vehicle, while under- utilizzig the motion and interaction context of traffic participants observed from cooperative devices. In this paper, we propose forecasting orienttd representation paradigm to utilize motion and interaction features from cooperative information. Specifically, we present V2X- Graph,, a representative framework to achieve inter- pretable and end-to-end trajectory feature fusion for cooperative motion forecasting. V2X- 22X-Graph is evaluated on V2X-Seq in pooll--t-inffasttuctum (V2I) scenarios. To further evaluate vehicle-to-everythin real-world V2X motion forecasting dataset V2X-Traj. which contains multiple autonomous vehicles and infrastructure in every scenario. Experimental results on both V2X-Seq and V2X-Tral Traj show the advantage of our method. We hope both V2X Graph and V2X-Traj will benefit the further development of cooperative motion forecasting. Find the project at https//gittbb.oom//iRRTTHUUA Graph
(V2X) scenario, we construct the first
Introduction
In recent years, autonomous driving has made significant progress. However, single-vehicle au- tonomous driving still faces substantial safety challenges due to its limited perception ability. Utiliz- ing external information, such as data from other connected autonomous vehicles and infrastructure sensors through vehicle-to-evee toveertt-n (V2X), has shown great potential to enhance autonomous driving capabilities. this paper, we focus on motion forecasting, a fundamental task for autonomous driving that has received significant attention in recent years [40
33 31 19]. Specifically, considering
currently practical communication conditions, we transmit perception results and input trajectories from the ego vehicle and external sources for cooperative motion forecasting.
Cooperative motion forecasting involves the ego vehicle aggregating its data with data transmitted from other connected vehicles or infrastructure devices to predict future waypoints for each agent in traffic scenarios. To accommodate limited communication conditions, consider data in the form of perception results. These perception results form historical trajectories of agents from respective views, termed cooperative trajectories. The autonomous vehicle utilizes these trajectories to enhance its motion forecasting capabilities. High-definition (HD) maps are also used in this task.
To leverage cooperative trajectories for improving motion forecasting performance, two critical issues must be addressed: (1) Observations of the agents from different views may different due various sensor perspectives and configurations; (2) In the cooperative scenario, there are multi-
Conresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Other Veew
Stew
Cge
Profitiio
Perception Complrioon
Fig Viww
(6) Exiiting Couperative Nechol
Other View
Medion
Weladiin
Intepretable Ausodation
I| Cooperativs Trajectoyy Raprosntatiors Parudige
Figure 1: Scheme Comparison. (a) Existing methods utilize cooperative perception information at each frame individually then performs forecasting. (b) Our V2X-Graph considers this information from a typical forecasting perspective and employs interpretable trajectory feature fusion in an end- to-end manner, to enhance the historical representation of agents for cooperative motion forecasting.
view observations of multi-agents, the redundant data need to be leveraged interpretably. Existing research mainly focuses on single-frame feature fusion to support real-world applications and enhance detection performance [50
A recent method (51) attempts to complement perception over the historical horizon to improve forecasting performance. These methods are depicted in fig. [Ta.. However, the above single-frame frame methods obtain the agent state at each frame individually, which may lead trade-off between the states observed from distinct views, and cannot utilize motion and interaction context sufficiently, thus failing to sufficiently model the historical behavior of agents, Instead, considering historical observations from each view holistically could address these shortcomings. Compared to previous approaches, this paper explores novel forecasting -earsttinn-rreeteeddd oriented trajectory feature fusion method, which aims to enhance the historical representation of agents, including their historical motion and surrounding interactions, for motion forecasting.
To address the challenges and effectively utilize the cooperative information, we propose V2X-Graph, a graph-basee based framework to achieve cooperative trajectory feature fusion for motion forecasting. Theoretically. "2X-Graph offers two advantages for enhancing motion forecasting performance. (I) Forecasting-orientt -rrennted cooperative representation. For accurate motion forecasting, it is common practice to represent motion features from an agent's 's historical trajectory and interaction features from other agents 10
19 Instead of perception complement, our method is the first to consider cooperative perception information from the typical motion forecasting perspective, which independently represents motion and interaction representations of cooperative trajectories, customized relative spatial-temporal encodings are designed to support trajectories feature fusion of each agent over historical horizon. (2) Graph-guided heterogeneous feature fusion. To support multi-agent motion forecasting in a cooperative scenario, is essential to interpretably integrate heterogencous motion and interaction features for each specific agent from cooperative trajectories. Drawing inspiration from graph link prediction [12 (55. a classical task that analyzes the relationship of nodes in graph, this may further support downstream applications like node feature propagation. To achieve end-to-end optimization in V2X-Graph, we constructed a graph that represents the cooperative scenario, interpretable association is established to guide heterogeneous feature fusion based on agent identification across views. The framework is represented in fig. (b).
25
58
V2X- is evaluated on V2X-Seq [51], which contains vehicle-to-infrassrnccuu (V2I) cooperative scenarios. evaluate its effectiveness in vehicle-to-vehick (V2V) and further more cooperative devices scenarios, we construct the first public and real-world vehicle-oo-everything (V2X) motion forecasting dataset V2X-Traj This dataset is the first to include multiple autonomous vehicles and infrastructure in every scenario, broadening the research devoted to V2X motion forecasting task. Extensive experiments conducted on V2X-Seq and V2X-Traj show the advantages of V2X-Graph in utilizing additional cooperative information to enhance the motion forecasting capability.
Our contributions are four fold: (1) We propose forecasting -orienttig--oreeneed oriented representation paradigm to utilize motion and interaction features from cooperative information. (2) We design 12X-Graph, a representative framework to achieve interpretable and end-to-end trajectory feature fusion for cooperative motion forecasting.
(3) We construct V2X-Traj. which is the first public and real-world dataset for V2X motion forecasting. It includes not only V2I but also V2V cooperation in every cooperative scenario. (4) Our approach achieves state-of-the-art on both V2X-Seq and V2X-Traj.

---

2 Related Work
Cooperative Autonomous Driving. In recent years, more and more researchers pay attention to cooperative autonomous driving, which leverages additional information from infrastructure-sinn devices and other vehicles to achieve system-wide performance improvement. As several public cooperative perception datasets [45 49 15 have been released. most of them focus on cooperative perception. Different from the single-side object detection [56] 24
8 47 48) cooperative detection methods aim to promote performance and transmission latency trade-offs to support real-world
9 4 Some works also dive into cooperative segmentation task [38 43]. However, as the downstream task of cooperative perception and directly influences the actions of autonomous vehicles, cooperative motion forecasting has not been well studied. A recent endeavor supplies historical observations with perception information from infrastructure devices to improve motion forecasting performance [51]. Instead of perception completion then forecasting, this paper presents an end-to-end cooperative motion forecasting framework for cooperative trajectory feature fusion, to achieve comprehensive utilization of motion and interaction contexts from cooperative information. To further broaden the research into V2X cooperative motion forecasting, we construct the first real-world and public motion forecasting dataset for general V2X scenario, termed V2X-Traj. including not only V2I but also V2V cooperations in every scenario.
applications [46 22
44
41
50
Motion Forecasting, Motion forecasting is an indispensable task in autonomous driving systems, which takes sequential perception results of agents and map elements into account to predict future trajectories of agents. Early works rasterize scenarios as images and deploy convolution neural networks to extract information 34
[21] The research community turns to vectorize the represen- tations of agents and maps for motion and interaction contexts [T0]. Some works consider pooling mechanism for feature fusion [14 1O 35) Others utilize the convolution technique to extract local
features 25 52 5) Inspired by the effectiveness and widespread usage of the Transformer model [36]. recent works adopted attention mechanism for learning representations in motion forecasting
40] Instead of sequential perception supplementation then forecast- ing, V2X-Graph explores novel trajectory feature fusion to comprehensively utilize information. represents and decouples trajectory features into motion and interaction features for each view independently, and employs customized Transformer modules for aggregating interpretable features based on agent identification, which facilitates cooperative motion forecasting.
task 126 39
58
33
19
GNNs for Motion Forecasting. Graph neural network (GNN) 20 37 is common structure for motion forecasting. A graph consists of nodes and edges, with each node typically representing information related to an agent or map element. While edges represent the relative information between pairs of nodes. The message-passing mechanism aggregates and updates node features from their neighbors. Previous methods adopted homogeneous GNN for unified but coarse scene 57 25 [.]. While recent research introduced heterogeneous GNN [54]
representation [30 10 23
16) to distinguish and further extracting features based on various settings of agents [32 29 I8 19] Compared to previous approaches, V2X- V2X-Graph explores leveraging heterogeneous edge encodings and interpretable graph link prediction for trajectory- based motion and interaction features fusion.
3 Preliminary
Cooperative motion forecasting can play an important role for autonomous driving as it better reasons the future movements of surrounding agents. The ego vehicle receives sequential perception results from cooperative devices, including infrastructures and other vehicles, to enhance the capability of motion forecasting. The contextual information in the vector map is also taken into account.
Problem Formulation. The inputs of cooperative motion forecasting include multiple-soure trajectories and vector maps. The cooperative motion forecasting scenario is represented as S LJ, where T and L are described as follows. (1) Trajectory. In a typical V2X cooperation scenario, each cooperative device independently captures the historical status of agents as trajectories. The multi-source trajectories are denoted as T = {T ego: Tother) }. here T Tother can include received multi-source cooperative trajectories such as Tan and och from the views of infrastructure and cooperative vehicles. The total number of trajectories is N₂ = Nego + Nother- Trajectory information is summarized as T E R N₂ XTxx, where T is the historical horizon and C₁ is the attributes of each trajectory to depict corresponding agent (e.g., tracking id, location, heading angle, detection bounding box and agent type). Specifically, the historical spatial status of each trajectory is formulated as 1D), where pf E RR2 the trajectory i's location, and r/ represents the heading theta vector at time step t. (2) Vector Map. Vectorized representation [10 is usually adopted for representing the
3

---

map elements in motion forecasting task, which leverages the sample points of the centerline within each lane and enables an efficient vectorized representation of spatial information. In this paper, we further consider vectorized lane segment, i.e., the vector between each two neighboring sample points. The set of vectorized lane segments denoted as: L E R--xxxxC, where N₂ is the number of lane segments and Cj is the attributes of each lane segment (e.g., location and road type). The start point and the end point of the lane segment are formulated as {p start p, j. here E [1, 2..,,...,NN Ni].
The output is K. future trajectories of the specified target agent in each scenario, the best one is chosen for evaluation, here = 6. Evaluation metrics are minADE, minFDE and MR, standard metrics for motion forecasting. Lower number is better.
Evaluation.
Challenges. To enhance the capability of motion forecasting considering abundant cooperative trajectories, it is essential to: (1) effectively represent the cooperative scenario, (2) efficiently utilize valuable information from redundant cooperative trajectories.
Methodology
This section presents V2X-Graph, a graph-based framework designed to achieve interpretable tra- jectory feature fusion for cooperative motion forecasting. To represent the cooperative scenario, constructs a graph with node and edge encodings. To enhance cooperative trajectory feature fusion, an interpretable graph consisting of three subgraphs is designed for the aggregation of heterogeneous motion and interaction features. The overall architecture is depicted in fig.
4.1 Scene Representation with Graph
In the graph that represents the cooperative scenario, trajectories from each view and their corre- sponding lane segments are independently encoded as nodes, while the relative spatial and temporal features between these nodes are encoded as edges.
Graph Node Encodings. We encode the node features from three perspectives: trajectory motion features, trajectory spatial-temporal features, and lane segment spatial features.
Compared to previous methods that leverage cooperative information at each frame individually, encode differential information from each view then fuse it with correlation to mitigate the deviation caused by direct single-frame fusion. The embeddings of differential coordinates {p¹ P - of the trajectory are considered as the motion features at each timestep. The self-attention mechanism (6) is adopted to incorporate temporal dependency. Here, missing frames are padded with learnable tokens, and the attention mechanism is enforced to only attend to the preceding time steps.
For the purposes of trajectory identification for interpretable association and motion correlation measurement, we also encode temporall features for trajectories from each view. The spatial- temporal features are encoded by incorporating the temporal dependency between the ego-centric normalized coordinates of the trajectory. Missing frames are masked in the attention module.
Overall, node encodings of trajectory from each view are formulated
mot V
= SelfAtt((MLP(1) (pi p -)) + PE"), V = SelfAtt(MMP((p)) + PE'), (I)
where PE signifies the learnable positional embedding timestep 1, SelfAut(()) .) is the multi-head self-attention module, and MIP((-) represents multi-layer perceptron.
To enhance the representation of trajectories for future intention reasoning, the feature of the vector map structure incorporated. Specifically, the spatial feature of lane segments are represented by the relative coordinates (from the start point to the end point of each lane segment) in an agent-centric frame [35], and further encoded nodes for feature aggregation. The formulation is:
map
(P pi ))t
(2)
where r., is the relative heading vector between trajectory with lane segment in its current frame.
Graph Edge Encodings. For effective trajectory feature fusion, design heterogeneous edge encodings, including spatial-temporal encoding and relative spatial encoding.
We use an attention module to aggregate the spatial-temporal encodings between each pair of cross- view trajectories. The spatial-temporal encoding captures the spatial-temporal correlations between two trajectories at each timestep, which facilitates motion feature fusion.

---

Input
Interpretable Graph
Trajectory Feature Fusion
Motion Festure
Interaction Feature
Interpretable
Graph Link Prediction
Other View Tocker
Adjarent
Spattsi-bemperal Alteniion
Output
MFG Siee
CIG
Teps
Multi-agent Prediction
Trajectory Decoder
0 Map
ST-ddges
Filges
Agont Frurm
Figure 2: V2X-Graph overview. Trajectories from the ego-view and other views, along with vector map information, are encoded nodes and edges for graph construction represent a cooperative scenario. The novel interpretable graph provides guidance for precasting-orrinter trajectory feature fusion, including motion and interaction features. this figure, solid rectangles represent encodings of ego-view trajectories, hollow circles represent encodings of cooperative trajectories, distinguished by distinct colors. Specifically, within the same view, the use of the same color indicates interruptions caused by occlusion. Triangles represent encodings of lane segments. In trajectory feature fusion, grey arrow indicates an missing frame motion case, lane segment vector in interaction case.
To capture the interaction features, we also introduce relative spatial encodings as edges.
Edge encodings can be formulated as follows:
SelfAtt(concat(( (",", v) C - MLP -p)),,
(3)
where e[= represents both agent-agent and agent-lane interaction relations. For agent-agent inter- action, the coordinates of the current frame for trajectories are denoted by p, and py. While for agent-lanc interaction, the encoding represents the feature between current coordinate p, of trajectory i and the starting point coordinates pf pttrtt of the lane segment.
4.2 Feature Fusion with Interpretable Graph
To achieve comprehensive historical representations of agents in cooperative scenario, in- terpretable graph is designed for multi-view trajectory feature fusion. Serving as a guidance for heterogeneous feature representations, the Interpretable Association component (IA) establishes plicit associations between trajectories of the same agent across views. The Motion Fusion subGraph (MFG) represents cooperative motion features by considering both explicit associations and implicit spatial-temporal encodings. The Agent-Lane subGraph (ALG) fuses the features from each view with lane segment features. The Cooperative Interaction subGraph (CIG) represents dense interaction representations by leveraging the spatial encodings between different agents from all views.
Algorithm 1: Pseudo Labels Generator
Input: Ego-view trajectories T Tego, Other-view trajectories Tother
Output: Cross-views trajectories matching pesudo labels Á
for f. 0 to do
Bego Bother + Collect ego-view, other-view detection bounding boxes from Tt T ego other
Calculate bounding boxes IOU matrix M, E |
A' + Solving the optimal matching: Hungarian Algorithm(M'))
Update greedy trajectory matching at t: Ar:
Á 4- Solve error matching with length intersection threshold Clnngth from A.
Interpretable Association. achieve end-to-end optimization of heterogeneous feature fusion, we formulate the association process as a graph link prediction problem. An interpretable association component is introduced establish interpretable associations of cross-view trajectories of agents, providing explicit guidance for the fusion of motion and interaction features. Additionally, we propose

---

pseudo label generator, described in alg. [ to collect trajectory matching labels for trajectories from the ego and another view over the historical horizon within the training set for knowledge distillation.
We denote an adjacency matrix for the two sets of trajectories = {as.yli E Nego, E Nothher ) where elements referring to associations are supervised by A which can be predicted as:
Classifer (c ,) E {0, } [egolx[[Notterl
(4)
here Cllassifier refers to a MLP for binary classification to determine whether there exists an association between two trajectories, thereby instructing the perception information belonging to the same agent. The classification is based relative spatial-temporal encodings of trajectories across views.
Motion Fusion SubGraph. To comprehensively represent the historical motion of agents, we employ the Motion Fusion subGraph (MFG) to aggregate cooperative motion feature representations from view associated trajectories. MFG models cooperative motion representations by incorporating both interpretable associations and temporal- -spatal correlations.
MFG defined as: Gmfg = (V,E) E) and Amfy A, where the node set V (v:li E N.), and the edge set & = {ei.g) denotes the edges captured by Amfg- Feature fusion and update process is:
(k+1) FFN
(8) + CrossAttn( v
(R) V + +((b))),
(k)
where v
(k+1) represents the updated feature of node v. FFN is feed-forward network.
Agent-Lane SubGraph. We employ the Agent-Lane subGraph (ALG) to incorporate map informa- tion for cooperative motion forecasting. ALG is bipartile graph that enables agent motion features query relevant lane segment interaction features using relative spatial encodings. We consider all lane segments within the observation range of the current frame of agents from each view.
ALG is defined as: Galg (V,E), where V = {ur, w li of interaction feature aggregation and update from agents to lane segments can be formulated as:
Nr, Nr and & {(.,)) The process
(k+1) = FFN(v! + CrossAAtt(v (k) + c + ar)),
(6)
where represents learnable tokens of semantic attributes associated with the corresponding lane segment, such as turn direction and road type.
Cooperative Interaction SubGraph. The Cooperative Interaction subGraph (CIG) is employed for cooperative interaction features representation between trajectories of distinguished agents in all views. Incorporating both interpretable associations and relative spatial correlations, CIG models denser interaction representations in cooperative scenario.
CIG is defined as: Geig = (V, and Acig =~ A, where V= (o,é E Nt), E = {er,) denotes the of not associated cross-view trajectories, which is captured by the adjacency matrix A and all intra-view edges. The process of interaction feature fusion and update CIG formulated as:
v (k+1) FFN(v + CrossAttn
+ ef + ar,)))
where ai, represents the learnable features of interaction attributes associated between two trajecto- ries, such as relative headings at the current timestep. Note that CIG only considers the interaction among trajectories observed in the current frame.
4.3 Multimodal Future Decoder
The future motion traffic agents inherently multi-modal. We parameterize the distribution of future trajectories as Laplacian Mixture Model (LMM) following (58). Every agent from each view is predicted and supervised during training phase for representations of trajectory feature fusion. While during inference phase, the decoder predicts the future trajectory of distinct agents in the cooperative scenario from the ego-view, guided by interpretable associations. Technically, we employ a MLP as a prediction head aggregate all the intermediate features, which can be formulated as:
Z MLP(concat[(?? v! v. all (i
(8)
where Z.:7" includes K. Laplacian components Nix with multi-modal probability distributions PI:K- The formulation for predicting the future coordinate distribution of agent at time as:
Pa(o) Σ Pk .Nix Ons Hyg Tg. p):
(9)
k=
the future positions are generated by the center distributions. The distribution A (pas My) and corresponding probability pę generated by two MLPs separately.

---

Figure 3: V2X-Traj dataset. (a) Statistics of the total number and average length for the classes of agents. (b) Visualizations. Orange boxes represent autonomous vehicles, blue elements denote other traffic participants and the green box denotes the target agent needs to be predicted.
4.4 Training Losses
To achieve interpretable trajectory feature fusion, the framework is trained in an end-to-end manner with two components optimization objectives. The first part is the cross-entropy loss to optimize the graph link prediction. The second part includes the regression loss and classification loss to optimize the motion forecasting. Please refer appendix for more loss details.
Experiment
In this section, we evaluate the proposed V2X- y2X-Graph framework not only on V2I cooperative scenarios but also on V2V and broader V2X cooperative scenarios.
5.1 Experimental Setup
Dataset. V2X-Grapp evaluated on both V2I and broader V2X scenarios. (1) V2X-Seq 51.
public large-scale and real-world V2I dataset. V2X-Seq consists of 51,146 V2I scenarios, and each scenario is seconds long with the sample rate of Hz. The task is to predict the motion of agents for the next 5 seconds, given the initial 5-second observation from both infrastructure and ego-view. (2) V2x-Traj (Ours). study the effectiveness of V2X-Graph in V2V and broader V2X scenarios, especially its ability to handle more than two views of trajectories, including both V2I and V2V cooperation, we construct the first real-world and public V2X cooperative motion forecasting dataset, termed V2X-Traj. It comprises 10,102 scenarios in challenging intersections. Each scenario includes two intelligent vehicles and an infrastructure perception device. The statistics and visualization of V2X-Traj presented in fig- Each scenario lasts for 8 seconds with sample rate of 10 Hz. The 4econn observations from each view are used to predict the future motion in the next 4 seconds. We hope the V2X-Traj dataset can facilitate the development of cooperative motion forecasting for general V2X scenarios. More details are described in the Appendix.
Implementation Details. For scene graph representation, V2X V2X-Graph employs 4-layer temporal self-attention Transformer to encode motion features, a 2-layer temporal self-attention Transformer and 2-layer self-attention module for relative temporal-spatial feature encoding. In the interpretable graph, there are 3 layers of MFG, layer of ALG and 3 layers CIG. The dimensions of the hidden feature is set 128, and the number of heads in all multi-head attention blocks is 16. The lane segments corresponding to agents within a observation range of 50 meters are taken into consideration. For training, the initial learning rate is set to x 10 and is scheduled according cosine annealing 27]. The AdamW W optimizer (28) is adopted with a weight decay of x 10 10-4, The model is trained for 64 epochs with batch size of 64 on server with 8 NVIDIA RTX 4090s.
5.2 Main Results
Cooperative method comparison. We compare our method with other cooperative methods on V2X-Seq. To the best of our knowledge, PP-VIC (5T) is the only existing method of the same type. Typically, PP-VIC provides the ego vehicle with infra-side perception information in a frame-by- frame manner within the historical horizon. After supplementation, the perception information provided to popular and competitive vanilla forecasting methods DenseTTNT [3] and HiVT (58]. For comparison, we also provide the output of VPC V2X-Graph in a similar way to the ego- view perception. As shown in [able perception completion enhances the downstream forecasting performance of HiVT and V2X- P2r-Gr However, the trade-off in cross-view perception also leads to error propagation, resulting in performance degradation of DenseTNNT. Instead, V2X-Grap enhances the historical representation of agents through trajectory feature fusion, leading to performance improvements, evidenced by -0.07 in minADE, -0.19 in minFDE, and 55% in MR.

---

Table 1: Cooperative method comparison on V2X-Seq.
DenseTNT B
[S8]
V2X V2X-Graph
Method
Vehicle-only
PP- [PTT
Vehicle-only
PP-VIC[ST
Vehicle-only
PP-VIC|SI]
Feature Fusion
minADE minFDE
1.71
1.84
1.28
1.12
1.16
1.12
1.05
2.43
2,56
2.15
1.97
2.04
1.98
1.79
MR
0.27
0.28
0.31
0.30
0.30
0.30
0.25
Table 2: Graph-based methods comparison on V2X- Traj.
Vehicle-only minlDE
V2V
V21
V2VRI
Method
minlDD
minADE
MR
[3
1.23
2.09
0.25
1.20
2,04
0.25
1.32
2.34
0.29
1.26
2.24
0.28
19
0.91
1.48
0.14
0.94
1.57
0.17
0.94
1.59
0.16
0.94
1.56
0.17
V2X-Graph
0,90
1,56
0.17
0,77
1.26
0.12
0.80
1,30
0.13
0,72
1,13
0.11
Graph-based methods comparison. To reflect the unique advantages of V2X-Graph, we conduct evaluations on V2X-Traj and compared with representative graph-base methods. Similar to V2X- Graph, cooperative trajectories are encoded vanilla nodes of agents in each compared methods, for fair comparison. Experimental results in four typical settings are reported in table 2 As shown in the table, HDGT [9] achieves superior performance through precise heterogeneous design to represent relationship of agents, compared with Dense TNT [13], which employs homogeneous graph to represent the scenario. V2X- -K-Graph is also highly competitive in vehicle-only task, without sophisticated feature engineering and decoder design. However, our method outperforms compared methods by large margins in all cooperative settings and achieves the best performance in V2V& &I co- operation, with 0.22 in minADE, - -0.43 in minFDE, and - 6% MR, illustrating the effectiveness of aggregation heterogeneous motion and interaction features enhance cooperative forecasting.
5.3 Ablation Study
To further illustrate the effectiveness of the method for trajectory feature fusion and the final result, we conduct ablation studies the V2X-Traj validation set. Considering extensive ablation studies, experiments are conducted based on our small model with a hidden-size 64.
Effectiveness of Major Components. Firstly, we alternately removing one of the components to illustrate the contribution of each component to the cooperative motion forecasting perfor-
Table 3: Effect of major components.
MFG
ALG
CIG
minADE
minFDE
MR
mance. As shown in table 3] the components within the interpretable graph separately rep- resent the typical motion feature of historical states of agents and the interaction features with surroundings, demonstrating marked perfor- mance enhancement of motion forecasting.
1.16
2.30
1.26
2.60
0.30
1.08
2.07
0.25
0.95
1.79
0.21
Table 4: Effect of cooperative representations.
MFG
ALG
CIG
minADE
minHDD
MR
Effectiveness of Cooperative Representations. We further evaluate the effectiveness of the model in trajectory feature fusion by alternately masking the features of cooperative trajectories within each component. As demonstrated in t each customized component benefits tra- jectory feature fusion and results in performance improvements to certain degree. Specifically, the MFG effectively integrates the motion features of associated trajectories, leading to -0.08 in minADE. Relatively, the ALG and CIG components fuse the interaction features of lane segments and cooperative trajectories. These component primarily enhance the performance of long-term intention prediction, indicated -0.59 in minFDE and 7% in MR.
1.03
2.04
0.25
1.19
2.37
0.28
1.08
2.17
0.27
0.95
1.79
0.21
Effectiveness of Interpretable Feature Fusion. Moreover, we evaluate the effectiveness of the proposed interpretable graph in aggregating heterogeneous motion and interaction features within cooperative trajectories. In table 5 the first line presents the result of the vehicle-only setting, which has no cross-viee view motion and interaction features fusion. We simply employ fully connections in our graph to aggregate motion and interaction features, compared with the ego-setting, there is no obvious positive effect with large amount of features fusion. The last line shows the result of interpretable

---

Table 5: Effectiveness of feature fusion with interpretable graph. "Fusion Count" represents statistics average fusion counts of features per scenario. "Interpretable Fusion" indicates the aggregation of motion and interaction features through associations.
Motion Fusion
Fusion Count MFG
Interaction Fusion
Fusion Count CIG
minADE
minFDE
MR
Fusion
Ego
986
1.06
2.09
0.27
Full Fusion
14,191
Interpretahie Fusion
6,788
1.10
2.18
0.25
Interpretable Fusion
172
Full Fusion
6,982
1.08
2.32
0.27
Full Fusion
14,191
Full Fusion
6,982
1.04
2.09
0,24
Interpretable Fusion
175
Interpretahle Fusion
6,836
0.95
1.79
0.21
05
Figure 4: Effectiveness of pseudo label supervision.
Figure 5: Qualitative results on V2X-Traj. There are several interesting cooperative scenarios at the challenging intersection, including speed-up, lane changing and turning. We visualize only the forecasting results of the target agent in each scenario for clarity. The ground-truth trajectories are shown in red, and the multimodal predicted trajectories are shown green.
features fusion, it achieves improved performance with low computational cost, demonstrating the effectiveness to interpretably aggregate heterogeneous cooperative features.
Effectiveness of Pseudo Label Supervision. To demonstrate the quality of pseudo labels and influence on the final motion forecasting performance, we disturb labels randomly for supervision and evaluate the motion forecasting results. As shown in fig. 4 the proportion of disturbed pscudo labels increases, corresponding forecasting performance decreases, which illustrates the effectiveness of both interpretable feature fusion and pseudo label supervision. This also suggests that there is potential for further improvement in forecasting performance as the quality of labels increases.
Conclusion and Limitation
In this paper, we introduce a forecasting erreastinn-orriented oriented representation paradigm to utilize motion and interaction features from cooperative information, and present V2X-Graph, a graph-based framework to achieve interpretable and end-to-end trajectory feature fusion for cooperative motion forecasting. Comparing to existing methods that rely on single-frame perception information cooperation, our approach enhances the historical representation of agents from lightweight cooperative trajectories, and achieve improved performance in the downstream task, namely cooperative motion forecasting. Moreover, we construct V2X-Traj. Traj, the first real-world and public V2X cooperative motion forecasting dataset, expanding the research from V2I to broader V2X motion forecasting task. Experiments on both two datasets demonstrate the effectiveness of our method.
Limitation and future work. Compared to the single-frame method, the proposed v2X-Graph explores trajectory feature fusion, which mitigates errors from single-frame perception completion and achieves better motion and interaction representation of agents. Despite these advantages, the performance still relies on the tracking quality from each view. Jointly optimizing the performance from perception to forecasting is significant to explore in the future.

---

Acknowledgment
The paper is supported by funding from Wuxi Research Institute of Applied Technologies, Tsinghua University under Grant 20242001120.
References
[1] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition (CVPR). 2016.
[2] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving, 2020.
[3] Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2019.
[4] Ziming Chen, Yifeng Shi, and Jinrang Jia. Transiff: An instance-level feature fusion framework for iiaiill--iifrstructuur cooperative 3d detection with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[5] Fang Da and Yu Zhang. Path-aware graph attention for HD maps in motion prediction. In Proceedings of the International Conference on Robotics and Automation (ICRA), 2022.
[6] Jacob Devlin, Ming-Woo Wei Chang, Kenton Lee, and Kristina Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the North American Association for Computational Linguistics (NAACL), 2019.
[7] Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles Qi, Yin Zhou, Zoey Yang, Aurelien Chouard, Pei Sun, Jiquan Ngiam, Vijay Vasudevan, Alexander McCauley, Jonathon Shlens, and Dragomir Anguelov. Large scale interactive motion forecasting for autonomous driving The waymo open motion dataset, 2021.
[8] Siqi Fan, Zhe Wang, Xiaoliang Huo, Yan Wang, and Jingjing Liu. Calibration-fres bev representation for infrastructure perception. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023.
[9] Siqi Fan, Haibao Yu, Wenxian Yang, Jirui Yuan, and Zaiqing Nie. Quest: Query stream for intrastiiffastu cooperative perception. In Proceedings of the International Conference on Robotics and Automation (ICRA), 2024.
[10] Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference Computer Vision and Paltern Recognition (CVPR), 2020.
[11] Harshayu Girase, Haiming Gang, Srikanth Malla, Jiachen Li, Akira Kanehara, Karttikeya Mangalam, and Chiho Choi. Loki: Long term and key intentions for trajectory prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[12] Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the ACM SIGKDD international conference on knowledge discovery & data mining (KDD), 2016.
[13] Junru Gu, Chen Sun, and Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), 2021.
10

---

[14] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. Proceedings of the IEEE conference computer vision and pattern recognition (CVPR), 2018.
[15] Yue Hu, Yifan Lu, Runsheng Xu, Weidi Xic, Siheng Chen, and Yanfeng Wang. Collaboration helps camera overtake lidar in 3d detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[16] Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In Proceedings of the ACM Web Conference (WWW), 2020.
[17] Xinyu Huang, Peng Wang, Xinjing Cheng, Dingfu Zhou, Qichuan Geng, and Ruigang Yang. The apolloscape open dataset for autonomous driving and its application. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI), 2020.
[18] Xiaosong Jia, Liting Sun, Hang Zhao, Masayoshi Tomizuka, and Wei Zhan. Multi-agent trajectory prediction by combining egocentric and allocentric views. In Proceedings of the Conference on Robot Learning (CoRL), 2022.
[19] Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. Hdgt: Heteroge- neous driving graph transformer for multi-agent trajectory prediction via scene encoding. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI), 2023.
[20] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
[21] Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmo- han Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[22] Zixing Lei, Shunli Ren, Yue Hu, Wenjun Zhang, and Siheng Chen. Latency-aware collaborative perception. In Proceedings of the European Conference on Computer Vision (ECCV). 2022.
[23] Jiachen Li, Fan Yang. Masayoshi Tomizuka, and Chiho Choi. Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning. In Advances Neural Information Processing Systems (NeurIPS), 2020.
[24] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's hirdsseey--view representation from multi-camera images via spatiotemporal transformers. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
[25] Ming Liang. Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng. and Raquel Urtasun. Learning lane graph representations for motion forecasting. In Proceedings of the European Conference Computer Vision (ECCV), 2020.
[26] Yicheng Liu, Jinghuai Zhang, Liangji Fang, Qinhong Jiang, and Bolei Zhou. Multimodal motion prediction with stacked transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[27] Ilya Loshchilov and Frank Hutter. Sgdr:
Stochastic gradient descent with warm restarts. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
[28] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[29] Xiaoyu Mo, Zhiyu Huang, Yang Xing, and Chen Lv. Multi-agent trajectory prediction with heterogeneous edge-enhanced graph attention network. IEEE Transactions on Intelligent Transportation Systems (TITS), 2022.

---

[30] Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. Social-stgem:: A social ppatio-temporal graph convolutional neural network for human trajectory prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2020.
[31] Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang. Hao- Lewis Chiang. Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene transformer: A unified architecture for predicting future trajectories of multiple agents. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[32] Tim Salzmann. Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasine trajectory forecasting with heterogeneous data. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[33] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[34] Charlie Tang and Russ R Salakhutdinov. Multiple futures prediction. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[35] Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled Refaat, Nigamaa Nayakanti, Andre Cormman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, et al. Multipath++:: Efficient information fusion and trajectory aggregation for behavior prediction. In Proceedings of the International Conference on Robotics and Aulomation (ICRA), 2022.
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems (NeurIPS), 2017.
[37] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[38] Binglu Wang, Lei Zhang, Zhaozhong Wang, Yongqiang Zhao, and Tianfei Zhou. Core: Cooper- ative reconstruction for multi-agent perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[39] Jingke Wang, Tengju Ye, Ziqing Gu, and Junbo Chen. Ltp: for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2022.
Lane-based trajectory prediction
[40] Xishun Wang. Tong Su, Fang Da, and Xiaodong Yang. Prophnet: Efficient agent-centric motion forecasting with anchor-informen proposals. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[41] Zhe Wang, Sigi Fan, Xiaoliang Huo, Tongda Xu, Yan Wang, Jingjing Liu, Yilun Chen, and Ya-Qin Zhang. Vimi: rchiele-infrastrue multi-view intermediate fusion for cumera-based 3d object detection. In Proceedings of the International Conference on Robotics and Automation (ICRA), 2024.
[42] Sizhe Wei, Yuxi Wei, Yue Hu, Yifan Lu, Yiqi Zhong, Siheng Chen, and Ya Zhang. Robust asynchronous collaborative 3d detection via bird's eye view flow. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[43] Runsheng Xu, Zhengzhong Tu, Hao Xiang, Wei Shao, Bolei Zhou, and Jiaqi Ma. Cobevt: Cooperative bird's eye view semantic segmentation with sparse transformers. In Proceedings of the Conference on Robot Learning (CoRL), 2023.
[44] Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang, and Jiaqi Ma. V2x-vit: Vehieli-t--eeerything cooperative perception with vision transformer. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
12

---

[45] Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma. Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehiee communication. In Proceedings of the International Conference on Robotics and Automation (ICRA), 2022.
[46] Dingkang Yang, Kun Yang, Yuzheng Wang, Jing Liu, Zhi Xu, Rongbin Yin, Peng Zhai, and Lihua Zhang. How2comm: communication--ffiiio and collaboration collaboratioo-preaaatic multi-agent perception. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[47] Lei Yang, Kaicheng Yu, Tao Tang, Jun Li, Kun Yuan, Li Wang. Xinyu Zhang, and Peng Chen. Bevheight: A robust framework for vision-based roadside 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2023.
[48] Lei Yang, Xinyu Zhang, Jun Li, Li Wang, Minghan Zhu, Chuang Zhang, and Huaping Liu, Mix- teaching: A simple, unified and effective semi- eemi-supervised learning framework for monocular 3d object detection. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2023.
[49] Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, and Zaiqing Nie. Dair-v2x: A large-scale dataset for vehicle- infrastructure cooperative 3d object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[50] Haibao Yu, Yingjuan Tang, Enze Xie, Jilei Mao, Ping Luo, and Zaiqing Nie. Flow-based feature fusion for vehicle-infrastructure cooperative 3d object detection. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[51] Haibao Yu, Wenxian Yang. Hongzhi Ruan, Zhenwei Yang, Yingjuan Tang, Xu Gao, Xin Hao, Yifeng Shi, Yifeng Pan, Ning Sun, al. V2x-seq: A large-scale sequential dataset for vehicle-infrastructue cooperative perception and forecasting, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[52] Wenyuan Zeng, Ming Liang, Renjie Liao, and Raquel Urtasun. Lanerenn: Distributed repre- sentations for graph-centric motion forecasting, In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021.
[53] Wei Zhan, Liting Sun, Di Wang. Haojie Shi, Aubrey Clausse, Maximilian Naumann, Julius Kummerle, Hendrik Konigshof, Christoph Stiller, Arnaud de La Fortelle, and Masayoshi Tomizuka. Interaction dataset: An international, adversarial and cooperative motion dataset in interactive driving scenarios with semantic maps. arXiv preprint arXiv: r 1910199100...... 2019.
[54] Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. Hetero- geneous graph neural network. In Proceedings of the ACM SIGKDD international conference on knowledge discovery & data mining (KDD), 2019.
[55] Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. In Advances in Neural Information Processing Systems (NeurIPS). 2018.
[56] Renrui Zhang, Han Qiu, Tai Wang. Ziyu Guo, Xuanzhuo Xu, Yu Qiao. Peng Gao, and Hong- sheng Li. Monodetr: septh--uidel transformer for monocular 3d object detection. In Proceed- ings of the IEEE/CVF International Conference Computer Vision (ICCV), 2023.
[57] Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Ben Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, et al. Tnt: Target-driven trajectory prediction. In Proceedings of the Conference on Robot Learning (CoRL), 2020.
[58] Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, and Kejie Lu. Hivt: Hierarchical vector transformer for multi-agent motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
13

---

Appendix
In this supplementary material, detailed information about the training loss provided appendix A Comprehensive details regarding the V2X-Traj dataset are presented in appendix B Additional experimental results are presented in appendix C The implementation details of the compared methods on the V2X-Traj and V2X-Seq datasets are presented in appendix [ Additional qualitative results of the proposed 22X-Graph framework on the V2X-Seq dataset are provided in appendix
A Details of Training Loss
To achieve interpretable trajectory feature fusion in cooperative scenarios, the framework is trained in an end-to-end manner with two components optimization objectives.
For the first part of optimization objectives, we utilize the cross-entropy as the knowledge distillation
loss Ldis to optimize the prediction of the adjacency matrix A {as.y) which represents the associations between trajectories of agents across views and is supervised by the pseudo labels (A)), here ax-)) indicates a pre-pruning strategy to mitigate the class imbalance issue. This strategy is utilized when there is no intersection between the minimum bounding rectangles of two trajectories, when the agent types are different.
The second part includes the regression loss Greg and classification loss Cets optimize the motion forecasting. For optimization, we select the trajectory with the minimum average 1.2 distance from the ground truth among K. modalities. We utilize the cross-entropy loss as the selection loss to optimize Pi,k- The negative log-likelihood loss is employed as the regression loss and can be formulated as:
T+H
1
Crey
Σ PL. II T++1
logp(r, (p: p. [l B))).
(10)
NH
where P(-I-) represents the probability density function of the Laplace distribution, and (P 2 [b))= denote the coordinates and the corresponding uncertainties of the best-predicted trajectory for the agent. Overall, the final training loss can be formulated as follow:
1+1
C Lais + Lreg + Less-
(11)
B Details of V2X-Traj Dataset
V2X-Traj is the first and real-world V2X cooperative motion forecasting dataset. In this section, we provide more comprehensive description our V2X-Traj dataset. Dataset comparison presented in table 6 Other details includes dataset composition (appendix [.II). collection and annotation process (appendix B.2) and additional visualizations (appendix B.3).
B.1 Dataset Composition
V2X-Traj dataset contains a total of 10,102 scenarios, which are randomly split into the training, validation, and test set, consisting of 6,062, 2,020, and 2,020 scenarios, respectively. Each scenario comprises three independent sets of trajectories from two autonomous driving vehicles and an infrastructure-side perception device. Additionally, considering the agents are moved with constraints of traffic rules, we also provide the static vector map and real-time traffic light signals.
Trajectory. Each trajectory represents the information of agent detected and tracked independently by a single perception device. The trajectory information includes the timestamp, unique ID, agent type, location, 7- 7-dimensional detection bounding box, heading, and velocity.
Vector Map. Following 3 we collect map information the form of vectorized representations to provide valuable hints for motion forecasting. Vector maps contain lane, erosswalk, stopline, and junction elements. For each lane, we provide sample points of centerline and boundary, and semantic attributes such as turning direction, lane topology and traffic control.
Traffic Light. We provide real-time traffic light signals as they have substantial impact on the behavior of traffic participants. During the data collection and storage process, we simultancously
14

---

Table 6: Comparison with the public motion forecasting dataset. denotes that the information not provided or not available. V2X-Traj the first cooperative dataset that supports research on V2V and broader V2V&I cooperative motion forecasting. The dataset contains abundant real-world cooperative trajectories from infrastructure and cooperative vehicles, as well information about vector maps and real-time traffic lights.
Ego-view Tracks
Infra-view Tracks
Vch-view Tracks
With Vector Map
With Traffic Light
Dataset
Year
View
Scenes
Nuscenes
2
2019
Single-veh
43
1,000
Apolloscape
2019
Single-veh
103
Interaction
2019
Single-veh
X
40.054
Argovense
3
2019
Single-veh
36
324,000
Waymo motion
2021
Single-veh
10,4000
V2X-Scq [T
2023
V21
101
50
51,146
V2X-Traj
2024
V2V&I
10,
Table Detailed statistics on the total number and length of trajectories per class.
Class
Van
Car
Cyclist
Motoreyclist
Pedestrian
Bus
Tricyelist
Truck
Number
62.725
572.527
120.310 28
291,437 31
291.702
24,088
44,909
16,875
Length
54
51
21
56
48
record traffic light data at a frequency of 10 Hz. The traffic light signal information includes the timestamp, location, direction, corresponding lane ID, color status, and remaining time.
We provide detailed statistics in table 7 As the table shows, the V2X-Traj dataset contains abundant trajectories of 8 classes of agents to depict real-world V2X cooperative scenarios.
Dataset schema is represented in fig. O In each cooperative scenario, three sets of trajectories are independently collected by the ego vehicle, the infrastructure perception device, and another autonomous vehicle; simultaneously, data on traffic light synchronization is gathered. Additionally, we offer comprehensive vector map covering intersections.
B.2 Data Collection and Annotation
This subsection details the process to construct the V2X-Traj dataset.
We choose 28 urban traffic intersections in Beijing and deploy 4-6 pairs of 300-beam LiDAR and high- resolution cameras for each intersection. These infrastructure sensors can fully cover the intersection areas. We deploy one 40-beam LiDAR and six high-quality cameras for the autonomous vehicle. We provide the configuration sensor deployment of autonomous vehicles and infrastructure in fig. The perception devices of autonomous driving vehicles and infrastucture devices have trained 3D object detection and tracking models. These models are used to generate trajectory sequences.
To collect the trajectory data, the two autonomous driving vehicles were driven simultaneously and randomly through areas equipped with sensors. The V2X cooperative scenarios are collected when there is certain overlap in the perception range of the two vehicles and the infrastructure device, resulting in the V2X cooperative trajectory sequences repository. The trajectories from each view were stored independently.
Finally, we mined interesting segments from the repository create 10. cooperative scenarios. The trajectory mining process consisted of several steps, including scenario fragmentation, trajectory scoring and scenario selection. In the first step, we divided the sequences from each view into 8-second segments. In the second step, a score was assigned to each trajectory from ego-view based on the interesting behaviors agents such as turning, speeding up, slowing down and lane changing. In the third step, we retained a total of 10,102 sequences with high-score trajectories. Within each segment, one trajectory was designated as the target agent for prediction, while the remaining trajectories were assigned as others.
15

---

V2X-Traj Dataset
htpp/liffa/Net-Trrgeettresss
Traffic Light
Vector Map
city,,
city, Linestarp,
L/NE
Linestanp,
ud, type, type, agont_tag,
har Trafric conreot,
lare Type,
direction,
direction, tanc 1d, color_1, ressin rensin_tive_i, color tine_,, color_ tivein
1s. Interserrtos,
X,
pridecessers,
F_reighoz
centerling Line
Length, wadth, height, heading_ theta, ve velority_K, velocity.y,
STOPLINE strpp
intersection_ic
CROSSWALK
cressvalk Lt, palggon
intersection_1d
junction_iA po lygon
Figure 6: Schema of the V2X-Traj dataset.
Camera
UOAR
Camera
RADAR
Trunk Liner
GPS Revelver
KLPC
Figure Sensor deployment in autonomous vehicles (left) and infrastructure (right).
B.3
Additional Dataset Visualization
fig. presents four interesting scenarios from our V2X-Traj dataset. the figure describes, each scenario includes trajectories from the ego-vehicle, the cooperative infrastructure and the cooperative autonomous driving vehicle. The inclusion of cooperative trajectories from both the autonomous vehicle and the infrastructure device enhances the information available to the ego-vehicle.
C Additional Experimental Results
C.1 Effectiveness of Major Components
In this experiment, conduct an additional ablation study on the V2X-Seg dataset demonstrate the contribution of each key component to the cooperative motion forecasting performance. Similar before, we evaluate the effectiveness by removing one of the components alternately. As shown in Table 8 each component within the interpretable graph demonstrates marked enhancement in performance. The MFG efficiently integrates motion features from associated trajectories, leading to significant decrease in minADE by 0.15. The ALG incorporates interaction feature from lane segments and the CIG merges dense interaction features from cooperative trajectories, resulting in maximum reduction of 0.51 in minFDE and 7% in MR. In summary, these components con- tribute significantly to performance improvements by learning heterogenerous cooperative feature representations.
16

---

Table 8: Effect of major components,
Table 9: Effect of cooperative representations.
MFG
ALG
CIG
minADE
minFDR
MFG
ALG
CIG
minADE
minFDF.
MR
1.29
2.11
0.30
1.29
2.11
0.30
1.34
2.48
0.36
V
1.22
2.17
0.32
1.30
2.23
0.34
1.26
2.11
0.31
1.14
1.97
0.29
v
1.14
1.97
0.29
C.2 Effectiveness Cooperative Representation
We further evaluate the effectiveness of the model in cooperative feature representations on V2X-Sep by alternately masking out the infrastructure-Ve trajectories in each component. As demonstrated in Table the customized cooperative representation learning of each component results in performance improvements to a certaín degree. Concretely, the MFG effectively integrates the motion features of associated trajectories through explicit associations and implicit temporal-spatial encoding. This component leads to improved performance in motion forecasting for each future frame, as evidenced by the decrease of 0.15 in the minADE metric. Relatively, the ALG and CIG components fuse the interaction features of lane segments and cooperative trajectories through compact spatial encoding. These component primarily enhances the performance of long- long-term intention prediction, indicated by a maximum reduction of 0.20 in the minFDE and 3% in the MR metrics.
14
000
minPOR
11
Figure 8: Scalability study different dataset sizes.
C.3 Scalability Study
Compared with vanilla motion forecasting, collecting large-scale data in cooperative scenarios for cooperative motion forecasting poses some realistic limitations. Therefore, it is essential to study how the performance of the method scales with dataset size.
In this experiment, we randomly split data of different sizes from the training set of V2X-Seq to train the V2X-Graph framework and evaluate the model the full validation set. We compare the proposed v2X-Graph framework both vehicle-only and cooperative settings to assess how the scalability of the proposed framework for cooperative motion forecasting varies with dataset size.
As shown fig.8 as the amount of training data increases, both vehicle-only and cooperative settings achieve improved performance. Notably, V2X-Graph achieves greater advantages in cooperation, which illustrates the scalability the framework cooperative motion forecasting.
C.4 Robustness of Latency and Data Loss
We conduct robustness experiments on V2X-Seq dataset, taking both data synchronization problem and communication latency into consideration.
Specifically, we simulate latency by dropping the latest one or two frames infra-view data during transmission. And we address the latency issue with simple interpolation to obtain synchronized trajectory data. Experiment results table 10 shows that there is little performance degradation to communication latency.
Data loss and sensor failures are common practical challenges. The performance advantage on real-world datasets demonstrates the robustness of our method. We further evaluate V2X-Graph under different data loss ratios on V2X-Seq dataset. Specifically, we randomly drop perception results data in each frame from the infra-view in transmission with various dropping ratios. As shown in
17

---

Table 11: Robustness of data loss.
Table 10: Robustness of latency.
Loss Ratio (%)
minADE
minFDE
MR
Latency (ms)
minADE
minFDE
MR
1.05
1.79
0.25
1.0458
1.7855
0.2527
10
1.05
1.81
0.26
100ms
1.0468
1.7879
0.2562
1.08
1.85
0.27
200ms
1.0779
1.8385
0.2688
50
1.10
1.88
0.28
ablle 11 the forecasting performance decreases as the ratio increases, but is worth mentioning that our method outperforms the compared methods (table on page 8) even under extreme conditions with 50% loss rate.
C.5 Parameter Size and Inference Cost Comparison
Table 12: Parameter size comparison.
Dataset
Methods
V2X-Seq
TNTseTNTT 13
HiVT [88
V2X-Graph
Param. (M)
1.0
2.6
5.0
V2X-Treh Traj Param. (M)
DenseTNT 13
HDGT 19
V2X-Craph
1.0
12.1
4.9
As shown table 12 the parameter size of V2X-Graph is comparable with other forecasting methods.
Table 13: Inference cost comparison.
Dataset
Methods
V2X-Seq
PP-VIC [5] DenseTNT 13
PP-VIC [5] + HiVT 58
V2X- V2X-Graph 51.50
Latency (ms)
161.45 371.38
161.45 + 53.30
V2X-Traj
DNT [33
HDGT [1
V2X-Graph
Latency (ms)
168.88
1260.70
52.69
Then we conduct the inference experiment on single NVIDIA GTX 4090 and compare the inference cost. As for the experiment results shown table [3] the proposed V2X-Graph is even faster than the compared vanilla motion forecasting methods, benefiting from the synchronous temporal state modeling and integration.
D Additional Implementation Details
In this section, we provide implementation details of compared methods our experiemnts.
Implementation Details of HiVT. We compare our proposed V2X-Graph with the official evaluation of the V2X-Seq dataset [5T] from httpss//github.com//ll TAw.:///AAATAAIIIIIIIIIAA THU/DAIR HHU/DAIR-V2X-SS V2X-Seg For fair comparison, we re-implemented a larger model of HiVT (58) with hidden size of 128 for improved performance reported in the corresponding paper.
Implementation Details of DenseTNT For comparison, we re-implemented the classical homoge- neous graph method DenseTNN (13) our V2X-Traj Traj dataset using their official code package, from https://gthuub.com SwwnTTiiighua-MRS--LEE
The model is trained using the default settings of two stages the V2X-Traj training set, utilizing server with 8 NVIDIA RTX 3090s. During the first stage, all modules are trained, except for the goal set predictor, for 16 epochs. In the second stage, the goal set predictor is trained for epochs. The batch size is set to 64, the initial learning rate 0.001, and it decays by 30% every epoch. The hidden size of the feature vectors is set to 128, and the head number of goal predictor 12.
Implementation Details of HDGT. We further re-implemented the advanced heterogeneous graph method HDGT [9] based their official code from http://ghhhha..cmmmmpmmeDneeoood We
18

---

re-implemented larger model of HDGT with a hidden size of 256, and the number of heads in all multi-head attention blocks 64, for better performance for comparison. The size of the kernel the ggentTemporalEnndder in HDGT is set to 40-13-3 to accommodate the specific observation horizon of 40 time steps in V2X-Traj. During training phase, we follow the default settings, which uses the Adam optimizer with initial learning rate of 5 x 10-1, weight decay of 1 10 and batch size of 64. For the V2X-Traj dataset, the number of training epochs set to 30, with a I-epoch warmup and linear decay O. The type-specific agent distance buffer hyperparameters are empirically set to 30 meters for vehicles, 10 meters for pedestrians, and 20 meters for cyclists.
Additional Qualitative Results
In this section, present qualitative results V2X-Seq dataset, including the visualizations of interpretable association (appendix E.I) and the visualizations of our proposed V2X-Graph and compared method (appendix E.2
E.1 Visualizations of Interpretable Association
Here we present the visualizations of the interpretable association on V2X-Seq. For clarity, we only visualize the associations of the historical trajectories of the target agent from the ego-view (fig. (a)) and the infrastructure-side (fig. 1O (b)). Dashed circles indicate part of the additional information from infrastructure-side trajectories that can be utilized. As shown in the figure,
method enables the interpretable association of trajectories across views, serving as guidance for the end-to-end learning of cooperative trajectory representations.
E.2 Qualitative Results on V2X-Seq
We present additional qualitative results on the V2X-Seq dataset; three challenge scenarios are selected for methods comparison. In particular, the motion forecasting results of HiVT [58], which employs PP-VIC VIC [51] to utilize cooperative information, are shown in fig. (c), and the results of proposed V2X- D2X-rrap exceptional performance in motion forecasting, in particular of predicting long-range intentions, which demonstrates the ability of proposed V2X-Graph for further utilization of information within cooperative trajectories.
shown in fig. (d). In the methods comparison, our method exhibits
19

---

View
Infra-View
Veh View
Scenario
Scenario Geenard
Figure 9: Visualizations of the V2X-Traj dataset. Each scenario consists trajectories from the ego- vehicle, the cooperative infrastructure and the cooperative autonomous vehicle. In this figure, orange boxes represent autonomous vehicles, blue elements denote traffic participants, and green boxes denote the target agent that needs be predicted.

---

Scenario 1
Scenario 2
Scenario
(a) Ego View Trajectory
(b) Infra-View Trajectory
(c) Compared
Method
(d) V2X-Graph (Ours)
Figure 10: Qualitative results on three challenge scenarios over V2X-Seq. The historical trajectories of the target agent are shown in yellow. The red dashed circles indicate a part the enhanced information from the infrastructure view. The ground-truth trajectories are shown in red, and the predicted trajectories are shown green.
21

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we claim the contributions made in cooperative motion forecasting in this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In section 6 discuss the contributions of cooperative motion forecasting in this paper and the relationship of the proposed method to the vanilla forecasting task. In discuss how the performance of the method scales with dataset size.
appendix C.3
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
22

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide details of the proposed method in section and describe the experimental settings in section The code is provided in the supplemental material and will be released along with the dataset once accepted.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
23

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide details of the experimental settings in section 5
The code is
provided the supplemental material, the code and the dataset will be released along with sufficient instructions once accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//ipps.cc/pullic/uuiiee
sisssssssiniyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide details of the experimental settings in section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not formally reported because it would be too computationally expensive. However, we conduct the experiment multiple times and record universally representative results to support the main claims of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details of the computational resources in section S
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :ttp::////erureee hww.npsu...../. http//rrpss..ccuile/e/tttteeeeee http://bbllctttccccc....eee.eeee
Answer: [Yes]
Justification:
The research in this paper conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We hope both the V2X-Grap framework and V2X-Traj dataset can facilitate the further development of cooperative motion forecasting.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: The proposed real-world dataset does not contain personal information and has been encrypted to mitigate related risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The related code and data in this paper have proper licenses and have been cited appropriately.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
26

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide details of the new real-world dataset in appendix B. and the local government has granted permission for the release of this dataset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "11WmH2vMmN", "reviewer": "Reviewer_zJGd", "summary": "The paper introduces a novel graph-based framework called V2X-Graph for learning cooperative trajectory representations in motion forecasting for autonomous vehicles. V2X-Graph aims to enhance the motion prediction capabilities of autonomous vehicles by leveraging cooperative information from vehicles and traffic infrastructure. The framework represents trajectories as nodes in a graph and fuses motion and interaction features from multiple views to achieve interpretable end-to-end trajectory feature fusion. The authors also constructed the first real-world V2X motion forecasting dataset, V2X-Traj, which includes cooperative scenarios with multiple autonomous vehicles and traffic infrastructure. Experiments conducted on both the V2X-Seq and V2X-Traj datasets demonstrate the advantages of V2X-Graph in utilizing additional cooperative information to enhance motion forecasting capabilities.The creation of the V2X-Traj dataset, a real-world, public dataset for V2X motion forecasting, adds significant value to the field, promoting further research in this domain.", "questions": "While the experiments demonstrate the effectiveness of V2X-Graph on the V2X-Seq and V2X-Traj datasets, these datasets may not capture the full range of scenarios encountered in real-world driving. Expanding the evaluation to include more diverse and challenging driving environments would enhance the generalizability and robustness of the proposed method.\nV2X-Traj dataset, while a valuable contribution, might have inherent biases that are not discussed in the paper. Providing a more detailed analysis of the dataset's composition, including the diversity of scenarios and the representativeness of different driving conditions, would be beneficial (page 7, Section 4.1).", "limitations": "- The paper briefly mentions some limitations in the discussion section, but it does not provide an in-depth analysis. The authors should more thoroughly acknowledge the limitations of their approach, including potential biases in the V2X-Traj dataset and the constraints of their methodology. Specific limitations such as the scalability of the V2X-Graph framework in highly dynamic and dense traffic environments, potential issues with data communication latency, and the synchronization of information from multiple sources are not adequately discussed. Including a more detailed assessment of these factors would be beneficial.\n\nOthers:\n- The paper should address practical challenges such as the computational overhead required for real-time processing, the robustness of the system in adverse weather conditions, and the reliability of V2X communication in various urban and rural environments.\nDiscussing potential solutions or future work aimed at overcoming these challenges would strengthen the practical applicability of the proposed method.\n\n- While the paper makes a significant contribution to the field of cooperative motion forecasting, it would benefit from a more thorough discussion of its limitations and potential negative societal impacts. By addressing these aspects in detail and providing constructive suggestions for mitigating these issues, the authors can enhance the robustness, and applicability of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel forecasting-oriented representation paradigm, V2X-Graph, which leverages cooperative perception information from multiple sources (e.g., infrastructure and other vehicles) to enhance motion forecasting for autonomous driving. This approach is unique in focusing on the holistic historical motion and interaction features rather than single-frame perception.\n- The methodology section (pages 4-6) is detailed and provides a comprehensive description of the V2X-Graph framework, including scene representation with graphs and trajectory feature fusion. The use of graph neural networks (GNNs) to encode cooperative scenarios and perform motion forecasting is well-justified.\n- The paper includes extensive experimental results (pages 8-10) on both the V2X-Seq and V2X-Traj datasets, demonstrating the effectiveness of the proposed approach with clear performance improvements over baseline methods.\n- The paper is well-structured, with a clear flow from the introduction of the problem (pages 1-2), through the related work (pages 2-3), to the proposed methodology and experimental results. Each section builds logically on the previous one, making the paper easy to follow. Figures and tables (e.g., Figure 1 on page 3 and Table 1 on page 8) are effectively used to illustrate key concepts and results, enhancing the reader’s understanding.\n- By addressing the underutilization of cooperative perception information in motion forecasting, the paper tackles a critical challenge in autonomous driving. The proposed V2X-Graph framework has the potential to significantly improve the safety and efficiency of autonomous vehicles by leveraging a richer set of perception data.", "weaknesses": "- While the application of graph neural networks (GNNs) to encode cooperative scenarios is well-executed, the approach may lack significant novelty. GNNs have been widely applied in various fields, including autonomous driving. The paper would benefit from a more detailed comparison with existing GNN-based methods for motion forecasting to highlight the unique contributions and advancements of the proposed V2X-Graph framework (page 6, Section 3.2).\n- The experimental evaluation compares the proposed method primarily against a few baselines. Including a broader range of state-of-the-art methods for motion forecasting, especially those utilizing cooperative perception, would strengthen the validation of the proposed approach. This would provide a more comprehensive view of where V2X-Graph stands in the context of current research.\n- Limited Discussion on Real-World Implementation: lacking a thorough discussion on the practical challenges and considerations of implementing the V2X-Graph framework in real-world autonomous driving systems. Issues such as communication latency, data synchronization, and robustness to sensor failures are crucial for practical deployment but are not sufficiently addressed (page 10, Section 5, Discussion).\n- Some parts of the methodology section are dense and might be challenging for readers to follow, particularly those less familiar with GNNs. Simplifying the mathematical notation or providing additional explanatory text and illustrative examples would improve clarity and accessibility", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer zJGd: \\\nThanks for your thorough review and valuable suggestions on our work.\nWe have carefully considered your suggestions and would like to respond to each of your main comments regarding our weaknesses and questions.\n\n**W1.**\nGraph neural networks (GNNs) are common practice for the vanilla motion forecasting task.\nIn this task, GNNs with full edge connections can effectively model the interaction of the specific agent and its surroundings.\nEarly works adopted homogeneous GNNs, while recent research has introduced heterogeneous GNNs to address the assumption of varying patterns in different interaction cases (e.g., vehicle to lane, vehicle to pedestrian).\n\nWhile considering the cooperative motion forecasting task, the challenges rised: how to formulate the fusion paradigm to effectively utilize the trajectory data of each agent from each view?\n\n**Existing graph-based methods cannot leverage cooperative information effectively (Table 2 on page 8).** To address the challenges, **we explored the unique cooperative trajectory representation paradigm through a novel design of GNN**, which integrates motion and interaction features via heterogeneous edge encodings, and further introduces graph link prediction in this task to guide interpretable fusion.\n\n**W2.**\nMost of the existing works in cooperative autonomous driving community pay more attention on cooperative perception, especially the single-frame cross-view feature fusion methods.\n\nWhile considering the communication latency chanllenge in cooperative motion forecasting task, we focus on lightweight cooperative information, namely perception results, instead of raw sensor data or features. \n\n**PP-VIC stands out as the only existing method based on perception results and the representative method of the single-frame perception approaches for the downstream forecasting task.**\nIt analyzes multi-view perception relations along trajectories to achieve completion. \n**Our V2X-Graph achieve a better performance compared with it.**\n\nThanks for your suggestion, we will give more discussion in the revision paper.\n\n**W3.**\nThanks for pointing it out, we further report the experimental results in the Subsections 2.1-2.2 of General Rebuttal. \nConsidering cooperative data holistically, V2X-Graph shows robustness against errors in local frames due to latency or data loss.\n\n**W4.**\nThanks for your suggestion, we will simplify the representations in the methodology section for clarity in the revision paper.\n\n**Q.**\nThanks for your valuable question. **We report more details of the proposed V2X-Traj dataset in Section 3 of General Rebuttal.**\n\nWe allow multiple autonomous vehicles to drive independently and collect data on cooperative scenarios with specific perception coverage.\n**To ensure the diversity of traffic and weather conditions, the V2X-Traj dataset is collected from several challenging and representative urban intersections, and the data is collected along the whole year.**\n\nHowever, the collected scenarios are limited by the roadside sensor installation, which is mostly at the intersections. We are continuely collecting the data and will provide more diverse cooperative scenario data in the future, such as the scenarios with multiple vehicles in corridor scenes.\n\n**L1.**\nThanks for your suggestion, we will expand our discussion in the revision paper.\n\nCompared with the single-frame method, the proposed V2X-Graph explores trajectory feature fusion, which mitigates errors from single-frame perception completion and offers additional advantages such as better motion and interaction representation of agents.\n\nAlthough V2X-Graph has the above advantages, the performance still relies on tracking quality from each view.\nA tracking error could introduce noises in trajectory representation and association.\n\nTo jointly optimize the performance from perception to forecasting, we are going to further explore the end-to-end cooperative forecasting. \n\n**L2.**\nThe inference cost and the robustness under non-ideal communication condition are the two main concerns for practical applications. \n\nWe further report the experimental results in Section 2 of General Rebuttal. **It can be seen that the inference cost is acceptable and the performance degradation is limited in poor communication conditions.**\n\nAs for the impacts of weather conditions, it will directly influence the perception performance, and may further introduce input noise to our V2X-Graph. \nExpanding the framework to the end-to-end cooperative forecasting is a possible solution to address the challenge, and we are working on that as a future work.\n\n**L3.**\nThanks for your suggestion, we will provide additional discussion on potential negative societal impacts in the revised version.\n\nWith more and more autonomous vehicles on the road, it is becoming possible to share information among vehicles and infrastructure. \nHowever, there are potential negative societal issues such as privacy concerns and backdoor attacks. \nExploring security encryption and anomaly detection for shared cooperative information will benefit research in this field.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3SKmlMhJB3", "reviewer": "Reviewer_vdKw", "summary": "This paper tackles the cooperative motion forecasting problem for vehicles. This paper introduces additional information other than the ego view agent from other view to expand the perception field of the prediction. The authors propose a graph network to extract information for multimodal trajectory prediction. The authors introduce a new dataset, namely V2X-Traj, for this problem setting. Experiments show the efficacy of the proposed method.", "questions": "Please address the weaknesses.\n\n---Post rebuttal\nThank you for the author's response. The reply addressed most of my concerns and the added experiments on inference efficiency are important. I will increase my score", "limitations": "The authors have discussed the potential limitations.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The problem setting is new. With more and more autonomous vehicles on the road, it is possible to include cooperative information.\n2. The experimental results on both datasets show that V2X-Graph outperforms existing methods.", "weaknesses": "1. The cooperative setting is limited to two vehicles. \n2. The scalability of the V2X-Graph framework in terms of computational resources and its performance with larger datasets or in more complex environments is not extensively discussed. The training cost and inference cost should be compared with the baselines.\n3. There is not error analysis. When does the method/cooperative setting fail and why?\n4. Potential cooperative forecasting problems are not considered. How does the V2X-Graph framework handle communication delays or data loss in real-time applications?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer vdKw: \\\nThanks for providing valuable feedback on our work. \nWe will address each of the limitations you have pointed out in your comments.\n\n**W1.**\nYes, **we focus more on the representative scenario unit at the current stage, which involves two vehicles and one roadside device. It can represent the common cooperative scenarios, including vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I).**\n\nIn this setting, we evaluate the proposed method with real-world dataset, which reflects the real complex traffic situation. \n**The experimental results demonstrate that the proposed V2X-Graph can be expanded from two views to three views (see Table 2 on page 8). Theoretically, the method can also be directly expanded to more than three views.**\n\nTo collect the dataset, we allow multiple autonomous vehicles to drive independently and collect data on cooperative scenarios with specific perception coverage. \nThe V2V scenarios with more than two vehicles are limited due to data distribution. \n**As the data volume accumulates, more diverse cooperative scenarios will be introduced in the following works.**\n\n**W2.**\nThanks for your suggestions. We further report the model parameters and inference cost in Subsection 2.3 of General Rebuttal. \n\nV2X-Seq is a large-scale challenging cooperative dataset, and V2X-Traj has more complex scenarios. \n**It can be seen from the experimental result that the proposed V2X-Graph achieves a better performance-efficiency balance.**\n\nThe lack of public dataset hinders the further development of cooperative motion forecasting. We release a larger and more challenging dataset, i.e., V2X-Traj, but a more complex one is needed. \nWe are going to enlarge the dataset and introduce more complex scenarios.\n\n**W3.**\nTransimission latency and data loss are two common challenges in practice. We add error analysis in Subsections 2.1 and 2.2 of General Rebuttal. \n\nUnder non-ideal communication conditions, performance is inevitably influenced. \nHowever, **the effect of errors in several frames is limited since V2X-Graph models cooperative data holistically, and the framework still outperforms the compared methods.**\n\n**W4.**\nWe report the performance when transimission latency and data loss occurs in Subsections 2.1 and 2.2 of General Rebuttal. \nThe experimental results demonstrate the effectiveness in practical applications. \nThanks for your valuable suggestion, more disscusion will be added in the revision paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V69eHlZE8u", "reviewer": "Reviewer_vBSp", "summary": "This paper presents the V2X-Graph method for cooperative motion forecasting. In a cooperative autonomous driving setting, an autonomous driving vehicle receives sensor data from surrounding vehicles and infrastructure-side devices.\n\nExisting cooperative autonomous driving approaches focus on perception completion, where the vehicles receive only perception data, and motion forecasting is performed with completed perception data without considering the cooperative setting.\n\nThe proposed V2X-Graph method is a graph-based framework that performs cooperative motion forecasting with trajectory feature fusion. It fuses history trajectories from different views with an Interpretable Graph. The Interpretable Graph consistents of three subgraphs: Motion Fusion subGraph (MFG), Agent-Lane subGraph (ALG), and Cooperative Interaction subGraph (CIG).\n\nThe authors also provide a new dataset V2X-Traj. Compared to the existing V2X-Seq dataset which contains only vehicle-to-infrastructure (V2I) cooperative scenarios, V2X-Traj also includes vehicle-to-vehicle (V2I) cooperative scenarios, making it vehicle-to-everything (V2X).\n\nThe authors evaluated their V2X-Graph method on the existing V2X-Seq dataset as well as their new V2X-Traj dataset. They compared against the PP-VIC baseline. The result shows that adding trajectory feature fusion improves the performance over the PP-VIC baseline.", "questions": "* Table 1: Why are the PP-VIC + HiVT results different from those in the original PP-VIC paper?\n\n* Line 284: How are the observations of the same agent from different views aggregated?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* The proposed V2X-Graph method improves the cooperative motion forecasting performance over the PP-VIC baseline.\n\n* The new V2X-Traj dataset will be very useful to the community.", "weaknesses": "* I am not fully convinced by the motivation of why perception completion is not enough for cooperative motion forecasting. It will be useful to give some examples to explain how feature fusion helps.\n\n* The writing can be improved. For example, it will be useful to give an overview of the architecture either in the caption of Figure 2 or at the beginning of Section 4. The module names MFG, ALG, CIG, etc in the architecture diagram are not explained until Section 4.2. The evaluation setup of Table 2 (Graph-based methods comparison) is not clearly explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer vBSp: \\\nThank you for your valuable feedback on our work. \nWe have carefully considered your suggestions and would like to respond to each of your main comments regarding our weaknesses and questions.\n\n**W1.**\nInstead of single-frame perception completion method, the proposed trajectory-based feature fusion have advantages in two perspectives.\n\nFirst, the observation of the agents in different views could be different due to various sensor perspectives and configurations; direct fusion may lead to deviations. Instead, utilizing differential information from each view and integrating it by correlation for forecasting may mitigate the effect.\n\nSecond, given the aforementioned shortcomings, perception completion methods obtain the agent state at each frame separately, lacking modeling across historical timesteps. This deficiency negatively impacts motion and interaction feature modeling in downstream forecasting.\n\n**W2.**\nThanks for your suggestion. \nWe will give an overview of the architecture and briefly introduce the key modules at the beginning of Section 4, and elaborate on the evaluation setup in Table 2 for clarity and accessibility. \nWe will address other similar issues in the revised version.\n\n**Q1.**\nIn the original paper, HiVT is evaluated with a hidden size of 64. \nWe follow the updated evaluation in that paper, using a hidden size of 128 to achieve better performance of HiVT for a fair comparison.\n\n**Q2.**\nWe compare the proposed V2X-Graph with popular and effective graph-based methods in cooperative scenarios.\n\nBoth V2X-Graph and the compared graph-based methods encode trajectories from each view as different nodes. \nThe methods are trained in cooperative scenarios to integrate cooperative data for performance improvement.\n\nFor the compared graph-based methods, the feature integration among agents observed from all views follows the original settings, including full edge connections and edge encodings similar to those in vanilla forecasting tasks.\nIn contrast, for the proposed V2X-Graph, **we design heterogeneous edge encodings to integrate motion and interaction features from cooperative data, with the integration guided by graph link prediction**.\n\nGraph-based comparisons across various cooperative scenarios demonstrate the effectiveness and advantages of the proposed method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uz2lEESsK1", "reviewer": "Reviewer_w4tS", "summary": "This paper introduces V2X-Graph, a novel framework using trajectories of agents (include ego and others) and vector map as inputs for cooperative motion forecasting that fuses trajectory features in an interpretable, end-to-end manner. The authors evaluate V2X-Graph using V2X-Seq, a vehicle-to-infrastructure (V2I) motion forecasting dataset. Additionally, they create V2X-Traj, the first real-world dataset for vehicle-to-everything (V2X) motion forecasting, including scenarios with multiple autonomous vehicles and infrastructure. Author shows the V2X-Graph is achieving state-of-the-art results on both V2X-Seq and V2X-Traj datasets.", "questions": "1. There are more works of GNNs for motion forecasting such as [1][2][3].\n2. What’s the efficiency or latency of this framework given it has quite complicated network components?\n3. Please check the weakness.\n\n[1] Mohamed, Abduallah, et al. \"Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n\n[2] Li, Jiachen, et al. \"Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning.\" Advances in neural information processing systems 33 (2020): 19783-19794.\n\n[3] Girase, Harshayu, et al. \"Loki: Long term and key intentions for trajectory prediction.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.", "limitations": "The paper discusses the limitations of the work in section 6.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) The paper is well-written, with a clear introduction of methods and detailed explanations of the experiments conducted.\n2) This paper proposes a novel framework to address motion forecasting in V2X scenarios that not only encodes multiple trajectories with map information but also designs additional graphs for heterogeneous feature representations. Moreover, the paper discusses the effectiveness of different components.\n3) This paper introduces the V2X-Traj dataset, which includes rich information such as trajectories, traffic lights, and maps, making it a valuable resource for the community to study V2X scenarios.", "weaknesses": "The proposed method emphasizes V2X; however, the inputs do not include important factors such as traffic lights, lane markings, and pedestrians, which may significantly impact the future motion of the vehicle. There is concern that V2X-Graph oversimplifies the problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer w4tS: \\\nThanks for your valuable feedback on our work. \nWe have carefully considered your suggestions and would like to respond to each of your main comments regarding our weaknesses and questions.\n\n**W1.**\n**The structure and information of the input data for V2X-Graph primarily follow popular motion forecasting methods**, such as [1][2][3], which involve the geometry and semantic attributes of lanes, and all types of agents in the scenario, including vehicles, bicycles, and pedestrians. \nPedestrian trajectories are also modeled and predicted. We believe this formulation of the problem is representative. \n\nOur method focuses mainly on V2X. \nThe inclusion of other factors, such as traffic lights, could benefit our approach and is worth exploring in the future. \n\nWe apologize for any potential misunderstandings in our paper and will detail these factors in Section 3 and Figures 1-2 in the revised version for clarity.\n\n**Q1.**\nThanks for reminding us of these excellent works on GNNs for motion forecasting. \nThese works are insightful in leveraging GNNs for modeling interactions between agents and their surroundings.\nWe will include these papers as related works in the next version.\n\n**Compared with these works, we introduce a unique design of GNN to interpretably integrate motion and interaction features from cooperative data for motion forecasting.**\n\n\n**Q2.**\nInference efficiency is important for practical application. \nWe further report the parameters and the inference cost in Subsection 2.3 of General Rebuttal. \n**The experimental results demonstrate the inference efficiency of V2X-Graph in complex cooperative scenarios.**\n\n[1] Gao et al. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. CVPR 2020. \\\n[2] Shi et al. Motion transformer with global intention localization and local movement refinement. NeurIPS 2022. \\\n[3] Zhou et al. Hivt: Hierarchical vector transformer for multi-agent motion prediction. CVPR 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## General Rebuttal for Commen Concerns
We will respond to the common concerns raised by the reviewers here.

**1. Our V2X-Graph is a pioneering work exploring trajectory-based feature fusion for cooperative motion forecasting.**

Most of the existing works in cooperative autonomous driving community pay more attention on cooperative perception, especially the single-frame cross-view feature fusion methods. 
However, cooperative forecasting has not been well explored, which is a direct safety-related downstream task. 
Existing perception-based methods obtain the agent state at each frame individually, which can not model the temporal motion and interaction feature for forecasting tasks, leading to performance degradation in downstream tasks.

The proposed V2X-Graph exploring forecasting-oriented trajectory feature fusion, which considers cooperative data holistically and integrates temporal motion and interaction features interpretably and in an end-to-end manner. 


**2.1 Our V2X-Graph is robust to communication latency and data synchronization.**

We conduct robustness experiments on V2X-Seq dataset, taking both data synchronization problem and communication latency into consideration. Specifically, we simulate latency by dropping the latest one or two frames of infra-view data during transmission. And we address the latency issue with a simple interpolation to obtain synchronized trajectory data. Experiment results in the following table show that there is little performance degradation to communication latency.

| Latency (ms) | minADE | minFDE | MR |
| --- | --- | --- | --- |
| 0 | 1.0458 | 1.7855 | 0.2527 |
| 100 | 1.0468 | 1.7879 | 0.2562 |
| 200 | 1.0779 | 1.8385 | 0.2688 |
Table 1. Robustness experiment results on latency.

**2.2 Our V2X-Graph is robust to data loss and sensor failures.**

Data loss and sensor failures are common practical challenges. The performance advantage on real-world datasets demonstrates the robustness of our method. We furhter evaluate V2X-Graph under different data loss ratios on V2X-Seq dataset. Specifically, we randomly drop perception results data in each frame from the infra-view in transmission with various dropping ratios. 
As shown in the table, the forecasting performance decreases as the ratio increases, but it is worth mentioning that our method outperforms the compared methods (table 1 on page 8) even under extreme conditions with a 50% loss rate.

| Loss Ratio (%) | minADE | minFDE | MR |
| --- | --- | --- | --- |
| 0 | 1.05 | 1.79 | 0.25 |
| 10 | 1.05 | 1.81 | 0.26 |
| 30 | 1.08 | 1.85 | 0.27 |
| 50 | 1.10 | 1.88 | 0.28 |
Table 2. Robustness experiment results on data loss.

**2.3 Our V2X-Graph is more efficient than other methods according to inference latency.**

Firstly, the parameter size of V2X-Graph is comparable with other motion forecasting methods, such as DenseTNT and HiVT.

| Dataset | Method | | |
| --- | --- | --- | --- |
| V2X-Seq | DenseTNT | HiVT | V2X-Graph |
| Param. (M) | 1.0 | 2.6 | 5.0 |
| V2X-Traj | DenseTNT | HDGT | V2X-Graph |
| Param. (M) | 1.0 | 12.1 | 4.9 |
Table 3. Parameter size comparison.

Then we conduct the inference experiment on single NVIDIA GTX 4090 and compare the inference cost. As for the experiment results shown, the proposed V2X-Graph is even faster than the compared vanilla motion forecasting methods, benefiting from the synchronous temporal state modeling and integration.

| Dataset | Method |  |  |
| --- | --- | --- | --- |
| V2X-Seq | PP-VIC + DenseTNT | PP-VIC + HiVT | V2X-Graph |
| Latency (ms) | 161.45 + 371.38 | 161.45 + 53.30 | **51.50** |
| V2X-Traj | DenseTNT | HDGT | V2X-Graph |
| Latency (ms) | 168.88 | 1260.70 | **52.69** |
Table 4. Inference cost comparison.

**3. We provide more additional details of the proposed V2X-Traj dataset.**

The location and time distribution of the proposed V2X-Traj dataset are presented in the PDF.

To avoid potential bias in the behavior patterns of autonomous vehicles, which leads to a distribution shift in the dataset, we deploy roadside sensors at 28 challenging and representative urban intersections and allow multiple autonomous vehicles to drive independently in these areas (Figure 1 in PDF). 
A scenario is collected when the perception ranges of two vehicles and a roadside device overlap.

To ensure the diversity of traffic and weather conditions, the data is collected throughout the whole year (Figure 2 in PDF).

Meta-review: The authors present V2X-Graph, a framework for interpretable and end-to-end trajectory feature fusion for cooperative motion forecasting. An advantage of this approach compared to the state-of-the-art in this area is that it focuses on the historical motion and interaction features rather than single-frame perception. Multiple reviewers raised the concern that it is not clear how the proposed method differs from existing GCN methods. They also acknowledged the response of the authors on this point. Additionally, the point was raised whether the V2X-Traj dataset might have inherent biases benefiting the proposed method. The authors have better explained the rationale behind the dataset (that represents a challenging scenario) and have given additional details on the dataset’s properties. Furthermore, the authors have addressed comments on the inference cost and the robustness under non-ideal communication conditions in their rebuttal. 

All in all, the article represents a valuable contribution to the field and of sufficient level for presentation at NeurIPS. As there were quite some shared comments between the reviewers, which were addressed in the rebuttal, I would strongly advise the authors to modify the article according to the comments. This said, there were some expressed desires for more comparison and / or inclusion of more agents in the evalution, but since the current study is already considered very solid and quite extensive, I think these recommendations should be more taken onboard for future work.
Decision: Accept (poster)