GT-Space: Enhancing Heterogeneous Collaborative Perception with Ground Truth Feature Space
Authors: Wentao Wang
Haoran Xu
Guang Tan
Venue: ICLR 2026 Poster (2026)
Keywords: Collaborative perception
multi-modality
multi-agent
sensor fusion
Primary area: applications to robotics, autonomy, planning

Abstract: In autonomous driving, multi-agent collaborative perception enhances sensing capabilities by enabling agents to share perceptual data. A key challenge lies in handling heterogeneous features from agents equipped with different sensing modalities or model architectures, which complicates data fusion. Existing approaches often require retraining encoders or designing interpreter modules for pairwise feature alignment, but these solutions are not scalable in practice. To address this, we propose GT-Space, a flexible and scalable collaborative perception framework for heterogeneous agents. GT-Space constructs a common feature space from ground-truth labels, providing a unified reference for feature alignment. With this shared space, agents only need a single adapter module to project their features, eliminating the need for pairwise interactions with other agents. Furthermore, we design a fusion network trained with contrastive losses across diverse modality combinations. Extensive experiments on simulation datasets (OPV2V and V2XSet) and a real-world dataset (RCooper) demonstrate that GT-Space consistently outperforms baselines in detection accuracy while delivering robust performance. Our code will be released at https://github.com/KingScar/GT-Space.

Published as a conference paper at ICLR 2026
GT-SPACE: ENHANCING HETEROGENEOUS COLLABO- RATIVE PERCEPTION WITH GROUND TRUTH FEATURE SPACE
Wentao Wang! Haoran Xu
Guang Tan' (Corresponding author) Sun Yat-sen University, Shenzhen Campus, Shenzhen, China
2 Peng Cheng Laboratory (PCL), Shenzhen, China wangwt66,, xuhr9) @ mnai2..sysu.edu.. tanguang @ mail.sysu..ed..om
ABSTRACT
In autonomous driving, multi-agent collaborative perception enhances sensing capabilities by enabling agents to share perceptual data. key challenge lies in handling heterogeneous features from agents equipped with different sensing modalities or model architectures, which complicates data fusion. Existing ap- proaches often require retraining encoders or designing interpreter modules for pairwise feature aligmment, but these solutions are not scalable in practice. To address this, we propose GT-Space, flexible and scalable collaborative perception framework for heterogeneous agents. GT-Space constructs a common feature space from ground truth labels, providing unified reference for feature alignment. With this shared space, agents only need a single adapter module to project their features, eliminating the need for pairwise interactions with other agents. Furthermore, we design a fusion network trained with contrastive losses across diverse modality combinations. Extensive experiments simulation datasets (OPV2V and V2XSet) and a real-world dataset (RCooper) demonstrate that GT-Space consistently outper- forms baselines in detection accuracy while delivering robust performance. Our code will be released at https://gttub..omm/ https//ggthubbooomKKnnSs/CCCCC
INTRODUCTION
Collaborative perception can enhance the sensing capabilities of interconnected vehicles through data sharing (Xu et al. 2022c Wang et al./ 2024c Xu et al. 2022b Wang et al. 2024b Hao et al. 2024). The sharing is typically conducted at the feature level, where agents exchange compressed feature data rather than raw sensor data to ensure communication efficiency. The collaboration is called homogeneous when the shared feature data aligned in both semantics and granularity (Yang et al., 2023a CCu et al., 2022, Lei et al. 2022), for example through the same encoder model; otherwise. is considered heterogeneous. The latter case is common in practice, the perception units, or agents, often differ in sensor modality or model architecture. If not properly handled, such an issue can significantly degrade the collaboration performance of the agents.
Current solutions (Xu et al. 2022b Xiang et al. 2023) to fuse heterogeneous feature data typically involve a feature adaptation step betore performing fusion. Two main adaptation strategies commonly used: (1) Encoder retraining (Lu et al.) 2024), illustrated in Fig. [a). Here, the ego agent runs both perception model and fusion network. To align with the ego feature space, collaborating agent needs to retrain its encoder. In open environments, this means maintaining multiple encoders, which highly expensive and inefficient; (2) Feature interpreter (Luo al.. 2024), shown in Fig. [b). In this case, the ego must be equipped with distinct interpreter for each heterogeneous agent, which poses a scalability issue similar to the first strategy. In both cases, collaboration performance is constrained by the ego agent's model capacity. If the ego model underperforms, the features from its collaborators may yield only limited benefits.
To address these issues, we propose CT-Space, flexible and efficient framework for collaborative perception. Unlike prior methods that require pairwise feature adaptation between heterogencous agents foreing each agent to maintain many specialized encoders or adapters method needs

---

Published as a conference paper at ICLR 2026
Ego
Nebs
Ego
Nebs
Nebs
E11
E#2
#1
E
E41
E #2
#3
E #3
E #3
Interp #2 #3
GT-Space Fusion
Fusion
Fusion
D
(a) Encoders re-training
(b) Feature interpreter
(c) Scalable GT-Space Fusion
Comparison heterogencous collaborative strategies. Retraining the encoders: the fusion network is frozen while the local encoder (denoted as E) is retrained to adapt data integration: (b) Interpreter- based alignment: local encoder detection head (denoted as are frozen, the interpreter can project received features into the ego's o feature space; (c) Our method does not require any re-training of encoders or heads, utilizes modality-agnostic fusion module to integrate heterogeneous modalities using ground-truth derived common feature space for alignment.
Figure 1:
only a single adapter model per agent. Fig. Te) depicts the principle of GT-Space, in which the features from individual agents are projected into common feature space, and then fused for downstream object detections.
The common feature space is constructed explicitly from ground-truth object information. For each scene, the object locations, sizes, and other properties are encoded into bird's bird's-eye view (BEV) features, which collectively form the common space. By using precise spatial and semantic information of the objects, this space provides a shared, accurate reference for aligning heterogeneous features. When new agent joins the system, only needs deploy a lightweight adapter to transform its features. This design enables scalable collaboration in open environments with minimal deployment cost.
The fusion network is responsible for aggregating the adapted features from multiple agents. The ground-truth features provide strong intermediate supervision signals, guiding the fusion process more effectively than supervision from final object detection outputs. As result, the collaboration performance is no longer limited by the weaker local model; instead, the system achieves overall improvement by exploiting complementary strengths across agents.
To ensure modality generalizability, we train the fusion network with combinatorial loss across all modality pairs. For instance, given three models LiDAR-based PointPillar (Lang al.) (209)). SECOND Yan et al. 2018), and namera-based EfficientNet Tan & Le (019)) the loss is computed over all three possible pairs. This strategy enables the model fuse any combination of input modalities at inference time.
The main contributions this paper are as follows:
We present GT-Space, flexible and efficient collaborative perception framework. The main novelty is a ground-truth derived common feature space for heterogeneous agents to align with. This approach greatly simplifies collaboration among agents, especially in an open environment.
We propose train the fusion network using combinatorial contrastive losses, that the fusion network can take as input arbitrary combinations of modalities.
Comprehensive experiments conducted on the simulation and real-world datasets demon- strate GT-Space's state-of-the-ar performance in terms of generalization, plug-and-play functionality and robustness to under-performing agents.
2
RELATED WORK
Collaborative Perception. Collaborative perception studies how to efficiently utilize shared percep- tual data to improve perception performance (Xu et al. [2026] 20226)Huang et al.) (2024) Yu et al. [022) Shi al. 2022). The fusion of shared data generally be performed in three ways: early, intermediate
2

---

Published as a conference paper at ICLR 2026
2022a; Li et al., 2024: Shi et al. 2022). Intermediate fusion (Xu et al. 2022c Fan et al. 2023) involves transmitting encoded feature data instead of the bulky raw data, and thus strikes a balance between accuracy and transmission cost. As such, it has been adopted as major solution in existing works. V2VNet (Wang et al. 2020) uses multi-round message passing via graph neural networks to achieve better perception performance. To address communication overhead, Hu et al. (Hu et al. (022) propose Where2comm, which enhances communication efficiency by reducing redundancy. Coopernaut (Cui et al. 2022) uses V2V sharing information to generate control policies for end-to-end autonomous driving. DiscoNet (Li et al. 2021) leverages knowledge distillation to enhance training by constraining the corresponding features to the ones from the network for early fusion. Some methods Yu et al., 2024 Wei et al. 2024 Lei et al. 2022) consider employing multi-frame features to address data transmission interruption or latency.
and late fusion (Xu et al.
Heterogeneous Collaboration. Heterogeneous collaboration has been significant topic in real multi- vehicle systems. V2X-ViT etu al. 2022b) utilizes a vision transformer to aggregate point cloud features from different agents, including vehicles and infrastructures. HM-ViT (Xiang et al. 2023 proposes a hetero-modal vision transformer to implement the feature alignment for heterogeneous sensor modalities including point clouds and RGB images. These end-to-end methods require training the entire model to fit particular combinations of modalities, which makes them inflexible when new modalities introduced. HEAL (Lu et al. 2024) introduces backward alignment training strategy, creating heterogeneous models by fixing a base pyramid fusion module and training only the encoders. However, retraining the encoders for the purpose of collaboration can potentially compromise the performance compared with the original encoders. PnPDA (Luo et al. 2024) proposes a plug-and- play domain adapter for aligning heterogeneous features without re-training encoders. However, the proposed adapter can only handle point cloud features and overlook sensor heterogeneity.
Multi-modality. Learning from multiple modalities has been an integral part of machine learning research (Lei et al. 2021; Xia et al. 2024; Yun et al.] [222] Arandjelovic & Zisserman (017)) In
the domain of intell ligett vehicles, a typical combination of multi-modalitiie has been cameras and
LiDARS (Zheng et al. 2024). BEVfusion (Liang et al. 2022), BEVGuide (Man et al.) 2023) and
RobBEV (Wang et al. 2024a) utilize Lidar-camera fusion to improve BEV perception in autonomous driving. (Li 1021 proposes method to project features of available modalities into common space for fusion. (Zhang et al. 2023) proposes a unified learning encoder to simultaneously extract representations from multiple modalities with the same set of parameters. In this paper, aim to develop a moodality-agnostie collaborative perception method that can be easily deployed on any agent enhance perception capabilities.
METHODOLOGY
3.1
BASIC PIPELINE
In the multi-agent system, each agent is equipped with its own sensors and perception models. The processing pipeline of each agent generally involves feature encoding, compression, transmission, decompression, fusion and decoding. Specifically, upon obtaining input raw sensor data O, each agent encodes the data into BEV feature vector F. To reduce transmission time, a compressor is used to compress F, to F before transmitting them to the collaborative agents. As a receiver, the agent receives compressed BEV features Fj from another agent and decompresses them to Fy. Then the fusion network collects and aggregates all received BEV features generate a consolidated BEV features H, Finally, the decoder processes H;; to output the detections B;.
3.2 COLLABORATION FRAMEWORK
Fig. [ ssowws the pipeline of the GT-Sppaee For the training process, we first train the perception network of a single agent, where raw sensor inputs are encoded into BEV feature maps and processed by a detection head to produce detection results, yielding a set of trained encoders. Next, the ground truth encoder is trained to map object labels into BEV representations, which a decoder reconstructs as bounding boxes, defining the ground feature space. Finally, during the training of the fusion network, the encoder weights are kept frozen, heterogeneous features are projected into a shared space via modality-specific projectors, and the network is trained using combinations of a few agents.
3

---

Published as a conference paper at ICLR 2026
Detectton
Agent
Similonty loss
07
feature
Projector #1
Space
Contraseive loss
GT Encoder
Region Chop
Callaborative Detection Head
Transformr
Object bounding labels
feature
Similonity
12
Detection Head
Agent
BEV feature #2
The framework of GT-Space. Heterogeneous features undergo domain conversion before being input into the fusion network. The ground truth object bounding box labels are loveraged to generate a common feature space. Heterogeneous features from different agents projected into the common feature space for alignment and fusion.
Figure 2:
During inference, each heterogeneous agent aligns its features to the ground feature space through its own projector, which then passed through the fusion network and subsequently through the collaborative detection head to generate the final results.
When previously unseen agent joins the collaboration, freeze all parameters and only train the projector assigned to this agent, enabling it to adapt to the pre-trained fusion network and participate collaborative perception.
3.3 COMMON FEATURE SPACE
Previous methods typically align heterogeneous features by projecting them into fully learned latent space. The representations are learned under the supervision signals from the object detection output. In contrast, our proposed common feature space is able to introduce additional feature level supervision derived from ground truth labels. The can help with bridging domain gaps between heterogeneous agents.
As shown in Fig. (a), given the ground truth bounding box annotations of the objects in the scene, use such label information to generate BEV features. Each 3D bounding box can be formulated as a vector:
(I) where (Ir y, 2) is the center point, (1, w. h) the length, width, and height of the 3D bounding box, respectively; r the yaw angle with respect to predefined axis, and e the category of the object. Given bounding box B,, we utilize two fully-connected (FC) layers with layer normalization (LN) to encode object-relevant information,
B₁ (xr, y, 2, w,,,
3 LayerNorm(FC((B))
where B, denotes the encoded representation.
Then we map the representations of objects onto the BEV plane to construct BEV features. A BEV feature frame is a grid map, where each grid cell of the map represents a region the scene (Li et al.) 2022: Yang et al. 2023b). A grid cell e is square area and is assigned a pair of integer coordinates [ee Ve). Let Ua be the feature of the c-th cell, defined as:
Ue MLP(3,, PE( Bc)),
(3)
where PE(-) denotes the Sinusoidal position embedding (Vaswani et al. 2017).
An object can cover multiple grid cells, thus the object B; comprises set of features SB, {U] U where n is the number of cells in B, If multiple objects cover the same grid cell c, up c's features from all the covering objects, and assign the aggregated feature to This preserves information of overlapping objects. In this way, we can obtain BEV map For comprising all the cells' features.
4

---

Published as a conference paper at ICLR 2026
GT BEV
Combinatorial Contrastive Loss
Object-Level Enhancing
Lor
MLP
Ground Truth Encoding
Nx Transformer Blocks
LayerNorm
Negative
Fair
Positive Pair
Instance Formatting
Multi-Head Attention
LayerNorm
Projector #1
Projector #2
Object bounding box labels
M1
M3
(a)
(b)
(c)
(d)
Feature alignment in GT-Space. (a) Ground truth features generation: The object bounding box labels are encoded and mapped onto the BEV map: (b) Multi-man molii- transformer: Each modality is mapped into common feature space by specific projector; (c) Combinatorial contrastive learning: M1, M2, and M3 denote three different modalities, and computing contrastive loss over all pairs of these modalities enhances the model's general feature extraction ability; (d) Object-level alignment: Contrastive learning pulls fcatures of the same object closer and pushes those of different objects apart.
Figure 3:
To ensure the effectiveness of the generated BEV feature map, following BEVFormer (Li et al. 2022). we feed the features encoded by Eq. 2 and Eq. and supervise the training process with the Intersection over Union (IoU) loss LGT:
3g.3iint into the detection head to output detection results,
K (1- LGT K
IP GrI/IP U.rb
where Gg denotes the ground truth bounding box, and Ps represents the output of the detection head for the k-th object, and K the total number of objects. This ensures that the encoded object-level BEV features can be decoded into bounding box outputs, thereby obtaining the ground truth BEV feature map Fcr.
3.4 HETEROGENBOUS FEATURE ALIGNMENT
Heterogeneity Alignment. For heterogeneous features, even feature elements with the same seman- ties may differ in dimension and parameter scales because of the domain gaps. As shown in Fig. (b), we adopt specific projector & that can project the local BEV feature into the common space as
follows:
S = arg minl, (FGT, Fa).
(5)
L, | (FF))
(6)
where Fcr, F, denotes the common feature map and local feature map generated by the a-th agent, respectively, 7) denotes the feature transformation and Ly, denotes the feature similarity loss function (Wang et al. 2025).
Relevant Feature Enhancement. ensure the model effectively captures critical information for
detection, we expect the fusion network to concentrate object-relevant features. We introduce transformer model to handle arbitrary heterogeneous features. As shown in Fig. (b), each block consists of multi-hean self-attention and FC layer with two LN transforms. It should be noted that the feature enhancement achieved through the training strategy.
We leverage contrastive learning to supervise the fusion network learning. Specifically, the goal is to align the fused features with the ground-truth -trth features. First, we assume that modalities m and m' are fused, resulting in the fused feature F Fm.ma. To compute the contrastive loss, the pooling operation is

---

Published as a conference paper at ICLR 2026
used for each object to obtain the mean features enable representation consisteney. For the grid cell a covered by the object bounding box B, we calculate the temperature-controll cosine similarity between the local BEV feature and the ground truth feature.
(F TUP
where F denotes the feature of grid cell in fused BEV map Fm,m' and U is the averaged ground feature for the object bounding box P. and is the temperature parameter. As shown in Fig 3 (d), the cross-entropy loss used to maximize the similarity between the fused features and ground truth features for the same object, while minimizing the similarity for different objects:
Lm = - BEB EeEcelte(B) log
exp(sn.c.n) Eies exp(#8.c.)
(8)
where B denotes the set of objects. Then we adopt combinatorial contrastive to enhance the model's ability to handle different modalities. As illustrated Fig. 3 (c), all possible modality pairs are considered the inputs, which can be defined as:
LE Σ Limm'
(9)
where Lm,m' denotes the contrastive loss between the fused feature of modality pair m and m' and the ground truth feature in Eq. 8 This combinatorial joint optimization helps the model effectively capture the object-relevant information, which is the core of heterogeneous perception features.
3.5 TRAINING
During the training of our framework, the models of individual agents, including local encoders and detection heads are pre-trained and frozen during the training. thus enabling plug-and-pla function- ality for easy deployment. Since our fusion objective is achieve alignment across heterogencous features, in order to avoid the noise caused by spatial misalignment, we train the model using the observation data from a single agent, which spatially aligned.
We use a series of pretrained encoders to obtain heterogeneous BEV feature maps of the same scene, and use a trained ground-truth feature encoder obtain the ground-truth BEV features for constructing the common feature space. For agent a, each BEV feature is fed into its corresponding projector & and transformed into the common feature space, then passed through the fusion network to produce fused features, which are finally fed into the detection head to generate detection results. Three losses are considered in the process of training: (I) feature alignment loss Lo,, (Eq. 6); (2) heterogeneous fusion loss Lg enhance the object-specific features (Eq. 9 (3) base BEV detection loss Lg. The overall training objective is therefore:
L L=
La. LE LB.
(10)
The ground truth BEV features are applied exclusively during the training process via L and LE without adding additional network architectures and parameters. Thus, we can obtain a general fusion model using few agents that can handle arbitrary modality inputs and effectively capture object-relevant features. This endows the model with scalability: when new agent with unseen modalities joins, it only requires training its projector to adapt to the trained fusion model and enable collaboration.
4 EXPERIMENTS
4.1 DATASETS
OPV2V (u al. 2022c OPV2V provides LiDAR point cloud and RGB data containing multiple autonomous vehicles. Using the OpenCDA (Xu al. 202I), cooperative driving framework and the CARLA Dosovitskiy et al. 2017) simulator, the dataset contains a total of 10,915 frames, which split into trin/validationntes with quantities of 6,765/1 /0002,802200 respectively.

---

Published as a conference paper at ICLR 2026
V2XSet (Xu al. 2022b). V2XSet builds upon OPV2V by adding roadside infrastructures, thereby introducing heterogeneous V2I collaboration.
RCooper (Hao et al. 2024). RCooper is large-scale roadside cooperative perception dataset with LiDAR and camera data, 3D bounding boxes, and diverse traffic scenarios, supporting multi-modal 3D detection and tracking. The results shown in Appendix. D
4.2 EXPERIMENTAL. SETTINGS
Evaluation Metrics. We adopt 3D detection accuracy as the metric to evaluate the performance of the model, which is measured by average precision (AP) at IoU thresholds of 0.5 and 0.7. The perception area of the agents are set to E [-140.8m, 140.8m), y E [-40m, 40m].
Implementation details. As shown in Table. [ we assume four types of agents, including two LiDAR models and two camera models. Agent 2 acts as vehicle when using the OPV2V dataset, whereas in V2XSet,
set as an infrastructure unit. We build our model using the OpenCOOD framework (Xu et al.] 2022c) and train it on all datasets using an NVIDIA A100) GPU. For the local encoders and detection heads, we follow the same hyperparameters as prior works (Luo et al. 22024] Shao et al. 2024). Local models and ground truth encoder are trained for 30 epochs using the Adam optimizer Kingma & Ba 2014). During the training of the fusion model, the original encoders and detection heads of all the agents are frozen, ensuring
fusion model does not affect the original perception capability of individual agents when collaborative perception is not in use.
Compared Methods. We compare our method with existing methods on 3D object detection. Non- collaboration is considered as our baseline method, which only use the ego's perception data. We also evaluate the Late Fusion, in which agent transmits its detection results and the ego leverages Non- maximum suppression to generate the final predictions. For the intermediate collaborative methods, we benchmark the following approaches: End-to-end training (Xiang al. 2023), PnPDA (Luo et
2024.. HEAL
(Lu et al. 2024). Hetercooper (Shao et al. 2024) and S STAMP (Gao et al. 2025).
Table 1: Settings of heterogeneous agents in the experiments.
Agent
Agent 1
Agent 2
Agent 3
Agent 4
Carrier
Vehicle
Infrastructure Vehicle
Vehicle
Vehicle
Seasor
LIDAR
LIDAR
Camera
Camera
Model
SECOND (Yan al. 2018
PaintPillar Lang
2019
EfficientNet Tan Le 2019
ResNet50 (He al. 2016
4.3 QUANTITATIVE RESULTS
Performance of Different Modality Pairs, We compare the detection performance of GT-Spacee against other collaborative methods under heterogeneous perception settings, as shown in Tab. [ Agent is fixed as the ego agent, while agents A2, A3, and A4 serve as collaborators. Across all cases, GT-Space consistently outperforms competing methods, due to the shared common feature space, which provides explicit object-level supervision for feature alignment and leads to more effective feature fusion.
As expected, LiDAR-based perception achieves higher detection accuracy than RGB-baser methods, owing to the richer spatial information inherent in point clouds. Notably, our approach yields larger gains for agent pairs with higher modality heterogeneity. highlighting its ability to bridge representation gaps across different domains. Specifically, the use of contrastive learning in GT-Space enhances object-relevant features in way that end-to-end training or feature interpreter methods cannot achieve.
Heterogeneous Collaboration. We further evaluate the perception performance of four heteroge- neous agents, A1-A4, as listed Tab. 3 The results on OPV2V and V2XSet are in shown in Tab, 3 The interpreter-based PnPDA and STAMP, provide substantial improvements for LiDAR agents; however, their ability to enhance camera agents remains limited. This is because spatial information in point clouds can be lost during the interpretation process and is difficult to recover with frozen camera detection head. In contrast, GT-Space leverages the common feature space as a reliable reference for heterogeneous feature fusion, thereby achieving the best performance across all agents.
7

---

Published as a conference paper at ICLR 2026
Table 2:
Comparison of fusion performance for different heterogeneous modality pairs using A and
AP@ 70 metrics. A1. A2, A3 and A4 refer to agent 1, agent 2. agent and agent 4, respectively.
Method
AP@50 f
AP@
Dataset
Ego Al, collaborates with
A2
A3
A4
A2
A3
A4
Non-collaboration
0.738
0.738
0.738
0.614
0.614
0.614
Late fusion
0,821
0.772
0.766
0.680
0.615
0.613
HM-ViT Xiang
2023
0.857
0.814
0.807
0.731
0.646
0.652
OPV2V
PnPDA
Luo et al. 2024
0.878
0.809
0.802
0.792
0.651
0.653
HEAL
Lu al. 2024
0.887
0.827
0.823
0.801
0.724
0.728
Hetecooper
Shao al. 2024
0.881
0.812
0.807
0.804
0.703
0.684
STAMP
Gao ct al 2025
0,876
0.833
0.827
0.806
0.734
0.738
paace (ours)
0.891
0.848
0.844
0.810
0.766
0.762
Non-collaboration
0.671
0.671
0.671
0.537
0.537
0.537
Late fusion
0.719
0.674
0.667
0.629
0.541
0.548
HM-ViT Xiang
2023
0.803
0.741
0.736
0.710
0.625
0.631
V2XSet
PnPDA
Luo et al.
2024
0,835
0.749
0.743
0.763
0,623
0.634
HEAL Lu
2024
0.854
0.785
0.791
0.778
0.693
0.688
Hetecooper (Shao et al. STAMP
2024
0,860
0.759
0.747
0.782
0,684
0.680
Gao et al. 2025
0.858
0.801
0.796
0.774
0.703
0.692
GT-Space (ours)
0.874
0.826
0.830
0.802
0.741
0.738
Note that the improvements are more pronounced for camera agents, reflecting the strength of our approach in compensating for weaker agents.
Robustness Under-Performing Agents. Agents with weaker capabilities may contribute low- quality features, which may bring down the collaboration performance. We further investigate the robustness the system with respect to under-performing agents, as shown Fig. The weaker agents are obtained by modifying the training setups of their perception models. It can be seen that our method outperforms the baselines collaboration, because the common feature space offers a strong reference for the agents to align with, and the centralized fusion network is sufficiently trained with full cross- cooss-modlltty losses. Since LiDAR agents provide more precise perception information in the collaborative system, the performance gains in heterogeneous scenarios primarily depend LiDAR. Consequently, compared to camera agents, variations in LiDAR agents' performance lead to larger fluctuations in the overall collaborative performance.
Table
Perception performance different agents in multi-agent scenarios.
A
A AP f
Dataset
Method
A3
A4
A1
A2
A3
A4
Non-collaboration
0.738
0.744
0.402
0.396
0.614
0.620
0.354
0.337
Late fusion
0.818
0.818
0.818
0.818
0.690
0.690
0.690
0.690
HM-VIT
Xiang et al 2023
0.852
0.853
0.835
0.837
0.756
0.752
0.722
0.725
OPV2V
PnPDA
Luo et al. 2024
0.861
0.864
0.828
0.833
0.798
0.794
0.719
0.716
al 2024
0.842
0.843
0.806
0.801
0.726
0.733
Hetecooper
0.876
0.881
0.836
0.838
0,802
0.798
0.731
0.739
STAMP
Gao et al. 2025
0.883
0.886
0.840
0.831
0.815
0.801
0.7188
0.716
GT GT-Space (ours)
0.892
0.894
0,867
0.859
0.814
0.803
0.758
0.750
Non-collaboration
0.671
0.665
0.389
0.384
0.537
0.542
0.323
0.330
fusion
0.798
0.798
0.798
0.798
0.628
0.628
0.628
HM-ViT
Xiang
2023
0.824
0.818
0.809
0.811
0.715
0.713
0.698
0.692
V2XSet
PnPDA
2124
0.831
0.822
0.812
0.806
0.764
0.752
0.694
0.697
HEAL
Lu al. A144
0.859
0.860
0.823
0.826
0.786
0.781
0.698
0.701
Hetecooper
Staan el al.
2024
0.852
0.849
0.818
0.816
0.780
0.781
0..266
0.724
STAMP
Gao ot al. 2025
0.854
0.848
0.817
0.815
0.801
0.794
0.709
0.717
GT- GT-Space (ours)
0.873
0.866
0.845
0.848
0.806
0.804
0.742
0.733
Robustness to Imperfect Localization. Existing experiments typically assume that each agent has access to an accurate pose. However, in real-world scenarios, localization noise is inevitable, leading to imperfect spatial alignment across agents. To evaluate robustness under such conditions,
8

---

Published as a conference paper at ICLR 2026
Collaboration (mAP@50)
0.51
Collaboration (mAP@@50)
0.88
sEnquce
0.1
0.80
0.75
0.88
0.1
0.88
0.51
0.56
0.81
0.66
0.71
0.78
000
0.30
0.32
0.34
0.38
0.38
0.40
No-Coolabooation (mAP@50)
No-Collaborrtion (mAP@50)
Figure 4: Performance of under-performing agents OPV2V. The left column represents Agent 1, and the right column Agent The borizontal axis represents performance without collaboration, while the vertical axis represents collaboration.
Performance APS0
Performance AP70
Per Performannc APSO
6.3
6/4
Performance
0.4
0.4
HEAL,
HEAL
--- HEAL
PHHMA
PxPOA STANF
PxPDA
STAMP
STAMP
GT-Soace
G1 Space
Pose Noise Std (m & degl lal
Pose Noise Std (m & deg) [b]
Comm. Latency [ms]
Comm. Latency (ms)
(c)
(d)
Figure 5: Robust Experiment to pose error and communication latency.
we introduce pose-error experiment by adding Gaussian noise into the ground-truth poses, where the pose noise set to (0, o.) on x.y location and NO x²) on yaw angle (Lu et al., 2023). Al is set as the ego agent and the performance is shown in Fig. The results demonstrate that our method consistently maintains state-of- oftte-of-thearr performance across wide range pose eттоr conditions.
Robustness to Communication Latency. In real-world scenarios, communication latency is in- evitable. We further evaluate the robustness our method under different latency conditions. Since the sensor data in the OPV2V dataset is recorded at 10 Hz, simulate communication latency of k x 100 ms by replacing the first & transmitted feature frames with the current frame for each collaborative sample, thereby emulating the latency effect. Following (Yu al. 2024), the range of communication latency to 100-500 milliseconds, Results in Fig. demonstrate that even with 500 ms latency, our method still outperforms the baseline methods.
4.4 ABLATION STUDIES
To investigate the effectiveness of each component in system, we introduce four variants of our GT-Space: (1) w/o-GT features, which replaces the ground-truth feature space with unified feature space generated by PointPillar; (2) w/o-projector, which cludes the feature projector, i.e., feed the heterogeneous features directly into the trans- former for fusion; (3) w/o-combinatorial contrastive loss, which trains the fusion network solely with the basic detection loss, without the proposed combinatorial contrastive losses. Tab. reports the performance of Agent under these design variants. The results show that removing the feature projec- tor causes the largest performance degra- dation. This because heterogeneous features have distinct semantic represen- tations, making it difficult for the fusion network to effectively distinguish and ag- gregate them without proper alignment.
Table 4: Efffet of each design component in the model.
Configurations
OPV2V
V2XSet
mAP@:5
mAP@ 70
mAP@:50
mAP@ 70
who GT feature
0.868
0.795
0,830
0.782
wlo-projectre
w/o-contrastive loss
0.845
0.721
0.823
0.709
Full version
0.892
0.814
0.873
0.806
For Agent 1, a LiDAR agent, replacing the GT feature
space with the point cloud feature space still yields reasonably good performance, as the point

---

Published as a conference paper at ICLR 2026
(a) Feature map before fusion
(b) Feature map after fusion
Figure 6: Visualization of BEV features before and after fusion. (a) Features before fusion of A2 (PointPillar); (b) Enhanced features after fusion.
(a)
(b)
(c)
Figure 7: Visualization of (a) Ground truth feature, (b) projected camera feature a ceam Feam) and
(c) projected LiDAR feature Piddr Fiidar).
cloud representation inherently preserves rich geometric cues. Finally, removing the combinatorial contrastive loss results in noticeable decline in detection accuracy, since the contrastive objective not only enforces modslty alignment but also strengthens feature representations.
5 VISUALIZATION
Fig. 6 presents the intermediate feature maps of A2 before and after data fusion. A clear contrast be observed: prior to fusion, the object-relevant features exhibit weak activations and heavily contaminated by noise. After fusion, however, the feature map highlights significantly more object information, confirming the effectiveness of collaborative perception, while the noise notably suppressed, reflecting the role of ground-truth features strong supervision signals.
We further present the visualization of ground truth feature, projected camera feature C (Fcam) and projected LiDAR feature Ane (F).DARR)) As shown in Fig. 7 we can observe that the ground truth feature map contains only object-related features. Since point cloud features include richer spatial information, the mapped FiLDR still retains some road information. As for the camera features Fcam: note that the projector does not have the ability to enhance features; therefore, object-related features in the feature map not abundant in FLIDAR-
CONCLUSION
In this paper, we have presented GT-SSacce, scalable and high-accuracy collaborative perception method aimed at fusing shared data among agents that are heterogeneous in sensor modality or data encoders. Ground truth object labels are used to construct a common feature space for feature alignment and fusion. To enable the fusion network to handle arbitrary input modalities and enhance relevant features, we employ a combinatorial contrastive loss for training. Experiments on the OPV2V, V2XSet and RCooper datasets validate the effectiveness of GT-Space. GT-Space depends on ground truth annotations and ideal communication/pos conditions. Future work will focus on weakly-superviseed learning to enhance its
applicability.
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGMENTS
This work was supported by Shenzhen Natural Science Foundation under grant 2024| 002412023000612
REFERENCES
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, pp. 609-617, 2017.
Jiaxun Cui, Hang Qiu, Dian Chen, Peter Stone, and Yuke Zhu. Coopernaut: End-to-end driving with cooperative perception for networked vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17252-17262, 2022.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pp. 1-16. PMLR, 2017.
Siqi Fan, Haibao Yu, Wenxian Yang. Jirui Yuan, and Zaiqing Nie. Quest: Query stream for vehicle- infrastructure cooperative perception. arXiv preprint (xXv:2308.00800 2023.
Xiangbo Gao, Runsheng Xu, Jiachen Li, Ziran Wang, Zhiwen Fan, and Zhengzhong Tu. Stamp: Scalable task and model-agnostic collaborative perception. arXiv preprint arXiv:2501.186/1 2025.
Ruiyang Hao, Siqi Fan, Yingru Dai, Zhenlin Zhang, Chenxi Li, Yuntian Wang, Haibao Yu, Wenxian Yang. Jirui Yuan, and Zaiqing Nie. Rcooper: A real-world large-scale dataset for roadside cooperative perception. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22347-22332 2024.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Yue Hu, Shaoheng Fang, Zixing Lei, Yiqi Zhong, and Siheng Chen. Where2comm: Communication- efficient collaborative perception via spatial confidence maps. Advances in neural information processing systems, 35:4874-4886 2022.
Suozhi Huang, Juexiao Zhang, Yiming Li, and Chen Feng. Actformer: Scalable collaborative perception via active queries. arXiv preprint arXiv:2403.04966 2024.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980 2014.
Alex H Lang. Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Point- pillars: Fast encoders for object detection from point clouds. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12697-12705, 2019.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and aosoo-nn--lanuuae lnnguagg learning via sparse sampling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7331-7341, 2021.
Zixing Lei, Shunli Ren, Yue Hu, Wenjun Zhang, and Siheng Chen. Latency-aware collaborative perception. European Conference on Computer Vision, pp. 316-332. Springer, 2022.
Yiming Li, Shunli Ren, Pengxiang Wu, Siheng Chen, Chen Feng, and Wenjun Zhang. Learning dis- tilled collaboration graph for multi-agent perception. Advances Neural Information Processing Systems, 14:29541-2955 2021.
Yiming Li, Zhiheng Li, Nuo Chen, Moonjun Gong, Zonglin Lyu, Zehong Wang, Peili Jiang, and Chen Feng. Multiagent multitraversal multimodal self-driving Open mars dataset. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12051, 2024.
11

---

Published as a conference paper at ICLR 2026
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eye s-eye-view representation from multi-camera images via spatiotemporal transformers. In European conference on computer vision, pp. 1-18. Springer, 2022.
Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang. Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. Advances in Neural Information Processing Systems, 35: 35:10421-10433 2022.
Yifan Lu, Quanhao Li, Baoan Liu, Mehrdad Dianati, Chen Feng. Siheng Chen, and Yanfeng Wang. Robust collaborative 3d object detection in presence of pose errors. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 4812-4818. IEEE, 2023.
Yifan Lu, Yue Hu, Yiqi Zhong, Dequan Wang, Yanfeng Wang, and Siheng Chen. An extensible framework for open heterogeneous collaborative perception. arXiv preprint arXiv:2401. 13964, 2024.
Tianyou Luo, Quan Yuan, Guiyang Luo, Yuchen Xia, Yujia Yang, and Jinglin Li. Plug and play: A representation enhanced domain adapter for collaborative perception. In European Conference on Computer Vision, pp. 287-303. Springer. 2024.
Yunze Man, Liang- Yan Gui, and Yu-Xiong Wang. Bev-guided multi-modality fusion for driving per- ception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21960-21969, 2023.
Congzhang Shao, Guiyang Luo, Quan Yuan, Yifu Chen, Yilin Liu, Kexin Gong, and Jinglin Li. Hetecooper: Feature collaboration graph for heterogeneous collaborative perception. In European Conference on Computer Vision, pp. 162-177 178. Springer, 2024.
Shuyao Shi, Jiahe Cui, Zhehao Jiang, Zhenyu Yan, Guoliang Xing, Jianwei Niu, and Zhenchao Ouyang. Vips: Real-time perception fusion for infrastructure-asistte autonomous driving. In Proceedings of the 28th Annual International Conference on Mobile Computing And Networking. pp. 133 146, 2022.
Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas, Brian Kingsbury, Rogerio S Feris, David Harwath, James Glass, and Hilde Kuehne. Everything at mcc-multi-modal fusion transformer for video retrieval. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp. 20020-20029, 2022.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105-61 14. PMLR, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2017.
Jian Wang, Fan Li, Yi An, Xuchong Zhang, and Hongbin Sun. Towards robust lidar-camera fusion in bev space via mutual deformable attention and temporal aggregation. IEEE Transactions Circuits and Systems for Video Technology. 2024a.
Shaohong Wang. Lu Bin, Xinyu Xiao, Zhiyu Xiang, Hangguan Shan, and Eryun Liu. Iftr: instance-level fusion transformer for visual collaborative perception. In European Conference Computer Vision, pp. 124-141. Springer, 2024b.
Tsun-Hsuan Wang, Sivabalan Manivasagam, Ming Liang, Bin Yang, Wenyuan Zeng, and Raquel Urtasun. V2vnet: Vchicle-to-vehee vehicce communication for joint perception and prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 11 16, pp. 605 621. Springer, 2020.
Wentao Wang, Haoran Xu, and Guang Tan. Intercoop: Spatio-temporal interaction aware cooperative perception for networked vehicles. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 14443-14444 IEEE, 2024c.
Wentao Wang, Jiaqian Wang, Yuxin Deng, and Guang Tan. Bevsyne: Asynchronous data align- ment for camera-based cchicle-infrastrucee cooperative perception under uncertain delays. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 14699-14707, 2025.
12

---

Published as a conference paper at ICLR 2026
Sizhe Wei, Yuxi Wei, Yue Hu, Yifan Lu, Yiqi Zhong, Siheng Chen, and Ya Zhang. Asynchrony-robure collaborative perception via bird's eye view flow. Advances in Neural Information Processing Systems, 36, 2024.
Xinshuo Weng, Jianren Wang. David Held, and Kris Kitani. 3d multi-object tracking: A baseline and new evaluation metrics. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10359-10366. IEFE, 2020a.
Xinshuo Weng, Jianren Wang, David Held, and Kris Kitani. Ab3dmot: A baseline for 3d multi-object tracking and new evaluation metrics. arXiv preprint arXiv:2008. 08063, 2020b.
Yan Xia, Hai Huang, Jieming Zhu, and Zhou Zhao. Achieving cross modal generalization with multimodal unified representation. Advances in Neural Information Processing Systems, 36, 2024.
Hao Xiang, Runsheng Xu, and Jiaqi Ma. Hm-vit: Hetero-modal rehicle-to-vehicne cooperative perception with vision transformer. arXiv preprint arXiv:2304. 10628, 2023.
Runsheng Xu, Yi Guo, Xu Han, Xin Xia, Hao Xiang, and Jiaqi Ma. Opencda: open coopera- tive driving automation framework integrated with co-simulation. In 202/ IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 1155-1162. IEEE, 2021.
Runsheng Xu, Zhengzhong Tu, Hao Xiang, Wei Shao, Bolei Zhou, and Jiaqi Ma. Cobevt: Cooperative bird's eye view semantic segmentation with sparse transformers. arXiv preprint arXiv:2207.02200: 2022a.
Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuie Hsuan Yang. and Jiaqi Ma. Vix-vit: Vehicle-to-everything cooperative perception with vision transformer. In European conference on computer vision, pp. 107-124. Springer, 2022b.
Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma. Opv2v: An open bench- mark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. In 2022 International Conference on Robotics and Automation (ICRA), pp. 2583-2583 IEEE, 2022c.
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18 (10):3337, 2018
Kun Yang, Dingkang Yang, Jingyu Zhang, Mingcheng Li, Yang Liu, Jing Liu, Hanqi Wang, Peng Sun, and Liang Song. Spatio-temporal domain awareness for multi-agent collaborative perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2023a.
Lei Yang, Kaicheng Yu, Tao Tang, Jun Li, Kun Yuan, Li Wang, Xinyu Zhang, and Peng Chen. Bevheight: A robust framework for vision-based roadside 3d object detection. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pp. 2161 21611-2162D, 2023b.
Haibao Yu. Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, et al. Dair-v2x: A large-scale dataset for vehicle-infrastreeuure cooperative 3d object detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 21..1331.... 21361-21370, 2022.
Haibao Yu, Yingjuan Tang, Enze Xie, Jilei Mao, Ping Luo, and Zaiqing Nie. Flow-based feature fusion for rehhill-iifrastructttee cooperative 3d object detection. Advances in Neural Information Processing Systems, 36, 2024.
Sukwon Yun, Inyoung Choi, Jie Peng, Yangfan Wu, Jingxuan Bao, Qiyiwen Zhang, Jiayi Xin, Qi Long, and Tianlong Chen. Flex-moe: Modeling arbitrary modality combination via the flexible mixture-of-experts. Advances in Neural Information Processing Systems, 37:98782-98805, 2024.
Yiyuan Zhang, Kaixiong Gong. Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. treaatrrnsformer: A unified framework for multimodal learning. arXiv preprint ar0iv:2307. 10802, 2023.
13

---

Published as a conference paper at ICLR 2026
Chao Zheng, Li Liu, Yu Meng, Xiaorui Peng, and Meijun Wang. Few-shot point cloud semantic segmentation via support-quer, feature interaction. IEEE Transactions on Circuits and Systems for Video Technology, 2024.
Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4490-4499, 2018.
14

---

Published as a conference paper at ICLR 2026
A COLLABORATIVE PERCEPTION
collaborative perception system consists of multiple agents, each equipped with its own sensors and perception models. The collaboration model is generally composed of encoder, a compressor, a decompressor, a feature fusion module and decoder, where i denotes the agent.
The process of the i-th agent be formally represented as follows:
Feature Encoding:
F; (O,),
(1a)
Compression:
F. (F.),,
(1b)
Transmission:
f ")-i(fr), E S
(1c)
Decompression:
FJ (1F
(1d)
Fusion:
H: ((F()))))s,
(le)
Decoding:
B; (HH,),
(1f)
where E; is the encoder, o; is compressor, is the communication operator, ab is the fusion module, and H, the aggreagated feature and B, is the final output obtained by the detection decoder D,.
B SUMMARY OF SYMBOLS
We summarize all the relevant symbols in Tab.
Table Summary of Symbols.
Symbol
Meaning
Fa
Feature map from a-th agent. Ground truth feature map. i-th bounding box in frame. Encoded representation of B3
For
B,
B;
Ue
The feature of the e-th cell The feature of the n-th cell of box B; The set of features of box B, Intersection over Union (loU) loss Feature similarity loss function. Feature projector for the a-th agent. The fused feature of modality m and m'.
U
Sp.
Let
Ly
P
F Fm
F B.C
m
The feature of grid cell in fused BEV map Fm.m²
T
Temperature parameter.
UP
The averaged ground feature for the object bounding box P.
C
IMPLEMENTATION DETAILS
Compared Methods. GT-Space is compared with seven baselines:
Non-collaboration Method;
Late fusion which transmits detected results;
HM-ViT (Xiang et al. 2023) based different encoders using end-to-end training;
PnPDA (Luo et al. 2024). PnPDA adopts modality interpreter to transform heterogeneous
features to a semantic space for fusion.
HEAL (Lu al. 2024). HEAL firstly trains a base collaboration fusion network with frozen parameters and re-traii re-trains the encoder for generalized fusion.
Hetecooper (Shao et al., 2024). Hetecooper constructs a graph transformer to achieve heterogeneous collaboration.
15

---

Published as a conference paper at ICLR 2026
Encoder
Fuslion Mede
Fissenn Mindle
Detection Head
Output
Output
Detection
Hlead
Curpurs
(s) Indio-end Training
(b) Late fuston
(c) Encoder ra- re-training
Encodor
Eicoder
Encoder
Projector
Common feoture spare
Common fcoture
Fusion Model
Detection Head
Detection
Fusion Mocel
Fusion
Funed putppu
Derecctin
Detoction Head
Desection Head 2
Dellccion
Detection
Haad
Head
Head
Single output
output
output
Output
Output
nutput
Interprenee fusion
(W) Adaattan Revertte Reverter fusion
Outs
Figure 8: (a) End-to-end training; (b) Late fusion: (c) Encoder re tr-aiiniin:: (d) Interpreter- pessee based method; ReJAdapterRReverter Reverter method; (f) GT-Space(Ours)) Blue indicates frozen modules, while red represents modules that updated during training.
STAMP (Gao al. 2025). STAMP configures adapter and reverter for each modality, enabling flexible transformation from local modality features to protocol features.
Architectures of Different Methods. Fig. & shows various heterogeneous collaborative frameworks. Late fusion simply transmits and combines agent detected results after local processing. HM- ViT (Xiang et al. (023) adopts end-to-end training, which is effective but interferes with individual perception. HEAL (Lu al. 2024) employs the fixed detection decoder and fusion model and re- trains only the local encoders, but still faces the issue of the coexistence of individual perception and collaborative perception. Interpreter- based methods adopt modality sppecifcc interpreters to transform heterogeneous features into common semantic feature space for alignment. The Adapter- Reverter method is a variant of interpreter-be into common feature space, and then employs a reverter map the common feature space back to the local features.
approaches. first uses an adapter to project local features
Projector Architecture. We use the same architectures that is MLP for projectors across all heterogeneous agents. The dimension of the broadcasting feature is set as (128, 128, 64). The input dimension of projectors varies according to the feature dimension of different local models. We utilize a NVIDIA A100 GPU for training. All local models are trained on the OPV2V and V2XSet for 30 epochs, with learning rate of 0.001 and weight decay of 0.01.
16

---

Published as a conference paper at ICLR 2026
D ADDITIONAL RESULTS
System Delays. To demonstrate the deployability of our proposed collaborative system, we test the
system delays of different components of the model.
Tab. 6 presents the latency breakdown of different components in the system, revealing that the majority of the computational cost is attributed the encoding of raw perception data. In contrast, both the fusion network and the detection head consume relatively little computational resources. also worth noting that our method achieves better performance without significantly increasing the computational cost. This improvement stems from our enhanced training strategy, rather than the use of more complex model architectures, which would have resulted increased parameters.
Table 6: System delays
Method
Encoding (ms)
Fusion (ms)
Detection (ms)
HM- ViT
Xiang et al. 2023
7.08
1.38
0.95
PnPDA
Luo ct al. 2024
7.08
1.41
0.98
Hetecooper
(Shao
al. 2024)
7.08
1.47
0.97
HEAL
Lu et al.
2024)
7.08
1.54
0.95
STAMP (Gao et 2025
7.08
1.52
0.94
GT-Space
7.08
1.48
0.97
Experiments on Real-world Dataset. To further evaluate the performance of our proposed GT- Space, we conduct experiments on RCooper (Hao et al. 2024), a real-world dataset. Tab. presents the performance of approach RCooper (AP@ 50), where the baselines the same those in Tab. [ The scenario is an intersection and all 4 agents are roadside infrastructures. It can be seen that on the real-world dataset, our method still achieves the state-of-art performance, especially showing better results for camera agents with weaker perception capabilities.
Table 7: Performance on RCooper
Method
A1
A2
A3
A4
No fusion
0.441
0.438
0.231
0,228
HM-VIT
Xiang et al.
2023
0.458
0.467
0.294
0.287
PnPDA
(Luo et al. 2024
0.463
0.468
0.321
0.330
Hetecooper
Shao
al.
2024)
0.471
0.465
0.319
0.324
HEAL
(u et al. 2024)
0.478
0.473
0.343
0,346
STAMP
Gao al.
2025
0.473
0.469
0.338
0.342
GT-Space
0.477
0.475
0.349
0.351
Experiments on Tracking Task. We further evaluat the performance of framework the tracking task. Specifically, we conduct experiments on the RCooper (Hao et al.) (024) dataset, which adopts AB3Dmot tracker (Weng al. (2020b) to perform object tracking based on the detection results. We adopt two evaluation metrics in (Weng et al. (2020a). 1) average multi-object tracking accuracy (AMOTA) and 2) average multi-object tracking precision (AMOTP). As shown Tab. we can see that our method outperforms all baseline models, which is consistent with the 3D detection results.
Newly Added Agent. Our proposed framework can adapt to newly added, previously unseen agents. but this requires a specific training strategy. Specifically, we still adopt the pipeline shown Fig. 2 For the fusion network and the collaborative detection head, freeze their parameters. At the same time, we also keep the encoder parameters of the newly added agent frozen, and train only its specific projector. The loss function can be expressed as:
L Lo. + LB-
where d; is the projector for the newly agent which is used to project its local feature to the common feature space and La, is the same as Eqa Ln is the base detection loss. Tab, shows the performance of A when the collaborative system adds new agents A3 and A4.
17

---

Published as a conference paper at ICLR 2026
Table 8: Performance of tracking task on RCooper
Method
AMOTA
AMOTP
No fusion
0.083
0.227
HM- VIT
Xiaag et al. 2023
0.224
0,354
PnPDA
[Luo et al. 2024
0.228
0.357
Hetecooper
(Shao et
2024
0.227
0.361
HEAL
(Lu al.
0.232
0.364
STAMP
Gao et al.
2025
0.231
0,359
GT-Space
0.236
0.367
We can see that retraining-hased HEAL clearly outperforms interpreter-based PnPDA and Adapter- Reverter-based STAMP, but at the cost of impairing the newly added agent's local perception. Although the projector use also serves modality conversion function, it does not convert inputs between agent pairs. Instead, it maps them into the shared common space. Meanwhile, the fusion network effectively enhances object-relevant representations, thereby preventing the loss of critical information.
Table 9: Newly adding agents.
Method (Based on AI, A2, Add)
Base
+ A3
+ A4
PnPDA (uoo al. 2024
0.878
0.857
0.864
HEAL
Lu ct al. 2024)
0.887
0.891
0.895
STAMP
Gao et 2025
0.876
0.863
0.862
GT-Space
0.891
0.892
0.897
Efficiency Comparison. To evaluate the training efficiency for the newly added agents, we addition- ally include four agents: A5, A6, A7, and A8. Tab. reports the sensors, encoders, and encoder parameters for agents A1-A8. PointPillar (large) and SECOND (large) are obtained by increasing number of hidden-layer parameters of the original models.
We then conduct an efficiency comparison between our method and baselines. Fig. Issows the number of training parameters and estimated training GPU hours. It can be seen that the retraining- brased HEAL incurs high training costs. The interpreter-based PnPDA and the
Retieer-Reeeeer---e STAMP significantly reduce the cost, though with some performance degradation. Together with Tab. 9 retraining-bel based HEAL achieve good performance but at high cost, while the interpreter-basse method have lower cost but yield weaker results. In contrast, our proposed framework achieves strong performance with low cost.
Table 10: Settings of heterogeneous agents in the experiments.
Agent
Apent
Agent
Cartier
Veticle
Infrastracture / Vehicle
Vehicle
Veticle
Semsu
LIDAR
LIDAR
Canera
Camera
Yan 201s
PounPhllar
EdfiiienNett Tan Le 2019
(Tl 2010
Enxoder Purmm (M)
3.70
0:87
$6.85
6.83
Agent
Agent
Agent
Agent 7
Agent $
Canier
Vehicle
Vihide
Vehicle
Sessor
LIDAR
LIDAR
LIDAR
Cunera
Model
VoseNNa (Phon Tuxel Tuze]
PoiseP (1 (lagg [ Fang (019]
SRCOND (lange)
Vin al 2018
RecNet34 (He
2016
(M)
Robustness to Data Distribution Shifts. To validate the robustness to data distribution shifts, we further evaluated the fusion model trained using OPV2V the V2X-Set dataset. As shown in Tab. []] our method consistently outperforms the baseline models even under distribution shifts, demonstrating the robustness of the ground-truth features.
18

---

Published as a conference paper at ICLR 2026
Training Parameters (M)
Training GPU Time (A100)
E2E
E2E
120
HEAL
HEAL
Parameters (M)
PnPDA
PnPDA
90
STAMP
Time (hours)
STAMP
GT-Space
40
GT-Space
60
3
Number of Agents (M)
Number of Agents (M)
(a)
(b)
Figure 9:
Training efficiency comparison for the different number of newly adding agents.
Table 11:
Performance (AP50) of agent A A1-A4 in V2XSet using the fusion model trained on OPV2V.
Dataset
Method
A3
A4
fusion
0.798
0.798
0.798
0.798
HM- VIT
Xiang et al. 2023
0.809
0.793
0.792
0.788
PnPDA
Luo al.. 2024
0.819
0,804
0.801
0.790
V2XSet
HEAL
La al 2024
0.844
0,842
0.818
0.822
Hetecuoper
Shao et al. 2024
0.835
0.821
0,810
0,803
STAMP Gao ct al. 2025
0.842
0.829
0.801
0.803
GT GT GT-Space (ours)
0.858
0.846
0.828
0.833
Table 12: Performance (AP50) models trained with different temperature parameters
Dataset
Temperature
A1
A2
A3
A4
0.05
0,845
0.835
0,830
0,826
OPV2V
0.07
0,850
0.842
0.824
0.830
0.1
0.858
0.846
0.828
0.833
Sensitivity Analysis for Temperature Parameter. In contrastive learning, the temperature used to adjust the "sharpness" of the similarity distribution, thereby controlling how strongly the model distinguishes between positive and negative samples. We evaluate the model's performance under different temperature settings. Based empirical practice in multimodal tasks, the temperature parameter is typically set within the range of 0.05-0.1 (Shvetsova et al. 2022). Therefore, we evaluate three settings: 0.05, 0.07, and 0.1. The results in Tab. 12 show that temperature of 0.1 yields the best performance. This is because smaller temperatures make the distribution sharper and strengthen the separation between positive and negative samples, but lead to training instability. Hence, 0.1 serves as more optimal choice.
Scale Balancing for obejcts. Considering that large objects occupy greater weight in the loss function, we perform scale balancing accordingly. Specifically, we take the average based on the number objects, and then multiply by a weighting factor, which is set the average number of objects per frame across all training frames. The modified formula of Eq. g is as follows:
Lm
/ B
Ecccelte)))))) log
exp(sn.c.n)
(12)
Eien exp(s8...))
where |B|] denotes the number of objects in a feature frame and is the average number of objects per frame across all training frames. We conduct experiments the RCooper dataset, and the results are shown in the Tab. 13 It can be seen that the performance improves.
19

---

Published as a conference paper at ICLR 2026
Table 13: Results of scale balancing
Method
A1
A2
A3
A4
GT-Space
0.477
0,475
0.349
0.351
GT Space (average)
0.481
0.482
0.350
0.354
Figure 10: Visualization of collaborative perception results. We color predicted and GT boxes.
Additional Visualization. Fig. 10 shows the visualization of collaborative perception results on OPV2V.
20


Reviews:
[{"review_id": "07ecvRO7th", "reviewer": "Reviewer_dxid", "summary": "This paper introduces GT-Space, which is a scalable collaborative perception framework for heterogeneous autonomous driving agents. GT-Space uses a common feature space constructed from ground-truth labels to serve as a unified reference for feature alignment and a lightweight projector for each agent to map its features into that space. It achieves SOTA performance on OPV2V, V2XSet, and RCooper dataset.", "questions": "I would like to raise my rating if the authors can address my concerns shown in weakness section.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The idea of aligning all heterogeneous features in the GT space to construct a common, unified feature space is well-motivated and novel. \n2. The designed system is highly scalable and flexible. Because individual agents' encoders and detection heads are kept frozen, a new or unseen agent can be integrated simply by training a single, lightweight projector module to map its features to the GT-Space.\n3. The framework's performance is not bottlenecked by the capability of the ego agent. By using the GT-Space as a strong, objective reference, the fusion network can effectively leverage complementary strengths and compensate for weaker agents (like camera-only models).\n4. The paper demonstrates SOTA performance on multiple benchmarks.", "weaknesses": "1. I have a major concern in the scalability of fusion training. The fusion network is trained using a \"combinatorial contrastive loss\" across all pairs of modalities. The paper gives an example with 3 models, resulting in 3 pairs. This implies that for $M$ distinct modality types, the training complexity is $O(M^2)$. This is not scalable\n2. The visualization in Figure 5 is not very insightful. It simply shows that the fused feature map has stronger activations than the original. More compelling visualizations would have been:A visualization of the GT-Space $F_{GT}$ itself.A side-by-side comparison of a projected camera feature $\\Phi_{cam}(F_{cam})$ and a projected LiDAR feature $\\Phi_{lidar}(F_{lidar})$ to visually demonstrate the alignment quality in the common space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "28M2uYa7lj", "reviewer": "Reviewer_BvnQ", "summary": "GT-Space is a heterogeneous collaborative perception framework that avoids pairwise feature alignment between agents with different sensors/encoders. It builds a ground-truth–derived common BEV feature space from 3D box labels and learns a single lightweight projector per agent to map local features into this space. On OPV2V, V2XSet, and a real-world RCooper scene, GT-Space consistently outperforms end-to-end, interpreter, and retraining baselines, especially boosting weaker/camera agents. Ablations show the projector and GT-space supervision are key contributors to gains.", "questions": "What is the advantage of this work over interpreter-based methods? The author claimed that they are not scalable but did not explain why.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. GT-space requires only one projector per agent, without pairwise adapters or encoder retraining which is scalable.\n\n2. GT-space uses contrastive learning to handle the bottleneck effect in collaborative perception.\n\n3. The experiments show that GT-space has SOTA performance among widely used datasets.", "weaknesses": "This work requires accurate, dense 3D labels to build the common space, which is often impractical in real-world deployments.\n\nThe reviewer suggests adding robustness experiments, such as how the pipeline handles communication latency and localization errors.\n\nThis work evaluates only 3D detection; performance on other tasks, such as lane segmentation or tracking, is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sVUtFPFtbi", "reviewer": "Reviewer_AVxH", "summary": "This paper proposes GT-Space, a novel framework that constructs a common feature space from ground truth labels, offering a unified reference for feature alignment among heterogeneous agents. This approach eliminates the need for pairwise interactions between agents, enhancing scalability. Extensive experiments demonstrate that GT-Space achieves state-of-the-art performance in detection accuracy.", "questions": "See the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.Common Feature Space for Heterogeneous Agents: GT-Space introduces a common feature space for aligning heterogeneous agents, which promotes the practical deployment of heterogeneous collaborative perception systems.\n\n2.Contrastive Learning for Consistent Representation: By employing contrastive learning to supervise the fusion network, GT-Space encourages different agents to learn consistent feature representations for the same instances, improving robustness.\n\n3.Superior Performance Across Modality Combinations: Experimental results clearly indicate that GT-Space achieves advanced perception performance across various heterogeneous modality pairings, showcasing its effectiveness.", "weaknesses": "1.Generalization to Unseen Agent Types: The paper trains and tests with the same set of agent types (e.g., agents A1-A4 are used for both training and testing under various combinations). However, in real-world scenarios, new, unseen agent types may need to be integrated into the system. Without ground truth labels for these new agents, how can the projection layer be trained to adapt to the fusion model for collaborative perception? What would be the estimated amount of training data and training duration required for such adaptation?\n\n2.Handling Misaligned Feature Spaces in Real-World Scenarios: GT-Space trains the model using observations from a single agent that assumes spatially aligned data. In practical settings, the feature spaces of different agents are likely to be misaligned. While coordinate transformations can achieve spatial alignment, the resulting features might differ significantly from those used during training (e.g., containing more background features). How does the proposed method address this discrepancy and maintain performance robustness in such real-world, misaligned scenarios?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zC2JpfWhYY", "reviewer": "Reviewer_w8N3", "summary": "The paper proposes a heterogeneous collaborative perception framework named GT-Space. Instead of retraining or creating an \"interpreter\" for each heterogeneous agent, this framework projects the BEV (Bird's-Eye View) features from each vehicle's agent into a common feature space constructed from the ground truth, followed by fusion and detection.", "questions": "- The object-level contrastive learning \"aggregates grids within a bounding box and then computes similarity/cross-entropy.\" Larger objects, which cover more grids, might have a greater weight in the loss function. Is scale balancing or re-weighting necessary?\n- In Figure 1, the caption explicitly states, \"Interpreter-based alignment: local encoder and detection head (denoted as D) are frozen.\" However, diagram (b) shows that the local encoder is trainable. Furthermore, why do agents E#2 and E#3 point to the same detection head D?\n- In Figure 2, why is the similarity loss calculated between the GT feature and the BEV feature? The BEV feature is generated by a frozen encoder, so is calculating a loss on it meaningful? Shouldn't the loss be calculated on the feature after it has passed through the projector?\n- Equations (5) and (6) feel somewhat vague and lack rigor. I can understand the author's intention, but the formulation seems to have a circular definition. $\\Phi_{a}$ is presented as the function to be solved in the first equation, yet it is used as a known function within the loss in the second equation.\n- In Equation (8), is there a missing negative sign? In $L_{m,m'} = \\sum_{B} \\sum_{c} \\sum_{B'} \\log\\left( \\frac{\\exp(s_{B,c,B'})}{\\sum_{l} \\exp(s_{B,c,B'})} \\right)$, is the summation in the denominator performed over all $l$? However, the term inside the exponential function being summed is $\\exp(s_{B,c,B'})$, which uses the index $B'$.  Perhaps the correct formula is something like $L_{m,m'} = -\\sum_{B} \\sum_{c \\in \\text{cells}(B)} \\log \\frac{\\exp\\left(s_{B,c,B}\\right)}{\\sum_{l \\in \\mathcal{B}} \\exp\\left(s_{B,c,l}\\right)}$.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper has a clear motivation, aiming to solve the problem of needing to retrain for each new agent, thereby increasing the system's scalability.\n- The proposed method is designed to be plug-and-play, is orthogonal to other existing methods, and can be used in conjunction with them.\n- The experimental results demonstrate that the framework can effectively improve perception performance in a heterogeneous environment.", "weaknesses": "- The framework heavily relies on the \"ground-truth feature space\". The construction of GT-Space depends on ground truth. During inference, since ground truth is unavailable, the model can only \"imagine\" a space that aligns with it. This requires the training and deployment distributions to be sufficiently close; otherwise, the alignment will drift. The paper does not quantify the robustness under such distribution shifts.\n- The authors freeze the local encoders and detection heads, promoting a \"plug-and-play\" capability. However, this also limits the potential performance ceiling that could be achieved through end-to-end joint fine-tuning.\n- The authors claim that when adding a new, unseen agent, only the projector for that agent needs to be trained, thus enabling scalable collaboration with minimal deployment cost by deploying only a lightweight adapter. However, the paper does not explicitly report the comparison of training computation/time for \"adding a new agent\"; it only reports inference latency.\n- The implementation process is not clearly described, containing some errors or confusing points. The lack of open-sourced code makes the paper difficult to reproduce.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)