STAMP: Scalable Task- And Model-agnostic Collaborative Perception
Authors: Xiangbo Gao
Runsheng Xu
Jiachen Li
Ziran Wang
Zhiwen Fan
Zhengzhong Tu
Venue: ICLR 2025 Poster (2025)
Keywords: Autonomous Driving
Collaborative Perception
Domain Adaptation
Primary area: applications to robotics, autonomy, planning
TL;DR: We propose STAMP, a scalable, task- and model-agnostic collaborative perception framework that enables efficient feature sharing and fusion among heterogeneous agents in autonomous driving

Abstract: Perception is a crucial component of autonomous driving systems. However, single-agent setups often face limitations due to sensor constraints, especially under challenging conditions like severe occlusion, adverse weather, and long-range object detection. Multi-agent collaborative perception (CP) offers a promising solution that enables communication and information sharing between connected vehicles. Yet, the heterogeneity among agents—in terms of sensors, models, and tasks—significantly hinders effective and efficient cross-agent collaboration. To address these challenges, we propose STAMP, a scalable task- and model-agnostic collaborative perception framework tailored for heterogeneous agents. STAMP utilizes lightweight adapter-reverter pairs to transform Bird's Eye View (BEV) features between agent-specific domains and a shared protocol domain, facilitating efficient feature sharing and fusion while minimizing computational overhead. Moreover, our approach enhances scalability, preserves model security, and accommodates a diverse range of agents. Extensive experiments on both simulated (OPV2V) and real-world (V2V4Real) datasets demonstrate that STAMP achieves comparable or superior accuracy to state-of-the-art models with significantly reduced computational costs. As the first-of-its-kind task- and model-agnostic collaborative perception framework, STAMP aims to advance research in scalable and secure mobility systems, bringing us closer to Level 5 autonomy. Our project page is at https://xiangbogaobarry.github.io/STAMP and the code is available at https://github.com/taco-group/STAMP.

Published as a conference paper at ICLR 2025
STAMP: SCALABLE TASK- AND MODEL-AGNOSTIC COLLABORATIVE PERCEPTION
Xiangbo Gao', Runsheng Xu², Jiachen Li,, Ziran Wang', Zhiwen Fan', Zhengzhong Tu Texas A&M University, 'UCLA, *UC Riverside, "Purdue University, SUT Austin {xiangbog, tzz)@tam edd
ABSTRACT
Perception is crucial component of autonomous driving systems. However, single-agent setups often face limitations due to sensor constraints, especially un- der challenging conditions like severe occlusion, adverse weather, and long-range object detection. Multi-agent collaborative perception (CP) offers a promising solution that enables communication and information sharing between connected vehicles. Yet, the heterogeneity among agents- in terms of sensors, models, and misks-significantly hinders effective and efficient cross-ar aggett collaboration. To address these challenges, we propose STAMP, a scalable task- and agdestcgnostic collaborative perception framework tailored for heterogeneous agents. STAMP utilizes lightweight adapter-reverter pairs to transform Bird's Eye View (BEV) features between agent-specific domains and a shared protocol domain, facilitat- ing efficient feature sharing and fusion while minimizing computational overhead. Moreover, our approach enhances scalability, preserves model security, and ac- commodates a diverse range of agents. Extensive experiments on both simulated (OPV2V) and real-world (V2V4Real) datasets demonstrate that STAMP achieves comparable superior accuracy to state-of-the-arr models with significantly re- duced computational costs. As the first-of-its-kind task- and model-agnostic col- laborative perception framework, STAMP aims to advance research in scalable and secure mobility systems, bringing us closer to Level 5 autonomy. Our project page is https://iangbbogobbrrs wwttoobarrrrrrrbbbiinoee githuu.io/STAME and the code is available https://gituub.com/ttooo http://ttattooooSSS/SSSS group/STAN
INTRODUCTION
Multi-agent collaborative perception (CP) (Bai et al., 2022b; Han et al., 2023; Liu et al., 2023a) has emerged promising solution for autonomous systems by leveraging communication among mul- tiple connected and automated agents. It enables agents-such as vehicles, infrastructure, or even pedestrians to share sensory and perceptual information, providing a more comprehensive view of the surrounding environment to enhance overall perception capabilities. Despite its potential, CP faces significant challenges, particularly when dealing with heterogeneous agents that defer input modalities, model parameters, architecture,, or learning objectives. For instance, Xu et al. (2023b) observed that features from heterogeneous agents vary spatial resolution, channel number, and feature patterns. This domain gap hinders effective and efficient CP, particularly when employing fusion-based approaches.
To facilitate collaborative perception among heterogeneous agents- referred to as heteroge- neous collaborative perception-one might consider using early or late fusion. However, early fu- sion requires high communication bandwidth, making it impractical for real-time applications. Late fusion often results in suboptimal accuracy, and it is not viable across models with different down- stream tasks. Alternative methods attempt to achieve heterogeneous intermediate fusion by either incorporating adapters (Xu et al., 2023b) or sharing parts of the models (Lu et al., 2024). While these approaches can bridge the domain gap, they are limited in scalability or security, rendering them inefficient or unsafe for practical deployment. Additionally, recent studies have highlighted increased security vulnerabilities in CP systems compared to single-agent frameworks (Hu et al.,
"Corresponding author.

---

Published as a conference paper at ICLR 2025
2024; Tu et al., 2021; Li et al., 2023b). Notably, Li et al. (2023b) found that black-box attacks nearly ineffective without the knowledge of other agents' models, emphasizing the importance of task- and model-agnostic approaches in enhancing system-level security against adversarial threats.
To address these challenges, we propose STAMP, Scalable Task- And Model-agnostic collabora- tive Perception framework. Our approach employs lightweight adaptcr-reverter pairs to transform the Bird's Eye View (BEV) features of each heterogeneous agent into a unified protocol BEV feature domain. These protocol BEV features are then broadcasted to other agents and subsequently mapped back to their corresponding local domains, enabling collaboration within each agent's source do- main. We refer to this process collaborative feature alignment (CFA). Our proposed pipeline offers several key advantages. Firstly, it enables existing heterogeneous agents to collaborate with minimal additional disk memory (~1MB) and computational overhead, making it scalable for large number of heterogeneous agents. Secondly, the alignment process is designed to be task- and model-agnostic, allowing our framework to integrate with various models and tasks without retrain- ing the model or the need to share models among agents, enhancing both flexbbiliy and security.
We conducted comprehensive experiments to evaluate the performance of our collaborative per- ception framework. Using the simulated OPV2V dataset (Xu et al., 2022b) and the real-world V2V4Real dataset (Xu et al., 2023d), we demonstrated that our STAMP pipeline achieves compa- rable or superior accuracy with significantly lower training resource growth rate as the number of heterogeneous agents increases. Our method requires, on average, only 2.36 GPU hours (7.2x saving) of training time per additional agent, compared to 17.07 GPU hours per additional agent for existing heterogencous collaborative pipelines. We also demonstrated our pipeline's 's unique ability to perform task- and model-agnoste collaboration, a capability not supported in existing methods. This achievement establishes new benchmark for heterogeneous CP in autonomous driving, show- casing clear performance improvements in scenarios where other methods are unable to operate.
RELATED WORKS
2.1 MULTI-AGENT COLLABORATIVE PERCEPTION
Multi-agent CP has emerged as a promising solution to overcome the inherent limitations of single- agent perception systems, particularly in addressing occlusions and extending perception range (Hu et al., 2022a).
Information-sharing schemes. There are three main information-sharing schemes in multi-agent CP systems: early fusion, late fusion, and intermediate fusion. 1 Early fusion (Gao et al., 2018; Chen et al., 2019; Arnold et al., 2020) involves the direct sharing of raw sensor data, such as LiDAR point clouds or camera images, between agents. This method maximizes information transfer but requires high bandwidth for transmitting. 2 Late fusion (Melotti et al., 2020; Fu et al., 2020; Zeng al., 2020; Shi et al., 2022; Glaser & Kira, 2023; Xu et al., 2023a; Su al., 2023; 2024), involves sharing only final prediction results, such as object detection bounding boxes or occupancy pre- dictions. This approach significantly reduces communication bandwidth overhead, making it more feasible for implementation in real-world systems. However, late fusion often results in subopti- mal accuracy due to the loss of intermediate information that could be valuable for collaborative decision-making. @ Intermediate fusion (Wang et al., 2020; Liu et al., 2020b;a; Guo et al., 2021; Li al., 2021; Hu et al., 2022b; Bai et al., 2022a; Cui et al., 2022; Xu et al., 2022a; 2023e; Qiao & Zulkernine, 2023; Li et al., 2023a; Wang et al., 2023; Yu et al., 2023; Yang et al., 2024) has emerged as a promising middle ground, involving the sharing of mid-level information, typically in the form of Bird's Eye View (BEV) features. This approach strikes balance between communica- tion bandwidth efficiency and information richness. Intermediate fusion allows for more flexibility in collaborative processing while maintaining a reasonable data transfer load. However, intermediate fusion faces significant challenges in addressing domain gap issues for heterogeneous agents.
Collaborative perception datasets. Several significant collaborative perception datasets have emerged recently (Yazgan et al., 2024). The simulated OPV2V (Xu et al., 2022b) and V2X-Sim dataset (Li et al., 2022) each contain approximately 10k multi-agent scenes, featuring RGB images and LiDAR point clouds with 3D object detection, tracking, and segmentation annotations. Two real-world DAIR- V2X Yu et al. (2022) and 42V4Real (Xu et al., 2023d) datasets provide 39k and 20k: dual-agent samples, respectively, with object detection annotations only. For our framework
2

---

Published as a conference paper at ICLR 2025
evaluation, we selected two complementary datasets: the OPV2V dataset for its diverse downstream tasks, and the 42V4Real dataset to validate performance real-workd scenarios.
2.2 HETEROGENEOUS COLLABORATIVE PERCEPTION
In a CP system, the heterogeneity of agents can manifest three different types: heterogencous modalities, heterogeneous model architectures or parameters, and heterogeneous downstream tasks. O Heterogeneous modalities. Each model is expected to take input data of different modalities (e.g., RGB images, LiDAR point clouds (Liu et al., 2023b), Thermal images (Gao et al., 2024a)), requiring different encoders to process the data. Xiang al. (2023) propose a hetero-modal vision transformer to fuse heterogeneous BEV features, but this requires end-to-end model training, which impractical for existing heterogeneous agents. Xu et al. (2023b) introduce multi- perception domain adaptation (MPDA), which aligns feature maps between heterogeneous agent pairs. While effective for collaboration, this method's polynomial complexity limits its scalability as the num- ber of heterogeneous models increases. Lu al. (2024) introduce backward alignment training strategy, creating heterogeneous models by fixing a base network's decoder and training only the encoders. While this enables collaboration between existing heterogeneous agents, incurs high computational costs, especially for models with large encoders. @ Heterogeneous model archi- tectures or parameters. Model architectures or parameters may differ across agents, resulting in feature map in different domains, rendering existing heterogeneous intermediate fusion methods (Xiang al., 2023; Lu et al., 2024) inapplicable. However, late fusion methods (Xu et al., 2023b) remain viable as the model output for all models is in the same domain. & Heterogeneous down- stream tasks. The learning objectives are different across agents, which results in model outputs in different domains. Li al. (2023c) propose task-agnostic CP by training models with multi-robot scene completion objectives. Despite the effectiveness of task-agnostic collaboration, their method does not support heterogeneous modality inputs and model architectures.
METHODOLOGY
3.1 PRELIMINARIES: INTERMEDIATE COLLABORATIVE PERCEPTION
A CP system typically comprises multiple (N) agents, each equipped with its own CP model. This work mainly focuses on intermediate fusion, so we consider all CP models to be trained using an intermediate fusion strategy. The architecture of these models generally consists of an encoder E, a compressor &;. decompressor th; collaborative fusion layer U,, and decoder D,, where i {1,2,.... N} represents the agent index.
The CP process unfolds as follows: Upon receiving input data L, the encoder E, of agent trans- forms this data into a Bird's Eye View (BEV) feature representation F,. To save transmission band- width, each agent uses the compressor 4 to compress F, to F; before broadcasting them to other agents within predefined collaborative distance d. Here, ō denotes the maximum range for inter- agent collaboration. Each agent collects the BEV features from other agents and uses their own decompressor th, to decompress F5 to Fk. where agent é and agent & are within the distance 8 for collaboration. Then, the collaborative fusion layer U. collects and integrates the BEV features from all cooperating agents, producing a consolidated BEV feature F'. Finally, the decoder D; processes this fused feature F' to generate the final model output Oz. This process can be formally described as follows for each agent i 21, N}:
Encoding:
F. = E,(L,)
Compression:
F. = (F.(F))
(2)
Decompression:
F, (,)(F)),
Vj VjEN(i.J) 5 F Ui(LE, (N((i,)) o0)
Fusion:
Decoding:
O, D,(F()
where N(i,,) k) refers to the Euclidean distance between the agent i and agent k.
3.2 FRAMEWORK OVERVIEW
Our proposed framework, STAMP, enables collaboration among existing heterogeneous agents with- sharing model details downstream task information. We replace the compression (Equation 2)
3

---

Published as a conference paper at ICLR 2025
Before CFA (Non-collaborative)
(III) After CFA (heterogeneously collaborative)
Fj
F2
F2
domain gap
adapted local features
hetero agents
hetero agents
(II)
Collaborative Feature Alignment (CFA)
Step Training Protocol Network
Step 2: Training Local Adapters and Reverters
F
C00 OOO atapter $
OOO
local feature
adapted local feature
reverter 14
local fenture
Ladape
local feature
Off-line Training
poocol
000 reverter dbs
reverted protocol feature
feature
Vv 000
protocol Teatures
800
Figure 1: Initially, agents are non-collaborative (I), resulting degraded performance. Collaborative Feature Alignment (CFA) enables collaboration among heterogeneous agents through two-step process (II): training protocol network and training local adapters and reverters. The protocol network facilitates communication between Agent 1, Agent 2, and Agent 3, each with heterogeneous models and features. Gradient-colored feature maps represent features adapted or reverted between domains. After CFA implementation, agents become collaborative (III) with improved performance.
and decompression (Equation 3) with adaptation and reversion steps. Specifically, for each agent i, we introduce local adapter db; and a local reverter th. The adaptation process defined as follows:
Adaptation: Fip (FF),, Vi (1,
(6)
Here, the adapter db; maps the local BEV feature F; unified BEV feature representation, which we term the protocol feature, denoted as Fp. The resulting adapted feature denoted as Fip.
Following adaptation, the features from all heterogeneous agents i are broadcast to other agents within the collaborative distance 5. Each receiving agent j (j # i) then uses its local reverter 1h to map the received features back to its own local feature representation. The resulting reverted features are denoted as Pij. This reversion process formulated as:
(FeP), Reversion: Fij P..
if ifi=j
Vi.j E (1,
Note that F, is already in the local feature presentation, so stay intact. This adaptation and reversion process seamlessly integrates into the standard heterogeneous CP pipeline, forming the core of our STAMP framework.
The STAMP framework supports agents with different modalities, model architectures, and down- stream tasks, while maintaining collaboration process that is entirely agnostic to those characteris- tics of other agents. To the best of our knowledge. STAMP is the first framework that simultancously addresses all three aspects of agent heterogeneity. Furthermore, the adapters and reverters can be implemented in a highly lightweight manner, ensuring high scalability across a large number of het- erogeneous agents. A comprehensive comparative summary STAMP and other heterogeneous CP frameworks is presented in Table 1.
3.3
COLLABORATIVE FEATURE ALIGNMENT
We propose the Collaborative Feature Alignment (CFA) module to train a unified BEV feature representation and local adapter-reverter pair. As illustrated in Figure (I), before CFA, het-
4

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of heterogeneity support and scalability across existing heterogeneous collab-
oration frameworks. "P" indicates that while the current codebase does not support the specified heterogeneity, we believe the proposed method could accommodate it with minor modifications.
Frameworks
Modality
Model Architecture
Downstream Task
Scalability
Calibrator (Xu 2023a)
high
MPDA (Xu et al., 20236)
low
HEAL et al., 2014)
mechum
Scene Completion (Li al. 2023c)
high
STAMP (ours)
high
erogencous agents perform multiple tasks individually without the ability collaborate, resulting in suboptimal performance. After CFA (III), these agents can effectively collaborate, leading to significant performance boost.
Training protocol network. The first step is learn unified BEV embedding space by training a protocol network. This process follows the standard training process of a collaborative perception model, as described in Equations (I) (5). We denote the protocol encoder, fusion model, and decoder as Ep, Up, and Dp. respectively. The compressor and decompressor are set as identity functions. The input data, BEV feature, fused BEV feature, and final output of the protocol model are represented by Ip, Fp, Fjp, and Op, respectively.
@ Training local adapters and reverters. We introduce a notation X' = x x to denote set world states, where the total number of world states. modality transformation function T X + / transforms world state to sensor data of a given modality. For instance, in an autonomous driving scenario, x could represent the world state surrounding the ego vehicle, and T could be the matrix of six surround-vied RGB cameras, resulting in I as the six RGB images captured by these cameras.
Given a local model i, protocol model P. and a set of world states X' = fx' xx.....xx" xx) within the collaborative distance, we define 1 Tx(x*) as the input for local model and 12 (x(x") as the input for protocol model P. Here, Ti and Tp denote the sensor modalities of the local model and protocol model, respectively. By passing these inputs into their corresponding encoders, we get set of local BEV features FI:K and protocol BEV features F}:K A domain gap exists between F.lKK and F due to the heterogeneity of sensor modalities T; and Tp, as well as the encoders E; and Ep. To bridge this gap, we introduce local adapter 4b, that maps the local BEV feature F. to the protocol feature representation. The objective function for Φ
Φ arg min L., (F1. F.: where F (:(P..
where Lap represent the feature alignment loss for the adapter i.. Similarly, we introduce reverter vb; that maps the protocol BEV feature FP to the local feature representation, i.e., Fė, th, (F)). since Ft. is also in the protocol representation, by including F 1b: (FFt))) we provide additional supervision for tps The objective function for the reverter is formulated as:
tb. arg min (Li,, (FP: F.:K) + Lyp. (F1:K p!.k)
vb
where F (F$)) F (F.t)
Here, Tap, represents the feature alignment loss for the reverter 1h,- To achieve our objective func- tions, we conduct alignment in both the feature space and the decision space.
Feature space alignment. For given local model i, first align the feature pairs (F F)*K), (F); F.IKk,, ), and (F ,F!:K) for all & using the L2-norm. This direct alignment of feature spaces is formulated
1/
EIFip, 1 K
1 E (IFA, Fl 1F2.
(10)
Decision space alignment. When T, and Tp represent significantly different sensor modalities (e.g., RGB camera vs. LiDAR), the disparity between their intermediate feature representations can
5

---

Published as a conference paper at ICLR 2025
be substantial. In such cases, achieving exact equivalence between FiP and Fý for all k is neither feasible nor necessary. Nevertheless, since both F and Fÿ are derived from the same world state x", their corresponding downstream task outputs should align with the same ground truth labels. To enforce this alignment in the decision space, we introduce additional loss terms:
L. - Lr(Dp o UP(P.P)) ), GTp)
(11)
L.j. LD U.(FP)) GT,) Li(D; o (F(K GT;)
where Cp and L, represent the task-specific loss functtion for training the protocol model and local model i, respectively. GTp and GT, denote the corresponding ground truth labels for the protocol and local model i. Finally, balance the importance of adaptation and reversion, as well as feature
and decision space alignment, we introduce scaling factors Ng. Xy. Xg. and X The total loss function for local model i is:
= + +No + +gg +1444
(12)
3.4 ADAPTER AND REVERTER ARCHITECTURE
To bridge the domain gap between heterogeneous agents, we propose a flexible architecture for both the adapter & and reverter yb. This architecture addresses three main sources of domain gap caused by agent heterogeneity, as identified by Xu et al. (2023b): spatial resolution, feature patterns, and channel dimensions. Our design employs simple linear interpolation for spatial resolution alignment, three ConvNeXt layers (Liu et al., 2022) with hidden channel dimension Chidden for feature pattern alignment, and two additional convolutional layers for channel dimension alignment (input: Cn - Chidden. output: Chidden Com.). For the model architecture details, please refer to Appendix A.
Note that this high-level architecture is flexible and open various implementations. In Section 4.4, we evaluate alternative approaches for feature pattern alignment, demonstrating our framework's flexibility across different specific implementations.
4 EXPERIMENTS
Our STAMP framework enables collaboration among agents with heterogeneous modalities, models architectures, and downstream tasks without sharing model or task information. We first compare our framework with existing heterogeneous CP frameworks in Section 4.2. Given that no previous work supports simultaneous task- and model-agnostic heterogeneous collaboration, we concentrate our evaluation on the 3D object detection task. This focus ensures a fair comparison across two key dimensions: object detection average precision, trainable parameters and GPU hours required for training. Next, in Section 4.3, we demonstrate our framework's 's unique capability in a task- and model-agnostic setting, evaluating its performance using four existing collaborative models with heterogeneous architectures and downstream tasks. Then, we present ablation studies on channel sizes, model architectures, and loss functions in Section 4.4 to further analyze our framework's design choices. Finally, we present some feature and output visualization in Section 4.5.
4.1 EXPERIMENTAL SETUP
Our experiments utilize two CP datasets: the simulated OPV2V dataset (Xu et al., 2022b) and the real-world V2V4Real dataset (Xu et al., 2023d). We employ both datasets in Section 4.2 and Section 4.4 for method comparison and ablation studies. The task- and model-agnostic evaluation in Section 4.3 uses only OPV2V due to its multi-task annotations. This combination leverages the scale and diversity of simulated data with the realism of worl-world data, ensuring comprehensive model evaluation.
Implementation details. We use different setups for 3D object detection (Section 4.2) and task- agnostic settings (Section 4.3), detailed within each section. Unless using end-to-end training, local and protocol models are trained for 30 epochs using Adam optimizer (Kingma & Ba, 2014). For end-to-end training, we use Iters = 30N epochs, where N is the number of heterogencous models, to ensure all models receive the same amount of supervision. Local adapters and reverters up trained for epochs. We set loss scaling factors X Xf Aaapp Alvent = 0.5 empirically. For additional details, please refer to Section 4.2, Section 4.3, and Appendix A.

---

We prepared 12 heterogeneous local models (six with LiDAR modality and six with RGB camera modality) and one protocol model with LiDAR modality (details in Appendix A. Each agent has visible range of 51.2m 51.2m square units. Considering that samples of the OPV2V dataset contain no more than four agents, we only select the first four models for evaluation on the OPV2V Published as a conference paper at ICLR 2025
Table 2: Performance comparison using AP@ 30 and AP@50 metrics the OPV2V dataset. Agent positions are perturbed with Gaussian noise of standard deviations 0.0, 0.2, and 0.4. Al, A2, A3, and A4 refer to agent I, agent 2, agent 3, and agent 4, respectively.
AP@30 f
AP@50 1
Agent Index
A1
+A2
+A3
+A4
A1
+A2
+A3
+A4
Late Fusion
0.902
0.931
0.935
0.935
0.894
0.908
0.913
0.914
Calibrator
0.901
0.935
0.939
0.938
0.896
0.914
0.916
0.920
0.0
E2B Training
0.899
0.973
0.978
0.986
0.885
0.967
0.977
0.980
HEAL
0.897
0.975
0.986
0.985
0.889
0.972
0.978
0.983
STAMP (ours)
0.902
0.981
0.987
0.989
0.894
0.977
0.983
0.985
Late Fusion
0,900
0.910
0.902
0.905
0.882
0.783
0.797
0.792
Calibrator
0.897
0.908
0.898
0.902
0.885
0.778
0.800
0.791
0.2
E2F: Training
0.900
0.967
0.961
0.961
0.879
0.936
0.941
0.934
HEAL 0.899
0.971
0.965
0.962
0.881
0,938
0,940
0.940
STAMP (ours)
0.900
0.975
0.968
0.968
0.882
0.937
0.948
0.946
Late Fusion
0.888
0.882
0.864
0.870
0.874
0,639
0.614
0.617
Calibrator
0.874
0.889
0.887
0.871
0.867
0,644
0.622
0.628
0.4
E2E Training
0.885
0.952
0.956
0,952
0.863
0.883
0.892
0,899
HEAL
0.880
0.959
0.961
0.962
0.852
0,913
0.915
0,912
STAMP (ours)
0,888
0.961
0.963
0.966
0.874
0.915
0.915
0.909
4.2 HETEROGENEOUS COLLABORATIVE PERCEPTION FOR 3D OBJECT DETECTION
Performance comparison. We compare our method with existing heterogeneous CP approaches on the 3D object detection task. We select two late fusion methods (vanilla late fusion and calibrator (Xu et al., 2023a)) and two intermediate fusion methods (end-to-end training and HEAL (Lu et al., 2024)) for comparison. Late fusion methods offer simple way to mitigate domain gaps in collab- orative 3D object detection. Xu et al. (2023a) propose using a calibrator to address residual domain gaps late fusion, which arise from differences in training data and procedures among heteroge- neous models. For intermediate fusion, end-to-end training of all heterogeneous models together allows collaboration during the training stage to bridge domain gaps. Lu et al. (2024) introduce backward alignment technique, first training base network, then fixing its decoder while train- ing only the encoders to create heterogeneous models. An architectural comparison between these frameworks and our proposed STAMP framework is illustrated and visualized in Appendix A2.
dataset. Similarly, we select the first two models for the V2V4Real dataset since has two agents for each sample. All 12 models are used for efficiency comparison.
For the OPV2V dataset, simulate real-world noise by adding Gaussian noise with standard de- viations σ = {0.0, 0.2, 0.4) to the agents' locations. As shown in Table 2, late fusion methods underperform the number of agents increases, with performance degrading further at higher noise levels (a = 0.4). This is particularly evident when camera agents (agents 3 and 4) are involved, highlighting the late fusion methods' vulnerability to bottleneck agents' incorrect predictions. Our framework demonstrates superior or comparable performance to other heterogeneous fusion meth- ods across all noise levels.
Table 3 compares the average precision on the real-world V2V4Real dataset. Our STAMP pipeline demonstrates superior performance, achieving the highest AP@ for both agents (0.523 and 0.633) and competitive 50@ scores. STAMP outperforms Late Fusion methods and matches or exceeds the perfor- mance of existing heterogeneous intermedi- ate fusion approaches like HEAL. These re-
Table Performance comparison using AP@ @0 and AP@ metrics on the V2V4Real dataset.
AP@30
AP@50 f
Agent Index
A1
+A2
A1
+A2
Late Fusion
0.523
0.511
0.483
0.471
Calibrator
0.520
0.524
0.484
0.488
E2E Training
0.513
0.612
0.473
0.598
HEAL 0.515
0.628
0.480
0.595
STAMP (ours)
0.523
0.633
0.483
0.594
sults indicate that the CFA module in STAMP
is effective not only in simulated environments but also in real-world scenarios.
7

---

Published as a conference paper at ICLR 2025
Efficiency comparison. We conducted an efficiency comparison between our approach and existing heterogeneous CP pipelines, fo- cusing on the total number of parameters and
Thiring Parameters (M)
Troining GPU Time (turs
CPU Fram (TTTA MB000
HSAL
Skipe 31.69
2015
IEDAS
stoper
Pummetee ca
training GPU hours. Training GPU hours refers to the time required to complete model training on the OPV2V dataset using an RTA A6000 GPU. To analyze training costs at scale, we report the number of training pa- rameters and the estimated training time. Fig- ure illustrates the changes in number of training parameters and estimated training GPU hours as the number of heterogeneous agents increases from to 12. End-to-end training and HEAL exhibit steep increase in both parameters and GPU hours the number of agents grows. In contrast, although our pipeline shows higher parameters and GPU hours at the one or two number of agents (due to the training of the protocol model), it demonstrates much slower growth rate because our proposed adapter & and reverter yp is very light-weighted and only takes 5 epochs to finish training. This highlights the scalability of our pipeline.
2lope22.00
<app
Skpe
Numbx: ol Hotero Aganis
Number of Hetero Agenn
Figure 2: Training efficiency comparison of our framework and existing heterogeneous CP frame- works across a number of heterogeneous agents.
4.3 MODEL- AND TASK-AGNOSTIC FUSION
In this section, we evaluate our proposed framework's performance in a task-agnostic setting us- ing the OPV2V dataset. To simulate agent heterogeneity, we assign four agents with diverse input sensors, learning objectives, and evaluation metrics, equipping them with various backbones and fusion models. Agent I was equipped with SECOND encoder (Yan et al., 2018) and window attention fusion module (Xu et al., 2022a; 2024). For Agent 2, we implemented an EfficientNet- bo encoder (Tan, 2019). while Agents 3 and 4 were equipped with PointPillar encoders (Lang et al., 2019). Agents 2, 3, and all utilized the Pyramid Fusion module Lu et al. (2024). Ta- ble 4 summarizes these models' key characteristics. We compare STAMP framework against two baseline scenarios: non-collaborative (single-agent perception without information sharing) and collaborative without feature alignment (performing intermediate fusion despite domain gaps). Ta- ble 4 presents the evaluation results on the OPV2V dataset, with added Gaussian noise (standard deviations {0.0, 0.2, 0.4)) to the agents' locations.
Table 4: Heterogeneous CP results in a model- and task-agnostic setting. Tasks include 3D object detection ('Object Det'), static object BEV segmentation ('Static Seg'), and dynamic object BEV segmentation ('Dynamic Seg'). 3D object detection is evaluated using Average Precision at 50%
loU threshold (AP@50), while segmentation tasks Mean Intersection over Union (MIoU).
Agent Index
Agent
Agent 2
Agent 3
Agent 4
Metric
AP@50
AP@50 Object Det
MIoU Static Seg
MIoU Dynamic Seg Lidar
Downstream Task
Object Lidar SECOND 64 x 64 256 Window Attention
Agent Info
Sensor Modality
Camera
Lidar
Backbone
ElticientNNet-b 128 x 128
PointPillar 128 x 128
PointPillar
Feature Resolution
128 x 128
Channel Size
64 Pyramid Fusion
64 Pyramid Fusion
64 Pyramid Fusion
Fusion Method
Evaluation
Non-Collab Collab w/o. CFA
0.941
0.399
0.548
0.675 0.070
0.909_0.032
1.399->0.00
0.1 L0.444
0.0)
STAMP (ours)
0.936 10.005
0.760 10.362
0.624 10.076
0.690 10.014
Evaluation (.2)
Non-Collab Collab w/o. CFA
0.936
0.399
0.521
0.658
0.902 10.034
0.399 -.0.000
0.1 10.407
0.069 10.588
STAMP (ours)
0.930 10.006
0.734 70.336
0.615 10.094
0.676 70.018
Evaluation
Non-Collab Collab w/o. CFA
0.925
0,399
0.503
0.630
0.886 J0.039
0.400 10.001
0.114 L0.389
0.069 10.561
(a 0.4)
STAMP (ours)
0,923 10.002
0.585 18,186
0.600 10.097
0.650 00.020
Our method consistently outperforms single-agent segmentation for agents 3 and 4 in the BEV segmentation task. Conversely, collaboration without feature alignment significantly degrades per-
8

---

Published as a conference paper at ICLR 2025
formance compared to the single-agent baseline, underscoring the importance of adaptation mechanism aligning heterogeneous features. For agent 2's camera-based 3D object detection, pipeline achieves substantial gains (c.g., AP@50 improves from 0.399 to 0.760 in noiseless conditions), while collaboration without feature alignment shows negligible changes. These results demonstrate our pipeline's effectiveness in bridging domain gaps between heterogeneous, agents, abling successful collaboration across diverse models, sensors, and tasks. The consistent improve- ments, particularly under noisy conditions, highlight our approach's 's robustness and adaptability.
However, we observe that both collaborative approaches lead to performance degradation for Agent I compared to its single-agent baseline, despite method outperforming collaboration without feature alignment. This unexpected outcome attributed to Agent 2's limitations, which rely solely on less accurate camera sensors for 3D object detection. This scenario illustrates bottleneck effect, where a weaker agent constrains the overall system performance, negatively impacting even the strongest agents. This challenge in multi-agent collahoration systems prompts to introduce the concept of a Multi-group Collaboration System. In Appendix A.4, we elaborate on the advantages of such a system and demonstrate how our framework can be easily integrated potentially mitigate performance discrepancies in heterogeneous agent collaborations.
4.4 ABLATION STUDIES
In this section, we conduct ablation studies on three factors that may affect our pipeline's perfor- mance: BEV feature channel size, adapter & reverter architectures, and loss functions for collabora- tive feature alignment. All experiments are conducted on both the OPV2V and 42V4Real datasets.
(A) Clasnnel Siacs
& eo's Anduiteclure
Lase Fundikuu
0.1
Agent
A
Agent APM000
Ageut AProS
Agess Mlll
Agent AiMOco
Agent
Agent
Ageut
Misl
AProo
Miell
Klodl
0.0
-0.1
0..2
-0.4
-0.5
convixl
self-attention
Ly only
Figure 3: Ablation studies on the OPV2V dataset: (a) Model performance across different BEV feature channel sizes. (b) Performance comparison of various adapter and reverter architectures. (c) Performance results using different combinations of loss function components (I.g and T.a).
BEV feature channel size. Changing the BEV feature channel size is essentially form of feature compression, which is crucial for con- trolling communication bandwidth in multi- agent collaboration systems. Our collaborative feature alignment module inherently supports
(A)
Channel Stzes
di & gt's Architectures
Loss Finetioue
Agost
Agent
Agrat
Agouz
Agos APaco
Ages 2
0.000
-0.15
feature compression by adjusting the protocol
-0,050
-0.075
BEV feature's channel size. We experiment with two channel sizes for the protocol BEV feature, 32 and 16, and compare their perfor- mance our standard implementation with channel size of 64 (Figures 3 and 4). Surpris- ingly, reducing the channel size results in only minor performance changes for both datasets, revealing our model's resilience to high BEV feature compression rates.
0.100
-100-101
00150
cuacld
Auy mnly Lag only
sellattenboon
Figure 4: Ablation studies on the V2V4real set.
Adapter & reverter architecture. We evaluate two alternative architectures for the adapter and re- verter- a single x convolutional layer and three self-attention layers- to our standard implementation of three ConvNeXt layers. The results demonstrate that performance is not highly

---

Published as a conference paper at ICLR 2025
Before CFA
After CFA
Testure
Model
Fenure
Model outguls
Scone
Agent Agen 2
Scone
Figure 5: Visualization of feature maps and model outputs before and after Collaborative Feature Aligmment (CFA) for two scenes with different agents and tasks. For 3D object detection, green boxes indicate the ground truth labels and red boxes indicate the predictions. CFA enhances feature clarity and information preservation, resulting in improved perception accuracy across heteroge- neous agents.
sensitive to the adapter and reverter architecture, showcasing our framework's 's flexibility across var- ious specific implementations.
Loss function. The final loss function in our collaborative feature alignment module comprises two components: Ly for feature space alignment and La for decision space alignment. We evaluate our method's performance using each loss function individually. Figure 3 shows that the OPV2V dataset, using only La leads to significant performance drop, while using only L, results in more fluctuating and generally lower performance. Figure 4 demonstrates that the V2V4Real dataset, dropping either La or Lj results in performance degradation. These findings underscore the neces- sity of using both loss functions combination for optimal performance.
4.5 VISUALIZATION
Figure illustrates the impact of our CFA method on feature maps and output results across vari- ous tasks. We visualize feature maps by averaging each channel of the fused feature map, Fy', to (W, H) shape and plotting in a grayscale. Without CFA, the fused feature maps appear noisy and lack critical information for downstream tasks, leading to poor output results. In contrast, CFA sig- nificantly enhances feature preservation, resulting in clearer feature maps and more accurate outputs across different tasks. This visualization demonstrates CFA's effectiveness in maintaining essen- tial information during the fusion process, which directly translates to improved performance CP tasks. More comprehensive visualization results are shown the Appendix A.5.
5 CONCLUSION
In this paper, we introduce STAMP, scalable, task- and model-agnostic multi-agent collaborative perception framework. This framework simultaneously addresses three aspects agent hetero- geneity: varieties in modalities, model architectures, and downstream learning tasks. By utilizing lightweight adapter-reverter pairs, STAMP enables efficient collaborative perception while main- taining high security, scalability, and flexibility. Experiments both the simulated OPV2V dataset and the real-world V2V4Real datasets demonstrate its superior performance and computational ef- ficiency over existing state-of-the-arts. This approach opens avenues for developing more reli- able, efficient, and secure collaborative systems in future autonomous driving applications.
Limitations. Our experiments revealed a bottleneck effect in Collaborative Perception (CP), where the performance of the weakest agent constrains the overall system performance. This finding under- scores the necessity for multi-group collaborative systems, where agents communicate only within defined groups. Such systems could mitigate the bottleneck effect by allowing for more selective collaboration. In Appendix A.4, we provide a more detailed discussion multi-group collaborative systems and the advantages of our framework in this context.
10

---

Published as a conference paper at ICLR 2025
Reproducibility statement. To ensure the reproducibility of our results, we have provided detailed information about our experimental setup, including dataset descriptions, model architectures, and training procedures in the main text and appendices. We encourage researchers to refer to Appendix A. for more implementation details. We also release the codebase at https//github..om/faaa group/STAMP.
Ethics statement. Our task- and model-agnostic framework enhances local model security, reduc- ing risks like model stealing (Oliynyk et al., 2023) and adversarial attacks (Tu et al., 2021). While limiting model sharing improves security, as assessed by (Li et al., 2023b), we recognize the need for further security analysis. We advocate for collaboration with experts to rigorously evaluate and strengthen approach in order to contribute safer and more trustworthy autonomous driving systems and advance privacy in collaborative perception.
REFERENCES
Eduardo Arnold, Mehrdad Dianati, Robert de Temple, and Saber Fallah. Cooperative perception for 3d object detection in driving scenarios using infrastructure sensors. IEEE Transactions on Intelligent Transportation Systems, (3(3):8852-188) 2020. 2
Zhengwei Bai, Guoyuan Wu, Matthew J Barth, Yongkang Liu, Emrah Akin Sisbot, and Kentaro Oguchi. Pillargrid: Deep tearning-based cooperative perception for 3d object detection from onboard-roadside lidar. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), pp. 7743-1749, IEEE, 2022a. 2
Zhengwei Bai, Guoyuan Wu, Matthew Barth, Yongkang Liu, Emrah Akin Sisbot, Kentaro Oguchi, and Zhitong Huang. A survey and framework of cooperative perception: From heterogeneous singleton hierarchical cooperation. arXiv preprint arXiv:2208. 10590, 2022b. 1
Qi Chen, Sihai Tang, Qing Yang, and Song Fu. Cooper: Cooperative perception for connected autonomous vehicles based on 3d point clouds. In 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS), pp. 114-524. IEEE, 2019.
Jiaxun Cui, Hang Qiu, Dian Chen, Peter Stone, and Yuke Zhu. Coopernaut: End-to-end driving with cooperative perception for networked vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17252-17262, 2022.
Chen Fu, Chiyu Dong, Christoph Mertz, and John M Dolan. Depth completion via inductive fusion of planar lidar and monocular camera. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10843-10848. IEEE, 2020.
Hongbo Gao, Bo Cheng, Jianqiang Wang, Keqiang Li, Jianhui Zhao, and Deyi Li. Object classi- fication using cnn-based fusion of vision and lidar in autonomous vehicle environment. IEEE Transactions on Industrial Informatics, 14(9):4224-4231, 2018. 2
Xiangbo Gao, Asiegbu Miracle Kanu-Asiegbu, and Xiaoxiao Du. play cross-spectral spatial-temporal fuser for efficient pedestrian detection. arXiv preprint arXiv:2408.01037 2024a.
Mambast: A plug-and-
Xiangbo Gao, Qinliang Lin, Cheng Luo, Weicheng Xie, Linlin Shen, Keerthy Kusumam, and Siyang Song. Scale-free and task-generic attack: Generating photo-realistic adversarial patterns with patch quilting generator. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2985-2989. IEEE, 2024b. 3
Nathaniel Moore Glaser and Zsolt Kira. We need to talk: Identifying and overcoming communication-critical scenarios for self-driving, arXiv preprint arXiv:2305.0433. 2023. 2
Jingda Guo, Dominic Carrillo, Sihai Tang, Qi Chen, Qing Yang. Song Fu, Xi Wang, Nannan Wang, and Paparao Palacharla. Coff: Cooperative spatial feature fusion for 3-d object detection on autonomous vehicles. IEFE Internet of Things Journal, 8(14):11078- 1087, 2021. 2
11

---

Published as a conference paper at ICLR 2025
Yushan Han, Hui Zhang, Huifang Li, Yi Jin, Congyan Lang, and Yidong Li. Collaborative percep- tion in autonomous driving: Methods, datasets, and challenges. IEEE Intelligent Transportation Systems Magazine, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Hanjiang Hu, Zuxin Liu, Sharad Chitlangia, Akhil Agnihotri, and Ding Zhao. Investigating the impact of multi-lidar placement object detection for autonomous driving. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 2550-2559, 2022a. 2
Senkang Hu, Zhengru Fang, Yiqin Deng, Xianhao Chen, and Yuguang Fang. Collaborative per- ception for connected and autonomous driving: Challenges, possible solutions and opportunities. arXiv preprint ar arXiv:2401.0155 2024.
Yue Hu, Shaoheng Fang. Zixing Lei, Yiqi Zhong, and Siheng Chen. Where2comm: Communication -ffficient collaborative perception via spatial confidence maps. Advances in neu- ral information processing systems, 35:4874-4886, 2022b. 2
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint
orXiv:1412.6980 2014.
Alex H Lang. Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang. and Oscar Beijbom. Point- pillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12697-12705, 2019. 8,
Jinlong Li, Runsheng Xu, Xinyu Liu, Jin Ma, Zicheng Chi, Jiaqi Ma, and Hongkai Yu. Learning for vehicle-to-vehicle cooperative perception under lossy communication. IEEE Transactions on Intelligent Vehicles, 8(4):2650-266 2023a. 2
Yiming Li, Shunli Ren, Pengxiang Wu, Siheng Chen, Chen Feng, and Wenjun Zhang. Learning dis- tilled collaboration graph for multi-agent perception. Advances in Neural Information Processing Systems, 34:29541-29552, 2021. 2
Yiming Li, Dekun Ma, Ziyan An, Zixun Wang. Yiqi Zhong, Siheng Chen, and Chen Feng. V2x- sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving. IEEE Robotics and Automation Letters, 7(4): 1094:-1991, 2022. 2
Yiming Li, Qi Fang, Jiamu Bai, Siheng Chen, Felix Juefei-Xu, and Chen Feng. Among Adver- sarially robust collaborative perception by consensus. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 186-195, 2023b. 2, 11
Yiming Li, Juexiao Zhang, Dekun Ma, Yue Wang, and Chen Feng. Multi-robot scene completion: Towards task-agnostic collaborative perception. In Conference Robot Learning, pp. 2062- 2072. PMLR, 2023c. 3.5
Si Liu, Chen Gao, Yuan Chen, Xingyu Peng, Xianghao Kong, Kun Wang, Runsheng Xu, Wentao Jiang. Hao Xiang, Jiaqi Ma, et al. Towards vehicle-to-everythingg autonomous driving: A survey on collaborative perception. arXiv preprint arXiv:2308. 16714, 2023a. 1
Yen-Cheng Liu, Junjiao Tian, Nathaniel Glaser, and Zsolt Kira. When2com: Multi-agent perception via communication graph grouping. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 4106-4115, 2020a. 2
Yen-Cheng Liu, Junjiao Tian, Chih-Yao Ma. Nathan Glaser, Chia-Wen Kuo, and Zsolt Kira.
Who2com: Collaborative perception via learnable handshake communication. In 2020 IEEE International Conference on Robotics and Automation (ICRA). pp- 6876-6883, IEEE, 2020b. 2
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's-eye view representation. In 2023 IEEE international conference on robotics and automation (ICRA), pp. 2774-2781. IEEE, 2023b.
12

---

Published as a conference paper at ICLR 2025
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-11986, 2022.
Yifan Lu, Yue Hu, Yiqi Zhong, Dequan Wang, Siheng Chen, and Yanfeng Wang. An extensible framework for open heterogeneous collaborative perception. arXiv preprint arXiv:2401. 2024. 1, 3, 7, 8, 2
Gledson Melotti, Cristiano Premebida, and Nuno Gonçalves. Multimodal deep-learning for ob- ject recognition combining camera and lidar data. In 2020 IEEE International Conference Autonomous Robot Systems and Competitions (ICARSC), pp. 177-182. IEEE, 2020.
Daryna Oliynyk, Rudolf Mayer, and Andreas Rauber. know what you trained last summer: A survey on stealing machine learning models and defences, ACM Computing Surveys, 55(14s): 1-41, 2023. 11
Donghao Qiao and Farhana Zulkernine. Adaptive feature fusion for cooperative perception using li- dar point clouds. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 1 1186-1195, 2023. 2
Shuyao Shi, Jiahe Cui, Zhehao Jiang, Zhenyu Yan, Guoliang Xing, Jianwei Niu, and Zhenchao Ouyang. Vips: Real-time perception fusion for infrastructure-assiinn assisted autonomous driving. In Proceedings of the 28th annual international conference on mobile computing and networking,
pp. 133-146, 2022.
Sanbao Su, Yiming Li, Sihong He, Songyang Han, Chen Feng, Caiwen Ding, and Fei Miao. Un- certainty quantification of collaborative detection for self-driving. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 3588-5594 IEFF, 2023. 2
Sanbao Su, Songyang Han, Yiming Li, Zhili Zhang, Chen Feng, Caiwen Ding, and Fei Miao. Col- laborative multi-object tracking with conformal uncertainty propagation. IEEE Robotics and Au- tomation Letters, 2024.7
Mingxing Tan. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: 1/946, 2019. 8,
James Tu, Tsunhsuan Wang, Jingkang Wang, Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun. Adversarial attacks on multi-agent communication. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7768-7777, 2021. 2, 11,3
Binglu Wang, Lei Zhang, Zhaozhong Wang, Yongqiang Zhao, and Tianfei Zhou. Core: Coopera- tive reconstruction for multi-agent perception. In 2023 IEEF/CVF International Conference on Computer Vision (ICCV), pp. 8676-8686. IEEE Computer Society, 2023. 2
Tsun-Hsuan Wang, Sivabalan Manivasagam, Ming Liang, Bin Yang, Wenyuan Zeng, and Raquel Urtasun. V2vnet: Vehicle-to-vehicle communication for joint perception and prediction. In Com- puter Vision-ECCV 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020, Pro- ceedings, Part II 16, pp. 605-621. Springer, 2020. 2
Hao Xiang, Runsheng Xu, and Jiaqi Ma. Hm-vit: Hetero-modal vehicle-to-vehicl. cooperative perception with vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 284-295, 2023. 3,4
Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Hing-Hsuann Yang, and Jiaqi Ma. V2x-vit: Vehicle-to-everything cooperative perception with vision transformer. In European conference on computer vision, pp. 107-124. Springer, 2022a. 2, 8,4
Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma. Opv2v: An open bench- mark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. In 2022 International Conference on Robotics and Automation (ICRA), pp.
IEEE, 2022b. 2,
13

---

Published as a conference paper at ICLR 2025
Runsheng Xu, Weizhe Chen, Hao Xiang. Xin Xia, Lantao Liu, and Jiaqi Ma. Model Model-agnostie -agostic multi- agent perception framework. 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 1471-1478. IEEE, 2023a. 2,5.7
Runsheng Xu, Jinlong Li, Xiaoyu Dong, Hongkai Yu, and Jiaqi Ma. Bridging the domain gap for multi-agent perception. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 6035-6042. IEEE, 2023b. 6,
Runsheng Xu, Zhengzhong Tu, Hao Xiang, Wei Shao, Bolei Zhou, and Jiaqi Ma. Cobevt: Coop- erative bird's eye view semantic segmentation with sparse transformers. In Conference on Robot Learning, pp. 989-1000. PMLR, 2023c. 2,4
Runsheng Xu, Xin Xia, Jinlong Li, Hanzhao Li, Shuo Zhang, Zhengzhong Tu, Zonglin Meng. Hao Xiang, Xiaoyu Dong, Rui Song, al. V2v4real: A real-worll acage-scale dataset for vehicle-to- vehicle cooperative perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13712-13722, 2023d. 2,
Runsheng Xu, Chia-Ju Chen, Zhengzhong Tu, and Ming-Hsur Yang. V2x-vitv2: Improved vi- sion transformers for vehicle-to-evee tveeryyhhnn cooperative perception. IEEE transactions on pattern analysis and machine intelligence, 2024. 8
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018. 8,
Dingkang Yang, Kun Yang, Yuzheng Wang, Jing Liu, Zhi Xu, Rongbin Yin, Peng Zhai, and Lihua Zhang. How2comm: Communication--fficcii and collaboration praggaatiio multi-agent percep- tion. Advances in Neural Information Processing Systems, 36, 2024.
Melih Yazgan, Mythra Varun Akkanapragada, and Marius 7öllner. Collaborative perception datasets in autonomous driving: A survey. In 2024 IEEE Intelligent Vehicles Symposium (IV), pp. 2269-2276. IEEE, 2024.
Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, and Zaiqing Nie. Dair-v2x: large-scale dataset for rehicle-infrastructur cooperative 3d object detection. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 21361-21370, 2022. 2
Haibao Yu, Yingjuan Tang. Enze Xie, Jilei Mao, Jirui Yuan, Ping Luo, and Zaiqing Nie. Vehicle- infrastructure cooperative 3d object detection via feature flow prediction. arXiv preprint arXiv:2303. 10552, 2023. 2
Wenyuan Zeng, Shenlong Wang, Renjie Liao, Yun Chen, Bin Yang, and Raquel Urtasun. Dsdnet: Deep structured self-driving network. In Computer Vision-ECCC 2020: 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pp. 156-172. Springer, 2020.
Yin Zhou and Oncel Tuzel. Voxelnet: In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4490- 4499, 2018. 1
learning for point cloud based 3d object detection.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 IMPLEMENTATION DETAILS
For training all models, we initialize the learning rate at 0.001 and reduce it by a factor of 0.1 at 50% and 83% of the total epochs. We utilize single NVIDIA RTX A6000 GPU for both model training and inference. Training time for each model varies between 7 to 30 GPU hours, de- pending the specific model architecture. For adapters and reverters, we start with learning rate of 0.01, reduc- ing it by a factor of 0.1 after the first epoch. These com- ponents are trained pairs, requiring to GPU hours depending the specific encoder and decoder architec- tures.
input
(W Hr
spatial resolution alignment (bilinear interpolation)
(W ow out Cor
input alignment (conv Ixl)
(W ous" H ou C niddonn
feature pattern alignment (ConvNext x3)
(W our H ow" C hidden
Adapter and Reverter's Architecture. We use the same architectures for both adapters and reverters across all CP models, as visualized in Figure A1. The dimension of the broadcasting feature map is set to (128, 128, 64). Ghidden is set to be 64. W Hin Cin, oomts Hom: and Con of adapters and reverters vary according to the feature dimensions of each local model and the broadcasting feature map dimension. For instance. in the task- and model-agnostic setting, Agent I's feature dimension is 128 x 128 64, so we set (Wi.- Hm: Cin) = (Waut: Hout, Cou) = (128, 128, 64) for both its adapter and reverter. For Agent 2, with a feature dimension of 64 x 64 x 256, we configure the adapter with (Win; Hin Cin) (64, 64, 256) and (Wom: om: Hout, Cout) = (128, 128, 64), while the reverter is set with (Win, Hins Cin) = (128, 128, 64) and (W Homty Cout) = (64, 64, 256).
output alignment (conv 1xl)
(W owr H ow
C. onn
output
Figure A1: Architecture of adapter and reverter.
Table Al: Modality, encoder, and encoder parameters (M) of each heterogeneous model in the 3D object detection setting.
Index
Agent
Agent Lidar
Agent 3
Agent
Modulity
Lidar
Camera
Cumera
Encoder
PointPillar (Lang et al., 2019)
SECOND) (Yan et al., 2018)
EfficientNetH00
ResNet101 (He et al., 2016)
(Tan, 2019)
Encoder Param,(M)
0.87
3.79
56.85
6.88
Index
5
Agent
Agent 7
Agent
Modality
Camera
Lidar
Camera
Lidar
Encoder
ResNet34 (He et al., 2016)
VoxeINet (Zhou & Tuzel, 2018)
EfficieniNeeBl (Tan, 2019)
PointPillar (lurge) (Lang al., 2019)
Encoder (M))
6.51
2.13
66,41
1.91
Index
Agent
Agent
Agent 11
Agent
Modality
Camera
Lidar
Cameru
Lidar
Encoder
ResNet50 (He et al., 2016)
SECOND (large) (Yan et al., 2018)
NeficientNet822 (Tan, 2019)
VoxeINet (large)
(Zhou & Tuzel, 2018)
Encoder Param.(M)
6.88
4.82
71.43
3.18
3D object detection setting. Under the experiments on 3D object detection task, we prepared 12 heterogeneous models. Table Al displays the Modality, Encoder, and Encoder Parameters (M) information of each the 12 heterogeneous models. For model 7, 9, and 11, we enlarge the encoders by increasing the size of hidden layers. For all heterogeneous models, layers proposed by Lu et al. (2024) to be the fusion module and three 1 1 convolutional layers for classification, regression, and direction, respectively.
choose pyramid fusion

---

Published as a conference paper at ICLR 2025
A.2 ARCHITECTURAL COMPARISON BETWEEN EXISTING FRAMEWORKS
Figure A2 illustrated various frameworks that address heterogeneous CP. Late fusion simply com- bines agent outputs through post-processing. Calibrator (Xu et al., 2023a) enhances this approach by using calibrators address domain gaps between heterogeneous agent outputs. End-to-end train- ing, while effective, lacks scalability due to its requirement of re-training all agents' models. It also compromises security and task flexibility by shared fusion models and decoders. HEAL (Lu et al., 2024) improves upon this by fixing decoders and fusion models, re-training only the encoders, ducing training resources but still facing scalability issues due to the computational cost of encoder retraining as well as the security issue due to the shared fusion models and decoders. Our proposed framework, STAMP, introduces a novel approach using lightweight adapter and reverter pairs to align feature maps for collaboration. The lightweight nature of these components ensures scalabil- ity, while the maintenance of local fusion and decoders ensures both security and task agnosticism. This design effectively addresses the limitations of previous methods.
Late Fusion
Calibrator (Xu ct al., 2023a)
End-to-end Training
Ipput
Impar
Ingun
Daca Ingux
Da Irpor
Escodar
Encadar
Encodkr
Enceder:
Usaligred Deemitation Intennedizee CP)
Usalignal
Distibotion Innomndivse
Distibution
Fasturs
Feature
Feature
Feaure
Fasuue
Feguue
Shand Fusizn Model
Traeing
Dansler
Oupui
Ougput
Outpu
Ouipu
M
(Late Piston)
Posed
Callmater
(Late Piston)
Fosed
Shared Desoder (NOT Safe)
(NOT Tasc-sgnoolk)
Oulput
HEAL (Lu et al., 2024)
STAMP (ours)
Data Imput
Jipur
Ixpert
(NOT Scalable)
Aligned Diaribation (Callaberstive)
Adester
(Scalabla)
Reventer Wil
Reverter 2tt
Frature
Feature
M
Stared Model
Forum
Allaned
Aligned
Stazod Fusion Model & Dezader (NOT Safe) (NOT Tesk agrostic)
Fasice Model
Faúco Model
Feead
M
Local Decoders (Sate & Task- agnostic)
Ougpa
Desoder
Decoder
Outper
Figure A2: Architectural comparison of collaborative perception frameworks: existing approaches versus our proposed STAMP method. Blue boxes represent models with fixed parameters, while red boxes indicate models whose parameters are trained during the collaboration process.
2

---

Published as a conference paper at ICLR 2025
A.3
ADDITIONAL EXPERIMENTS
A.3.1 DIFFERENT PROTOCOL MODELS
We conducted complementary experiments comparing different protocol model designs, analyzing variations in both encoder types and downstream tasks.
Table A2: Comparison of protocol designs, encoder types, and tasks across different agents.
Protocol
Encoder Type
Protocal Task
Agent (Lidar+Ohjj)
Agent 2 (Cam.+Obj.)
Agent (Lidar+Static Seg.)
(Lidar+Dyn. Seg.)
Agent
Non- Collab
0,941
0.399
0.548
0.675
STAMP
CNN-based
Object Det.
0.936 J0.005
0.760
10.362
0.624 10.076
0.690 70.014
Camern-modality
Object Det.
0.931 J0.010
0.777
10.368
0,580 10.032
0.671 10.004
Camera + Lidar
Object Det.
0.937 J0.004
0.762 10.363
0.632 10.084
0.714 10.039
STAMP
(ablations)
Point-tranfformer
Object Det.
0.942 70.001
0.775 10.376
0,634 10.086
0.696 70.021
CNN hased
Dyn.
0.935 J0.006
0.743 10.344
0.624 10.076
0.723 70.048
CNN-base
Statie. Seg.
0.747 J0.194
0.412 10.013
0.681 10.133
0.235 J0.440
Impact of Model Objectives The experimental results shown in Table A2 demonstrate that the alignment's success significantly depends on the learning objectives between protocol models and agent architectures. When there strong aligmment between the protocol model and an agent's ob- jectives, we observe performance improvements. For examples, A Gamera-modalle protocol model improves camera- camera-based Agent 2's performance from 0.760 to 0.777; a segmmnttggmentttionn pro- tocol model enhances Agent 4's performance from 0.690 to 0. 0.723. Similarly, a static-segmentation protocol model boosts Agent 3's performance from 0.624 0.681.
However, significant objective mismatches can lead to severe performance degradation. For in- stance, using segmentegttnntnioon protocol model causes Agent 4's mAP to drop dramatically from 0.690 to 0.235. This highlights the importance of careful protocol model selection.
Encoder Architecture Variations While our baseline experiments primarily used CNN-based en- coders, we explicitly tested different encoder architectures to understand their impact. As shown in our results table, we evaluated: CNN-based encoders and Noint-transformet encoders.
The Point-transformer protocol model outperforms the original CNN-based protocol model, show- ing framework's compatibility with different encoder architectures. Notably, the Point- transformer protocol model achieved slightly superior performance (AP@ 50 = 0.991) compared its CNN-based counterpart (AP@50 0.973). This observation suggests important insight: the overall performance of the protocol model is more crucial than its specific architectural design. In other words, well rell-performin protocol model tends to benefit all agent types, regardless of their individual architectures.
A.3.2 ADVERSARIAL ROBUSTNESS EVALUATION
Adversarial attacks involve intentionally perturbing inputs to mislead machine learning models while keeping the modifications nearly imperceptible(Tu et al., 2021; Gao et al., 2024b). Follow- ing James et al. Tu et al. (2021)'s collaborative white-box adversarial attack method with the same hyperparameters, we conducted adversarial attack experiments on the V2V4Real dataset with two agents per scene. We designated agent I as the attacker and agent 2 the victim, comparing three settings:
End-to-end training: Models trained end-to-end with full parameter access, enabling direct white-hox attacks on the victim.
HEAL: Agents share encoders but have different fusion models/decoders. assuming no victim model access.
STAMP: Agents share no local models, using protocol representation for communication, assum- ing victim model access.
The results shown in table A3 demonstrate that adversarial attacks have minimal impact on HEAL and STAMP frameworks due to local model security, while significantly degrading performance in
3

---

Published as a conference paper at ICLR 2025
Table A3: Performance of different frameworks under adversarial attack the V2V4Real dataset.
AP@50
End-to-end
HEAL
STAMP (ours)
Before Attack
0.513
0.515 0,506
0.523
After Attack
0.087
0,503
end-to-end training where models are shared. This empirically supports our framework's 's robustness against malicious agent attacks.
A.3.3 MORE COMPARISON WITH THE STATE-OF-THER THE-ARR METHODS
We conducted additional experiments comparing with - et al., 2022a), CoBEVT(Xu et al., 2023c), HM-ViT(XXinn et al., 2023), HEAL(Lu et al., 2024) in a heterogeneous input modal- ity setting. We configured four agents: two LiDAR agents using PointPillar and SECOND encoders, and two camera agents using EfficientNet-bo and ResNet-100 encoders. For CoBEVT, HM-ViT, and HEAL, we followed their standard architecture and port camera modality, images to BEV features. For STAMP, we used pyramid fusion layers and three 1 convolutional layers (for classification, regression, and direction) across all heterogeneous models.
setup. V2X- VIT does not sup- follow HEAL to use ResNet-100 101 with Split-slat-shot for encoding RGB
Table A4: Comparison of performance metrics (AP@50) for different models across heterogeneous agents.
Agent (PointPillar)
Agent 2 (SECOND)
Agent 3 (EfficientNet-BO))
Agent 4 ((ReNNet-101)
AP@50
Average
V2X-VIT CoBEVT
0.905
0.899
HM-ViT
0,918
HEAL
0.971
0.958
0.776
0.771
0.934
STAMP (ours)
0.971
0.963
0.771
0.756
0.934
Note that CoBEVT, HM-VIT, and V2X-VIT use single fusion layer and output layer for all modal- ities, while HEAL and our framework maintain separate fusion and output layers for each agent to preserve model independence. The reported accuracy is averaged across all samples.
A.4 MULTI-GROUP AND MULTI-MODEL COLLABORATIONS SYSTEM
In experimental findings, observed bottleneck effect in CP systems, where the overall sys- tem performance is constrained by the capabilities of the weakest agent. This limitation underscores the need for more selective collaboration, leading Group a set of agents that collaborate under specific criteria. These criteria are essential for maintaining the quality and integrity of CP, admitting agents that meet predefined standards while excluding those with inferior models, potential malicious intent, or incompatible alignments. As illustrated in Figure A3, we can distinguish between three collaborative system types:
introduce the concept of Collaboration
Single-group systems, where agents either operate independently or are compelled to collaborate with all others, are susceptible performance bottlenecks caused by inferior agents and vulnera- bilities introduced by malicious attackers.
Multi-group single-model systems, allowing multiple collaboration groups but restricting agents to a single group because each agent can only equip single model.
Multi-group multi-mode systems, enabling agents to join multiple groups if they meet the prede- fined standards.
The multi croti structure offers significant advantages over traditional single- single-group group systems. en- hances agents' potential for diverse collaborations, consequently improving overall performance. This approach mitigates the bottleneck effect by allowing high herfo-irn agents to maintain ficiency within groups of similar capability while potentially assisting less capable agents in other
4

---

Published as a conference paper at ICLR 2025
Single group System
Multi group Single-model System
Mulli group Multi model System
Institate
I Group
Cellaborabae
Idin ond greup ONLY
Mutts group ALLOWED
Group
Institutt (vetiicle))
Single
Caltaborative
Institute (Netisl))
Collaborative Group
Group
Figure A3: Comparison of collaborative perception systems: (Left) Single-group system where all agents collaborate within one group. (Middle) Multi-group single-model system allowing agents to join only of multiple collaboration groups. (Right) Multi-group multi-model system enabling agents to participate multiple collaboration groups simultaneously. The figure illustrates how different system architectures impact agent interactions and group formations in autonomous driving scenarios.
groups. Furthermore, it enhances system flexibility, enabling dynamic group formation based on specific task requirements or environmental conditions.
However, implementing such multi-group system poses challenges for existing heterogeneous collaborative pipelines. End-to-end training approaches require simultaneous training of all models, conflicting with the concept of distinct collaboration groups. Methods like those proposed by Lu et al. (2024) require separate encoders for each group, becoming impractical as the number of groups increases due computational and memory constraints.
Our proposed STAMP framework effectively addresses these limitations, offering a scalable solu- tion for multi-group CP. The key innovation lies in its lightweight adapter and reverter pair (approx- imately IMB) required for each collaboration group an agent joins. This efficient design enables agents to equip multiple adapter-reverter pairs, facilitating seamless participation in various groups without significant computational overhead. The minimal memory footprint ensures scalability, agents join numerous collaboration groups, making STAMP particularly well-suited for multi-group and multi model collaboration systems.
A.S MORE VISUALIZATION RESULTS
Figure A4 and A5 illustrate more feature map and result visualizations before and after collaborative feature alignment (CFA), Prior to CFA, agents' feature maps exhibit disparate representations. For instance, in Figure A4, the pre-fusion feature maps of agents 1, 3, and 4 appear entirely black, indicating significantly lower scale compared to agent 2's feature map. This discrepancy leads to instability in feature fusion. Post-CFA, the features are aligned to the same domain, resulting in more coherent fusion and accurate inference outputs.

---

Published as a conference paper at ICLR 2025
Features before fusion
Before CFA
Agent
Agent
Agent
Agent
Fused Features
Outputs
Agent 1
Agent 2
Agent 3
Agent 4
After CFA
Adapted Features before fusion
A1
A2-AA1
A1-A2
A2
A1-A3
A2-5A3
A1-A4
A2-A4
A3-AI
A4-AI
A3-A2
A4-A2
A4-
A3-A4
Fused Features
Outputs
Agent
Agent 2
Agent 3
Agent 4
Figure A4: Visualization of feature maps and inference results before and after Collaborative Feature Alignment (CFA) in three-agent scene. A, Ay denotes the feature map aligned from agent i's domain to agent j's domain, also represented as F.j.

---

Published as a conference paper at ICLR 2025
Before CFA
Features before fusion
Agent
Agent 3
Agent
Fused Features
Outputs
After CFA
Adapted Features before fusion
Agent
Agent
Agent
Agent
Agent
Agent
Agent
Fused Features
Outputs
Agent 1
Agent 3
Agent 4
Figure A5: Visualization of feature maps and inference results before and after Collaborative Feature Alignment (CFA) four-agent scene. A, - A, denotes the feature map aligned from agent i's domain to agent j's domain, also represented as F,j.


Reviews:
[{"review_id": "DQcADhVaHF", "reviewer": "Reviewer_MyQk", "summary": "This paper presents a framework for collaborative perception that is argued to be salable, tasks-independent and model agnostic, which is also argued to be capable of dealing with heterogeneity in the agents and enhancing flexibility and security.", "questions": "See above.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ Heterogeneity in collaborative agents, especially in robotics outside of collaborative driving, is an important problem.\n\n+ The proposed method is straightforward, and the writing is easy to follow.", "weaknesses": "- Novelty is a key concern. The proposed method is based on BEV representations and performs intermediate fusion using the BEVs. Each component also uses or marginally extends existing methods, and the novelty of each component is not well justified. For example, Collaborative Feature Alignment (CFA) simply uses the BEV from each agent and projects it to a unified feature space.\n\n- The paper argues that the proposed approach can address heterogeneity in the agents; however, the solution is simply based on the BEV representation that is assumed to be computed by each agent.\n\n- Similarly, the claim that the proposed method is model-agnostic is also an overstatement, primarily due to its reliance on the BEV representation.\n\n- Why is the proposed method secure? The method uses a set of neural networks, so how can we ensure that the use of these networks is secure?\n\n- What is the coordinate frame of the BEV representation used by each vehicle? If the BEV is ego-centric for each vehicle, how can the correspondence of street objects between agents be found?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AbiCPX8pyP", "reviewer": "Reviewer_fgQh", "summary": "This paper introduces STAMP (Scalable Task- and Model-Agnostic Collaborative Perception), a framework designed to enable efficient, secure, and scalable multi-agent collaborative perception (CP) in autonomous driving systems. Recognizing the challenges posed by heterogeneous agents—such as varying sensors, models, and tasks—STAMP employs lightweight adapter-reverter pairs to align Bird’s Eye View (BEV) features to a unified protocol, allowing agents to collaborate without sharing model details. The framework is validated on simulated (OPV2V) and real-world (V2V4Real) datasets.", "questions": "Could the authors illustrate task-agnostic collaborative perception more (especially the difference compared to the prior work [1])? As this prior work can be trained without knowing downstream tasks. However, the proposed framework in this paper seems to be trained on some specific tasks and is hard to generalize to novel downstream tasks. The authors are suggested to illustrate the limitations and setups clearly.  \n\n[1] Yiming Li, Juexiao Zhang, Dekun Ma, Yue Wang, and Chen Feng. Multi-robot scene completion: Towards task-agnostic collaborative perception. In Conference on Robot Learning, pp. 2062–2072. PMLR, 2023c. 3, 5", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. STAMP initiates the first study of task-agnostic and model-agnostic collaborative perception, and verifies the effectiveness of the proposed method in connected and autonomous driving scenarios.\n\n2. STAMP introduces a unique adapter-reverter mechanism to bridge heterogeneity gaps in multi-agent collaboration.\n\n2. STAMP addresses a critical need in autonomous driving by enabling heterogeneous agents to collaborate effectively, setting a foundation for more secure, scalable CP frameworks applicable to various downstream tasks.", "weaknesses": "1. While the proposed method claims scalability, the experiments include only three agents, which limits the demonstration of scalability in larger, more complex multi-agent systems.\n\n2. Although the method is presented as task-agnostic, generalizing to untrained downstream tasks may prove challenging, suggesting that the adaptability across broader task domains could benefit from further validation.\n\n3. The writing needs to be polished. The main paper contains numerous intricate details that might be better suited for the appendix, and the main figure contains redundant information that could be streamlined for clarity.\n\n4. The framework focuses on vehicle-to-vehicle communication, which may narrow its broader impact within the ICLR community. Expanding the scope or exploring applications outside autonomous driving could increase its relevance.\n\n5. The paper could benefit from a more comprehensive literature review, as some highly relevant works on efficient and scalable collaborative perception are missing [1-7]. Including a broader range of recent studies would provide a stronger context for the contributions and better situate the proposed framework within the current state of research.\n\n[1] Li, Y., Ren, S., Wu, P., Chen, S., Feng, C. and Zhang, W., 2021. Learning distilled collaboration graph for multi-agent perception. Advances in Neural Information Processing Systems, 34, pp.29541-29552.\n\n[2] Li, Y., Ma, D., An, Z., Wang, Z., Zhong, Y., Chen, S. and Feng, C., 2022. V2X-Sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving. IEEE Robotics and Automation Letters, 7(4), pp.10914-10921.\n\n[3] Hu, Y., Fang, S., Lei, Z., Zhong, Y. and Chen, S., 2022. Where2comm: Communication-efficient collaborative perception via spatial confidence maps. Advances in neural information processing systems, 35, pp.4874-4886.\n\n[4] Huang, S., Zhang, J., Li, Y. and Feng, C., 2024. Actformer: Scalable collaborative perception via active queries. ICRA 2024.\n\n[5] Yang, D., Yang, K., Wang, Y., Liu, J., Xu, Z., Yin, R., Zhai, P. and Zhang, L., 2024. How2comm: Communication-efficient and collaboration-pragmatic multi-agent perception. Advances in Neural Information Processing Systems, 36.\n\n[6] Su, S., Li, Y., He, S., Han, S., Feng, C., Ding, C. and Miao, F., 2023, May. Uncertainty quantification of collaborative detection for self-driving. In 2023 IEEE International Conference on Robotics and Automation (ICRA) (pp. 5588-5594). IEEE.\n\n[7] Su, S., Han, S., Li, Y., Zhang, Z., Feng, C., Ding, C. and Miao, F., 2024. Collaborative multi-object tracking with conformal uncertainty propagation. IEEE Robotics and Automation Letters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ayCFkYDRGP", "reviewer": "Reviewer_XjTw", "summary": "This work focuses on collaborative perception for heterogeneous agents and proposes a task- & model- agnostic framework that is scalable, efficient & secure. The framework involves learning a shared latent representation space, referred to as the protocol feature, using BEV features from LiDAR input. For each agent, a learned local adapter and reverter modules are used to map the agent-specific BEV features to the protocol feature space and vice versa. The protocol feature space is learned jointly across agents and tasks, while the local adapters and reverters are trained separately. Since the protocol feature space is learned once and adapter-reverter modules are lightweight, the proposed framework is scalable and efficient. Moreover, using a shared feature space prevents the need to share information about modalities & models from different agents, thus improving security. Extensive experiments on OPV2V and V2V4Real datasets show the effectiveness of the proposed framework in various scenarios.", "questions": "Some aspects need clarification, which are mentioned in the Weaknesses above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work studies collaborative perception for heterogeneous agents and focuses on efficiency, scalability & security, which is important from a practical perspective.\n- The shared latent representation space only needs to be learned once and the adapter-reverter modules are lightweight (~1MB), thereby reducing computational overhead (7.2x saving) and improving efficiency.\n- Using a shared feature space avoids sharing information about the sensors, models & tasks, making it task-& model-agnostic and secure.\n- The ideas are intuitive and the paper is well written & easy to follow.\n- Experiments in both simulated (OPV2V) and real (V2V4Real) scenarios show the effectiveness of the proposed framework over several baselines in terms of detection performance (Tab.2,3), scalability (Fig.2), efficiency (Fig.2), better robustness to noise (Tab.1), being task-agnostic (Tab.4) & model-agnostic (Tab.4).\n- Ablation study on different architectural components (Fig.3,4) and visualization of features & outputs (Fig.5) help to the capabilities of the proposed framework.", "weaknesses": "- The protocol feature space is learned using BEV features from LiDAR data. Is there a way to extend this to incorporate other modalities like RGB as well since dense semantic features from RGB complement sparse geometric features from LiDAR. It would also enhance the modality-agnostic aspect of the proposed framework and might scale better to real-world datasets.\n- In the current experiments, the model- and task- agnostic setting is considered on the simulated OPV2V dataset. Is there any reason why this cannot be extended to real-world datasets like nuScenes? This would be helpful to verify if the trends hold on real-world datasets as well. This is not required for rebuttal but additional clarifications would be helpful.\n- For multi-group collaborative systems, it seems like the agents might need to share extra information to form groups, e.g. which is the weaker modality. This might affect the modality agnostic or security aspects of the proposed framework. It'd be useful to provide some more insights into this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZKFWr64RFu", "reviewer": "Reviewer_m4Vi", "summary": "The manuscript proposed a collaborative perception framework for heterogeneous agents, highlighting its feature of task- and model-agnostic. The framework contains a lightweight adapter-reverter pair, transforming the features between agent-specific domains and a shared protocol domain. The framework is tested on both simulated(OPV2V) and real-world (V2V4Real) datasets.", "questions": "1. Following the weakness section, have authors considered the risk of alignment failure when there are significant capability differences between models? How do you pick the protocol model to maintain the robustness of the framework?\n\n2. Another follow-up to the weaknesses section, this reviewer noticed that, in the experimental setup, all encoders are CNN-based. Has the author tried different combinations of protocol models and agent encoders, such as using a transformer-based protocol model while allowing agents to have either CNN-based or transformer-based encoders or other combinations.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using an adapter-reverter pair in collaborative perception system for heterogeneous agents is intuitive. The design of the adapter & reverter is light-weight and does support solving most of the issue caused by heterogeneous encoders, such as resolution feature dimension. \n\n2. The experimental results validate the effectiveness of the framework and prove it is task- and model-agnostic. More surprisingly, the training efficiency is significantly higher than the existing methods.\n\n3. The manuscript organizes very well and the visualization is clear and easy to understand.", "weaknesses": "In the methodology section, the authors propose using the L2 norm as the training loss to align the agent-specific features with the protocol features. However, to fully understand this approach, more information on the architecture and capability of the protocol model is needed. Knowledge-distillation designs like this can sometimes risk alignment failure if there is a significant capability gap between the models. This may also explain the limitation noted in A.3, where the system performance is constrained by the weakest agent. \n\nThis reviewer is concerned that this may pose a drawback, as finding a suitable protocol model that meets the requirements of various modern encoders could be challenging. To address this concern, it would be helpful if the authors can provide more details about criteria of the protocol model selection, the current protocol model architecture, model size and its capabilities like task performances, as well as the comparison with those of the agent models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wPL7eq0gaQ", "reviewer": "Reviewer_6FAr", "summary": "This paper proposes STAMP, a task- and model-agnostic collaborative perception framework. The core idea is to first obtain a protocol BEV feature space, then align other local models' BEV features to this space using a simple DNN projection to achieve model agnosticism. A simple DNN is also used to map the aligned BEV features to a specific decoder and task head, achieving task agnosticism. Finally, experiments were conducted on OPV2V and V2V4Real.", "questions": "1. In experiment, why use AP@30 and AP@50 rather than AP@50 and AP@70? I think AP@30 is not usually used in detection task.\n2. Why not conduct experiments about the communication efficiency.\n3. Since the task is different, how to align the decision space with different tasks (GT) in Sec. 3.3\n4. For the feature space alignment, I don’t think it always works, some times it may has negative influence, because the BEV feature distribution is different across different agent. From Figure 5, we can see that the styles of different agents’ feature are not the same. As a result, just simply forcing the features to be same is not a good idea.\n5. The current trend in autonomous driving models is towards increasing model size and vehicle computational power. Additionally, there is a shift towards end-to-end models, significantly enhancing the autonomous capabilities of individual vehicles. Given these advancements, how much market potential remains for multi-vehicle cooperative perception based on intermediate BEV feature communication? How does the author view this issue?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposed a task- and modal-agnostic CP framework, which is new and the first work to address heterogeneous modalities, heterogeneous model architectures or parameters, and heterogeneous downstream tasks simultaneously.\n2. The writing is good and fluent.", "weaknesses": "1. On line 227, the authors claim that the protocol model is not limited to any specific architecture or downstream task, making it a task- and model-agnostic framework. However, I disagree. The framework is task- and model-agnostic because it allows newly added agents to use different models or tasks, rather than due to the flexibility of the protocol model itself.\n2. I think the author should compare more baseline methods, such as HM-ViT, DiscoNet, V2VNet, V2X-ViT, Where2comm, When2com, and What2com, not just compare with HEAL. I know your idea comes from HEAL, but comparing with other methods is necessary.\n3. In Tab.2 and 3, I observe that STAMP achieves the best performance. However, I have some concerns about E2E training. The E2E training supposes to be the best, since it has the entire parameters to adapt the domain gap, which STAMP just use two projection DNN to adapt the features between different modalities and models, which is not make sense.\n4. In Tab. 4, I find that the STAMP just has very little improvement even degradation. I don’t think there is much significance. \n5. On line 52, the authors claim that this framework is robust against malicious agent attacks. However, they haven't proven this or conducted even a single experiment to support it. Moreover, I believe this claim is questionable. Although an attacker might not know the other agents' models, they could still inject malicious information into the protocol BEV features to attack the ego vehicle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)