STAMP: Scalable Task- And Model-agnostic Collaborative Perception

Authors: Xiangbo Gao, Runsheng Xu, Jiachen Li, Ziran Wang, Zhiwen Fan, Zhengzhong Tu
Primary area: applications to robotics, autonomy, planning
Keywords: Autonomous Driving, Collaborative Perception, Domain Adaptation

Abstract:
Perception is a crucial component of autonomous driving systems. However, single-agent setups often face limitations due to sensor constraints, especially under challenging conditions like severe occlusion, adverse weather, and long-range object detection. Multi-agent collaborative perception (CP) offers a promising solution that enables communication and information sharing between connected vehicles. Yet, the heterogeneity among agents—in terms of sensors, models, and tasks—significantly hinders effective and efficient cross-agent collaboration. To address these challenges, we propose STAMP, a scalable task- and model-agnostic collaborative perception framework tailored for heterogeneous agents. STAMP utilizes lightweight adapter-reverter pairs to transform Bird's Eye View (BEV) features between agent-specific domains and a shared protocol domain, facilitating efficient feature sharing and fusion while minimizing computational overhead. Moreover, our approach enhances scalability, preserves model security, and accommodates a diverse range of agents. Extensive experiments on both simulated (OPV2V) and real-world (V2V4Real) datasets demonstrate that STAMP achieves comparable or superior accuracy to state-of-the-art models with significantly reduced computational costs. As the first-of-its-kind task- and model-agnostic collaborative perception framework, STAMP aims to advance research in scalable and secure mobility systems, bringing us closer to Level 5 autonomy. Our project page is at https://xiangbogaobarry.github.io/STAMP and the code is available at https://github.com/taco-group/STAMP.

Decision: accept (poster)
Reviewer scores: 3, 8, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1729593779314

thread_type:
official_review

<thread_object>
creator: reviewer_6far

<content>
summary:
This paper proposes STAMP, a task- and model-agnostic collaborative perception framework. The core idea is to first obtain a protocol BEV feature space, then align other local models' BEV features to this space using a simple DNN projection to achieve model agnosticism. A simple DNN is also used to map the aligned BEV features to a specific decoder and task head, achieving task agnosticism. Finally, experiments were conducted on OPV2V and V2V4Real.

strengths:
1. This paper proposed a task- and modal-agnostic CP framework, which is new and the first work to address heterogeneous modalities, heterogeneous model architectures or parameters, and heterogeneous downstream tasks simultaneously.
2. The writing is good and fluent.

weaknesses:
1. On line 227, the authors claim that the protocol model is not limited to any specific architecture or downstream task, making it a task- and model-agnostic framework. However, I disagree. The framework is task- and model-agnostic because it allows newly added agents to use different models or tasks, rather than due to the flexibility of the protocol model itself.
2. I think the author should compare more baseline methods, such as HM-ViT, DiscoNet, V2VNet, V2X-ViT, Where2comm, When2com, and What2com, not just compare with HEAL. I know your idea comes from HEAL, but comparing with other methods is necessary.
3. In Tab.2 and 3, I observe that STAMP achieves the best performance. However, I have some concerns about E2E training. The E2E training supposes to be the best, since it has the entire parameters to adapt the domain gap, which STAMP just use two projection DNN to adapt the features between different modalities and models, which is not make sense.
4. In Tab. 4, I find that the STAMP just has very little improvement even degradation. I don’t think there is much significance. 
5. On line 52, the authors claim that this framework is robust against malicious agent attacks. However, they haven't proven this or conducted even a single experiment to support it. Moreover, I believe this claim is questionable. Although an attacker might not know the other agents' models, they could still inject malicious information into the protocol BEV features to attack the ego vehicle.

questions:
1. In experiment, why use AP@30 and AP@50 rather than AP@50 and AP@70? I think AP@30 is not usually used in detection task.
2. Why not conduct experiments about the communication efficiency.
3. Since the task is different, how to align the decision space with different tasks (GT) in Sec. 3.3
4. For the feature space alignment, I don’t think it always works, some times it may has negative influence, because the BEV feature distribution is different across different agent. From Figure 5, we can see that the styles of different agents’ feature are not the same. As a result, just simply forcing the features to be same is not a good idea.
5. The current trend in autonomous driving models is towards increasing model size and vehicle computational power. Additionally, there is a shift towards end-to-end models, significantly enhancing the autonomous capabilities of individual vehicles. Given these advancements, how much market potential remains for multi-vehicle cooperative perception based on intermediate BEV feature communication? How does the author view this issue?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730503500544

thread_type:
official_review

<thread_object>
creator: reviewer_m4vi

<content>
summary:
The manuscript proposed a collaborative perception framework for heterogeneous agents, highlighting its feature of task- and model-agnostic. The framework contains a lightweight adapter-reverter pair, transforming the features between agent-specific domains and a shared protocol domain. The framework is tested on both simulated(OPV2V) and real-world (V2V4Real) datasets.

strengths:
1. The idea of using an adapter-reverter pair in collaborative perception system for heterogeneous agents is intuitive. The design of the adapter & reverter is light-weight and does support solving most of the issue caused by heterogeneous encoders, such as resolution feature dimension. 

2. The experimental results validate the effectiveness of the framework and prove it is task- and model-agnostic. More surprisingly, the training efficiency is significantly higher than the existing methods.

3. The manuscript organizes very well and the visualization is clear and easy to understand.

weaknesses:
In the methodology section, the authors propose using the L2 norm as the training loss to align the agent-specific features with the protocol features. However, to fully understand this approach, more information on the architecture and capability of the protocol model is needed. Knowledge-distillation designs like this can sometimes risk alignment failure if there is a significant capability gap between the models. This may also explain the limitation noted in A.3, where the system performance is constrained by the weakest agent. 

This reviewer is concerned that this may pose a drawback, as finding a suitable protocol model that meets the requirements of various modern encoders could be challenging. To address this concern, it would be helpful if the authors can provide more details about criteria of the protocol model selection, the current protocol model architecture, model size and its capabilities like task performances, as well as the comparison with those of the agent models.

questions:
1. Following the weakness section, have authors considered the risk of alignment failure when there are significant capability differences between models? How do you pick the protocol model to maintain the robustness of the framework?

2. Another follow-up to the weaknesses section, this reviewer noticed that, in the experimental setup, all encoders are CNN-based. Has the author tried different combinations of protocol models and agent encoders, such as using a transformer-based protocol model while allowing agents to have either CNN-based or transformer-based encoders or other combinations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730610858917

thread_type:
official_review

<thread_object>
creator: reviewer_xjtw

<content>
summary:
This work focuses on collaborative perception for heterogeneous agents and proposes a task- & model- agnostic framework that is scalable, efficient & secure. The framework involves learning a shared latent representation space, referred to as the protocol feature, using BEV features from LiDAR input. For each agent, a learned local adapter and reverter modules are used to map the agent-specific BEV features to the protocol feature space and vice versa. The protocol feature space is learned jointly across agents and tasks, while the local adapters and reverters are trained separately. Since the protocol feature space is learned once and adapter-reverter modules are lightweight, the proposed framework is scalable and efficient. Moreover, using a shared feature space prevents the need to share information about modalities & models from different agents, thus improving security. Extensive experiments on OPV2V and V2V4Real datasets show the effectiveness of the proposed framework in various scenarios.

strengths:
- This work studies collaborative perception for heterogeneous agents and focuses on efficiency, scalability & security, which is important from a practical perspective.
- The shared latent representation space only needs to be learned once and the adapter-reverter modules are lightweight (~1MB), thereby reducing computational overhead (7.2x saving) and improving efficiency.
- Using a shared feature space avoids sharing information about the sensors, models & tasks, making it task-& model-agnostic and secure.
- The ideas are intuitive and the paper is well written & easy to follow.
- Experiments in both simulated (OPV2V) and real (V2V4Real) scenarios show the effectiveness of the proposed framework over several baselines in terms of detection performance (Tab.2,3), scalability (Fig.2), efficiency (Fig.2), better robustness to noise (Tab.1), being task-agnostic (Tab.4) & model-agnostic (Tab.4).
- Ablation study on different architectural components (Fig.3,4) and visualization of features & outputs (Fig.5) help to the capabilities of the proposed framework.

weaknesses:
- The protocol feature space is learned using BEV features from LiDAR data. Is there a way to extend this to incorporate other modalities like RGB as well since dense semantic features from RGB complement sparse geometric features from LiDAR. It would also enhance the modality-agnostic aspect of the proposed framework and might scale better to real-world datasets.
- In the current experiments, the model- and task- agnostic setting is considered on the simulated OPV2V dataset. Is there any reason why this cannot be extended to real-world datasets like nuScenes? This would be helpful to verify if the trends hold on real-world datasets as well. This is not required for rebuttal but additional clarifications would be helpful.
- For multi-group collaborative systems, it seems like the agents might need to share extra information to form groups, e.g. which is the weaker modality. This might affect the modality agnostic or security aspects of the proposed framework. It'd be useful to provide some more insights into this.

questions:
Some aspects need clarification, which are mentioned in the Weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685348505

thread_type:
official_review

<thread_object>
creator: reviewer_fgqh

<content>
summary:
This paper introduces STAMP (Scalable Task- and Model-Agnostic Collaborative Perception), a framework designed to enable efficient, secure, and scalable multi-agent collaborative perception (CP) in autonomous driving systems. Recognizing the challenges posed by heterogeneous agents—such as varying sensors, models, and tasks—STAMP employs lightweight adapter-reverter pairs to align Bird’s Eye View (BEV) features to a unified protocol, allowing agents to collaborate without sharing model details. The framework is validated on simulated (OPV2V) and real-world (V2V4Real) datasets.

strengths:
1. STAMP initiates the first study of task-agnostic and model-agnostic collaborative perception, and verifies the effectiveness of the proposed method in connected and autonomous driving scenarios.

2. STAMP introduces a unique adapter-reverter mechanism to bridge heterogeneity gaps in multi-agent collaboration.

2. STAMP addresses a critical need in autonomous driving by enabling heterogeneous agents to collaborate effectively, setting a foundation for more secure, scalable CP frameworks applicable to various downstream tasks.

weaknesses:
1. While the proposed method claims scalability, the experiments include only three agents, which limits the demonstration of scalability in larger, more complex multi-agent systems.

2. Although the method is presented as task-agnostic, generalizing to untrained downstream tasks may prove challenging, suggesting that the adaptability across broader task domains could benefit from further validation.

3. The writing needs to be polished. The main paper contains numerous intricate details that might be better suited for the appendix, and the main figure contains redundant information that could be streamlined for clarity.

4. The framework focuses on vehicle-to-vehicle communication, which may narrow its broader impact within the ICLR community. Expanding the scope or exploring applications outside autonomous driving could increase its relevance.

5. The paper could benefit from a more comprehensive literature review, as some highly relevant works on efficient and scalable collaborative perception are missing [1-7]. Including a broader range of recent studies would provide a stronger context for the contributions and better situate the proposed framework within the current state of research.

[1] Li, Y., Ren, S., Wu, P., Chen, S., Feng, C. and Zhang, W., 2021. Learning distilled collaboration graph for multi-agent perception. Advances in Neural Information Processing Systems, 34, pp.29541-29552.

[2] Li, Y., Ma, D., An, Z., Wang, Z., Zhong, Y., Chen, S. and Feng, C., 2022. V2X-Sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving. IEEE Robotics and Automation Letters, 7(4), pp.10914-10921.

[3] Hu, Y., Fang, S., Lei, Z., Zhong, Y. and Chen, S., 2022. Where2comm: Communication-efficient collaborative perception via spatial confidence maps. Advances in neural information processing systems, 35, pp.4874-4886.

[4] Huang, S., Zhang, J., Li, Y. and Feng, C., 2024. Actformer: Scalable collaborative perception via active queries. ICRA 2024.

[5] Yang, D., Yang, K., Wang, Y., Liu, J., Xu, Z., Yin, R., Zhai, P. and Zhang, L., 2024. How2comm: Communication-efficient and collaboration-pragmatic multi-agent perception. Advances in Neural Information Processing Systems, 36.

[6] Su, S., Li, Y., He, S., Han, S., Feng, C., Ding, C. and Miao, F., 2023, May. Uncertainty quantification of collaborative detection for self-driving. In 2023 IEEE International Conference on Robotics and Automation (ICRA) (pp. 5588-5594). IEEE.

[7] Su, S., Han, S., Li, Y., Zhang, Z., Feng, C., Ding, C. and Miao, F., 2024. Collaborative multi-object tracking with conformal uncertainty propagation. IEEE Robotics and Automation Letters.

questions:
Could the authors illustrate task-agnostic collaborative perception more (especially the difference compared to the prior work [1])? As this prior work can be trained without knowing downstream tasks. However, the proposed framework in this paper seems to be trained on some specific tasks and is hard to generalize to novel downstream tasks. The authors are suggested to illustrate the limitations and setups clearly.  

[1] Yiming Li, Juexiao Zhang, Dekun Ma, Yue Wang, and Chen Feng. Multi-robot scene completion: Towards task-agnostic collaborative perception. In Conference on Robot Learning, pp. 2062–2072. PMLR, 2023c. 3, 5
</content>
</thread_object>

</thread>
<thread>
creation date:
1730764129820

thread_type:
official_review

<thread_object>
creator: reviewer_myqk

<content>
summary:
This paper presents a framework for collaborative perception that is argued to be salable, tasks-independent and model agnostic, which is also argued to be capable of dealing with heterogeneity in the agents and enhancing flexibility and security.

strengths:
+ Heterogeneity in collaborative agents, especially in robotics outside of collaborative driving, is an important problem.

+ The proposed method is straightforward, and the writing is easy to follow.

weaknesses:
- Novelty is a key concern. The proposed method is based on BEV representations and performs intermediate fusion using the BEVs. Each component also uses or marginally extends existing methods, and the novelty of each component is not well justified. For example, Collaborative Feature Alignment (CFA) simply uses the BEV from each agent and projects it to a unified feature space.

- The paper argues that the proposed approach can address heterogeneity in the agents; however, the solution is simply based on the BEV representation that is assumed to be computed by each agent.

- Similarly, the claim that the proposed method is model-agnostic is also an overstatement, primarily due to its reliance on the BEV representation.

- Why is the proposed method secure? The method uses a set of neural networks, so how can we ensure that the use of these networks is secure?

- What is the coordinate frame of the BEV representation used by each vehicle? If the BEV is ego-centric for each vehicle, how can the correspondence of street objects between agents be found?

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733105341012

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to Reviewers: Clarifying STAMP's Novelty and Addressed Concerns

comment:
## **Dear Reviewers,**

We sincerely thank you for your thoughtful and detailed feedback on our manuscript. We would like to address major concerns about the novelty of our work and summarize the revisions we made to address the reviewers' concerns.

> ### **Novelty and Contributions** 

Our work makes several novel contributions to collaborative perception (CP):

**First Heterogeneity Framework**: While previous works have addressed individual aspects of agent heterogeneity, STAMP is the first framework to **simultaneously handle all three dimensions**: input modalities, model architectures, and downstream tasks.

**Lightweight, Scalable Design**: Our adapter-reverter mechanism provides an efficient solution for feature alignment, requiring only ~1MB additional parameters per agent. This represents a **7.2x reduction** in computational overhead compared to existing methods [1], while maintaining or even improving performance. 

**Enhanced Security**: Our protocol framework inherently enhances security by eliminating the need to share agent details with other agents. The experiments show that STAMP maintains 98% performance under adversarial attack where end-to-end approaches degrade to 17% performance.

Regarding the novelty concern, we would like to emphasize that while the model architectures we used in this work are not the key novelty contribution, the heterogeneous collaborative framework design and the training and inference pipelines represent significant novel contributions. We believe this approach opens new possibilities for **scalable and secure heterogeneous collaborative perception system research**.

---

> ### **Our revisions**

**Protocol Model Selection**: In this revised version, we further clarify the architecture design of the protocol model. Our additional experiments with protocol models of different input modalities, encoder architectures, and training downstream tasks reveal that encoder architecture has minimal impact on framework performance, while input modalities and training objectives have relatively major effects. This discovery provides valuable guidance for future researchers in designing better protocol models.

**Adversarial Robustness**: We have included complementary experiments on the adversarial robustness of our proposed framework, demonstrating that STAMP enhances adversarial robustness by eliminating access to ego agent's information from other agents.

**Experimental Comparison with Existing Methods**: We conducted new experiments comparing STAMP with V2X-ViT, CoBEVT, and HMViT under heterogeneous input modality scenarios. The results show that STAMP not only supports all three types of agent heterogeneity (while other methods only support heterogeneous input modality) but also surpasses them in performance.

**Paper Writing Enhancement**: Following reviewer suggestions, we have: 1. Included additional highly related works, 2. Streamlined the main paper to focus on key concepts while moving detailed discussions to the appendix, and 3. Refined the main figure for a more concise and clear information presentation.

**Additional Performance Analysis**: Our training analysis comparing single-agent and end-to-end training reveals why STAMP outperforms end-to-end training strategies. We discovered that training multiple agents together in an end-to-end manner can cause overfitting or underfitting of some models, resulting in suboptimal checkpoint selection. This problem is effectively avoided by training each agent separately.

Since the PDF editing window closed before the completion of certain experimental results, we are committed to including these valuable complementary experiments and detailed analyses in the next version of our manuscript.

---

> ### **Future Research Opportunities**

**Investigating Protocol Model Selection**: Our experiments reveal that protocol model design significantly impacts framework performance, particularly regarding input modalities and training objectives. A promising direction is exploring input or task generic protocol models that potentially improve the framework's adaptability and robustness.

**Reducing Reliance on Collaborative Datasets**: STAMP currently requires each agent to be trained with collaborative datasets. Reducing this dependency could significantly improve training efficiency and cost-effectiveness.

**Enhancing Communication Efficiency**: While STAMP adapts to various BEV feature dimensions, incorporating more efficient communication strategies such as selective sharing warrants further investigation. We consider this a promising direction for future research.

We sincerely thank all reviewers for their valuable suggestions regarding complementary experiments and paper improvements, as well as for inspiring these promising future research directions.


Regards, 

Authors of Submission3152

---

> ### **Reference**

[1] Lu et al. (2024). An extensible framework for open heterogeneous collaborative perception. ICLR.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper addresses multi-agent collaborative perception, a setting where multiple agents exchange perception-related information to improve sensing capabilities. Specifically, this paper assumes heterogeneous agents (that may be equipped with different sensors or models performing different perception tasks) and follows a mid-level fusion approach that fosters the exchange of mid-level information exchange and fusion.  
The paper proposes BEV representation for feature exchange among agents and a shared communication protocol for feature exchange and fusion. Key findings (experiments on synthetic and real data) show the advantages of collaborative perception, with graceful degradation to baseline performance (single system). 

The paper received ratings of 3, 6, 6, 8, 8. Overall, reviewers are on the positive side (avg. rating 6.2). Two endorsements are strong, two positive, and one reviewer argues against acceptance. 

Reviewers agree that the problem is important and interesting (especially in the context of autonomous driving), the paper is very well written and structured, recognize that this paper is the first to study task-agnostic *and* model-agnostic collaborative perception, find that the effectiveness of the proposed contributions is clearly demonstrated in the context of AV data, comment that this work is an important foundation for future efforts in this domain, appreciate proposed systems efficiency and scalability, appreciate thorough ablation on the different architectural components, appreciate fantastic feature visualizations (before&after fusion). 

Reviewers provided constructive feedback and engaged in a discussion with authors, after which all provided a positive rating, with the exception of reviewer MyQk. A detailed summary of this discussion is below- the key takeaway is that the reviewer did justify their claims regarding the lack of novelty or responded to author clarifications (and ACs reminders to provide a response and final ratings). I agree with the reviewer's remarks that the "heterogeneous" claim made in the intro is misleading. However, I agree with the author's justification and urge the authors to clarify this in the revised version of this paper.

AC specifically appreciates well-structured coverage of related work that portrays a broad coverage of collaborative perception. This makes this paper's contribution accessible to an unfamiliar audience with multi-agent perception. Overall, the presentation clarity is exceptional. Discussion on limitations and failure cases strengthens this work further.