NegoCollab: A Common Representation Negotiation Approach for Heterogeneous Collaborative Perception
Authors: Shao Congzhang
Quan Yuan
Guiyang Luo
Yue Hu
Danni Wang
Liu Yilin
Rui Pan
Bo Chen
Jinglin Li
Venue: NeurIPS 2025 poster ()
Keywords: collaborative perception
autonomous driving
Primary area: applications

Abstract: Collaborative perception improves task performance by expanding the perception range through information sharing among agents. Immutable heterogeneity poses a significant challenge in collaborative perception, as participating agents may employ different and fixed perception models. This leads to domain gaps in the intermediate features shared among agents, consequently degrading collaborative performance.
Aligning the features of all agents to a common representation can eliminate domain gaps with low training cost. However, in existing methods, the common representation is designated as the representation of a specific agent, making it difficult for agents with significant domain discrepancies from this specific agent to achieve proper alignment.
This paper proposes NegoCollab, a heterogeneous collaboration method based on the negotiated common representation. It introduces a negotiator during training to derive the common representation from the local representations of each modality's agent, effectively reducing the inherent domain gap with the various local representations.
In NegoCollab, the mutual transformation of features between the local representation space and the common representation space is achieved by a pair of sender and receiver. To better align local representations to the common representation containing multimodal information, we introduce structural alignment loss and pragmatic alignment loss in addition to the distribution alignment loss to supervise the training. This enables the knowledge in the common representation to be fully distilled into the sender.
The experimental results demonstrate that NegoCollab significantly outperforms existing methods in common representation-based collaboration approaches. The mechanism of obtaining common representations through negotiation provides a more reliable and flexible option for common representations in heterogeneous collaborative perception.

NegoCollab: A Common Representation Negotiation Approach for Heterogeneous Collaborative Perception
Congzhang Shao!
Quan Yuan
Guiyang Luo Yue Hu Danni Wang!
Yilin Liu Rui Pan Bo Chen' Jinglin Li
State Key Laboratory of Networking and Switching Technology,
Beijing University of Posts and Telecommunications
Cooperative Medianet Innovation Center, Shanghai Jiaotong University
{shaocongzhang, yuanquan, 1uoguiyang edu. cn
Abstract
Collaborative perception improves task performance by expanding the perception range through information sharing among agents. Immutable heterogeneity poses a significant challenge in collaborative perception, as participating agents may employ different and fixed perception models. This leads to domain gaps in the intermediate features shared among agents, consequently degrading collaborative performance. Aligning the features of all agents common representation can eliminate domain gaps with low training cost. However, in existing methods, the common representation is designated as the representation of specific agent, mak- ing it difficult for agents with significant domain discrepancies from this specific agent achieve proper alignment. This paper proposes NegoCollab, hetero- geneous collaboration method based on the negotiated common representation. It introduces negotiator during training to derive the common representation from the local representations of each modality's 's agent, effectively reducing the inherent domain gap with the various local representations. In NegoCollab, the mutual transformation of features between the local representation space and the common representation space is achieved by pair of sender and receiver. To better align local representations to the common representation containing multimodal information, we introduce structural alignment loss and pragmatic alignment loss in addition to the distribution alignment loss to supervise the training. This enables the knowledge in the common representation to be fully distilled into the sender. The experimental results demonstrate that NegoCollab significantly outperforms existing methods in common representation- based collaboration approaches. The mechanism of obtaining common representations through negotiation provides a more reliable and flexible option for common representations heterogeneous collaborative perception.
Introduction
Collaborative perception has gained significant attention in recent years. By sharing intermediate features among agents, it expands the perception range and provides more supporting information for downstream tasks. In autonomous driving, collaborative perception enables vehicles detect obstacles in blind spots, thereby preventing traffic accidents and effectively enhancing driving safety, Heterogeneity is one of the key challenges practical applications of collaborative perception
"Correspondigg author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Xu et al. (2023b); Lu et al. (2024): Gao et al. (2025). When there are differences in sensors and perception models among collaborating agents, creates domain gaps in the shared intermediate features. This prevents effective fusion of features from heterogeneous agents and consequently degrades collaborative performance.
Current research on heterogeneity issues includes approaches that achieve heterogeneous collaboration by retraining specialized collaborative modules Xiang al. (2023) sharing partial networks in model Lu et al. (0024). However, in practical deployment, perception model are crucial for autonomous driving safety and tightly coupled with downstream tasks, making difficult to replace or retrain. These limitations lead to the challenge of immutable heterogeneous collaborative perception Xia et al.] (2024). To address this issue, methods like Xu al. (2023b); Luo et al. Xia et al. (2024) employ domain adapters or polymorphic prompts to eliminate domain gaps through
one-to-one adaptation for heterogeneous agents, as shown Figure Dh. requiring only single-step feature transformation but incurring higher training costs. A Alternatively, Gao et al. (2025) aligns the representations of each modality's s agent to a common representation by training a pair of adapter and reverter, which has low training cost. However, since the common representation is designated the representation of a specific agent, as shown in Figure Db. alignment becomes difficult to achieve when there exists large domain gap among the representations of other agents and the designated agent.
This paper presents NegoCollab, heterogeneous collaborative framework based on negotiated common representation. The framework introduces an additional negotiator during training to generate common representation from local representations of each modality's 's agent, as is shown in Figure D. supervised by a cyclic distribution consistency loss. This design minimizes information loss during bidirectional transformation between local representations and the common representation, effectively reducing inherent domain discrepancies between them. During collaboration, NegoCollab facilitates heterogeneous information exchange through pair of plug-and-play sender-receiver. The sender first maps features to the common representation space for sharing with collaborators, while the receiver subsequently projects the received features back to the local representation space, thereby eliminating domain gaps with collaborators' features. Furthermore, to better align local representations with the common representation containing multimodal information, structural alignment loss and pragmatic alignment loss are introduced in addition to the commonly used distribution alignment loss. These losses collectively form multi-dimensional alignment loss to supervise the training, enabling the knowledge in the common representation to be fully distilled into the sender.
The main contributions this work are summarized as follows:
Introducing negotiator to generate the common representation from local representations of each modality's agent, effectively reducing the alignment difficulty between the local representations and common representation while providing more diverse and reliable options for the common representations required in heterogencous collaborative perception.
A multi-dimennional alignment loss comprising distribution alignment loss, structural align- ment loss, and pragmatic alignment loss is introduced to supervise the training process, enabling more effective alignment of local representations to the multimodal common representation.
Experimental results on collaborative perception datasets demonstrates that NegoCollab achieves Mfte-of-the--ae ofe-ttr-rrt performance among common representation-bite methods, out- performing even one-to-one adaptation approaches in certain collaborative scenarios.
Related Work
2.1
Collaborative Perception
In recent years, collaborative perception has attracted widespread attention due to its potential to enhance autonomous driving safety. By sharing perception data among agents-including raw sensor data Rauch et al. 2012); Luo et al. (2023) ; Liu et al. (0024), intermediate features Wang et al. (2020); Li al. (2021) Chen et al. (2019) 9 at (2022). and detection results et al. (2023a)) Rawashdeh and Wang 2018). collaborative perception effectively expands the perception range and overcomes

---

(a) One-to-one Adaption
(b) Align to Common (Designated)
(c) Align to Common (Negotiated)
Figure 1: Two paradigms for eliminating domain gaps. The method (a) eliminates the domain gap by adapting domain adaptation modules between every pair of collaborating agents. The methods in (b) and (c) both eliminate domain gaps by unifying the representations of each agent into the common representation, where the common representation in (b) is designated the local representation of specific agent, and the common representation in (c) is negotiated from the local representations of each modality's agent.
blind spots and occlusion issues inherent in single-agent perception. However, in real-world scenarios, collaborative perception faces multiple challenges including: limited communication bandwidth (Hu et allll (2022 2023] 2024), location noise Lu et al. (2023); Lei et al. (2024). communication delay
and computation asynchronously Lei et (2022): Wei al. (2024a). communication interruptions
Ren et al. (002)), heterogeneity Xu et al. (2023b); Xiang et al. (2023 Lu et al. (2024); | Luo et al.)
Gao al. 2025); Xia et al. (2024), security and privacy concerns Li (2023) ) Zhao al. (2023),
and simulation-to-ret
generalization issues Kong et al. (2023) Wei et al. (2024b), all of which pose challenges collaboration. This paper focuses on the heterogeneity challenge collaborative perception, proposing negotiated common representation-batt approach to achieve common representation-bate based heterogeneous collaboration.
Multi-modal Representation Learning
Multi-modal representation learning Manzoor et al. (2023) enables information fusion and transfor- mation across different modalities (c.g., images, LiDAR point clouds, text, speech) by learning shared representation space. In autonomous driving, approaches like Zhang et al. (2025); Liu et al. (2023): Lu et al. (2024) employ network designs such sparse transformers and feature pyramids learn fused multi modllll representations from LiDAR point clouds and camera images, significantly enhancing vehicles' environmental perception capabilities. Knowledge distillation serves as com- mon method for cross modal knowledge transfer, approaches like Zhou et al. (2023): ) Wang et al. (2024): Chen et al. (022) apply various distillation losses, including dense distillation loss, relative relation distillation loss, and response distillation loss, between multi-modal features to achieve mutual enhancement of multi moltilmodal information, thereby improving task performance. This paper generates the common representation from the local representations of each modality using feature pyramid network, while introduces multi-dimensional alignment loss composed of distribution alignment loss, structural alignment loss, and pragmatic alignment loss during training to enable more effective alignment of local representations the multi-modal common representation.
Chenn
3 Method
3.1 Framework
NegoCollab achieves heterogeneous collaboration through the negotiated common representation. As shown in Figure by introducing plug-and-play sender-receiver pairs for each agent, the mutual conversion of features between the local representation space and the common representation space achieved, thereby eliminating domain gap. Let ") (-) denote the model used by the agent with modality m, where denotes the name of any module in the model, E 21, ..., M) and M is the total number of modalities (specific sensor and perception encoder constitute a modality). The structures of the sender and receiver, well the collaboration process, described below:

---

Visualized Features
Collaborators
Encoder
Encoder
Neb1
F
Camera
UDAR Point Cloud
Sender
Sender
Ego Feature
Rep. Space
P
Feature
@ Neb Festure in Ego Rep.
0 Neb Feature Common Rep.
Pis
Local
Sender
Receiver
Local Prompt
Local Rep. Space
Fitei
Detection
Encoder
Fusion
G
Ego
Local Feature
Fused Feature
LIDAR Point Cloud
Figure 2: Overview of NegoCollab. Each agent shares features in the negotiated common represen- lation space. Through the sender-receiver pairs, the features mutually converted between local representation space and the common representation space, thereby enabling the mutual transforma- tion of features across modalities and eliminating domain gaps.
3.1.1 Sender
The sender's role is to transform features from the local representation space to the common rep- resentation space, consisting of two modules: recombiner and aligner. The recombiner employs a ConvNeX et al. (2022) structure enhance local features beneficial for collaboration. It also includes a size-channel alignment module to adjust the dimensions and channels of local features to standard settings. The aligner a fused axial attention Xu et al. (2022) to capture both global and local dependencies within features, therehy mapping features from the local representation space to the common representation space.
During collaboration, for agent i with modality m in the scene, where N the total number of agents, its local observation O is first encodes by perception encoder (m(m) to extract initial feature eucoder (O2) Then the initial feature are transformed into the common representation
r((m) (mm)
space by the sender and shared with the collaborators, formalized process is follows:
(((m)
recombiner (F(())
m(m) =
s(m)
(m())
aligner
(2)
3.1.2 Receiver
The role of receiver is to transform the received features from collaborators from the common representation space back to the local representation space, consisting of two modules: converter and recombiner. The converter adopts fused axial attention transform features from the common representation space to the local representation space. The query vector Q in its input comes from the
recombiner (-) in sender, providing local modality guidance information for the transformation of collaborative features. The recombiner employs a ConvNeXt architecture to further reorganize and adjust local feature information, enabling adaptation to the local fusion module.
output N.!T! of the (m))
Let p(*) denote the features received from collaborator j A, with modality n, where Af, represents the set of collaborators for agent i. The formalized process of the receiver is as follows:
(m))
(m)
(R("))
p(n)
(3)
F (m)
R (m)
(T(m)
7-+1
Recombinee
(4)

---

first input the observation Finally, the transformed features F | from the collaborator and the local initial feature r2(m) are fused to obtain the fused feature = The fused feature is then processed the task head obtain the task result ( completing the process of collaborative perception. Formalized process is as follows:
G. H(m) (I:(T), _ (m)
(5)
D(m) head
((m))
(6)
3.2 Training
In the heterogeneous collaboration method based common representations, whether the domain converter can effectively achieve the mutual conversion of features between local representation space and the common representation space is of crucial importance to the collaboration performance. To address this, we introduce a negotiator that generates the common representation from each modality's s local representations, thereby reducing the inherent domain gap between the common representation and local representations and consequently decreasing the training difficulty for sender-receiver pairs. The training process consists of two stages: The objective of the first stage is negotiate common representations and to enable the sender-received to transform features from the local representation to and from the common representation. The training loss includes two components: cyclic distribution consistency loss and multi-dimensional alignment loss. The objective of the second stage to adapt the framework to downstream collaborative tasks. This is achieved by fine-tuning the receiver parameters using the collaborative task loss. Detailed training procedure is described below, diagram provided in the appendix.
3.2.1 Pairwise Local Representation Extraction
Since both the distribution cycle-consistent loss and multii-dimensional alignment loss require paired representations for computation, provide each modality's observation encoder with observation data from the same perspective during training. Let = [0,, Oz..., ONl denote the observation data from all N perspectives in the scene. At the start of training, data O into each modality's perception encoder to obtain the initial local representations for each modality. Then, we use resizer align the sizes and channels of these representations to the standard configuration. The formalized process is as follows:
((m) (m))
encoder (0),
(7)
(m)) (m) (F(m)
(8)
3.2.2 Generates Common Representation by Negotiator
After obtaining the standardized local representations (mm)), we use the negotiator to generate the common representation from each modality's 's local representations. The main structure of the negotiator is a feature pyramid network, where each level contains an estimator to evaluate the contribution of each modality's representation to the common representation that level, detailed illustrations is in appendix. Specifically, pyramid network is first used to extract multi-level features (!("") from (1m), and the corresponding estimators each level is used evaluate their contribution weights to the common representation, producing an importance matrix (,()) Next, at each level, the (:(m) and (i(m) from modalities are multiplied and then averaged to obtain the common representations P, for that level. Subsequently, all Pt concatenated after alignment through upsampling. Afterward, their sizes and channels are restored to standard settings via a shrink header,
yielding the common representation P. Let the input (m) level 0 of the pyramid be (m)) The
5

---

formalized process of the negotiator is as follows:
v(m)_ = =Mayer,
(U(
1= 1,2, I,
(9)
c(m)
Nestimaton
(U(T))
1 1,,,, 1,,,... L.
(10)
Pt sum {ufm (m)m) cc
c((m) {
m= /M,
(11)
(lu (PN)I=o)
P contact
(12)
P = NNturiinn header (P),
(13)
where denotes the pyramid level, m represents the modality of the representation, @ Hadamard product, and uq (.) stands for the upsampling operation.
indicates the
Next, the common representation P is fed into each modality 's receiver and transformed back to the local representation [m);
(Tm) -R(me (m) coaverter ((m).)))
(14)
L (m) R (m) recombiner
(T(m)
(15)
At this stage, the eyclic distribution consistency loss be computed follows:
(m) eycle (mm) (m) - B Std (F(m)) Std [L(ml
(16)
Through the constraint of cyclic distribution consistency loss, the information loss during mutual transformation between the common representation and local representations minimized, thereby effectively reducing the inherent domain gap between them.
3.2.3 Multi-dimeensinal Information Alignment
We impose a multi-dimensional alignment loss constraint between the common representation output by senders and the negotiator. This constraint consists of three components: distribution consistency loss, structural alignment loss, and pragmatic alignment loss. Its purpose to fully distill the representational information from the multimodal common representations into the sender, thereby facilitating the transformation from local representations to the common representation. The formulation process is follows:
First, we use the sender to transform the local representations F(m) into common representation:
(m)
((m) (m)
(F(m)
(17)
recombiner
p(m) aligner
(m)
(((m)
(18)
Next, we compute the multi diltiidimmnsionall alignment loss between common representations p(m) output by senders and the common representation /' output by the negotiator. This loss enforces distribution consistency, structural consistency, and pragmatic consistency between p(m) and P. Here, distribution consistency ensures that the statistical characteristics This achieved by applying distribution alignment loss that constrains p(m) and P to have identical means and standard deviations, computed as follows:
representations match.
(m)) uni dis = p(m) - pl. + a ||sddd (psmd(pm) (t)
(19)
Structural consistency ensures that the spatial relationships between scene components remain coherent across representations. This is achieved by enforcing consistent relative relationships between different parts of samples. Specificallyy, for each sample 8, where & E 21,,..., S} and S is the total number of samples, we consider the interrelationships among key points {(x; 1)) Features of keypoints collected from samples sampled from the common representations p(m)

---

and P. and the relative relation matrix of sample is obtained by calculate the similarity between keypoints:
plm) Mij
=C =C(p/m)) (p(m) (I (c). p!!)) (z).
(20)
M = (P, Ui), P. (Iz, Uj)),
(21)
where & ú & 9, and ....) denotes the cosine similarity between elements. The relative relationship matrices of all sample pairs p(m) and P are made consistent to achieve structural consistency. The structural alignment loss is calculated follows:
s
(m)) uni
Σ
plm) M
M, /81.
(22)
stru
S=
1E
Pragmatic consistency refers to the consistent organization of foreground information in the represen- tation space. It achieved by training a shared 2D occupancy prediction network for the common representations p(m) and P. which aligns the organization foreground information through reverse alignment. Let (-)) denote the shared 2D occupancy prediction network, and Y be the 2D occupancy labels corresponding to observation data O. The pragmatic alignment losses for p(m) and P are computed as follows, respectively:
((m)
uni- "uni-pragma f.coal (N (p(m)) v).
(23)
(P)
= f (N(P),
(24)
pragma
where focal is the focal loss Lin al. 2017))
Then, the milei-dimmensoonl alignment loss of modality m is obtained by summing the distribution consistency loss, the structural consistency loss, and the pragmatic consistency loss:
((m) uni
A + (m mii + -
(25)
Finally, the first-stage training loss is calculated as a weighted sum of the distribution cycle-consistent losses, the multi-dimensional alignment losses from all modalities, and the pragmatic alignment loss of the common representation P:
M
Lstagel = da L())
d (m) eyate + (m))
(26)
pragma
3.2.4 Task Adaption
To enable the receiver focus restoring information beneficial for collaboration, fine-tune the receivers of each modality using the downstream collaborative task loss for the second stage of training. During this process, the data loading method and feature flow identical those during inference (Section 3.I. the parameters of the senders are fixed, and the loss is calculated as follows:
Loottab (D.m), Yi).
Estage2
(27)
Here, Lestiab is the collaborative task loss, (m)) is derived from Equation 6 and represents the task prediction output by the collaborative model, while Y, denotes the task label for agent
Experiment
4.1 Settings
We configure four collaborating agents ml, m2, m3, m4 and one protocol agent in the scenario. Among them, the protocol agent, ml, and m3 are equipped with LiDAR sensors, while m2 and m4
7

---

are equipped with cameras. The perception encoders used by mi and m3, well as those used by m2 and m4, are different. Detailed configurations are provided in the Appendix.
To evaluate the performance of the common representation and its generalization capability to new agents, form an initial collaborative alliance between agent m1 and agent m2, from which the common representation are negotiated. Agents m3 and m4 are newly added agents that align their features with the common representation. The training process consists of three stages:
Step 0: Homogeneous collaborative training. For each of the 4 agent types, train a homogeneous collaborative perception model.
Step 1: Initial alliance negotiation. Following the method Section 3.2 the training is conducted in two stages. In the first stage, sender-receiver pairs are introduced to ml and m2, respectively. A common representation obtained through training assisted by the negotiator to complete the training of sender-receiver pairs. In the second stage, the parameters of the receivers for ml and m2 are adjusted to adapt to the downstream collaborative task. During the training process, the parameters of the perception encoder, fusion module, and task head in the homogeneous collaborative perception model for ml and m2 are frozen.
Step 2:
New agent joins. The training when new agents m3 and m4 join is also divided into two stages. The loss calculation in the first stage is the same as in Section 3.2 but the common representation is obtained directly from the perception encoders of ml and m2 and the negotiator. The collaborative task loss in the second stage is calculated as the collaborative task loss of the new agents and the existing agents in the alliance. During the training process, the parameters of the negotiator, the perception encoders of ml and m2, and the parameters of the homogeneous collaborative model for m3 and m4 are frozen. Specific illustration is provided in the appendix.
Table 1: Performance comparison of heterogeneous collaboration OPV2V-H. "NegoCollab-P", "MPDA-P" and "PnPDA-P" after added "-P" are special implementations of the corresponding methods, which feature sharing is achieved by using the representation of the protocol agent as the common representation.
Metric
A.@
AP@0.7
Agent Types
mlm2
m1m3
m2m4
All
m1m2
mlm3
m2m4
All
No Fusion
0.482
0.794
0.221
0,480
0.350
0.687
0.106
0.342
One-to-one
MPDA
0.815
0.922
0.520
0.512
0.692
0.850
0.331
0.435
Adaptation
PnPDA
0.865
0.949
0.532
0.494
0.755
0.903
0.351
0.424
MPDA-P
0.561
0.811
0.354
0.465
0.409
0.697
0.173
0.353
PnPDA-P
0.552
0.875
0.365
0.434
0.447
0.805
0.216
0.346
Align to Common
STAMP
0.545
0.770
0.264
0.382
0.448
0.708
0.134
0.286
NegoCollab-P
0.792
0.772
0.499
0.676
0.615
0.710
0.289
0.457
NegoCollab
0.872
0.911
0.512
0.745
0.765
0.854
0.319
0.555
4.2
Quantitative Analysis
Performance of heterogeneous collaboration. We evaluated each method on the OPV2V-H Lu et al. (2024). V2V4Real et al. (2023c), and DAIR-V2X4 Yu al. (2022) datasets, as shown in Table and 2 Since the common representation of MPDA-P, PnPDA-P, and STAMP are all derived from the single-modality protocol agent, for fair comparison, we implement NegoCollab-P, which derives the common representation from the protocol agent. In Table [ the columns ml2, m1m3, m2m4, and mlm2m3m4 correspond the performance of: initial alliance agents, heterogeneous LiDAR agents, heterogeneous camera agents, and all agent types collaborative, respectively. The results demonstrate that among heterogeneous collaboration methods based on common representation, NegoCollab achieves the best performance in all test conditions. Compared with one-to-one adaptation methods, NegoCollab also maintains optimal collaborative performance when agents ml and m2 within the initial alliance collaborated. For collaboration with new agents m3 and m4, although m3 and

---

Table Performance comparison of heterogeneous collaboration on real- datasets 42V4Real and DAIR-V22 V2X, with collaborating agents being ml and m3, ml and m2 respectively.
V2V4Real
DAIR-V2X
Methods
AP@0.5
AP@0.7
AP@0.5
AP@0.7
No Fusion
0.504
0.358
0.329
0.219
One Adaption
MPDA
0.613
0.400
0.344
0.235
PnPDA
0.598
0.385
0.443
0.277
MPDA-P
0.467
0.334
0.258
0.211
PnPDA-P
0.485
0.324
0.230
0.192
Align to Common
STAMP
0.466
0.345
0.299
0.161
NegoCollab-P
0.482
0.333
0.376
0.195
NegoCollab
0.605
0.397
0.397
0.241
0.9
0.9
MIDA
NegoCollab Baseline
KL Divergence
0.7
0.7
AP@0.5
AP@0.7
0.55
0.5
NegoColab-
STAMP NegoCatte
0.3
PAPOA
0.3
METUA-P
PHPOAP
0.1
0.0
0.2
0.4
0.1 0.0
0.4 Pose Noise Std
0.2
m1
m12
m.3
m4
Noise Std
Agent
Type
Figure 3: Robustness Analysis of Localization Errors. Pose noise is set to N (0, on both x.y location and yaw angle. The collaborating agents are m1 and m2.
Figure 4: Comparison of domain gaps between local and common representation.
m4 did participate in the negotiation process of the common representation, their collaborative performance is slightly lower than that of one-to-one adaptation methods, but still achieves competitive results. This strongly demonstrates NegoCollab's 's superior performance and the excellent adaptability of the common representation to new agents. Additionally, the results in Table show that NegoCollab also has excellent heterogeneous collaboration performance in real-world environments.
Table 3: Comparison of homogeneous collaboration performance when sharing features in the common representation space. "Local" denotes direct feature sharing through local representation spaces. Evaluation conducted the OPV2V-H dataset.
Metric
0.5
AP@0.7
Agent Type
ml
m2
m3
m4
ml
m2
m3
m4
Local
0.952
0.540
0.930
0.497
0.919
0.378
0.886
0.322
MPDA-P
0.837
0.515
0.804
0.439
0.712
0.305
0.684
0.230
PnPDA-P
0.950
0.545
0.926
0.499
0.910
0.362
0.883
0.309
STAMP
0.945
0.555
0.925
0.497
0.892
0.373
0.868
0.304
NegoCollab-P
0.951
0.566
0.932
0.513
0.916
0.378
0.881
0.317
NegoCollab
0.953
0.570
0.933
0.521
0.911
0.385
0.888
0.317
Performance of homogeneous collaboration. Table presents the homogeneous collaboration performance of different methods when using the common representation share feature. As shown, NegoCollab achieves the best performance among all methods. For agents ml, m3, and m4, it even surpasses the original homogeneous collaboration performance. This improvement stems from the multi-dimensional alignment loss distilling multi-modal knowledge from common representation into local senders, thereby enhancing the feature's 's representational capacity.
Comparison of domain gaps. validate the effectiveness of the negotiator reducing domain gaps, employ KL divergence Kullback and Leibler [951] measure the domain gap between common

---

representation and local representations of each modality across different methods. Comparision illustrated in Figure Since MPDA-P, PnPDA-P, and STAMP all use the representation of the protocol agent as the common representation, they are aggregated as the "Baseline' in the figure. It can be seen that the domain gap between the common representation generated by the negotiator and each local representation is significantly reduced. Compared to the method of directly designating the representation the protocol agent as the common representation, the domain gap measured by KL divergence is reduced by average of approximately 93.5
Localization error robustness. We introduced Gaussian noise the accurate poses to evaluate the noise robustness of each method, as shown in Figure 3 The results demonstrate that under various error conditions, NegoCollab maintained superior performance on the A.5 evaluation metric.
4.3 Ablation Study
Negotiating from different initial alliances. In practical applications, heterogeneous agents form multiple collaborative groups based on collaboration needs Gao et al. (2025), using different com- representations for information sharing within each group. NegoCollab's 's negotiation- baaed mechanism enables the free selection agents from a collaborative group to negotiate the common representation, thereby providing more diverse and reliable common representation. To further explore how to negotiate a better common representation, we investigate the impact of common representations negotiated from different initial alliances on collaborative performance. Two key ob- servations are summarized. with detailed content and experimental results provided in the Appendix.
Training Setting Ablation. We conducted ab- lation studies on the negotiator and the multi- dimensional alignment loss within the training setup on the OPV2V-H dataset. The results be- fore adaption for the downstream collaborative task are presented in Table [ Under the initial setup, the multi-dimensional alignment loss in- cludes only the distribution alignment loss, with- out assistance from the negotiator during train- ing. The common representation is obtained by directly constraining the outputs of each modal- ity's senders to be consistent through the align- ment loss. comparison between the upper and lower sections of the table demonstrates that negotiate common representation by the nego- tiator effectively enhanced the performance in heterogeneous collaboration. The performance improvements observed in the "uni-stru" and "uni-pragma" columns indicate that the structural and pragmatic alignment losses effectively facilitated the transformation local representations into the common representation.
Table 4: Ablation study the traning setting. The collaborating agents are ml and m2.
uni- stru
uni- pragma
Nego
AP@0.5
AP@0.7
0.617
0.490
0.609
0.485
0.627
0.499
/
0.635
0.508
0.609
0.496
d
0.655
0.532
/
0.671
0.538
0.711
0.566
5 Conclusion
This paper proposes NegoCollab, a heterogeneous collaboration method based on negotiating com- mon representation. NegoCollab uses a negotiator generate the common representation from the local representations of each modality's agent, effectively reducing the domain gap between the common representation and the local representations. Furthermore, by introducing a multi- dimensional alignment loss, it effectively promotes better alignment of the local representations to the multi-modal common representation. Evaluation results from both simulated and real-world environments collectively demonstrate the outstanding heterogeneous collaboration performance of NegoCollab. limitation NegoCollab that once common representation is negotiated, becomes fixed. Aligning new agents to this pre-negotiated common representation inevitably leads to greater information loss. We will explore methods make the common representation generalize better to agents in the future.
10

---

6 Acknowledgement
This work was supported in part by the National Key Research and Development Program of China under Grant 2023 0023YF4430190 1900, in part by the Natural Science Foundation of China under Grant 62272053 and Grant 62472048, in part by the Beijing Nova Program under Grant 20230484364, and in part by Beijing Natural Science Foundation under Grant L242081.
References
Qi Chen, Xu Ma, Sihai Tang, Jingda Guo, Qing Yang, and Song Fu. F-cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3d point clouds. Proceedings of the 4th ACM/IEEE Symposium on Edge Computing, pages 88-100, 2019.
Zehui Chen, Zhenyu Li, Shiquan Zhang. Liangji Fang, Qinhong Jiang, and Feng Zhao. Bevdistill: Cross-modal bev distillation for multi-view 3d object detection. arXiv preprint arXiv:221.009366 2022.
Xiangho Gao, Runsheng Xu, Jiachen Ii, Ziran Wang, Zhiwen Fan, and Zhengzhong Tu. Stamp: Scalable task and model-agnostic collaborative perception. arXiv preprint arXiv:2501.18616 2025.
Yue Hu, Shaoheng Fang, Zixing Lei. Yiqi Zhong, and Siheng Chen. Where2comm: Communication-effcieeet collaborative perception via spatial confidence maps. Advances in neural information processing systems, 35: 2022.
Yue Hu, Yifan Lu, Runsheng Xu, Weidi Xie, Siheng Chen, and Yanfeng Wang. Collaboration helps camera overtake lidar in 3d detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9243-9252, 2023.
Yuc Hu, Juntong Peng. Sifci Liu, Junhao Gc. Si Liu, and Siheng Chen, Communication-effiienn collaborative perception via information filling with codebook. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15481-15490, 2024.
Xianghao Kong, Wentao Jiang, Jinrang Jia, Yifeng Shi, Runsheng Xu, and Li Dusa: Decoupled unsupervised sim2real adaptation for cehicle-to-everything collaborative perception. In Proceedings of the 3 ACM International Conference on Multimedia, pages 1943-1954, 2023.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-86, 1951.
Zixing Lei. Shunli Ren, Yue Hu, Wenjun Zhang, and Siheng Chen. Latency-aware collaborative perception. In European Conference on Computer Vision, pages 316-332. Springer, 2022.
Zixing Lei, Zhenyang Ni, Ruize Han, Shuo Tang, Chen Feng, Siheng Chen, Yanfeng Wang. Robust collaborative pereeption without external localization and clock devices. arXiv preprint arXiv:2405.02965 2024.
Yiming Li. Shunli Ren, Pengxiang Wu, Siheng Chen, Chen Feng, and Wenjun Zhang. Learning distilled collaboration graph for multi-agent perception. Advances in Neural Information Processing Systems, 34: 29541-29552 2021.
Yiming Li, Qi Fang. Jiamu Bai, Siheng Chen, Felix Juefei-Xuu and Chen Feng. Among us: Adversarially robust collaborative perception by consensus. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 186-195, 2023.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár, Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pages 2980-2988, 2017.
Si Liu, Zihan Ding, Jiahui Fu, Hongyu Li, Siheng Chen, Shifeng Zhang, and Xu Zhou. V2x-pe: Vehicle-to- everything collaborative perception via point cluster. arXiv preprint arXiv:2403. 16635, 2024.
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11976-11986, 2022.
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang. Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task sensor fusion with unified bird's-ey view representation. In 2023 IEEE international conference robotics and automation (ICRA), pages 4774-278) IEEE, 2023.
11

---

Yifan Lu, Quanhao Li, Baoan Liu, Mehrdad Dianati. Chen Feng. Siheng Chen, and Yanfeng Wang. Robust collaborative 3d object detection in presence of pose errors. In 2023 IEEE International Conference on Robotics and Automation (ICRA). pages 4812-4818. IEEE, 2023.
Yifan Lu, Yue Hu, Yiqi Zhong, Dequan Wang, Yanfeng Wang, and Siheng Chen. An extensible framework for open heterogeneous collaborative perception. arXiv preprint 1rXiv:2401.13964, 2024.
Guiyang Luo, Chongzhang Shao, Nan Cheng, Haibo Zhou, Hui Zhang, Quan Yuan, and Jinglin Li. Edgecooper: Network-aware cooperative lidar perception for enhanced vehicular awareness. IEEE Journal on Selected Areas in Communications, 2023.
Tianyou Luo, Quan Yuan, Guiyang Luo, Yuchen Xia, Yujia Yang, and Jinglin Li. Plug and play: A representation enhanced domain adapter for collaborative perception. re-scarch.
Muhammad Arslan Manzoor, Sarah Albarri, Ziting Xian, Zaiqiao Meng, Preslav Nakov, and Shangsong Liang. Multimodality representation learning: A survey on evolution, pretraining and its applications. ACM Trunsactions on Multimedia Computing, Communications and Applications. 20(3:1134. 2023.
Andreas Rauch, Felix Klanner, Ralph Rasshofer, and Klaus Dietmayer. Car2x-based perception in high-level fusion architecture for cooperative perception systems. In 2012 IEEE Intelligent Vehicles Symposium, pages 270-275. IEFF, 2012.
Zaydoun Yahya Rawashdeh and Zheng Wang. Collaborative automated driving: A machine learning-bas based method enhance the accuracy shared information. 2018 2 International Conference on Intelligent Transportation Systems (ITSC), pages 3961-3966, 2018.
Shuali Ren, Zixing Lei, Wang, Mehrdad Dianati, Yafei Wang. Siheng Chen, and Wenjun Zhang. Interruption- aware cooperative perception for v2x communication-aa Intelligent Vehicles, 2024.
autonomous driving. IEEE Transactions on
Sijie Wang, Rui She, Qiyu Kang, Xingchao Jian, Kai Zhao, Yang Song, and Wee Peng Tay. Distilypr: Cross- modal knowledge distillation for visual place recognition. In Proceedings of the AAAI conference on artificial intelligence, pages 10377-10385, 2024.
Tsun- Hsuan Wang. Sivabalan Manivasagam, Ming Liang, Bin Yang. Wenyuan Zeng, and Raquel Urtasun. V2vnet: Vehicle-to-veihe communication for joint perception and prediction. In Computer Vision-ECCV ECCV 2020: 16th Europeon Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 605-621. Springer, 2020.
Sizhe Wei, Yuxi Wei, Yue Hu, Yifan Lu, Yigi Zhong, Siheng Chen, and Ya Zhang. Asynchrony-robust collaborative perception via bird's eye view flow. Advances Neural Information Processing Systems, 36, 2024a.
Yuxi Wei, Zi Wang. Yifan Lu, Chenxin Xu, Changxing Liu, Hao Zhao, Siheng Chen, and Yanfeng Wang. Editable scene simulation for autonomous driving via collaborative Ilm-agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15077-15087, 2024b.
Yuchen Xia, Quan Yuan, Guiyang Luo, Xiaoyuan Fu, Yang Li, Xuanhan Zhu, Tianyou Luo, Siheng Chen, and Jinglin Li. One plenty: A polymorphic feature interpreter for immutable heterogeneous collaborative perception. arXiv preprint xxXv:2411.1177 2024.
Hao Xiang, Runsheng Xu, and Jiaqi Ma. Hm-vit: detero-modll schicle-to-vehice cooperative perception with vision transformer. arXiv preprint 2334 10628, 2023.
Runsheng Xu, Zhengzhong Tu, Hao Xiang, Wei Shao, Bolei Zhou, and Jiaqi Ma. Cobevt: Cooperative bird's eye view semantic segmentation with sparse transformers. arXiv preprint arXiv:2207.0223 02202,, 2022.
Runsheng Xu. Weizhe Chen, Hao Xiang, Xin Xia, Lantao Liu, and Jiaqi Ma. Model-agnostic multi-agent perception framework. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1471-1478. IEEE, 2023a.
Runsheng Xu, Jinlong Li, Xiaoyu Dong, Hongkai Yu, and Jiaqi Ma. Bridging the domain gap for multi-agent perception. In 2023 IEEF International Conference on Robotics and Automation (ICRA), R,), pages 6035-6042. IEFF, 2023b.
Runsheng Xu, Xin Xia, Jinlong Li, Hanzhao Li, Shuo Zhang, Zhengzhong Tu, Zonglin Meng, Hao Xiang. Xiaoyu Dong. Rui Song, et al. V2v4real: A world large dataset for vehicle-to- ve-hillle cooperative perception. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13712-13722. 2023c.
12

---

Lucas D Young, Fitsum A Reda, Rakesh Ranjan, Jon Morton, Jun Hu, Yazhu Ling, Xiaoyu Xiang, David Liu, and Vikas Chandra. Feature-align network with knowledge distillation for efficient demoising. In Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision, pages 709-718, 2022.
Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang. Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, al. D2i:-v2x: A large-scale dataset for vehicle-ifrrastructure cooperative 3d object detection. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21361-21370, 2022.
Hongcheng Zhang. Liu Liang. Pengxin Zeng, Xiao Song, and Zhe Wang. Sparselif: tieh-peeffrrance sparse lidar-camera fusion for 3d object detection. In European Conference on Computer Vision, pages 109- 128. Springer. 2025.
Yangheng Zhao, Zhen Xiang, Sheng Yin, Xianghe Pang, Siheng Chen, and Yanfeng Wang. Malicious agent detection for robust multi-agent collaborative perception. arXiv preprint arXiv:2310. /990,, 2023.
Shengchao Zhou, Weizhou Liu, Chen Hu, Shuchang Zhou, and Chao Ma. Unidistill: A universal cross- roos-mod knowledge distillation framework for 3d object detection in bird's-cye view. In Proceedings of the IEFFICVF conference computer vision and pattern recognition, pages 5116-5125, 2023.
13

---

A Detailed Setup of Experiment
A.1 Dataset
OPV2V-H. OPV2V-H Lu et al. (2024) dataset contains 73 scenes covering 6 road types across 9 cities. Each Connected Autonomous Vehicle(CAV) in the scenes equipped with one 16-channel, one 32-channel, and one 64-channel LiDAR, along with 4 monocular cameras and 4 depth cameras. The dataset comprises 36K frames LiDAR point clouds, 12K frames of RGB camera images, 12K frames of depth camera images, and 230K annotated 3D bounding boxes.
DAIR-V2.. DAIR- DAXR-V2X] Yu et al. (2022) is real-world collaborative perception dataset. The dataset has 9K frames featuring one vehicle and one roadside umit (RSU), both equipped with a LiDAR and 1920x camera. RSU LiDAR 300-channel while the vehicle's is 40-channel.
V2V4Real. 42V4Real Xu et al. (023c) is a real-world Vehicle-to soocle-to-Vehicle Vehicle (V2V) cooperative perception dataset. The dataset inc muluude 20,000 LiDAR scans and 240,000 annotated 3D bounding boxes across five vehicle classes. It supports benchmarks for three key task: 3D object detection, object tracking, and Rem2Real domain adaptation-enabling evaluation with state-of-the-art models.
A.2 Training Setup
We conducted testing and training using a single RTX 4090 GPU, with an initial learning rate of 0.001 and Adam optimizer for parameter adjustment. The first training phase required approximately 4-12 GPU hours with about 23GB memory usage, while the second phase took around 2-5 GPU hours consuming approximately 14GB memory. The exact values depend the specific agent model architecture.
A.3 Detailed Configuration of Agents
Section 4.1 mentions types of agents ml, m2, m3, and m4, as well as protocol agents. The detailed configurations of their sensors and perception encoders are shown in Table
Table Settings for sensors and perception encoders of agents.
Agent Type
Sensor
Perception Encoder
Protocol
LiDAR of 64-channel
PointPillars
m1
LiDAR of 64-channel
PointPillars
m2
Camera, resize img. to height 384 px
Lift-Splat w.
EfficientNet as img. encoder
m3
LiDAR of 32-channel
SECOND
m4
Camera, resize img. to height 336 px
Lift-Splat w. ResNet50 as img. encoder
B More Experiments
B.1 Negotiating from Different Initial Alliances
We investigate the impact of negotiating common representation from different initial alliances on collaborative performance, as shown in Table It can be observed that in the heterogencous collaboration scenario, for common representations negotiated from different initial alliances, when the participating agents are consistent with those their initial alliance, the optimal performance is achieved in the corresponding collaboration scenario. In homogeneous collaboration, compared to directly sharing features using local representations, sharing features using different common representations results in nearly unchanged collaboration performance for agents ml and m3, and even better performance for agents m2 and m4. This is because the dinli--dmensional alignment loss effectively distills multimodal knowledge from the common representation into the local senders and receivers, thereby enhancing the performance of the representations.
Furthermore, we derive two key observations from the results in Table 6b
14

---

Table Performance comparison when negotiating common representations from different initial alliances. The "Imitial Alliance" column indicates the agents in the initial alliance, while the remaining agents are new agents. The training process the same as that described in Section 4.1
(a) Performance of heterogeneous collaboration
Initial
A.@0.5
AP@0.7
Alliance
m mlm2
m3m4
mlm3
m2m4
All
mlm2
m3m4
mlm3
m2m4
All
Protocol
0.792
0.785
0.772
0.499
0.676
0.615
0,564
0.710
0.289
0.457
m1m3
0.869
0.832
0.951
0.484
0.830
0.761
0.720
0.904
0.280
0.718
mlm2
0.872
0.770
0.911
0.512
0.745
0.759
0.578
0,805
0.319
0.555
m3m4
0.727
0,840
0.914
0.506
0.737
0.550
0.726
0.840
0.289
0.562
(b) Performance of homogeneous collaboration
Initial
A.@0.5
AP@0.7
Alliance
m2
m3
m4
m1
m2
m3
m4
Local
0.952
0.540
0.930
0.497
0.919
0.378
0.886
0.322
Protocol
0.951
0.566
0.932
0.513
0.916
0.378
0.881
0.317
mlm3
0.953
0.568
0.932
0.512
0.913
0.378
0.882
0.315
mlm2
0.953
0.570
0.933
0.521
0.911
0.385
0.888
0.317
m3m4
0.953
0.575
0.932
0.511
0.914
0.391
0.883
0.313
Common representations negotiated from more types of agents demonstrate superior per- formance. As shown rows 1 ("Protocol") and ("mlm2") of Table [6b] compared to representation negotiated solely from LiDAR-equipped protocol agent, those obtained from the initial alliance comprising both LiDAR-equipper agent m1 camera-equipped agent m2 achieve better performance in m1m2, mlm3, m2m4, and all types of agent collaboration scenarios.
Common representations negotiated from agents with superior perception encoder perfor- mance yield better results. As evidenced by rows ("m3m4") and 2 ("m ("mlm3") in Table 6b]] representations negotiated from agents ml and m2 which have better perception perfor- mance when using identical sensors demonstrate stronger generalization to new agents m3 and m4. Conversely, representations derived from agents m3 and m4 with inferior perception exhibit degraded performance when collaborating with new agents m1 and m2. Therefore, when sensors identical, agents with better-performing perception encoders should be prioritized to form the initial alliance.
B.2 Comparison with Late Fusion
We further contrast the performance of NegoCollab with late fusion, shown in Table Late fusion generally performs better when there is no localization error in different scenario. This is because, compared intermediate fusion, late fusion directly merges detection results, which can mitigate the impact of model heterogeneity on collaboration. As the localization error increases, the performance of late fusion declines significantly. contrast, NegoCollab-P and NegoCollab, based on the intermediate fusion, demonstrate greater robustness and achieve performance substantially superior to late fusion. This because feature-level fusion combines the features from collaborative agents based on semantic similarity, which can mitigate the impact of locaization enror to some extent. Since localization errors are almost unavoidable practical scenarios, the more robust NegoCollab exhibits stronger practicality.
15

---

Table 7: Performance Comparison with Late Fusion under different localization error conditions. The agent positions are perturbed with Gaussian noise, where d represents the standard deviation of the Gaussian noise. The "vg. Inc." column corresponds to the increase in the average evaluation results of NegoCollab and NegoCollab-P across various collaborative scenarios under different noise conditions, compared to late fusion.
AP@0.5
AP@0.7
Avg.
Avg.
mlm2
mlm2
Agent Types
mlm2
m1m3
m2m4
Inc.
mlm2
m1m3
m2m4
Inc.
m3m4
m3m4
Late Fusion
0.873
0.952
0.482
0.854
0.743
0.893
0.290
0.725
0.0
NegoCollab-P 0.792
0.772
0.499
0.676
-133%
0.615
0.710
0.289
0.457
-2.9%
NegoCollab 0.872
0.911
0,512
0.745
-3.8%
0.765
0.854
0.319
0.555
-0.06%
Late Fusion 0.564
0.626
0.299
0.543
0.201
0.271
0.077
0.197
0.3
NegoCollab-P 0.676
0.711
0.391
0.591
+ +16.6%
0.403
0.527
0.149
0.388
+96.6%
NegoCollab 0.719
0.837
0.387
0.616
+25.9%
0.425
0.582
0.146
0.365
+ +103.4%
Late Fusion 0.278
0.328
0.154
0.264
0.115
0.169
0.043
0.106
0.6
NegoCollab-P 0.477
0.574
0.256
0.500
+79.5%
0.283
0.397
0.099
0.353
+161.1%
NegoCollab
0.483
0.693
0.229
0.462
+82.3%
0.276
0.427
0.086
0.292
+149.7%
B.3
Component Ablation
We conducted ablation experiments on the recombiner and aligner in the sender, the negotiator, and the local prompt on OPV2V-H, as shown in Table It can be seen that NegoCollab achieves optimal performance when the recombiner and aligner are set to Convext and FAX(fused axial attention), respectively. This is because we divide the feature transformation process into two steps: adjusts local detail information, and transforms global representation style. The characteristics of Convext and FAX are respectively more suitable for local information adjustment and representation style transformation. For the Negotiator, the FPN structure adopted in this paper achieves the best performance with the smallest parameter count, indicating that the FPN structure can better extract common representation from each modality's 's local representation. After using Local Prompt to guide the transformation from the common representation to local representation, the performance significantly improved. The above results fully demonstrate the rationality of the component design in NegoCollab.
C
Additional Illustrations
C.1 Training Process of Initial Alliance Negotiation
Figure SigurrBiillustees iillustrases the first-stage training process when the initial alliance negotiates the common representation as described in Section 3.2 The specific steps are follows:
The perception encoder of each modalitiy's 's agent is fed with observational data from the same perspective, encoding them into paired initial local representations ((m).
The local representations F(m) from each modality's 's agent are input into the negotiator for fusion, producing common representation P.
The common representation P is fed into the receiver of each modality's agent to obtain the restored local representation (m))
The initial local representation ((m) of each modality's 's agent input into its respective sender to yield common representation p(m))
The training loss is calculated, which includes the cyclic distribution consistency loss Leyele (F(m), I (mm) between the receiver's output, and the initial local representation (mm the multi-dimennional alignment loss Luni (P, p(m)) between the common representation output by the senders and the negotiator,
16

---

Table Component ablation study. The collaborating agents are ml and m2, and the results are the performance without downstream collaborative task adaptation. The component name in bold in the settings column indicates the default configuration. Column corresponding to #Paramsf shows number of parameters when the module uses the corresponding configuration. "M' standing for 'MB'.
"ResMIp" network with a multi-layer perceptron as backbone. FANe(Youug al. (2022)
featuring encoder-decoder structure, which can be used adjust the feature space.
Components
Settings
AP@0.5
AP@0.7
#Params#
ResMIp
0.633
0,510
0.1 M
FANet
0.649
0.492
1.7 M
Recombiner
Convext
0.711
0.566
0.3 M
FAX
0.596
0.487
0.2 M
ResMIp
0.697
0.527
0.1 M
FANet
0.696
0.563
1.7 M
Aligner
Convext
0.702
0.542
0.3 M
Fused Axial Attention
0.711
0.566
0.2 M
ResMIp
0.705
0.565
1.8 M
Convext
0.706
0.566
2.7 M
Negotiator
Sparse Transformer
0.706
0.564
2.1 M
FPN
0.711
0.566
1.2M
w/o
0.672
0.547
Local Prompt
0.711
0.566
The parameters the negotiator, as well as the sender and receiver each modality's agent, are iteratively updated via backpropagation.
The objective of the second-stage training is to adapt the receiver for the downstream collaborative task. During this training process, the parameters of the negotiator, the perception encoders and senders of each modality's 's agent are frozen. The feature flow is consistent with that during inference. The loss is computed the collaborative loss of the agents within the initial alliance.
Sender
Receiver
Encoder
Recambiner
Aligner
Converter
Recombiner
K.
Arll
Resizer
Occ Head
Negotiator
Resizer
Same Perspective
Sender
Receiver
K.
Encoder
Recombiner
Aligner
Converter
Recombiner
Figure 5: Training process of initial alliance negotiation.
C.2 Training Process of New Agent Join
Figure Gillusttall the training process of the first stage when new agent joins. This stage aims to enable the new agent's sender and receiver to map local representations to and from the negotiated
17

---

Initial Alliance
Resizar
F
Negotiator
E
Data From
Same Persective
New Agent
Sendier
Receiver
K
Frooder
Rarsmbkiner
Aligner
Racambinne
R
Figure 6: Training process of new agent join.
common representation, respectively. The loss calculation for this process is identical that used during the common representation negotiation. The key difference that the common representation is generated by leveraging the negotiator and the perception encoder of the agents within the initial alliance. The specific steps are as follows:
Observational data from the same perspective is fed into the agents within the initial alliance and the new agent, encoding them into paired local representations F(m), p(o),
The local representations of the agents the initial alliance F(m) are input into the negotiator produce the common representation P.
The common representation P is fed into the new agent's 's receiver to obtain the reconstructed local representation I(e),
The new agent's local representation F(e) input into its sender to yield a common representation p(a),
The training loss is calculated, which includes the multi-dimennional alignment loss Luni (P, p())) between the common representation output by the negotiator and the sender of the new agent, and the cyclic distribution consistency loss Ceyele (F (a)) between the receiver's output and the initial local representation,
The parameters of the new agent's s sender and receiver are iteratively updated via backprop- agation, while the parameters of the negotiator and the encoders of the agents within the initial alliance remain frozen during this process.
In the second training stage, only the parameters of the new agent's 's receiver are adjusted, while the parameters of all other modules remain frozen. The feature flow during training is consistent with that during inference. The loss is calculated as the collaborative detection loss of the new agent and the agents within the alliance.
C.3 Sender and Receiver
The detailed structure of the sender and receiver shown in Figure 7 Both the sender and receiver adopt hybrid architecture combining Transformer and ConvNeXt. The sender consists of recom- biner and an aligner. responsible for transforming local features into the common representation space. The receiver comprises a recombiner and converter, which converts collaborators' features into the local representation space. The query vector the converter is derived from the output of the recombiner in the sender.
C.4 Negotiator
Figure & iillustrates the process of negotiating common representation P from initial local representa- tions ((m) and ((n) of agents with modalities and Agents first extracts local representations
18

---

P
Aligner
MLP
Fused Axial
Layer Norm
Attention
Fused Axial
Layer Norm
Attention
MLP
Recombiner
Recombiner
R!
ConvNeXt
ConvNeXt
Channel Unify
Interpolation
Interpolation
Channel
F!
(a) Sender
(b) Receiver
Figure 7: Detailed structure of the sender and receiver. Both the sender and receiver employ a hybrid architecture integrating Transformer and ConvNeXt.
F(m) and ((a) using its native perception encoder, then aligns them to standard size through the resizer to obtain (1m) and ('n),, Subsequently, the negotiator generates the common representation P from (mm) and (()) through the following steps:
Extract representations of each scale U,(T) and 7 from (mm) and (n)) respectively, using the pyramid network,
At each level, employ the corresponding estimator assess the contribution of U.mm) and 07(*) to the common representation, yielding the importance matrices c(()) and c(()) respectively,
For each level. multiply c(()) with (,(m), and c(") with U("), then average the results to obtain the level-wise common representation P,.
Upsample and concatenate all Pt. then restore the dimensions and channels to the standard configuration via shrink header to produce the final common representation P.
Negotistor
Reiioo
Lapen
p
Figure 8: Architecture negotiator. Layer-x and Estimator-x x is the network of pyramid and the estimator corresponding level.
19

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] " or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Contributions to heterogencous collaboration perception are outlined in the concluding section of the abstract and introduction.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: It is mentioned in the concluding section of the contribution that there will be inevitable loss of information when the new agent aligned to the negotiated common representation
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: No theoretical derivation is involved.
4. Experimental result reproducibility
20

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The detailed configuration of experiments has presented in the appendix.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code has been open-sourced github.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The detailed setup of the experiments has been given in the appendix, and the division of the optimizer and dataset can be viewed in the code.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: During testing, we observed that the experimental results exhibited minimal randomness, therefore statistical significance analysis was not performed.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The detailed setup of computing resources has been placed in the Appendix.
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp////arrrpp...ppppia//attat
Answer: [Yes]
Justification: The research in this paper accordance with the NeurIPS Code of Ethical in every respect.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The research in this paper is expected to effectively promote the development of collaborative perception and the arrival of the 1.5 era of autonomous driving.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper is tested on public datasets, and there is no such risk.
21

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: has been referenced the relevant point.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: No new assets provided in this paper.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The study in this paper does involve humans
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The study in this paper does not involve humans
16. Declaration LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used core methodology,
Question:
only for writing, editing, formatting purposes and does not impact scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
22


Reviews:
[{"review_id": "mioBdmdx6v", "reviewer": "Reviewer_K5st", "summary": "This paper addresses the challenge of immutable heterogeneity in collaborative perception—where agents with different, fixed perception models share intermediate features that suffer from domain gaps—by introducing NegoCollab, a framework that negotiates a common representation from each agent’s local representations via a feature-pyramid-based negotiator, and then employs plug-and-play sender–receiver modules to bidirectionally map between local and common feature spaces. To supervise this process, the authors propose a multi-dimensional alignment loss comprising distribution alignment, structural alignment (enforcing consistency of scene-component relations), and pragmatic alignment (via a unified 2D occupancy prediction task)", "questions": "N/A", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The negotiator generates a common representation by weighting multi-level features from each modality, substantially reducing domain gaps and easing sender–receiver alignment\n\n2. Combining distribution, structural, and pragmatic alignment fosters richer knowledge distillation from the common space, improving the expressiveness of local senders and boosting collaborative performance.\n\nWeakness:\n\n1. Once negotiated, the shared representation is static, leading to inevitable information loss and limited adaptability when integrating entirely new agent types.\n\n2. All experiments are conducted on the synthetic OPV2V-H dataset; real-world validation on diverse traffic scenes is absent, leaving open questions about generalization in more complex environments.\n\n3. While domain-gap reduction is well addressed, the approach does not consider bandwidth limitations, latency, or security/privacy aspects essential for real-world collaborative perception deployments.", "was_revised": true, "final_justification": "Thanks for the rebuttal. I keep my score Borderline accept", "rebuttal": "To the esteemed reviewer, I deeply appreciate the time and effort you've put into reviewing my work. Your suggestions are very helpful and will be carefully considered. Now let me address some concerns for you.\n\n> **W1:** Once negotiated, the shared representation is static, leading to inevitable information loss and limited adaptability when integrating entirely new agent types.\n\nSince the negotiation of common representations involves only agents within the initial alliance, it inevitably causes information loss for new agents. **Optimizing the composition of the initial alliance to enhance the generalization capability of common representations can partially mitigate this issue**. We provide an experimental analysis of collaborative performance when starting from different alliances in Appendix B.1, as shown in the following table: \n\n|  |      |       |    AP\\@0.5   |       |       |  |       |   AP\\@0.7    |       |       |\n|------------------|--------|-------|-------|-------|-------|--------|-------|-------|-------|-------|\n| **Initial Alliance** | m1m2   | m3m4  | m1m3  | m2m4  | All   | m1m2   | m3m4  |  m1m3 |  m2m4 | All   |\n| **Protocol**         | **0.792**  | 0.785 | 0.772 | **0.499** | 0.676 | **0.615**  | 0.564 | 0.710 | **0.289** | 0.457 |\n| **m1m3**             | 0.869  | 0.832 | **0.951** | 0.484 | **0.830** | 0.761  | 0.720 | **0.904** | 0.280 | **0.718** |\n| **m1m2**             | **0.872**  | 0.770 | 0.911 | **0.512** | 0.745 | **0.759**  | 0.578 | 0.805 | **0.319** | 0.555 |\n| **m3m4**             | 0.727  | 0.840 | **0.914** | 0.506 | **0.737** | 0.550  | 0.726 | **0.840** | 0.289 | **0.562** |\n\nTable 1: Performance comparison when negotiating common representations from different initial alliances. The \"Initial Alliance\" column indicates the agents in the initial alliance, while the remaining agents are new agents.\n\nThe comparison between line 1 (\"Protocol\", initial alliance containing only LiDAR-equipped agents) and line 3 (\"m1m2\", initial allliance containing both LiDAR and camera-equipped agents) demonstrates that common representations negotiated from the initial alliance composed with more types of agents achieve better performance in most collaborative scenarios. The comparison between line 2 (\"m1m3\") and line 3 (\"m3m4\") further shows that common representations negotiated from the initial alliance composed of relatively better-performing agents, m1 and m3, yield superior results in most cases. These findings support the **qualitative conclusion** that **selecting agents with better perception encoders for the initial alliance and including more agent types can enhance the generalization capability of common representations**, thereby improving collaboration with new agents.  \n\n&nbsp;\n\n> **W2:** All experiments are conducted on the synthetic OPV2V-H dataset; real-world validation on diverse traffic scenes is absent, leaving open questions about generalization in more complex environments.\n\nWe evaluated and compared the performance of NegoCollab against baseline methods on the real-world dataset V2V4Real [1], as shown in the following table:  \n\n|                   | Methods    | AP\\@0.5 | AP\\@0.7 |\n|-------------------|-----------|--------|--------|\n| One to one Adaptation  | MPDA      | 0.613  | 0.400  |\n|                       | PnPDA     | 0.598  | 0.385  |\n|-------------------|-----------|--------|--------|\n|   Align to Common    | MPDA-P    | 0.467  | 0.334  |\n|                | PnPDA-P   | 0.435  | 0.323  |\n|                   | STAMP     | 0.466  | 0.345  |\n|                   | NegoCollab| **0.605**  | **0.397**    |\n\nTable 3: Performance comparison of heterogeneous collaboration on V2V4Real. Agents involved in the collaboration are m1 and m3.\n\nAs shown, NegoCollab achieves outstanding collaborative performance even in real-world environments. These experimental results will be included in the revised version. \n\n[1] V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception. CVPR2023\n\n&nbsp;\n\n> **W3:** While domain-gap reduction is well addressed, the approach does not consider bandwidth limitations, latency, or security/privacy aspects essential for real-world collaborative perception deployments.\n\nNegoCollab eliminates the domain gap by introducing a pair of sender and receiver modules, which serve as additional plug-and-play components to the original homogeneous collaboration model of the agent. Aspects essential for real-world collaborative perception deployments. such as bandwidth limitations, latency, and security/privacy are **handled by the original homogeneous collaboration model**. As long as the original homogeneous collaboration model incorporates additional designs addressing these challenges, NegoCollab can readily adapt to real-world environments.  \n\nTo verify this, we simulated the **positioning errors** existing in the real world in the synthetic dataset OPV2V-H, and evaluated NegoCollab's performance under these conditions (see Figure 3 in the main text). The results show that NegoCollab maintains strong robustness even with increasing localization errors, **demonstrating its preliminary adaptability to complex real-world environments**."}, {"review_id": "FiumQ5d0Q2", "reviewer": "Reviewer_py3y", "summary": "This paper introduces NegoCollab, a framework for collaborative perception among heterogeneous agents, particularly in autonomous driving scenarios. The key idea is to negotiate a common representation space by aggregating local representations from multiple modalities (e.g., LiDAR, camera) through a dedicated negotiator during training, thus minimizing domain gaps between agents with differing sensors and backbones. Training is supervised by distribution, structure, and pragmatic alignment losses to ensure comprehensive and meaningful alignment. The experimental results on the OPV2V-H dataset show that NegoCollab achieves superior performance compared to existing one-to-one and common representation-based approaches, demonstrating improved adaptability and knowledge transfer across heterogeneous agent types.", "questions": "See Weekness.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "*Strengths*\n1. The paper addresses a relevant and practical challenge in collaborative perception: handling immutable heterogeneity among agents by introducing a negotiator for common representation. This is a meaningful contribution to autonomy and multi-agent systems.\n2. The methodology is well constructed, combining a plug-and-play sender/receiver architecture with a negotiator for representation fusion. The distinction from solely protocol-agent-based approaches is well articulated both in the Introduction and Figure 1.\n\n\n*Weakness*\n\n1.Lack of Architectural Component Ablation. \n\nThe paper introduces several novel architectural components, such as the ConvNeXt-based Recombiner, the Sparse Transformer-based Aligner/Converter, and the FPN structure within the Negotiator.  However, the ablation studies do not sufficiently justify these specific design choices. It remains unclear how much performance gain is attributable to these sophisticated modules compared to simpler alternatives (e.g., using MLPs for recombination or a non-pyramidal structure for the negotiator). A more thorough ablation on these architectural sub-components would be necessary to validate their effectiveness and contribution to the overall framework.\n\n2. The Role of the \"Local Prompt\" is Not Experimentally Validated. \n\nA key design element of the Receiver is the use of a \"local prompt\"—the output of the sender's recombiner as the query vector for the converter. This is proposed to provide local modality guidance during the feature transformation process.  However, the importance of this mechanism is not ablated", "was_revised": true, "final_justification": "Overall, it is a good paper. The author have solved my main concerns. It's glad to see the author's contribution, I prefer to accept this paper.", "rebuttal": "We sincerely appreciate your recognition of our work's contributions to multi-agent system development. Your constructive suggestions are invaluable to refining this work. In the following, we will address your concerns regarding the weaknesses and show additional experimental results.\n\n>  **W1:** Lack of Architectural Component Ablation. \n\nWe respectively validated the performance of NegoCollab when setting Recombiner, Aligner, and Negotiator to different structures (while keeping other modules unchanged when modifying one component's structure), as shown in the table below:\n\n| Components   | Settings         | AP\\@0.5 | AP\\@0.7 | #Params# |\n|--------------|------------------|--------|--------|----------|\n| Recombiner   | ResMlp           | 0.633  | 0.510  | 0.1 M    |\n|              | FANet            | 0.649  | 0.492  | 1.7 M    |\n|              | Conevxt           | **0.711**  | **0.566**  | 0.3 M    |\n|              | Sparse Transformer | 0.596  | 0.487  | 0.2 M    |\n| Aligner      | ResMlp           | 0.697  | 0.527  | 0.1 M    |\n|              | FANet            | 0.696  | 0.563  | 1.7 M    |\n|              | Convext           | 0.702  | 0.542  | 0.3 M    |\n|              | Sparse Transformer | **0.711**  | **0.566**  | 0.2 M    |\n| Negotiator   | ResMlp           | 0.705  | 0.565  | 1.8 M    |\n|              | Convext           | 0.706  | 0.566  | 2.7 M    |\n|              | Sparse Transformer | 0.706  | 0.564  | 2.1 M    |\n|              | FPN              | **0.711**  | **0.566**  | 1.2 M    |\n\nTable 1: Architectural Component Ablation. \n\n\nIn the table, **#Params#** represents the size of parameter of module when it uses different settings. The evaluation involves collaborating agents m1 and m2, showing their collaborative performance prior to downstream task adaptation training. The \"ResMLP\" is a network with a multilayer perceptron (MLP) backbone, FANet [1] employs a 2D convolutional backbone in an encoder-decoder architecture for feature space adaptation.  \n\nThe comparative results demonstrate that **NegoCollab achieves optimal performance when the Recombiner and Aligner are configured as Convext and Sparse Transformer respectively in their corresponding experimental groups**. This superior performance arises from our two-stage feature transformation pipeline: local detail refinement via the Recombiner and global representation style transfer through the Aligner. **The architectural properties of Convext and Sparse Transformer make them ideally suited for local refinement and representation transformation tasks respectively**, allowing each module in NegoCollab to operate at maximum effectiveness. **For the Negotiator component, our implemented FPN structure attains peak performance with minimal parameter requirements.** The above results fully demonstrate the rationality of each component's design in NegoCollab.\n\n[1] Feature-Align Network with Knowledge Distillation for Efficient Denoising. WACV2022\n\n&nbsp;\n\n>  **W2:** The Role of the \"Local Prompt\" is Not Experimentally Validated.\n\nWe evaluated NegoCollab's performance with and without Local Prompt integration, as shown in the following table:  \n\n| Local Prompt | AP\\@0.5 | AP\\@0.7 |\n|--------------|--------|--------|\n| w/o          | 0.672  | 0.547  |\n| w            | 0.711  | 0.566  |\n\nTable 2: Component ablation of local prompt. The collaborating agents involved were m1 and m2, with the table presenting the collaborative performance before downstream task adaptation training.\n\nThe results demonstrate that integrating **local prompt improves NegoCollab's performance by 5.8% in AP@0.5 and 3.5% in AP@0.7**, confirming the effectiveness of this module.  \n\n&nbsp;\n\n Finally, we sincerely appreciate your constructive feedback on our work, and we will incorporate all the aforementioned ablation studies in the revised version."}, {"review_id": "NPB69qqvcd", "reviewer": "Reviewer_s7iD", "summary": "The authors present NegoCollab, a framework for collaborative perception among agents with different sensors and models. The key idea is to \"negotiate\" a common representation space that all agents can map to, rather than forcing everyone to conform to one specific agent's representation. This is done using a \"negotiator\" module trained with a complex, multi-part loss function involving distribution, structural, and pragmatic alignment. The authors show strong performance on the OPV2V-H dataset.", "questions": "1. The same as point 4 in the Weaknesses. How is this different from a fusion network? \n2. The loss function is complex. The \"pragmatic alignment\" part, which needs a whole separate network, seems like a lot of extra work. Have you tried removing it? I'm curious how much of your performance gain is due to this versus the core idea.\n3. The paper's biggest blind spot is scalability. What's the plan when a new agent type, one that wasn't part of the initial \"negotiation,\" wants to join the collaboration? Do you have to retrain everything from scratch? If so, the \"low training cost\" claim from the abstract seems misleading, as the real cost is in re-integration.  If you can clarify this, I would be pleased to improve the rating.", "limitations": "The primary limitation of this work is the challenge of dynamically integrating new agents into the system.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. For a fixed set of heterogeneous agents, the proposed \"negotiation\" process and the complex alignment losses do achieve excellent performance.\n2. The idea that a democratically \"negotiated\" space might be better than a \"dictated\" one is appealing at first glance. \n\nWeaknesses:\n1. This paper tackles a \"closed-world\" problem where the set of agent types is predefined. This is a significant step back from the \"open-world\" formulation presented in recent work like HEAL (Lu et al., ICLR 2024), which explicitly addresses the critical challenge of continuously integrating new, unseen agent types. NegoCollab's approach, by design, ignores the most crucial aspect of real-world deployment: scalability and extensibility.\n2. The core design of NegoCollab is inherently non-extensible. When a new agent needs to join, the entire \"negotiated\" space becomes suboptimal. This forces a choice between two bad options: either the new agent suffers from poor alignment, or the entire collective must be retrained at a massive cost. This contrasts sharply with HEAL's \"backward alignment,\" which provides a clear, low-cost path for new agents to join.\n3.  The paper introduces a highly complex loss function with distribution, structural, and even a \"pragmatic\" alignment component , kind of over-engineered.\n4. The core concept is \"negotiation,\" which implies a dynamic, give-and-take process. But your negotiator is trained once and then fixed. How is this different from a multi-input fusion network? The term feels like a bit of a stretch.", "was_revised": true, "final_justification": "In light of the fact that the authors have addressed some of my concerns, I have increased my score.", "rebuttal": "Thank you very much for dedicating your time and expertise to review our submission. The depth of your analysis and the constructive criticism you have provided are immensely beneficial. I sincerely hope my reply can answer your questions.\n\n> **W4:** The core concept is \"negotiation,\" which implies a dynamic, give-and-take process. But your negotiator is trained once and then fixed. How is this different from a multi-input fusion network? The term feels like a bit of a stretch. \n> \n> **Q1:** The same as point 4 in the Weaknesses. How is this different from a fusion network?\n\nArchitecturally, the **Negotiator** does resemble **multi-input fusion networks** in structure. However, **their functional roles within the collaborative framework differ fundamentally**, as we elaborate below:  \n\n- **Input/Output**: The **negotiator** takes local representations encoded by each modality's encoders from the **same perspective** of environmental information as **input**, and **outputs a common representation**. In contrast, the **input** of the **multi-input fusion network** is the features obtained by agents from encoding the environmental information from **different perspectives**, and **outputs features fused with multi-perspective information**.\n\n- **Estimator:** The estimator in the Negotiator **evaluates each modality's contribution to the shared representation**, whereas the foreground estimator in multi-input fusion networks **assesses the probability of foreground objects** appearing at corresponding locations in BEV.\n\n- **Role of Module:** The Negotiator's role is to **negotiate the common representations** while implicitly preserving their information through model parameters, whereas multi-input fusion networks are designed to **integrate multi-view information**.  \n\nIn NegoCollab, **the concept of \"negotiation\" is embodied in the mutual compromise among agents within the initial alliance** when negotiating the common representation from their respective local representations. For dynamically joined new agents participating in collaboration, they only need to align with the common representation negotiated by the initial alliance.\n\n&nbsp;\n\n> **W3:** The paper introduces a highly complex loss function with distribution, structural, and even a \"pragmatic\" alignment component, kind of over-engineered.\n>  \n> **Q2:** The loss function is complex. The \"pragmatic alignment\" part, which needs a whole separate network, seems like a lot of extra work. Have you tried removing it? I'm curious how much of your performance gain is due to this versus the core idea.\n\nThe multi-dimensional alignment loss plays a critical role in facilitating better alignment between unimodal local representations and the multimodal common representation. In Section 4.3, we perform comprehensive ablation studies on the individual components of the multi-dimensional alignment loss. The comparative results after completing the first training phase are summarized in the table below:\n\n| uni-dis | uni-stru | uni-pragma | AP\\@0.5 | Inc.   | AP\\@0.7 | Inc.    |\n|:---------:|:----------:|:------------:|:--------:|:--------:|:--------:|:---------:|\n| ✓       |          |            | 0.609  | -      | 0.496  | -       |\n| ✓       | ✓        |            | 0.655  | +7%    | 0.532  | +6.8%   |\n| ✓       |          | ✓          | 0.671  | +9.2%  | 0.538  | +7.8%   |\n| ✓       | ✓        | ✓          | **0.711** | **+14.3%** | **0.566** | **+12.4%** |\n\nTable 1. Ablation study of the multi-dimensional alignment loss. The collaborating agents are m1 and m2.\n\nTaking the performance under AP@0.5 as an example: when solely adding the structural **consistency loss, performance improves by 7%**; when only incorporating the **pragmatic consistency loss, performance gains approximately 10%**; when **both consistency losses** are included, performance **increases by about 14%**. These results clearly demonstrate the significance of multi-dimensional alignment loss for enhancing heterogeneous collaboration performance. Furthermore, the **separate network required for auxiliary pragmatic alignment** is a 2D occupancy detection head, implemented as a single-layer 2D convolutional network **with approximately 1.3K parameters, introducing virtually no additional training overhead**.\n\nThe design of the multi-dimensional alignment loss stems from this **analysis: The essence of aligning local representations to the common representation lies in transforming single-modal features into multi-modal features, which cannot be adequately achieved solely through conventional distribution consistency loss that merely constrains feature distributions.** This is because the common representation is negotiated from multiple different types of agents and contains multi-modal information, while the local representation is encoded by the local encoder from observed data and contains only single-modal information. Taking the common representation negotiation between m1 and m2 as an example: m1 and m2 employ LiDAR and camera sensors respectively, with their local representations containing only single-modal information from the corresponding sensors. The common representation negotiated from m1 and m2 combines camera-LiDAR fused modalities. However, **inherent differences** exist between camera and LiDAR modalities **in terms of data format, spatial structure, and pragmatic information**. These differences are consequently reflected in the disparities between single-modal (camera/LiDAR) features and fused-modal features. Therefore, **aligning such cross-modal features requires multi-dimensional consistency constraints encompassing distribution, structural relationships, and pragmatic information between features.**\n\nIn the revised version, we will refine the descriptions of the multi-dimensional alignment loss and its ablation studies (Lines 189-195 and 293-301) to better clarify their functions.\n\n&nbsp;\n\n>**W1:** This paper tackles a \"closed-world\" problem where the set of agent types is predefined.\n>\n> **W2:** The core design of NegoCollab is inherently non-extensible. \n> \n> **Q3:** The paper's biggest blind spot is scalability. What's the plan when a new agent type, one that wasn't part of the initial \"negotiation,\" wants to join the collaboration? Do you have to retrain everything from scratch? If so, the \"low training cost\" claim from the abstract seems misleading, as the real cost is in re-integration. If you can clarify this, I would be pleased to improve the rating.\n\nFor dynamically joining **new agents**, collaborative participation can be achieved by training a pair of **plug-and-play sender-receiver** to align with the negotiated common representation. The training process **only requires parameter updates for the new agent's sender and receiver modules while eliminating the need to retrain the negotiator network**, thus providing **an efficient solution for heterogeneous collaboration in \"open-world\" scenarios with minimal training cost**.  \n\nWe explained the training process when a new agent joins in Section 4.1. Specifically, in the first stage, **the common representation is generated from the negotiator and the encoder of the agents in initial alliance.** The cyclic consistency loss and multi-dimensional alignment loss are calculated in the same way as described in Section 3.2.2 and Section 3.2.3. In the second stage, the collaborative task loss is calculated as the collaborative detection loss between the new agent and the agents in the initial alliance. We will add a diagram of the training process for a new agent in the appendix of the revised version.\n\nNotably, since new agents do not participate in the negotiation process, aligning them to the common representation inevitably incurs greater information loss. This indeed represents a limitation of NegoCollab in \"open-world\" collaboration scenarios. To mitigate this limitation, we investigate NegoCollab's collaborative performance when the common representation is negotiated from different initial alliances. Our analysis concludes that selecting initial alliance agents with optimally performing perception encoders and including more diverse agent types in the initial alliance can enhance the generalization capability of the common representation, thereby improving collaborative performance in \"open-world\" settings. The relevant results are presented in Appendix B.1, with selected findings shown below:\n\n|  |      |       |    AP\\@0.5   |       |       |  |       |   AP\\@0.7    |       |       |\n|------------------|--------|-------|-------|-------|-------|--------|-------|-------|-------|-------|\n| **Initial Alliance**  | m1m2   | m3m4  | m1m3  | m2m4  | All   | m1m2   | m3m4  |  m1m3 |  m2m4 | All   |\n| **Protocol**          | 0.792  | 0.785 | 0.772 | 0.499 | 0.676 | 0.615  | 0.564 | 0.710 | 0.289 | 0.457 |\n| **m1m3**              | 0.869  | 0.832 | 0.951 | 0.484 | 0.830 | 0.761  | 0.720 | 0.904 | 0.280 | 0.718 |\n| **m1m2**              | 0.872  | 0.770 | 0.911 | 0.512 | 0.745 | 0.759  | 0.578 | 0.805 | 0.319 | 0.555 |\n| **m3m4**              | 0.727  | 0.840 | 0.914 | 0.506 | 0.737 | 0.550  | 0.726 | 0.840 | 0.289 | 0.562 |\n\nTable 2: Performance comparison when negotiating common representations from different initial alliances."}, {"review_id": "BKNEVptgm2", "reviewer": "Reviewer_fF95", "summary": "In this paper, the authors propose a heterogeneous collaboration approach, NegoCollab, which is based on negotiated common representation. This framework facilitates heterogeneous information exchange through a pair of sender and receiver, specifically comprising a recombiner, an aligner, and a converter. Several supervision losses are introduced to achieve bidirectional conversion between local representations and the common representation. The experimental results on the OPV2V-H dataset demonstrate the effectiveness of the proposed approach.", "questions": "Result-level collaboration can naturally address the heterogeneous challenge and is more computationally efficient. How about the performance comparison?\n\nWhy is the performance of NegoCollab-FT worse than one-to-one adaptation in the m1m3 and m2m4 settings? Is it due to the negotiation mechanism?\n\nWhat is the coordinate system of the BEV common feature map? Does it depend on the relative positions of the collaborating vehicles?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The authors propose a heterogeneous collaboration approach to address the practical heterogeneity challenge in collaborative perception. The proposed NegoCollab is technically sound, training a negotiator to generate the common representation. Several supervision strategies are introduced to better align local representations with the common representation. The paper is well-organized, and the experimental analysis is comprehensive and in-depth, especially in the studies of generalization capability and negotiation mechanism.\n\nHowever, there are still some concerns. The performance advantage is not significant compared with one-to-one adaptation. The proposed approach is only evaluated on a simulation dataset, whereas the heterogeneous problem in real-world scenarios is more challenging. The generalization of the common representation to new agents is still constrained by the training process.\n\nAdditionally, there are typos in lines 164 and 171. Table 1 is too wide.", "was_revised": true, "final_justification": "The experimental results address my concerns to a large extent. I have carefully read the authors' response as well as the discussion with other reviewers. I have decided to keep my relatively positive score.", "rebuttal": "We thank the reviewer for providing valuable comments, we will address your concerns regarding the questions and show additional experimental results.\n\n>  **Q1:** Result-level collaboration can naturally address the heterogeneous challenge and is more computationally efficient. How about the performance comparison?\n\nWe evaluated the performance of late fusion across different collaborative scenarios on the OPV2V-H dataset, testing its robustness against progressively increasing localization errors. Comparative results with NegoCollab and NegoCollab-P are presented in the table below:\n\n|  |            | |     AP\\@0.5   |       |         | Avg. Inc. |  |    AP\\@0.7   |       |         | Avg. Inc. |\n|------------|------------------|--------|-------|-------|---------|-----------|--------|-------|-------|---------|-----------|\n|    σ       | Agent Types      | m1m2   | m1m3  | m2m4  | All |           | m1m2   | m1m3  | m2m4  | All|           |\n| 0.0        | Late Fusion      | **0.873**  | **0.952** | 0.482 | **0.854**| -         | 0.743  | **0.893** | 0.290 | **0.725**| -         |\n|            | NegoCollab-P     | 0.792  | 0.772 | 0.499 | 0.676   | -13.3%    | 0.615  | 0.710  | 0.289 | 0.457   | -21.9%    |\n|            | NegoCollab    | 0.872  | 0.911 | **0.512** | 0.745 | -3.8%     | **0.765** | 0.854  | **0.319** | 0.555   | -0.06%    |\n| 0.3        | Late Fusion      | 0.564  | 0.626 | 0.299 | 0.543   | -         | 0.201  | 0.271  | 0.077 | 0.197   | -         |\n|            | NegoCollab-P     | 0.676  | 0.711 | **0.391** | 0.591 | +16.6%    | 0.403  | 0.527  | **0.149** | **0.388** | +96.6%    |\n|            | NegoCollab    | **0.719** | **0.837** | 0.387 | **0.616** | +25.9%    | **0.425** | **0.582** | 0.146 | 0.365   | +103.4%   |\n| 0.6        | Late Fusion      | 0.278  | 0.328 | 0.154 | 0.264   | -         | 0.115  | 0.169  | 0.043 | 0.106   | -         |\n|            | NegoCollab-P     | 0.477  | 0.574 | **0.256** | 0.500 | +79.5%    | 0.283  | 0.397  | **0.099** | **0.353** | +161.1%   |\n|            | NegoCollab    | **0.483** | **0.693** | 0.229 | 0.462   | +82.3%    | 0.276  | **0.427** | 0.086 | 0.292   | +149.7%   |  \n\nTable 1: Performance Comparison with Late Fusion under different localization error conditions. Agent positions are perturbed with Gaussian noise of standard deviations 0.0, 0.3, and 0.6. \"All\" refers to the scenario when all four types of intelligent agents, m1, m2, m3, and m4, are involved in collaboration. \"Avg. Inc.\" column in the table indicates the average performance improvement of NegoCollab and NegoCollab-P over late fusion across different collaborative scenarios under varying noise conditions.\n\n\nThe results demonstrate that late fusion achieves superior performance when no localization errors exist, as it can effectively mitigate the impact of model heterogeneity. However, as localization errors progressively increase, late fusion shows significant performance degradation, while the intermediate fusion-based NegoCollab-P and NegoCollab maintain robust performance, substantially outperforming late fusion. \n\nNotably, although late fusion performs well under ideal conditions without localization errors, such perfect conditions rarely exist in real-world scenarios where localization errors are practically inevitable. Therefore, the more robust NegoCollab framework offers greater practical utility. We will include these experimental results in the revised version.\n\n&nbsp;\n\n>  **W:** The generalization of the common representation to new agents is still constrained by the training process.\n> \n> **Q2:** Why is the performance of NegoCollab-FT worse than one-to-one adaptation in the m1m3 and m2m4 settings? Is it due to the negotiation mechanism?\n\n This result is indeed caused by the negotiation mechanism. As detailed in Section 4.1, the common representation is negotiated between agents m1 and m2 from the initial alliance. **The newly joined agents m3 and m4 do not participate in this negotiation process**, leading to inevitably greater information loss during alignment with the pre-established common representation. In contrast, **the one-to-one adaptation method involves m3 and m4 in training their respective domain adaptation modules**, resulting in relatively smaller information loss. This explains why NegoCollab demonstrates weaker performance than one-to-one adaptation in m1m3 and m2m4 collaboration scenarios. Compared to one-to-one adaptation, **NegoCollab's advantage lies in its lower training cost**: For N agent types involved in collaboration, one-to-one adaptation requires training N² domain adaptation modules, whereas NegoCollab only needs 2N modules.  \n\n**The** limitations of NegoCollab when collaborating with new agents can be mitigated by optimizing the composition of agents in the initial **alliance**. In Appendix B.1, we present a comparative analysis of collaborative performance when starting from different initial alliances, as shown in the following table:\n\n|  |      |       |    AP\\@0.5   |       |       |  |       |   AP\\@0.7    |       |       |\n|------------------|--------|-------|-------|-------|-------|--------|-------|-------|-------|-------|\n| **Initial Alliance** | m1m2   | m3m4  | m1m3  | m2m4  | All   | m1m2   | m3m4  |  m1m3 |  m2m4 | All   |\n| **Protocol**         | **0.792**  | 0.785 | 0.772 | **0.499** | 0.676 | **0.615**  | 0.564 | 0.710 | **0.289** | 0.457 |\n| **m1m3**             | 0.869  | 0.832 | **0.951** | 0.484 | **0.830** | 0.761  | 0.720 | **0.904** | 0.280 | **0.718** |\n| **m1m2**             | **0.872**  | 0.770 | 0.911 | **0.512** | 0.745 | **0.759**  | 0.578 | 0.805 | **0.319** | 0.555 |\n| **m3m4**             | 0.727  | 0.840 | **0.914** | 0.506 | **0.737** | 0.550  | 0.726 | **0.840** | 0.289 | **0.562** |\n\nTable 2: Performance comparison when negotiating common representations from different initial alliances. The \"Initial Alliance\" column indicates the agents in the initial alliance, while the remaining agents are new agents.\n\n Based on the comparisons of line 1 and line 3, as well as line 2 and line 4, a qualitative conclusion can be drawn: forming the initial alliance with agents possessing optimally performing perception encoders and including more diverse agent types enhances the generalization capability of the common representation, thereby improving its adaptability to collaboration with new agents.\n\n&nbsp;\n\n > **Q3:** What is the coordinate system of the BEV common feature map? Does it depend on the relative positions of the collaborating vehicles?\n\n **During training, both the common representation and each modality's local representations share the same coordinate system**, representing environmental information from identical perspectives, **independent of the relative position of the vehicle**. During the training phase when loading data, NegoCollab uniformly batches perception data from all perspectives in the scene, duplicates it N times, and feeds identical input to N different modality's encoders to obtain paired features of each modality, and subsequently calculating the loss and generating the common representation.\n\n**During inference**, the data loading process distinguishes between different perspectives. The **BEV common features** are transformed from each vehicle's local features, **maintaining coordinate systems aligned with their respective ego perspectives and incorporating relative vehicle positions**.\n\nWe will refine the description in lines 158-162 in the revised version to improve reader comprehension.\n\n&nbsp;\n\n> **W:**  The proposed approach is only evaluated on a simulation dataset, whereas the heterogeneous problem in real-world scenarios is more challenging. \n\nWe evaluated and compared the performance of NegoCollab against baseline methods on the real-world dataset V2V4Real [1], as shown in the following table:  \n\n|                   | Methods    | AP\\@0.5 | AP\\@0.7 |\n|-------------------|-----------|--------|--------|\n| One to one Adaptation  | MPDA      | 0.613  | 0.400  |\n|                       | PnPDA     | 0.598  | 0.385  |\n|-------------------|-----------|--------|--------|\n|   Align to Common    | MPDA-P    | 0.467  | 0.334  |\n|                | PnPDA-P   | 0.435  | 0.323  |\n|                   | STAMP     | 0.466  | 0.345  |\n|                   | NegoCollab| **0.605**  | **0.397**  |\n\nTable 3: Performance comparison of heterogeneous collaboration on V2V4Real. Agents involved in the collaboration are m1 and m3.\n\nAs shown, NegoCollab achieves outstanding collaborative performance even in real-world environments. These experimental results will be included in the revised version.\n\n&nbsp;\n\n[1] V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception. CVPR2023"}]

Author rebuttal:

Meta-review: The paper proposes NegoCollab, a novel framework for heterogeneous collaborative perception that negotiates a common representation across multiple agents and leverages sender–receiver mappings together with multi-dimensional alignment losses. 

Reviewers agree that the idea of negotiated representation is both original and meaningful for reducing domain gaps in multi-agent perception. The methodology is technically sound, combining distributional, structural, and pragmatic alignment in a modular design. Empirical results on OPV2V-H show consistent improvements over strong baselines, and the rebuttal further added results on V2V4Real, strengthening confidence in the approach. Other highlighted strengths include clear motivation, strong architectural design, and the ability to generalize to different agent types without retraining entire models.

At the same time, some limitations remain. The negotiated representation is essentially static and scalability to unseen agents is not fully explored. Evaluation is still simulation-heavy, and the complexity of the alignment loss design may pose implementation challenges. 

Nevertheless, these issues do not undermine the core contributions, which are both technically sound and practically relevant. We recommend acceptance, with the following suggestions for the camera-ready: (1) providing deeper ablations on the role of each alignment loss, (2) clarifying scalability to dynamic and real-world heterogeneous agents, and (3) including the experiments added during the rebuttal.
Decision: Accept (poster)