Point Cluster: A Compact Message Unit for Communication-Efficient Collaborative Perception
Authors: Zihan Ding
Jiahui Fu
Si Liu
Hongyu Li
Siheng Chen
Hongsheng Li
Shifeng Zhang
Xu Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: Point Cluster
Collaborative Perception
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: The objective of the collaborative perception task is to enhance the individual agent's perception capability through message communication among neighboring agents. A central challenge lies in optimizing the inherent trade-off between perception ability and communication cost. To tackle this bottleneck issue, we argue that a good message unit should encapsulate both semantic and structural information in a sparse format, a feature not present in prior approaches. In this paper, we innovatively propose a compact message unit, namely point cluster, whose core idea is to represent potential objects efficiently with explicitly decoupled low-level structure information and high-level semantic information. Building upon this new message unit, we propose a comprehensive framework CPPC for communication-efficient collaborative perception. The core principle of CPPC is twofold: first, through strategical point sampling, structure information can be well preserved with a few key points, which can significantly reduce communication cost; second, the sequence format of point clusters enables efficient message aggregation by set matching and merging, thereby eliminating unnecessary computation generated when aligning squared BEV maps, especially for long-range collaboration. To handle time latency and pose errors encountered in real-world scenarios, we also carefully design parameter-free solutions that can adapt to different noisy levels without finetuning. Experiments on two widely recognized collaborative perception benchmarks showcase the superior performance of our method compared to the previous state-of-the-art approaches.

Published as a conference paper at ICLR 2025
POINT CLUSTER: A COMPACT MESSAGE UNIT FOR COMMUNICATION-EFFIIII EFFICIENT COLLABORATIVE PER- CEPTION
Zihan Ding', Jiahui Fu', , : Hongyu Li', Siheng Chen',
Hongsheng L,8.4, Shifeng Zhang Xu Zhou
1 Institute of Artificial Intelligence, Beihang University.
2 School of Artificial Intelligence, Shanghai Jiao Tong University,
3 MMLab, CUHK, Centre for Perceptual and Interactive Intelligence, Sangfor Technologies dingzihan73788 2egggphhnn77@m......... com
ABSTRACT
The objective of the collaborative perception task is enhance the individual agent's perception capability through message communication among neighbor- ing agents. central challenge lies optimizing the inherent trade-off between perception ability and communication cost. To tackle this bottleneck issue, we argue that a good message unit should encapsulate both semantic and structural information in sparse format, feature not present in prior approaches. In this paper, we innovatively propose compact message unit, namely point cluster, whose core idea is to represent potential objects efficiently with explicitly decou- pled low-level structure information and high-level semantic information. Build- ing upon this new message unit, we propose a comprehensive framework CPPC for communication -fffccient collaborative perception. The core principle of CPPC is twofold: first, through strategical point sampling, structure information can be well preserved with a few key points, which can significantly reduce communica- tion cost; second, the sequence format of point clusters enables efficient message aggregation by set matching and merging, thereby eliminating unnecessary com- putation generated when aligning squared BEV maps, especially for long-range collaboration. To handle time latency and pose errors encountered in real-world scenarios, we also carefully design parameter- free solutions that can adapt to dif- ferent noisy levels without finetuning. Experiments serval widely recognized collaborative perception benchmarks showcase the superior performance of our method compared to the previous state-of-the-art approaches.
INTRODUCTION
Collaborative perception aims to enhance the perception capabilities of individual agents by utilizing complementary information exchanged between surrounding agents. This approach offers novel strategy to address several inherent challenges
single-agent perception He et al. (2017); Cheng
et al. 2022),, including occlusion and long-range limitations. There is pressing need for related
methods and systems across a wide spectrum of practical applications, such as vehicle-to-everyyne autonomous driving Yu al. (2022). automated multi-robot warehouse systems [i al.)
(0020)
and multi-UAVs for search and rescue Scherer et al. (2015). Recent efforts have made valuable contributions in terms of high-quality real and simulated datasets Yu et al. (2022); Xu al. (2022b);
Li et al. (2022); Xu et al. (2022a] 2023b), as well as effective solutions Xu et al. (2023a): Li et al.
2021a)) Wang et al. (2020p; Xu et al. (2022a); Hu et al. (2022); Chen et 2019a):
Xu et
20226 for collaborative perception.
The paramount challenge in this field involves optimizing perceptual ability in the face of lim- ited communication bandwidth in real-world scenarios, i.e., communicatio--fffiiinnne. Based on
the type of observation transmission medium (i.e., message unit), previous studies can be catego- rized into three types: early collaboration with raw point cloud data Chen et al. (2019b); Arnold
ctt al. (2020), late collaboration using bounding boxes Shi et al. (2022); Zeng et al. (2020); Rauch
al. 2012); Glaser & Kira 2023), and intermediate collaboration through bird's 's eye view (BEV)
maps Xu et al. (20226 a 2023a): Hu et al.] (2022). Early collaboration can simplify subsequent
*Corresponding author

---

Published as a conference paper at ICLR 2025
analyses and enrich information for downstream models. Despite fostering high performance, this approach incurs huge bandwidth consumption due to the transmission of complete raw observa- tions. Late collaboration ensures efficient economic bandwidth consumption; however, it suffers from performance bottlenecks and reduced robustness due to searce object information. Intermedi- ate collaboration compresses representative information into BEV feature maps, resulting in reduced communication bandwidth compared to early collaboration, while also enhancing perception capa- bilities in comparison late collaboration. Despite significant development, the manual projection of point clouds to BEV feature maps suffers from quantization error Shi et al. (2019) and creates a bottleneck in collaboration, hindering these methods from effectively completing 3D structural information for precise object boundary predictions.
Considering the above issues, we innovatively propose a compact message unit, called point cluster, which describes objects with point coordinates representing the object structure, a cluster center representing the object position, and cluster feature representing the high-level semantics of the object. The point cluster has several merits compared to existing message units: 1) Unlike raw point clouds, point clusters intrinsically capture only the foreground objects within a scene in a sparse manner, leading efficient communication; 2) In contrast to bounding boxes, point clusters offer sufficient information about potential objects, enabling more robust aggregation and decoding for enhanced regression accuracy; 3) Compared BEV maps, point clusters explicitly preserve the structural information of objects in original coordinate space, enabling fine-grained point alignment and complementary structure information fusion between different agents.
To fully unleash the potential of this powerful message unit, we propose CPPC, a comprehensive Collaborative Perception framework based on Point Cluster that revolutionizes existing communica- tion mechanism, while featuring low bandwidth usage, high-performance perception, and real-noise robustness. The main body of our CPPC includes two key modules: 1) point cluster packing mod- ule, which flexibly controls the number of points contained in each point cluster while maintaining their geometric structure to cope with different bandwidth constraints; 2) a point cluster aggregation module, which integrates point clusters from other agents with set matching and merging to achieve a comprehensive understanding of the surrounding scene for the ego agent. In addition to optimizing bandwidth and performance trade-off, our CPPC also has careful designs for two common robust- ness challenges, i.e., pose error Lu et al. 2023) and time delay Lei et al. (002);; Wei et al. (2024), benefited from the low-level information in point cluster. To explicitly solve the pose error, pro- pose to reformulate cluster centers as vertices a graph and optimize to promote pose consistency between agents and point clusters. In order to compensate for the time latency, we link point clus- ters of the same object along the time dimension and measure its speed for position prediction in the current timestamp.
As an intermediate collaboration method, using point cluster as message unit gives our CPPC three distinct advantages compared with previous methods based on BEV map: 1) Decoupling of struc- ture and semantic information allows our CPPC to efficiently represent each potential object with a few key points and significantly fewer feature channels, all while avoiding the loss of high-level information during message packing. 2) The computational complexity of our CPPC during mes- sage aggregation efficiently related to the number of possible objects the scene, rather than quadratically related to the perception range, which is more suitable for long-range collaboration. 3) Our graph optimization and speed estimation methods possess a strong generalization capability as they do not rely on any training parameters during the optimization process. This enables our method to adapt to varying levels of noise with ease. To validate the effectiveness of our proposed three popular collaborative perception benchmarks, V2X-Set Xu
CPPC, we conduct experiments let al. (2022a), OPV2V Xu et al. (2022b), and DAIR-V2X Yu et al. (002)). Our method achieves the new state-of-the-a -her-- performances with significant performance gains of 5.7%, 7.3% and 12.8% on the strict mAP@0.7 metric, respectively. Not only that, we also contributes this community a series of new metrics demonstrate the benefits of different message units brings across various collaboration phases in fine-grained manner, where our CPPC still achieves the best performance.
2 PROBLEM FORMULATION
Suppose there are N Nagent agents present in the scene, whose observations and the ground truth an- notation are denoted as {X' Ti=1 mize the perception performance of all agents while taking into account the constraint available
[X'), and ? respectively. Collaborative perception aims to maxi-
2

---

Published as a conference paper at ICLR 2025
Map
Cluster
Extraction
Cluste Fosture (d) Object leature preservation during meessage packing.
Custer
Clusse Certer
(a) Object feature destruction during message parking
Message Packing
Polnt Samplia
Timp
Chanzel Compresitn
Spatial Selection
Denze
Spurso
(b) Inefficient message aggregation for long-range cillaboratoon.
Sange--rreeee efficient massage aggregation.
E3 Bunge
Message Aggregation
B3 Zara Fredlng
Computaiinal compleaciy terressge DOH [<] Empiliiit siructure right fonntation comimniiialli
Compurationa cumplecty formesode tgumeptoon HW ie (0 Explicit structure representation communication.
Collabocation
Message Decading
Colaboraton
lou: 0.3
Sore: lat:
Scace: 0.5 1aU: 0.8
Figure 1: Illustration of using BEV map and point cluster message units for intermediate collaboration.
bandwidth B, which can be formulated as:
arg max 8.M
o( (Mitt
N DELLSAE)
s.t. Σ
IMP
1 (1)
Here, 0 denotes trainable parameters of the network and g(.-)) is the evaluation metric. Mü:t-ri is the message transmitted from the j-th agent at the timestamp L 7--+;: and re- ceived by the i-th agent at the timestamp 4, where Tjーt the transmission latency. includes the j-th agent's is extracted collaborative message units M and 6DoF pose Ç Note that 1) there is no collaboration when 3 = and the objective reflects the single-agent perception performance; 2) when accounting for pose error, it is necessary to correct the pose & for feature alignment; 3) will omit the superscripts related to the timestamp in the following when r = 0 for simplicity.
Existing intermediate collaboration approaches employing dense BEV maps basic message units have several limitations: 1) Object destruction during message packing (Figure (a)). Chan- nel compression Xu al. (2022a) enables the sender to preserve spatially complete scene informa- tion, while the receiver may suffer from object feature degradation during reconstruction due to heterogeneity representation across channels. In contrast, spatial selection Hu et al. (2022); Wang et al. (2023b) transmits only informative regions pointed out by spatial confidence maps, which may result in potential object loss when bandwidth constraints become more stringent. 2) Inefficient message aggregation for long-range collaboration (Figure [ (b)). Collaboration can bring larger perception range to the ego agent, but computational complexity also grows quadratically with the expansion of dense BEV feature maps. Moreover, limited by the square map structure and convo- lution operation requirements, the received BEV features are inevitably filled to the same shape for aggregation with zero paddings, resulting in unnecessary calculation Implicit structure representation communication (Figure The voxelization operation sacrifices 31) geometric details in comparison to the raw point clouds. While aggregating BEV representations from different agents can enhance the response of potential object regions, the precision of predicted box boundaries may be constrained by incomplete object structure modeling.
non-overlapped areas. 3)
Our CPPC based on point clusters for communication can overcome these issues: 1) Object fea- ture preservation during message packing. (Figure | (d)). Point clusters inherently contain only the information of foreground objects present in the scene, eliminating the need for filtering out irrelevant backgrounds by handcrafted rules. Moreover, we can control the transmission bandwidth by explicitly reducing the number of points, as opposed to implicitly compressing feature channels. efficient message aggregation (Figure I (e)). The number of point clusters more related to the number of objects in the scene rather than the collaboration range. Furthermore, point clusters can be conveniently associated with the same object and aggregated through set merg- ing, without the need for padding to the same shape of joint field of view. 3) Explicit structure representation communication ((Figure (f)). Point clusters fully preserve the geometric struc- tural information of objects in the original coordinate space, enabling fine- tine-graieed point alignment and complementary structure information fusion between different agents, which can improve the precision of predictions.
2) Range-irreleve
3

---

Published as a conference paper at ICLR 2025
Puint
Agent
Pose Comrettonn
Thine Cloaae Thelang
Cluster Punn
Point Cluster Encoder
Lavesyy Compensation
Agezi
Shard
Cluster
Trasforstion
0y0
A Aggzz Cluster Graph
Ego Porceptioo
Prediction
Figure 2: Overview of our CPPC system, including shared PCE module to extract point clusters from raw point clouds, a PCP module to selectively packing informative point clusters, and a PCA to complete object information. Pose error and time latency are revised before aggregation phase.
3 METHOD
To begin with, we will present overview of our CPPC framework in $3.1 The we describe details about how we encode, pack, and aggregate point clusters $3.3 and $3.4 respectively, Finally, we will address time latency and pose enror problems to improve our method's 8 robustness in $3.5
3.1 OVERVIEW
Figure 2 illustrates the overall architecture of our proposed method. Initially, the raw point clouds of all agents are processed by shared Point Cluster Encoder (PCE), which segments foreground points on the surface of objects and divides them into clusters based on the distance metric. From each of these point clusters, we extract and formulate the point coordinates, the center coordinates, and the cluster feature as corresponding intermediate representations. Then we propose Point Cluster Packing (PCP) module to filter noisy background clusters and correct the involved points of foreground clusters via proposal generation. The reduction in bandwidth usage of the point cluster can be achieved by decreasing the number included points. After receiving messages from other agents, address the pose eπror and time latency with parameter-free approaches and then align the coordinate space point clusters from multiple agents via pose transformation, which allows the ego agent obtain an agent-cluster graph as the comprehensive scene representation from its own coordinate space. As for message aggregation, we propose a Point Cluster Aggregation (PCA) module, where point cluster matching is performed object and merge them into new point cluster that contains complete object information with linear complexity. Finally, we refine cluster features of point clusters via point-based operators, based on which we output the final detection results.
find point clusters belonging to the same
3.2 POINT CLUSTER ENCODER
Since the original BEV representations for collaborative perception suffer from object feature de- struction, inefficient message aggregation for long-range collaboration, and implicit structure rep- resentation communication, we propose a brand new representation called point cluster to address these issues. We adopt the encoder-decoder point cloud backbone (hi al. (2020b) based on 3D sparse convolution and deconvolution as the sparse voxel feature extractor. To construct each point feature, we concatenate the voxel feature where the point is located and the corresponding off- set from the point to the voxel center. These point features are then passed through a MLP for foreground segmentation. Since objects are naturally well-separated, 3D box annotations in au- tonomous driving scenes directly provide semantic masks for supervision (Shi et al. (201)). We use focal loss [in al. (2017) for segmentation loss, denoted as Leeg. For foreground points, we use another MLP to predict their offsets to the corresponding object centers, which is supervised by LI loss Resn et al. (013)), denoted as Leenter Next, we measure the distance among the predicted centers of foreground points, where two points belong the same point cluster if their predicted centers' Euclidean distance smaller than a certain threshold épeint.
After we directly extract cluster features via point-based operators, such as PointNet Qi et al. (2017)). DGCNN Wang et al. (2019). Meta-Kernell Fan et al. (2021). and SIR Fan et al. (2022). We select SIR in this paper and stack L2 layers to encode all cluster features in parallel. To express more clearly, we take the processing process of the q-th cluster in the 1-th SIR layer as an example. Concretely, assume there are No
ppii foreground points, we denote the included point coordinates N and Foo! point
and features Pe R channels. We take the average coordinates of all predicted cluster centers the cluster center,
No
R Mmmx> respectively, where D is number of feature
4

---

Published as a conference paper at ICLR 2025
denoted as C E R 1x33
The processing process of SIR can be formulated as:
Foil point
Fooi point
P e C²)),
F point = MLP (MM((FFWL)))) Fioi point
maxpool Foint) ))),
(2)
where [:] denotes concatenation along the channel dimension, @ means applying subtraction each point in Pº, and F (Frit 1aa along the channel dimension, and apply linear transformation and max-pooling on it to
point E g XD is the processed point cluster feature. We concatenate
obtain the final cluster feature F E IR IxD.
3.3 POINT CLUSTER PACKING The intermediate representations of point clusters extracted from PCE by the i-th agent can be for- mulated as:
M
{m S
{(Pbg C F(:) Nasa 1q=1
Navae
F==
where m is the representation of the q-th cluster, Netweer is the number of point clusters extracted by the i-th agent, pio denotes the set of included point coordinates, C denotes the cluster center, and Fés denotes the cluster feature. However, during the point cluster grouping process, errors may occur, resulting in the loss of points on the object's surface and the inclusion of background distractors. In order to correct this issue, we propose to generate proposal bounding box for each cluster. To achieve this, we feed the cluster features to two separate MLPs for proposal classification and regression. During the training phase, clusters are classified as positive if their predicted centers are located in the ground truth bounding boxes. We adopt L1 loss Ren et al. (2015) and focal loss Lin et al. (2017) as regression loss Lreg and classification loss Less, respectively. After proposal generation, we only retain point clusters with positive proposals. The point coordinates in these clusters are overridden with the coordinates of points within the positive proposals. Consider there No. cluster+ positive clusters, modify the formulation of M; as following:
M {m 6:0 Ni g-=l
= {(Pig Chi F ())) } Nauoer
(4)
where B (#. ÿ, 2, h, w i, az 2) includes the center coordinates, the size, the yaw angle and the confidence score of the proposal bounding box of the g-th cluster. The final message distributed can be formulated as M = (M'stt) where ç² is the 6DoF pose used for pose transforming in the later aggregation phase.
Optimizing the trade-off between perception performance and communication bandwidth is vital in collaborative perception. Unlike BEV maps, our point clusters are inherently sparse in the spa- tial dimensions. Therefore, the communication cost in our method is mainly on the point coor- dinates (i.e. thousands of 3-dimensional coordinates) rather than the object features (i.e. one 128- dimensional features). Considering that the geometric structure of the object can be represented by keypoints, we propose Semtanic and Distribution guided Farthest Point Sampling (SD-FPS) to compress the transmission data, which can effectively exclude raw points with ambiguous semantic features or redundant structural features. Specifically, not only considering the distance dpoint be- tween points like naive farthest point sampling, but also select keypoints based on the object's semantic confidence score 8; and distribution density score Sμ. The semantic confidence score is derived from segmentation head in PCE, with higher score indicating richer semantic information for distinguishing objects. The distribution density score for point is inversely proportional to its density estimation:
[X(p) EgEN(p) K ((p, q), where A(p) is point set in the nearby area of p. and K(--)) is a gaussian kernel function to measure the similarity between position of two points. Thus, lower indicates removing it will not significantly affect the object's shape. We show the algorithm details of applying SD-FPS on the q-th cluster of the i-th agent in Appendix A.
3.4 POINT CLUSTER AGGREGATION
After message communication, we need to appropriately aggregate point clusters from surrounding agents to form holistic perception. We take the aggregation process from the j-th agent to the i-th agent as example, which can be extended to all agents easily. Firstly, we align the coordinate space of M to that of M' through the transform matrix calculated from 5² and E?. The transformed message units from the j-th agent are denoted Mi Similar to the foreground point grouping process, we match M and Mi-i based on their clusters' centers, where the q-th point cluster of the i-th agent m f and the r-th point cluster of the j-th agent mi-ii belong to the same object if
5

---

Published as a conference paper at ICLR 2025
their centers' distance ||C - -ii-iiill is less than a predefined threshold Cage- After matching, organize all point clusters as two disjoint sets Munqque and M The set Manqqe comprises point clusters exclusively observed by single agent, which do not need to be involved in the following aggregation process. Differently, Mahree is the set of tuples including point clusters belonging to the same object in the scene. We combine each tuple form novel point cluster that encompasses comprehensive low-level and high-level object information. In detail, assume that m= and m +ir belong to the s-th object in the scene, the aggregated point cluster i (Pa, C Fo, B²) can be formulated as follows:
C + Ci-ii
p== pio U Ui Ö
(5)
B Bi-trr
if f - ~- 2
F = avgpool (F Fi+br), B
(6)
otherwise
Since there are no convolution operations and unnecessary zero-padding, the computational com- plexity of aggregating point clusters in our PCA is only strongly related to the number of potential objects, which is more efficient for long-range collaboration than BEV-based aggregation methods.
3.5 ROBUSTNESS
In realistic communication settings, the presence of pose error and time latency is unavoidable and it leads to misalignment of point clouds that significantly affect the reliability of transferred informa- tion in collaborative perception. Benefiting from the low-level object information in point clusters, we propose parameter-fres approaches that can generalize to different noise settings. For simplicity, take the correction processes that happen between the i-th (ego) and j-th agents
example.
Pose Correction. To address this issue, we propose to align the clusters from different agents belonging to the same object. After receiving the message from the j-th agent, we first align the coordinate space of M to the i-th agent with relative pose ę = (E*) o E where means multiplying their homogeneous transformation matrices. We denote m and m as point clusters belonging to the s-th unique object in the scene after spatial cluster matching with threshold épose- In the following, we simplify each pose in 2D space. The pose of the s-th unique object is defined as x" = 5 o (C + /i- Inspired by CoAlign Lu et al. (0023), we define the pose consistency error vector as e C ((E)) x"), which is zero when there is no pose error. The overall optimization problem can be formatted as follows:
Naw Najaa
((')), arg min (E
ΣΣ (ejt) J=1
Different from existing BEV-based methods that need additional detection results for pose correc- tion, we can directly utilize low-level information contained in the point cluster.
Latency Compensation. SyncNet Lei et al. (0022) proposes to complete BEV maps at the current timestamp using locally stored historical BEV maps from other agents. Differently, we propose to directly predict the location of point clusters in the current timestamp via speed estimation based on the low-level coordinate information, which is more efficient and interpretable. We denote the received point clusters from the j-th agent at the 1-th timestamp as M where TJーi is the time latency. The stored point clusters from the j-th agent during the last communication round denoted as 1 where l' < -i Similar the spatial matching process in 3.4
match point clusters along the time dimension by measuring whether their Euclidean distance is in predefined range [Simency [latency]- Assume mirrit and miggt-- belong the same object, can infer its speed pii and offset A Addigit -/g
to the current timestamp t follows:
1hin Oirrsl ||2
Adisgtt +J
+ t'
cluster
Finally, we can obtain the estimated point cluster miti at the 1-th timestamp by coordinates trans- lation with A 1 cluster

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
4.1 DATASETS AND EVALUATION METRICS
We conducted experiments on three widely used benchmarks for collaborative perception, i.e.. V2XSet Xu al. (202a), OPV2V Xu et al. (2022b). and DAIR-V2X-C Yu et al. (2022) We follow previous works Hu et al. (2022) Xu et al. (022a)) to select one of the agents in the scene as the ego agent, whose detection results are assessed by Average Precision (AP) at Intersection- over-Union (IoU) thresholds of 0.5 and 0.7, denoted as AP@0.5 and AP@0.7. Nevertheless, they treat all elements in the scenario uniformly, disregarding the input of collaborative participants. To compare the collaborative perception ability of different methods in more fine-grained manner, we calculate the number of points observed by the ego agent Nego in target objects and categorize them as: 1) Single-agent Perception of Other agents (SP-O), which signifies objects that are scarcely perceived by the ego agent; 2) Single-agent Perception of Ego agent (SP-E), which denotes objects that are more effectively scanned by the individual agent, with additional information provided by other agents primarily serving as supplementary: 3) Collaborative Perception (CP), which indicates objects that are partially observable by the individual agent and require assistance from other agents to obtain comprehensive information about the objects. We evaluate the AP@0.7 of each group, denoted as APsr-o, APcr, and AP SP-E, respectively.
4.2 IMPLEMENTATION DETAILS
We set the perception range along the x, y, and z-axis [-140.8m, 140.8m] x [-40m, 40m] x [-3m, 1m] for V2XSet and [- 100.8m, 100.8m] x [-40m, 40ml x [-3m, 1m] for DAIR-V2X-C, respectively. The communication results measure message size in bytes using logarithmic scale with base 2. The thresholds Cagg- Cpuse> Sattnnyy and Elatenyy for cluster matching are set as 0.6, 1.5, 0.5 and 2.0, respectively. The number of SIR layers L1 = in PCE and L2 = during message decoding. The channel number of cluster features is D = 128. Adam Kingma & Ba (2014) is employed as the optimizer for training our model end-to-end on NVIDIA Tesla VTOO GPUs, with total of 35 epochs. The initial learning rate is set as 0.001 and we reduce it by 10 after 20 and 30 epochs, respectively. Our method is implemented with Py Torch.
Table 1: Comparison with stae-of-the-ar methods on the test sets of V2XSet, OPV2V, and DAIR- V2X-C C-C on perfect setting, The second Table 2: Comparison with state-of- highest performance accuracy is highlighted in blue in the table.
the-art methods on the test set of DAIR-V2X-C with different target
OPV2V
VAX--VX--
categories.
Method
AP810.
AP20.7
APO0.5
A
APBOS
AP8007
DiscoNer al
202in
86.2
13.3
69.23
58.56
Merric APer
Where2cocm Hiu 21
V2X-ViI
Kureral
2022
85.00
36.77
79.02
91.4
82.3
71,48
60.36
Methol
APpro
APaaz
OPV2V Ki 0
207210
91.88
84.75
89.9
76.5
66.07
50.92
WhereZeomm V2X-VIT
38.54
73.17
67.18
COBEVT
Xi ah
2023
90.33
82.09
63.90
51.07
36.85
74.94
67.27
CoAligh
94.5
86.8
74.60
Ours
40.13
82.63
76.72
Oun
92.3
89.35
94.66
93.1
76,89
69.39
4.3 COMPARISON WITH STATE-OF-THE-EE METHODS
VIX-VI (PP33))
Wherloomm (AP205)
Ond (AP90.)
V2X (AP00.))
Whacloam (AP800))
Oan
Average prodision
75
Avenggo procision
Avenge Procision
80
Averge Precision
20
70
&
8
3
8
30
8
16
100
6.5
066
0.2
(a) Communicazion volume (B)
(b) Time latency (124)
(c) Std (m)
(d) Std (*)
Figure 3: Comparison with state-of-the art methods considering (a) performance-bandwidde trade-
off, (b) time latency, (c) positional error, and (d) heading error.
As shown in Table our CPPC outperforms previous BEV-based approaches on the test sets of all evaluated benchmarks, indicating the effectiveness of adopting point cluster as the basic collabora- tive message unit in both simulated and realistic scenarios. Our CPPC outperforms other methods significantly in the most stringent metric AP@0.7, 7, and is 5.7%, 7.3%, and 12.8% higher than the previous best OPV2V, CoAlign. and V2X-VIT ViT on V2XSet, OPV2V, and DAIR- V2X-CV respectively, demonstrating its ability to accurately locate the correct object through collaborative perception and regress more precise bounding box to cover the object. In addition, compare our point cluster
7

---

Published as a conference paper at ICLR 2025
aggregation method with late collaboration, which directly use B" for point clusters matching the same potential objects, i.c., in Mauee. and B for objects exclusively observed by a single agent, i.., in M Munquee as the final outputs, respectively. Our method improves late collaboration by 15.0% and 68.2% on V2X-Set and DAIR-V21 V2X on AP@0.7. This finding confirms that our PCA module can effectively utilize rich object irformatioo-sseciffcaall specificallyy semantic features and structural infor- mation-in point clusters received from surrounding agents, enabling the discovery of objects that cannot be accurately detected by individual agents.
trade-off of our CPPC and previous typical methods in Fig- ure 3 (a). Thanks to the sparse nature of point clusters, the communication volume under 16 even if we pack all cluster points. which is close to the lower bound of bandwidth usage of other methods. Under the same communication volume, our method achieves significant performance im- provements of around absolute 10.0 AP@0.7. The little performance drops with bandwidth decrease indicate that appropriate point sampling strategies can reduce representation redundancy.
We explore the iefformance-bandwine
Time latency poses a pervasive challenge in real-world V2X communication, leading to asynchro- nization between ego features and received collaborative features. We compare the model robustness against time latency ranging from 0 to 500ms in Figure 3 (b). Compared to previous interme- diate methods, our CPPC achieves superior performance, with similar AP degradation as latency increases. It is worth noting that both V2X-Vi1 ViT and Where2comm need to be finetuned with data on different noisy levels, while our CPPC can adapt to arbitrary noise levels in a zero-shot manner.
Collaborative agents depend precise pose data from others to transform coordinates of re- ceived messages. Despite advanced localization technologies like GPS, pose error is unavoid- able. Therefore, collaborative approaches need be resilient localization errors. We compare different methods with pose noises following Gaussian distribution with standard deviations from {0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6} for positional error (m) (Figure and heading error (*) (Figure (d)). respectively. Results show that there is no significant AP drop. which validates that our CPPC can handle large noise disturbances without further finetuning and additional model parameters.
The existing evaluation metrics in collaborative perception treat all objects equally, which is not conducive to fine -me-graiedd analysis. As shown in Table [ we split targets into different cate- gories based on the number of points scanned by the ego agent and evaluate 0.@ for each category, denoted as APSP-O. APcp, and APsP-E. Experimental results demonstrate the superiority of our CPPC over the state-of-the--e BEV-based approaches across all evaluation metrics. In detail, high APSP.o means that PCP module keep more complete object information since we avoid feature destruction caused by channel compression and spatial selection. The results in the 2-nd column demonstrate the significant improvement of our CPPC over BEV-based methods in terms of the APcp metric. This indicates that utilizing point clusters as the basic collaborative message unit is advantageous for both message packing and aggregation phases, resulting enhanced collaboration compared to earlier techniques relying on dense BEV maps.
Table 3: Different numbers of feature channels in PCE. Channel Number
Metric
64
AP
74.41
75.32
77.12
76.63
77.30
70
64.79
67.08
69.32
68.92
69.55
69.43
4.4 ABLATION STUDIES
Number of Feature Channels in PCE. We assess AP@ 50 and AP@ 70 using various chan- nel numbers of cluster features PCE on the test set of DAIR-VE -AX---VX-CCn in Table 3 The results indicate that there is no notable decrease in per- formance when the number of channels is re- duced to 16. We argued that since we explicitly include structure representation in point clusters, the semantic information can be compressed to large extent, leading to small bandwidth consumption. In contrast to prior approaches that compress solely before the message packing phase, our framework incorporates a small number of channels throughout the entire encoding phase, thereby decreasing network size and computational overhead without destroying object features across channels (3-rd and 4-st rows).
Table 4: Different sampling ratios and methods.
Mebhod
1/198
1/64
1/32
1/16
1/8
1/4
RPS
64.60
67.80
66.61
08
68.5
S49S
66.21
67.08
69.01
89.05
D-FP5
65.65
66.52
67.51
63.57
69:21
69.16
SD.FPS
66.12
67.05
68.08
09.22
09.19
69.41
Sampling Point Clusters with Different Ratios and Methods. We evaluate AP@ 70 of different sampling ratios and methods during message packing on the test set of DAIR- V2X-C in Table 4 The ratio is the number of sample points divided by the total number of points. The first and second
8

---

Published as a conference paper at ICLR 2025
Table 5: Ablation experiments on Epoints Eagg, Epose. and Elatency.
Faaner
0.1
0.2
0.3
0.4
0.3
0.6
0.7
1.0
1.5
2.0
1.5
2.0
2.5
A.er0.5
91.94
92.01
91.92
91.75
91.47
51.97
91.95
91.99
92.01
91.97
89.86
90..00
89.80
72.50
72.69
72.68
AP
89.86
89.99
86.93
64.61
64.59
rows represent the baseline methods of random sampling and basic FPS. The second, third, and
fourth rows represent the results of introducing the semantic score, density score, and their joint application. It can be seen that the semantic score performs better than the baseline under smaller sampling ratios, because it preserves the semantic of object categories in extremely sparse structure. The density score performs better than the baseline under larger sampling ratios, because it can remove local redundant information. By combining the two scores, our method has significant performance advantages over the baseline at all sampling ratios. Moreover, SD-FPS does not incur significant time costs, less than 3% of time consumption in entire pipeline (Details in Apendix ( The reason is that our method applies SD-FPS to each point cluster, which contains limited object- level points (about average 10 points in the V2X-Set dataset) rather than large-scale scene-level points (about average 10* points in the V2X-Set dataset).
Ablation on Hyperparameters We evaluate AP@O.5/AP@O.7 of different lower bounds Gatency for matching during latency compensa- tion in Table 6 In line with Figure 3 (b), an increase in time latency may result in per- formance degradation due to notable position shifts that complicate temporal alignment. If our method encounters relatively high time la- tency (e.g., 300ms, 400ms, and 500ms), performance significantly decreases when Clatency 0 or
Table 6: Lower bound for matching during latency compensation.
Time Latency
100
200
300
4001
500
36367
74 EETSSAEn
75.
7134m1 79
34667.8
75.39066
78.20663.53
0.2
76. 16.34667.5 88
75.21/66
74. 1364,.99
73. 1716366346
72.37/54.00 00
0.3
76,
13.99666
14/644
73. 19963.50
72.72/64-00 46
0.4
76.42/68.01
75.26/6605
73,50064.42 14.11/94.55
72.76/64.47
0.5
76,44168.01
75.25/65.34
74.3666.15
73,79964.2
72/75/64-48
is very low. Our research revealed that certain vehicles remain stationary, rendering the assumption of uniform motion in Section invalid. By adjusting Siateney as demonstrated in the 3-rd, 4-th, and 5-th columns, we can mitigate this issue by filtering out stationary targets during latency compen- sation. Other involved hyperparameters i.e., Epoint Cagg. Eposes and Elatency, in our CPPC are evaluated with different values in T 4.5 QUALITATIVE ANALYSIS
As shown in Figure uation of our CPPC w/ and w/o collaboration on the test sets of both V2XSet and DAIR- DAIR-V2X-C datasets. We can observe that the absence of collaboration leads to the overlooking of many objects to the ego agent's inability to perceive sufficient information regarding long-range or obscured targets. These issue can be effectively addressed using our message packing and ag- gregation mechanisms based on point clusters.
4 we show qualitative eval-
We also visualize the detection results before and after latency compensation on the test set of the DAIR-V2X-C dataset in Figure [ The latency results in a delay of point clusters from road infrastructure (marked by yellow dots) compared to the point clusters from the ego car (marked by blue dots) at the current timestamp. Consequently, the ego car erroneously detects them as distinct objects, resulting in an escalation of false- passiiooitivve predictions. After refinement by our latency compensation module, the delayed point clusters can be adjusted to the correct positions, enhancing the detection results. Further, we illustrate the detection results before and after pose cor- rection on the test set of the V2XSet dataset in Figure 6 By aligning clusters among all agents, we can correct the noisy poses and obtain bounding boxes with high precision.
collabrrriii
Figure 4: Qualitative comparison results of CPPC with and without collaboration on the test sets of (a) V2XSet and (b) V2X-C. respectively. The green bounding boxes represent the ground-truth, and the red ones depict our predictions.
5 RELATED WORK
5.1 COLLABORATIVE PERCEPTION
Collaborative perception can be systematically classified into three primary types based on distinct fusion stages: early Chen al. (2019h); Aroold et al. 2000)) intermediate, and late Shi et al. (002))

---

Published as a conference paper at ICLR 2025
B8
8
Refirenani
Afler
Figure 5:
Qualitative comparison results of our Figure 6: Qualitative comparison results hefore and CPPC before and after latency compensation.
after pose correction.
Zeng et al. 2 Rauch et al. (2012); Glaser & Kira (023) collaboration. We focus on interme- diate collaboration in this paper, which enables the exchange of intermediate features created by the involved agents and has demonstrated significant potential in recent years. Considering the band- width constraints in practical application scenarios, various cooperation strategics proposed to
decide who Liu et al. (2020b), when Liu et al. (2020a), and where Hu et al. (2022) ; Wang et al.] (2023b) communicate. After receiving features from other agents, existing methods adopt atten- tion mechanism Xu et al. (2022b a a Zhang et al. (2022a); Wang et al. (2023a): Yang ct al.
(2023), graph neural network Wang al. 2020); Li et (202 la), maxout Bai al. (0222); Guo
et al. (202T); Qiao & Zulkernine (2023), and addition Marvasti et al. (2020) to aggregate comple- mentary scene information. This paper proposes novel collaborative message unit named point cluster and demonstrates its superiority over the BEV map for collaborative perception.
5.2 SPARSE DETECTORS
To address quantization errors caused by voxelization, point- based detectors Shi et al. 22019; Yang al. (2020); Qi al. (2019): Zhang et al. (20226); Qi et al. (2017b); Faan et al. (2022) Chen et al. 22033; Huang et al. (2023); Fu et al. (2024) have emerged
popular research topic. PointR- CNN al. (2019) is recognized as groundbreaking research in the advancement of this line of work. Taking inspiration from Hough voting, VoteNet Qi et al. (2019) initially casts votes for object centroids and subsequently generates high-quality proposals based on the voted center. FSD Fan et al. (2022) is the pioneering fully sparse 3D object detector, which treats instances as groups and gets rid of the dependence on the neighborhood query. In this paper, we study the shortcomings of BEV-based collaboration methods and propose the point cluster as the collaborative message unit, which keeps the low-level structure information to support effective and efficient collaboration.
CONCLUSION AND DISCUSSION
In this paper, we concentrate on the multi multi-agent collaborative perception task, illustrating that current approaches hampered by the inherent limitations of existing message units leading to sub-optimal collaboration. To handle this issue, we create brand new message unit for collaborative perception, namely point cluster, and based this further present novel collaborative framework CPPC. The core idea of CPPC involves representing scenes through object-level point clusters, which are sparse and encompass comprehensive information about objects. These clusters can be compressed efficiently without losing geometric or high-level object information and can be integrated through set matching and merging. To deal with time latency and pose errors encountered ments, we align point clusters from spatial and temporal dimensions and propose parameter-fre free solutions for them. Extensive experiments on two collaborative perception benchmarks show our method outperforms previous state-of-the-an methods.
real environ-
Limitation and future work. By leveraging the low-level information within point clusters, we introduce farameter-free solutions to enhance the robustness of our approach. Despite good results, we may need to manually adjust hyperparameters when facing new environment. In the future, plan to address this issue using novel techniques specifically designed for adapting to environ- mental changes. addition, we also plan to extend our CPPC to handle tasks considering temporal modeling, like tracking and forecasting Yu 20033
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research supported in part by National Key R&D Program of China (2022D00115002) Na- tional Natural Science Foundation of China (NO. 62461160306 U23B2010), "Pionee" and "Lead- ing Goose" " R&D Program of Zhejiang (No. 2024C01161).
REFERENCES
Eduardo Arnold, Mehrdad Dianati, Robert de Temple, and Saber Fallah. Cooperative perception for 3d object detection in driving scenarios using infrastructure sensors. IEEE Transactions Intelligent Transportation Systems, (3(3):1852-186) 2020.
Zhengwei Bai, Guoyuan Wu, Matthew Barth, Yongkang Liu, Emrah Akin Sisbot, and Kentaro Oguchi. Pillargrid: Deep learning- carding-based cooperative perception for 3d object detection from onboard-roadside lidar. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), pp. 1743-1749. IEEE, 2022.
Qi Chen, Xu Ma, Sihai Tang, Jingda Guo, Qing Yang, and Song Fu. F-cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3d point clouds. In Proceedings of the 4th ACM/IEEE Symposium on Edge Computing, pp. 88-100, 2019a.
Qi Chen, Sihai Tang, Qing Yang. and Song Fu. Cooper: Cooperative perception for connected autonomous vehicles based 3d point clouds. In 2019 IEEE 39th International Conference Distributed Computing Systems (ICDCS), pp. $14-524. IEEE, 2019b.
Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi. and Jiaya Jia. Voxelnext: Fully sparse voxelnet for 3d object detection and tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11674-21688 2023.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1290-1299, 2022.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pp. 1-16. PMLR, 2017.
Lue Fan, Xuan Xiong, Feng Wang. Naiyan Wang, and Zhaoxiang Zhang. Rangedet: In defense of range view for biser- 3d object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2918 2918-2927, 2021.
Lue Fan, Feng Wang, Naiyan Wang, and ZHAO XIANG ZHANG. Fully sparse 3d object detection. Advances in Neural Information Processing Systems, 35:351-363 2022.
Jiahui Fu, Chen Gao, Zitian Wang, Lirong Yang, Xiaofei Wang, Beipeng Mu, and Si Liu. Eliminat- ing cross-modal conflicts in bev space for lidar-camera 3d object detection. In ICRA, 2024.
Nathaniel Moore Glaser and Zsolt Kira. We need to talk: Identifying and overcoming communication-critte scenarios for self-driving arXiv preprint arXiv:2305.0433 2023.
Jingda Guo, Dominic Carrillo, Sihai Tang, Qi Chen, Qing Yang, Song Fu, Xi Wang, Nannan Wang, and Paparao Palacharla. Coff: Cooperative spatial feature fusion for 3-d object detection on autonomous vehicles. IEEE Internet of Things Journal, 0(14):11788 1087, 2021.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Yue Hu, Shaoheng Fang, Zixing Lei, Yigi Zhong, and Siheng Chen. Where2comm: Communication-effce -ffficient collaborative perception via spatial confidence maps. Advances neu- ral information processing systems, 35:4874-4888 2022.
Zhe Huang, Yongcai Wang, Jie Wen, Peng Wang, and Xudong Cai. An object detection algorithm combining semantic and geometric information of the 3d point cloud. Advanced Engineering Informatics, 56: 2023.
11

---

Published as a conference paper at ICLR 2025
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990, 2014.
Zixing Lei, Shunli Ren, Yue Hu, Wenjun Zhang, and Siheng Chen. Latency-aware collaborative perception. In European Conference on Computer Vision, pp. 316-332. Springer, 2022.
Yiming Li, Shunli Ren, Pengxiang Wu, Siheng Chen, Chen Feng, and Wenjun Zhang. Learning dis- tilled collaboration graph for multi-agent perception. Advances in Neural Information Processing Systems, 34:29541-29552, 2021a.
Yiming Li, Dekun Ma, Ziyan An, Zixun Wang, Yiqi Zhong, Siheng Chen, and Chen Feng. V2x- sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving, IEEE Robotics and Automation Letters, 7(4): 199141091,- 2022.
Zhi Li, Ali Vatankhah Barenji, Jiazhi Jiang, Ray Y Zhong, and Gangyan Xu. A mechanism for scheduling multi robot intelligent warehouse system face with dynamic demand. Journal of In- telligent Manufacturing, 31 31:4944880 2020.
Zhichao Li, Feng Wang, and Naiyan Wang. Lidar r-cmn: An efficient and universal 3d object detec- tor. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7546-7555, 2021b.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017.
Yen-Cheng Liu, Junjiao Tian, Nathaniel Glaser, and 7solt Kira. When2com: Multi-agent perception via communication graph grouping. In Proceedings of the IEEE/CVE Conference on computer vision and pattern recognition, pp. 4106-4115, 2020a.
Yen-Cheng Liu, Junjiao Tian, Chih- Yao Ma, Nathan Glaser, Chia-Wen Kuo, and Zsolt Kira. Who2com: Collaborative perception via learnable handshake communication. In 2020 IEEE International Conference Robotics and Automation (ICRA), pp. 6876-6883, IEEE, 2020b.
Yifan Lu, Quanhao Li, Baoan Liu, Mehrdad Dianati, Chen Feng, Siheng Chen, and Yanfeng Wang. Robust collaborative 3d object detection in presence of pose errors. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 4812-4818. IEEE, 2023.
Ehsan Emad Marvasti, Arash Raftari, Amir Emad Marvasti. Yaser Fallah, Rui Guo, and Hong- sheng Lu. Cooperative lidar object detection via feature sharing in deep networks. In 2020 IEEE 92nd Vehicular Technology Conference (VTC2020-Faa) Fall), pp. 1-7. IEEE, 2020.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedingy of the IEFE conference on computer vision and pattern recognition, pp. 652-660, 2017a.
Charles R Qi, Or Litany, Kaiming He, and Leonidas Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 9277 9277-9286, 2019.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical fea- ture learning point sets in metric space. Advances in neural information processing systems, 30, 2017b.
Donghao Qiao and Farhana Zulkernine. Adaptive feature fusion for cooperative perception using lidar point clouds. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pp. 1186-1195, 2023.
Andreas Rauch, Felix Klanner, Ralph Rasshofer, and Klaus Dietmayer. Car2x-based perception in a high-level fusion architecture for cooperative perception systems. 2012 IEEE Intelligent Vehicles Symposium, pp. 270-275. IEEF, 2012.
12

---

Published as a conference paper at ICLR 2025
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Jürgen Scherer, Saeed Yahyanejad, Samira Hayat, Evsen Yanmaz, Torsten Andre, Asif Khan, Vladimir Vukadinovic, Christian Bettstetter, Hermann Hellwagner, and Bernhard Rinner. An autonomous multi-uav system for search and rescue. In Proceedings of the first workshop on micro aerial vehicle networks, systems, and applications for civilian use, pp. 33-38, 2015.
Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 770-779 1000-79,220 2019.
Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang. Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn: Point-voxel feature set abstraction for 3d object detection. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 05529-10538, 2020a.
Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts:
3d object detection from point cloud with part-aware and part-aggregation network. IEEE trans- actions on pattern analysis and machine intelligence, 43(4):2647-2644, 2020b.
Shuyao Shi, Jiahe Cui, Zhehao Jiang, Zhenyu Yan, Guoliang Xing, Jianwei Niu, and Zhenchao Ouyang. Vips: Real-time perception fusion for cntasttrctuur-assiiteer autonomous driving. In Proceedings of the 28th Annual International Conference Mobile Computing And Networking.
pp. 133-146, 2022.
Binglu Wang, Lei Zhang. Zhaozhong Wang, Yongqiang Zhao, and Tianfei Zhou. Core: Cooperative reconstruction for multi-agent perception. arXiv preprint 2rX.v:2307.I1. /15/3007.11..,.... 2023a.
Tianhang Wang. Guang Chen, Kai Chen, Zhengfa Liu, Bo Zhang, Alois Knoll, and Changjun Jiang. Ume: A unified bandwidth-efficient and multi-resolution based collaborative perception frame- work. arXiv preprint ar 2rv::3333 12400, 2023b.
Tsun-Hsuan Wang, Sivabalan Manivasagam, Ming Liang, Bin Yang, Wenyuan Zeng, and Raquel Urtasun. V2vnet: Schicle-to-vehile communication for joint perception and prediction. In Com- puter EEctonECCVVVA 2020: 16th European Conference, Glasgow, UK, August 23 28,28, 2020, Pro- ceedings, Part II 16, pp. 605-621. Springer, 2020.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning point clouds. ACM Transactions Graphics (tog), 38(5): 1-12, 2019.
Sizhe Wei, Yuxi Wei, Yue Hu, Yifan La, Yiqi Zhong, Siheng Chen, and Ya Zhang. Asynchrony- robust collaborative perception via bird's eye view flow. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Runsheng Xu, Yi Guo, Xu Han, Xin Xia, Hao Xiang, and Jiaqi Ma. Opencda: an open cooper- ative driving automation framework integrated with co-simulation. 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 1155-1162. IEEE, 2021.
Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang, and Jiaqi Ma. V2x-vit: Vehiele-to-everything cooperative perception with vision transformer. In European conference on computer vision, pp. 107-124. Springer, 2022a.
Runsheng Xu, Hao Xiang. Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma. Opv2v: An open bench- mark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. In 2022 International Conference on Robotics and Automation (ICRA), pp. 2583-2589. IEBE, 2022b.
Runsheng Xu, Zhengzhong Tu, Hao Xiang, Wei Shao, Bolei Zhou, and Jiagi Ma. Cobevt: Coop- erative bird's 's eye view semantic segmentation with sparse transformers. In Conference on Robot Learning, pp. 989-1000. PMLR, 2023a.
13

---

Published as a conference paper at ICLR 2025
Runsheng Xu, Xin Xia, Jinlong Li, Hanzhao Li, Shuo Zhang, Zhengzhong Tu, Zonglin Meng, Hao Xiang, Xiaoyu Dong, Rui Song, et al. V2v4real: A real-world large-scale scale dataset for vehicle-to- vehicle cooperative perception. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 13712-13722, 2023b.
Kun Yang, Dingkang Yang, Jingyu Zhang, Mingcheng Li, Yang Liu, Jing Liu, Hanqi Wang, Peng Sun, and Liang Song. Spatio-temporal domain awareness for multi-agent collaborative percep- tion. arXiv preprini arXiv:2307. 13929, 2023.
Zetong Yang. Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point- based 3d single stage object detector. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11040-1 1048, 2020.
Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi. Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, al. Dair-v2x: A large-scale dataset for vehiele-infrastructure cooperative 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21361-21370, 2022.
Haibao Yu, Wenxian Yang, Hongzhi Ruan, Zhenwei Yang. Yingjuan Tang, Xu Gao, Xin Hao, Yifeng Shi, Yifeng Pan, Ning Sun, et al. V2x-seq: A large-scale sequential dataset for vehicle- infrastructure cooperative perception and forecasting. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 5486-5495, 2023.
Wenyuan Zeng, Shenlong Wang, Renjie Liao, Yun Chen, Bin Yang. and Raquel Urtasun. Dsdnet: Deep structured self-driving network. In Computer Vision-ECCC 2020: 16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pp. 156-172. Springer, 2020.
Hui Zhang, Guiyang Luo, Yuanzhouhan Cao, Yi Jin, and Yidong Li. Multi-modal virtual-real fusion based transformer for collaborative perception. In 2022 IEEE 13th International Symposium Parallel Architectures, Algorithms and Programming (PAAP), pp. 1-6. IEEE, 2022a.
Yifan Zhang, Qingyong Hu, Guoquan Xu, Yanxin Ma, Jianwei Wan, and Yulan Guo. Not all points equal: Learning highly efficient point- basid detectors for 3d lidar point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18953-18962, 2022b.
A METHOD DETAILS
A.1 ALGORITHM: SEMTANIC AND DISTRIBUTION GUIDED FARTHEST POINT SAMPLING
A.2 POINT CLUSTER DECODING
We denote the intermediate representation of point clusters in the scene after aggregation M [m*) Nates where Nobject = || + Mahreel is the potential object number the scene observed by all involved agents. Different from BEV-based message aggregation methods, p= contains complete low-level structure information, which can be utilized to enhance the precision of the proposal bounding box B. detail, we apply an additional SIR module that includes L2 layers on M, which predicts the box residual A to corresponding ground truth box. For each point cluster m we generate its point feature Foin by concatenating offsets from the cluster proposal 8= and the cluster feature F. This introduces the proposal boundary information to the SIR module, which can handle the size ambiguity problem to a certain extent Li et al. (2021b). We define the residual loss Lres as the LI distance between Are and the ground-trun -ruth residual Å In addition, define the soft classification label min (1, max(0, 2u 0.5)) following previous works Shi et al.] (2000a b), where u is the 3D Intersection-of Union (IoU) between the predicted proposal and the ground truth. We adopt cross-entroy loss the IoU loss Lion- Considering all losses in our framework, the total training loss L can be formulated as:
L Lieg + Leenter freg + Lete + Lies Liou-
(9)
14

---

Published as a conference paper at ICLR 2025
Algorithm Semantic and Distribution guided Farthest Point Sampling Algorithm. Npoi is the number of input points and Numple puin X is the number of sampled points controlled predefined sampling rate
Input: coordinates P= {p' ..... P p00) E R poin x3, semantic scores St = (8),.... ..... 6f
Mpmn J R n Npoin min.. Namule)
distribution scores Sa {sa. (sa..... S
Output: sampled key point set
(F)
1: initialize empty sampling point set P
2: initialize distance array pain of length Npoir with all +00 initialize a visit aray V length Nppii with all zeros
4: for n to Nample do
5: if n = then
6:
arg max(1(())) (a ( = 0 21
else
puine = {( (s a diuiinl V+ k]
pain
9:
arg max D poim)
10:
end if
11: add po P. =
12: for
to N do dipin. min (dppii::: -r"")
13:
14:
end for
15:
end for
16:
return P
B DATASETS
DAIR- V2X-C Yu al. (022 is the first provide large-scale collection of real-world scenar- ios for chhicl--iffrastructure collaborative autonomous driving. It contains 38,845 frames point cloud data annotated with almost 464k 3D bounding boxes representing objects in 10 different classes. Since the original DAIR-V2X-C does not include objects beyond the camera's view, we have adopted the complemented annotations encompassing the 360-degree detection range, which are relabeled by Hu el al. Hu et al. (2022).
v2XSer Xu et al. 2022al is large-scale V2X perception dataset founded CARLA Dosovitskiy et al. (2017) and OpenCDA Xu et al.] (202T), which explicitly takes into account real-world noises like localization error and transmission latency. V2XSet has 11,447 frames (6.694/ 1,920/2,833 for train/validation//
respectively) captured in 55 representative simulation scenes that cover the most common driving scenarios in real life. Each scene typically involves 2-7 agents engaged in collaborative perception.
OPV2V Xu OpenCDA Xu et al. (202T) and Carla [Dosovitskiy et al. (2017). which includes 12K frames of 3D LiDAR point clouds and RGB images with 230K annotated 3D boxes.
al. (20226) vehicle-to-vehicle collaborative perception dataset, cosimulated by
C EXPERIMENTS
C.I DATASET STATISTICS.
C.2 COMMUNICATION VOLUME.
In real-world applications, collaborative perception methods must achieve delicate equilibrium be- tween communication volume and precision due to the typically limited and variable communication bandwidth. The communication volume is calculated as follows:
Comm log,(N C 16/8),
(10)
where N represents the number of collaborative message units, C represents the number of channels, and the data is transmitted in fp16 data type, resulting in minimal performance impact. The volume in bits is then converted to bytes using the logarithm base 2.
15

---

Published as a conference paper at ICLR 2025
SP-
CP
SP-E
1200
(37%)
(31%)
(32%)
10000
8000
6000
4000
2000
0-10
10-20
20-30
30-50
50-70
70-90
90-110
110-130
130
130-150
1s0+
Figure 7: Illustration of histogram of all largets and proportions those belonging SP-O. SP-E, and CP
categories in the test of DAIR-V2--C.
C.3 COMPUTATIONAL COMPLEXITY OF POINT CLUSTER AGGREGATION
The computational complexity discussed primarily aims to demonstrate the efficiency of message aggregation based on sparse point clusters in comparison to dense BEV map representations. The computational complexity of message aggregation based on point clusters is solely related to the number of potential objects, which is significantly smaller than the scale of the quadratic relationship with the collaboration scope.
Assume there are Nagent agents in the scene, so each tuple in Manare contains at most Nagent matched point clusters. According to (5) and eq (6). the computational complexity of aggregating single tuple in M in point cluster and and C' is the channel number of cluster features. After point sampling during point cluster packing. Pi| can be reduced significantly as shown in Table And Table shows that C' can also be reduced to small constant. Overall, the computational complexity of aggregating single tuple in Mature can be reduced to (D), where D is a constant. Assume there are Notjeet possible objects in the scene, there at most Notject tuples in Msture, while Munique 0. The computational complexity of aggregating all tuples is (DNaajjaa..
pi N aggent + C. + Nagem). where p is the set points
C.4 ADDITIONAL ABLATION STUDIES
Table 7: Features for object pose calculation during pose correction. "0.1" label of each column means experiments with standard deviation 1.1 for both heading error (9) and positional error (m).
Pose Error
Feature
0.1
0.2
0.3
0.4
0.5
Point Center
13.15/45.44
76.89/46.59
78.50/48.22
79.60/51.03
78.90/52.07 89.97/86.94
Cluster Center
90.22/86.95
90.34/87.00
90.04/87.06
90.10/87.06
Features for Object Pose Calculation during Pose Correction. We evaluate AP@0.5/AP@0.7 of different features for object pose calculation during pose correction in Table 7 "Point Center" denotes representing the object pose with the mean coordinates of cluster points. "Cluster Center" denotes representing the object pose with the estimated cluster center. The experiments demon- strate that utilizing the "Point Center" method for determining the object's 'ss pose correction results in notable performance deterioration. Due Lidar typically scanning objects partially, directly rep- resenting a point cluster by averaging the coordinates of all cluster points can lead to significant offset from the true object center.
Filtering background information. We evaluate the recall of points belonging to target objects after segmentation. The results demonstrate that segmentation can recall more than 90% of target object points. This finding confirms that point clusters generated from segmentation results ef- fectively represent potential object parts, enabling the fusion module to complete object information utilizing information from all agents and achieve better performance.
16

---

Published as a conference paper at ICLR 2025
Table 8: Time cost (mss analysis.
CPPC
VIT
Total
PCP
PCA
Decoder
Others
153.31
79.52
53.71
3.21
9.69
12.70
0.21
Table 9: Different based operators.
AP@0.5
AP@0.7
PointNet
89.67
85,82
DGCNN
91.39
87.68
Meta-Kernel
91.52
88.48
STR
92.01
89.99
Inference time cost analysis. We evaluate the time cost of method's entire pipeline and com- pare it with one the state-of-the-art methods in Table 8 i.c., V2X-VIT. Comparing the first and second columns, the results show that method significantly reduces the time cost compared with previous collaborative sensing methods, indicating the efficiency communication with our new message unit point cluster. Moreover, the collaborative modules (PCP and PCA) occupy relatively small portion of our entire pipeline, which verifies collaboration based on point cluster can impose little additional burden existing single-vehicle detectors.
Different point- operators. We evaluate AP@ 70 of different existing point-based operators on V2XSet in T where SIR achieves the best performance.
C.5 QUALITATIVE COMPARISON
V2X-VIT
Where2comm
Ours
(c)
(d)
Figure 8: Qualitative comparison results of our CPPC with state-of-the-art methods on V2XSet (a, b) and
DAIR-V2X-CC (c, d). The green bounding boxes represent the ground truth, and the red ones depict predictions.
17


Reviews:
[{"review_id": "o47kP5cx7w", "reviewer": "Reviewer_tujs", "summary": "This paper presents a new message unit, the \"point cluster,\" to improve collaborative perception efficiency in multi-agent systems. Unlike existing message units such as raw point clouds, bounding boxes, or BEV maps, the point cluster format minimizes bandwidth usage while retaining essential structural and semantic information. Representing objects with point coordinates, a cluster center, and semantic features, this approach allows efficient inter-agent information exchange, enhances object alignment, and preserves object structure for more accurate detection. A new framework, CPPC, combines point packing and aggregation modules, addressing issues like bandwidth constraints, time delay, and pose errors, and achieves state-of-the-art performance on several benchmarks.", "questions": "No questions.", "limitations": "", "rating": 6, "confidence": 1, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Bandwidth Efficiency**: The proposed \"point cluster\" message unit significantly reduces communication bandwidth by capturing only essential foreground object information in a sparse format, making it highly efficient compared to dense representations like raw point clouds and BEV maps.\n\n2. **Enhanced Object Detection Accuracy**: By preserving detailed structural and semantic information, the point cluster improves object detection accuracy, especially in complex multi-agent scenarios, demonstrating superior performance on established collaborative perception benchmarks.\n\n3. **Robustness to Real-world Challenges**: The CPPC framework includes robust mechanisms for handling pose errors and time delays, crucial for real-world applications. Parameter-free solutions for pose and latency issues make it adaptable to various levels of noise without additional tuning.\n\n4. **Clear and Effective Writing**: The paper is well-written, with a clear explanation of the proposed methods and thorough descriptions of experiments, which makes the complex technical content accessible and supports the credibility and reproducibility of the research findings.", "weaknesses": "In terms of methodology, there is a noticeable reliance on FSD, which slightly reduces the originality. However, overall, the approach is still reasonable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SAyo0Ietdo", "reviewer": "Reviewer_1x7n", "summary": "This paper proposes a novel message exchange unit called point cluster for collaborative perception. Point clusters can efficiently and compactly represent an object's location, structure, and semantic information. The proposed CPPC framework includes point cluster-based encoding, packing, exchange, and integration methods. CPPC improves both communication efficiency and perception accuracy while being robust to various real-world noises. Experimental results on various benchmarks demonstrate that CPPC significantly outperforms existing methods.", "questions": "If we assume some objects are moving at high speeds, I think the errors caused by delay would vary for each object. It seems this aspect might not be covered, does the proposed method handle this problem?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper systematically analyzes the advantages and disadvantages of existing collaborative perception methods (early, intermediate, late collaboration) and proposes a rational solution considering the inherent trade-off between communication efficiency and perception performance. \nThe proposed point cluster is a compact message unit that selectively combines the strengths of existing methods and efficiently represents both structural and semantic information of objects. \nCPPC introduces parameter-free approaches to handle pose error and time latency issues encountered in real-world scenarios, ensuring robustness to various noise levels. \nThis paper clearly identifies the current challenges and limitations in the field of collaborative perception and demonstrates how the proposed method can rationally solve them.\nThrough clear problem formulation, the collaborative perception problem is mathematically defined, and based on this, the motivation and principles of the proposed method are lucidly explained.", "weaknesses": "Based on the proposed technique, the amount of data that can be transmitted is limited. Therefore, it is crucial to prioritize foreground objects and bring them in a sparse manner to distinguish the overall shape, rather than bringing only a few parts. Regarding this aspect, I wonder if there are any criteria or tendencies for determining which points should be selected and what rules should be followed. Additionally, I am curious about the extent of performance degradation when background information is included instead of solely focusing on the foreground. \n\nIf we follow the logic mentioned above, I wonder if we can expect sufficient performance improvement in early or late collaboration by only bringing foreground information and adjusting it according to the available bandwidth. \n\n- In Figure 4(a), there is a section where the performance of the proposed method rises sharply. I am curious about the reason behind this phenomenon and how the performance of the proposed method would differ from existing technologies if the communication volume is lower than this point. \n- On Page 7, Line 373, it is mentioned that the proposed method shows improvements of 5.7%, 7.3%, and 12.8%, but it is unclear which methods are being compared. \n- On Page 7, Line 377, the term \"late collaboration\" is used. I am curious about which specific technique this refers to.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rlwppfD3aB", "reviewer": "Reviewer_xrYN", "summary": "This paper introduces a communication-efficient collaborative perception (CPPC) framework for vehicle-to-everything autonomous driving. Unlike previous methods that mainly rely on BEV features, the CPPC framework leverages point clusters to control its computational complexity and alleviate issues like high-level information loss. Specifically, the CPPC framework consists of a point cluster picking module, a pose alignment and latency compensation module, and a point cluster aggregation module to generate cluster features for subsequent predictions. The CPPC framework outperforms existing BEV-based methods on three public datasets, including V2XSet, OPV2V, and DAIR-V2X-C.", "questions": "My current concerns are mainly about the technical contributions of the proposed method. I may change my rating after reading other reviewers' comments and the authors' rebuttal. Here are some questions I wish could be addressed:\n\nQ1: In the proposed SD-FPS method, how to balance the weights of semantic and spatial distances ($\\lambda_s$ and $\\lambda_d$)? How will they affect the performance of the proposed method?\n\nQ2: Does the proposed method perform well in a crowded environment where many objects are exhibited?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "\\+ This paper is written well and the motivation behind the proposed method is easy to follow.\n\n\\+ The proposed method is a systematic solution for multi-agent perception, which exploits point clusters as the intermediate representation with controllable communication costs, to balance efficiency and effectiveness. \n\n\\+ The proposed method outperforms several previous methods on three public datasets consistently. The extensive experimental results validate the effectiveness of the proposed modules.", "weaknesses": "\\- There are four thresholds used in the proposed method, which may be hard to choose in complex or unseen scenarios.\n\n\\- The proposed method mainly considers BEV-based methods for comparisons, however, it is unclear how significant its technical contributions are compared with previous point-cloud-based methods (e.g., Cooper, F-cooper). I am concerned about this because the techniques used in the main components of the proposed method, including point cluster picking, pose correction, and the SD-FPS, are quite common in the research communities of point cloud processing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rhWg9CSzSH", "reviewer": "Reviewer_4p6F", "summary": "This paper proposes to use Point Cluster as the message unit for collaborative perception. Previous intermediate methods use BEV map as the message unit, which suffer from weak object features, inefficient message aggregation and vague boundary. The proposed point cluster-based framework can solve these problems well. Extensive experiments on V2XSet, OPV2V, and DAIR-V2X-C demonstrate the effectiveness of the proposed method.", "questions": "In section 3.5, can the authors detail \"spatial cluster matching\"? If the pose between two agents is not accurate, how to match point clusters from different agents into a single object?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Using point cluster as an intermediate message unit is novel and well-motivated.\n2. The proposed CPPC framework, as well as the PCE, PCP and PCA modules are solid.\n3. State-of-the-art performance on mainstream benchmarks.", "weaknesses": "My main concerns are about the effectiveness and efficiency of the proposed method, for which I think more ablation studies are required.\n1. Ablation studies on PCP, PCA, pose correction and latency compensation are required.\n2. The authors claim the BEV representation is inefficient during aggregation. Is there any comparison between BEV and Point Cluster-based methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)