An Extensible Framework for Open Heterogeneous Collaborative Perception
Authors: Yifan Lu
Yue Hu
Yiqi Zhong
Dequan Wang
Yanfeng Wang
Siheng Chen
Venue: ICLR 2024 poster (2024)
Keywords: Collaborative Perception
Sensor and Model Heterogeneity
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose an extensible framework to integrate previously unseen heterogenous agents into collaborative perception with high collaboration performance and ultra low training costs.

Abstract: Collaborative perception aims to mitigate the limitations of single-agent perception, such as occlusions, by facilitating data exchange among multiple agents. However, most current works consider a homogeneous scenario where all agents use identity sensors and perception models. In reality, heterogeneous agent types may continually emerge and inevitably face a domain gap when collaborating with existing agents. In this paper, we introduce a new open heterogeneous problem: how to accommodate continually emerging new heterogeneous agent types into collaborative perception, while ensuring high perception performance and low integration cost? To address this problem, we propose HEterogeneous ALliance (HEAL), a novel extensible collaborative perception framework. HEAL first establishes a unified feature space with initial agents via a novel multi-scale foreground-aware Pyramid Fusion network. When heterogeneous new agents emerge with previously unseen modalities or models, we align them to the established unified space with an innovative backward alignment. This step only involves individual training on the new agent type, thus presenting extremely low training costs and high extensibility. To enrich agents' data heterogeneity, we bring OPV2V-H, a new large-scale dataset with more diverse sensor types. Extensive experiments on OPV2V-H and DAIR-V2X datasets show that HEAL surpasses SOTA methods in performance while reducing the training parameters by 91.5\% when integrating 3 new agent types. We further implement a comprehensive codebase at: https://github.com/yifanlu0227/HEAL

Published as a conference paper at ICLR 2024
AN EXTENSIBLE FRAMEWORK FOR OPEN HETEROGE- NEOUS COLLABORATIVE PERCEPTION
Yifan Lu Yue Hu Yiqi Zhong', Dequan Wang 13 Yanfeng Wang 1.3 Siheng Chen 1,3,4
1 Shanghai Jiao Tong University. University of Southern California, 3 Shanghai Al Lab
Multi-Agent Governnnce & Intelligence Crew (MAGIC)
(yifan.lu, 18671129361,
dequanwang, wangyanfeng, sihengc)esjtu.
2 rigizhon@usc.edu
ABSTRACT
Collaborative perception aims to mitigate the limitations of single-agent percep- tion, such occlusions, by facilitating data exchange among multiple agents. However, most current works consider a homogeneous scenario where all agents use identity sensors and perception models. In reality, heterogeneous agent types may continually emerge and inevitably face a domain gap when collaborating with existing agents. In this paper, we introduce new open heterogeneous problem: how to accommodate continually emerging new heterogeneous agent types into collaborative perception, while ensuring high perception performance and low integration cost? To address this problem, we propose HEterogeneous ALliance (HEAL), a novel extensible collaborative perception framework. HEAL first establishes unified feature space with initial agents via novel multi-scale foreground- Pyramid Fusion network. When heterogeneous new agents emerge with previously unseen modalities models, we align them the es- tablished unified space with an innovative backward alignment. This step only involves individual training on the new agent type, thus presenting extremely low training costs and high extensibility. To enrich agents' data heterogeneity, we bring OPV2V-H, a new large-scale dataset with more diverse sensor types. Ex- tensive experiments on OPV2V-H and DAIR-VES V2X datasets show that HEAL sur- passes SOTA methods in performance while reducing the training parameters by 91.5% when integrating 3 new agent types. We further implement a comprehen- sive codebase at: http::/gihub..om///fttti//////
INTRODUCTION
Multi-agent collaborative perception promotes better and more holistic perception by enabling mul- tiple agents to share complementary perceptual information with each other (Wang al. 2020 Xu et al. 2022c Li al. 2021). This task can fundamentally overcome several stag-standing issues in single-agent perception, such as occlusion (Wang et al.] 2020). The related methods and systems have tremendous potential in many applications, including multi-UAV (unmanned aerial vehicles) for search and rescue (Hu et al., 2022), multi-robot automation and mapping (Carpin) 2008). and vehicle-to-vehicle (V2V) and vehiele-f--everything (V2X) collaboration.
In this emerging field, most current works (Lu et al., 2023 Lei et al. 2022) make a plausible, yet oversimplified assumption: all the agents have to be homogeneous; that all agents' perception systems use the same sensor modality and share the same detection model. However, in the real world, the modalities and models of agents are likely to be heterogeneous, and new agent types may continuously emerge. Due to the rapid iteration of sensor technologies and perception algorithms, coupled with the various attitudes of agent owners (like autonomous driving companies) towards col- laborative perception, it is inherently challenging to definitively determine all agent types from the outset. When a heterogeneous agent, which has never appeared in the training set, wishes to join the collaboration, it inevitably encounters domain gap with the existing agents. This gap substantially impedes its capability to fuse features with the existing collaborative agents and markedly limits the extensibility of the collaborative perception. Thus, the problem of open heterogeneous collabora- tive perception arises: how to accommodate continually emerging new agent types into the existing collaborative perception while ensuring high perception performance and low infegration cost?

---

Published as a conference paper at ICLR 2024
(6) Homogeneous scenario (b) Heterageneous scenario
(c) Open Heterogeneous scenario
(d) Performance Training Cost
HEAL
UDAR
UDAR
HMAVIT
Camera1
Unccollet
Join
LIDAR
DAR
LDAR
Join
LIDAR
T-Cooper
Trainnn Farams (MI
Figure I: (a) homogeneous setting, where agents have identical modality and model. (b) hetero- geneous setting, where agents' modalities and models are distinct but pre-determined. (c) Open heterogeneous setting, where new types of agents want to join collaboration with previously unseen modalities or models. (d) HEAL holds the SOTA performance while minimizing the training cost (model parameters here) when integrating a new agent type. The bullseye represents the best.
The designation open heterogeneous underscores the unpredictable essence of the incoming agent's modality and model; see Figure. D for an illustration. To address this issue, one viable solution is late fusion. By fusing each agent's detection outputs, late fusion bypasses the heterogeneity among new agents and existing agents. However, its performance is suboptimal and has been shown par- ticularly vulnerable to localization noise (Lu et al.] 2023) and communication latency (Wang et al, 2020). Another potential approach is fully collective training like HM-ViT (Xiang al.] (023), which aggregates all agent types in training to fill domain gaps. However, this approach requires retraining the entire model every time a new agent type is introduced. This becomes increasingly training-expensive as new agents continuously emerge.
To address this open heterogeneous collaborative perception problem, we propose HEterogeneous ALliance (HEAL), novel extensible framework that integrates new agent types into collaboration with ultra-low costs. The core idea is to sustain unified feature space for multi-agent collaboration and ensure new agent types align their features to it. HEAL has two training phases: collaboration base training and new agent type training. In the first phase, HEAL sets initial agents as the collabo- ration base and undertakes collective end-to-end training create a robust unified feature space for all agents. It uses the innovative Pyramid Fusion, a multi-scale and foreground- aware network, to fuse features and promote the learning of the unified space. In the next phase, when agents with new heterogeneous type aim to join the collaboration, HEAL designs a novel backward alignmens mechanism for their individual training. The inherited Pyramid Fusion module acts detection back-end, with only new agents' front-end encoders updated. This prompts new agents to align their features with the unified feature space. Such individual training eliminates the high costs associated with collective retraining when adding new agent types, presenting extremely low model size, FLOPs, training time, and memory consumption. Further, it preserves the new agents' model and data privacy. As the backward alignment can be conducted locally, it protects new agents' model details and allows agent owners to use their sensor data for training. significantly addressing automotive companies' data- and model-privacy concerns. Once the training is complete, all agents of the new type can join the alliance with feature-level collaboration. By repeating the second phase, the alliance can continuously incorporate new types of agents as they emerge.
agents'
To evaluate HEAL and further promote open heterogeneous collaborative perception, we propose a large-scale heterogeneous collaborative perception dataset, OPV2V-H. which supplements more sensor types based on the existing OPV2V (Xu et al. 2022c). Extensive experiments on OPV2V-H and real-world dataset DAIR-V V2X (Yu et al. 2022) show HEAL's remarkable performance. In the experiment of successively adding 3 types of heterogeneous agents, HEAL outperforms the other methods in collaborative detection performance while reducing 91.5% of the training parameters compared with SOTA. We summarize our contributions as follows:
In considering the scenario of continually emerging new heterogeneous agents, we present HEAL, the first extensible heterogeneous collaborative perception framework. HEAL en- sures extensibility by establishing a unified feature space and aligning new agents to it.
We propose a powerful Pyramid Fusion for the collaboration base training, which utilized multiscale and foreground-awarr designs to sustain potent unified feature space.
2

---

Published as a conference paper at ICLR 2024
To integrate new types of agents, introduce novel backward alignment mechanism to align heterogencous agents to the unified space. This training is conducted locally on single agents, reducing training costs while also preserving model details.
We propose a new dataset OPV2V-H to facilitate the research of heterogeneous collabora- tive perception. Extensive experiments on OPV2V-H and real-world DAIR- V2X datasets demonstrate HEAL's SOTA performance and ultra-low training expenses.
2 RELATED WORKS
2.1 COLLABORATIVE PERCEPTION
The exchange of perception data among agents enables collaborative agents to achieve a more com- prehensive perceptual outcome (Wang et al. 2020 Yu et al.] [022] Li et al. 2021; Hu et al.) [023] al. 2023). Early techniques transmitted either raw sensory data (known as early fusion) or perception outputs (known as late fusion). However, recent studies have been studying the transmission of intermediate features to balance performance and band- width. To boost the research of multi-agent collaborative perception, V2X-Sim (Li et al., 2022c) and OPV2V Xu al. (2022c) generated high-quality simulation datasets, and DAIR- V2X (Yu et al. 2022 collects real real-world data. To achieve an effective trade-off between perception per- formance and communication costs, Who2com (Liu et al., (020b), When2com [iu et al. (2020a) and Where2comm (Hu et al. 2022) select the most critical message to communicate. To resist pose errors, Vadivelu et al. (2021) and Lu et al. (2023) use learnable or mathematical methods to correct the pose errors. Collaborative perception
Wei et al. 2023 Liu et al. 2020a
also directly help the driving planning and control task Chen & Krähenbühl ) Cui et al. (2022); Zhu et al. (2023) with more accurate perceptual results. Most papers assume that agents given the same sensor modality and model, which is deemed impractical in the real world. A contemporaneous work solving agent's modality hetero- geneity is HM-ViT (Xiang al., (203)) but it neglects the framework's 's extensibility and requires retraining the whole model when adding new agent types. In this paper, we address the issues of heterogeneity and extensibility together.
2.2
MULTI-MODALITY FUSION
In the field of 3D object detection, the fusion of LiDAR and camera data has demonstrated promis-
ing results (Chen et al. 2022a; Li et al. 2022d Yang et al. 2022; Bai et al.
2022; Borse et al,
2023 Li et al. 2022b Xu et al. 2022d) -iDAR-t--camen
Ma et al. 2019) methods project Li-
DAR points to camera planes. By using this technique, a sparse depth map can be generated and combined with image data. Camera-to-LiDAR (Vora et al.) 2020 Li et al. (022d) methods deco- rated LiDAR points with the color and texture information retrieved from images. Bird's eye view (BEV) (Liu et al. 2022a Li et al. 2022a| Borse al. 2023 Chen et al. 2022b) provides spa- tial unified representation for different modalities to perform feature fusion. As stated in (Xiang jet al., 2023). in single-agent multi-yodalltyy settings, sensor types/numbers and their relative poses are fixed, with most LiDARRcamera fusion algorithms (Chen et al., 2022a Li et al., 2022d Yang jet al., 2022] developed based on this fact. In contrast, heterogeneous multi-agent collaboration has random sensor positions and types, differing fundamentally from Ainle-vehicle multi-modality fu- sion. However, the aforementioned BEV representation a highly efficacious approach to facilitate spatial alignment among agents of diverse modalities and models.
3 OPEN HETEROGENEOUS COLLABORATIVE PERCEPTION
Multi-agent collaborative perception allows a group of agents to collectively perceive the whole environment, exchanging complementary perceptual information with each other. Within this do- main, open heterogeneous collaborative perception considers the scenario where new agent types with unseen sensor modalities or perception models can be continually added to the existing collab- orative system. Without loss of generality, consider N homogeneous agents in the scene initially. These agents, uniformly equipped with identical sensors and perception models, have the capability to observe, communicate, and compute, thereby fostering a homogeneous collaborative network. Subsequently, some new types of agents with previously unseen sensor modalities or models, are introduced into the scene sequentially to join the collaboration. Such dynamism characterizes the attributes of deploying collaborative perception in the real world: agent types will not be fully de- termined at the beginning and the number of types is likely to increase over time. It highly differs from traditional heterogeneous frameworks, where agent types are predetermined and fixed.
3

---

Published as a conference paper at ICLR 2024
To tackle open heterogeneous collaborative perception, straightforward approach is to conduct training for both existing and new agents collectively. However, this approach is computationally expensive with repetitive integrations primary goals: i) minimizing the training overhead associated with each integration, and ii) maxi- mizing perception performances across all agent types post-integration. The pursuit of these twin goals is fundamental to the successful implementation of open heterogeneous collaborative percep- tion in diverse and dynamically changing real-world scenarios.
time. Therefore, an effective solution must balance two
HETEROGENEOUS ALLIANCE (HEAL)
Collaboration Base Training
(collective training)
Feanure
|
Encoder
Agent
BEV Feature
Detection Head
Encoder
Agent
BEV
Festure Pose
Pyramid Fusion
Collaborative Detections
Message Transmisiinn
Feature Fusion
Dacoding
(i1) New Agent Type Training
(ii)) Join Collaboration
(local training)
From (8 and foed Trabuble
Collab, Base
Align
H
Desection Head
Encoder
New
BEV Feature
Heterogenous
Agent
Bacloward Aligament
Pyramid Fusion
Collaborative Detections
Single
Collaborate feature level
Figure 2: Overview of HEAL. (i) We train the initial homogeneous agents (collaboration base) with our novel Pyramid Fusion to establish a unified feature space; (ii) We leverage the well-trained Pyramid Fusion and detection head the new agents' detection back-end. With the back-end fixed, pushes the encoder to align its features within the unified feature. This step is performed on the new agent type only, presenting extremely low training costs. (iii) New agents join the collaboration.
To address open heterogeneous collaborative perception problem, we propose HEterogeneous ALliance (HEAL). It is an extensible framework to seamlessly integrate new agent types into the existing collaborative network with both minimal training overhead and optimal performance. As shown in Figure. 2 HEAL includes two phases realize growing alliance: i) collaboration base training, which allows initial agents to collaborate at feature-level and create unified feature space; and new agent type training, which aligns new agents' feature with the previously established unified feature space for collaboration. For every integration of new agent type, only the second phase required. We now elaborate on each training phase in the following subsections.
4.1 COLLABORATION BASE TRAINING
In this phase, designate the initial homogeneous agents our collaboration base and train feature-level collaborative perception network. Let Sny be the set of N agents with the same base agent type b. For the ith agent in the set Spy,, we denote O, as observation, Vencoder|o] (-) as its perception encoder and B; as its final detection output. Then, the collaborative perception network of the ith agent works as follows:
F fencoder[b] (O:).
E SM
> Feature Encoding
(1a)
F Ji 17 (Fz),
j jE Sw
Message Transmission
(Ib)
H. Spyramid.fusi fusion ((F)-())ss
>.eatture Fusion
(Ic)
B. Shead(H;))
>-Dcooding Feature
(Id)
where F; is the initial feature map from the encoder with BEV representation, r-)>(.)) is an operator that transmits jth agent's 's feature to the ith agent and performs spatial transformation, F Fj is the spatially aligned BEV feature in ith's 's'' coordinate (note that F F. = F,). H, the fused feature and B, is the final detection output obtained by detection head fhead
4

---

Published as a conference paper at ICLR 2024
Pyramid Fusion
Foreground
Fonsground Entrnnoo
Poxal-wise Multiply
Pixel-wise Addition
Scrmanrr
Concateration
Softmax Operator
Multiscale Feat. Enc.
Map
D O
ReaNiX:: F
Foregroand Mip
ResNeXt Layers
F
J-rd
Layors
Detection Results
F(1)
ResNeXt
Detection Head
Layors
Upsampll
Upsample
Final Fused Feature
Figure 3: Pyramid Fusion uses multiscale and foreground-avee aware designs to fuse features and create robust unified feature space. Foreground estimators produce foreground possibility maps each BEV position. These foreground maps are then normalized to weights for feature summation. Fore- ground maps are subject supervision during training. Blue and green represent different agents.
To provide well-establish feature space for multi-agent collaboration, we propose novel Pyramid Fusion fpyramid.fusion (-) in Step (Ic). designed with multiscalc and foreground- structure. Let
(0) be Fj Ji we elaborate fpyramid fusion! (-) in detail:
(
3-+ j+i R2 (F
E Spi and = 1,2, L,
(2a)
s0
(((F(( j-t));
E Spy, and = 2,
(2b)
( H+i
= softmax($) )-1.)
j E Sph and l 1.2,
(2c)
F(O) sum( j+i
(2) ( jot
(2d)
H. concat([F) (1) u2 (2
(P))
(2e)
where l indicates the scale, R₂(-) is the 9th ResNeXt (Xie et al. 2017) layer with a downsampling
rate of 2, F/2, is encoded features at 9th scale; He(.) is êth foreground estimator that outputs
the foreground map s/2,. measuring the possibility of foreground object each BEV position; softmax(.) normalizes the foreground possibility the weights for multi-agent feature fusion w/ ue(+) is an upsampling operator for the éth scale.
The proposed Pyramid Fusion facilitates the establishment of unified feature space for multi-agent collaboration by harnessing two key designs: a multi-scale structure and foreground awareness. First, the multi-scale ResNeXt layers create comprehensive unified feature space by fusing features at varying BEV scales. This not only promotes feature fusion in the collaboration base, but also ensures adaptability for future new agents, allowing their alignment to this unified feature space with coarse and fine grain. Furthermore, fusing at higher scales mitigates the discretization errors introduced by spatial transformation, thereby enhancing the robustness of multi-agent feature fusion and future alignment. Second, the foreground awareness design leverages foreground estimators to obtain foreground maps, which will guide Pyramid Fusion to select the perceptually critical features for fusion. It also enables the the model to learn how to differentiate between the foreground and the background, leading to a more robust unified space.
To train the collaborative perception model for the collaboration base, the overall loss is:
L Spi
= Laet (B,, Y.) +
ΣV
art (5)2 (2.).
(3)
2=13=1
The first term is detection supervision, where Teacs() is the detection loss, including focal loss (Lin et al. 2017) for classification and Smooth- I, loss (Girshick, (015) for regression, Y, is the ground- truth detections and B, is the detection output by our model. In addition the supervision on the collaborative detection, we design the foreground map supervision at multiple BEV scales, where
T refers to focal loss (L et al. 2017), (()), is the estimated foreground map from Step 2h
5

---

Published as a conference paper at ICLR 2024
and Y the ground-truth BEV mask of foreground object for each agent. The hyperparameter Og controls the effect of foreground supervision at various scales.
4.2 NEW AGENT TYPE TRAINING
Now we consider the integration of a new heterogeneous agent type, leveraging a novel backward alignment. The core idea is to utilize the Pyramid Fusion module and detection head from the previous phase as new agents' single detection back-end and only update the front-end encoder module, prompting the encoder to generate features with the pre-established unified feature space.
Specifically, we denote the new agent type as n4 and new agent set as Sta.). and the full set of current agents set becomes S Spj U Smnj)- For agent & in the agent set [mr.]: we define Og as its observation and Seneoder(n.]] )-)) as its detector encoder. We keep sorramid_fusion (-) and (+) from
previous stage unchanged, where denotes fixed, and train Sencoder(m1] (-) single agents:
FR fencuder(a1] (Ok),
k S(m]]
(4a)
F/ f pyramiid fusion (Fx),
k Smml)
(4b)
B₁ Shead (F)))
k S(mJ]:
(4c)
where F& is the feature encoded from the new sensor and model. Fú, is the feature encoded by Pyramid Fusion module and B3 is the corresponding detections. Note that here perform individ- ual training for single agents; thus the input to the Pyramid Fusion is single-agent feature map F; in (Ic). With the pretrained Pyramid Fusion module and the detection head established as the back-end and fixed, the training process naturally evolves into adapting the front-end encoder Sencoder(n. to the back-end's parameters. thereby enabling new agent types to align with unified space.
in (b) instead of multi-agent feature maps {F {F
Our backward alignment works well for two reasons. First, BEV representation offers shared co- ordinate system for varied sensors and models. Second, with the intrinsic design of Pyramid Fusion, feature domain aligmment can be conducted with high efficiency: i) The alignment is performed across multiple scales, capturing and bridging the plausible feature scale differences between dif- ferent modalities and models. ii) The foreground estimators are also retained, thereby preserving effective supervision the alignment with the most important foreground feature.
In addition to enabling new and existing agents to collaborate the feature level with robust per- formance, our backward alignment also shows a unique advantage: Training is only conducted on single agents of the new type. This significantly reduces the training costs of cach integration, avoiding collecting spatio-temporal synchronized sensor data for multi-agent collective training and expensive retraining. Further, it prevents the new agents' model details from disclosure and allows the owner of the new agents can their sensor data. It would remarkably address many privacy concerns automotive companies might have when deploying the V2V techniques.
To supervise the training of new agent type with backward alignment, the loss is the same Eq. (3). but we refer the detection bounding boxes B, and ground-truth bounding boxes Y, to belong the single agents. The supervision on the confidence score at different scales is also preserved.
4.3 HEAL DURING INFERENCE
Once the new agent of agent type n, has trained its encoder, all agents of new agent type nu can collaborate with base agents in the scene. Mathematically, for Agent in the set Siy U U its
feature after multi-agent fusion is obtained as H, = fpyramidd fusion ((Tz-: (F (F3) (es This is feasible because two training phases ensure that for all j. F, lies in the same feature space.
Following this, we can continually integrate emerging heterogencous agents into our alliance by revisiting the steps outlined in See. [4.2 creating highly extensible and expansive heterogeneous alliance. Assuming there are total of 7 new agent types, once training for each of these new agent types is completed, the collaboration among all heterogeneous agents can be written as:
H; = fpramid fusion (T (Fz))
18,
Then, decode the feature and obtain the final detections B, fhead (H:).

---

Published as a conference paper at ICLR 2024
5 EXPERIMENTAL RESULTS
5.1 DATASETS
OPV2V-H. We propose simulation dataset dubbed OPV2V-H. OPV2V (Xu et al.; 2022c) dataset, the LiDAR with 64-channe] shows significant detection advantage over the camera modal- ity. Evaluations of heterogeneous collaboration on OPV2V may not truly represent the collaboration performance between these two modalities since LiDAR agents can provide most detections (Xiang et al. 2023). For this purpose, we collected more data to bridge the gap between LiDAR and camera modalities, leading to the new OPV2V-H dataset. OPV2V-H dataset has average approximately 3 agents with a minimum of 2 and a maximum of 7 in each frame. Except for one 64-channel Li- DAR and four 4 RGB cameras (resolution 800+600) of each agent from original OPV2V dataset, OPV2V-H collects extra 16- and 32-channel LiDAR data and 4 depth camera data.
DAIR-V2X. DAIR-V2X (Yu et al. 2022 is a real-world collaborative perception dataset. The dataset has 9K frames featuring one vehicle and one roadside unit (RSU), both equipped with a LiDAR and a 1920x1080 camera. RSU" LiDAR 300-channel while the vehicle's is 40-channel.
5.2 OPEN HETEROGENEOUS SETTINGS
We consider the sequential integration of new heterogeneous agents into the collaborative system to evaluate the collaborative performance and training costs of HEAL. We prepared four agent types, including 2 LiDAR models and 2 camera models; see Table. D
Table 1: Agent type setting for open heterogeneous collaborative perception experiments.
Agent Type
Agent Sensor and Model Setup
LiDAR of a-channel, PointPillars (Lang et 2019
2019
Camera, resize img. to height px. Lift-Splat (Philion & Fidler 2020
EfficientNe
Tan & Le 2019 as img. encoder.
LiDAR of a-chhamel, SECOND Yan el 2018
Camera, resize img. height px. Lift-Splat (Philion & Fidler 2020
ReaNer50 (H al. 2016 as img. encader.
Implementation details. We first adopt L (64) agents as the collaboration base to establish the
unified feature space. And then select (C (C(381)
(32) ((336)) to be new agent types. Both PointPil- E L S
lars (Lang et al. 2019). SECOND (Yan et al. 2018) and Lift-Splat-Shoot (Philion & Fidler 2020) encode input data with grid size [0.4m, 0.4m) and further downsample the feature map by 2 and shrink the feature dimension to 64 with CenvNe (Liu al. (022b) blocks for message sharing. The multi-scale feature dimension of Pyramid Fusion is [64, 28,256]. The ResNeXt layers have [3,5,8] blocks each. Foreground estimators
1 X 1 convolution with channel [64, 226).. The hyper-parameter ag = (0.4, 0.2, 0.1) We incorporated depth supervision for all camera de- tections help convergence. We train the collaboration base and new agent types both for 25 epochs end-to-end with Adam, reducing the learning rate from 0,002 by 0.1 at epochs 15. Training costs 5 hours for the collaboration base on 2 RTX 3090 GPUs and hours for each new agent's training, while Xiang et al. (2023) takes more than day to converge with 4 agent types together. We adopt Average precision (AP) different Intersection-overr Union (loU) to measure the perception perfor- mance. The training range is E [- 102. +102.4m), E [-51.2m, +51.2m] but we expand the range to æ E [-204..mm + 2m.. y E [-102.4m, +102.4m] in evaluation for holistic view. Late Fusion aggregates all detected boxes from single agents. Considering most scenarios in the test set involve fewer than 4 agents, we initialized the scenario with one (64) agent and progressively
introduced C Lg
(384) (32) _c
(336) agents into the scene for evaluation.
5.3 QUANTITATIVE RESULTS
Performance and training cost. Table. compares the detection performance and training cost. We see that HEAL surpassed all existing collaborative perception methods in perception perfor- mances while maintaining the lowest training cost. This is attributed to our powerful Pyramid Fusion structure and cost-efficient backward alignment design. Pyramid Fusion selects the most important feature for fusion in a multiscale manner and sustains potent unified space. Backward alignment further reduces the training costs of integrating new agents remarkably. This step does not involve collective training, thus ensuring consistent and extremely low training costs. It presents significant advantages in model size, FLOPs, training time, and memory usage. In contrast, other baseline methods necessitate retraining all models at each integration with computational complexity of
7

---

Published as a conference paper at ICLR 2024
Table 2: We evaluate the performance and training cost on OPV2V-H dataset when increasingly
adding 3 new heterogeneous agents to the scene for collaboration (starting with one 4 agent). Every method requires retraining the whole model except for 'no fusion', 'late fusion', and HEAL. Metrics related to the training cost are all measured with batchsize on RTX A40. Optimal values among intermediate fusion methods are bolded. Percentage comparison is made with HM-ViT.
Metric
AP50 ?
AP701 1
Train Throughput (ftsec.))
Based on L," .dd New Agent
+CE
+C
+C
+L.
(12)
+C
-C
+C
No Fusion
0.748
0.748
0,748
0.606
0.606
0.606
Late Pusion
0.275
0.833
0.834
0.599
0.685
F-Creger
Chen
2019
0.778
0.742
0.261
0.628
0.517
0.494
2.41
2.93
2.54
DiscoNet
0.798
0.833
0.830
0.653
0.682
0.695
2.37
2.87
2.47
AnHusion
0.796
0.821
01813
0.635
2.36
2.39
VIXVI
AU Cl AT226
0.832
0.888
0.882
0.655
0.765
0.753
1445
1.58
1.45
CaBEVT
AU 2f 20226
0.822
0.885
0.885
0.671
0.742
0.742
1.94
2.27
1.99
HM-VIT
Kiing
0.813
0.871
0.876
0.616
0,743
0.755
1.22
1.33
1.18
HEAL
0.892
0.726
3.27
(.880)
Metric
Model APaaaus (M)
FLOPST) L
Poak Menoxy (GB)
Based on L. Add New Agent
+
+
+
+
+1.
+C
-C
+L
+
(50)
Fusson
Late Pusicn
20:25
7.15
0:149
3.70
F-Cooper
Chen al 2019
30.70
39.59
49.22
0.190
0.263
0.322
20.27
14.56
18,36
DisooNet
30.80
39.67
-9.29
0.194
0.270
0.332
19.92
17.10
15.95
AtfFusion
30.78
49.22
0.190
19.77
V2NVI
Xu al
20226
36.17
44.99
54.62
0.315
0.352
0.393
25.92
21.25
21.00
CaBEVT
XB 27
20124
33.24
42,05
51.68
0.243
0,280
0.321
20.94
20.16
23.29
HM-MI
Xhang
2023
47.71
65.08
83.34
0.236
0.331
0.442
35.53
26.55
26.88
HEAL
20.25
6.34
7.15 91.5%)
0.149
0.074
0.091 (L 79.65))
3.71
(L 88.85))
Table 3: Heterogeneous type agents are added in the order presented from left to right in each type
combination. Σ M.#P. represents the accumulation of model parameters for collaboration base
training and each integration. Percentage comparison is made with HM-ViT. HEAL holds the best performance and the lowest training cost under various agent type combinations.
Dutasel
OPV2V-H agents)
DAIR-VIN (2agenis)
Agent Types
-C
+1.50
+/
1 +C
1 +1.5
L
-Cy
Metric
AP10
M
AP20
M.P.R.
AP50
X M.A..
AP50
M.P.
AP50
M.
Fusion
5.47
5.47
0.392
5.47
0.392
Late Fusion
0.600
30.2
0.457
39.2
0.344
25.7
0.376
11.8
0.341
12.6
P-Cooper
0.400
127.6
0.299
127.6
0.626
38.8
0.545
24.9
0.611
25.8
Discoder
0.612
127.9
0.420
127.9
0,576
38.9
0.634
25.1
0.621
25.9
AtlFessio
0.57]
127.7
0.394
127.7
0.619
35.8
0.661
21.7
0.623
25.8
VIXVI
0.688
149.2
0.298
149.2
0.654
0.709
35.7
0.669
36.6
137.5
137.5
0.638
29.8
IIM-VE VIT
0.695
216.3
0.506
216.3
0.638
67.8
0.538
53,9
0.677
51.8
HEAL
0.738
39.2 61.99)
0.578
39.2 61.99)
0.658
25.7C_ 62.19)
0.770
11,80 78.15)
0.681
12,611 77.1%)
O(m) or even 0(m²) (Xiang al. 2023), where m denotes the number of agent types. This limits their scalability, especially when there are numerous and growing types. Here when adding (3336) agent, HEAL outperforms previous SOTA HM-VIT by 7.6% in AP70 with only 8.3% parameters.
Agent type combination. We present the final collaboration performance and accumulated train- ing parameters with different agent type combinations on OPV2V-H and DAIR-V V2X datasets in Table. 3 Specifically, we configure the LiDAR agent with lower channel numbers to show the performance degraded LiDAR scenarios. Experiments show that no matter what kind of agent combination, HEAL always maintains the best performance and the lowest training cost.
Imperfect localization. Existing experiments hold the assumption that each agent has an accurate pose. However, in real-world scenarios, due to the presence of localization noise, features between agents might not align precisely. Consequently, we introduce robustness experiment against pose errors, adding Gaussian noise to the accurate pose, as depicted in Figure. ! Experimental results show that HEAL retains state-of-the-an performance even under various pose error conditions.
Feature compression. We use an autoencoder to reduce feature channels for bandwidth saving. Using well-trained HEAL and baseline methods, we finetune the new autoencoder. The compression ratio indicates the reduction in channels. Results in Figure. @emmonstrate that even with a 32-fold compression, we still retain exceptionally high performance, surpassing baseline methods.
8

---

Published as a conference paper at ICLR 2024
0.900
APTO
0.65
66-
0.6
Performace
0.00
8.A
0.825
0.775
0.0
066
0.0
033
16
32
Pose Nose Std deg)
Pose Noise Std degl
Compression Ratio
Compression Ratio
Figure 4: Robust Experiment to pose error and compression ratio. Pose noise to N o on
x, location and N ( of) on yaw angle.
Component ablation. We carried out ablation experiments on HEAL's components and de- sign, as shown in Table. Results show that of them are highly beneficial to collaboration performance. Through multiscale pyramid fea- ture encoding at various scales, HEAL is capa- ble of learning features at different resolutions and subsequently aligning new agent types via backward alignment. Further, the foreground supervision can help the HEAL distinguish the foreground from the background and select the most important features. These components help to construct a robust unified feature space and realize the alignment comprehensively.
Table 4: Ablation study of multiscale pyramid,
foreground supervision, and backward alignment.
Agent Types:
( + C(384) L
(336)
Multiscale Pyramid
Foreground Back.
Supervision
Align.
AP50 f
AP70 t
0.729
0,578
0.746
0.628
0.753
0.633
0.764
0.662
0.841
0.732
0.864
0.781
0.879
0.782
0.894
0.813
5.4 QUALITATIVE VISUALIZATIONS
HEAL makes features align within the domain via backward alignment in Figure. and
achieves the best detection results in Figure. 6 LI,C1,L2 refer to (64) (3844) L. respectively.
(a) L2 Feature before Back. Align
(b) L2 Feature after Back. Align Figure 5: Visualization of HEAL's backward alignment from 1.2 to Ll's unified space.
(c) Unified Feature Space from L
88
(d) HMVIT (LI+CI+L2) Figure 6: Visualization of open heterogeneous collaborative perception results OPV2V-H. In (a)( we show the process of gradually adding new agents to HEAL. The features of the added agent are displayed in the upper left corner. Note that the point cloud of the camera agent C1 is only used to indicate the agent's position. We color the predicted and GT boxes.
(a) HEAL (LI)
(b) HEAL (LI+CI)
(c) HEAL (LI+CI+L2)
6 CONCLUSION
This paper proposes HEAL, novel framework for open heterogencous collaborative perception. HEAL boasts exceptional collaborative performance, minimal training costs and model-detail pro- tection. Experiments
proposed OPV2V-H and DAIR-V2X datasets validate the efficiency of the real-
HEAL, offering a practical solution for extensible collaborative perception deployment world scenario. The limitation of HEAL is that requires BEV features, which requires extra effort for some models (e.g., keypoint-based LiDAR detection) to be compatible to the framework.
7 ACKNOWLEDGE
This research is supported the National Key R&D Program of China under Grant 202 002IZD01122 12801, NSFC under Grant 62171276 and the Science and Technology Commission of Shanghai Municipal under Grant 2151 100000, 2251 106101 and 22D72229005.

---

Published as a conference paper at ICLR 2024
REFERENCES
Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1090-1199, 2022.
Shubhankar Borse, Marvin Klingner, Varun Ravi Kumar, Hong Cai, Abdulaziz Almuzairee, Senthil Yogamani, and Fatih Porikli. X-align: Cross-modal cross-view alignment for bird's-eye-view segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3287-3297, 2023.
Stefano Carpin. Fast and accurate map merging for multi-robot systems. Autonomous robots, 25: 305-316, 2008.
Dian Chen and Philipp Krähenbühl. Learning from all vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17222-172311 2022.
Qi Chen, Xu Ma, Sihai Tang, Jingda Guo, Qing Yang, and Song Fu. F-cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3d point clouds. In Proceedings of the 4th ACM/IEEE Symposium on Edge Computing, pp. 88-100, 2019.
Zchui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. Autoalignv2: Deformable feature aggregation for dynamic multi-modal 3d object detection. arXiv preprint arXiv:2207. 103 2022a.
Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. Bevdistill: Cross-modal bev distillation for multi-view 3d object detection. arXiv preprint 1rxXi:22111003380 2022b.
Jiaxun Cui, Hang Qiu, Dian Chen, Peter Stone, and Yuke Zhu. Coopernaut: End-to-end driving with cooperative perception for networked vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17252-17262, 2022.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urhan driving simulator. In Conference on robot learning, pp. 1-16. PMLR, 2017.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 1440-1448, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Yue Hu, Shaoheng Fang. Zixing Lei, Yiqi Zhong, and Siheng Chen. Where2comm: Communication-efficeen collaborative perception via spatial confidence maps. Advances in neu- ral information processing systems, 35:4874-4886 2022.
Yue Hu, Yifan Lu, Runsheng Xu, Weidi Xie, Siheng Chen, and Yanfeng Wang. Collaboration helps camera overtake lidar in 3d detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9243-9252 2023.
Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Point- pillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 12697-12705, 2019.
Zixing Lei, Shunli Ren, Yue Hu, Wenjun Zhang, and Siheng Chen. Latency-aware collaborative per- ception. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII. pp. 316-332. Springer. 2022.
Xin Li, Botian Shi, Yuenan Hou, Xingjiao Wu, Tianlong Ma, Yikang Li, and Liang He. Homoge- neous multi-modal feature fusion and interaction for 3d object detection. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXVIII, pp. 691-707. Springer, 2022a.
10

---

Published as a conference paper at ICLR 2024
Yanwei Li, Xiaojuan Qi, Yukang Chen, Liwei Wang. Zeming Li, Jian Sun, and Jiaya Jia. Voxel field fusion for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1120-1129, 2022b.
Yiming Li. Shunli Ren, Pengxiang Wu, Siheng Chen, Chen Feng, and Wenjun Zhang. Learning dis- tilled collaboration graph for multi-agent perception. Advances in Neural Information Processing Systems, 14:29541-295555 2021.
Yiming Li, Dekun Ma. Ziyan An, Zixun Wang. Yiqi Zhong, Siheng Chen, and Chen Feng. V2x- sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving. IEEE Robotics and Automation Letters,
2022c.
Yiming Li, Qi Fang, Jiamu Bai, Siheng Chen, Felix luefei-Xu, and Chen Feng. Among us: Adver- sarially robust collaborative perception by consensus. In Proceedings of the IEEE/CVF Interna- lional Conference on Computer Vision (ICCV), pp. 186-195, October 2023.
Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Yifeng Lu, Denny Zhou, Quoc V Le, et al. Deepfusion: Lidar-camera deep fusion for multi- modal 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 17182-17191, 2022d.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017.
Yen-Cheng Liu, Junjiao Tian, Nathaniel Glaser, and Zsolt Kira. When2com: Multi-agent perception via communication graph grouping. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 4106-4115, 2020a.
Yen-Cheng Liu, Junjiao Tian, Chih- Yao Ma, Nathan Glaser, Chia-Wen Kuo, and Zsolt Kira. Who2com: Collaborative perception via learnable handshake communication. In 2020 IEEE International Conference on Robotics and Automation (ICRA). pp. 6876-6883. IEEE, 2020b.
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's-ey' s-eye view representation. arXiv preprint arXiv:2205.133: 2022a.
Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11976-1 1986, 2022b.
Yifan Lu, Quanhao Li, Baoan Liu, Mehrdad Dianati, Chen Feng, Siheng Chen, and Yanfeng Wang. Robust collaborative 3d object detection in presence of pose errors. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 4812 4818. IEEE, 2023.
Fangchang Ma, Guilherme Venturelli Cavalheiro, and Sertac Karaman. Sclf-supervieed sparse-to- dense: Self-supervised depth completion from lidar and monocular camera. In 2019 International Conference on Robotics and Automation (ICRA), pp. 3288-3295. IEEE, 2019.
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 194-210. Springer, 2020.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105-6114 14. PMLR, 2019.
Nicholas Vadivelu, Mengye Ren, James Tu, Jingkang Wang, and Raquel Urtasun. Learning to communicate and correct pose errors. In Conference on Robot Learning, pp. 1195-1210. PMLR, 2021.
Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom. Pointpainting: Sequential fusion for 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4604-4611 2020.

---

Published as a conference paper at ICLR 2024
Tsun-Hsuan Wang, Sivabalan Manivasagam, Ming Liang, Bin Yang, Wenyuan Zeng, and Raquel Urtasun. V2vnet: Vehicle-to-vehicle communication for joint perception and prediction. In Com- puter Viston-ECCe Eeronn-CCVVA 2020: 16th European Conference, Glasgow, UK, August 23 23-28, 2020, Pro- ceedings, Part II 16, pp. 605-621. Springer, 2020.
Sizhe Wei, Yuxi Wei, Yue Hu, Yifan Lu, Yiqi Zhong, Siheng Chen, and Ya Zhang. Asynchrony- robust collaborative perception via bird's eye view flow. In Advances in Neural Information Processing Systems, 2023.
Hao Xiang, Runsheng Xu, and Jiaqi Ma. Hm-vit: Hetero-modal vehicle-to-vehicle cooperative perception with vision transformer. arXiv preprint arXiv:2304. 10628, 2023.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1492-1500, 2017.
Runsheng Xu, Yi Guo. Xu Han, Xin Xia, Hao Xiang, and Jiaqi Ma. Opencda: an open cooper- ative driving automation framework integrated with co-simulation. In 202/ IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 1155-1162 1162. IEEE, 2021.
Runsheng Xu, Zhengzhong Tu, Hao Xiang, Wei Shao, Bolei Zhou, and Jiaqi Ma. Cobevt: Cooperative bird's eye view semantic segmentation with sparse transformers. arXiv preprint DxXiv:2007.02202, 2022a.
Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang, and Jiaqi Ma. V2x-vit: Vehiclc-to-everythinng cooperative perception with vision transformer. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX, pp. 107-124. Springer, 2022b.
Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma. Opv2v: An open bench- mark dataset and fusion pipeline for perception with vehicle-to-vchicle communication. In 2022 International Conference Robotics and Automation (ICRA), pp. 2583-2589. IEEE, 2022c.
Runsheng Xu, Xin Xia, Jinlong Li, Hanzhao Li, Shuo Zhang, Zhengzhong Tu, Zonglin Meng, Hao Xiang, Xiaoyu Dong, Rui Song, et al. V2y4real: A real-world large-scale dataset for vehicle-to- vehicle cooperative perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13712-13722, 2023.
Xinli Xu, Shaocong Dong, Lihe Ding, Jie Wang, Tingfa Xu, and Jianan Li. Fusionrenn: Lidar- camera fusion for two-stage 3d object detection. arXiv preprint arXiv:2209. 10733, 2022d.
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018.
Zeyu Yang, Jiaqi Chen, Zhenwei Miao, Wei Li, Xiatian Zhu, and Li Zhang. Deepinteraction: 3d object detection via modality interaction. arXiv preprint arXiv:2208. 0122081/1112 //1/2, 2022.
Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, al. Dair-v2x: A large-scale dataset for vehicle-infrastrucuure cooperative 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21361-21337 2022.
Ruizhao Zhu, Peng Huang, Eshed Ohn-Bar, and Venkatesh Saligrama. Learning to drive anywhere. In 7th Annual Conference Robot Learning, 2023.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.I REAL-WORLD SOLUTION DETAILS
Vehicle
Road-Side Unit
Collective Training
Collaboration
Base
Pyramid Fusion Architecture & Weight
Distribute to Companies
Company
Company2
Company3
Local Data,
Local Data, Local Model, Local Training
Local Data, Local Model,
Local Model,
Local Training
Local Training
can create infinitely large Heterogeneous Alliance!
Figure 7: Real-world solution illustration. Take the DAIR-V22 dataset example, which sists one vehicle and one Road-side Unit(RSU). We first trained the Pyramid Fusion using the vehicle and the RSU's data the collaboration base. Subsequently, distributed the vehicle's 's raw data and the Pyramid Fusion's weights to various companies, allowing them to train their respective models locally.
Deploying collaborative sensing in the real world is not trivial task, largely due to the challenges in collecting training data. It necessitates the acquisition of multi-vehicle sensor data that has tem- poral and spatial synchronization, which is often a formidable endeavor. Precisely for these reasons, existing real-world collaborative perception datasets (Yu et al. 2022] Xu et al. (023) typically en- compass no more than 2 agents.
Given 2 agents in the scene, existing intermediate fusion methods is difficult to train three or more heterogeneous agent types together. However, due our extensible design, HEAL shows the poten- tial to achieve collaboration between infinite agent types even with the only 2 agent data. We think this is very efficient training paradigm in V2X collaborative perception, and it has great potential in the future. See Figure. 7 for illustration.
practice for automotive companies to join
Take the DAIR-V2 V2X dataset as an example, We can collectively train the Pyramid Fusion using vehicle-side and RSU-side LiDAR data. To expand the alliance, we distributed the Pyramid Fusion (and detection head)'s weights to companies that want to join the V2X collaboration. They can integrate their own vehicle's 's model encoder with the Pyramid Fusion module and train their model locally use their own data. After the backward alignment, they can collaborate with RSU at feature- level for robust and holistic perception. It is noteworthy that the data, model, and associated training are all conducted locally. This approach significantly safeguards the technical privacy of autonomous driving companies while simultaneously reducing training costs.
13

---

Published as a conference paper at ICLR 2024
A.2 VISUALIZATION FEATURE ALIGNMENT
(a) Ll's (
j= feature, channel 12 (b) Cl's F j-s feature, channel
(c) L2's (()) feature, channel
Ll's F J-s
feature, channel
(e) Cl's ((2)
feature, channel
(f) L2's
((2) j-i feature, channel
L.'ss (2) J-s
feature, channel 29 (h) C1's
[2) j-s feature, channel
( (i) 1.2's j-+i feature, channel
1.1's (3) feature, channel
(k) Cl's [(3) feature, channel
(I) 1.2's (13)
feature, channel 49
(m) Ll'ss (3) feature, channel (n) Cl's F feature, channel (o)
(3) j-i feature, channel
Figure 8: Visualization of the feature after ResNeXt Layer 1, and 3 within the Pyramid Fusion. Visualization shows that they are the same feature space and have similar response values for foreground and background pixels, which allows them fuse together.
14

---

Published as a conference paper at ICLR 2024
A.3 PERFORMANCE OF PYRAMID FUSION ON HOMOGENEOUS SETTING
We provide the collaborative perception performance comparison with existing methods under the homogeneous setting, where all agents are equipped with the same sensor and detection model. Such benchmark has appeared in number of existing works (Xu et al. 2022c; Xiang et al., 2023), which offers an intuitive performance comparison of fusion modules. We conduct experiments on SiDAR-based homogeneous collaboration and camera-based homogeneous collaboration to show the sota performance of Pyramid Fusion.
For the LiDAR-based setting, we use PointPillars (Lang et al.- 2019) as the encoder and use the 64-channel LiDAR as input. The detection range is set to æ E 102 102.4m, 2m, +102 y E [-102.4m, + 102.4mm) The voxelization grid size [0.4m 0.4m]. and we further downsample the feature map by 2x and shrink the feature dimension to 64 for message sharing. The multi-scale feature dimension of Pyramid Fusion is [64,128,256] and the ResNeXt layers have [3.5,8] blocks cach. Foreground estimators are x convolution with channel [64,128,256]. The
0g {0.4, 0.2, 0.1 1=1,223
For the camera-bart baaeer setting, we use Lpft-Splat (Philion & Fidler, 2020) with EfficientNet (Tan & Le 2019) as the image encoder, and resize the image imput to 384 px height. The detection range is The BEV feature map is initialized with size [0.4m, 0.4m)]. downsampled by 2x and shrinked to dimension 64 for message sharing. The hyperparameters of Pyramid Fusion is the same as the LIDAR-bascd model. Depth supervision is incorporated.
set to x E [-51.2m1 +51 +51.2m], y E [-51.2m, +51
2m]. +51.2m).
We present our collaborative perception results in Table. 5 and all methods are collectively trained an end-to-end manner. Experiments show the superiority of our multiscale and foreground-awe -waae fusion strategy.
Table 5: Comparison of the performance of different collaborative 3D object detection model in ho- mogeneous setting on OPV2V and DAIR-V22 Results show that our Pyramid Fusion outperforms the state-of-ther the-ahhrrr art methods in LiDAR- -bsed homogeneous collaboration and camera- based homo- geneous collaboration.
Dataset
OPV2V
DAXR-V2X
LiDAR-based AP501
Camera-based
LiDAR-based
Camera-based
Method
AP70 1
AP50
AP70
AP30 f AP50 f
AP30
AP50 1
No Collaboration
0.782
0.634
0.405
0.216
0.421
0.405
0.014
0.004
F-Cooper
Chen et al.
2019
0.763
0.481
0.469
0.219
0.723
0.620
0.115
0.026
DiscoNet
al.
0.882
0.737
0.517
0.234
0.746
0.685
0.083
0.017
AttFusion
2022c
0.878
0.751
0.529
0.252
0.738
0.673
0.094
0.021
V2XViT
Xu
20226
0.917
0.790
0.603
0.289
0.785
0,521
0.198
0.057
CoBEVT
Xu
2022a
0.935
0.821
0.571
0.261
0.787
0.692
0.182
0.042
HM-ViT
Xiang
22023
0.950
0.873
0.643
0.370
0.818
0.761
0.163
0.044
Pyramid Fusion
0.963
0.926
0.689
0.468
0.832
0.790
0.282
0.128
A.4 OPV2V-H DATASET DETAILS
The existing collaborative perception benchmarks focus solely on single sensor type. However, in real-world scenarios, agents often have varied sensor setups due to the different deployment time- lines and expenses. To bridge this gap, we propose a heterogeneous dataset, OPV2V-H. where agents are set up with diverse configurations. OPV2V-H is collected with OpenCDA (Xu et al.) 202T and CARLA (Dosovitskiy et al. (nn7) MIT license). Figure. shows the simulation environment. OpenCDA provides the driving scenarios that ensure the agents drive smoothly and safely, including the vehicle's initial location and moving speed. CARLA provides the maps, and weather and controls the movements of the agents.
The simulation settings and traffic patterns are derived from the playback of OPV2V, as referenced in (Xu et al.) 2022ch. Specifically, position both the autonomous and background agents based on the standard configuration of OPV2V. Additionally, we outfit each au- tonomous agent with broader range of sensors, totaling 11, including 4 cameras, 4 depth sensors,
Data collection.
15

---

Published as a conference paper at ICLR 2024
Figure 9: OPV2V-H is co-simulated by OpenCDA Xu et al. (2021) and CARLA Dosovitskiy et al. (017).
and 3 LiDAR sensors. The cameras provide views from the front, left, right, and rear. The depth sensors are arranged in a similar fashion to the cameras. The LiDAR configurations comprise 64- channel, 32-channel, and 16-channel setups, which typical in real-world situations. The sensor data and the sensor configurations are recorded during data collection.
Sensor configuration. The four RGB and depth cameras face forward, backward, right, and left with a yaw degree of 0%, 100% -100", and 180", respectively. And the corresponding offset to the agent center is (2.5, 0.0, 1.0). (0.0, 0.3, 1.8), (0.0, -0.3, 1.8), and (-2.0, 0.0, 1.5), respectively. The resolution of the captured image is 800 600. On each agent, all the cameras are fixed and their internal relative position and rotation degree invariable. Camera intrinsics and extrinsic in global coordinates are recorded to support coordinate transformation across various agents. The 3 LiDAR sensors are attached the same location on the agent with rotation frequency of 20 Hz, the upper field of view 2, and lower field of view -55, and channels 16, 32, and 64, respectively, resulting in about 10k, 20k and 40k point clouds, respectively. The translation (x, y. 2) and rotation (w, x, y, in quaternion) of each camera and LiDAR in both global and ego coordinates are recorded during data collection.
Data statistics. OPV2V-H provides fully annotated data, including synchronous images and point clouds with 3D bounding boxes vehicles. In total, OPV2V-H has 10,524 samples, including 6374/1980/2170 in train/validationttsst split, respectively. Each sample has 2-7 agents. Figure I0 shows captured sensor data the same agent, including the 4 RGB images, 4 depth images from four views (front, left, right, back), and LiDAR point clouds from the same view.
16

---

Published as a conference paper at ICLR 2024
LiDAR 16-channel
(b) LiDAR 32-ch2n 32-channel
LiDAR 64-channel
(d) Camera front
(e) Camera left
(f) Camera right
g Camera back
(h) Depth front
(i) Depth left
(i) Depth right
(k) Depth back
Figure 10: Each agent is equipped with 3 LiDARR cameras, and depth sensors in OPV2V-H.
A.5 FEATURE ENCODING DETAILS
PointPillars Lang et al. 2019) (L) in the paper) divides the point cloud data into pillars, which are vertical columns of points. They are then converted to a stacked pillar tensor and pillar index tensor. The encoder uses the stacked pillars to learn a of features that can be scattered back 2D pseudo-image for a convolutional neural network. This forms the BEV feature.
(x)
S in the paper) begins by voxelizing the 3D point cloud data, cre- ating 3D grid of voxels. Each voxel contains the point cloud data within its boundaries. SECOND then uses sparse convolutions to process these voxels. Stacking the voxel features along the height dimension results in a BEV feature map.
SECOND Yan et al. 2018
(L (ar)
Lift-Splat Philion & Fidler (2020) (Ch)) and C(a) in the paper) represents more recent and innovative approach for camera modality. It starts by "lifting" the image features from an image encoder (here C(2) uses EfficientNet and C(2) uses ResNet50) with a predicted depth distribution into 3D space. After this, it "splats" these features onto a 2D grid, effectively projecting features onto BEV plane. In our paper, use this feature projection mechanism with two different image encoders.
A.6 TRAINING DETAILS
We introduce how train the baseline model and our HEAL model on OPV2V-H dataset. The training procedure DAIR-V2X dataset is the same.
Baseline Model Training Baseline intermediate fusion, including F-Cooper Chen al. 2019) AuFusion (Xu et al., 2022c). DiscoNet (Li et al.) 2021). V2X-ViT (Xu et al. 2022b), CoBEVT (Xu ct al., 2022a), HMVIT (Xiang al.) 2023), will conduct collectively end-to-end training to inte- grate new agent type. Assume that when there are a total of m distinct agent types in the scenario, inclusive of the new intelligent agent. Dur- ing the training, an agent type from one of the m distinct types with uniform probability. Concurrently, each of the other agent in the scene is assigned an agent type with the same uniform probability. All agents
agent type seeks to join the existing collaboration,
randomly select an agent from the scenario to
the ego agent and assign
17

---

Published as a conference paper at ICLR 2024
encode their features according to the encoder specific to their type and transmit the information to the ego agent for feature fusion. During the validation phase, we prepared JSON file in advance, which contains the pre- assignedg agent types for each agent in the scene. Note that the agent types within the JSON file are also assigned with a uniform probability distribution (but since it stored in the file, it keeps the same for each validation and for each model). We select specific agent from the scenario to serve as the ego agent for computing the validation loss. In OPV2V-H, this would be the agent with the smallest agent ID.
The rationale behind this design is that during training, we can randomly permute the types of agents within the scenario and randomly select an agent serve the ego, thereby achieving data augmentation. Concurrently, during validation, the validation scenarios for all methods are consistent and fixed. This ensures an accurate reflection of whether the training has converged and facilitates fair comparison between methods.
HEAL Training As stated in Sec. 4 the training of HEAL can be divided into collaboration base training and new agent type training, For collaboration base training, it is the same as Baseline Model Training which is trained collectively in an end-to-end manner. So the training strategy is also the same with total agent type number
1. For new agent training, all agents in the scenario will be assigned to new agent type. However, because the training is only carried out on single agents, will randomly select agent from the scene ego and use its sensor data for individual training. During the validation phase, we consistently select a specific agent within the scenario the ego compute the validation loss. In OPV2V-H, this would be the agent with the smallest agent ID.
B SPATIAL TRANSFORMATION
Each agent generates BEV features in its own coordinate system, and the ego agent requires spatial transformation unify all coming features into its coordinate system. This step requires knowing the (i) relative pose 'L, E IR between vehicles and (ii) the physical size of the BEV feature D Dy meters. We assume these meta information (global pose and BEV feature map physical size) will be sent in V2V communication. Note that relative pose can be calculated between global poses.
To perform the spatial transformation, we adopt the affine_griiii grid and grid_sample functions from torch. nr. funct
onal innal package.
Since the feature in BEV space, we deprecate the z-axis component in 'E, and normalize with D2 and Dy. It results in normalized and unitless affine matrix, which will be the input of affineeggre -egrrd and
together with the feature map. With these two functions executed continuously, we will transform the feature map from other agent's ego perspective to our own coordinate. Just like the affine transformation a 2D image.
18


Reviews:
[{"review_id": "zuSYiimQo2", "reviewer": "Reviewer_hUnZ", "summary": "This paper presents a novel question surrounding the task of multi-agent collaborative perception, duly considering the scenario where new collaborators continually join the perception system in real-world settings. This is a highly intriguing question, directly impacting the deployment of multi-agent collaborative perception systems.\n\nThe author proposes a highly concise solution, introducing the HEAL framework, which is capable of accommodating the features acquired by the agents newly joining the system. The experimental results demonstrate that the proposed new framework is effective in accommodating agents newly incorporated into the system, and achieves state-of-the-art results. \n\nIn conclusion, this is a highly intriguing piece of work.", "questions": "1. In this paper, the authors claim that agents newly joining the system may struggle to align well in the feature space due to data distribution differences. However, is training a unified feature space an effective solution? Given that data discrepancies arising from different sensors inherently result in domain differences, this discrepancy poses a significant challenge in the domain adaptation field. Is the method proposed in this paper suitable for addressing this issue?\n\n2. The author raises a novel question, thus it would be prudent to utilize more datasets to verify the efficacy of the proposed method. This is because some schemes[1] solely employing distillation can achieve significant improvements in accuracy. In the open-source datasets they used, there are also newly joining agents, similar to the DAIR-V2X dataset used in this paper. It is hoped that the author can supplement with more extensive experiments to substantiate the reliability of the raised question and the effectiveness of the proposed method.\n\n3. Referring to the article on late fusion[2], I believe that late fusion seems to be a more effective solution to the problem posed in this paper. While the process of late fusion indeed has some issues with error accumulation, [2] has adeptly mitigated some of the past problems of late fusion through trajectory prediction. At the same time, employing late fusion can maximally avoid the issue of aligning features extracted by different agents, fundamentally resolving the problem posed in this paper. I hope that the author can conduct a comparative analysis between the methods of these two papers.\n\n[1]. Z. LI, et al. MKD-Cooper: Cooperative 3D Object Detection for Autonomous Driving via Multi-teacher Knowledge Distillation. IEEE Transactions on Intelligent Vehicles, 2023.\n\n[2]. S. Wei, et al. Asynchrony-Robust Collaborative Perception via Bird's Eye View Flow. NIPS 2023.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Presents a highly intriguing new question, effectively addressing the challenges faced during the deployment of multi-agent collaborative perception systems.\n\n2. The approach in this paper is notably succinct and efficient; the authors design a novel backward alignment mechanism for individual training. This method constructs an alignable feature space, facilitating subsequent updates of features transmitted by other agents.", "weaknesses": "1. The intermediate fusion method employed in this paper doesn't seem to address the issue of new agents joining as effectively as late fusion does. \n\n2. This paper has only conducted experiments on two datasets, one of which is generated for the first time in this paper. It is hoped that the author can introduce more experiments to substantiate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v5xVDAIQlq", "reviewer": "Reviewer_wcWo", "summary": "This paper introduces HEterogeneous ALliance (HEAL), an innovative framework designed to enhance collaborative perception by accommodating continually emerging heterogeneous agent types, addressing the domain gap issue in existing systems. The framework establishes a unified feature space that aligns new, diverse agents through a cost-effective and secure backward alignment process, requiring only individual training for the new agents. This approach not only minimizes training costs and maximizes extensibility but also safeguards the model details of the new agents. The authors also introduce a new extensive dataset, OPV2V-H, to advance research in heterogeneous collaborative perception. Experiments reveal that HEAL outperforms state-of-the-art methods, showing a remarkable reduction in training parameters by 91.5% while integrating three new agent types.", "questions": "1. The paper mentioned new agent privacy issue. I assume the late participate will require the new agent to access the fused feature to do the update according to equations in Section 4.3. Will the fused feature release some privacy of the old agents to the new agent?\n2. For the experiments setting, are there any case agent exits the cooperation during training? How the framework will do to deal with this situation.\n3. I am wondering if there are some simple experiments to show real world cases as I mentioned in the weak point.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a interesting open heterogeneous collaborative perception setting. Agents with different sensor can collaborate for vision tasks. This is an interesting and practical setting.\n2. Multi-scale feature fusion and the 'late participation' strategy is reasonable for such tasks.\n3. A dataset contribution. Experiments are extensive. Presentation of the paper is good.", "weaknesses": "1. There is no real-world experiments. There are some dataset like nuScene/nuPlan, Waymo and etc including data of different sensors. It would be nice to show some real examples.\n2. It would be interesting to include a bit discussion on related works for cooperation for driving tasks, e.g. [1][2][3]\n3. I don't find a code release. Would be nice to release the code for supplementary or public github repo.\n\n[1] D Chen and et al. Learning from All Vehicles. CVPR 2022.\n[2] J Cui and et al. Coopernaut: End-to-end driving with cooperative perception for networked vehicles. CVPR 2022.\n[3] R Zhu and et al. Learning to Drive Anywhere. CoRL 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UYzF2570tt", "reviewer": "Reviewer_bqtt", "summary": "This work presents a novel training schema and fusion network for heterogeneous cooperative perception where agents may have different kinds of modalities/sensor configurations. The proposed framework aligns the BEV features from different modalities to the common feature space and fuses them to reason 3D object detections.", "questions": "* The author argued that \"late fusion is suboptimal due to the communication latency\". However, as shown in DiscoNet/V2X-ViT/V2VNet etc., intermediate fusion methods usually require larger bandwidth requirements than late fusion, which can even lead to potentially larger communication latency compared with late fusion. \n* How the result of HM-ViT is reproduced? Is the heterogeneity used for all 4 modalities? Or only two modalities are used as the original paper? \n* What is the inference time of the proposed method? \n* What is the influence of the modality choice in the base collaboration training?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed training design is simple yet effective for faster convergence and higher performance. \n* The proposed methods achieve outstanding performance.", "weaknesses": "* When comparing with other SOTA algorithms, is the same training strategy used for a fair comparison? Is the main performance boost from the training strategy or the multi-modal fusion design? \n* The fusion model design (except the residual part) shows similarity with existing methods like who2com/where2comm/disconet. Please justify and highlight the differences and novelty. Please also benchmark the performance under the same training strategy with only different fusion networks so as to demonstrate the effectiveness of the proposed fusion module. \n* How to ensure each modality can be aligned to the common feature space? For example, it may be hard to extract the 3D features \nfrom camera/radar data which are expected to be as good as (aligned) as the LiDAR features. From 6, we can see that camera bev features are more vague than LiDAR features, which could not show the aligned effect. Please justify this design choice.  \n* Can the network scale to different sensor modality combinations? For example, changing/fixing the ego modality with dynamic collaborator sensor modalities. What is the sensitivity of the network with respect to this sensor modality combination ratio?\n* The 5 hour training time is for single modality. What is the overall training time and the associated time for the compared model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RfZvnVjyE0", "reviewer": "Reviewer_prVY", "summary": "This article introduces a novel problem about collaborative perception, which is extensible heterogeneous agent collaborative perception. To address this problem, a novel extensible collaborative perception framework.is proposed, which includes a novel Pyramid Fusion method. In addition, a new large-scale database is proposed based on OPV2V. The introduced problem is interesting and has significance in engineering. However, the explanation of certain content is not clear enough.", "questions": "1. What’s the structure of the encoder in the proposed framework?\n2. Will the number of input channels for the Pyramid Fusion module change when a new agent is added?\n3. Why can the performance of this method be improved so much on AP70? \n\nSincerely,", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The introduced extensible heterogeneous agent collaborative perception problem is interesting and has significance in engineering.\n2. The proposed HEAL framework is novel and holds state-of-the-art performance.\n3. A new large-scare database is proposed to facilitate the research of heterogeneous collaborative perception.", "weaknesses": "1. The content of the Section 3 is somewhat limited, please make some extensions or merge it with other sections.\n2. The article has not mentioned how to extract features from different modalities and convert them into the required BEV features.\n3. The article has not mentioned how to perform spatial transformation and alignment of the features from heterogeneous agents in different coordinate systems. Please clarify this.\n4. During the training of the new-type agent, the new agent seems not to get collaborative information from other agents, which may have an impact on perception performance. This should be explained.\n5. On the other hand, the Pyramid Fusion module and detection head haven’t been affected by new type agents. It means that new-type agents did not participate in collaborative perception during the training process. This could be better clarified.\n6. The claim of the performance of the proposed Pyramid Fusion should be supported by experiment results and observations, please supplement relevant ablation experiments.\n\nSincerely,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)