RCDN: Towards Robust Camera-Insensitivity Collaborative Perception via Dynamic Feature-based 3D Neural Modeling
Authors: Tianhang Wang
Fan Lu
Zehan Zheng
Zhijun Li
Guang Chen
changjun jiang
Venue: NeurIPS 2024 poster (2024)
Keywords: collaborative perception
Primary area: machine_vision
TL;DR: robust camera-insensitivity collaborative perception

Abstract: Collaborative perception is dedicated to tackling the constraints of single-agent perception, such as occlusions, based on the multiple agents' multi-view sensor inputs. However, most existing works assume an ideal condition that all agents' multi-view cameras are continuously available. In reality, cameras may be highly noisy, obscured or even failed during the collaboration. In this work, we introduce a new robust camera-insensitivity problem: how to overcome the issues caused by the failed camera perspectives, while stabilizing high collaborative performance with low calibration cost? To address above problems, we propose RCDN, a Robust Camera-insensitivity collaborative perception with a novel Dynamic feature-based 3D Neural modeling mechanism. The key intuition of RCDN is to construct collaborative neural rendering field representations to recover failed perceptual messages sent by multiple agents. To better model collaborative neural rendering field, RCDN first establishes a geometry BEV feature based time-invariant static field with other agents via fast hash grid modeling. Based on the static background field, the proposed time-varying dynamic field can model corresponding motion vector for foregrounds with appropriate positions. To validate RCDN, we create OPV2V-N, a new large-scale dataset with manual labelling under different camera failed scenarios. Extensive experiments conducted on OPV2V-N show that RCDN can be ported to other baselines and improve their robustness in extreme camera-insensitivity setting. Our code and datasets will be available soon.

RCDN: Towards Robust Camera-Insensitivity Collaborative Perception via Dynamic Feature-based 3D Neural Modeling
Tianhang Wang Tongji University
Fan Lu Tongji University
Zehan Zheng
Tongji University
waayyowwng@toongi.com edu. cn
luf lufan@tongji edu. en
zhengzehan@tongji edu. cn
Guang Chen Tongji University
Changjun Jiang
Tongji University
guangchen@tooggi. edu, cn
cj cjjiangStongji. edu.
Abstract
Collaborative perception is dedicated to tackling the constraints of single-agent perception, such occlusions, based the multiple agents' multi-view sensor inputs. However, most existing works assume an ideal condition that all agents" multi-view cameras are continuously available. In reality, cameras may be highly noisy, obscured or even failed during the collaboration. In this work, we introduce a new robust camera cumerrainsensitivii problem: how to overcome the issues caused by the failed camera perspectives, while stabilizing high collaborative performance with low calibration cost? To address above problems, we propose RCDN, a Robust Gamera-insensitivine collaborative perception with a novel Dynamic deature-based 3D Neural modeling mechanism. The key intuition of RCDN is to construct collaborative neural rendering field representations to recover failed perceptual messages sent by multiple agents. better model collaborative neural rendering field, RCDN first establishes geometry BEV feature based time-ine time-invariint static field with other agents via fast hash grid modeling. Based on the static background field, the proposed varying dynamic field can model corresponding motion vectors for foregrounds with appropriate positions. To validate RCDN, we create OPV2V-N, new large-scale dataset with manual labelling under different camera failed scenarios. Extensive experiments conducted on OPV2V-N show that RCDN be ported other baselines and improve their robustness in extreme camera- insensitivity settings.
Introduction
Multi-agent collaborative perception 115 obtains better and more holistic perception by allowing multiple agents exchange complementary perceptual effectively address various persistent challenges in information. This field has the potential to such as occlusion[6] []). The associated techniques and systems also process significant promise in various domains, such as the utilization of multiple unmanned aerial aircraft for search and rescue operations [- the automation and mapping of multiple robots [K] As emerging field, the research of collaborative perception faces several issues that need to be addressed. These challenges include the need for high-quality
datasets 17, the formulation of models that are agnostic to specific tasks and models
[9] and
the ability to handle pose error and adversarial attacks(20 21].
*Corresponding Author, code available at: https:/guthub..cm//ppppttidRCCC
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Neisy camera sibuatio duriag collaboraties
However, vast majority of existing works do not seriously account for the harsh realities [22]
Palled
Ocsludd
@
23 of real-world sensors in the collaboration, such as blurred, high noise, interruption and
even failure. These factors directly undermine the basic collaboration premise (atiise[224]]] (2, [5] of recon- structing the holistic view based on the multi- view sensors that severely impact the reliability and quality of collaborative perception process. This raises a critical inquiry: how to overcome the issues caused by the failed cameras' per- spectives while stabilizing high collaborative performance with low calibration cost? The designation camera insensitivity overcomes the unpredictable essence of the specific failure cam- era numbers and time; see Figure | for an il- lustration. To address this issue, one viable solution is adversarial defense(26). By robust defense strategy, adversarial defense bypasses camera insensitivity among blurred and noise. However, its performance suboptimal and has been shown to be particularly vulnerable to noise ratios 20 and failed camera numbers.
Ego agent Citee
No
Whether RCDN?
+ Yes o
Repaired Views
Perpeption under comera failures
Pespeption normal camera
Perception results
RCDN
Figure 1: Illustration of noisy camera situations
(blurred, occluded and even failed) during collabora- tion and the perception result w.o./w. RCDN. orange for drivable areas segmentation, blue for lanes and teal for dynamic vehicles.
To address this robust camera insensitivity collaborative perception problem, we propose RCDN, a Robust cameer--nsensitivityy collaborative perception with Dynamic feature-based 3D Neural mod- cling mechanism. The core idea is to recover noisy camera perceptual information from other agents' views by modeling the collaborative neural rendering field representations. Specifically, RCDN has two collaborative field phases: a Lime-invariant static background field and time-varying dynamic foreground field. In the static phases, RCDN sets other baselines' backbone as the collaboration base and undertakes end end-to-end training to create robust unified geometry Bird-eye view (BEV (28] 29] feature space for all agents. Then, the geometry BEV feature combines the hash grid modeling. an explicit and multi-resolution network, to generate static background views through a-composed accumulation of RGB values along a ray fast speed. In the dynamic phase, RCDN utilizes 4D spatiotemporal position features to model the dynamic motion of 3D points, which learns an accurate motion field under optical priors and spatiotemporal regularization. The proposed RCDN has two major ad advantages: i) RCDN can handle camera insensitivity collaboration under unknown noisy timestamps and numbers; ii) RCDN does not put any extra communication burden into inference stage and costs little computation burden.
In our efforts to validate the effectiveness of RCDN, we identified a gap: the lack of a comprehensive collaborative perception dataset that accounts for different camera noise scenarios. To address this, we create the OPV2V-N, an expansive new dataset derived from OPV2V, featuring meticulously labeled timestamps and camera IDs. This advancement aims to support and enhance research in camera- insensitive collaborative perception. Extensive experiments on OPV2V-N show RCDN's remarkable performance when other baselines equipped with RCDN under extreme camera-insensitiiity setting, improving w.o. RCDN baseline methods by about 157.91%.
2
Related Works
Robust Single Perception. Single-agent perceptions 30 31 27 32-34) have tackled the robust camera setting with other sensor modals. [27 reveals that camera- based methods [4 can be easily effected by camera working conditions. Some works 32 31 introduce LiDAR into perception system and design a soft-association mechanism between the LiDAR and the inferior camera-side to relieve the negative impacts caused by cameras. MVX-Net improves the combination pipeline of LiDAR and cameras by leveraging the VoxeINet architecture. CRN 30 introduces the low-cost Radar to replace the LiDAR, which can provide precise long-range measurement and operates reliably in all environments. However, as for the camera-only situation, few work seeks to solve this because recovering just from the single-view is highly ill-posed (with infinitely many solutions that match the

---

input image). With the recent rapid development of V2X and multi-view based collaborative perception setting explore this extreme situation.
(36, we now can introduce the multi-agent
Collaborative Perception. Perception tasks for single agents can be adversely affected by factors such as limited sensor fields of view and physical ambient occlusions. To address the aforementioned challenges, collaborative perception (3739] can attain more comprehensive perceptual output by exchanging perception data. Early techniques involved the transmission of either unprocessed sensory input (referred to as early fusion) the results of perception (referred to as late fusion). Nevertheless, recent research has been examining the transfer of intermediate features to achieve a balance between performance and bandwidth. Some works 40 devote selecting the most informative messages to communicate. DiscoNet(44] utilizes knowledge distillation to achieve a better trade-off between performance and bandwidth. 12X-ViTA55 presents a unified V2X framework based Transformer that takes into account the heterogeneity of V2X system. Meanwhile, some learnable or mathematical based methods 49 have also been proposed correct the pose errors and latency. Moreover, some works (50, 51 reveal that the holistic character of collaborative perception can improve the effect of driving planning and control tasks. However, most existing papers do not take the harsh realities of real-world sensors into account, such as blurred, high noise, occlusion and even failure, which directly undermine the basic collaboration premise of multi-view based modeling, negatively impacting performance. This work formulates camera- iaaneraiisensstivityy collaborative perception, which considers real-world camera sensor conditions.
Neural Rendering. Neural radiance fields 52 aim to utilize implicit neural representations to encode densities and colors of the scene. This approach takes advantage of volumetric rendering to synthesize views, and it can be effectively optimized from 2D multi-view images. Hence, numerous works have enhanced NeRF in terms of rendering q efficiency [56 etc. For example, (66-Ne utilizes cone tracing instead of ray tracing in standard NeRF volume rendering by introducing integrated positional encoding, which greatly improves the render quality. To improve the efficiency of training and inference processes, Instant-NGP(61] proposes learned parametric multi-resolution hash for efficient encoding, which also leads to high compactness. Some works have also extended NeRF to large-scale urban autonomous scenes 662 64 In this work, first introduce neural rendering to collaborative perception. The proposed collaborative neural rendering field representations will address the problem of recovering highly noisy perceptual messages.
3 Problem Formulation
Consider N agents in a scene, where each agent can send and receive collaboration messages from other agents. For the n-th agent, let X = (Ih j and )it, be the raw observation and the perception ground-truth at time current 4, respectively, where Ih is the e-th camera images recorded at i-th timestamp, and Pri.., is the collaboration message sent from the agent m at time t,, The key of the camera insensitivity is that the specific noisy camera number and corresponding timestamp are unpredictable. Therefore, each agent has to encounter invalid view information, which contains both local observation and collaboration messages sent from other agents. Then, the task of camera insensitivity collaborative perception is formulated as:
N
max 1.0
Σ
g
Y
Y!:
==
subject to Y4 = co. co, (W(2) (Phh-n)) )))),
is the perception result of the nith agent at time t, ()( is the camera noise function to simulate the harsh realities of the real-world situation, πT. (-) is the proposed collaborative neural rendering field network RCDN with trainable parameters 01. and co, is the existing collaborative perception network with trainable parameters og. Note that the proposed RCDN is to recover the noisy camera views caused by the & function, making collaborative perception system more robust to the unpredictable situation of noisy camera data.
where o(.,-) is the perception evaluation metries, Y
Given such high noisy camera view, the performances of collaborative perception system would be significantly degraded since the mainstream collaborative perception utilizes the multi-view camera- based BEV features for communication and downstream tasks, and using such damaged features
3

---

Geometry BEV Generation
Collaborative Statie Newral Field
Messoge Exchange
V
Sompling
(200)
Concat.
M
MLP
Render
MCP Encoder
3D Hash
Geometry BEV Volume
Foature Mash Table
Statte Scrrn
Collaborative Dynamic Neural Field
RCDN
RCDN
BEV Decoder
F. (x)
Optical Flore
4D Hash
run
Blonding
Interpolation
Lign
Aggregated BEV Feature
Palfed
Sherree
Figure System overview. The geometry BEV generation module provides feature sampling for later processes. The collaborative static and dynamic fields are performed in parallel to model the background and foreground, respectively. Note that MCP is short for the multi-agents collaborative perception process.
would contain erroneous information during the perception process. In the next section, will introduce RCDN to address this issue.
RCDN
This section proposes robust camera-insensitivity collaborative perception system, RCDN. Figure overviews the framework of the RCDN module in Sec The details of three key modules of RCDN be found in Sec
4.1 Overall Architecture
The problem of noisy camera view results in the sub-optimization the holistic multi-view based BEV features generation in the collaboration messages. That is, the collaboration messages from both self and other agents would be noisy or damaged for the fusion process. The proposed RCDN addresses this issue with two key notions: i) we construct novel collaborative neural rendering field representations, enabling collaborative perception to recover from the noisy camera view; and ii) we establish time- imeeinvariian and time-vare varying fields for background and foreground, respectively, making the collaborative neural rendering field more accurate.
Mathematically. let the v--hh agent be the ego agent and X be its raw observation at the I, timestamp of agent n. The proposed camera- inness-insenssttviiy collaborative perception system RCDN formulated as follows:
F Seno (bb((, (25) )
(2a)
Vt: fgeo (F kev
(2b)
(0" c²)
faratic (aaic(r(uug) (r(ux), Vul ien (r(w)))), faynamic (r (u(u), Vh tr(
(2c)
w Sbws 0 C/ b)
d
(2d)
24. N j==1
frender (a", c² of b), b),
(2c)
t fimp (1) (A poa )
(2
where Ft. E RCCH is the BEV feature maps of the n-th agent timestamp f, with H, W the
size of BEV map and C the number of channels; V igo RCx xxw xW is the implicit collaborative geometry volume feature of the scenarios; which is lifted from BEV plane with the Z height; R² through given pixel on the image + u(k)d, where d E R is the normalized viewing direction; fatatic is
r(u(k)) is the ray from the failed camera center plane as r(u(k)) explicit hash grid based representation to model the collaborative static scenarios volume density o² E IR_ and corresponding color c* E 33, faynamic is the dynamic collaborative neural network

---

takes the interpolated 4D-tuple (r(u(k)),tt) t;) and sampled VL. ice feature as input and predict 3D collaborative scene flow vectors Sww: Sbw E R dynamic volume density of color ci, and blending weight E RR;; and Rh: (2) N-1 is the recovered noisy camera images at timestamp 4 after 1j=1"
collaborative rendering; and Y is the final output of the system. In summary, Step 2a extracts BEV perceptual features from observation data. feature map for each timestamp, enabling feature sampling in Step and 2d Step 2d models the static background field of collaboration scenarios. 2d models the dynamic foreground field of collaboration objects. 2tep/2egget gets the global volume density and color information by combining both static and dynamic field models to recover the failed camera perspective images. Finally, Step 21 outputs the final perceptual results with repaired images.
the collaborative geometry BEV volume
Note that i) Step 2a is done locally, Step 2b 2f are performed after receiving the messages from others. The proposed RCDN does not require any extra transmission during the inference process, which is bandwidth friendly; and ii) Step 2c and 2d]are are performed in parallel to save inference time; and iii) Same [44] 49], RCDN adopts the feature representations in bird's eye view (BEV), where the feature maps of all agents are projected to the same glohal coordinate system. We now elaborate on the details of Steps 2b Zelin the following subsections.
4.2 Collaborative Geometry BEV Volume Feature
Given the BEV feature map of each agent, Step 26 aims to construct unified collaborative geometry BEV volume feature for each timestamp of the scenario. The intuition is that 665 points out that combing with generic feature representations can avoid the per-scene "network memorization" phenomenon [52) which will improve the efficiency of the optimization process. Therefore, using the geometry BEV feature can enable the subsequent Step 2c 2d to learn more generie networks for both static and dynamic collaborative neural fields, respectively.
To implement, we use a geometry-aware decoder Deco to transform the BEV feature Ft: into the intermediate feature F'. E RXXIXXXY and F height, E R xlIxxxXXY and this feature is lifted from BEV plane to implicit collaborative volume feature sev RRxxZxXXY
icD = sigmoid
height, (heightt,) -F':
where represents dot production along the channel. Eq. lifts the items on the BEV plane into 3D collaborative volume with the estimated height position sigmoid(F mhight,.)) sigmoid(F (FeigiidFFeigtr..... height, represents whether there is an item the corresponding height. Ideally, the collaborative volume feature Vic.
icv contains all the scene items information in the corresponding position.
4.3 Static Collaborative Neural Field
After getting the collaborative volume feature V4, icu: Step aims to construct the background of camera views with the static collaborative neural field. Given an arbitrary 3D scenario position x E I and a 2D viewing direction d E TR.2, we aims to estimate static scenarios volume density o² and emitted RGB color c* using the fast hash grid- [61) neural network:
(c*,o*) (L (ccott/cot d); f= f= V
(4)
where f VL, (x) is the neural feature trilinearly interpolated from the collaborative geometry BEV volume V at the location x. G- is explicit multi-level hash grid representation with the generic f features for fast static collaborative neural field training. Meanwhile, owing to the collaborative scenarios are unbounded, we utilize contract( (533 to map 3D scenario position into a bounded ball of radius 2 with regularization, making the estimation optimization process faster and better. Hence, we can compute the color of the pixel (corresponding the ray r(u;) using numerical quadrature for approximating the collaborative volume rendering interval [66]
K
(+
(ux)a* (0 (wx))) (ux).
(5a)
1=1
T (wx) exp
Σ
of (ux)os
(5b)
=
5

---

where (x) 1 - exp(-z) and 6₁ uk-1 - is the distance between two quadrature points. The K quadrature points (u [= are drawn uniformly between Un and Uy, which denotes the near and far of the bounded collaborative scenarios. T (wx) indicates the accumulated transmittance from Un to Ug. Here, we denote r; as the rays passing through the pixel Then, the collaborative static neural loss Lisattic is defined to minimize the 12-loss between the estimated colors C* (r,) and the ground truth colors C (r.) in the static regions (where M(r.) = 0):
Lstaati
- (r,) (1 (rr))
M(r:))
4.4 Dynamic Collaborative Neural Field
While the static collaborative neural field is being modeled, Step 2d]is building the dynamic collabo- rative neural field to construct the foreground of camera views. Our dynamic collaborative neural field takes 4D spatiotemporal position features as input to model dynamic motion of 3D scene flow 8f Stur, volume density of color ci, and blending weight b (Note that blending weights learns how to blend the results from both the static and dynamic collaborative neural fields in an unsupervised manner, avoiding background's ss structure and appearance conflict the moving objects.):
(s Sbw C, of b) = MLP(A(G)) (contract(() d), f = Vicc
(7)
where G% shares the same hash grid representations, but for the dynamic collaborative neural field optimization; (): is the temporal interpolation functions, which makes the MLP can efficiently learn the features between keyframes in a scalable manner. Meanwhile, to improve the temporal consistency of the proposed field, we compute the collaborative scene flow neighbors r(ug) + Syw and r(wx) - Sbw with the predicted collaborative scene flow f Sbw to warp the collaborative neural field from the neighboring time instance to the current time. Note that the term S fw stands for forward scene flow, while Sbe refers to backward scene flow. Specifically, the forward scene flow (s) estimates the flow from time I to t+1, whereas the backward scene flow (Stw) estimates the flow from time to t-1. Hence, we obtain the corresponding density and color of adjacent time by querying the same MLPs model at r ((x)) + s:
(c, 1 d = ( (contract (x Stw), d), + 1))
(8a)
((,1) = (I. [I (contract(x
Sbw)), d), 4 1))
(8b)
We can compute the color of a dynamic pixel of collaborative view at time f;. Hence, with both the static and dynamic collaborative neural fields model, we can easily compose them into a complete model using the predicted blending weight and render full color fful (r) frames at noisy views and time. We utilize the following approximate of collaborative volume rendering integral:
K
cf full (r) = > fill (o4 -)cc + (o* bb))c²)
(9)
Similar to the static collaborative rendering loss, we train the dynamic collaborative neural model by minimizing the le reconstruction loss under time unit (t,, 1, + 1}:
Lidgn
EEI(C! LET
(r:) C (r:))
(10)
To reduce the amount of ambiguity caused by the sparse views during collaborative perception process, we construct motion matching loss to constrain the proposed dynamic collaborative neural field. As we do not have direct 3D supervision for predicted collaborative scene flow from the motion MLP model, we utilize 2D optical flow f as indirect supervision. Specifically, we first use the estimated collaborative scene flow to obtain the corresponding 3D point. Then, we project these 3D points onto the 2D reference frame with sp-)) function. Hence, we can compute the projected collaborative scene optical flow and enforce it match the estimated optical flow follows:
Lept Σ
- [ow (r.)
(11)
Meanwhile, we also regularize the consistency of the collaborative scene flow by minimizing the cycle consistency loss Leye. See more details in the Appendix B.7.

---

Table 1: Map-view segmentation of different baseline methods w.//w the proposed RCDN on OPV2V-N camera-track with one random noisy camera failure in the testing phase. We report IoU for all classes.
Static Part (Perf. Comparison)
Dynamic Part Vehicle
Model/ Metric
Drivable Area
Lame
Failure
Failure
Fuilure
Normal
Normal
Normal
waa/w. RCDN
woo/w. RCDN
w.a/w, RCDN
F-Cooper D
45,44
33.17
63.33
29.70/61.76(1 107 (55%)
AttRuse I6
45.59
27.99/44.38/ 33994444881188569)
33.76
18.77/31.500167..22 (22%)
54.14
24.76/52. 15.661)
DiscoNet [4
42.30
24.31/38.54 (5(855545))
24.24
12.29/22.97 90%)
46.56
9.25/43.03(
(13833033336611101 1365.199)
V2V 41.70
27.99/39.72(141.) (99119992((11.9999)
27.14
(1.22/5.244(139.00
42.57
11.28/2.76611773)
CoBEVT 51.96
32.08/47.19(147.10888 19(147.109)
34.19
14.45/29.55(1 15045505
56.61
32,41/55.110 (01//..)))7
4.5
Training Details and Optimization
To train the overall system, we supervise two tasks: static and dynamic collaborative neural fields, respectively. Meanwhile, during the training process, the static collaborative field and dynamic collaborative field are trained separately. The initial learning rate is 5e-4 with the exponential learning rate decay strategy. The weight values are set to 1.0, 1.0, 0.1, and 1.0, respectively:
Ltotal =
Lstatic + X2Ldyn + AsCopt + daCaye
(12)
5 Experimental Results
We create the first camera-insensee inneessiiisestttiittty collaborative perception dataset and conduct extensive ex- periments on OPV2V-N, To ensure the consistency of the input noisy camera data and verify the effectiveness of RCDN, we set the noisy camera data to be in the failed situation [7]] Meanwhile, the task of the experiments is map segmentation, including the performance of the vehicle, drivable area (Dr. area) and lane, totaling three classes. We utilize the Intersection over Union (IoU) between map prediction and ground truth map-view labels as the performance metric.
5.1 Datasets
OPV2V-N. To facilitate research on camera-insensitivity for collaborative perception, we propose simulation dataset dubbed OPV2V-N. In OPV2V dataset, there is a lack of mask labels for distin- guishing between foreground and background views, as well as optical flow labels for supervising the scene flow. For this purpose, we collect more data to bridge the gap between neural field and collabo- rative perception, leading to the new OPV2V-N datasets. Specifically, we utilize the OneFormer (67) detector to extract the foreground mask labels and mainstream RAFT(68] detector to compute the optical flow between image pairs. Meanwhile, we manually annotate which part of the performance degradation is triggered by camera failure in different scenarios. See more details in the Appendix A.
5.2 Quantitative Evaluation
Benchmark comparison. The baseline methods include F-Cooper [1], [16), DiscoNet 44) V2VNet and CoB All methods use the same BEV feature encoder based on CVT[69 To validate the portability of the RCDN, we compare different baseline methods w.o/w. RCDN under unpredictable camera failure settings. Table [ shows that the map-view segmentation per- formance of different baseline methods w.o/w. the proposed RCDN with only one number ran- dom noisy camera failure in the testing phase on the OPV2V-N dataset. We see that i) for static part, each baseline method with one camera failure drops about 37. (Avg/Max/Min) and 52.93%/0 /61 25%/44.400 ever, each baseline method w. RCDN under the same camera failure situation only decreases about 5.34% 99.1%%9997///// (1.7%%/1.22/ /1.22% and 7.08%/13.0 19.59% /2.8%% respectively. Compared to the w.o RCDN baseline methods, RCDN can improve the performance of drivable area and lane for 52.32% //////58.44/4...../
7...54%%/...777 /3.877% for drivable area and lane, respectively. How-
and 100.37% 100037%/133...0 00t.92%667.88. part, as we all know, the fusion stage in collaborative perception process needs more effort on the multi-view based BEV feature map to highlight the corresponding dynamic part. Hence, the
respectively; ii) compared to the static
7

---

P-copperw BCDN
AulFuse RCDN AnrFuse
DiscoNet RCDN DiscoNet
VIVNR RCDN V2V
COBEVT RCDN COBEUT
P-cooger
25-
65-
1et tnt/owaal cleenee
15-
15-
Randoon masp canara fallim ncmbbes Performance static part Ama
Rendon mayy carara fallee nunbas (b) Performanos shitc part
Randon mass caner fullmn sumbers Perforance dynamic gort
Figure 3: Comparison the performance of other baseline methods w/w/w the proposed RCDN under random noisy (failed situation) camera numbers from to 3. RCDN can be ported to other baseline methods and stabilize the performance under different level camera failure situations on OPV2V-N dataset.
model
model
Table 2: Ablation Study on OPV2V-N dataset.
Modules
Dr. Area
Lanes
Dynamic Veh.
Neural Field
Time Model
30.67
x
24.47
11.71
41.55
V
27.37
10.63
46.65
Figure 4: Effectiveness of dynamic neural field.
baseline methods' dynamic performance suffers more from camera failure than the static part, caus- ing about 60.75% 42. 72% /10%//880.%/ performance drop. Nevertheless, RCDN also demonstrates robustness to the dynamic foreground object modeling, with only 3.31% 1/11 /1/5//558/////// /0.4%% perfor- mance decrease for the dynamic part, improving the w.o RCDN baseline methods' performance by 186.57% 111///35..199 10.55.99/700 70.000109 .1.00 Meanwhile, as for the communication cost, similar to [44,, we only utilize the C labels during the training stage, meaning we leave the communication burden the training stage and do not introduce extra information during the inference.
Robust to extremely noisy camera data. We conduct experiments validate the performance under the impact of random noisy camera numbers. Figure 3 shows the map-view segmentation performance of the different basclines methods w.o/w. the proposed RCDN under varying levels of camera failures situation on OPV2V-,, where the z-axis is the expectation of the number of random failed cameras during the inference stage and y-axis the segmentation performance. Note that, when the z-axis is at O, it represents standard collaborative perception without any camera failures. We see that the proposed RCDN can stabilize all the baseline methods in both static and dynamic part of map-view segmentation performance at all camera failure settings; ii) as for the static part, with the RCDN can maintain the 87.84 87.84% 44/8..777 ///8/88..7. 7 22%%//6.477 64% Dr. area performance of the standard setting even under three random failed views during the collaboration process, compared with the w.o. RCDN only about 47. 68%/57.488 /37. 15%. Note that the V2VNet baseline method's performance degrades sharply as the failed camera number increases, however, with RCDN, the V2VNet can settle considerable performance even with the failed camera number increases; iii) as for the dynamic part, some baseline methods are crashed even with only one random camera failure situation, e.g. DiscoNet only maintains about 19.87% performance of the standard collaborative perception setting, and almost every baseline method is unusable when there three random camera failures, only about 20.73% /8. 11% 13.09% of the standard situation. Nevertheless, with the RCDN, we see that all baseline methods still perform well when three random failed camera situation appear, maintaining the 84.95% 19.5%%/90.8. /9..999..119 1% 75.9%% dynamic performance of the standard situation.
5.3 Qualitative Evaluation
Visualization of segmentation. We illustrate the map-view segmentation of other baseline methods w.o/w. RCDN and the corresponding repaired camera view Figure The random camera failure number is one. The orange represents the drivable area, the blue represents the lanes and the teal
8

---

CaBEVT
Repaired Views
(b) Origin Views
mto. RCDN
(d) w. RCDN
(e) Origin segmentation
Figure 5: Visualization of different baseline methods w. RCDN with one random camera failure.
represents the vehicles. We that i) other baseline methods show significant improvement in RCDN under noisy camera data; ii) V2VNet that collapses with noise camera data can also achieve the same level of performance the origin data with the help of RCDN.
5.4 Ablation Study
Components analysis We conduct ablation studies on OPV2V-N with the CoBEVT base- line method. Table Dassesssssssess the effectiveness of the proposed two field phases. We see that i) only one neural field can recover most static part performance from the noisy camera data; ii) the proposed time model in collaborative dynamic fields can handle the motion blurry caused by the vehicles, shown in Figure 4 Meanwhile, we compare the training efficiency of the proposed RCDN with existing dynamic fields modeling methods [70] as illustrated in Figure 6 Our ap- proach, which leverages explicit grid and geometry feature- representations, accelerates the training process by approximately 24x compared to the existing implicit MLP-based modeling, while also achieving superior PSNR quality. See more discussions in the Appendix B.2.
Impticil MLFF P-barol dynumi: Mocking (b) Explicit Grid-bsoo RCDS Mol-ting
Braining (PSN 21,33 About baining
Figure 6: Comparison between existing dynamic field modeling and the proposed RCDN.
Performance bottlenecks Regarding the increasing number of agents and cameras, we validated the impact of adding more cameras using the OPV2V-N dataset (corresponding scenario types are T section and midblock respectively) with the CoBEVT baseline. From Table 3 we observe the following: i) With a single overlapping camera view, the proposed method significantly improves baseline performance, and ii) While theoretically, more cameras can provide larger overlap range, addition of multiple cameras (depending on their positions) may introduce redundant viewing angles, resulting in less significant performance improvements.

---

Table 3: Map-view segmentation performance validation about the increasing number of cameras under OPV2V-N datasets with CoBEVT baseline. Note that the failure setting is under one random noisy camera failure in the testing phases. report IoU for all classes.
Overlap Cameras
Methods
Metrics
Scene
Failure
+1
+2
+3
T Section
23.23
26.97
26.91
27.23
Dr. Area
Midblock 23.43
38.87
38.94
39.51
CoBEVT
Section 18.83
40.72
41.38
42.29
Dyn. Vehicles
Midblock
16.57
45.60
48.31
49.88
Table 4: Detection performance of CoBEVT and V2VNet baseline meth- ods w.a/w. the proposed RCDN OPV2V-N dataset with one random noisy camera failure in the testing phase. We report Average Precision (AP) Intersection over-Union Union (loU) thresholds of 0.50 and 0.70.
COBEFT
Methods
Nommal
MMtrrcs
AP@0.5001)
AP@0.70(1)
COBEVT
V2VNe<37
Methods
H.Wfe RCDN
Metries
AP@0.5001)
AP@0,70(f)
Nurmal
RCDS wnder moise
w.O HC 'DN umder noroe
CoBEVT H
46.67/55.56
34.57/43.21
V2VNe
45.45/53.85
36.37/38.18
Figure 7: Visualization of proposed RCDN for detection downstream task performance. Note that red and green boxes denote detection results and ground-truth respectively.
Different downstream tasks Our proposed RCDN is general to different downstream tasks and limited to just BEV segmentation. We focus on BEV segmentation due to its crucial role in autonomous driving, with direct applications to other tasks such as layout mapping, action prediction, route planning. and collision avoidance. Additionally, we have validated RCDN for detection tasks, shown in Figure 7 We replaced the original segmentation header with a detection header in our experiments. Table 4 shows that for CoBEVT, using RCDN improves metrics of AP@0.50 and AP@0.70 by 19.05% and 24.99%, respectively.
6 Conclusion and Limitation
We formulate the camera- camera-insentn insensitivity collaborative perception task, which considers harsh realities of real-world sensors that may cause unpredictable random camera failures during collaborative communication. We further propose RCDN, robust camera-insensittvity collaborative perception with a novel dynamic feature-based
neural modeling. The core idea RCDN is to construct collaborative neural rendering field representations to recover failed perceptual messages sent by multiple agents. Comprehensive experiments show that RCDN can be portable to other baseline methods and stabilize the performance with considerable level under all settings and far superior robustness with random camera failures.
Limitation and future work. The current work focuses on addressing the cumera-insensitie problem in collaborative perception. It is evident that accurate reconstruction can compensate for the negative impact of noisy camera features on collaborative perception. In the future, we expect more works on exploring real-time collaborative neural field modeling with 3D Gaussian splatting.
7 Acknowledgments
This work was supported by the National Key Research and Development Program of China (No. 2021 YFB2501 104), in part by the National Natural Science Foundation China (No. 62372329), in part by Shanghai Scientific Innovation Foundation (No. 23DZ1203400, in part by Pongji-Qomolo Autonomous Driving Commercial Vehicle Joint Lab Project, and in part by Xiaomi Young Talents Program.
10

---

References
[1] Qi Chen, Xu Ma, Sihai Tang, Jingda Guo, Qing Yang. and Song Fu. F-cooper: Feature hased cooperative perception for autonomous vehicle edge computing system using 3d point clouds. Proceedings of the 4th ACM/IEEE Symposium on Edge Computing, pages 88-100, 2019.
[2] Runsheng Xu, Yi Guo, Xu Han, Xin Xia, Hao Xiang, and Jiaqi Ma. Openeda: an open cooperative driving automation framework integrated with co-simulation. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 1 1155-1162. IEEE, 2021.
[3] Yiming Li, Dekun Ma, Ziyan An, Zixun Wang, Yiqi Zhong, Siheng Chen, and Chen Feng. V2x-sim: ageti- collaborative perception dataset and benehmark for autonomous driving. IEEE Robotics and Automation Letters, 7(4):10914 14-11921, 2022.
[4] James Tu, Tsunhsuan Wang. Jingkang Wang, Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun. Adversarial attacks multi- agent communication. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7768 7777, 2021.
[5] Jiaxun Cui, Hang Qiu, Dian Chen, Peter Stone, and Yuke Zhu. Coopernaut: End-to-end driving with cooperative perception for networked vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17252-17262. 2022.
[6] Runsheng Xu, Zhengzhong Tu, Hao Xiang, Wei Shao, Bolei Zhou, and Jiaqi Ma. Cobevt: Cooperative bird's eye view semantic segmentation with sparse transformers. 2022.
[7] Hao Xiang, Runsheng Xu, and Jiaqi Ma. Hm-vit: Hetero-modal vehicle-to-vehicle cooperative perception with vision transformer. arXiv preprint arXiv:23044. 10628, 2023.
[8] Ebtehal Turki Alotaibi, Shahad Saleh Alqefari, and Anis Koubaa. Lsar: Multi-uav collaboration for search and rescue missions. JEEE Access, 7:55817 17-5532, 2019.
[9] Jürgen Scherer, Saeed Yahyanejad, Samira Hayat, Evsen Yanmaz, Torsten Andre, Asif Khan, Vladimir Vukadinovie, Christian Betistetter, Hermann Hellwagner, and Berhaard Rinner. An autonomous multi-uav system for search and rescue. In Proceedings of the First Workshop on Micro Aerial Vehicle Networks, Systems, and Applications for Civilian Use, page 33-38, New York, NY, USA, 2015.
[10] Yue Hu, Shaoheng Fang, Weidi Xie, and Siheng Chen. Acrial monocular 3d object detection. IEEE Robotics and Automation Letters, 1996 2023.
[11] Lukas Bernreiter, Shehryar Khattak, Lionel Ott, Roland Siegwart, Marco Hutter, and Cesar Cadena. A framework for collaborative multi-robot mapping using spectral graph wavelets. The International Journal of Robotics Research, 000)027783642241224888
[12] Luiz Eugènio Santos Araújo Filho and Cairo Lúcio Nascimento Júnior. Multi-robot autonomous exploration and map merging in unknown environments. In 2022 IEEE International Systems Conference (SysCon), pages 1-8, 2022.
[13] Yiming Li, Juexiao Zhang, Dekun Ma, Yue Wang, and Chen Feng. Multi-rohot scene completion: Towards task agnostic collaborative perception. In 6th Annual Conference on Robot Learning. 2022.
[14] Runsheng Xu, Xin Xia, Jinlong Li, Hanzhao Li, Shuo Zhang, Zhengzhong Tu, Zonglin Meng, Hao Xiang, Xiaoyu Dong, Rui Song, et al. V2v4real: A real-world large-scale dataset for sehicle-to-vehicke cooperative perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13712-13722, 2023.
[15] Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang. Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, et al. Dair-v2x: large-scale dataset for vehicle-infrastmictuee cooperative 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21361-21370 2022.
[16] Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma. Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. 2022 International Conference on Robotics and Automation (ICRA), pages 2583-2589. IEEE, 2022.
[17] Walter Zimmer, Gerhard Arya Wardana, Suren Sritharan, Xingcheng Zhou, Rui Song, and Alois C. Knoll. Tumtraf v2x cooperative perception dataset. In 2024 JEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR). IEEE/CVE, 2024.
11

---

[18] Runsheng Xu, Jinlong Li, Xiaoyu Dong. Hongkai Yu. and Jiaqi Ma. Bridging the domain gap for multi-agent perception. arXiv preprint arX1v:2210.08451, 2022.
[19] Yunsheng Ma, Juanwu Lu, Can Cui, Sicheng Zhao, Xu Cao, Wenqian Ye, and Ziran Wang. Macp: Efficient model adaptation for cooperative pereeption. In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision, pages 3373-3382, 2024.
[20] Yiming Li, Qi Fang, Jiamu Bai, Siheng Chen, Felix Juefei-Xu Xu, and Chen Feng. Among us: Adversarially robust collaborative perception by consensus, In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 186- 195, October 2023.
[21] James Tu, Tsunhsuan Wang. Jingkang Wang, Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun. Adversarial attacks on multi-agent communication. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7748 2021,
[22] Francesco secci and Andrea Coccarelli. On failures of rgh cameras and their effects in autonomous driving applications. In 2020 IEEE s1 International Symposium on Software Reliability Engineering (ISSRE), pages 13-24, 2020.
[23] Kui Ren, Qian Wang, Cong Wang, Zhan Qin, and Xiaodong Lin. The security of autonomous driving: Threats, defenses, and future directions. Proceedings of the IEEE, 108(2):357372,, 2020.
[24] Rui Song, Chenwei Liang, Hu Cao, Zhiran Yan, Walter Zimmer, Markus Gross, Andreas Festag, and Alois Knoll. Collaborative semantic occupancy prediction with hybrid feature fusion in connected automated vehicles. 2024.
[25] Xiang Li, Junbo Yin, Wei Li, Chengzhong Xu, Ruigang Yang, and Jianbing Shen. Di-v2x: Learning domain-invariant representation for vehicle-infrastrucuuee collaborative 3d object detection. Proceedings of the AAAI Conference on Artificial Intelligence, 38(4):3208-322 Mar. 2024.
[26] Hao Xiang, Runsheng Xu, Xin Xia, Zhaoliang Zheng, Bolei Zhou, and Jiaqi Ma. V2xp-asg: Generating adversarial scenes for vehicle-to-ewen -erer-tttterriinggggeee. perception. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3584-3591, 2023.
[27] Kaicheng Yu, Tang Tao, Hongwei Xie, Zhiwei Lin. Tingting Liang, Bing Wang, Peng Chen. Dayang Hao. Yongtao Wang, and Xiaodan Liang. Benchmarking the robustness of lidar-camera fusion for 3d object detection. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3188-3198 2023.
[28] Zhiqi Li, Wenhai Wang. Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer:
Learning bird's s-yeevviw representation from multi-camera images via spatiotemporal trans- formers. In European conference computer vision. pages 1-18. Springer, 2022.
[29] Chenyu Yang. Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang. Hongyang Li, Yu Qiao, Lewei Lu, et al. Bevformer v2: Adapting modern image backbones to bird's e-ye- view recognition via perspective supervision. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 17830-17839, 2023.
[30] Youngseok Kim, Juyeb Shin, Sanmin Kim, In-Jac Lec, Jun Won Choi, and Dongsuk Kum. Cm: Camera radar net for accurate, robust, efficient 3d perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 17626, October 2023.
[31] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Trans- fusion: Robust lidar-camera fusion for 3d object detection with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1090-1099, June 2022.
[32] Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang. and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 10421-10434. Curran Associates, Inc., 2022.
[33] Vishwanath A. Sindagi, Yin Zhou, and Oncel Tuzel. Mvx-net: Multimodal voxelnet for 3d object detection. In 2019 International Conference on Robotics and Automation (ICRA), pages 7276-7282, 2019.
[34] Yue Wang, Vitor Campagnolo Guizilini. Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors. Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 180-191. PMLR, 08 11 Nov 2022.
12

---

[35] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4490-4499, 2018.
[36] Maria Christopoulou, Sokratis Barmpounakis, Harilaos Koumaras, and Alexandros Kaloxylos. Artificial intelligence and machine learning key enablers for v2x communications: A comprchensive survey. Vehicular Communications, 39: 100569, 2023.
[37] Houn-Hsuan Wang, Sivabalan Manivasagam, Ming Liang, Bin Yang. Wenyuan Zeng, and Raquel Urtasun. V2vnet: Vehicle-to-vehicle communication for joint perception and prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pages 605-621, 2020.
[38] Yifan Lu, Yue Hu, Yiqi Zhong, Dequan Wang, Siheng Chen, and Yanfeng Wang. An extensible framework for open heterogeneous collaborative perception. In The Twelfth International Conference on Learning Representations, 2024.
[39] Yue Hu, Yifan Lu, Runsheng Xu. Weidi Xie, Siheng Chen, and Yanfeng Wang. Collaboration helps camera overtake lidar in 3d detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9243-9252, 2023.
[40] Yen- Ben-Cheng Liu, Junjiao Tian, Chih Yao Ma, Nathan Glaser, Chia- Wen Kuo, and Zsolt Kira. Who2com: Collahorative perception via learnable handshake communication. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 6876-6883, 2020.
[41] Yen-Cheng Liu, Junjiao Tian, Nathaniel Glaser, and Zsolt Kira. When2com: multi-agent perception via communication graph grouping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4106-4115, 15, 2020.
[42] Yue Hu, Shaoheng Fang. Zixing Lei, Yiqi Zhong, and Siheng Chen. Where2comm: Communication- efficient collaborative perception via spatial confidence maps. Advances in neural information processing systems, 35:4874-4886, 2022.
[43] Tianhang Wang. Guang Chen, Kai Chen, Zhengfa Liu, Bo Zhang, Alois Knoll, and Changjun Jiang. Ume: A unified handwidth-efffcient and multi-resolution hased collahorative perception framework. In 2023 IEFF/CVF International Conference on Computer Vision (ICCV), pages 1166- 2023.
[44] Yiming Li, Shunli Ren, Pengxiang Wu, Siheng Chen, Chen Feng, and Wenjun Zhang. Learning distilled collaboration graph for multi-agent perception. Advances Neural Information Processing Systems, 34:29541 295522, 2021.
[45] Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang, and Jiaqi Ma. V2x-vit: Vehicle- to-everything cooperative perception with vision transformer. ArXiv, 1ttp/2233.16633 2022.
[46] Nicholas Vadivelu. Mengye Ren, James Tu, Jingkang Wang, and Raquel Urtasun. Learning to communicate and correct pose errors. In 4th Conference on Robot Learning (CoRI.), 2020.
[47] Yifan Lu, Quanhao Li. Baoan Liu, Mchrdad Dianati, Chen Feng, Siheng Chen, and Yanfeng Wang. Robust collaborative 3d object detection in presence of pose errors. In 2023 IEEE International Conference on Robotics and Automation (ICRA). pages 4812-4818. IEEE, 2023.
[48] Zixing Lei, Shunli Ren, Yue Hu, Wenjun Zhang, and Siheng Chen. Latency-aware collaborative perception. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII, pages 316-332. Springer, 2022.
[49] Sizhe Wei, Yuxi Wei, Yue Hu, Yifan Lu, Yigi Zhong, Siheng Chen, and Ya Zhang, roynchrony-robust collaborative perception via bird's eye view flow. In Advances in Neural Information Processing Systems, 2023.
[50] Dian Chen and Philipp Krähenbühl. Learning from all vehicles. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 17222-17231, 2022.
[51] Ruizhao Zhu, Peng Huang, Eshed Ohn-Bar, and Venkatesh Saligrama. Learning to drive anywhere. In 7th Annual Conference on Robot Learning, 2023.
[52] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[53] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti- alissed neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470- 5479, 2022.
13

---

[54] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti- aliased grid-based neural radiance fields. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 19697-19705, 2023.
[55] Wenbo Hu, Yuling Wang. Lin Ma. Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma. Tri-miprf: Tri-mip representation efficient anti- aati-aliasing neural radiance fields. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19774-19783, 2023.
[56] Sara Fridovich- Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5501 -5100, 2022.
[57] Cheng Sun, Min Sun, and Hwann- Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[58] Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowei Zhou, Zexiang Xu, and Hao Su. Tensoir: Tensorial inverse rendering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2023.
[59] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkütler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4): 14, 2023.
[60] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2021.
[61] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG), 41())11-15, 2022.
[62] Konstantinos Rematas, Andrew Liu, Pratul Srinivasan, Jonathan Barron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Ferrari. Urban radiance fields. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 12932-12942, 2022.
[63] Fan Lu, Yan Xu, Guang Chen, Hongsheng Li, Kwan- Yee Lin, and Changjun Jiang. Urban radiance field representation with deformable neural mesh primitives. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 465-476 2023.
[64] Fan Lu, Kwan-Yee Yee Lin, Yan Xu, Hongsheng Li, Guang Chen, and Changjun Jiang. Urban architect: Steerable 3d urban scene generation with layout prior, arXiv preprint arXiv:2404.06780, 2024.
[65] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[66] Robert A Drebin, Loren Carpenter, and Pat Hanrahan. Volume rendering. ACM Siggraph Computer Graphics, 22(4):65-74, 1988.
[67] Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. OneFormer: One Transformer to Rule Universal Image Segmentation. 2023.
[68] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow, In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part 16, pages 402-419. Springer, 2020.
[69] Brady Zhou and Philipp Krähenbühl. Cross-view transformers for real-time map-view semantic seg- mentation. In 2022 JEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13750-13755 2022.
[70] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEFE International Conference on Computer Vision, 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: This paper does discuss the limitations of the work performed by the authors.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper fully discloses all the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
This paper provides open access to the data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee 7/nips.com / cassiicyncccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: This paper specify all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: This paper reports appropriate information about the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: This paper provides sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http::
Answer: [Yes]
Justification: This research conducted in the paper conform, in evrery respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is no societal impact of the work performed. No harm technical paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We mentioned creators original owners of assets and properly respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper introduces assets well documented.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "onAGybRG08", "reviewer": "Reviewer_qUQv", "summary": "The paper introduces RCDN, a novel method for robust camera-insensitivity collaborative perception. This method aims to overcome challenges associated with noisy, obscured, or failed camera perspectives by using dynamic feature-based 3D neural modeling. RCDN constructs collaborative neural rendering field representations to recover failed perceptual messages sent by multiple agents. The proposed system consists of two collaborative field phases: a time-invariant static background field and a time-varying dynamic field. To validate RCDN, a new dataset called OPV2V-N was created. The paper demonstrates that RCDN improves the robustness of baseline methods in extreme camera-insensitivity settings.", "questions": "*Dataset Diversity*: Have you tested RCDN on any datasets other than OPV2V-N? How does it perform on real-world data?\n\n*Real-Time Feasibility*: What are the computational requirements of RCDN, and how feasible is it for real-time applications in autonomous systems?\n\n*Scalability*: How well does the method scale with an increasing number of agents and cameras? Are there any performance bottlenecks?", "limitations": "Please see weakness and questions.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "*Innovative Problem Addressing*: The paper tackles a significant real-world problem of camera insensitivity in multi-agent collaborative perception, which is crucial for autonomous systems.\n\n*Novel Methodology*: The introduction of dynamic feature-based 3D neural modeling and the construction of collaborative neural rendering field representations are innovative approaches.\n\n*Comprehensive Dataset*: The creation of the OPV2V-N dataset, which includes various camera failure scenarios, provides a robust platform for testing and validating the proposed method.\n\n*Performance Improvement*: The extensive experiments and quantitative evaluations show significant improvements in robustness and performance over baseline methods.\n\n*Detailed Evaluation*: The paper includes both quantitative and qualitative evaluations, along with ablation studies, which thoroughly demonstrate the effectiveness of RCDN.", "weaknesses": "*Complexity and Computation*: The proposed method involves complex modeling and multiple steps. The author should provide the latency.\n\nGeneralizability: The performance of RCDN is primarily validated on the OPV2V-N dataset, which may limit the generalizability of the results to other datasets or real-world scenarios.\n\n\n*Failure Cases*: It would be nice if the authors provide failure cases, which is important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time in reviewing our work and your feedback on the paper's value and clarity! Please note our top-level comment with additional experimental and theoretical results. Below we address specific questions.\n\n>\"Q.1: Have you tested RCDN on any datasets other than OPV2V-N? How does it perform on real-world data?\"\n\n**A.1** This point was also raised by other reviewers, and we have addressed it in our general response (above). Please refer to our top-level comment and the attached PDF for detailed information. In summary, i) we validated our approach on the newly collected V2XSet-N-mini dataset, demonstrating that the proposed RCDN stabilizes performance under noisy camera conditions; ii) we also deployed the OPV2V-N pre-trained RCDN module for direct inference on the V2X-Sim 2.0 dataset, where RCDN continued to effectively stabilize perception results; iii) we will keep track of real-world dataset development, implementing the RCDN in real-world settings when the camera settings meet our requirements.\n\n\n>\"W.1: The performance of RCDN is primarily validated on the OPV2V-N dataset, which may limit the generalizability of the results to other datasets or real-world scenarios.\"\n\n**A.1** Since the proposed RCDN operates as a post-processing step on top of BEV features, its direct applicability to other datasets or real-world scenarios depends on the domain gap between the BEV feature space learned by OPV2V-N and those of other datasets or the real world. i) for other datasets, we performed open-set validation by directly applying the OPV2V-N pre-trained RCDN module to the V2X-Sim 2.0 dataset. The corresponding results, shown in Figure 1 of the attached PDF, indicate that RCDN effectively stabilizes perception results. This is expected since both V2X-Sim 2.0 and OPV2V-N are recorded using the CARLA simulator, resulting in a minimal domain gap. ii) for real-world scenarios, the recently published DUSA (ACM-MM 2024) identified a significant domain gap between simulated and real-world environments. DUSA proposed an unsupervised BEV feature adaptation module to bridge this gap. Therefore, in theory, the proposed RCDN could leverage DUSA to mitigate the BEV feature space gap between OPV2V-N and real-world datasets.\n\n>\"Q.2: What are the computational requirements of RCDN, and how feasible is it for real-time applications in autonomous systems?\"\n\n**A.2** We provide the corresponding latency times in the supplementary materials: the proposed static module takes approximately 4.47 ms, the dynamic module takes about 3.94 ms, and each rendering process takes about 21 ms. The training time ranges from 20 to 30 minutes. For more details, please refer to lines L556-583. Currently, RCDN requires further code optimization to meet real-time application requirements.\n\n>\"W.3: It would be nice if the authors provide failure cases\"\n\n**A.3** Based on the assumption of the multi-view setting, the multi-view RCDN may fail when agents are at the edge of the communication range, resulting in minimal view overlap. This limitation motivated our effort to collect the OPV2V-N dataset for our experiments.\n\n>\"Q.4: How well does the method scale with an increasing number of agents and cameras? Are there any performance bottlenecks?\"\n\n**A.4** Regarding the increasing number of agents and cameras, we validated the impact of adding more cameras using the OPV2V-N dataset (corresponding scenario types are T section and midblock respectively) with the CoBEVT baseline. From Table 2 in the attached PDF, we observe the following: i) With a single overlapping camera view, the proposed method significantly improves baseline performance, and ii) While theoretically, more cameras can provide a larger overlap range, the addition of multiple cameras (depending on their positions) may introduce redundant viewing angles, resulting in less significant performance improvements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bAIV9dxvmP", "reviewer": "Reviewer_y98a", "summary": "The paper proposed Bird Eye View (BEV) semantic segmentation pipeline from collaborative perception, robust to motion blur, sensor noise, occlusion and even failure. The proposed a pipeline that adapts neural rendering techniques to overcome the noise/malfunction in camera capture and occlusion. With the proposed method combined with prior methods, performances on OPV2V-N (the proposed BEV semantic segmentation dataset) are improved.", "questions": ".", "limitations": ".", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper proposed to apply neural rendering concept for ‘robust’ collaborative-perception BEV segmentation. It is natural way of thinking to overcome noise/malfunction in the caption system but the way the paper adapts neural rendering to BEV segmentation is novel. And, the performance is verified with OPV2V-N dataset.", "weaknesses": "Evaluation is only performed with OPV2V-N dataset which may result in overfitting. More evaluation with different dataset is required. The author may need to compare methodologies on other dataset although the existing dataset do not have noise. The author also may add random noise to the prior dataset and run experiments.\n\nThe manuscript was uneasy to read and understand. The paper should re-written. The comments below are without understanding supplemental materials fully.\n- The way proposed algorithm is combined with prior method is unclear. The reviewer guessed that the MCP module can be replaced with prior methods, but it is not stated.\n- Many abbreviations are not explained sufficiently and terminologies the author defined are ambiguous and may be incorrect. \n- MCP is short for the multi-agents collaborative perception process but the paper did not explain MCP module in details with no reference\n- BEV, no full name, no reference.\n- “Camera-insensitivity” can be understood terminologies related to camera sensor sensitivity (how much the camera sensor accept photon…).\n- Robust Camera-Insensitivity: Robust == Camera-sensitivity? The latter one may be redundant\n- Line 6. introduce a new robust camera-insensitivity problem: cam be replaced “introduce BEV segmentation when the camera capture are unreliable (or noisy)?” Should be more concrete without ambiguous words\n- Line19 “Ported to” mean?\n- There are more unclear sentences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and thoughtful comments. Please note our top-level comment with additional experimental and theoretical results. Below we address specific questions.\n\n>\"W.1: Evaluation is only performed with OPV2V-N dataset which may result in overfitting. More evaluation with different dataset is required. \"\n\n**A.1** This point was also raised by other reviewers, and we have addressed it in the general response (above). Please refer to our top-level comment and the attached PDF for detailed information. In summary, i) validated our approach on a newly collected V2XSet-N-mini dataset, demonstrating that the proposed RCDN stabilizes performance under noisy camera conditions.; ii) utilized the OPV2V-N pre-trained RCDN module for direct inference on the V2X-Sim 2.0 dataset, showing that RCDN effectively stabilizes the perception results.\n\n>\"W.2: The way proposed algorithm is combined with prior method is unclear. The reviewer guessed that\nthe MCP module can be replaced with prior methods, but it is not stated\"\n\n**A.2** The MCP module stands for the Multi-agent Collaborative Perception module. Existing state-of-the-art (SoTA) MCP modules share a common pipeline: an encoder-fusion-decoder architecture. To ensure fairness in collaborative perception experiments, different MCP modules use the same encoder-decoder architecture but differ in the fusion process. The fusion process is responsible for the bird-eye view (BEV) feature aggregation. Therefore, the MCP module can be replaced by simply switching between different BEV feature aggregation processes.\n\n>\"W.3: Many abbreviations are not explained sufficiently and terminologies the author defined are\nambiguous and may be incorrect. MCP is short for the multi-agents collaborative perception process but the paper did not explain MCP module in details with no reference\"\n\n**A.3** Thank you for pointing out the misuse of abbreviations. We will ensure to double-check and eliminate any ambiguous abbreviations. For the multi-agent collaborative perception (MCP) module, we employ mainstream collaborative methods based on intermediate BEV features for our experiments. The general setting of the MCP module is described in L570-578 supplements. To avoid any ambiguity, we will include additional technical details about the different MCP baseline methods, such as their respective pipelines and feature fusion processes, in the final supplements.\n\n\n>\"W.4: BEV, no full name, no reference\"\\\"There are more unclear sentences.\"\n\n**A.4** We will reintroduce the concept of bird’s-eye view (BEV) and include relevant references, such as BEVFormer (ECCV 2022). Additionally, we will address the misuse of the terms s\\_fw and s\\_bw, which refer to scene flow for forward and backward directions, respectively. We will also revise the paper to clarify any unclear sentences.\n\n>\"W.5: “Camera-insensitivity” can be understood terminologies related to camera sensor sensitivity (how much the camera sensor accept photon…).\n\"\\\"Robust Camera-Insensitivity: Robust == Camera-sensitivity? The latter one may be redundant\nLine 6. introduce a new robust camera-insensitivity problem: cam be replaced “introduce BEV segmentation when the camera capture are unreliable (or noisy)?” Should be more concrete without ambiguous words\"\n\n**A.5** Thank you for your careful and valuable feedback. Our intent in using the term “robust” was to emphasize the concept, but we will revise the description to avoid redundancy based on your suggestions. We will replace the phrase “introduce a new robust camera-insensitivity problem” with “introduce BEV segmentation when the camera capture is unreliable (or noisy).”\n\n>\"W.6: Line19 “Ported to” mean?\"\n\n**A.6** Apologies for any confusion regarding the term \"Ported to\". The \"Ported to\" means that it is fairly easy to apply to different existing feature backbones, as it is the post-processing step built on top of BEV features.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5j8eWuHRGU", "reviewer": "Reviewer_YPsN", "summary": "The paper presents RCDN, a method to aggregate multi-sensor perception signals in dynamic environment.\nThe key idea, is to improve the aggregated multi-agent feature with the multi-view rendering loss.\nAt its core, RCDN gathers input streams at varying timesteps of multiple agents. The gathered images are fused into Birds Eye view (BEV) then further decoded into volume. \nThe volumetric features are learned into static scene and dynamic scene components with NGP based representation.\nOverall procedure is supervised with rendering loss, (cyclic) optical flow consistency.\n\n\nThe method is evaluated on new dataset, OPV2V-N, which is an updated version of OPV2V, with additional masking and optical flow. \nThe results show that RCDN helps BEV segmentation with various backbones, compared to the model used without RCDN.", "questions": "These are few questions that I would like the authors to answer in the rebuttal.\n1. How real is the synthetic OPV2V-N dataset? In other words, how can features learned in OPV2V-N dataset be translated to real-world usage? Moreover, are there any real-world quantitative results on model trained on synthetic data?\n\n2. Have authors evaluated the method on different down streaming task other than segmentation? How does one verify that the volumetric features are geometrically correct? (how accurate is the Geometric BEV features?)\n\n3. How is BEV segmentation evaluation differ on non-flat surfaces like hills or bridges?", "limitations": "No concerning limitations are found.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 1, "strengths": "The main benefit of the RCDN, is that it is fairly easy to apply into different existing feature backbones, as it is the post-processing step built on top of BEV features. \nExperimentally, the usage of RCDN significantly improves the segmentations which implies that the features are better aligned throughout the noisy signals. \nThis makes the work to be a great off-line data augmentation / preparation pipeline for generating BEV segmentation features. \nThe paper additionally proposes OPV2V-N dataset, which may be somewhat valuable addition to the community.\n\nAside from technical perspective, the paper is easy to follow and well-written.", "weaknesses": "The paper's main weaknesses are two folds. \n1. The paper does not evaluate on tasks other than BEV segmentation. \nWhile I believe that the pixel-aligned features from NGP would give benefits over various vision tasks, the paper only demonstrates on smaller domain of work which undermines its actual potential. It would have been more interesting to compare how it impacts in different downstreaming tasks, such as detection / tracking.\n\n2. Technical contribution seems to lack novelty. \nThe paper is a mix of two known-to-work solutions; BEV feature decoding for segmentation (used with various baselines in the experiments), and NGP (or radiance field based) multi-view pixel / density alignment through rendering loss. Usage of rendering loss to improve segmentation map is well-investigated in different literatures in the NeRF community (e.g, semantic-nerf).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and valuable comments. Please note our top-level comment with additional experimental and theoretical results. Below we address specific questions.\n\n>\"W.1: Have authors evaluated the method on different down tasks other than segmentation?\"\n\n**W.1** Our proposed RCDN is general to different downstream tasks and is not limited to just BEV segmentation. We focus on BEV segmentation due to its crucial role in autonomous driving, with direct applications to other tasks such as layout mapping, action prediction, route planning, and collision avoidance. Additionally, we have validated RCDN for detection tasks.  In our experiments, we replaced the original segmentation header with a detection header (see Figure 2 and Table 3 in the attached PDF). Table 3 shows that for CoBEVT, using RCDN improves the metrics of AP@0.50 and AP@0.70 by 19.05\\% and 24.99\\%, respectively.\n\n>\"W.2: Technical contribution\"\n\n**A.2** Robust perception is a significant challenge in single-agent systems; however, few studies have addressed this issue in the context of collaborative perception. Instead of adding additional modal sensors, such as LiDAR, as done in single-agent systems, our approach leverages the unique multi-view properties to mitigate the impact of noisy views. Notably, NeRF (Neural Radiance Fields) achieves photorealistic rendering by optimizing 2D multi-view images. To the best of our knowledge, we are the first to apply NeRF to the field of collaborative perception to handle robust settings. Meanwhile, to make NeRF a better adaption with the collaborative perception setting, we propose the geometry BEV features, which can improve the PSNR by about 9.30\\%  (see more details in Appendix.B.6) and avoid NeRF network memorization. Additionally, we optimize the collaborative neural field by decoupling it into static and dynamic parts for better performance, which improves the segmentation performance of dynamic vehicles by 12.27\\% compared to the non-decouple approach.\n\n>\"Q.1: How real is the synthetic OPV2V-N dataset? In other words, how can features learned in OPV2V-N dataset be translated to real-world usage?\"\n\n**A.1** The OPV2V-N dataset is recorded by the co-simulation with SUMO (traffic manager) under the realistic platform CARLA simulator. As for the \"How can features learned in OPV2V-N dataset be translated to real-world usage?\", this pertains to the broader issue of sim-to-real transfer. Whether the RCDN pre-trained on OPV2V-N can be effectively applied to other datasets or real-world scenarios depends on the domain gap between the BEV (Bird’s Eye View) feature space learned from OPV2V-N and that of other datasets or real-world environments. Recent research, such as the DUSA approach (ACM-MM 2024), addresses sim-to-real adaptation for collaborative perception. DUSA proposes a unified unsupervised BEV feature adaptation module. Since our RCDN operates as a post-processing step built upon BEV features, it is theoretically possible for RCDN to leverage DUSA to bridge the BEV feature space gap between OPV2V-N and real-world scenarios.\n\n>\"Q.2: Moreover, are there any real-world quantitative results on model trained on synthetic data?\"\n\n**A.2** Please refer to our top-level comment and the attached PDF for further details. In summary, we can employ sim2real BEV feature space adaptation techniques, such as DUSA, to validate models trained on synthetic data in real-world scenarios. However, the current real-world collaborative perception datasets can not meet the RCDN's camera setting demands, and we will keep track of real-world dataset development, implementing the RCDN in real-world settings if the camera setting meets the demands.\n\n>\"Q.3: How does one verify that the volumetric features are geometrically correct? (how accurate is the Geometric BEV features?)\"\n\n**A.3** Our proposed module only inputs RGB images without introducing additional features such as depth images. Specifically, we supervise the geometry of volumetric features by using multi-view consistency and corresponding downstream tasks. From the experimental results, the visualization results (please kindly refer to Figure 5 in the manuscript Page 9.) demonstrate the high multi-view consistency of the rendered results. The generated BEV features successfully improve the performance of perception tasks in 3D space (i.e., BEV segmentation/detection), which further indicates the geometrical correctness of our BEV feature.\n\n>\"Q.4: How is BEV segmentation evaluation differ on non-flat surfaces like hills or bridges?\"\n\n**A.4** Thank you for your questions. Currently, the OPV2V-N segmentation labels are available only in 2D image format. As a result, non-flat surfaces such as hills or bridges are projected onto these 2D segmentation images. Consequently, from the perspective of static part segmentation, our BEV decoder classifies static elements into two categories: road and line.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iiMlZ8w2Q7", "reviewer": "Reviewer_ymZH", "summary": "The paper introduces a new problem: how to overcome the issues caused by the failed camera perspectives, while stabilizing high collaborative performance with low calibration cost? Therefore, RCDN, a Robust Camera-insensitivity collaborative perception with a novel Dynamic feature-based 3D Neural modeling mechanism is introduced. To validate the new method, the authors also provide a new dataset: OPV2V-N. RCDN serves as baseline here. Ablation Study shows for 5 models (F-cooper, Att-Fuse, Disco-Net, V2VNet, CoBEVT a significant improvement over their baselines, w/o RCDN.", "questions": "What is difference between sf w , sbw in equation (7)?\nHow many message exchange tasks could be used overall (Figure 2.)  Will baseline code be publishe in combination with the dataset? When?", "limitations": "The most relevant limitation is the missibg real-time applicability.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper builds up on three pillars: single perception, collaborative perception and neural rendering. The base idea is novel to the best of my knowledge. The problem formulation is clear and well sounded, easy to follow. The System architecture is strong. The authors also focus on the differentiation between static and dynamic scenarios, especially for the neural fields both based on the BEV volume feature space. This differention is very important, not very often in detail discussed. The ablation study especially table 5.1 shows very accurate an increase of performance for different tasks static (lanes, free space) and dynamic perception. The experimentsl part introduces a new dataset, which is necessray for the investigation.", "weaknesses": "The overall system architecture sounds good. However, there are some open points for me, the impact of section 4.3 and 4.4, i.e. the neural fields part, seems open in terms of clarification. Example: What is difference between sf w , sbw in equation (7)?\nThe experimental section is a bit too short. I feel its not finished yet. However, there is limited space. The overall approach is not usable for realtime.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We have carefully addressed all the questions raised. Please find our responses below.\n\n>\"W.1: What is difference between sfw, sbw in equation (7)?\"\n\n**A.1** Apologies for any confusion regarding the terms s\\_fw and s\\_bw. The term s\\_fw stands for forward scene flow, while s\\_bw refers to backward scene flow. Specifically, the forward scene flow (s\\_fw) estimates the flow from time t  to  t+1, whereas the backward scene flow (s\\_bw) estimates the flow from time t  to  t-1. By leveraging both s\\_fw and s\\_bw, we can achieve a more consistent representation of scene movements.\n\n>\"W.2: The experimental section is a bit too short. I feel its not finished yet. However, there is limited space.\"\n\n**A.2** Thanks for your understanding. Due to the limited space available, we aimed to present the key experimental results and analyses as concisely as possible. However, we greatly appreciate your suggestion and agree that providing additional details could enhance the clarity and comprehensiveness of our work. In response to your comment, we will make the following improvements to the experimental section in the final version: i) include a more detailed description of the experimental setup and procedures ii) add additional experimental results and figures (such as detection results) to better support our conclusions.\n\n>\"Q.1: How many message exchange tasks could be used overall (Figure 2.)\"\n\n**A.1** The proposed RCDN does not add extra message exchange times, and as for communication cost, similar to DiscoNet (NIPS 2021), we only utilize the RGB labels during the training stage, meaning we leave the communication burden to the training stage and do not introduce extra information during the inference.\n\n>\"Q.2: Will baseline code be published in combination with the dataset? When?\"\n\n**A.2** We will publish the corresponding codes and dataset as soon as we get accepted.\n\n>\"L.1: The most relevant limitation is the missibg real-time applicability.\"\n\n**A.1** We provide the corresponding latency times in the supplementary materials: the proposed static module takes approximately 4.47 ms, the dynamic module takes about 3.94 ms, and each rendering process takes about 21 ms. The training time ranges from 20 to 30 minutes. For more details, please refer to lines L556-583. Currently, RCDN requires further code optimization to meet real-time application requirements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "77fB79DyUY", "reviewer": "Reviewer_fQwk", "summary": "In this paper, the authors proposed an essential problems: how to overcome the issues caused by the failed camera perspectives, while stabilizing high collaborative performance with low calibration cost? The authors presented a robust camera-insensitivity collaborative perception with a novel dynamic feature-based 3d neural modeling mechanism to address the issue. Moreover, to verify the effectiveness of the model, the authors also provided a new large-scale dataset, OPV2V-N for this field. The experiments result showcase the model’s robustness in proposed dataset.", "questions": "see weakness part", "limitations": "The current work focuses on addressing the camera-insensitivity problem in collaborative perception. It is evident that accurate reconstruction can compensate for the negative impact of noisy camera features on collaborative perception.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Strength:\n1.\tThe paper presents an interesting viewpoint that is to recover noisy camera perceptual information from other agents’ views by modeling the collaborative neural rendering field representation, in which the model is divided into two stages: a time-invariant static background and time-varying dynamic foreground.s\n2.\tThe paper develops a new dataset to fill the gap of the lack of a comprehensive collaborative perception dataset that accounts for different camera noise scenarios.\n3.\tThe paper is well-organized and interesting to read.", "weaknesses": "1.\tFrom my perspective, the paper lacks the theory analysis for the proposed method. Moreover, the authors fail to introduce the motivation of each sub-module in the presented model. For example, can the authors showcase the motivation of using Nerf for the static and dynamic fields, are there any dominant advantages of nerf, compared to other 3d reconstruction methods in this method?\n2.\tIt is necessary to give more rigorous mathematic analysis of equations in this paper. Furthermore, the authors are required to introduce the details of each networks, including the training parameters, learning rate, weight values in eq. 12.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our work. We will take your suggestions into account when preparing the final version. Below please find responses to specific questions.\n\n>\"W.1: Can the authors showcase the motivation of using Nerf for the static and dynamic fields, are there any dominant advantages of nerf, compared to other 3d reconstruction methods in this method?\"\n\n**A.1** The proposed geometry BEV feature avoids the issue of per-scene “network memorization” inherent in NeRF by employing generic feature representations. The decoupling of dynamic and static neural rendering adapts to real-world collaborative autonomous scenarios. We selected NeRF for its photorealistic rendering capabilities. However, research such as Mip-NeRF 360 (CVPR 2022) and Zip-NeRF (ICCV 2023) demonstrates that other reconstruction methods, such as Structure from Motion (SfM) and Multi-View Stereo (MVS), are less effective at novel view synthesis compared to NeRF-based methods. Meanwhile, to make NeRF a better adaption to the collaborative perception setting, we optimize the collaborative neural field by decoupling it into static and dynamic parts for better performance, which improves the segmentation performance of dynamic vehicles by 12.27\\% compared to the non-decoupled one.\n\n>\"W.2: It is necessary to give more rigorous mathematic analysis of equations in this paper. The authors are required to introduce the details of each networks, including the training parameters, learning rate, weight values in eq. 12.\"\n\n**A.2** Thank you for pointing this out. The static and dynamic neural components of our model follow the architecture of Instant-NGP (SIGGRAPH 2022). The architecture for the collaborative perception component is detailed on L566-583 of the supplementary material.  The initial learning rate is 5e-4 with the exponential learning rate decay strategy. The weight values in eq. 12 is set to 1.0, 1.0, 0.1, and 1.0, respectively. Further details about the loss function are provided on L632-636 of the supplementary material. We will also include these parameters in the main text for improved clarity.\n\n>\"L.1: The current work focuses on addressing the camera-insensitivity problem in collaborative perception. It is evident that accurate reconstruction can compensate for the negative impact of noisy camera features on collaborative perception.\"\n\n**A.1** Thanks for your deep understanding of our proposed RCDN. To the best of our knowledge, We are the first to introduce the NeRF into the collaborative perception field to handle robust settings. Our proposed RCDN has shown significant improvements in our experiments, particularly in scenarios under multi-source noises.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Please see the attached PDF for a one-page PDF with a summary of added experimental results.**

We thank all reviewers for their constructive comments on our work. We found one comment that was common amongst more than one reviewer, hence we highlight it here.

>"Have you tested RCDN on any datasets other than OPV2V-N?[qUQv]"\
>"The author may need to compare methodologies on other dataset.[y98a]"\
>"are there any real-world quantitative results on model trained on synthetic data?[YPsN]"

**A1.1** **For other datasets**, to adapt the multi-view based robust collaborative perception setting, we spent considerable time manually recording the multi-view overlaps existing start time, end time, and corresponding car IDs, and generated corresponding masks and flows (more details are shown in Appendix A). Due to time constraints, we tried our best to convert a single scene from the V2XSet dataset (ECCV 2022) into the V2XSet-N-mini dataset for our experiments. As shown in Table 1 of the attached PDF, the RCDN significantly improves the average performance of the drivable area, lane, and dynamic vehicle detection by 74.40\%, 110.69\%, and 201.74\%, respectively, compared to baseline methods without RCDN. Additionally, we conducted open-set inference on the V2X-Sim 2.0 dataset (RA-L 2022), deploying the pre-trained RCDN model from OPV2V-N directly onto the V2X-Sim 2.0 dataset. Figure 1 in the attached PDF demonstrates that RCDN effectively stabilizes the perception results.


**A1.2** **For the real-world datasets**, the existing open-source real-world collaborative perception datasets are DAIR-V2X (CVPR 2022) and V2V4Real (CVPR 2023). i) V2V4Real focuses on the two connected vehicle-to-vehicle collaboration and currently only provides LiDAR data without camera modality, targeting the downstream of 3D detection. As for our proposed RCDN, we do not introduce lidar modality just like single-agent perception to deal with the robust perception setting, we want to focus on utilizing the special property of collaborative perception: multi-view based. Hence, V2V4Real cannot be utilized for our purposes. ii) DAIR-V2X focuses on the collaboration between one road-side infrastructure (equipped with one camera and one lidar) with one connected vehicle (equipped with one camera and one lidar). During our experiments, we found that the infrastructure cameras are positioned much higher than those on the vehicle, resulting in fewer overlapping views compared to vehicle-to-vehicle collaboration. Due to these limitations, we are also unable to utilize DAIR-V2X. Theoretically, the proposed RCDN compensates for the negative impact of noisy cameras through multi-view accurate reconstruction. The distinction between synthetic and real datasets primarily affects the distribution of RGB images, influencing later BEV feature space but not conflicting with the fundamental principles of RCDN. Therefore, the effectiveness of RCDN remains unchanged whether using synthetic or real datasets. RCDN has been thoroughly validated in synthetic OPV2V-N experiments, demonstrating stable performance under noisy camera conditions. However, we will keep track of real-world dataset development, implementing the RCDN in real-world settings when the camera settings meet our requirements.

Meta-review: The paper received 3 Borderline Accept and 1 Weak Accept. The reviewers agree that the solution and the problem is interesting. However, reviewers think that the evaluation is weak, but enough to show the benefits of the proposed approach. After reading the paper, the reviews, and discussion, I also think the problem and solution is interesting and can cause excitement in the community. Thus, I recommend acceptance. Nevertheless, I encourage the authors to revise the paper given the feedback the reviewers gave.
Decision: Accept (poster)