RCDN: Towards Robust Camera-Insensitivity Collaborative Perception via Dynamic Feature-based 3D Neural Modeling

Authors: tianhang  wang
fan  lu
zehan  zheng
guang  chen
changjun  jiang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 13:19:08

Abstract: Collaborative perception is dedicated to tackling the constraints of single-agent perception, such as occlusions, based on the multiple agents' multi-view sensor inputs. However, most existing works assume an ideal condition that all agents' multi-view cameras are continuously available. In reality, cameras may be highly noisy, obscured or even failed during the collaboration. In this work, we introduce a new robust camera-insensitivity problem: how to overcome the issues caused by the failed camera perspectives, while stabilizing high collaborative performance with low calibration cost? To address above problems, we propose RCDN, a Robust Camera-insensitivity collaborative perception with a novel Dynamic feature-based 3D Neural modeling mechanism. The key intuition of RCDN is to construct collaborative neural rendering field representations to recover failed perceptual messages sent by multiple agents. To better model collaborative neural rendering field, RCDN first establishes a geometry BEV feature based time-invariant static field with other agents via fast hash grid modeling. Based on the static background field, the proposed time-varying dynamic field can model corresponding motion vector for foregrounds with appropriate positions. To validate RCDN, we create OPV2V-N, a new large-scale dataset with manual labelling under different camera failed scenarios. Extensive experiments conducted on OPV2V-N show that RCDN can be ported to other baselines and improve their robustness in extreme camera-insensitivity setting. Our code and datasets will be available soon.

Main paper: 
 introduction: 
 Multi-agent collaborative perception [1] [2] [3] [4] [5] obtains better and more holistic perception by allowing multiple agents to exchange complementary perceptual information. This field has the potential to effectively address various persistent challenges in single-perception, such as occlusion [6, 7] . The associated techniques and systems also process significant promise in various domains, such as the utilization of multiple unmanned aerial aircraft for search and rescue operations [8] [9] [10] , the automation and mapping of multiple robots [11] [12] [13] . As an emerging field, the research of collaborative perception faces several issues that need to be addressed. These challenges include the need for high-quality datasets [14] [15] [16] [17] , the formulation of models that are agnostic to specific tasks and models [18, 19] , and the ability to handle pose error and adversarial attacks [20, 21] . Repaired Views However, a vast majority of existing works do not seriously account for the harsh realities [22, 23] of real-world sensors in the collaboration, such as blurred, high noise, interruption and even failure. These factors directly undermine the basic collaboration premise [24, 25] of reconstructing the holistic view based on the multiview sensors that severely impact the reliability and quality of collaborative perception process. This raises a critical inquiry: how to overcome the issues caused by the failed cameras' perspectives while stabilizing high collaborative performance with low calibration cost? The designation camera insensitivity overcomes the unpredictable essence of the specific failure camera numbers and time; see Figure 1 for an illustration. To address this issue, one viable solution is adversarial defense [26] . By robust defense strategy, adversarial defense bypasses camera insensitivity among blurred and noise. However, its performance is suboptimal [27] and has been shown to be particularly vulnerable to noise ratios [20] and failed camera numbers.To address this robust camera insensitivity collaborative perception problem, we propose RCDN, a Robust Camera-insensitivity collaborative perception with a Dynamic feature-based 3D Neural modeling mechanism. The core idea is to recover noisy camera perceptual information from other agents' views by modeling the collaborative neural rendering field representations. Specifically, RCDN has two collaborative field phases: a time-invariant static background field and time-varying dynamic foreground field. In the static phases, RCDN sets other baselines' backbone as the collaboration base and undertakes end-to-end training to create a robust unified geometry Bird-eye view (BEV [28, 29] ) feature space for all agents. Then, the geometry BEV feature combines the hash grid modeling, an explicit and multi-resolution network, to generate static background views through α-composed accumulation of RGB values along a ray at a fast speed. In the dynamic phase, RCDN utilizes 4D spatiotemporal position features to model the dynamic motion of 3D points, which learns an accurate motion field under optical priors and spatiotemporal regularization. The proposed RCDN has two major advantages: i) RCDN can handle camera insensitivity collaboration under unknown noisy timestamps and numbers; ii) RCDN does not put any extra communication burden into inference stage and costs little computation burden.In our efforts to validate the effectiveness of RCDN, we identified a gap: the lack of a comprehensive collaborative perception dataset that accounts for different camera noise scenarios. To address this, we create the OPV2V-N, an expansive new dataset derived from OPV2V, featuring meticulously labeled timestamps and camera IDs. This advancement aims to support and enhance research in camerainsensitive collaborative perception. Extensive experiments on OPV2V-N show RCDN's remarkable performance when other baselines equipped with RCDN under extreme camera-insensitivity setting, improving w.o. RCDN baseline methods by about 157.91%. 
related works: 
 Robust Single Perception. Single-agent perceptions [30, 31, 27, [32] [33] [34] have tackled the robust camera setting with other sensor modals. [27] reveals that camera-based methods [34] can be easily effected by camera working conditions. Some works [32, 31] introduce LiDAR into perception system and design a soft-association mechanism between the LiDAR and the inferior camera-side, to relieve the negative impacts caused by cameras. MVX-Net [33] improves the combination pipeline of LiDAR and cameras by leveraging the VoxelNet [35] architecture. CRN [30] introduces the low-cost Radar to replace the LiDAR, which can provide precise long-range measurement and operates reliably in all environments. However, as for the camera-only situation, few work seeks to solve this because recovering just from the single-view is highly ill-posed (with infinitely many solutions that match the input image). With the recent rapid development of V2X [36] , we now can introduce the multi-agent and multi-view based collaborative perception setting to explore this extreme situation.Collaborative Perception. Perception tasks for single agents can be adversely affected by factors such as limited sensor fields of view and physical ambient occlusions. To address the aforementioned challenges, collaborative perception [37] [38] [39] can attain more comprehensive perceptual output by exchanging perception data. Early techniques involved the transmission of either unprocessed sensory input (referred to as early fusion) or the results of perception (referred to as late fusion). Nevertheless, recent research has been examining the transfer of intermediate features to achieve a balance between performance and bandwidth. Some works [40] [41] [42] [43] devote selecting the most informative messages to communicate. DiscoNet [44] utilizes knowledge distillation to achieve a better trade-off between performance and bandwidth. V2X-ViT [45] presents a unified V2X framework based on Transformer that takes into account the heterogeneity of V2X system. Meanwhile, some learnable or mathematical based methods [46] [47] [48] [49] have also been proposed to correct the pose errors and latency. Moreover, some works [50, 51] reveal that the holistic character of collaborative perception can improve the effect of driving planning and control tasks. However, most existing papers do not take the harsh realities of real-world sensors into account, such as blurred, high noise, occlusion and even failure, which directly undermine the basic collaboration premise of multi-view based modeling, negatively impacting performance. This work formulates camera-insensitivity collaborative perception, which considers real-world camera sensor conditions.Neural Rendering. Neural radiance fields [52] aim to utilize implicit neural representations to encode densities and colors of the scene. This approach takes advantage of volumetric rendering to synthesize views, and it can be effectively optimized from 2D multi-view images. Hence, numerous works have enhanced NeRF in terms of rendering quality [53] [54] [55] , efficiency [56] [57] [58] [59] , etc. For example, Mip-NeRF [60] utilizes cone tracing instead of ray tracing in standard NeRF volume rendering by introducing integrated positional encoding, which greatly improves the render quality. To improve the efficiency of training and inference processes, Instant-NGP [61] proposes a learned parametric multi-resolution hash for efficient encoding, which also leads to high compactness. Some works have also extended NeRF to large-scale urban autonomous scenes [62] [63] [64] . In this work, we first introduce neural rendering to collaborative perception. The proposed collaborative neural rendering field representations will address the problem of recovering highly noisy perceptual messages.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: This paper does discuss the limitations of the work performed by the authors. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem formulation: 
 Consider N agents in a scene, where each agent can send and receive collaboration messages from other agents. For the n-th agent, let X ti n = {I ti c } cn c=1 and Y ti n be the raw observation and the perception ground-truth at time current t i , respectively, where I ti c is the c-th camera images recorded at i-th timestamp, and P ti m→n is the collaboration message sent from the agent m at time t i . The key of the camera insensitivity is that the specific noisy camera number and corresponding timestamp are unpredictable. Therefore, each agent has to encounter invalid view information, which contains both local observation and collaboration messages sent from other agents. Then, the task of camera insensitivity collaborative perception is formulated as:EQUATION)subject to Y ti n = c θ2 (π θ1 (ψ(X ti n , {P ti m→n } N -1 m=1 ))),where g(•, •) is the perception evaluation metrics, Y ti n is the perception result of the n-th agent at time t i , ψ(•, •) is the camera noise function to simulate the harsh realities of the real-world situation, π θ1 (•) is the proposed collaborative neural rendering field network RCDN with trainable parameters θ 1 , and c θ2 is the existing collaborative perception network with trainable parameters θ 2 . Note that the proposed RCDN is to recover the noisy camera views caused by the ψ function, making collaborative perception system more robust to the unpredictable situation of noisy camera data.Given such high noisy camera view, the performances of collaborative perception system would be significantly degraded since the mainstream collaborative perception utilizes the multi-view camerabased BEV features for communication and downstream tasks, and using such damaged features The collaborative static and dynamic fields are performed in parallel to model the background and foreground, respectively. Note that MCP is short for the multi-agents collaborative perception process.would contain erroneous information during the perception process. In the next section, we will introduce RCDN to address this issue.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [Yes] Justification:This paper provides open access to the data and code.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
rcdn: 
 This section proposes a robust camera-insensitivity collaborative perception system, RCDN. Figure 2 overviews the framework of the RCDN module in Sec.4.1. The details of three key modules of RCDN can be found in Sec.4.2-4.4.The problem of noisy camera view results in the sub-optimization of the holistic multi-view based BEV features generation in the collaboration messages. That is, the collaboration messages from both self and other agents would be noisy or damaged for the fusion process. The proposed RCDN addresses this issue with two key notions: i) we construct novel collaborative neural rendering field representations, enabling collaborative perception to recover from the noisy camera view; and ii) we establish time-invariant and time-varying fields for background and foreground, respectively, making the collaborative neural rendering field more accurate.Mathematically, let the n-th agent be the ego agent and X ti n be its raw observation at the t i timestamp of agent n. The proposed camera-insensitivity collaborative perception system RCDN is formulated as follows:EQUATIONEQUATIONEQUATIONEQUATIONEQUATIONEQUATIONwhere F ti n ∈ R C×H×W is the BEV feature maps of the n-th agent at timestamp t i with H, W the size of BEV map and C the number of channels; V ti icv ∈ R C×Z×H×W is the implicit collaborative geometry volume feature of the scenarios; which is lifted from BEV plane with the Z height; r(u(k)) is the ray from the failed camera center o ∈ R 2 through a given pixel on the image plane as r(u(k)) = o + u(k)d, where d ∈ R 3 is the normalized viewing direction; f static is a explicit hash grid based representation to model the collaborative static scenarios volume density σ s ∈ R 1 and corresponding color c s ∈ R 3 ; f dynamic is the dynamic collaborative neural network takes the interpolated 4D-tuple (r(u(k)), t i ) and sampled V ti icv feature as input and predict 3D collaborative scene flow vectors s f w , s bw ∈ R 3 , dynamic volume density σ d ti , color c d ti and blending weight b ∈ R 2 ; and X ti n , { X ti j } N -1 j=1 is the recovered noisy camera images at timestamp t i after collaborative rendering; and Y ti n is the final output of the system. In summary, Step 2a extracts BEV perceptual features from observation data. Step 2b generates the collaborative geometry BEV volume feature map for each timestamp, enabling feature sampling in Step 2c and 2d. Step 2d models the static background field of collaboration scenarios. Step 2d models the dynamic foreground field of collaboration objects. Step 2e gets the global volume density and color information by combining both static and dynamic field models to recover the failed camera perspective images. Finally, Step 2f outputs the final perceptual results with repaired images.Step 2a is done locally, Step 2b-2f are performed after receiving the messages from others. The proposed RCDN does not require any extra transmission during the inference process, which is bandwidth friendly; and ii) Step 2c and 2d are performed in parallel to save inference time; and iii) Same as [44, 49] , RCDN adopts the feature representations in bird's eye view (BEV), where the feature maps of all agents are projected to the same global coordinate system. We now elaborate on the details of Steps 2b-2e in the following subsections.Given the BEV feature map of each agent, Step 2b aims to construct a unified collaborative geometry BEV volume feature for each timestamp of the scenario. The intuition is that [65] points out that combing with generic feature representations can avoid the per-scene "network memorization" phenomenon [52] , which will improve the efficiency of the optimization process. Therefore, using the geometry BEV feature can enable the subsequent Step 2c, 2d to learn more generic networks for both static and dynamic collaborative neural fields, respectively.To implement, we use a geometry-aware decoder D geo to transform the BEV feature F ti n into the intermediate featureF ′ ti n ∈ R C×1×X×Y and F ti height,n ∈ R 1×Z×X×Y, and this feature is lifted from BEV plane to an implicit collaborative volume featureEQUATIONwhere • represents dot production along the channel. Eq. 3 lifts the items on the BEV plane into 3D collaborative volume with the estimated height position sigmoid(F ti height,n ). sigmoid(F ti height,n ) represents whether there is an item at the corresponding height. Ideally, the collaborative volume feature V ti icv contains all the scene items information in the corresponding position.After getting the collaborative volume feature V ti icv , Step 2c aims to construct the background of camera views with the static collaborative neural field. Given an arbitrary 3D scenario position x ∈ R 3 and a 2D viewing direction d ∈ R 2 , we aims to estimate static scenarios volume density σ s and emitted RGB color c s using the fast hash grid-based [61] neural network:EQUATIONwhere f = V ti icv (x) is the neural feature trilinearly interpolated from the collaborative geometry BEV volume V ti icv at the location x, G s θ (•, •) is explicit multi-level hash grid representation with the generic f features for fast static collaborative neural field training. Meanwhile, owing to the collaborative scenarios are unbounded, we utilize contract(•) [53] to map 3D scenario position into a bounded ball of radius 2 with regularization, making the estimation optimization process faster and better. Hence, we can compute the color of the pixel (corresponding to the ray r(u k ) using numerical quadrature for approximating the collaborative volume rendering interval [66] :EQUATIONEQUATIONwhere α s (x) = 1 -exp(-x) and δ k = u k+1 -u k is the distance between two quadrature points. The K quadrature points {u k } K k=1 are drawn uniformly between u n and u f , which denotes the near and far of the bounded collaborative scenarios. T s (u k ) indicates the accumulated transmittance from u n to u k . Here, we denote r i as the rays passing through the pixel i. Then, the collaborative static neural loss L static is defined to minimize the l 2 -loss between the estimated colors C s (r i ) and the ground truth colors C gt (r i ) in the static regions (where M(r i ) = 0):L static = i ∥C s (r i ) -C gt (r i ) • (1 -M(r i ))∥ 2 2 (6)While the static collaborative neural field is being modeled, Step 2d is building the dynamic collaborative neural field to construct the foreground of camera views. Our dynamic collaborative neural field takes 4D spatiotemporal position features as input to model dynamic motion of 3D scene flow s f w , s bw , volume density σ d ti , color c d ti and blending weight b (Note that blending weights learns how to blend the results from both the static and dynamic collaborative neural fields in an unsupervised manner, avoiding background's structure and appearance conflict the moving objects.):EQUATIONwhere G d θ shares the same hash grid representations, but for the dynamic collaborative neural field optimization; ∆(•, •) is the temporal interpolation functions, which makes the MLP can efficiently learn the features between keyframes in a scalable manner. Meanwhile, to improve the temporal consistency of the proposed field, we compute the collaborative scene flow neighbors r(u k ) + s f w and r(u k ) -s bw with the predicted collaborative scene flow s f w , s bw to warp the collaborative neural field from the neighboring time instance to the current time. Note that the term s f w stands for forward scene flow, while s bw refers to backward scene flow. Specifically, the forward scene flow (s f w ) estimates the flow from time t to t+1, whereas the backward scene flow (s bw ) estimates the flow from time t to t-1. Hence, we can obtain the corresponding density and color of adjacent time by querying the same MLPs model at r(u k ) + s:EQUATIONWe can compute the color of a dynamic pixel of collaborative view at time t i . Hence, with both the static and dynamic collaborative neural fields model, we can easily compose them into a complete model using the predicted blending weight b and render full color C f ull (r) frames at noisy views and time. We utilize the following approximate of collaborative volume rendering integral:EQUATIONSimilar to the static collaborative rendering loss, we train the dynamic collaborative neural model by minimizing the l 2 reconstruction loss under time unit τ = {t i , t i -1, t i + 1}:EQUATION)To reduce the amount of ambiguity caused by the sparse views during collaborative perception process, we construct motion matching loss to constrain the proposed dynamic collaborative neural field. As we do not have direct 3D supervision for predicted collaborative scene flow from the motion MLP model, we utilize 2D optical flow f as indirect supervision. Specifically, we first use the estimated collaborative scene flow to obtain the corresponding 3D point. Then, we project these 3D points onto the 2D reference frame with φ(•) function. Hence, we can compute the projected collaborative scene optical flow and enforce it to match the estimated optical flow as follows:EQUATIONMeanwhile, we also regularize the consistency of the collaborative scene flow by minimizing the cycle consistency loss L cyc . See more details in the Appendix B.7. To train the overall system, we supervise two tasks: static and dynamic collaborative neural fields, respectively. Meanwhile, during the training process, the static collaborative field and dynamic collaborative field are trained separately. The initial learning rate is 5e-4 with the exponential learning rate decay strategy. The weight values are set to 1.0, 1.0, 0.1, and 1.0, respectively:EQUATION 
experimental results: 
 We create the first camera-insensitivity collaborative perception dataset and conduct extensive experiments on OPV2V-N. To ensure the consistency of the input noisy camera data and verify the effectiveness of RCDN, we set the noisy camera data to be in the failed situation [27] . Meanwhile, the task of the experiments is map segmentation, including the performance of the vehicle, drivable area (Dr. area) and lane, totaling three classes. We utilize the Intersection over Union (IoU) between map prediction and ground truth map-view labels as the performance metric.OPV2V-N. To facilitate research on camera-insensitivity for collaborative perception, we propose a simulation dataset dubbed OPV2V-N. In OPV2V dataset, there is a lack of mask labels for distinguishing between foreground and background views, as well as optical flow labels for supervising the scene flow. For this purpose, we collect more data to bridge the gap between neural field and collaborative perception, leading to the new OPV2V-N datasets. Specifically, we utilize the OneFormer [67] detector to extract the foreground mask labels and mainstream RAFT [68] detector to compute the optical flow between image pairs. Meanwhile, we manually annotate which part of the performance degradation is triggered by camera failure in different scenarios. See more details in the Appendix A.Benchmark comparison. The baseline methods include F-Cooper [1] , AttFuse [16] , DiscoNet [44] , V2VNet [37] and CoBEVT [6] . All methods use the same BEV feature encoder based on CVT [69] .To validate the portability of the RCDN, we compare different baseline methods w.o/w. RCDN under unpredictable camera failure settings. Robust to extremely noisy camera data. We conduct experiments to validate the performance under the impact of random noisy camera numbers. Figure 3 shows the map-view segmentation performance of the different baselines methods w.o/w. the proposed RCDN under varying levels of camera failures situation on OPV2V-N, where the x-axis is the expectation of the number of random failed cameras during the inference stage and y-axis the segmentation performance. Note that, when the x-axis is at 0, it represents standard collaborative perception without any camera failures. We see that i) the proposed RCDN can stabilize all the baseline methods in both static and dynamic part of map-view segmentation performance at all camera failure settings; ii) as for the static part, with the RCDN can maintain the 87.84%/88.72%/86.64% Dr. area performance of the standard setting even under three random failed views during the collaboration process, compared with the w.o. RCDN only about 47.68%/57.48%/37.15%. Note that the V2VNet baseline method's performance degrades sharply as the failed camera number increases, however, with RCDN, the V2VNet can settle in a considerable performance even with the failed camera number increases; iii) as for the dynamic part, some baseline methods are crashed even with only one random camera failure situation, e.g. DiscoNet only maintains about 19.87% performance of the standard collaborative perception setting, and almost every baseline method is unusable when there are three random camera failures, only about 20.73%/28.11%/13.09% of the standard situation. Nevertheless, with the RCDN, we see that all baseline methods still perform well even when three random failed camera situation appear, maintaining the 84.95%/90.81%/75.93% dynamic performance of the standard situation.Visualization of segmentation. We illustrate the map-view segmentation of other baseline methods w.o/w. RCDN and the corresponding repaired camera view in Figure 5 . The random camera failure number is one. The orange represents the drivable area, the blue represents the lanes and the teal represents the vehicles. We can see that i) other baseline methods show significant improvement in w.RCDN under noisy camera data; ii) V2VNet that collapses with noise camera data can also achieve the same level of performance as the origin data with the help of RCDN. Performance bottlenecks Regarding the increasing number of agents and cameras, we validated the impact of adding more cameras using the OPV2V-N dataset (corresponding scenario types are T section and midblock respectively) with the CoBEVT baseline. From Table 3 , we observe the following: i) With a single overlapping camera view, the proposed method significantly improves baseline performance, and ii) While theoretically, more cameras can provide a larger overlap range, the addition of multiple cameras (depending on their positions) may introduce redundant viewing angles, resulting in less significant performance improvements. Different downstream tasks Our proposed RCDN is general to different downstream tasks and is not limited to just BEV segmentation. We focus on BEV segmentation due to its crucial role in autonomous driving, with direct applications to other tasks such as layout mapping, action prediction, route planning, and collision avoidance. Additionally, we have validated RCDN for detection tasks, shown in Figure 7 . We replaced the original segmentation header with a detection header in our experiments. Table 4 shows that for CoBEVT, using RCDN improves the metrics of AP@0.50 and AP@0.70 by 19.05% and 24.99%, respectively. 
conclusion and limitation: 
 We formulate the camera-insensitivity collaborative perception task, which considers harsh realities of real-world sensors that may cause unpredictable random camera failures during collaborative communication. We further propose RCDN, a robust camera-insensitivity collaborative perception with a novel dynamic feature-based 3D neural modeling. The core idea of RCDN is to construct collaborative neural rendering field representations to recover failed perceptual messages sent by multiple agents. Comprehensive experiments show that RCDN can be portable to other baseline methods and stabilize the performance with a considerable level under all settings and far superior robustness with random camera failures.Limitation and future work. The current work focuses on addressing the camera-insensitivity problem in collaborative perception. It is evident that accurate reconstruction can compensate for the negative impact of noisy camera features on collaborative perception. In the future, we expect more works on exploring real-time collaborative neural field modeling with 3D Gaussian splatting.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: This paper specify all the training and test details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: This paper reports appropriate information about the statistical significance of the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: This paper provides sufficient information on the computer resources.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We mentioned creators or original owners of assets and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: This research conducted in the paper conform, in evrery respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no societal impact of the work performed. No harm technical paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper received 3 Borderline Accept and 1 Weak Accept. The reviewers agree that the solution and the problem is interesting. However, reviewers think that the evaluation is weak, but enough to show the benefits of the proposed approach. After reading the paper, the reviews, and discussion, I also think the problem and solution is interesting and can cause excitement in the community. Thus, I recommend acceptance. Nevertheless, I encourage the authors to revise the paper given the feedback the reviewers gave.

Reviews: review: paper_summary: paper_summary: The paper introduces RCDN, a novel method for robust camera-insensitivity collaborative perception. This method aims to overcome challenges associated with noisy, obscured, or failed camera perspectives by using dynamic feature-based 3D neural modeling. RCDN constructs collaborative neural rendering field representations to recover failed perceptual messages sent by multiple agents. The proposed system consists of two collaborative field phases: a time-invariant static background field and a time-varying dynamic field. To validate RCDN, a new dataset called OPV2V-N was created. The paper demonstrates that RCDN improves the robustness of baseline methods in extreme camera-insensitivity settings.
strength_weakness: strength_weakness: *Complexity and Computation*: The proposed method involves complex modeling and multiple steps. The author should provide the latency.

Generalizability: The performance of RCDN is primarily validated on the OPV2V-N dataset, which may limit the generalizability of the results to other datasets or real-world scenarios.


*Failure Cases*: It would be nice if the authors provide failure cases, which is important.
questions: questions: *Dataset Diversity*: Have you tested RCDN on any datasets other than OPV2V-N? How does it perform on real-world data?

*Real-Time Feasibility*: What are the computational requirements of RCDN, and how feasible is it for real-time applications in autonomous systems?

*Scalability*: How well does the method scale with an increasing number of agents and cameras? Are there any performance bottlenecks?
limitations: limitations: Please see weakness and questions.
paper_summary: paper_summary: The paper proposed Bird Eye View (BEV) semantic segmentation pipeline from collaborative perception, robust to motion blur, sensor noise, occlusion and even failure. The proposed a pipeline that adapts neural rendering techniques to overcome the noise/malfunction in camera capture and occlusion. With the proposed method combined with prior methods, performances on OPV2V-N (the proposed BEV semantic segmentation dataset) are improved.
strength_weakness: strength_weakness: Evaluation is only performed with OPV2V-N dataset which may result in overfitting. More evaluation with different dataset is required. The author may need to compare methodologies on other dataset although the existing dataset do not have noise. The author also may add random noise to the prior dataset and run experiments.

The manuscript was uneasy to read and understand. The paper should re-written. The comments below are without understanding supplemental materials fully.
- The way proposed algorithm is combined with prior method is unclear. The reviewer guessed that the MCP module can be replaced with prior methods, but it is not stated.
- Many abbreviations are not explained sufficiently and terminologies the author defined are ambiguous and may be incorrect. 
- MCP is short for the multi-agents collaborative perception process but the paper did not explain MCP module in details with no reference
- BEV, no full name, no reference.
- “Camera-insensitivity” can be understood terminologies related to camera sensor sensitivity (how much the camera sensor accept photon…).
- Robust Camera-Insensitivity: Robust == Camera-sensitivity? The latter one may be redundant
- Line 6. introduce a new robust camera-insensitivity problem: cam be replaced “introduce BEV segmentation when the camera capture are unreliable (or noisy)?” Should be more concrete without ambiguous words
- Line19 “Ported to” mean?
- There are more unclear sentences.
questions: questions: .
limitations: limitations: .
paper_summary: paper_summary: The paper presents RCDN, a method to aggregate multi-sensor perception signals in dynamic environment.
The key idea, is to improve the aggregated multi-agent feature with the multi-view rendering loss.
At its core, RCDN gathers input streams at varying timesteps of multiple agents. The gathered images are fused into Birds Eye view (BEV) then further decoded into volume. 
The volumetric features are learned into static scene and dynamic scene components with NGP based representation.
Overall procedure is supervised with rendering loss, (cyclic) optical flow consistency.


The method is evaluated on new dataset, OPV2V-N, which is an updated version of OPV2V, with additional masking and optical flow. 
The results show that RCDN helps BEV segmentation with various backbones, compared to the model used without RCDN.
strength_weakness: strength_weakness: The paper's main weaknesses are two folds. 
1. The paper does not evaluate on tasks other than BEV segmentation. 
While I believe that the pixel-aligned features from NGP would give benefits over various vision tasks, the paper only demonstrates on smaller domain of work which undermines its actual potential. It would have been more interesting to compare how it impacts in different downstreaming tasks, such as detection / tracking.

2. Technical contribution seems to lack novelty. 
The paper is a mix of two known-to-work solutions; BEV feature decoding for segmentation (used with various baselines in the experiments), and NGP (or radiance field based) multi-view pixel / density alignment through rendering loss. Usage of rendering loss to improve segmentation map is well-investigated in different literatures in the NeRF community (e.g, semantic-nerf).
questions: questions: These are few questions that I would like the authors to answer in the rebuttal.
1. How real is the synthetic OPV2V-N dataset? In other words, how can features learned in OPV2V-N dataset be translated to real-world usage? Moreover, are there any real-world quantitative results on model trained on synthetic data?

2. Have authors evaluated the method on different down streaming task other than segmentation? How does one verify that the volumetric features are geometrically correct? (how accurate is the Geometric BEV features?)

3. How is BEV segmentation evaluation differ on non-flat surfaces like hills or bridges?
limitations: limitations: No concerning limitations are found.
paper_summary: paper_summary: The paper introduces a new problem: how to overcome the issues caused by the failed camera perspectives, while stabilizing high collaborative performance with low calibration cost? Therefore, RCDN, a Robust Camera-insensitivity collaborative perception with a novel Dynamic feature-based 3D Neural modeling mechanism is introduced. To validate the new method, the authors also provide a new dataset: OPV2V-N. RCDN serves as baseline here. Ablation Study shows for 5 models (F-cooper, Att-Fuse, Disco-Net, V2VNet, CoBEVT a significant improvement over their baselines, w/o RCDN.
strength_weakness: strength_weakness: The overall system architecture sounds good. However, there are some open points for me, the impact of section 4.3 and 4.4, i.e. the neural fields part, seems open in terms of clarification. Example: What is difference between sf w , sbw in equation (7)?
The experimental section is a bit too short. I feel its not finished yet. However, there is limited space. The overall approach is not usable for realtime.
questions: questions: What is difference between sf w , sbw in equation (7)?
How many message exchange tasks could be used overall (Figure 2.)  Will baseline code be publishe in combination with the dataset? When?
limitations: limitations: The most relevant limitation is the missibg real-time applicability.
paper_summary: paper_summary: In this paper, the authors proposed an essential problems: how to overcome the issues caused by the failed camera perspectives, while stabilizing high collaborative performance with low calibration cost? The authors presented a robust camera-insensitivity collaborative perception with a novel dynamic feature-based 3d neural modeling mechanism to address the issue. Moreover, to verify the effectiveness of the model, the authors also provided a new large-scale dataset, OPV2V-N for this field. The experiments result showcase the model’s robustness in proposed dataset.
strength_weakness: strength_weakness: 1.	From my perspective, the paper lacks the theory analysis for the proposed method. Moreover, the authors fail to introduce the motivation of each sub-module in the presented model. For example, can the authors showcase the motivation of using Nerf for the static and dynamic fields, are there any dominant advantages of nerf, compared to other 3d reconstruction methods in this method?
2.	It is necessary to give more rigorous mathematic analysis of equations in this paper. Furthermore, the authors are required to introduce the details of each networks, including the training parameters, learning rate, weight values in eq. 12.
questions: questions: see weakness part
limitations: limitations: The current work focuses on addressing the camera-insensitivity problem in collaborative perception. It is evident that accurate reconstruction can compensate for the negative impact of noisy camera features on collaborative perception.

Comments: title: Thanks for your comment !
Thanks for your comment !
Thanks for your comment !
Thank You
Thanks for your comment !
Thanks for your comment !
Thank you for the rebuttal
comment: Thank you for your positive feedback on our proposed collaborative perception system via RCDN. We appreciate your acknowledgment that our approach is resistant to noise, blurriness, and interruption. Additionally, we confirm that there are no ethical concerns associated with the solution presented in this paper.
Thank you for your insightful comment regarding the potential limitations and vulnerabilities of our methodology in real-world scenarios, particularly in the context of autonomous driving. We agree that this is an important aspect to consider. In response to your feedback, we have included a detailed discussion in the global rebuttal where we address these concerns. Specifically, the proposed RCDN compensates for the negative impact of noisy cameras through multi-view accurate reconstruction. The distinction between synthetic and real datasets primarily affects the distribution of RGB images, influencing later BEV feature space but not conflicting with the fundamental principles of RCDN. Therefore, the effectiveness of RCDN remains unchanged whether using synthetic or real datasets. RCDN has been thoroughly validated in synthetic OPV2V-N experiments, demonstrating stable performance under noisy camera conditions. However, we will keep track of real-world dataset development, implementing the RCDN in real-world settings when the camera settings meet our requirements.
Thank you for your feedback and for acknowledging our efforts to address your comments. We appreciate your recommendation regarding W.1/A.1 and will ensure that these points are carefully addressed in the final version of the paper. Your insights have been invaluable in improving our work, and we are committed to refining it further.
The authors addressed my comments very well. Thank you.
I would recommend to address W.1/A.1 for a final paper.
Overall, I stay at my decision due to the weaknesses described above.
Thanks for your positive feedback and valuable insights! In our revised manuscript, we will ensure that the writing is clear and accessible, assuming that the reader may be unfamiliar with the topic. We will also replace any ambiguous or subjective language with more concrete and specific terms. Additionally, we will clearly outline all technical details to ensure a comprehensive understanding for all readers.
Thank you for your thorough review and constructive feedback. We sincerely appreciate your valuable reviews and are glad to know that our rebuttal and new experiments have addressed most of your concerns.
When revising the writing, it is recommended below:
1. Assume that the reader is unfamiliar with the topic.
2. Replace ambiguous or subjective words/sentences to more concrete and specific ones.
The authors have processed all my concerns. Thanks!