UC-NERF: Neural Radiance Field for Under-Calibrated Multi-View Cameras in Autonomous Driving
Authors: Kai Cheng
Xiaoxiao Long
Wei Yin
Jin Wang
Zhiqiang Wu
Yuexin Ma
Kaixuan Wang
Xiaozhi Chen
Xuejin Chen
Venue: ICLR 2024 poster (2024)
Keywords: Neural radiance field
Multi-cameras
Under-Calibration
Color Inconsistency
Primary area: applications to robotics, autonomy, planning

Abstract: Multi-camera setups find widespread use across various applications, such as autonomous driving, as they greatly expand sensing capabilities. 
Despite the fast development of Neural radiance field (NeRF) techniques and their wide applications in both indoor and outdoor scenes, applying NeRF to multi-camera systems remains very challenging. This is primarily due to the inherent under-calibration issues in multi-camera setup, including inconsistent imaging effects stemming from separately calibrated image signal processing units in diverse cameras, and system errors arising from mechanical vibrations during driving that affect relative camera poses.
In this paper, we present UC-NeRF, a novel method tailored for novel view synthesis in under-calibrated multi-view camera systems.
Firstly, we propose a layer-based color correction to rectify the color inconsistency in different image regions. Second, we propose virtual warping to generate more viewpoint-diverse but color-consistent virtual views for color correction and 3D recovery. Finally, a spatiotemporally constrained pose refinement is designed for more robust and accurate pose calibration in multi-camera systems.
Our method not only achieves state-of-the-art performance of novel view synthesis in multi-camera setups, but also effectively facilitates depth estimation in large-scale outdoor scenes with the synthesized novel views.

Published as a conference paper at ICLR 2024
UC-NERF: NEURAL RADIANCE FIELD FOR UNDER- CALIBRATED MULTI-VIEW CAMERAS IN AU-
TONOMOUS DRIVING
Kai Cheng"", Xiaoxiao Long2.3, Wei Yin', Jin Wang', Zhiqiang Wu', Yuexin Ma², Kaixuan Wang'. Xiaozhi Chen', Xuejin Chen
1 MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China 2 The University of Hong Kong
PKU-Wuutt Institute for Artificial Intelligence
DJI Technology ShanghaiTech University
ABSTRACT
Multi-camera setups find widespread use across various applications, such au- tonomous driving, as they greatly expand sensing capabilities. Despite the fast de- velopment of Neural radiance field (NeRF) techniques and their wide applications in indoor and outdoor scenes, applying NeRF to multi-camera systems remains very challenging. This is primarily due to the inherent under-calibration issues in multi-camers setup, including inconsistent imaging effects stemming from sep- arately calibrated image signal processing units in diverse cameras, and system errors arising from mechanical vibrations during driving that affect relative cam- era poses. this paper, we present UC-NeRF, novel method tailored for novel view synthesis in layer-based color correction to rectify the color inconsistency in different image regions. Second, we propose virtual warping to generate more viewpoint-diverse but color-consistent virtual views for color correction and 3D recovery. Finally, spatiotemporally constrained pose refinement is designed for more robust and ac- curate pose calibration in multi-camera systems. Our method not only achieves the tfate-of-the-an performance of novel view synthesis in multi-camera setups but also effectively facilitates depth estimation in large-scale outdoor scenes with the synthesized novel views.
multi-view camera systems. Firstly, we propose
INTRODUCTION
Neural radiance field (NeRF) is revolutionary approach that enables the synthesis of highly detailed and photorealistic 3D scenes from 2D images. This technology has opened up multitude of new possibilities autonomous driving, such as generating synthetic data from diverse viewpoints for robust training of perception models and providing effec- tive 3D scene representations to enhance comprehensive environmental understanding (Fu et al. (2022): Zhang et al. )2023)).
Multi-camera systems (Sun et al. (2020); Caesar et al. (2000;; Guizilini
et al. (2020)) are commonly used for autonomous driving, while involv- ing the strategic placement of multiple cameras to capture a holistic per- spective of the surrounding environment, as shown in Fig- [ supplying spatially consistent information to complement the temporal data for per- ception tasks (Mei al.] (2022); [ang al.) (2023)). Incorporating NeRF in multi-camera systems could provide a way to efficiently and economically produce extensive high-quality video data for training various models in autonomous driving systems.
Figure Illustration of multi-camera system in autonomous driving.
However, naively combining images captured from multi-camera systems into NeRF's training of- ten results significant deterioration of rendering quality, as depicted in the first two rows of
First two authors contributed equally.
1 'Coresponding athuor (xjchen99 uust.cddu.c)..

---

Published as a conference paper at ICLR 2024
Zip-NeRF
Ours
Figure 2: In under-calibrated multi-camera camera systems, the NeRF quality significantly degrades (the first row), along with color discrepancies (red boxes), object ghosts (red arrows). and wrong geom- etry (the second row). Our UC-NeRF achieves high-quality rendering and accurate geometry in the challenging cases.
Fig. 2 The underlying cause of this degradation is the inconsistent color supervision from different views, since the multi-cameral are usually under-calibrated. This under-calibration issue manifests in two distinct ways. First, the image signal processing (ISP) units involve various techniques like white balance correction, gamma correction, etc., to convert the raw data captured by the sensors to discretized pixel colors. However, these ISP units fluctuate over time across different cameras, re- sulting in color disparities within the same scene region, as shown in the red box in Fig. Secondly, even with delicate camera pose calibration beforehand, systematic errors during vehicle production and vibrations during driving inevitably introduce misalignment and further exacerbate color incon- sistency, as indicated by the arrows in Fig. 2
Several NeRF methods (Martin- MMrttnnBruullaa et al. 2021 Rematas et al. 2022 Tancik et al. (2022)) at-
tempt to alleviate the inconsistent color supervision by modeling image-dependee appearance with global latent code for each image. However, the capacity of a global latent code uniformly rect colors in different regions of an image limited, especially when different regions correspond different color transformations. Furthermore, learning one color correction code for each image usually leads to overfitting when the training images lack color and viewpoint diversity. This lim- itation pronounced for areas observed by side-view cameras, which have fewer observations and limited overlapping with front-view areas. To correct inaccurate poses, some approaches perform joint NeRF optimization with pose refinement using photometric losses. Unfortunately, utilizing such joint optimization under multi-camera setup, the photometric consistency across cameras can be ensured and spatial relations among cameras are not fully utilized, making optimization more challenging and prone to local minima.
To address these challenges, we introduce UC-NeRF, method for high-quality neural rendering with multiple under-calibrated cameras. We introduce three key innovations: 1) Layer-based Color Correction. To address color inconsistencies in the training images, especially for those taken by different cameras, design novel layer-based color correction module. This module separately adjusts the colors the foreground and sky regions using two learned affine transformations for each image. 2) Virtual Warping. We introduce "virtual warping" strategy that generates viewpoint-
2

---

Published as a conference paper at ICLR 2024
observations for each camera at each moment. These warped images under virtual viewpoints offer stronger constraints on the latent codes for color correction, especially for multi-camer systems where the overlapping region between cameras is limited. Moreover, the virtual warping strategy naturally expands the range of the training views for NeRF, enhancing its effectiveness in learning both the scene's appearance and geometry. 3) Spatiotemporally Con- strained Pose Refinement. We propose spatiotemporally constrained pose optimization strategy that explicitly models the spatial and temporal constraints between cameras for pose optimization. This approach also improves the robustness against photometric differences by utilizing reprojection errors during pose optimization.
diverse yet
Experiments on the public datasets Waymo (Sun et al. (2020) and NuScenes (Caesar et al. (2020)) show that our method achieves high-quality renderings with multi-camera system and outperforms other baselines by large margin. Moreover, we show that the obtained high-quality renderings of novel views can facilitate downstream perception tasks like depth estimation.
RELATED WORK
Multi-view Stereo Multi-view stereo (MVS) is fundamental 3D vision task that aims to recon- struct a 3D model from posed images. Traditional methods (Campbell et al. (2008); Furukawa &
Ponce (2009); Bleyer et al. (201T); Furukawa et al. (2015); Schönberger et al. (016)) exploit pixel
correspondences between images from hand-crafted features to infer 3D structure. Deep learning
methods Yao et al. (2018): Vakalopoulou et al. (2018); Long al. (2020) Chen et al. 2019); 9
Long et al. (2021) Ma et al. (2022) Feng et al. (2023)) generally build multi-view view correspondences
implicitly and regress the depth maps or 3D volumes an end-to-end framework. Despite the increasing capability of MVS techniques in reconstructing accurate 3D models, it is strenuous to integrate 3D models into the traditional rendering pipeline to achieve photorealistic rendering.
NeRF for Outdoor Scenes NeRF (Mildenhall et al. (2021)) is revolutionary technique that al- lows realistic novel view rendering without explicitly reconstructing 3D models. Its effectiveness in high-quality novel view synthesis has been demonstrated
indoor scenes and small-scale outdoor scenes. But faces challenges when applied to unbounded outdoor scenes due to infinite depth range, complex illumination, and dynamic objects. To make NeRF more effective for infinite depth range, NeRF++ (Zhang et al.] (2020) divides the scene into foreground and background regions with inverted sphere parameterization. In the following works (Barron al. (2022); Wang et al. (223a)), more complicated non-linear scene parameterization is proposed to model the outdoor space more compactly and sample points more efficiently. Some other works (Deng et al. (2022); Xie et al. (203);; Wang et al. (2023b); Yang et al. (2023)) learn the complex geometry of outdoor scenes by introducing depth and surface normal priors. Moreover, to adapt to the view-dependent appearance due to surface reflection, camera parameters, and environment change, several works -Martin-rualla et al. (202T): Rematas et al. (2022): ) TTancik et al. (2022) Turki et al. (2022); [ et al.) 2023) learn appearance- related latent codes independently to control the view-dependent effect. Besides, some works (Xic et al. 2023); Turki et al. (2023) model dynamic objects separately based on semantic priors, using 3D detection or semantic segmentation. In comparison, primarily ad- dress the rendering quality deterioration problem caused by under-calibration of photometry and poses in multi-camera setup for large-scale outdoor scenes.
NeRF with Pose Refinement NeRF methods always require accurate camera poses to optimize neural 3D scene. However, the camera poses obtained from Prruttre-from-Mootoo (SfM) usu- ally contain subtle errors that could significantly degrade the quality of the reconstructed NeRF. NeRF-- (Wang et al. (202T)) jointly optimizes camera parameters with NeRF training via the photometric loss. However, the pose optimization struggles to achieve effective updates due to the increased non-linearity of NeRF arising from position encoding. BARF (Lin al. (202T) elimi- nates this negative impact with a coarse-to-fin training strategy. SiNeRF (Xia et al. (2022) and GARF (Shi et al.) (2022) replace the positional encoding with different activation functions to re- duce non-linearity while maintaining the same rendering quality. Besides, SCNeRF (Jeong et al. (021 and SPARF (Truong et al. (2023)) propose different geometric losses to improve the pose accuracy further. However, directly employing these methods in multi-camera systems will lead to a bottleneck, since the spatial relation between cameras is not taken into consideration. MC-NeRF [ao et al. 2023) is contemporaneous work multi-camera systems, but it focuses on optimiz-
3

---

Published as a conference paper at ICLR 2024
Volume Rendering
Layer-based Color Correction
Sky Loss
Photometric
Regularization Loss
Rendered
Spatiotemporaly Constrained
Virtual Warping
Pose Refinement
12 tasa La 47 Initial & Refined & Virtual Poses (Initial) Refined Pases Multi-view Images
Virtual Poses Virtual Images
Figure 3: Overview of UC-NeRF framework. To mitigate the inconsistency color supervision in multi-camera systems, the spatiotemporally constrained pose refinement module optimizes poses and the layer-based color correction module models the image-dependent appearance from varying cameras and timestamps. The virtual warping module generates diverse virtual views with geometric and color consistency, enriching data for color correction and 3D scene recovery.
ing the camera intrinsics during pose optimization while we propose to impose the spatiotemporal constraint between cameras for the pose optimization.
METHOD
Our UC-NeRF extends the general NeRF algorithm the multi-camera setup in autonomous driv- ing. We begin by reviewing the common NeRF pipeline. Then we introduce the layer-based color correction (Sec. 3.2) to reformulate the color rendering for handling the inconsistent color supervi- sion in multi-camera systems. In Sec. 3.3 we introduce our virtual warping strategy to assist color color-consistent images. Finally, the spatiotempo- rally constrained pose refinement is explained in Sec. 3.4
correction by generating viewpoint-diverse
3.1 PRELIMINARY
NeRF models a 3D scene as a continuous implicit function and regresses the density a and color 3 of every individual 3D point given its 3D coordinate p E RR3 and unit-norm viewing direction d E 2 To synthesize 2D image, NeRF employs volume rendering which samples sequence of 3D points along camera ray I(r) Σn T., where Tn is the accumulated transmittance of the sampled points, an and c, the alpha value and the color the sampled nith point. Detailed definitions can be referred to NeRF (Mildenhall et al. (202I)) ).
To optimize a NeRF, the photometric loss between the rendered color I(r) and the ground truth color i(r) from set of sampled rays R. is applied as:
Σ [(r)) - 1r))
Lpho (0)
FER
3.2 LAYER-BASED COLOR CORRECTION
In multi-camera systems, different cameras always have distinct Image Signal Processor (ISP) con- figurations, resulting in inconsistent imaging colors for the same 3D region. As a result, optimizing a NeRF representation using such inconsistent images always causes low-quality renderings. The work Urban-NeRF (Rematas et al. (2022)) attempts to approximate global linear compensation transformation for each view to alleviate the discrepancies the views from different cameras. is worth noticing that due to the non-linear property of the ISP process, it is insufficient to model the spatially varying color patterns using a single global compensation transformation. To balance the rendering quality and efficiency, the color compensation transformation for each layer separately. This because the sky regions are always much brighter than the foreground objects and they present distinct ISP imaging effects.
propose split the scene into foreground-sky layers and model
4

---

Published as a conference paper at ICLR 2024
Camera Trajectory
Real Views
Virtual Views
Poses Virtual Views
Geometric Consistency Check
MVS
Warping
Geumetric Consistent Mask
Images Real Views
Depths in Real Views
Images in Virtual Views
in Real Views
Generation warped images virtual views. For each known viewpoint, we estimate its depth map using deep MVS method and filter out inaccurate depths through geometric con- sistency check. Each virtual view is obtained by warping the image from a known viewpoint to the virtual viewpoint.
Figure 4:
We first model the foreground and sky as two independent NeRF models Osg and Osky. The color of rendered pixel from ray is obtained by the weighted combination of foreground color (r) and sky color Skk (r), as illustrated in Eq. 2
I(r) f + 0f
(2)
where 0sg = -> fn is the accumulated weight of foreground NeRF along r. To encour- age Ofg approach 1 in the foreground area and approach 0 in the sky area, binary cross-entropy loss is employed as Eq.
Lisky Liky(Ofo (Ofgs Maky) = -aaky log(1 ofa) - may log(osa),
where maky is the sky mask generated from pretrained segmentation model Yin et al. (2022)
After modeling the foreground and sky NeRF, we approximate the color correction of the foreground and the sky using separate affine transformations. Considering the color variance across both cam- eras and times, for each training image I, from camera k at time foreground correction code and sky correction code are assigned to represent the image-dependent color variation (Subscripts i, k are omitled in the following descriptions for clarity.. These correction codes are further de- coded by multi-layer perceptron (MLP) as the affine transformations [A, x] and yC, where A, C e 33x3, and x, - E p331 For the rendered pixel which emits in I, the final pixel color in Eq. can be rewritten as:
I(r) (r) + (1 (1-00 059 (CIkkk)))) +)-
To stabilize the optimization process and ensure that the adjusted color does not significantly devi- ate from the original observation, we add a regularization term for the transformation matrices, as illustrated in Eq. 5
Lreg IA MA/ll/ + |C Ea
x lvl.
(5)
where Eg refers the identity matrix.
3.3 VIRTUAL WARPING
In multi-camera systems, images from different viewpoints often have limited overlapping areas, making it more challenging to align their colors compared to aligning frames from a single camera. To align the image colors of multiple cameras and prevent the optimized latent codes for color correction from overfitting to specific view, we propose virtual warping, which simulates more diverse yet color-consistent images under set of virtual viewpoints for training. Furthermore, virtual warping naturally expands the range of perspectives available to NeRF, thereby enhancing its capability to reconstruct the 3D scene. Fig. shows the pipeline of our virtual warping. We employ deep MVS method (Ma et al. (2022)) to estimate depth maps of all views. To remove outliers and retain the consistent depths across multiple views, we further leverage a geometric consistent check (cchöbeerger al. (2016)) generate mask M that only keeps reliable depth values.
5

---

Published as a conference paper at ICLR 2024
With estimated depths, we generate multiple virtual poses and warp colors and color correction codes to the virtual positions. Specifically, we perturb an existing pose T, with an additional trans- formation [Rg- tg-vo] as virtual pose T. The rotation matrix R is generated by randomly selecting one of the three axes with a random angle E [-20", 20 The translation t-- is a 31) vector of random direction with length [0m, 1m]. Each pixel P. in existing image taken under camera pose To is warped to an image point with the virtual pose Te as:
d...P KRoo... doPo + to-so)
where Po and P. is the homogeneous coordinates of P. and pe. K is the camera intrinsic matrix, d, and d, are the pixel depth in the virtual view and the corresponding pixel depth in the original real view. Considering object occlusions, there could be multiple pixels in the original views mapped to the same position in the virtual view. We keep the warped pixel with the minimum depth.
After warping, the geometric consistency mask M is applied to the warped pixels to remove pixels with noisy depth. Then the color and the color correction code of the pixel the original view are assigned to the corresponding warped pixels in the virtual view. This provides more clues to recover consistent appearance and geometry of the scene.
3.4
SPATIOTEMPORALE CONSTRAINED POSE REFINEMENT
The rendering quality of NeRF heavily relies the accuracy of camera poses. Previous approaches (Tancik et al. (2022); :::::;(]xx) Xie et al.]
T/ATn
T/AT,
(023 model the camera poses independently and jointly optimize them within the NeRF framework. They do not fully exploit the spa- tial correlations between cameras multi-camera systems, leading to under-constrained pose optimization. Additionally, camera pose optimization depends on the photometric consistency assumption, which is usually violated in long-time videos captured in multi- camera systems. Given the condition that the cameras have a fixed spatial relationship with the main capturing device (i. e. the driving car) during the whole process, we explicitly establish the temporally fixed geometric transformation between cameras.
T/AT2
T ATo
T NT,
T'AT
Figure 5: Pose modeling.
While capturing multi-view images with cameras, the pose Tý the kth camera at time is denoted as the combination of the car's ego pose T and the relative transformation AT,, which is temporally consistent and optimizable, as Fig. Showss. Explicitly modeling the spatial relation- ship between cameras provides more restrictions for pose refinement, thus effectively enhancing robustness against incorrect point matches across all frames.
After building the spatio-temporal constraint between camera poses, point correspondences between images captured by different cameras different times are established as correlation graph E. Then we employ bundle adjustment to minimize the reprojection error defined as:
Lrpj
Σ
p - II, ((T'AT)) 'T (T'TT)) AT (qb))
(O.A).G.))EE
where p/ and qi are pixels in the images captured by camera I at time j and camera k at time i, II: and II,"" projection function of camera I and unprojection function of camera k.
3.5 TRAINING STRATEGY
Our NeRF training process consists of two stages. The first stage is pose refinement and depth estimation. We initialize the camera poses from sensor-fusion SLAM and further optimize them using our proposed spatiotemporally constrained pose refinement module, as described in Eq. 7 With these refined poses, we generate a depth map and geometric consistency mask for each image, following the procedure outlined in Sec. 3.3 The second stage is the end-to-end NeRF optimization. Specifically, the proposed layer-based color correction and virtual warping strategies are employed in the NeRF optimization to achieve high-quality renderings. In each training batch, we randomly sample B real images and employ our virtual warping module to create virtual views for each real image. The pixels randomly sampled from these real and virtual views the ground truth

---

Published as a conference paper at ICLR 2024
Table I: Comparison Waymo and NuScenes. Our method significantly outperforms other state-
of-the-art methods on all the evaluation metrics.
Waymo SSIM f
NuScenes SSIM 1
Method
PSNR
LPIPS +
PSNR
LPIPS
Mip-NeRF (Barron al. 2021))
22.42
0.698
0.471
23.31
0.758
0,489
Mip-NeRF 360 (Barron al. 2022
24.46
0.769
0.406
25.15
0.809
0.436
Instant-NGP (Müller et al. S S-NcRF Xie ct al. (2025)
23.84
0.702
0.494
23.81
0.777
0.476
24.89
0.772
0.401
26,02
0.824
0.415
Zip-NeRI Barron èl al.
2023
26.21
0.815
0.389
27.06
0.831
0,435
UC-NeRF (Ours)
28.13
0.842
0.356
30.20
0.876
0.374
for NeRF training. Our UC-NeRF renders these pixels based on Eq. and is supervised by the loss function in Eq. 8
L Lihho + ALoky + Lregs
(8)
where À and y are the weights Lay and Lreg-
4 EXPERIMENTS
4.1 DATASETS AND IMPLEMENTATION DETAILS
Datasets. We conduct experiments on two urban datasets with images captured with multi-camera settings, i.e., Waymo (Sun et al. (2020) and NuScenes (Caesar al.] (2020)). We select ten static scenes in Waymo and five static scenes NuScenes for evaluation. To evaluate the performance of novel view synthesis, following common settings, select one of every cight images of each camera as testing images and the remaining ones as training data. We apply the three widely-used metrics for evaluation, i.c., peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and the learned perceptual image patch similarity (LPIPS) (Zhang et al. (2018)).
Baselines. We choose Zip-NeRF (Barron al. (2023) as our baseline using the codes imple- mented by Gu (023). We compare our method with the baseline and other NeRF methods, includ- ing Mip-NeRF (Barron al. (202T)) Mip-NeRF 360 ([Barron et al. (2022), Instant-NGiP (Müller ct al. 2022)), ) and S-NeRF (Xie et al. (2023)..
We provide implementation details in the appendix.
4.2 RESULTS OF NOVEL VIEW SYNTHESIS
The comparison of neural rendering results for urban scenes with multi-camera settings is shown Tab. [ With the proposed layer-based color correction, virtual warping, and spatio-temporally constrained pose refinemen,, our UC-NeRF outperforms the other methods on both datasets. We also show the panoramic rendering results in Fig. 6 Without any anti anti-aliasing design in the rendering process, images generated by Instant-NGP and S-NeRF exhibit notable blurriness. Although Zip- NeRF features an anti-aliasing mechanism, it also amplifies the artifacts caused by inconsistent color supervision across different views. Our approach excels at rendering consistent colors and sharp details, as highlighted the regions of texts, cars, and buildings. Additionally, our method provides more accurate 3D reconstruction, demonstrated by the depth maps. We show more results of the Waymo and NuScenes datasets the appendix.
Table 2: Ablation study.
4.3 ABLATION STUDY
LCC
STPR
VW
PSNR 1
SSIM t
LPIPS 4
We conduct extensive ablation studies on ten scenes from the Waymo dataset to explore the
X
X
X
26.21
0.815
0.389
X
26.95
0.839
0.360
effect of each proposed module in our UC- NeRF, We investigate the effect of each mod- ule, i.e., based color correction (LCC), spatiotemporally constrained pose refinement (STPR), and virtual warping (VW). As shown in Tab. the layer-based color correction module brings significant improvement (the third row)
X
27.18
0.820
0.375
X
27.26
0.825
0.372
X
27.82
0.838
0.371
V
28.13
0.842
0.356

---

Published as a conference paper at ICLR 2024
(a) Instant-NCr -GP
(b) S-NeRF
Zip-NeRF
STEEE
STEYE IIVENT
(d) Ours
TOPE
Figure 6: Comparison of rendered panoramas at a resolution 5760 1280. Notable enhancements are indicated in the blue and red boxes. The close-ups of some patches are displayed to emphasize the rendering details. Compared to other methods, our results present consistent colors and sharp details, even faithfully recovering the slogans. Please refer to the appendix for additional results.
compared with the baseline model, since it solves the problem of inconsistent color supervision between views in training. Fig. 7 (b) also illustrates that the LCC module reduces hazy artifacts and presents sharper renderings. By incorporating the spatiotemporally constrained pose refinement (STPR) module, the quality of rendering is further improved. Moreover, virtual warping (VW) strategy can enrich the diversity training views for learning color correction, appearance, and ge- ometry. Even the object details, e.g. the car lights and the car emblem, become more discernible in Fig. 7 (d). One notable thing is that the accuracy of the virtual views provided by virtual warping closely related to the accuracy of the poses. Thus, virtual warping provides more noticeable boost when the pose refinement module is added (the fourth and sixth row in Tab. 2 More detailed discussions can be found the appendix.
Benefit of Virtual Warping Our virtual warping enriches NeRF's training perspectives of each
camera by generating images with consistent geometry and appearance. In addition to the overall improvement in rendering quality shown in Tab. 2 we present icant enhancement in rendering quality after incorporating virtual warping in Fig. X We can see better color correction (the first row) and enhanced image details (the second row).
cases that demonstrate signif-
8

---

Published as a conference paper at ICLR 2024
a Baseline
(b) Layer-based Color Correction
(c) + Spatiotemporally Constrained Pose Refinement
(d) Virtual Warping (Full Model)
Figure
Each proposed module enhances the rendering quality of novel views.
GT
W/O Virtual Warping
W/ Virtual Warping
5721
5721
Figure 8: Our virtual warping contributes to color correction (top) and edge sharpness (bottom).
Efficiency Analysis We com- pare the efficiency of different
Table 3: Efficiency Analysis. Tested on one NVIDIA Tesla V100
GPU with image resolution 1920 x 1280.
methods in Tab. 3 All meth- ods are tested on one NVIDIA Tesla V100 GPU with an im- age resolution of 1920 1280. Note that our training time in- cludes both steps (pose refine- ment and NeRF training) de-
Method
Training
Inference
PSNR
Mip-NeRF (Barron
2021
20h
70s
22.42
Mip-NeRF-3G0) (Harron et al. (2022)
14h
42s
24.46
Instant-NGP (Müller el al. 2022
30min
0.35s
23.84
S-NeRF (Xie et al. (2023)
15h
80s
24.89
Zip-NeRI (Barron UC-NeRF (Ours)
(0233
2h
2s
26.21
3.2s
28.13
scribed in Sec. 3.5 Zip-NeRF
more efficient than other methods except Instant-NGP, which is specifically designed for NeRF acceleration. Since our method built upon Zip-NeRF, our method consumes a bit more time than Zip-NeRF but achieves significant improvement in rendering quality.
CONCLUSION AND FUTURE WORK
In conclusion, we propose UC-NeRF that effectively addresses the challenges integrating multi- camera systems into the NeRF paradigm by layer-based color correction module, virtual warping module, and spatiotemporally constrained pose refinement module. Experiments on the Waymo and NuScenes datasets demonstrate a significant improvement in rendering quality, setting new benchmark for neural rendering within multi-camera setups. With the high-quality novel view ren- dering of our UC-NeRF, the synthesized images can provide plentiful training data for depth es- timation, semantic segmentation, and object detection in autonomous driving scenarios. With the development of more neural reconstruction techniques, such 3D Gaussian splatting (Kerbl et al. (2023)), ) the strategies layer-based correction, virtual warping, and pose refinement in our UC- NeRF can be easily extended to facilitate the reconstruction quality using these techniques.

---

Published as a conference paper at ICLR 2024
Acknowledgements Thanks to DJI for providing sufficient computing resources. This work was supported by National Natural Science Foundation of China (NSFC) under Grants 62076230 and the Fundamental Research Funds for the Central Universities under Grant WK3490000008.
REFERENCES
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Brrtinn-ruuala,, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 5855-5864,
2021.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5470-5479, 2022.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip- nerf: Anti-aliased grid-based neural radiance fields. Proceedings of the IEEF/CVF International Conference on Computer Vision, 2023.
Michael Bleyer, Christoph Rhemann, and Carsten Rother. Patchmatch stereo-stereo matching with slanted support windows. In Brvc, volume 11, pp. 1-11, 2011.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Osear Beijbom. nuscenes: multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1621--1631, 2020.
Neill DF Campbell, George Vogiatzis, Carlos Hernández, and Roberto Cipolla. Using multiple 2008: 10th
hypotheses to improve depth-maps for multi-view stereo. Computer European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part 10, pp. 766-779. Springer, 2008
Rui Chen, Songfang Han, Jing Xu, and Hao Su. Point-based multi-view stereo network. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pp. 1538-1547, 2019.
Kangle Deng, Andrew Liu, Jun-Yan Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12882-12891, 2022.
Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest. point detection and description. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 224-236, 2018.
Ziyue Feng, Leon Yang, Pengsheng Guo, and Bing Li. Cvrecon: Rethinking 3d geometric feature learning for neural reconstruction. arXiv preprint arXiv:2304. 14633, 2023.
Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, Andreas Geiger, and Yiyi Liao. Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmenta- tion. In 2022 International Conference on 3D Vision (3DV), pp. 1-11. IEEE, 2022.
Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. IEEE trans- actions on pattern analysis and machine intelligence, 1362):136211377 2009.
Yasutaka Furukawa, Carlos Hernández, et al. Multi-view stereo: A tutorial. Foundations and TrendyS in Computer Graphics and Vision, (1-2): 2015.
Yu Gao, Lutong Su, Hao Liang, Yufeng Yue, Yi Yang. and Mengyin Fu. Mc-nerf: Muti-camera neu- ral radiance fields for muti-camera image acquisition systems. arXiv preprint arXiv:2309.07846, 2023.
Chun Gu. Zipnerf-pytoree 2023.
URL https://githu... www.:/tthhu..c..uuuuuutut wttp://.oooommtuutaaaaaiiiiii/
zipnerf- -pttorchhh Accessed: 2023-04-23.
10

---

Published as a conference paper at ICLR 2024
Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon. 3d packing for self--upervised monocular depth estimation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 2485-2494, 2020.
Yoonwoo Jeong, Seokjun Ahn, Christopher Choy, Anima Anandkumar, Minsu Cho, and Jaesik Park. Self-calibrating neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5846-5854, 2021.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions Graphics (TOG), 42(4):1-14, 2023.
Peihao Li, Shaohui Wang, Chen Yang, Bingbing Liu, Weichao Qiu, and Haoqian Wang. Nerf- ms: Neural radiance fields with multi-sequence. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 18591-18600, 2023.
Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey, Barf: Bundle Bundle-adjusting neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5741-5751, 2021.
Ce Liu, Suryansh Kumar, Shuhang Gu, Radu Timofte, and Lue Van Gool. Va-depthnet: A variational approach to single image depth prediction. In The Eleventh International Conference Learning Representations, 2023.
Xiaoxiao Long, Lingjie Liu, Christian Theobalt, and Wenping Wang. Occlusion-awand aware depth es- timation with adaptive normal constraints. In Computer Dision-ECCH 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16, pp. 640-657. Springer, 2020.
Xiaoxiao Long. Lingjie Liu, Wei Li, Christian Theobalt, and Wenping Wang. Multi-viww depth estimation using epipolar spatio-temporal networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8258-8267, 2021.
Zeyu Ma, Zachary Teed, and Jia Deng. Multiview stereo with cascaded epipolar raft. In European Conference on Computer Vision, pp. 734-750. Springer, 2022.
Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collec- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7210-7219, 2021.
Jieru Mei, Alex Zihao Zhu, Xinchen Yan, Hang Yan, Siyuan Qiao, Liang-Chieh Chen, and Hen- rik Kretzschmar. Waymo open dataset: Panoramic video panoptic segmentation. In European Conference on Computer Vision, pp. 53-72. Springer, 2022.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), (14) 2022.
Ziqi Pang, Jie Li, Pavel Tokmakov, Dian Chen, Sergey Zagoruyko, and Yu-Xiong Wang. Standing between past and future: Spatio-temporal modeling for multi-camera 3d multi-object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 17928-17938, 2023.
Konstantinos Rematas, Andrew Liu, Pratul P Srinivasan, Jonathan T Barron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Ferrari. Urban radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19932-12942 2022.

---

Published as a conference paper at ICLR 2024
Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part III 14, pp- 501-518. Springer, 2016.
Yue Shi, Dingyi Rong. Bingbing Ni, Chang Chen, and Wenjun Zhang. Garf: Geometry-ave aware generalized neural radiance field. arXiv preprint arXiv:2212.02288, 2022.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference com- puter vision and pattern recognition, pp. 2446-2454, 2020.
Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srini- vasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2022.
Fabio Tosi, Alessio Tonioni, Daniele De Gregorio, and Matteo Poggi. Nerf-supervised deep stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 855-866, 2023.
Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari. Sparf: Neural radiance fields from sparse and noisy poses. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4190-4200, 2023.
Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12922-12931, 2022.
Haithem Turki, Jason Y Zhang. Francesco Ferroni, and Deva Ramanan. Suds: Scalable urban dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12385-12385, 2023.
Maria Vakalopoulou, Guillaume Chassagnon, Norbert Bus, Rafael Marini, Evangelia I Zacharaki, M-P Revel, and Nikos Paragios. Atlasnet: Multi-atlas non-linear deep networks for medical image segmentation. In Medical Image Computing and Computer Assisted Intervention MICCAI 2018: 2 st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part IV 11, pp. 658-666. Springer, 2018
Peng Wang. Yuan Liu, Zhaoxi Chen, Lingjie Liu, Ziwei Liu, Taku Komura, Christian Theobalt, and Wenping Wang. F2-nerf: Fast neural radiance field training with free camera trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
4150 4150-4159, 2023a.
Zian Wang. Tianchang Shen, Jun Gao, Shengyu Huang, Jacob Munkberg. Jon Hasselgren, Zan Gojeic. Wenzheng Chen, and Sanja Fidler. Neural fields meet explicit geometric representations for inverse rendering of urban scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8370-8380, 2023b.
Zirui Wang. Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint
2021.
Yitong Xia, Hao Tang, Radu Timofte, and Luc Van Gool. Sinerf: Sinusoidal neural radiance fields for joint pose estimation and scene reconstruction. arXiv preprint arXiv:2210.04533 2022.
Ziyang Xie, Junge Zhang, Wenye Li, Feihu Zhang, and Li Zhang. S-nerf: Neural radiance fields for street views. In The Eleventh International Conference on Learning Representations, 2023.
Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed- llop sensor simulator. In Proceedings of the IEEÉ/CVF Conference Computer Vision and Pattern Recognition, pp. 1389-1399, 2023.
12

---

Published as a conference paper at ICLR 2024
Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstruc- tured multi-view stereo. In Proceedings of the European conference on computer vision (ECCV), pp. 767-783, 2018.
Wei Yin, Yifan Liu, Chunhua Shen, Anton van den Hengel, and Baichuan Sun. The devil is in the labels: Semantic segmentation from sentences. arXiv preprint arXiv:220..02002 2022.
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Xiaoshuai Zhang, Abhijit Kundu, Thomas Funkhouser, Leonidas Guibas, Hao Su, and Kyle Gen- ova. Nerflets: Local radiance fields for efficient structure-aware 3d scene representation from 2d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8274 6274-8284, 8284, 2023.
A APPENDIX
A.1 IMPLEMENTATION DETAILS
We train our UC-NeRF for 40k iterations using Adam optimizer with batch size of 32384. The learning rate is logarithmically reduced from 0.008 to 0.001, with a warm-up phase consisting of 5000 iterations.
Layer-based Color Correction In UC-NeRF, we model a scene as the foreground and sky. The foreground represented by Zip-NeRF while the sky modeled by the vanilla NeRF (Mildenhall et al. (2021)). The weight of sky loss is set to 2 x 10-3 The dimension of sky latent code and foreground latent code is set to For the MLP that decodes the latent code, we use three layers with 256 hidden units. The weight of transformation regularization is set 2 x 10
Virtual Warping For virtual warping, we randomly sample 9 virtual poses for each existing pose. The occlusion needs to be considered in the case of multiple pixels of the known view warping to the same pixel in the virtual view, which is shown in the red boxes of Fig. g We resolve this conflict by taking the warped pixel with the smallest depth value. For generating the geometric consistency mask, we set target views check the depth consistency. Only pixels with depth absolute relative error within the range of 0.01 for at least neighboring views are retained. For each training batch, we sample the rays from the real and virtual images at a ratio of respectively.
Spatiotemporally Constrained Pose Refinement We use the reprojection error to optimize the camera poses. To calculate the reprojection error, the feature points need to be extracted from images. We use Superpoint ( (DeTone et al. (2018)) to detect and describe the keypoints. The keypoints are matched by mutual nearest neighbors and the confidence higher than 0.95 preserved. For each view, we match it with the subsequent ten frames captured by the same camera and the subsequent twenty frames from different cameras. Image pairs with more than 30 matching points are retained.
A.2 SPECIAL CASE FOR SPATIOTEMPORAN CONSTRAINT
In Sec. 3.4 we model the pose of kth camera at timestamp i as T = T T' Here pose refers the transformation from the camera coordinate to the world coordinate. T refers to the car's ego pose at time i. AT; refers to the transformation from the kth camera's 'ss coordinate to the ego coordinate, which is temporally consistent. During bundle adjustment, Tí and T/ are optimized as Eq. in Sec. 3.4 where -TT) expressed
13

---

Published as a conference paper at ICLR 2024
Vanilla Warping
Ocelusion-awaro Warping
Figure 9: Occlusion problem in warping.
(T)) TT TAT
(9)
AT,
T
AT
Rij ti
AR
At;
AR
At
Expressing (T')-1T)
AT,
and AT,
1
0
1
0
1
then
TT/
AR
-AR
At
R
t
ARe
AtE
1
0
1
0
(10)
AR R
AR
R
AR.
tt
ART
At
0
When the vehicle is moving straight without any rotation, R equals to the identity matrix. Thus, the Eq. 10 is simplified as:
T AR
AR
AR At. + AR ti ART At
(1I)
0
the image correspondences are not established across cameras, i.e., I k; then Eq. 11 can further simplified as:
I AR T 0
(T -T-
(12)
which suggests that the relative transformation between any two neighboring poses of the same camera & remains unaffected by Atg, thus resulting in lack of constraint the camera's 's translation At during the optimization process. This implies that the image correspondences across both cameras and timestamps ensure robust constraint on inter-camera transformation.
A.3 EXPERIMENTS
A.3.1 APPLICATION: SYNTHESIZED VIEWS FOR MONOCULAR DEPTH ESTIMATION
With the obtained 3D NeRF, we can generate additional photo-realistic images from novel view- points. The synthesized images can facilitate downstream perception tasks like monocular depth estimation. We first train VA-DepthNet, state-of-the-art monocular depth estimation model (Liu) et al. (203)). on the original real images. We then train the model by combining the original real images and the new synthesized images "VA-DepthNet*). As Tab. 4 iillustrates, the accuracy of the estimated depth is improved with such a data augmentation. Fig. 10 also shows such an operation
leads to sharper edges and more accurate predictions.
14

---

Published as a conference paper at ICLR 2024
Table 4: The accuracy of depth estimation using VA-DepthNet before and after adding rendered novel views (VA-DepthNet") for training.
Method
Abs Rel +
RMSE
51
VA-DepthNet
0.078
2.82
93.7
VA- Ae-DepthNet"
0.076
2.64
94.2
Image
VA-DepthNet
VA-DepthNet*
Image
VA-DepthNet
VA-DepthNet*
Figure 10:
Compared training VA-DepthNet (Liu et al. (2023) on the original data, augmenting it with our rendered novel views (VA-DepthNer") leads to improved depth estimation. Depth of VA-DepthNer" in the red boxes exhibits sharper edges and smoother surfaces.
A.3.2 MORE ABLATION STUDY RESULTS
Color Correction Strategies. We compare color correction with other strategies that also model the image-dependee appearance with latent codes, as done in NeRF in the wild [Martin- Brualla et al. (202T)) and Urban-NeRF (Rematas et al. (2022)) ). As shown in Fig. II] the baseline (a) exhibits noticeable color discontinuities in regions where camera views overlap (indicated by red arrows). Although NeRF in the wild (b) models image-dependent appearance through latent codes, the absence of constraints on latent codes results in the disentanglement of attributes not related the cause of color inconsistency. significant blurriness on both sides (emphasized by red boxes), along with additional texture artifacts the sky (red arrows). Urban-NeRF (c) also decodes the latent code into affine transformations to model image-dependent appearance. However, due to the absence of separate modeling for color transformations across different image regions, color discontinuities, as indicated by the red arrows, persist in the overlapping regions of the cameras. Additionally, the areas observed by different cameras are not unified into the same color space. When employing single latent code for rendering the panorama image, the color within the central region adheres closely to reality. However, severe color deviations occur in the peripheral area, such as the grass in the red box, which appears black instead of its natural color. In contrast, method addresses the color inconsistencies and ensures clear details with consistent colors. Tab. [ also demonstrates that our approach achieves the best rendering results.
a result, the panoramic rendering produced by it exhibits
(a) Baseline
(b) NeRF in the wild
(c) Urban NeRF
(d) Ours
Figure Comparison of color correction strategies for rendering images with large field of view.
15

---

Published as a conference paper at ICLR 2024
Table 5: Comparison of different strategies for color correction.
Method
PSNR
SSIM
LPIPS +
NeRF in the wild Martin- Brualla et al.
2021
25.59
0.839
0,389
Urban NeRF Rematas et al. 2022
27.89
0.849
0.378
Ours
28.15
0.851
0.374
(a) Baseline
(b) NeRF-..
(c) Ours w/o STC
(d) Ours
Figure 12: Quantitative comparison of different pose optimization strategies. We effectively climi- nate the ghosting of the pole. STC refers to the proposed spatiotemporal constraint.
Pose Refinement Strategies S-NeRF ( Xie et al. (2023) explores the performance of various NeRF algorithms that jointly optimize poses urban scenes and found that NeRF- (Wang al. (202T) yields the best results. Thus, we compare our method with NeRF-- and validate the signif- icance of the spatiotemporal constraint proposed in our paper. As demonstrated in Tab. 6 N
indeed enhances rendering results compared to the baseline which does not refine poses, and perfor- mance does not exhibit a significant change with the inclusion of the spatiotemporal constraint. In contrast, our spatiotemporally constrained pose refinement achieves a remarkable 238% improve- ment compared to N Fig- 12 clearly demonstrates how our method effectively resolves rendering artifacts. Due color variation among different cameras, NeRF- struggles achieve precise pose optimization based on the photometric error. The ghosting of the pole (b) does not change significantly compared to the baseline (a). However, when poses optimized by explicit pixel correspondences, the ghosting is noticeably reduced (c). Furthermore, with the addition of our spatiotemporal constraint, the artifact completely disappears (d).
Table 6: Ablation study on different strategies for pose refinement. NeRF- refine poses within the NeRF framework by photometric loss while we refine poses based on explicit pixel correspondences among images. STC refers to the proposed spatiotemporal constraint.
Method
PSNR
SSIM f
LPIPS
Baseline
28.09
0.851
0.374
NeRF
28.48
0.851
0.383
NeRF STC
28.51
0.852
0.383
Ours w/o STC
29.01
0.866
0.376
Ours
29.14
0.867
0.355
16

---

Published as a conference paper at ICLR 2024
Table 7: Ablation study the weight of sky loss.
Waky
PSNR t
SSIM t
LPIPS
0
27.89
0.849
0.378
0,001
28.05
0.851
0.379
0,002
28.15
0.851
0.374
0.004
27.98
0.850
0.378
Table 8: Comparison of diverse weather conditions. (Waymo Segment-100170, Waymo Segment- 150908)
Rainy SSIM
Night
Method
PSNR 1
LPIPS
PSNR 1
SSIM 1
LPIPS
Zip-NeRF[3] UC-NeRF (Ours)
27.65
0.831 0.866
0.434
30.69
0.864
0.512
30.03
0.387
31.32
0.869
0.491
Weight for Sky Loss shown in Tab. 7 we compare the performance of our UC-NeRF using different weights for sky loss. Our UC-NeRF is not very sensitive to the changes in the loss weights. Using a large weight sky loss might diminish the weight of the photometric loss, leading to slight performance decline. [0.001, 0.002) is the reasonable range for loss weight.
Results on diverse weather conditions We further validate the robustness of our method un- der night and rainy conditions. As shown in Tab. g our approach still significantly outperforms
Night
Zip-NeRF
Ours
Rainy
Zip-NeRF
15
SPEED LIMIT 15
Ours
Figure 13: 180 paranoma rendering results night and rainy conditions.
17

---

Published as a conference paper at ICLR 2024
ZipNeRF in these scenarios. As illustrated in Fig. E facts caused by color inconsistency (indicated by red arrows) and achieve better rendering of details (highlighted by the green boxes).
manage eliminate the rendering arti-
present additional rendering results in overlap regions of different cameras. The original NeRF exhibits significant color inconsistencies overlapping areas. Through color correction, we able to render images that maintain global color consistency.
More Results Layer-based Color Correction As shown in Fig 14
More Results on Spatiotemporally Constrained Pose Refinement As shown in Fig 15 the is- sue of rendering artifacts and blurriness caused by pose errors is quite common in the multi-camera setup. Based on our observations, they typically occur in the overlapping regions captured by dif- ferent cameras. This implies that these rendering artifacts result from errors in the relative transfor- mations between the cameras. By explicitly modeling the relative transformations between cameras and ensuring spatiotemporal constraints during optimization, it is evident that these problems have been significantly addressed (shown in red boxes).
w/o Color Correction
w/ Color Correction
Figure More results on color correction.
18

---

Published as a conference paper at ICLR 2024
w/o Posc Refinement
w/ Posc Refinement
Figure 15: More results on pose refinement.
A.3.3 MORE RESULTS ON WAYMO AND NUSCENES
We further present rendering results on the Waymo and NuScenes datasets, which are compared with state-of-th of-the--rtttr art methods S-NeRF [Xie et al. (2023)) and Zip-NeRF (Barron et al. (2023)) As shown in Fig. 16 [8 it evident that both S-NeRF and Zip-NeRF exhibit color inconsistencies on the sides of the mmges, where they overlap with other cameras. In contrast, we have addressed this issue through layer-based color correction. Furthermore, we achieve improved rendering quality, such as clear contours, patterns, and text, and more accurate geometry by incorporating virtual warping and spatiotemporally constrained pose refinement.
19

---

Published as a conference paper at ICLR 2024
S-NeRF
Zip-NeRF
Ours
Figure 16: Comparison of rendering results with the state-of-the-art S-NeRF and Zip-NeRF in Waymo.
20

---

Published as a conference paper at ICLR 2024
S-NeRF
Zip-NeRF
Ours
Figure 17: Comparison of the rendering results and dept maps with S-NeRF Tosi et al, (2023) (left)
and Zip-NeRF Barron et al. (2023) (middle) in NuScenes.
21

---

Published as a conference paper at ICLR 2024
(a) S-NoRF
(b) Zip-NeRF
(c) Ours
Figure 18: 180 paranoma rendering results in NuScenes.
22


Reviews:
[{"review_id": "CZnb9odop1", "reviewer": "Reviewer_oaML", "summary": "The paper proposes a neural rendering system for automotive multi-camera temporally captured data (which they should explicitly mention in title and is misleading if not mentioned), which accounts for color variation, extrinsic errors and lack of sufficient training data typically effecting previous automotive applied Nerf methods. They first handle extrinsic errors by SLAM and again do something similar to SLAM as post-processing to recalibrate the cameras across time. To handle color variation across cameras at a given time instant, an affine color correction matrix is learnt separately from sky and foreground (as they can get sky mask from previous work) and thus there are separate NeRF models. Since NeRF requires dense sampling of input images, they also propose to get novel views via a separate pre-existing MVS method and use that depth map to render novel views. These novel views are then added to the training set of images. A new NeRF optimizing function is then defined taking all these into account. The results show improvement compared to many existing SOTA methods.", "questions": "Kindly address the weakness as much as possible. I will update my review based on the rebuttal. Currently its borderline for me.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper handles a relevant problem in real life data. The proposed method looks sound. The references look adequate. The comparative results look good.", "weaknesses": "- First, I would like the title to be more specific. Its very misleading as the paper does use automotive setting (sky+fg) as necessary input. The title seems to suggest that its a generic method for uncalibrated cameras. Also all results are on automotive. Please correct it.\n\n- In Eq. 5, what is b and d. I think the authors missed defining those params.\n\n- In Eq.7, isn't it  better to reduce reprojection error as a function of direct 4x4 transformation between the cameras across time. For example directly modeling the transformation between cameras labeled T^i*delta_T1 and T^j * delta_T1 in Fig4. The reason for this is that at time t=0, its convenient to bring all the three cameras in Fig4 to the car's coordinate system, but later assuming that this transformation is fixed and won't perturb due to camera shake, bad roads, bumps etc. is an unrealistic assumption. Then propagating this incorrect assumption across time from T^i to T^j can lead to erroneous extrinsics estimation in Eq. 7.\n\n- When virtual views are created and added to the training set as discussed in Section 3.3, it has holes either due to low confidence or occlusion as shown in Fig 9, then how does the sky segmentation from Yin 2022 perform in these images. What happens to the mask value in the missing image regions in virtual view and how does in impact Equation 3.\n\n- The training strategy is not very clear. So, you train using Eq.4 upto some convergence, Then you get A and C color correction matrices. Then again you apply the corrected A and C matrices to virtual views in Section 3.3. Then you get color corrected virutal views. Then you again train the original set of spatial+temporal data but this time include virtual views? I think Section 3.5 needs more explanation because it joins all your individual modules and is critical to understand how the complete system is working.\n\n- Section 4.4 is redundant I think. It has nothing to do with the main goal of your paper and that space could have been used to explain your main parts e.g. Section 3.5 in detail.\n\n\n- In Fig5, the part of the image where the road appears to merge, the green region adjoining the bright sky appears to be hazy in the proposed result compared to Zip-Nerf results. In other words Zip-Nerf results are much sharper in that region. What could be the reason for that?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NN1bEU2xj3", "reviewer": "Reviewer_9RAi", "summary": "In this work, a novel method tailored for novel view synthesis is proposed for under-calibrated multi-view camera systems. In particular, a layer-based color correction is designed to rectify the color inconsistency in different image regions. To generate more viewpoint-diverse but color-consistent virtual views for color correction and 3D recovery, the authors further propose the virtual warping technique. And a spatiotemporally constrained pose optimization strategy is presented to explicitly model the spatial and temporal connections between cameras for pose optimization. Experiments on the Waymoand and NuScenes datasets show that this work achieves high-quality renderings with a multi-camera system and outperforms other baselines by a large margin.", "questions": "How's the time cost to filter out inaccurate depths through a geometric consistency check?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed layer-based color correction well addresses color inconsistencies in the training images, especially for those taken by different cameras.\n+ The virtual warping strategy naturally expands the range of the training views for NeRF, enhancing its effectiveness in learning both the scene's appearance and geometry.\n+ The experimental results look promising, and the proposed work significantly leads state-of-the-art methods.", "weaknesses": "- The whole pipeline seems verbose since three independent modules are stitched together with few connections. Could the proposed UC-NeRF be trained in an end-to-end manner? Additionally, the efficiency comparisons of different methods are expected to be provided in the experiments.\n- The first two contributions, i.e., Layer-based Color Correction and Virtual Warping, are kind of trivial and have limited novelty. They are constructed based on existing methods like the pretrained segmentation model, the MVS model, and a geometric consistent check approach. The procedures of these two parts perform a preprocessing-like role in the proposed method. The authors are suggested to give more clarifications and highlight their specific contributions.\n- For the color correction part, it seems that the accuracy of the correction performance highly depends on the sky segmentation. However, the cases shown in the paper only contain clean skies and sunny weather. I am wondering how this work performs under diverse weather conditions. Because this work aims at multi-camera systems that are widely used in outdoor scenes (such as autonomous driving), the real-world application would be preferred over the method itself.\n- For the proposed Spatiotemporally Constrained Pose Refinement, please clarify its relationship and difference to the bundle adjustment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zM4WNDgeaL", "reviewer": "Reviewer_5Tdv", "summary": "The paper introduces UC-NeRF, a novel approach designed specifically for under-calibrated multi-view camera systems, addressing the challenges faced when applying NeRF techniques in such setups. The method incorporates layer-based color correction, virtual warping, and spatiotemporally constrained pose refinement to achieve exceptional performance in novel view synthesis and enhance the sensing capabilities of multi-camera systems. The contributions of the paper encompass the introduction of a new dataset tailored for under-calibrated multi-view camera systems, a novel layer-based color correction method, and an algorithm for spatiotemporally constrained pose refinement. The effectiveness of UC-NeRF is demonstrated through experiments conducted on the new dataset, and comparisons are made against state-of-the-art methods.", "questions": "Please check the weaknesses listed above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. This paper is well-written and easy to follow. \nS2. The proposed method is technically sound. \nS3. The experiment design especially the ablation study is solid and the results are noticeable.", "weaknesses": "W1. The novelty of this paper is somewhat limited to me:\nW1-1. In terms of the first key innovation, namely layer-based color correction, why we can not use some classical multiple views color correction solutions in the structure-from-motion field as a pre-processing step instead of a module inside the NeRF? It should be justified. Besides, some existing NeRFs also addresses similar problem such as RAWNeRF and block-NeRF, what are the main differences between the proposed method and these works? \nW1-2 In terms of the spatiotemporally constrained pose refinement, there are some similar NeRFs that also consider the spatial and temporal connections between cameras for pose optimization. Name a few but not completed lists such as BARF (Lin et al. ICCV 2021) and BAD-Nerf (Wang et al. CVPR 2023). What is the novelty of the proposed method over these works? \n\nW2. The experiment comparisons are limited since only Mip-NeRF was used. Why not compare to some large-scale NeRDs such as block-NeRF or multi-views NeRFs such as MC-NeRF and NeRF-MS. The authors should justify the reason.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wMF7wlLWhZ", "reviewer": "Reviewer_oEhA", "summary": "The paper presents UC-NeRF, a method for new view image synthesis in multicamera systems. They introduce models for color correction, virtual warping, and pose refinement to improve upon the results of Zip-NeRF and NeRF. Each of these operations defines a loss function L_{sky}, L_{reg}, and L_{rpj}. The results seem to suggest that they are achieving state-of-the-art results. Yet, the code to verify this claim is not available. Further details may be needed to implement their ideas completely.", "questions": "In 3.5, I understand that UC-NeRF is NeRF trained on the original NeRF’s photometric loss and L_{sky} and L_{reg}, but you are also using L_{rpj}, correct? Is the total loss the sum of the individual losses? Are there weights on the losses before adding them?\n\nWhat is mathbf{b} and mathbf{d} in (5)?\n\nDefine d_v and d_o in (6)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper presents state-of-the-art results. They benchmark the performance of UC-NeRF with several other strategies that have been recently introduced. Their ablation study suggests that each term in the loss function improves the results.", "weaknesses": "It would be great if the authors could share their code; even promising to share upon acceptance will be understandable. NeRF code is readily available. Otherwise, the authors should increase the clarity of their presentation to explain how their ideas could be implemented and the results reproduced for verification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wP6pNbKC1c", "reviewer": "Reviewer_ehJ8", "summary": "This work proposed UC-NeRF, a method for novel view synthesis in under-calibrated multi-view camera systems. The authors propose a three-step approach to address these challenges:\n\n1. Layer-based color correction: This step rectifies the color inconsistency in different image regions by applying color correction to each layer of the image pyramid.\n\n2. Virtual warping: This step generates more viewpoint-diverse but color-consistent virtual views for color correction and 3D recovery. The authors show that virtual warping benefits color correction and edge sharpness.\n\n3. Spatiotemporally constrained pose refinement: This step is designed for more robust and accurate pose calibration in multi-camera systems. The authors demonstrate that this step improves the accuracy of depth estimation in large-scale outdoor scenes.\n\nThe paper includes experimental results on several datasets and comparisons with other methods. The authors show that UC-NeRF achieves state-of-the-art performance in novel view synthesis and improves the sensing capabilities of multi-camera systems.", "questions": "- The pose refinement step relies on keypoints, which could be a shortcoming. Did the author consider DBARF (CVPR 2023) and FlowCam (NeurIPS 2023), which jointly optimize consecutive camera poses and NeRF?\nActually, due to the vibrations during driving, the relative camera poses in a rig could change. I think the author mentioned it in the introduction, but the case is not handled in the formulation (Eq. (7)).\n\n- Did you reimplement Zip-NeRF or use others' reimplementation of Zip-NeRF? If it is the latter case, the URL should be provided since Zip-NeRF does not release its code.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The problem setting is interesting. The proposed method can be an enhancement of current NeRF techniques.\n\n- The proposed method is sound and effective, regardless of its simplicity - complexity is not a criterion for us to judge whether a paper is good or not.\n\n- The authors did exhaustive experiments to show the effectiveness of the proposed method. Ablation studies also show the effectiveness of each module.", "weaknesses": "- I think the introduction of this paper is not well written. It took me some time to understand why this work needs virtual warping and what are the differences between single-camera NeRF and multi-camera NeRF (since each camera in a multi-camera system can be deemed as a single camera).\n\n- The virtual warping step relies on the MVS method to generate dense depth maps, which may not generalize to street views (I'm not certain about this) and may need further pertaining.\n\n- In Eq.(5), the author does not explain what is $\\mathbf{b}$ and $\\mathbf{d}$ denote.\n\n- In Eq. (7), it is unclear whether the relative transformation $\\Delta \\mathbf{T}_k$ is optimized.\n\n- The final training loss is missing, e.g. $\\mathcal{L} = \\mathcal{L}_{\\text{pho}} + \\lambda_1 \\mathcal{L}_{\\text{reg}} +  \\lambda_1 \\lambda_2 \\mathcal{L}_{\\text{rpj}} $. \n\n- The pose refinement step is quite straightforward. Since the relative pose constraints $\\Delta \\mathbf{T}$ in the same rig can be obtained through calibration, I think it is naive to decompose the camera pose into the ego pose and a relative transformation $\\Delta \\mathbf{T}$. Moreover, the pose refinement step requires point correspondences, which could introduce outliers since it is well known that SOTA point matching methods are prone to repetitive structures and moving objects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)