Neural Radiance Fields with Geometric Consistency for Few-Shot Novel View Synthesis
Authors: Jiuhn Song
Min-Seop Kwak
Seungryong Kim
Venue: Submitted to ICLR 2023 (2023)
Keywords: NeRF
3D Computer Vision

Abstract: We present a novel method to regularizes neural radiance field (NeRF) in few-shot setting with geometry-based consistency regularization. The proposed approach leverages NeRF's rendered depth map to warp source images to unobserved viewpoints and impose them as pseudo ground truths to facilitate learning of detailed features. By encouraging consistency at feature-level instead of using pixel-level reconstruction loss, we regularize the network solely at semantic and structural levels while allowing view-dependent radiance to model freely after color variations. Our application of proposed consistency term for the network is twofold: between and observed and unobserved viewpoints, image rendered at unseen view is forced to model after the image warped from input observation, while between observed viewpoints the warped image undergoes optimization for geometry-specific regularization. We also demonstrate an effective method to filter out erroneous warped solutions, along with relevant techniques to stabilize training during optimization. We show that our model achieves competitive results compared to concurrent few-shot NeRF models.

Under review as a conference paper at ICLR 2023
GECONERF: FEW-SHOT NEURAL RADIANCE FIELDS VIA GEOMETRIC CONSISTENCY
Anonymous authors Paper under double-bling review
ABSTRACT
We present novel framework regularize Neural Radiance Field (NeRF) in few-shot setting with geometry-aware consistency regularization. The proposed approach leverages a rendered depth map at unobserved viewpoint warp sparse input images to the unobserved viewpoint and impose them as pseudo ground truths facilitate learning NeRF. By encouraging such geometry-awane consistency at a feature-level instead of using pixel-Ievel reconstruction loss, we regularize the NeRF semantic and structural levels while allowing for modeling view- dependent radiance to account for color variations across viewpoints. We also propose an effective method to filter out erroneous warped solutions, along with training strategies to stabilize training during optimization. We show that our model achicves competitive results compared to state-of-the-ar few-shot NeRF models.
INTRODUCTION
Recently, representing a 3D scene as a Neural Radiance Field (NeRF) Mildenhall et al. (2020) has proven to be powerful approach for novel view synthesis and 3D reconstruction Barron al. (202T): Jain et al. (20TT); Chen et al. (02T)). However, despite its impressive performance, NeRF requires large number of densely, well distributed calibrated images for optimization, which limits its applicability. When limited to sparse observations, NeRF easily overfits to the input view images and is unable reconstruct correct geometry Zhang al. (2020).
The task that directly addresses this problem, also called few-shot NeRF, aims optimize high- fidelity neural radiance field such sparse scenarios Jain et al. (202T): Kim al. (2022): Niemeyer et al. 2022). countering the underconstrained nature of said problem by introducing additional priors. Specifically, previous works attempted to solve this by utilizing a semantic feature Jain et al.
(202 entropy minimization Kim al. (022), SIM depth priors Deng et al. (2022) or normalizing flow Niemeyer et al. 2022), but their necessity for handcrafted methods or inability to extract local and fine structures limited their performance.
To alleviate these issues, we propose a novel regularization technique that enforces geometric consistency across different views with depth-guided warping and a geometry-aware consistency modeling. Based on these, we propose a novel framework, called Neural Radiance Fields with Geometric Consistency (GeCoNeRF), for training neural radiance fields in a few-shot setting. Our key insight is that we can leverage depth rendered by NeRF to warp sparse input images to novel viewpoints, and use them as pseudo ground truths to facilitate learning of fine details and high- frequency features by NeRF. By encouraging images rendered at novel views to model warped images with consistency loss, we can successfully constrain both geometry and appearance to boost fidelity of neural radiance fields even in highly under-constrainee few-shot setting. Taking into consideration aon-Lambertian nature of given datasets, we propose a feature- regularization loss that captures contextual and structural information while allowing for modeling view-dependent color differences. We also present a method to generate a consistency mask to prevent inconsistently warped information from harming the network. Finally, we provide coarse-to- fone training strategies for sampling and pose generation to stabilize optimization the model.
We demonstrate the effectiveness of our method synthetic and real datasets Mildenhall al. (2020); et al. (2014). Experimental results prove the effectiveness of the proposed model over the latest methods for few-shot novel view synthesis.

---

Under review as a conference paper at ICLR 2023
Inverse Warping
Feature Extractor fø
Gradient Backgroppgation
Ground Truth
Consistency Modeling Loss
Field yr-
Occlusion Mask Af
Unseen View Warped /t-s
Seen Depth Rendered D,
Unseen Depth Rendered
Unseen View
Pase Offfrrnce Ri-s)
Rendered
Mask
Generation
Figure 1: Illustration of our consistency modeling pipeline for few-shot NeRF. Given an image and estimated depth map D, of j-th unobserved viewpoint, we warp the image 1; to that novel viewpoint as ii by establishing geometric correspondence between two viewpoints. Using the warped image as a pseudo ground truth, we cause rendered image of unseen viewpoint, Ij. to be consistent in structure with warped image, with occlusions taken into consideration.
RELATED WORK
Neural radiance fields. Among the most notable of approaches regarding the task of novel view synthesis and 3D reconstruction is Neural Radiance Field (NeRF) Mildenhall et al. (2020), where photo-realistic images are rendered by simple MI.P architecture. Sparked by its impressive performance, a variety of follow-up studies based on its continuous neural volumetric representation have been prompted, including dynamic and deformable scenes Park et al. (202I): Tretschk et al. (2021) Pumarola et al. (2021); y Attal et al. 22221, real-time rendering Yu et al.
(202 lay: Hedman et al.
(2021); Reiser et al. (2021); Müller et al. 2022 self-calibration Jeong et al. (2021) and generative
modeling Schwarz et al. 2020); Niemeyer & Geiger (2021); Xu al. (2021); Deng al. (2021).
Mar-NeRFBBrrron et al. (2021) eliminates aliasing artifacts by adopting cone tracing with a single multi-scale MLP. In general, most of these works have difficulty in optimizing single scene with few number of images.
Few-shot NeRE One key limitation of NeRF its necessity for large number of calibrated views in optimizing neural radiance fields. Some recent works attempted to address this in the case where only few observed views of the scene are available. PixeINeRI Yu et al. (202 conditions a NeRF on image inputs using local CNN features. This conditional model allows the network to learn scene priors across multiple scenes. Stereo radiance fields Chibane et al. (202T) use local CNN features from input views for scene geometry reasoning and MVSNeRF Chen et al. 2222 combines cost volume with neural radiance field for improved performance. However, pre-traning with multi-view images of numerous scenes essential for these methods for them to learn reconstruction priors.
Other works attempt different approach optimizing NeRF from scratch in few-shot settings: DSNeRF al. (2022) makes use of depth supervision to network to optimize scene with few images. RReesle et al. (2021) also utilizes sparse depth prior by extending into dense depth map by depth completion module guide network optimization. On the other hand, there models that tackle depth prior-free few-shot optimization: DietNeRF Jain et al. (202I) enforces semantic consistency between rendered images from unseen view and seen images. RegNeRF Niemeyer et al. (2022) regularizes the geometry and appearance of patches rendered from unobserved viewpoints.
InfoNeRF Kim al. (2022) constrains the density's entropy each ray and ensures consistency across rays in the neighborhood. While these methods constrain NeRF into learning more realistic geometry, their regularizations are limited in that they require extensive dataset-specific fine-tuning and that they only provide regularization at global level in a generalized manner. Improving upon above works, our method tackles prior-free few-shot optimization without using any depth priors, achieving more local and scene-specific regularization with warping-basee consistency modeling.
Self-supervised photometric consistency. In the field of multiview stereo depth estimation, con- sistency modeling between stereo images and their warped images has been widely used for self-
2

---

Under review as a conference paper at ICLR 2023
supervised training Godard et al. (2017 ; Garg al. (2016); Zhou al. (2017) In weakly supervised or unsupervised settings Huang et al. (2021 1 Khot et al. (2019) where there is lack of ground truth
depth information, consistency modeling between images with geometry-based warping is used as
a supervisory signal Zhou et al. (2017): Huang et al. (02T); Khot et al. (2019) formulating depth
learning as a form of reconstruction task between viewpoints.
Recently, methods utilizing self-supervised photometric consistency have been introduced to NeRF: concurrent works such NeuralWarp Darmon et al. (2022), StructNeRF Chen al. (2022) and Geo-NeuS Fu et al. 2022) model photometric consistency between source images and their warped counterparts from other source viewpoints improve their reconstruction quality. However, these methods only discuss dense view input scenarios where pose differences between source viewpoints are small, and not address their behavior few-shot settings where sharp performance drop is expected due to scarcity of input viewpoints and increased difficulty in the warping procedure owing to large viewpoint differences and heavy self-occlusions. RapNeRF geometry-baseee reprojection method to enhance view extrapolation performance, and Bortolon et al. (2022) uses depth rendered by NeRF as correspondence information for view-morphing module to synthesize images between input viewpoints. However, these methods do not take occlusions into account, and their pixel-level photometric consistency modeling comes with downside suppressing view-dependent specular effects.
al. (2022) uses
PRELIMINARIES
Neural Radiance Field (NeRF) Mildenhall et al. (2020) represents scene as a continuous function fe represented by a neural network with parameters 0, where the points are sampled along rays, represented by for evaluation by the neural network. Typically, the sampled coordinates and view direction d E IR. are transformed by a positional encoding y into Fourier features Tancik et al. (2020) that facilitates learning of high-frequency details. The neural network fo takes input the transformed coordinate (xx) and viewing directions (dd) and outputs a view-invariant density value a E E and view-dependent color value с E IR such that
R3
{c, } = fo()(x), (d)).
(1)
With ray parametrized as (t) = + tdp from the camera center o through the pixel along direction dp. the color is rendered as follows:
C(rp) L"d ((t))((,((t))((((((()) d,),
(2)
where T(t) exp (epp-// (-f,, (-f(o(rp(s))ds
where C(rp) is a predicted color value at the pixel along the ray (t)() from t, to ty, and ((t) denotes accumulated transmittance along the ray from fn t. To optimize the networks fe- the observation loss Lobs enforces the rendered color values to be consistent with ground truth color value Cr(r):
Lobs Σ - c(r,)l
(3)
E,ER
where R represents batch of training rays.
METHODOLOGY
4.1 MOTIVATION AND OVERVIEW
Let us denote image at i-th viewpoint as 7, In few-shot novel view synthesis, NeRF is given only a few images } for E [1, N} with small N, e.g., novel view synthesis is to train the mapping function fo that can be used to recover an image I, at j-th unseen or novel viewpoint. As we described above, in the few-shot setting, given {I,), directly optimizing fo solely with the pixel-wise reconstruction loss Lobs is limited by its inability to model view-dependent effects, and thus an additional regularization to encourage the network fo to generate consistent appearance and geometry is required.
or N = 5. The objective of
3

---

Under review as a conference paper at ICLR 2023
Nelt MIP
4
Teature Extrctor f
fe
La
Vieupaint
Dj
Warping
R(-1
Dj
Lrgg
Seen viewpoint/
Ri-j
Dj
Insers Vaarping
Feature f
NutF MLP fa
cons
Unseen
vieapoint j
Figure 2: Illustration of the proposed framework, GeCoNeRF regularizes the networks with consistency modeling. Consistency loss function C is applied between unobserved viewpoint image and warped observed viewpoint image, while disparity regularization loss freg regularizes depth at seen viewpoints.
To achieve this, we propose a novel regularization technique to enforce geometric consistency across different views with depth-guided warping and consistency modeling. We focus on the fact that MeRF/Mildenhaal
al. (2020) inherently renders not only color image but depth image as well. Combined with known viewpoint difference, the rendered depths can be used to define geometric correspondence relationship between two arbitrary views.
Specifically, consider depth image rendered by the NeRF model, Dj at unseen viewpoint j. By formulating a warping function (I;;; Dj. Rı-s)) that warps an image I, according to the encourage consistency between warped image T= = (T; ; Dj, Re-z)) and rendered image 7, at j-th unseen viewpoint, which in turn improves the few-shot novel view synthesis performance. This framework can overcome the limitations of previous few-shot setting approaches Mildenhall et al. (2020); Chen et al. (202I); Barron et al. (2021). improving not only global geometry but also figg-frreeency details and appearance well.
depth D, and viewpoint difference Rı-zj.
In the following, we first explain how input images can be warped to unseen viewpoints in our framework. Then, demonstrate how impose consistency upon the pair of warped image and rendered image for regularization, followed by explanation of occlusion handling method and several training strategies that proved crucial for stabilization NeRF optimization in few-shot scenario.
4.2 RENDERED SEPTH-GUIDER WARPING
To render an image at novel viewpoints, we first sample random camera viewpoint, from which corresponding ray vectors are generated in a patch-wise manner. As NeRF outputs density and color values of sampled points along the novel rays, we use recovered density values render consistent depth map. Following Mildenhall et al.] (2020), we formulate per-ray depth values as weighted composition of distances traveled from origin. Since ray rp corresponding to pixel p parameterized as (t) = o tdp, the depth rendering is defined similarly to the color rendering:
D(rp) = D(rp)==/-
[(t)a(r,(t))td;
4
where D(rp) is predicted depth along the ray rp. As described in Figure we use the rendered depth map Dj to warp input ground truth image 7, to j-th unseen viewpoint and acquire a warped image Ti-ij. which is defined a process such that Tı=;j (T;;; Dj, Ra-z))- More specifically, pixel location pý in target unseen viewpoint image is transformed to py-+i at source viewpoint image by viewpoint difference Ry and camera intrinsic parameter К such that
PJ-+ M...RR, (D)
where indicates approximate equality and the projected coordinate Pj-ı is a continuous value. With a differentiable sampler, we extract color values of Pー- on I, More formally, the transforming components process can be written as follows:
Tp-5(P;) = sampler (;
where sampler(:) is a bilinear sampling operator Daderberg al. 2015
4

---

Under review as a conference paper at ICLR 2023
(a) GT patch
(b) Rendered patch
(c) Warped patch
(d) Occlusion mask (c) Masked patch
Figure 3: Visualization of consistency modeling process. (a) ground truth patch, (b) rendered patch
novel viewpoint, (c) warped patch, from input viewpoint to novel viewpoint, (d) occlusion mask with threshold masking, and (e) final warped patch with occlusion masking at novel viewpoint.
Acceleration. Rendering full image with NeRF voluemtric rendering is computationally heavy and extremely timetaking, requiring tens seconds for single iteration. To overcome the computational bottleneck of full image rendering and warping, rays are sampled on a strided grid to make the patch with stride 8, which we have set as After the rays undergo volumetric rendering, we upsample the low-resolution depth map back to original resolution with bilinear interpolation. This full-resolution depth map is used for the inverse warping. This way, detailed warped patches full-resolution can be generated with only fraction of computational cost that would be required when rendering the original sized ray batch.
4.3 CONSISTENCY MODELING
Given the rendered patch 1 at j-th viewpoint and the warped patch Ti-s, with depth D, and viewpoint difference Rı-jj we define the consistency between the two encourage additional regularization for globally consistent rendering. One viable option is to naïvely apply the pixel-wise image reconstruction loss Lpxx such that
Spix = 1| 1
However, we observe that this simple strategy prone to cause failures in reflectant non-Lambertian surfaces where appearance changes greatly regarding viewpoints Zhan et al.) (2018). In addition, geometry-related problems, such as self-occlusion and artifacts, prohibits naive usage of pixel-wise image reconstruction loss for regularization in unseen viewpoints.
Feature-level consistency modeling. To overcome these issues, we propose masked feature-level regularization loss that encourages structural consistency while ignoring view-dependent radiance effects, as illustrated in Figure??
Given an image I as an input, we use convolutional network extract multi-level feature maps such that (T) E # with channel depth C; for I-th layer. To measure feature-level consistency between warped image Tı-ク and rendered image [,, we extract their features maps from I, layers and compute difference within each feature map pairs that are extracted from the same layer.
In accordance with the idea of using the warped image T,-, as pseudo ground truths, we allow a gradient backpropagation to pass only through the rendered image and block it for the warped image. By applying the consistency loss at multiple levels of feature maps, we cause Ij to model after I both semantic and structural level.
Formally written, the consistency loss Lcons is defined as such that,
Lcons Σ 1=
1 - f(a1l
(8)
For this loss function Lcons, we find 1-1 distance function most suited for our task and utilize it to measure consistency across feature difference maps. Empirically, have discovered that VGG-19 network[Simonyan (Sioonyan & Zisserman| (2014) yields best performance in modeling consistencies, likely due the absence of normalization layers Johnson al. (2016) that scale down absolute values of feature differences. Therefore, we employ VGG19 network as our feature extractor network fo throughout all of our models.
should be noted that our loss function differs from that of DietNeRF Jain et al. 2202 in that while Diet consistency loss is limited to regularizing the radiance field in a globally semantic level,
5

---

Under review as a conference paper at ICLR 2023
our loss combined with warping module is also able give the network highly rich information local, structural level as well. In other words, contrary to DietNeRF giving only high-level feature consistency, method of using multiple levels of convolutional network for feature difference calculation can be interpreted as enforcing mixture of all levels, from high-level semantic consistency to low-level structural consistency.
Occlusion handling. In order prevent im- perfect and distorted warpings caused by erro- ncous geometry from influencing the model, which degrade overall reconstruction quality, we construct consistency mask Mj to let NeRF ignore regions with geometric inconsistencies, as demonstrated Figure 3 Instead of apply- ing mask to the images before inputting them into feature extractor network, we apply resized masks Mj directly to the feature maps, after ing nearest-neige neightr samplinn to make them match the dimensions of 1-th layer out- puts.
Rendered Surface
(b)
Artifacts
Seen View
Unseen View /
I-) Mi
Ij
Figure Occlusion-aware mask generation. Mask generation by comparing geometry between novel view j and source view i, with Li-s being warped patch generated for view j. For (a) and (b), warping does not occur correctly due to artifacts and self-occlusion, respectively. Such pixels are masked out by Mr, allowing only (c), with accurate warping, training signal for rendered image Ij-
We generate M by measuring consistency be- tween rendered depth values from target view- point and source viewpoint such that
((p)) (p,) - D.(P)-0)Il <T]. (9)
where [-] is Iverson bracket, and Pj-- refers to the corresponding pixel in source viewpoint i for reprojected target pixel p, of j-th viewpoint. Here measure euclidean distance between depth points rendered from target and source viewpoints as a criterion for a threshold masking. As illustrated in Figure if distance between two points are greater than given threshold value т, we determine and mask out the corresponding pixel in viewpoint 7,- The process takes place over every pixel in viewpoint 1j to generate a mask M the same size as rendered pixels. Through this technique, filter out problematic solutions at feature level and regularize NeRF with only high-confidence image features.
rays as rendering depths of separate surfaces
Based on this, the consistency loss Ccons is extended as such that
1 Cim
M cons
|| (f(()))))
(10)
1=1
where my is the sum of non-zero values.
Edge-aware disparity regularization. Since our method is dependent upon the quality of depth rendered by NeRF, we directly impose additional regularization on rendered depth to facilitate optimization. We further encourage local depth smoothness on rendered scenes by imposing I-1 penalty on disparity gradient within randomly sampled patches of input views. In addition, inspired by Godard al. (2017), we take into account the fact that depth discontinuities in depth maps are likely to be aligned to gradients of its color image, and introduce an edge-aware term with image gradients ØI to weight the disparity values. Specifically, following (Goaadd et al. (2017), we regularize for edge-aware depth smoothness such that
Creg
10.4. + |0, 18,411
(11)
where D
D:/D, the mean- mean-normalized inverse depth from Godard al. (017)) to discourage shrinking of the estimated depth.
4.4 TRAINING STRATEGY
In this section, we present novel training strategies to learn the model with the proposed losses. Total losses. We optimize our model with a combined final loss of original NeRF's pixel-wise reconstruction loss Lobs and two types of regularization loss, cons for unobserved view consistency modeling and Creg for disparity regularization.

---

Under review as a conference paper at ICLR 2023
(a) GT
(b) DietNeRF
(c) InfoNeRF
(d) RegNeRF
(e) Ours
(f) Ours (D)
Figure 5: Qualitative comparison on NeRF -eRt-Synthetic Mildenhall al. (2020) show that in 3-view setting, our method captures fine details more robustly (such as the wire in the mic scene) and produces less artifacts (background in the materials scene) compared to previous methods. We show GeCoNeRF's results (e) with its rendered depth (f).
Progressive camera pose generation. Difficulty of of accurate warping increases the further target view is from the source view, which means that sampling far camera poses straight from the beginning of training may have negative effects on our model. Therefore, we first generate camera poses near source views, then progressively further as training proceeds. We sample noise value uniformly between an interval of [-3, +38 and add it to the original Euler rotation angles of input view poses, with parameter 3 growing linearly from
degrees throughout the course of optimization. This design choice can be intuitively understood as stabilizing locations near observed viewpoints at start and propagating this regularization to further locations, where warping becomes progressingly more difficult.
Positional encoding frequency annealing. We find that most of the artifacts occurring are high- frequency occlusions that fill the space between scene and camera. This behaviour can be effectively suppressed by constraining the order of fourier positional encoding Tancik et al. (2020) to low dimensions. Due to this reason, we adopt coarse-to-fine frequency annealing strategy previously used by Park et al. al. (202T) to regularize our optimization. This strategy forces our network to primarily optimize from coarse, low-frequency details where self-occlusions and fine features are minimized, easing the difficulty of warping process in the beginning stages of training. Following Park et al. (2021). the annealing equation is axt)= = mt/ K, with m as the number of encoding frequencies, as iteration step, and we set papper-rrramter K as 15k.
5
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Baselines. We use mip-NeRF Barron al. (2021) backbone. We give comparisons to the baseline and several state-of of-te-of-the-an art models for few-shot NeRF: InfoNeRF Kim et al. (2022).
DietNeRFJain al. (2021), and RegNeRF ReegNeRFNiemeyer
al. (2022).
Datasets and metrics. We evaluate our model on Nent-Synthetic Mildenhall et al. (2020) and LLFF|Mildenbal
al. (2019). NeRF-Synthetic is a realistically rendered 360" synthetic dataset comprised of 8 scenes. We randomly sample 3 viewpoints out of 100 training images in each scene, with 200 testing images for evaluation. We also conduct experiments on LLFF benchmark dataset, which consists of real-life forward facing scenes. Following RegNeRF Niemeyrr et al. (2022), apply standard settings by selecting test set evenly from list of every 8th image and selecting reference views from remaining images. We quantify novel view synthesis quality using PSNR, Structural Similarity Index Measure (SSIM) Wang et al. (2004), LPIPS perceptual metric Zhang al. 2018) and "average" error metric introduced in Barron al. 2021 metrics for all scenes in each dataset.
report the mean value of
Implementation details. Our main model is built on top of the JAX mip-NeRF codebase Barron al. (222)). We use Adam optimizer using exponential learning rate decay. Our model is trained for 60k iterations for hours on two NVIDIA RTX3090Ti GPUs. We provide more implementation details in supplementary materials.
7

---

Under review as a conference paper at ICLR 2023
Table 1: Quantitative comparison on NeRF-Synthetic Miilennaall et al. 2020) and LLFF (Mildenhall et al. 2019) datasets.
NeRF-Syuthetic (Mildenhall al.
2020
LLFF
(Mildenhall ct al. 2019)
Methods
PSNR 1 SSIM T
LPIPS 1
Avg.
PSNR
SSIM
LPIPS
Avg. :
NeRF [Mildenhall et al.
2020
14.73
0.734
0.451
0.199
13.34
0.373
0.451
0.255
Mip-NNKFFTT [Barron et al 2021
17.71
0.798
0.745
0.178
14.62
0.351
0.495
0.246
DietNeRF Jain of al. 2021
16.06
0.795
0,306
0.131
14,94
0.370
0.496
0.232
InfoNeRF (Km ara
18.65
0.811
0.230
0.111
14.37
0.349
0.457
0.238
RegNeRF Niemeyer et al. 2022
18.01
0.842
0.747
0.167
19.08
0.587
0.336
0.146
GeCoNeRF (Ours)
19.23
0.866
0.723
0.148
18.55
0.578
0.340
0.150
(a) Ground truth
(b) mip-NeRF
(c) mip-NeRF (D)
(d) Ours
(e) Ours (D)
Figure 6: Qualitative results on LLFF Mildenhall et (011)) Comparison with baseline mip-
NeRF shows that our model learns of coherent depth and geometry in extremely sparse 3-view setting.
5.2 COMPARISONS
Qualitative comparisons. Qualitative comparison results in Figure 5 and 6 demonstrate that our model shows superior performance to baseline mip-NeRF Barron et al. (202T) and previous state-of-the-art model, RegNeRF Niemeyer RegneeF et al.) (2022), in 3-view settings. We observe that our warping-based consistency enables GeCoNeRF to capture fine details that mip-NeRF and RegNeRF struggle to capture in same sparse view scenarios, as demonstrated with the mic scene. Our method also displays higher stability in rendering smooth surfaces and reducing artifacts background in comparison previous models, as shown in the results of the materials scene. We argue that these results demonstrate how our method, through generation of warped pseudo ground truth patches, is able to give the model local, scene-specific regularization that aids recovery of fine details, which previous few-shot NeRF models with their global, generalized priors were unable to accomplish.
Quantitative comparisons. Comparisons in Table T shows our model's competitive results in LLFF dataset, whose PSNR results show large increase in comparison to mip-NeRF baseline and competitive compared RegNeRF. We see that our warping-based consistency modeling successfully prevents overfitting and artifacts, which allows our model to perform better quantitatively.
5.3 ABLATION STUDY
Table 2: Ablation study.
We validate our design choices by perform- ing an ablation study on LLFF Mildenhal]
Components
PSNR
SSIM
LPIPS|
Avg..l
(a) Bascline
14.62
0.351
0.495
0.246
et al.19) dataset. Quantitative and qualita-
(b) (a) Leuns
18.10
0.529
0.408
0.164
tive results are given Table 2 and Figure 7 respectively.
(c) (b) + Ad (O. mask)
18.24
0.535
0.379
0.159
(d) (c) + Progressive
18.46
0.552
0.349
0.151
(d) Lieg (Ours)
18.55
0.578
0.340
0.150
Feature-level consistency loss. We ob- serve that without the consistency loss Lcons: our model suffers both quantitative and qualitative
decrease reconstruction fidelity, verified by incoherent geometry in image (a) of Figure 7 Absence of unseen view consistency modeling destabilizes the model, resulting divergent behaviours such as artifact generation in empty space.
Occlusion mask. We observe that addition of occlusion mask M improves overall appearance as well as geometry, as shown in image (c) Figure Its absence results broken geometry throughout the overall scene, demonstrated in (b). Erroncous artifacts pertaining projections from different viewpoints were detected in multiple scenes, resulting lower quantitative values.
8

---

Under review as a conference paper at ICLR 2023
(a) Baseline
(b) (a) + Leons
(c) (b) M (O. mask) (d) (c) + Progressive (e) (d) Lreg (Ours)
Figure Qualitative ablation. Our qualitative ablation results Horns scene shows the contribution of each module in performance of our model at 3-view scenario.
Progressive training strategies. In Table 3, we justify our progressive training strategies with additional experiments on NeRF-Synthetic dataset, while the main ablation we conduct ablation with progressive annealing only. For pose generation, we sample pose angle from large interval in the beginning, instead of slowly growing the interval. For positional encoding, we replace progressive annealing with naïve po- sitional encoding used in NeRF, We observe that their absence causes destabilization of the model and degradation in appearance, respectively.
Table 3: Progressive training ablation.
Components
PSNRT
SSIM
LPIPS,
Avg.
w/o prog. anneal
18.50
0.852
0.781
0.161
w/o prog. pose
16.96
0.799
0.811
0.194
w/o both
17.04
0.788
0.823
0.197
GeCuNeRF (Ours)
19.23
0.866
0.723
0.148
Table 4: Pixel-level consistency ablation.
Components
PSNR
SSIM
LPIPS
Avg.
w/ Lpix w/ Levas
17.98
0.528
0.431
0.165
18.55
0.578
0.340
0.150
Feature-level loss vs. pixel-level loss. In Table 4, we con- duct a quantitative ablation comparisons between feature- level consistency loss Lcons and pixel-level photometric con- sistency loss fpix: both with occlusion masking. As shown in Figure 8 naïvely applying pixel-level loss for consistency modeling Teads to broken geometry. This phenomenon can be attributed to Lpix being agnostic view- fie-ddependent spec- ular effects, which the network tries to model by altering or erasing altogether non-Lambertian surfaces.
Pixel-level
(b) Feature-level
Figure 8: Lpix vs. Loons comparison.
5.4 COMPARISON CONSISTENCY MODELING BETWEEN KNOWN VIEWS
In order to compare GeCoNeRF with contemporary meth- ods Darmon et al. (2022); Chen et al. (2022); Fu et al. (2022) that model consistency between known views, we conduct an experiment to observe how such consistency modeling performs few-shot NeRF setting. In our experiment, we replace our consistency modeling with above setting, warping source im- ages to other known views for consistency between the warped image and ground truth (reference) image.
(a) Between known views
Its result, shown in (a) of Figure 9 displays divergent be haviours such as heavy artifact generation, while method (b) succeeds in recovering detailed geometry of the scene under the same setting. As discussed in Section 2] we argue that large view differences and scarcity of reference images make it difficult for NeRF refine geometry with consistency mod- eling between known views. Our work's 's novel contributions allow consistency modeling be adopted few-shot NeRF to facilitate stable training under such extreme conditions, distinguishing our work from above methods.
(b) GeCoNeRF (Ours)
Figure 9: Consistency between known views vs. method.
CONCLUSION
We present GeCo-NeRF, novel few-shot NeRF regularization method. We regularize geometry by modeling feature-level consistency at unobserved viewpoints between using the warped images, regularizing NeRF for learning of robust geometry. Further techniques and training strategies propose prove to have stabilizing effect and facilitate optimization of our network. Our experimental evaluation demonstrates method's competitiveness in regards to other otate-of-the-ar models.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Benjamin Attal, Eliot Laidlaw, Aaron Gokaslan, Changil Kim, Christian Richardt, James Tompkin, and Matthew O'Toole. Törf: Time-of-fligh radiance fields for dynamic scene view synthesis. Advances in neural information processing systems, 34, 2021.
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin Bariina, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2021.
Matteo Bortolon, Alessio Del Bue, and Fabio Poiesi. Data augmentation for nerf: geometric consistent solution based on view morphing, 2022. URL
04214
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pp. 14124-14133, 2021.
Zhengjie Chen, Chen Wang. Yuanchen Guo, and Song-Hal Hai Zhang. Structnerf: Neural radiance fields for indoor scenes with structural hints. ArXiv, (ttp/2229.052777 2022.
Julian Chibane, Aayush Bansal, Veriea Lazova, and Gerard Pons-Moll. Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 791 1-7920, 2021.
François Darmon, Bénédicte Bascle, Jean-Clément Devaux, Pascal Monasse, and Mathieu Aubry. Improving neural implicit surfaces geometry with patch warping. 2022.
Kangle Deng, Andrew Liu, Jun- Yan Zhu, and Deva Ramanan. Depth-supervised NeRF: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong. Gram: Generative radiance manifolds for 3d-aware image generation. arXiv preprint arXiv:2112.08867. 2021.
Qiancheng Fu, Qingshan Xu, Yew-Soon Ong, and Wenbing Tao. Geo-neus: Geometry-ccnniiseet
neural implicit surfaces learning for multi-view reconstruction, 2022. URL lttps:://axiv
org
abs/ orrg/bs// 2205. 15848
Ravi Garg, Vijay Kumar Bg, Gustavo Carneiro, and lan Reid. Unsupervised cnn for single view depth estimation: Geometry to the rescue. In European conference on computer vision, pp. 740-756. Springer, 2016.
Clément Godard, Oisin Mac Aodha, and Gabriel Brostow. Unsupervised monocular depth estima- tion with left-right consistency. In CVPR, 2017.
Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5875-5884, 2021.
Baichuan Huang, Hongwei Yi, Can Huang, Yijia Hc, Jingbin Liu, and Xiao Liu. M3vsnet: Unsuper- vised multi-metric multi-view stereo network. In 202/ IEEE International Conference on Image Processing (ICIP), pp. 3163 3163-3167, 2021. doi: 10. 1091ICIP429282221..999
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et Spatial transformer networks. Advances in neural information processing systems, 28, 2015.
Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 5885-5894, 2021.
Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi- view stereopsis evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 406-413, 2014.
10

---

Under review as a conference paper at ICLR 2023
Yoonwoo Jeong, Seokjun Ahn, Christopher Choy, Anima Anandkumar, Minsu Cho, and Jaesik Park. Self-calibrating neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Viston, pp. 5846-5854, 2021.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision, 2016.
Tejas Khot, Shubham Agrawal, Shubham Tulsiani, Christoph Mertz, Simon Lucey, and Martial Hebert. Learning unsupervised multi-view stereopsis via robust photometric consistency, arXiv preprint 1rXXiv:9005.07066 2019.
Mijeong Kim, Sconguk Sco, and Bohyung Han. Infonerf: Ray entropy minimization for few-shot neural volume rendering. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022.
Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ra- mamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. arXiv preprint arXiv:2201.05905 2022.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11453-11464, 2021.
Michael Niemeyer, Jonathan Barron, Ben Mildenhall, Mehdi M. Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022.
Keunhong Park. Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5865-5874, 2021.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10318-10327, 2021.
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14335-14345, 2021.
Barbara Roessle, Jonathan Barron, Ben Mildenhall, Pratul Srinivasan, and Matthias Nießner. Dense depth priors for neural radiance fields from sparse input views. arXiv preprint arXiv:2112.03288, 2021.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems, 33: 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CORR, abs/1 Abs/1 1556, 2014. Ukl.htttp.//arttt..rr///bbb//
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS, 2020.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 19979, 2021.
11

---

Under review as a conference paper at ICLR 2023
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612 2004. doi: 10.1 109/0/111220008118866..16
Xudong Xu, Xingang Pan, Dahua Lin, and Bo Dai. Generative occupancy fields for 3d surface-aware image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng. and Angjoo Kanazawa, PlenOctrees for real-time rendering of neural radiance fields. ICCV, 2021a.
Alex Yu, Vickie Yc, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one few images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4578 4587, 2021b.
Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and lan Reid. Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 340-300340 349, 2018.
Jian Zhang, Yuanqing Zhang, Huan Fu, Xiaowei Zhou, Bowen Cai, Jinchi Huang, Rongfei Jia, Binqiang Zhao, and Xing Tang. Ray priors through reprojection: Improving neural radiance fields for novel view extrapolation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18355-18365, 2022. doi: 10.1 1111009C95228882222.001111
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 2020.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1851-1858, 2017.
12


Reviews:
[{"review_id": "ydVhsCIx-Wy", "reviewer": "Reviewer_Copi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focus on the novel view synthesis with only sparse inputs. The authors propose a method called GeCo-NeRF that enforces geometric consistency to regularize the training of NeRF. Specifically, input images are warped to unseen viewpoints for supervision at feature level. They also filter out erroneous warped patches and engineer several training strategies to improve training. Experiments on both synthetic and real data demonstrated the effectiveness of GeCo-NeRF.", "summary_of_the_review": "Overall, I think this paper falls into the borderline. The techniques are not novel enough and some details are missing, but I appreciate the engineering effort and the results are quite promising. Therefore, I lean towards acceptance.", "strength_and_weaknesses": "Strength:\n- The results are quite promising and achieve the state-of-the-art in the few-shot NeRF setting\n- The authors put a lot of engineering efforts to make the warping technique work\n\nWeaknesses:\n- The method is not novel enough for me. Many methods propose to supervise at unseen viewpoints, e.g. DietNeRF.  Also, in [1], input image color was used to supervise unseen viewpoints at pixel level.\n- The authors doesn't discuss the computational cost of the proposed method. In the paper, they report that 8 hours is needed to train 120k iterations with 2 RTX 3090Ti. As far as I know, training vanilla NeRF for 200k iterations only need 8 hours on a single 1080Ti.\n- Unless I am missing something, ablation is not provided for the training strategy in sec 4.5, which I think is part of the core contribution of this paper\n- Some references can be incorporated and discussed: concurrent work [2], [3] also uses estimated depth to warp input images to other camera poses to regularize NeRF (This point doesn't influence my rating, but including them would make this paper more complete)\n- It seems that I cannot find the mentioned supplementary material\n\nMinor: \n- some references are not compiled successfully, as can be seen in sec 5.1\n- In \"Plugging in NeRF'' in sec 5.3, table 3 should be table 1\n\n[1] Ray Priors through Reprojection: Improving Neural Radiance Fields for Novel View Extrapolation\n\n[2] Data augmentation for NeRF: a geometric consistent solution based on view morphing\n\n[3] StructNeRF: Neural Radiance Fields for Indoor Scenes with Structural Hints", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gQvhrVJWHtx", "reviewer": "Reviewer_8ZEs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method that regularize nerf in a few-shot setting. The method utilize NeRF rendered depth as pseudo ground truth to warp rgb value to other near view. The error between source view and target view are taken as an additional loss to train NeRF in a few shot setting.", "summary_of_the_review": "Based on the weakness discussed above, I think the paper has limited novelty, and there is also some defacts in terms of technical and writing issues. Therefore, I vote for rejection at this stage. ", "strength_and_weaknesses": "Strengths:\n\n1. A novel way to generate pseudo ground truth by warpping multiple input views to the novel view location. The local features are aligned using NeRF generated depth map by an occlusion-aware mask map.\n\n2. Instead of vanilla NeRF only using pixel-level reconstruction loss, the paper proposes to add multi-level perceptual loss between wrapped pseudo ground truth image and generated novel view image. By adding such constrain at the feature level, the framework improves quality of generated novel views by a large margin in comparison to the baselines.\n\nWeaknesses:\n\n1. The novelty is not significant. The main contribution of this paper is the depth rewarping loss. However, using depth to for few-shot nerf is not new [1][2]. Also, warping input to unseen views is widely used in self-supervised MVS [3][4].\n\n2. It is not convincing that depth estimation (Eq. 4) is accurate enough with few viewpoints. Also, the resolution could be limited given the fact that the depth is generated with grid sampling. \n\n3. It is weird to mention Monocular depth estimation in related work, which may be self-supervised photometric consistency.\n\n4. There are a lot of missing citations. \n\n[1] Depth-supervised NeRF: Fewer views and faster training for free\n\n[2] Dense depth priors for neural radiance fields from sparse input views. \n\n[a] M3vsnet: Unsupervised multi-metric multi-view stereo network \n\n[b] Learning Unsupervised Multi-View Stereopsis via Robust Photometric Consistency", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RUYWTUMKnw", "reviewer": "Reviewer_m8Jd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a geometric regularization technique for nerf training under a sparse view setting. The regularization is based on cross-view warping between seen/unseen views using rendered depth, where the warped image is utilized for supervision in feature space.\n\n", "summary_of_the_review": "See above. I would like to hear the author's feedback on the missing references and I would raise my rating if my concerns could be well-addressed.", "strength_and_weaknesses": "Pros:\n\n1. The overall idea is tidy and easy to follow\n\n2. Experiment results seem promising\n\nCons:\n\n1. Some important references which address the multi-view photo-consistency are missing, e.g., MVSDF, NeuralWarp, and GeuNeuS. Specifically, MVSDF also applies feature consistency to constrain the neural rendering optimization.\n\n2. Following Cons 1, I would expect a comparison between the proposed method with the explicit multi-view consistency formulation in MVSDF/NeuralWarp/GeuNeuS. (maybe an ablation study plus some explanations). From my perspective, they are all some kind of similar photoconsistency but implemented in different ways.\n\n3. There are several wrong citations in the [Datasets and metrics] section.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "M6gtsr8eeWz", "reviewer": "Reviewer_w2pC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method to enable only sparse image set as an input to a NeRF network and enable rendering of artifact free novel views which are typically not possible using current existing methods. Their main contribution is to warp given input images to novel camera locations given the current estimate of the depth at novel view and minimize the error between the warped image and the NeRF rendered image at that novel viewpoint. The warped image is assumed to be a ground truth at the novel view-point. Since warping across translated views can lead to occlusion effects, a best estimate of occlusion mask is also calculated and used to mask out the cost error regions. The loss function is designed to be minimizing feature maps at multiple scales instead of direct pixel intensity difference because the pixel intensities can be different in reality if the scene contains non-Lambertian surfaces. The results are show to be better than standard and existing approaches.", "summary_of_the_review": "Please see Strengths above.", "strength_and_weaknesses": "Strength:\n* The paper addresses a very relevant area of NeRF application to real world where capturing a scene with many images maybe impossible due to scene motion or having a 100-camera multi-camera system to capture a scene is economically and practically prohibitive.\n* The paper is well written and all aspects are well discussed.\n* The results look good and show improvement over existing works.\n\nWeakness:\n* No major weakness except a few \"?\" in the paper which can be resolved.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject