3D Reconstruction with Generalizable Neural Fields using Scene Priors
Authors: Yang Fu
Shalini De Mello
Xueting Li
Amey Kulkarni
Jan Kautz
Xiaolong Wang
Sifei Liu
Venue: ICLR 2024 poster (2024)
Keywords: 3D Reconstruction
Generalizable Neural Fields
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: High-fidelity 3D scene reconstruction has been substantially advanced by recent progress in neural fields. However, most existing methods train a separate network from scratch for each individual scene. This is not scalable, inefficient, and unable to yield good results given limited views. While learning-based multi-view stereo methods alleviate this issue to some extent, their multi-view setting makes it less flexible to scale up and to broad applications. Instead, we introduce training generalizable Neural Fields incorporating scene Priors (NFPs). 
The NFP network maps any single-view RGB-D image into signed distance and radiance values. A complete scene can be reconstructed by merging individual frames in the volumetric space WITHOUT a fusion module, which provides better flexibility.  The scene priors can be trained on large-scale datasets, allowing for fast adaptation to the reconstruction of a new scene with fewer views. NFP not only demonstrates SOTA scene reconstruction performance and efficiency, but it also supports single-image novel-view synthesis, which is under-explored in neural fields. More qualitative results are available at: https://oasisyang.github.io/neural-prior.

Published as a conference paper at ICLR 2024
3D RECONSTRUCTION WITH GENERALIZABLE NEURAL FIELDS USING SCENE PRIORS
Yang Fu' i
Shalini De Mello
Xueting Li Sifei Liu?
Amey Kulkarni
Jan Kautz² Xiaolong Wang'
I University of California, San Diego NVIDIA
ABSTRACT
High-fidelity 3D scene reconstruction has been substantially advanced by recent progress in neural fields. However, most existing methods train a separate network from scratch for each individual scene. This is not scalable, inefficient, and unable to yield good results given limited views. While learning-based multi-view stereo methods alleviate this issue to some extent, their multi-view setting makes it less flexible to scale up and to broad applications. Instead, we introduce training general- izable Neural Fields incorporating scene Priors (NFPs). The NFP network maps any single-vier view RGB-D image into signed distance and radiance values. A complete scene can be reconstructed by merging individual frames in the volumetric space WITHOUT a fusion module, which provides better flexibility. The scene priors can trained on large-scale datasets, allowing for fast adaptation to the reconstruction of scene with fewer views. NFP not only demonstrates SOTA scene recon- struction performance and efficiency, but it also supports single-image novel-view synthesis, which is underexplored in neural fields. More qualitative results are available at: https: / oas
ctt.uu..oo//uuuuuu www.ssasssaaaaa.oooaao.....a...a syang. git.ub. io wtt/euraa-prio.
INTRODUCTION
Reconstructing a large indoor scene has been long-standing problem in computer vision. common approach is to use the Truncated Signed Distance Function (TSDF) (Zhou et al., 2018; Dai et al., 2017b) with depth sensor on personal devices. However, the discretized representation with TSDF limits its ability to model fine- une-grained details, e.g., thin surfaces in the scene. Recently, a continuous representation using neural fields and differentiable volume rendering (Guo et al., 2022; Yu et al., 2022; Azinović et al., 2022; Wang et al., 2022b; Li et al., 2022) has achieved impressive and detailed 3D scene reconstruction.
Although these results are encouraging, all of them require training distinct network for every scene, leading to extended training durations with the demand of substantial number of input views.
To tackle these limitations, several works learn generalizable neural network so that the representa- tion can be shared among different scenes (Wang et al., 2021b; Zhang et al., 2022; Chen et al., 2021; Long et al., 2022; Xu et al., 2022). While these efforts scale up training large-scale scene datasets, introduce generalizable intermediate scene representation, and significantly cut down inference time, they all rely intricate fusion networks handle multi-view input images at each iteration. This adds complexity to the training process and limits flexibility in data preprocessing.
In this paper, propose to perform 3D reconstruction by learning generalizable Neural Fields using scene Priors (NFPs). Such priors are largely built upon depth-map inputs (given posed RGB-I images). By leveraging the priors, our NFPs network allows for simple and flexible design with single-view inputs during training, and it can efficiently adapt to each novel scene using fewer input views. Specifically, full scene reconstruction achieved by directly merging the posed multi-view frames and their corresponding fields from NFPs, without the need for learnable fusion blocks.
direct way generalize per-scene Nerf optimization into an intermediate representation in the volumetric space. Yet, co-learning the encoder and the
encode each single-view input image
This work done while Yang Fu research intern NVIDIA.

---

Published as a conference paper at ICLR 2024
direct fusion
apply independeatly with shared weights
(a) generalizable prior
(b) per-scene optimization
Figure 1: We propose Neural Fields scene Prior (NFP) to enable fast reconstruction of geometry and texture of indoor scenes. Our method first (a) learns generalizable network as scene prior that obtains a coarse scene reconstruction in a feed-forward manner. Next, we directly fuse the per-view results and (b) perform per-scene optimization in more accurate and efficient way leading to high-quality surface reconstruction and realistic texture reconstruction.
NeRF presents significant challenges. Given that single-view image captures only thin segment of a surface, it becomes considerably harder to discern the geometry compared to understanding the texture. Thus, to train NFPs, we introduce two-stage paradigm: (i) We train geometric reconstruction network to map depth images to local SDFs; (ii) We adopt this pre-trained network as a geometric prior to support the training of a separate color reconstruction network, as a texture prior, in which the radiance function can be easily learned with volumetric rendering (Wang et al., 2021a; Yariv et al., 2021), given the SDF prediction.
popular choice in many NeRF-based rendering techniques (Yen-Chen et al., 2020; Chen et al., 2021; Liu et al., 2020; Huang et al., 2021; Takikawa et al., 2021; Sun et al., 2022b; Wang al., 2022b). However, for the single-view input context, they fall short for two main reasons. First, the single-view image inherently captures just thin and confined segment of surfaces, filling only minuscule fraction of the entire voxel space. Second, dense voxel grids employ uniform sampling, neglecting surface priors like available depth information. Instead, we resort surface representation: we build set of projected points in the 3D space keypoint, from where continuous surface can be decoded. The keypoint representation spans a compact 2D surface representation, allowing dense sampling close the surface, which significantly enhances scalability.
Dense voxel grids
NFPs can easily facilitate further fine-tuning on large-scale indoor scenes. Given the pretrained geometry and texture network as the scene prior, the single-scene scene reconstruction can be performed by optimizing the aggregated surface representation and the decoders.
With coarse reconstruction from the generalized network and highly compact surface representation, our approach achieves competitive scene reconstruction and novel view synthesis performance with substantially fewer views and faster convergence speed. In summary, our contributions include:
We propose NFPs, a generalizable scene prior that enables fast, large-scale scene reconstruc- tion.
NFPs facilitate (a) single-view across-scene input, (b) direct fusion of local frames, and (c) efficient per-scene fine-tuning.
We introduce a continuous surface representation, taking advantage of the depth input and avoiding redundancy in the uniform sampling of a volume.
With the limited number of views, we demonstrate competitive performance on both the scene reconstruction and novel view synthesis tasks, with substantially superior efficiency than existing approaches.
2 RELATED WORK
Reconstructing and rendering large-scale indoor scenes is crucial for various applications. Depth sensors, on the other hand, are becoming increasingly common in commercial devices, such as Kinect (Zhang, 2012; Smisek al., 2013), iPhone LiDAR (Nowacki & Woda, 2019), etc. Leveraging
2

---

Published as a conference paper at ICLR 2024
Eeepch (3)
Geometric
training
r0c) (x0
gonmerry muoza
-naar
Quay
goometay decoder
Grve (5)
normal
Texture
Prior training
y(r)
readering
encodeer
Lege (10)
teochure decoder
Quers JUNA
stage= 1
stago-2
stage-182
Figure 2: Overview of NFP. Given the RGBD input, we first extract the geometric and texture pixel feature using two encoders (Sec. 3.1). Then, we construct the continuous surface representation upon the discrete surface feature (Sec. 3.2). Next, introduce two-stage paradigm learn the generalizable geometric and texture prior, optimized via multiple objectives (Sec. 3.3).
depth information in implicit neural representations is trending. We discuss both these topics in detail, in the following.
Multi-view scene reconstruction. Reconstructing 3D scenes from images was dominated by multi- view stereo (MVS) (Schönberger et al., 2016; Schonberger & Frahm, 2016), which often follows the single-view depth estimation (e.g., via feature matching) and depth fusion process (Newcombe et al., 201 1; Dai al., 2017b; Merrell et al., 2007). Recent learning-basee et al., 2020; Düzçeker et al., 2020; Huang et al., 2018; Luo et al., 2019) substantially outperform the conventional pipelines. For instance, Yao et al. (2018); Luo et al. (2019) build the colt-volume based on 2D image features and use 3D CNNs for better depth estimation. Another line of works (Sun al., 2021; Bi al., 2017) fuse multi-view depth and reconstruct surface meshes using techniques such as TSDF fusion. Instead of fusing the depth, Wei et al. (2021), Wang et al. (2021b), Zhang et al. (2022), and Xu et al. (2022) directly aggregate multi-view inputs into radiance field for coherent reconstruction. The multi-view setting enables learning generalizable implicit representation, however, their scalability is constrained as they always require multi-view RGB/RGB-D data during training. Our approach, for the first time, learns generalizable scene priors from single-view images with substantially improved scalability.
MVS methods (Cheng
Neural Implicit Scene Representation. A growing number of approaches (Yariv et al., 2020; Wang al., 2021a; a; Yariv et al., 2021; Oechsle et al., 2021; Niemeyer et al., 2020; Sun et al., 2022a) represent a scene by implicit neural representations. Although these methods achieve impressive reconstruction of objects and scenes with small-scale and rich textures, they hardly faithfully reconstruct large-scale scenes due to the shape-radiance ambiguity suggested in (Zhang al., 2020; Wei et al., 2021). To address this issue, Guo et al. (2022) and Yu et al. (2022) attempt to build the NeRF upon a given geometric prior, i.e., sparse depth maps and pretrained depth estimation networks. However, these methods take a long time to optimize on an individual scene. As mentioned previously, generalizable NeRF representations with mutli-view feature aggregation et al., 2021b; Zhang et al., 2022; Johari et al., 2022; Xu et al., 2022), However, they still focus on reconstructing the scene's 's appearance, e,g., for novel view synthesis, but cannot guarantee high-quality surface reconstruction.
studied (Chen et al., 2021; Wang
Depth-supervised reconstruction and rendering.
With the availability of advanced depth sensors, many approaches seek depth-enhanced supervision of NeRF (Azinović et al., 2022:
al., 2022; Zhu et al., 2022; Sucar et al., 2021; Yu al., 2022; Williams et al., 2022; Xu et al., 2022; Deng et al., 2022) since depth information is more accessible. For instance, Azinović et al. (2022) enables detailed reconstruction of large indoor scenes by comparing the rendered and input RGB-D images. Unlike most methods that use depth as supervision, Xu et al. (2022), Williams et al. (2022) and Dong et al. (2023) build the neural field conditioned on the geometric prior. For example, Point-NeRI pretrains monocular depth estimation network and generates a point cloud by lifting the depth prediction. Compared to ours, their geometric prior is less integrated into the main reconstruction stream since it is separately learned and detached. Furthermore, these methods only consider performing novel view synthesis (Xu et al., 2022; Deng al., 2022), where the geometry is not optimized, or perform pure geometric (Yu al., 2022; Li
3

---

Published as a conference paper at ICLR 2024
et al., 2022; Williams et al., 2022; Azinović et al., 2022) reconstruction. In contrast, approach makes the scene prior and the per-scene optimization a unified model that enables more faithful and efficient reconstruction for both color and geometry.
METHOD
Given sequence of RGB-D images and their corresponding camera poses, our goal is to perform fast and high-quality scene reconstruction. To this end, we learn generalizable neural scene prior, which encodes an RGB image and its depth map as continuous neural fields in 3D space and decodes them into signed distance and radiance values. As illustrated in Fig. 2, we first extract generalizable surface features from geometry and texture encoders (Sec. 3.1). Then, pixels with depth values are back- projected to the 3D space as keypoints, from which continuous fields can be built with the proposed surface representation (Sec. 3.2). Motivated by previous works (Wang et al., 202 la; Yariv al., 2021), utilize two separate MLPs to decode the geometry and texture representations, which are further rendered into RGB and depth values (Sec. 3.3). To obtain high-quality surface reconstruction, we further propose optimize the neural representation top of the learned geometric and texture prior for a specific scene (Sec. 3.4).
3.1 CONSTRUCTING SURFACE FEATURE
Given an RGB-D image (I, DJ, we first project the depth map into 3D point clouds the world coordinate system using its camera pose {R, 1) and intrinsic matrix K. We sub-sample M points via Farthest Point Sampling (FPS), denoted as [Pm]: pmm.m m E [0, M - 1], which are used as keypoints representing the discrete form of surfaces. We extract generalizable point-wise geometry and texture features, as described below, which are further splatted onto these keypoints. Both encoders are updated when training the NFP.
Geometry encoder. For each surface point, we apply the K-nearest neighbor (KNN) algorithm to find K points and construct local region with K points. Thus, obtain collection of M local regions, (Pm, {p& kee Vm E [0, .M 1]. where Vm is the neighbor index set of point pm and | K 1. Then, we utilize stack of PointConv (Wu et al., 2019) layers extract the geometry feature from each local region fgeo = PointConvi ((pm [p*]ReN..) }).
Texture encoder. In addition, we extract RGB features for the keypoints via a 2D convolutional neural network. In particular, we feed an RGB image / into an UNet (Ronneberger et al., 2015) with ResNet34 (He et al., 2016) as the backbone, which outputs a dense feature map. Then, we splat the pixel-wise features fiee onto the keypoints, according to the projection location of the surface point Pm from the image plane. Thus, each surface point is represented by both a geometry feature and texture feature, denoted by f((m) Ugeo(Pnn f Frx (Pm))
3.2 CONTINUOUS SURFACE IMPLICIT REPRESENTATION
Given the lifted keypoints and their projected geometry and texture features, in this section, we introduce how construct continuous implicit fields conditioned on such discrete representations. We follow a spatial interpolation strategy: for any query point x (e.g.. in typical volume rendering process, can be a sampled point along any ray), we first find the K nearest surface points (p> vev,, where V is a set of indices of the neighboring surface points. Then, the query point's feature can be obtained via aggregation of its neighboring surface points. In particular, we apply distance-based spatial interpolation as
fxx) Evey wof(Pu) Evev Wa
Wo exp (-11x poll),
where fx) represents either the geometry fgeo (x) the texture fex (x) feature, and p. is the position of the v-th neighbouring keypoint. With distance-basee spatial interpolation, we establish continuous implicit fields for any point from the discrete keypoints.
The continuous representation suffers from two drawbacks: First, when a point is far away from the surface, fxx is no longer valid representation, but will still contribute decoding and rendering.
4

---

Published as a conference paper at ICLR 2024
Second, the distance we is agnostic to the tangent direction and hence is likely to blur the boundaries. To mitigate the first problem, we incorporate an additional MLP layer that takes into account both the original surface feature f(P.)) and its relative distance to the query point - Pu, and outputs distance-aware surface feature ((p = MLP (pxx), x - pv) Subsequently. this refined surface feature f f((") replaces the original surface feature in Eq. 1 to obtain the feature of query point In addition, we ensure that the sampled points lie near the surface via importance sampling. We resolve the second issue via providing the predicted normal the decoders an input. We refer to Sec. 3.3
and 3.4 for details.
3.3 GENERALIZABLE NEURAL SCENE PRIOR
To reconstruct both geometry and texture, i.e., textured mesh, a direct way decode the geometry and texture surface representation (Sec. 3.2) into signed distance and radiance values, render them into RGB and depth pixels (Guo al., 2022; Yu et al., 2022), and then supervise them by the ground-truth RGB-D images.
Unlike the multi-view setting that covers a significant portion of the volumetric space, the single-view input only encompasses small fraction of it. From our experiments, we found that the joint training approach struggles to generate accurate geometry.
Hence, we first learn geometric network that maps any depth input to its corresponding SDF (Sec. 3.3.1). Once a coarse surface is established, learning the radiance function initialized by becomes much easier pose it in the second stage where a generalizable texture network is introduced similarly (Sec. 3.3.2).
3.3.1 GENERALIZABLE GEOMETRIC PRIOR
We represent scene geometry as a signed distance function, where our case, it is conditioned on the geometric surface representation fgeo (2) to allow for generalization ability across different scenes. Specifically, along each back-projected ray with camera center o and ray direction r, we sample N points as x; o + d,r, Vi E j0, N 1]. For each sampled points x,, its geometry feature fgeo (x,) can be computed by equation 1. Then, the geometry decoder &G. taking the point position and its geometry feature inputs, maps each sampled point to signed distance, which is defined as s(x,) = (G (gco x.) Note that we also apply positional encoding 7()) to the point position as suggested in Mildenhall al. (2020). We omit it for brevity.
Following the formulation of NeuS (Wang al., 2021a), the estimated depth value d is the expected values of sampled depth de along the ray:
d d= Taadi =
II az)
j=1
04 max (ox()()))) - (s(xxx+1))
o, (s(x)))
where T, represents the accumulated transmittance point x,, a₂ the opacity value and or is Sigmoid function modulated by a learnable parameter s.
Geometry objectives. To optimize the generalizable geometric representation, we apply pixel-wise rendering loss on the depth map,
Caepth ld D(z, v)l.
Inspired by (Azinović et al., 2022; Li et al., 2022), approximate ground-truth SDF based on the distance to observed depth values along the ray direction, b(x,) = D(x, y) d, Thus, for points that fall in the nurfeserface region ((b(x;)) T, 7 is truncation threshold), apply the following approximated SDF loss
Luear |s(xx) b(xx))
We also adopt free-space loss (Ortiz et al., 2022) to penalize the negative and large positive predictions.
Liron = max (0,e-ss -s((x)) ) - -1, s(, - M(xi))
(5)
5

---

Published as a conference paper at ICLR 2024
where is the penalty factor. Then, approximated SDF loss is
Cuear
if 6(x;) S |r]
Cads
(6)
Liree otherwise
The approximated SDF values provide us with more explicit and direct supervision than the rendering depth loss (Eq. equation 3).
Surface regularization. To avoid artifacts and invalid predictions, further the Eikonal regularization term (Yariv et al., 2021; Ortiz et al., 2022; Wang et al., 2021a), which aims to encourage valid SDF values via the following,
Leik [x, 111.
where (x,)(x)) is the gradient of predicted SDF w.r.t. the sampled point X;-
Therefore, we update the geometry encoder and decoder with the generalizable geometry loss as follows,
goo Adepth Seeph + dutt Ladt
deikCeeik
3.3.2 GENERALIZABLE TEXTURE PRIOR
We build the 2nd stage the generalizable texture network following the pretrained geometry network, as presented in Sec. 3.3.1, 1 which offers the SDF's prediction as an initialization. Specifically, we learn pixel-wise RGB features, keypoints. Following the spatial interpolation method in Sec. 3.2, we query the texture feature of any sampled point in 3D space. As aforementioned, the spatial interpolation in Eq. equation 1 is aware of the surface tangent directions. For instance, point at the intersection of two perpendicular planes will be interpolated with keypoints coming from both planes. Thus, representations the boundary regions can be blurred. To deal with it, we further concatenate the surface normal V <x,)(x;) predicted in the first stage with the input compensate for the missing information.
described in Sec. 3.1, and project them onto the corresponding
With separate texture decoder étex, the color of point is estimated, conditioned on the texture feature frex(x; and the surface normal sx;;
c(x;) (rexx(frxx(Xe) (x,), Vx,s(x;)),
where r is the ray direction. Here we omit the positional encoding of the point's position and ray direction for conciseness. Therefore, the predicted pixel color can be expressed è Σ, Ta, where T; and ag are defined same as Eq. equation 2. We supervise the network by minimizing the L2 loss between the rendered pixel RGB values and their ground truth values
Lagh l - 12x,1)1
(10)
Meanwhile, we jointly learn the geometry network including the PointConv encoder and geometry decoder introduced in Sec. 3.2, via the same Egeo- Thus, the total loss function for generalizable texture representation learning is
Lrox Adepth Cdepth + Auat + desk Leike
(11)
Augh
During volumetric rendering, to restrict the sampled points from being concentrated on the surface, we perform importance sampling based on: (i) the predicted surface as presented in Wang al. (2021a), and (ii) the input depth map. More details are in the supplementary material.
3.4 PRIOR-GUIDED PER-SCENE OPTIMIZATION
To facilitate large-scale, high-quality scene reconstruction, we can further finetune the pretrained generalizable geometric and texture prior to individual scenes, with multi-view frames. Specifically, we first directly fuse the geometry and texture feature multi-view frames via the scene prior networks. No further learnable modules required, in contrast, to (Chen et al., 2021; Zhang et al., 2022; Li et al., 2022). Then, we design a grior- pruning and sampling module, which lets optimization happen near surfaces. In particular, we initialize the grid in the volumetric space via

---

Published as a conference paper at ICLR 2024
Ours-prior min
Man 640 min
Go-surf 35
Ours 15 min
GT
Figure 3: Qualitative comparisons for mesh reconstruction on ScanNet. Our method achieves the most complete and fine-detailed reconstruction. The prior reconstruction results and the ground-truth are provided as reference. Better viewed when zoomed in.
learn NSP and estimate the SDF value of each grid by its corresponding feature, and remove the grids whose SDF values are larger than threshold. We note that the generalizable scene prior can be combined with various optimization strategies (Xu et al., 2022; Yu et al., 2022; Wang et al., 2022b). More details can be found in the supplementary materials.
During the finetuning, update the scene-prior feature, and the weights of the MLP decoders fit the captured images for a specific scene. Besides the objective functions described in Eq. equation 11, we also introduce the smoothness regularization term to minimize the difference between the gradients of nearby points
Lsmooth ((x, Vx, Vx + +o)ll,
(12)
where a is a small perturbation value around point x4. Thus, the total loss function for per-scene optimization is
Lsceue Adepth Cdepth + Aadt + delx Lrgh + Xemooth Csmooth.
(13)
EXPERIMENTS
In this work, we introduce generalizable network that can be applied to both surface reconstruction and novel view synthesis from RGB-D images in offline manner. To our best knowledge, there is no prior work that aims for both two tasks. To make fair comparisons, we compare our work with the state-of-the-art (STOA) approaches of each task, respectively.
4.1 BASELINES, DATASETS AND METRICS
Baselines. To evaluate surface reconstruction, we consider the following two groups of methods: First, we compared our method with RGB-based neural implicit surface reconstruction approaches: ManhattanSDF (Guo et al., 2022) and MonoSDF (Yu et al., 2022) which involve additional network to extract the geometric prior during training. Second, we consider several RGB-D surface reconstruction approaches that share similar settings with ours: Neural-RGBD (Azinović al., 2022) and Go-surf (Wang et al., 2022b). In addition, to have a fair comparison, we finetune ManhattanSDF and MonoSDF with ground-truth depth maps two additional baselines and denoted as ManhattanSDDF and MonoSDF". We follow the setting in (Guo et al., 2022; Azinović et al., 2022) and evaluate the quality of the mesh reconstruction different scenes. We note that all the above approaches perform per-scene optimization.
To evaluate the performance in novel view synthesis, we compare our method with the latest NeRF- based methods in novel view synthesis, including NeRF (Mildenhall et al., 2020), NSVF (Liu et al., 2020), NerfingMVS (Wei al., 2021), IBRNet (Wang al., 2021b) and NeRFusion Zhang al. (2022). As most of existing works are only optimized with RGB data, we further evaluate the Go-surf for novel view synthesis from RGB-D images as another baseline. We adopt the evaluation setting in NerfingMVS, where we evaluate our method on scenes, and for each scene, pick 40 images covering a local region and hold out 1/8 of these as the test set for novel view synthesis.
ScanNetV2 (Dai et al., 2017a) for both surface reconstruction and novel view synthesis tasks. Specifically, we first train the generalizable neural scene prior the ScanNetV2 training set and then evaluate its performance in two testing splits proposed by Guo et al. (2022) and Wei et al. (2021) for surface reconstruction and novel view synthesis, respectively. The GT of Sean produced by BundleFusion Dai et al. (2017b), is
Datasets. We mainly perform experiments
7

---

Published as a conference paper at ICLR 2024
Method
depth
opt. (min)
Acc|
Compl
Prect
Recall
F-score
ManhattanSDF (Guo et al., 2022)
SIM
640
0.072
0.068
0.621
0.586
0.602
MonoSDF (Yu et al., 2022)
network
720
0.039
0.044
0.775
0.722
0.747
NeuRIS (Wang et al., 2022a)
network
480
0.051
0.048
0.720
0.674
0.696
FastMono (Dong ct al., 2023)
network
30
0.042
0.056
0.751
0.678
0.710
HelixSurf (Liang et al., 2023)
network
30
0.038
0.044
0.786
0.727
0.755
ManhattanSDF* (Guo et al., 2022)
GT.
640
0.027
0.032
0.915
0.883
0.907
MonoSDF (Yu et al., 2022)
GT.
720
0.033
0.026
0,942
0,912
0.926
Neural RGBD (Azinović et al., 2022)
GT.
240
0.055
0.022
0.932
0.918
0,925
sorsurf (Wang et al., 2022b)
GT.
35
0.052
0,018
0,946
0.956
0,950
Ours- prrs- (w/o per-scene opt.)
0.084
0.057
0.695
0.764
0.737
Ours (w per-scene opt.)
GT.
15
0.049
0.017
0.947
0.962
0.954
Table 1: Quantitative comparisons for mesh reconstruction on ScanNet. We compare with a number of baselines. "." is our re-implementation with dense ground-truth depth map. "opt." stands for the optimization time for per-scene fine-tuning.
Method
#frame|
Acc L Comp | C-4 | NC F-scoret
BundleFusion (Dai et al., 2017b)
1,000
0.0191
0.0581 0,0386 0.9027
0.8439
COLMAP (Schönberger al., 2016)
1,000
0.0271
0.0322 0.0296 0.9134
0.8744
ConvOccNer (Peng et al., 2020)
1,000
0.0498
0.0524 0.0511 0.8607
0.6822
SIREN (Sitzmann et al., 2020)
1,000
0.0229 0.0412 0.0320 0.9049
0.8515
Neural RGBD (Azinović et al., 2022)
1,000
0.0151
0.0197 0,0174 0,9316
0,9635
Go-surf (Wang et al., 2022b)
1,000
0.0158
0.0195 0.0177 0.9317
0.9591
Ours
1,000
0.0172
0.0192 0.0177 0.9311
0,9529
Go-surf (Wang ct al., 2022b)
30
0.0246 0.0442 0.0336 0.9117 0,9042
Ours
30
0.0177 0.0292 0.0234 0.9207 0.9311
Table 2: Quantitative evaluation of the reconstruction quality on 10 synthetic scenes. Our method show
competitive results when being reconstructed using only 30 frames used per room, in the lower part of the table.
known to be noisy, making accurate evaluations against it challenging. To further validate our method, we also conduct experiments on 10 synthetic scenes proposed by Azinović et al. (2022).
Evaluation Metrics. For 3D reconstruction, we evaluate our method in terms of mesh reconstruction quality used in Guo et al. (2022). Meanwhile, we measure the PSNR, SSIM, and LPIPS for novel view synthesis quality.
4.2 COMPARISONS WITH THE STATE-OF-TIIEART METHIODS
Surface reconstruction. Table provides quantitative comparison of our methods against STOA approaches for surface reconstruction (Guo et al., 2022: Yu et al., 2022; Wang et al., 2022a; Liang et al., 2023). Within our methods, the feed-forwart NFPs are denoted as Ours-prior, while the per-scene optimized networks are labeled as Ours. We list the RGB- and RGB-D-based approaches as in the top and the middle rows, whereas ours are placed in the bottom section. While we include ManhattanSDI (Guo et al., 2022) and MonoSDF (Yu et al., 2022) that are supervised by predicted or sparse depth information as in the top row, to ensure fair comparisons, we re-implement them by replacing the original supervision with ground-truth depth, as in the middle row (denoted by ***). Generally, using ground- truth depths can always enhance the reconstruction performance.
Comparison with NFPs on ScanNet. In contrast to all the other approaches that all require time- consuming per-scene optimization, the NPFs can extract the geometry structure through a single forward pass. The results in Table demonstrate that, even without per-scene optimization, the NFPs network not only achieves performance on par with RGB-based approaches but also operates hundreds of times faster. Note contrast to all the other approaches in Table that use around 400 frames to optimize the scene-specific neural fields, Ours-prior only takes around 40 frames per scene as inputs to achieve comparable mesh reconstruction results without per-scene optimization.
Comparison with optimized NFPs on ScanNet. We further perform per-scene optimization on top of the NFPs network. Compared with methods using additional supervision or ground truth depth maps, method demonstrates more accurate results on the majority of the metrics. More importantly, our method is either much faster, compared with the SOTA approaches. Some qualitative results shown in Fig. and more results can be found in the supplementary materials.
Comparison on synthetic scenes. Table 2 compares our approach with the most recent works on neural surface reconstruction from RGB-D images. The results demonstrate that method achieves
8

---

Published as a conference paper at ICLR 2024
Ground-truth
NerfingMVS
Go-surf
Ours
Figure 4: Qualitative comparison novel view synthesis on ScanNet. compare our method with baselines which achieves the competitive geometry reconstruction performance. Our approach produces more realistic rendering results than two other baselines.
comparable performance with most existing works, even when optimizing with limited number of frames, such as 1,000 30.
Results on novel view synthesis. To validate the learned radiance representation, we further novel view synthesis. The quantitative results and qualitative results are shown in Table 3 and Fig. 4. Table 3 shows that the proposed method achieves comparable if not better results compared to SOTA novel view synthesis methods (Wang et al., 2021b; Zhang et al., 2022; Liu et al., 2020). We note that our method outperforms Go-surf this in- stance, even when both methods achieve com- parable geometric reconstruction performance. offers distinct advantages for novel view synthesis. In addition, from Fig. 4, both NerfingMVS (Wei et al., 2021) and Go-surf (Wang et al., 2022b) fail on scenes with complex geometry and large camera motion. The generalized representation enables the volumetric rendering focus on more informative regions during optimization and improves its performance for rendering RGB images of novel views.
Method
PSNR1
SSIM
LPIPS]
NeRF (Mildenhall ct al., 2020)
24.04
01860)
0.334
conduct experiments
NSVF (Liu et al., 2020)
26.01
0.881
NeRFingMVS (Wei al., 2021)
26.37
0,903
0.245
IBRNet (Wang et al., 2021b)
25.14
0.871
0.266
NeRFusion (Zhang ct al., 2022)
26.49
0.915
0.209
Go-surf (Wang et al., 2022b)
25.47
0.894
0,420
Ours
26.88
0.909
0.244
Table 3: Quantitative comparisons for novel view synthesis on ScanNet. The best two results of different metrics are highlighted.
This suggests that our learned prior representation
4.3 ABLATION STUDIES
We further perform the ablation studies to evaluate the effec- tiveness and the efficiency of the neural prior network. Effec- tiveness of generalized representation. Table shows the results with and without the generalized representation. For the model without generalized representation, we randomly initialize the parameters of feature grids and decoders while keeping the other components unchanged. We observe that the model integrated with geometry prior and/or color prior can consistently improve the performance on 3D reconstruction and novel view synthesis.
Geo. prior
Acc|
Compl
F-scoref
0.079
0.031
0.851
0.046
0.030
0.862
Color prior
PSNR
SSIM
LPIPS
25.87
0.899
0.415
26.88
0.909
0.246
Table 4: Ablation studies geomet- ric and texture prior. We report both mesh reconstruction metrics and novel view synthesis metrics.
Fast optimization. Our approach can achieve high-quality reconstruction at approximately 1.5K iterations within 15 minutes. As illustrated in Fig. 5, our method achieves high F-score at the very early training stage, while Man- hattan SDF* (Guo et al., 2022) and MonoSDF" (Yu et al., 2022) take much more iterations to reach similar perfor- mance.
86
F-acore (96)
25
MonhutteuSDp
MoneSiie
2000
1000
6000
8000
10000
Number Herraionss
Figure 5: Ablation studies the number training iterations for per-scene opti- mization.
5 CONCLUSION
In this work, we present generalizable scene prior that enables fast, large-scale scene reconstruction of geometry and texture. Our model follows a single-view RGB-D input setting and allows non- learnable direct fusion of images. We design two-stage paradigm to learn the generalizable geometric and texture networks. Large-scale, high-fidelity scene reconstruction can be obtained with efficient fine-tuning our approach can achieve state-of-the quality of indoor scene reconstruction with fine geometric details and realistic texture.
the pretrained scene priors, even with limited views. We demonstrate that

---

Published as a conference paper at ICLR 2024
Acknowledgement This work was supported, in part, by NSF CAREER Award IIS-2240014, the Qualcomm Innovation Fellowship, Amazon Research Award.
REFERENCES
Dejan Azinović, Ricardo Martin Martin-Brualla, Dan Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6290-6301, 2022. 3, 5, 7, 8, 15, 16, 18, 19
Sai Bi, Nima Khademi Kalantari, and Ravi Ramamoorthi. Patch-based optimization for image-based texture mapping. ACM Trans. Graph., 36(4):106-1, 2017.
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mysnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14124-14133, 2021. 1, 2, 3,
6, 14
Rui Chen, Songfang Han, Jing Xu, and Hao Su. Point-based multi-view stereo network. In ICCV, pp. 1538-1547, 2019. 18
Shuo Cheng, Zexiang Xu, Shilin Zhu, Zhuwen Li, Erran Li, Ravi Ramamoorthi, and Hao Su. Deep stereo using adaptive thin volume representation with uncertainty awareness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2524-2534, 2020.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly- 3d reconstructions of indoor scenes. In CVPR, pp. 5828-5838 2017a. 7, 14, 15, 16, 18, 19
Angela Dai. Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. Bundlefu- sion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration. ACM Transactions on Graphics (ToG), 36(4):1, 2017b. 7,8
Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12891, 2022.
Wei Dong, Christopher Choy, Charles Loop, Or Litany, Yuke Zhu, and Anima Anandkumar. Fast monocular scene reconstruction with global-sparse local-dense grids. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4263-4272, 2023, 3, 8
Arda Düzşeker, Silvano Galliani, Christoph Vogel, Pablo Speciale, Mihai Dusmanu, and Marc View Stereo on Video with Recurrent Pentiol Fusion.
Pollefeys. DeepVideoMVE Multi-View
arXiv preprint arXiv:20/2.02 2020.
Arda Duzceker, Silvano Galliani, Christoph Vogel, Pablo Speciale, Mihai Dusmanu, and Marc Pollefeys. Deepvideomvs: Multi-view stereo on video with recurrent spatio-temporal fusion. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pp.
15324-15333, 2021. 18
Haoyu Guo, Sida Peng, Haotong Lin, Qianqian Wang, Guofeng Zhang, Hujun Bao, and Xiaowei Zhou. Neural 3d scene reconstruction with the manhattan world assumption. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 551 5511-5520, 2022. 1, 3,5,7. 7, 8, 9, 15, 16
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. 4, 14
Yuxin Hou, Juho Kannala, and Arno Solin. Multi-view stereo by temporal nonparametric fusion. In ICCV, pp. 2651-2660, 2019. 18
Jiahui Huang, Shi- Shi-Sheng Huang, Haoxuan Song, and Shi-Min Hu. Di-fusion: Online implicit 3d reconstruction with deep priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8932-8941, 2021.
10

---

Published as a conference paper at ICLR 2024
Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang. Deepmvs: Learning multi-view stereopsis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2821-28328 2018. 3
Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf with geometry priors. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18365-18375, 2022. 3
Kejie Li, Yansong Tang, Victor Adrian Prisacariu, and Philip HS Torr. Bnv-fusion: Dense 3d reconstruction using bi-level neural volume fusion. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 6166-6175, 2022. 1, 5,
Zhihao Liang, Zhangjin Huang, Changxing Ding, and Kui Jia. Helixsurf: A robust and efficient neural implicit surface learning of indoor scenes with iterative intertwined regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13165-13174, 2023. 8
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In NeurlPS, 2020. 2, 7,9
Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang. Sparseneus: Fast generalizable neural surface reconstruction from sparse views. arXiv preprint arXiv:2266.05337, 2022.
Keyang Luo, Tao Guan, Lili Ju, Haipeng Huang, and Yawei Luo. P-mvsnet: Learning patch- wise matching confidence aggregation for multi-view stereo. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10452-10461, 2019.
Paul Merrell, Amir Akbarzadeh, Liang Wang. Philippos Mordohai, Jan-Michael Frahm, Ruigang Yang, David Nistér, and Marc Pollefeys. Real-time visibility-basee fusion of depth maps. In 2007 IEFE 11th International Conference on Computer Vision, pp. 1-8. Ieee, 2007. 3
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV, pp. 405 421. Springer, 2020. 5, 7,7.9
Zak Murez, Tarrence van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich. Atlas: End-to-end 3d scene reconstruction from posed images. In ECCV, 2020. 18
Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In 2011 10th IEEE international symposiuum on mixed and augmented realify, pp. 127-136. leee, 2011.3
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In CVPR, pp. 2020. 3
Pawet Nowacki and Marek Woda. Capabilities of arcore and arkit platforms for ar/vr applications. In International Conference on Dependubility and Complex Systems, pp. 358-370. Springer, 2019.
Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf:
Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5589-5599, 2021.
Joseph Ortiz, Alexander Clegg, Jing Dong, Edgar Sucar, David Novotny, Michael Zollhoefer, and Mustafa Mukadam. isdf: Real-time neural signed distance fields for robot perception. arXiv preprint arXiv:2204.02296 2022. 5,
Adam Paszke, Sam Gross. Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Ginelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint 191.0:111 2019. 19
11

---

Published as a conference paper at ICLR 2024
Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolu- tional occupancy networks. In European Conference on Computer Vision, pp. 223-540. Springer, 2020. 8
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015. 4, 14
Johannes L Schonberger and Jan- Van-Michael Frahm. Structure-from comoomnttoon revisited. In CVPR, pp. 4104-41 4114-41132200 2016. 3
Johannes L Schönberger. Enliang Zheng, Jan-Michael Frahm, and Mare Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In ECCV, pp. 501-518. Springer, 2016. 3, 8
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im-
plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020. 8
Jan Smisek, Michal Jancosek, and Tomas Pajdla. with kinect. In Consumer depth camerus for computer vision, pp. 3-25. Springer, 2013.
Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J Davison. imap: Implicit mapping and positioning in real-time. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 6229-6238, 2021. 3
Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast conver- gence for radiance fields reconstruction. In CVPR, 2022a. 3
Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast conver- gence for radiance fields reconstruction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5459-5469, 2022b. 2
Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao, Neuralrecon: Real-time coherent 3d reconstruction from monocular video. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15598-15607, 2021. 3, 18
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezzhrrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11358-11367, 2021.
Matthew Tancik, Ethan Weber, Evonne Ng. Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. arXiv preprint arXiv:2302.0426: 2023. 19
Jiepeng Wang, Peng Wang, Xiaoxiao Long, Christian Theobalt, Taku Komura, Lingjie Liu, and Wenping Wang. Neuris: Neural reconstruction of indoor scenes using normal priors. In European Conference Computer Vision, pp. 139-155. Springer, 2022a. 8
Jingwen Wang, Tymoteusz Bleja, and Lourdes Agapito. Go-surf: Neural feature grid optimization for fast, figh-fideityy rgb-d surface reconstruction. arXiv preprint 220.1:2200..47353 2022b. 1, 7,8,9 9, 15, 16
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106. 2021a. 2, 4, 5, 6, 14
Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan Barron, Ricardo Bruallaa Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning milti-view image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4694 4699, 2021b. 1,3,7,9
12

---

Published as a conference paper at ICLR 2024
Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou. Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5610-5619, 2021. 3, 7,7, 7,9, 9, 16
Francis Williams, Zan Gojeic, Sameh Khamis, Denis Zorin, Joan Bruna, Sanja Fidler, and Or Litany. Neural fields as learnable kernels for 3d reconstruction. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 18500- 18510, 2022. 3.4
Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference Computer Vision and Putlern Recognition, pp. 9621-9630, 2019. 4, 14
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5438-5448, 2022. 3,3,7
Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. In Proceedings of the European conference on computer vision (ECCV), pp. 767-783, 2018.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 33:2492 22222222502, 2020. 3
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-4815 2021. 2, 3, 4, 6, 14
Lin Pete Florence, Jonathan T Barron, Alberto Rodriguez, Phillip Isola, and Tsung Yi Lin. iNeRF: Inverting Neural Radiance Fields for Pose Estimation. arXiv preprint arXiv:2012.05877, 2020. 2
Zehao Yu and Shenghua Gao. Fast-mvsnet. Sparse-to to-sse--t-eensee multi-viee stereo with learned propa- gation and gauss-newton refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1949-1958, 2020. 18
Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. arXiv preprint arXiv:2206.00665 2022.
3, 5, 7, 8, 15, 16
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020. 3
Xiaoshuai Zhang, Sai Bi, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Nerfusion: Fusing radiance fields for large-scale scene reconstruction. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5449-5458, 2022. 1, 6, 7, 14
Zhengyou Zhang. Microsoft kinect sensor and its effect. IEEE multimedia, 19(2):4-10, 2012. 2
Qian-Yr Zhou, Jaesik Park, and Vladlen Koltun. Open3d: A modern library for 3d data processing. arXiv preprini arXiv: /801.09877, 2018.
Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Mare Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12786-12796, 2022.
Zi-Xin Zou, Shi-Sheng Huang, Yan-Pei Cao, Tai-Jiang Mu, Ying Shan, and Hongbo Fu. Mononeu- ralfusion: Online monocular neural 3d reconstruction with geometric priors. arXiv preprint arXiv:2209.15/2 2022. 16, 18
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
In the supplementary document, we introduce more implementation details (Sec. A), a comparison with RGB-D surface reconstruction on (Sen B. Sec. B.2 and Sec. B.3), single-view novel view synthesis (Sec. B.4) and qualitative results (Sec. B.5 and Sec. B.6). We specifically discussed the importance of sampling methods w.r.t the two-stage generalizable prior training. which plays an important role in the surface representation, as described in Sec. 3.2 the main paper. We provide additional experiments including (i) quantitative results of the neural scene prior, ie., without per- scene optimization, (ii) quantitative comparisons with state-of-the-ar RGB-D surface reconstruction approaches, (iii) quantitative comparisons to the MVS-based approaches, (iv) single-view novel view synthesis, and (v) qualitative results on ScanNet and self ceff-collected data. Videos of full-size room reconstruction are included and recommended to watch.
A IMPLEMENTATION DETAILS
A.1 GENERALIZABLE NEURAL SCENE PRIOR
The generalizable neural scene prior is trained on the training split of ScanNet Dai et al. (2017a). We discuss some details of every component including the geometry encoder, texture encoder, generalizable geometric prior module, and generalizable texture prior in this section.
Geometry and Texture Encoder. For the geometry encoder, we first sample 512 keypoints from all the points projected from 2D pixels, via Farthest Point Sampling (FPS) for each frame. For each surface point, we apply the K-Nearest-Neighbor algorithm to select 16 adjacent points. Then, adopt two PointConv Wu et al. (2019) layers, to extract the geometry feature whose output channels set 64. To extract the RGB feature we use a U-Net Ronneberger al. (2015) with ResNet34 He et al. (2016) as the backbone network. We further use an additional convolutional layer to output per-point feature with the dimension as 32. All the encoder modules are jointly trained with the whole pipeline.
Generalizable Geometric Prior. Given an RGB-D image and its corresponding camera pose, we first randomly sample 256 rays from regions where depth values are valid, e.g., non-zero. Then for each ray, we define a small truncation region near the ground truth depth where 32 points are sampled uniformly. We then use two MLPs to map the geometry features to SDF values. The hyperparameters Adepth: Asar and deik are set to 1.0, 1.0 and 0.5, respectively.
Generalizable Texture Prior. Initialized with the geometric prior, we learn the texture prior via the volumetric rendering loss Wang et al. (2021a); Yariv et al. (2021). Different from the sampling strategy used in geometric prior learning, we restrict the importance sampling to the samples concentrated on the surface as described in Sec.3.4 of our main paper. In particular, we first sample 2048 rays from each RGB-D image where uniformly sample 64 points in the predefined near-far region. Following Wang et al. (2)21a), then, we sample 48 points that are close to the predicted surface. This sampling strategy is employed during both training and inference. Additionally, during training, for rays with non-zero depth values, we further sample 16 points within the truncation region around the ray's depth. Therefore, 128 points are sampled along each ray. For each point, we utilize MLPs in the texture decoder to estimate its RGB value. The hyperparamters Adepth: Andt Acik and Xzgb are set to 1.0, 1.0, 0.5 and 10.0, respectively.
Scene prior extraction and fusion. prior networks, we can directly aggregate the keypoints along with their geometry and texture features from these frames in the volumetric space. Then, the colored surface reconstruction can be decoded from the fused representation following the same procedure in Sec.3.1 and 3.2. No further learnable modules are required, in contrast, to Chen et al. (2021): Zhang et al. (2022); Li et al. (2022).
To leverage multiple views of RGB-D frames, with the scene
Prior-guided pruning and sampling. To optimize a single scene, we discard the encoders and treat the volume feature representation as learnable to be optimized together with the decoders. To further speed up the optimization, we accelerate the feature query process of sampled points, i.e., instead of optimizing the unstructured keypoints, from which the feature extraction we introduce the prior-guided voxel pruning to leverage the advantage of voxel-grit --rid sampling and surface representation. Specifically, we initialize uniform grids in the volumetric space and then query
be inefficient,
14

---

Published as a conference paper at ICLR 2024
each grid feature. Instead of optimizing large number of uniform grids, remove the redundant grids adaptively based on the geometric prior using the Algo. described below. To concentrate the sampled ray points near the surface, apply importance sampling strategy, similar to that used in training the generalizable texture prior, to mask out those far away from the surface. Starting from a large threshold at the early training stage, we decrease it gradually with more training iterations to prune more unnecessary grids. A similar procedure is also applied to the coarsely sampled points to remove some useless points and help more points concentrate around the surface region. Notably, compared to the voxel-based approach Wang et al. (2022b) having more than 4, 000, 000 uniform grids be optimized, the number of learnable keypoints in our case is around 40, 000 100x reduction in computational complexity.
Algorithm 1: Prior-guided voxel pruning
Input:
Grid feature {fi i=iNN
Grid position {x, i=I:Ni
Positional encoding 2());
Geometry decoder s(-);
Number of grids N:
Number of iterations T
Output: Grid feature after prune (fIN=
Initialization :70 :oo = 0.16
for / = 1: T do
T max(0.005, 0.8 m)
for 1 N do
8, < s
if ss 2 then
Prune i-th grid
end
end
end
B ADDITIONAL EXPERIMENTS
B.1 COMPARISON WITH RGB-D SURFACE RECONSTRUCTION ON SCANNET
Computational Resource. The geometric and texture priors network are trained on 8 NVIDIA V100 GPUs for 2 days until convergence. The per-scene optimization step is trained and tested on a single NVIDIA V100 GPU. All baselines reported in our paper are tested using the same computational resources.
Clarification of Table 2 in the main paper. Table 2 of the main paper presents the comparison of our method with ManhanttanSDF (Guo et al., 2022) and MonoSDF (Yu et al., 2022) with the depth supervision. For fairness, we keep every component of each method by involving an additional depth loss. Unlike some RGB-D surface reconstruction methods, we did not optimize the camera pose while during training. In the course of these modifications, both ManhattanSDF and MonoSDF were observed to have an architecture quite similar to NeuralRGBD Azinović et al. (2022). Given these cireumstances, we are confident that comparing our approach with ManhattanSDF and MonoSDF on ScanNet is indeed fair and effective.
Comparison with Go-surf (Wang et al., 2022b) and NeuraIRGBD (Azinović al., 2022). We compare our method with Go-surf and Neural RGB-D in Table 5. To have fair comparison, instead of optimizing camera poses and neural scene representation jointly, we fix the original camera poses as provided by ScanNet Dai et al. (2017a). Follow the same setting in the main paper, we report the performance of different models training with dense and sparse training views. As shown Table 5, approach achieves better performance over all metrics. More importantly, although Go-surf achieves similar performance within relatively similar time, it cannot produce any reasonable results without optimization demonstrated by our approach.
15

---

Published as a conference paper at ICLR 2024
Method
frames opt. time
Prect
Recall
F-scoret
Neural-RGBD Azinović et (2022)
400
0.932
0.918
0.925
Go-surf Wang et al. (2022h)
400
35
0.946
0.956
0.950
Ours
400
15
0.947
0.962
0.954
Neural-RGBD Azinović (2022)
40
240
0.837
0.855
0.846
Go-surf Wang et al. (2022h)
40
0.842
0.861
0.851
Ours
40
15
0.858
0.866
0.862
Table 5: Quantitative comparisons for mesh reconstruction on ScanNet.
Method
per-scene optim
opt.
Acc|
Compl
Pree
Recall
F-scoref
(min)
Manhattan SDF (Guo et al., 2022)
640
0.072
0,068
0.621
0.586
0.602
Mo (Yu et al., 2022)
720
0.039
0.044
0.775
0.722
0.747
Ours-prior
5
0.084
0.057
0.695
0.764
0.737
Table 6:
Quantitative comparisons of neural scene prior on ScanNet. Both Manhanttan SDF and MonoSDF require optimize specific scene for several hours, while the proposed neural scene prior achieve comparable performance without any optimizatin.
B.2 MODEL EFFICIENCY
We take Go-surf Wang et al. (2022b), which far of the most efficient offline scene reconstruc- tion approach, as the reference. Compared to it achieving average run-time of 35 mins per scene, our Neural Scene Prior network takes only 5 mins (note that the Neural Scene Prior is feed-forward network). The full pipeline leveraging the per-scene optimization stage takes an average run-time of 15 minutes, which is still obviously more efficient. More importantly, our model takes surface representation that facilitates scaling up to larger scenes, compared to dense voxels used in Go-surf. comprehensive comparison of running time can be found in Table of the main paper.
B.3 COMPARISON WITH MVS-BASED METHIODS
We show quantitative comparisons our method with the state-of-the-art approaches on surface reconstruction in Table 7. Different from what Table I reported in the main paper, mainly compare with the MVS-based methods here. For fair comparison, we follow the evaluation script used in Zou et al. (2022) for computing 3D metrics the ScanNet testing set. The top part of Table 7 includes offline methods while the middle one contains online methods with the fusion strategy. The bottom part of the table shows the methods that are finetined individual scenes. Compared to most MVS-based works that use fusion strategy, our method achieves much better results in terms of F-score and normal consistency. Moreover, our method outperforms MonoNeuralFusion Zou et al. (2022), which also performs finetuning for individual scenes, by large margin.
B.4 NOVEL VIEW SYNTHESIS
Novel View Synthesis. We show more qualitative results on novel view synthesis on ScanNet Dai al. (2017a) in Fig. 6 following the same setting described in the main paper. Both NerfingMVS (Wei et al., 2021) and Go-surf (Wang et al., 2022b) fail on scenes with complex geometry and large camera motion (bottom two rows). The generalized representation enables the volumetric rendering focus on more informative regions during optimization and improves its performance for rendering RGB images of novel views.
Single-view Novel View Synthesis. We demonstrate that NFP enables high-quality novel view synthesis from single-view input (Fig. 7, mid), which has been rarely explored especially at the scene level, and potentially enables interesting applications, e.g., on mobile devices.
B.5 QUALITATIVE RESULTS OF MESH RECONSTRUCTION
We show qualitative comparisons of our method with other baselines in Fig. 8. It demonstrates that the reconstructed mesh results of our approach are consistently more coherent and detailed than others. In addition. show the qualitative results of textured mesh for different scenes that obtained via neural scene prior in Fig. 9. More video demos of texture mesh reconstruction can be found in the supplementary video.
16

---

Published as a conference paper at ICLR 2024
Ground-truth
NerfingMVS MVS
Go-surf
Ours
Figure 6: Qualitative comparison for novel view synthesis on ScanNet.
Source view
Novel View
Ground-truth
Figure 7: Qualitative results for single-view novel view syuthesis. The left column shows the training source view, and the appearance reconstruction the novel view are reported the second column. The ground-truth images are listed at the last column as reference. Better viewed when zoomed in.
17

---

Published as a conference paper at ICLR 2024
Acc
Comp
Chamler
Precision
Recall
Fascure
NC
FastMVSNe (2020)
0.052
0.103
0,077
0.652
0,701
Poin Chen et al. (2019)
0.048
0.115
0,082
0.677
0.536
0.595
0.695
(2020)
GPMVS Hou et (2019)
0,058
0.078
0,068
0.621
0.543
0.578
0.715
DespVideeMVS Duzceker et (2021)
0.066
0.082
0,074
0.590
0.535
0.560
0.765
Transformnt Azinović
(2022)
NeuralRecon Sun et al. (2021)
0.038
0.123
0,080
0.769
0.506
0.608
0.816
MonoNeural Zou et al. (2022)
0.039
0.094
0,057
0.775
0.5U4
0.677
0.842
Ours
0,086
0.068
0.077
0,917
0.889
0.875
0.878
Table Quantitative comparisons of mesh reconstruction on ScanNet.
B.6 MESH RECONSTRUCTION THE LARGE-SCALE SCENE
Our results demonstrate that the neural scene prior propose generalize well to large-scale scenes, as shown Fig 10. In contrast to the previous four scenes, we selected larger room from ScanNet Dai al. (2017a) and applied our pre-trained model directly. The left figure in Fig 10 displays the mesh reconstruction obtained from the neural scene prior. Remarkably, our approach successfully recovers the geometry structure of the entire without requiring any optimization process. Furthermore, by optimizing the prior on this scene for only 20 minutes on single NVIDIA V100 GPU, we were able achieve high-quality mesh reconstruction.
with very sparse views (60 frames),
B.7 MESH RECONSTRUCTION ON THE SELF-CAPTURED SCENE
To further demonstrate the robustness of the neural scene prior, we evaluate the pretrained model on self-captured living room, and the reconstructed mesh w./w.o texture is shown in Fig. 11. Impressively, even without per-scene optimization, the proposed neural scene prior capable of feasibly reconstructing textured mesh.
C LIMITATION
The proposed neural scene prior could extract the geometric and texture prior for arbitrary scenes, but it does require the sparse RGB-D images the input. To adapt this neural scene prior for RGB images, one possibility would be to initially create a sparse point cloud using Structure from Motion (SfM) on RGB images. However, as of our submission time, we haven't yet experimented with this particular setup. Exploring this pathway in future research could certainly yield intriguing findings.
D REPRODUCIBILITY STATEMENT
All experiments in this paper are reproducible. We are committed to releasing the source codes once accepted.
E USE OF EXISTING ASSETS.
As mentioned in the NeurlPS 2023 checklist, we describe the existing assets we used in our paper and the corresponding license of these assets.
F PERSONAL DATA AND HUMAN SUBJECTS
The dataset does not include the facial or other identifiable information of humans.
G ETHICAL CONCERNS.
The datasets used are standard benchmarks proposed in previous works. Despite applying supervised learning, there may still be potential bias in model trained with these datasets.
18

---

Published as a conference paper at ICLR 2024
Manhattan SDF*
MonoSDF
Ours
GT
Figure 8: Qualitative comparisons of mesh reconstruction on ScanNet. Selected local regions are highlighted by the orange bounding box. Better viewed when zoomed in.
Datasets Most of the experiments are conducted on ScanNet dataset and 10 synthetic scenes collected by Dai et al. (2017a) and Azinović et al. (2022) which are released on their official website and public to everyone for non-commercial use.
Code. Our code built upon the Pytorch Paszke al. (2019). And we leverage the code from the released codes by nerfstudio Tancik et al. (2023) under the Apache License.
19

---

Published as a conference paper at ICLR 2024
O Figure 10: Mesh reconstruction results on the large-scale scene.
Figure 9: Qualitative results of Neural Scene Prior on ScanNet.
Scmne Prior
Per-scene Optimization
Figure 11: Mesh reconstruction results on the self-collected scene without any optimization.
20


Reviews:
[{"review_id": "KhBmvFycu4", "reviewer": "Reviewer_s8AK", "summary": "Input: One or more RGB-D images of an indoor scene\n\nOutput: Textured 3D mesh representing the indoor scene\n\nThe paper presents a generalizable neural framework, called Neural Field Priors (NFPs), for reconstructing 3D indoor scenes from a single as well as multiple RGB-D input images of the scene. Scene priors are obtained from depth map inputs, given posed RGB-D images. Results show significant performance improvement, especially on single-view 3D scene reconstruction, both in terms of speed and reconstruction quality.\n\nThe main contributions of the paper are two folds: (a) developing a two-stage generalizable neural framework using scene priors (i.e., not restricted to per-scene training) that is scalable to large-scale scenes, and (b) reconstructing the 3D scene by merging multiple view images (it is the features that are actually merged) in the volumetric space without using a fusion module. \n\nThe two-stage framework consists of a generalizable geometric prior and a generalizable texture prior. \nThe first network, which is the Geometric Prior network, is responsible for obtaining a signed distance field of the underlying scene. This is done by what are called Geometry Objectives and Surface Regularization. Geometry Objectives are based on the depth values. First, a pixel-wise rendering loss on depth maps is enforced to make the depth predictions at points sampled along a ray as close to the GT as possible (Importance sampling is used). The features of the points at the sampled locations are obtained using a modified form of weighted interpolation of surface-point geometric features. The surface points are nothing but the projection of depth image to 3D, and their features, a.k.a geometric features, are obtained by PointConv network. These interpolated point features, along with the point locations and their positional encoding, are passed through an MLP (called the Geometric decoder) to obtain signed distance values at the respective points. The signed distance value at a point is approximated to the GT SDF value by comparing the predicted SDF value with the difference of GT depth and predicted depth at that point. Surface Regularization is used for regularizing the SDF predictions to avoid artifacts. This is the Eikonal loss, which is a standard regularization term used in prior volume-rendering-based 3D reconstruction works.\n\nThe second network, which is the Texture Prior network, uses the SDF predictions from the first network as geometric initialization. The goal here is to learn RGB values for sampled points along the ray for which SDF values have been predicted by the Geometric Prior network. The texture features are a modified version of weighted interpolated surface-point texture features. The texture features here refer to convolutional features (image pixels to 3D correspondence yields surface-point convolutional features). These interpolated texture features for the points, along with the point locations and their positional encoding, are passed through an MLP (called the Texture decoder) to estimate the color at the respective points. During this process, the Geometric Decoder along with the PointConv encoder is jointly learned. So the loss for the Texture Prior network is the Geometric Prior loss plus the RGB loss.\n\nWhen multi-view images are used as input, Geometric and Conv features of these images are merged during the reconstruction process. This avoids the burden of learning fusion modules to fuse reconstruction results from multiple views, thereby making the training efficient (less training complexity). As well, it is claimed that it allows for flexible data processing (I have a few questions on this in the Questions section).\n\nDataset used:\nScanNet_v2 and 10 synthetic scenes from Azinovic et al. 2022\n\nUnderlying Neural Network:\nGeometry encoder: PointConv\nImage Encoder: U-Net\nDecoders are MLPs\nVolume rendering is what makes this possible\n\nLoss function: \nL_depth (L1 loss), L_sdf (L1 loss), L_surface (Eikonal loss), L_rgb (L2 loss)\n\nQuantitative Metric:\n3D scene mesh reconstruction – Accuracy, Completeness, Precision, Recall, F-score\nNovel view synthesis – PSNR (Power Signal-to-Noise ratio), SSIM (Structural Similarity for Image comparison metric) and LPIPS (Learned Perceptual Image Patch Similarity metric)", "questions": "1)\tIt is claimed in the second+third paragraph of the Introduction that not having a fusion module to handle multi-view images during training allows for flexible data processing. This is not substantiated and remains unclear to me. Can you elaborate how this is the case?\n2)\tIs there a reason behind using PointConv as geometric feature embedding network? Were alternative networks (like PointNet, DGCNN etc.) tried? Using different networks should have a bearing on the overall result. Were any experiments conducted to understand this? \n3) This has been touched upon but I would like to ask again -- what makes the proposed approach work on single images? To what extent are the reconstructed scenes reasonable? Or put differently, what \"gaps\" needs to be \"filled in\" to make the reconstruction results (from single input image) better than what can be currently achieved?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1)\tGeneralized framework for scene reconstruction using radiance fields (i.e., no per=scene training) from relatively few input views (relative to existing literature)\n\n2)\tAbility to reconstruct a 3D scene by merging individual frames in the volumetric space without a learnable fusion module\n\n3)\tNovel view synthesis from single-view input beats existing works\n\n4)\tSimple interpolation strategy for obtaining point features. Making use of surface points instead of dense volumetric grids for obtaining sampled point features\n\n5)\tThe paper is well-written", "weaknesses": "1)\tThe dependency on depth maps is limiting since such data is not always available. This is also a drawback of MonoSDF and other works that use additional priors for scene reconstruction, beyond just RGB images\n2)\tThe intuition behind the approximation of GT SDF values by observing depth values along a ray is unclear. This may result in erroneous signed distance predictions. An explanation of this is lacking. I am actually interested in this ablation experiment\n3)\tResults on images in the wild are missing. This will add value to the work\n4)\tLimited quantitative results\n5)\tAblation experiments are not thorough, in terms of the different components involved in Geometric and Texture Prior Networks\n6)\tDiscussion on limitations is lacking", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4mQOxz5eAg", "reviewer": "Reviewer_EoMe", "summary": "The paper addresses high-fidelity 3D reconstruction using cross-dataset (ie, -scene in particular) generalization using the popular conjunction of neural radiance fields and signed distance functions.\n\nThe proposed method shows favorable comparative results on Scannet, a well established public benchmark in the field against RGB and RGBD based baselines.", "questions": "The main questions I would have cover the aforementioned weaknesses that have been pinpointed. \n\nBesides those remaining grey areas, I would be happy to bump my initial rating were they to be addressed accordingly.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ ## Readability.\nAs it currently stands, the paper is very well written. The main ideas and concepts are mostly well explained and articulated throuthout.\n\n+ ## Organization of the contents and overall paper structure.\nThe contents are also very well structured and balanced.\n\n+ ## Related work section and discussion. \nIt is very well structued, articulated and populated with very relevant and up to date references.\n\n+ ## The disclosed performance of the proposed method is at the very least competitive and promising.\n\n+ ## The problem at hand (scene level generalization) is an important, impactfull one in the field.", "weaknesses": "+  ## 1. Missing bits of context information - How much does it cost?\nWhile indicative timings and thorough implementation details (in supMat) are provided, information regarding the resource usage, model size and complexity are yet underdescribed.\n\nA comparative disclosure of such information covering the main experimental baselines that are considered would help the reader better assess its relative positioning throughout the typical criteria.\n\nMentioning where the computation bottlenecks lie in terms of components would also be valuable in order to fully assess the practical usefullness of the proposed sequential pipeline, beyond rough timings (eg, Fig 3).\n\n+  ## 2. Comparative evaluation - Baselines and Benchmarks.\n\nWhile very recent work (eg, CVPR 2023) have been included in the setup, eg, HelixSurf (Liang et al.), there are a few missing players that currently hine by their absence.\n\nFor example, the RGB based references:\n\n-- Li, Z., Müller, T., Evans, A., Taylor, R. H., Unberath, M., Liu, M. Y., & Lin, C. H. (2023). Neuralangelo: High-Fidelity Neural Surface Reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8456-8465).\n\n-- Darmon, F., Bascle, B., Devaux, J. C., Monasse, P., & Aubry, M. (2022). Improving neural implicit surfaces geometry with patch warping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6260-6269).\n\n-- Zhang, J., Yao, Y., Li, S., Fang, T., McKinnon, D., Tsin, Y., & Quan, L. (2022). Critical regularizations for neural surface reconstruction in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6270-6279).\n\n-- Wang, Y., Skorokhodov, I., & Wonka, P. (2022). Hf-neus: Improved surface reconstruction using high-frequency details. Advances in Neural Information Processing Systems, 35, 1966-1978.\n\nSimilarly, the DTU public benchmark could have been envisionend, albeit partially just to compare other key papers from the state-of-the-art without having to re-run their public implementations. \n\nThe same goes for the Tanks and Temples public benchmark as well. At least one additional benchmark would have been a reasonable addition.\n\nThis would help better - and more thoroughly - assess the relative positioning of the proposed contribution, performance-wise.\n\n+  ## 3. The aforementioned references also lack in qualitative discussion and Related Work.\n\nThis is a direct consequence of (2) above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uMCKE0R7io", "reviewer": "Reviewer_2i3p", "summary": "This paper presents a generalizable 3D reconstruction framework from RGB-D sequences for indoor scenes. The key motivation is to design separate, progressive stages to learn the geometry field and color field. Experiments on various datasets have demonstrated the effectiveness of the design.", "questions": "I would consider to improve my rating if the authors can address my concern especially on the novelty of this work presented in the weakness part.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) Overall, the paper is well written and easy to follow.\n\n(2) The paper demonstrates comprehensive experiments and compares with different state-of-the-art (SOTA) methods, to highlight the advantage of the proposed method.", "weaknesses": "(1) To me, the novelty of this paper is limited. The key design of the geometry prior module is similar to PointNeRF (also employs a distance-wise feature aggregation from 3D point clouds). Also, learning a geometric prior (SDF) then pruning to facilitate the texture field is applied in previous neural 3D reconstruction methods such as NeRFusion and SparseNeuS. \n\n(2) The paper only conducts experiments on RGB-D sequences to demonstrate the generalizability. To me, the technical impact would be much higher if it also works well a RGB sequences, where obtaining precise geometry is challenging. For RGB-D sequences, some classic methods such as BundleFusion and COLMAP-MVS can achieve superior generalizability across different environments without any training.  \n\n(3) The advantage over Go-Surf is not convincing on both accuracy and speed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fc4m5yr00b", "reviewer": "Reviewer_ZqzA", "summary": "This works proposes a scene reconstruction and novel-view synthesis method by learning scene priors that leverage ground-truth RGB-D data. The proposed novel method allows to efficiently integrate features from multiplve views in order to obtain an implicit neural representation of the scene's geometry and texture. This scene representation can further be used to render images from novel viewpoints. Experimental results on ScanNetV2 show how it outperforms many state-of-the-art scene reconstruction methods while using fewer images and less computation time, the latter thanks to a accurate initial estimate of the scene representation before the optimization step. For novel-view synthesis evaluated on real-world scenes, results shows this method is comparable or better than a number of well-known methods in the literature such as NeRF and IBRNet.", "questions": "**A number of questions and suggestions:**\n\n- It's not clear what is the protocal for making a result bold in Table 1, making it diffucult to quickly see what performs best on each section of rows.\n\n- How exactly is importance sampling used? I couldn't find a technical explanation for how it's used. In particular, how is it used for novel-view synthesis tasks? Is it only used during training of the networks? In other words, for evaluation, I understand rendering of a novel view is done by uniform sampling of the ray?\n\n- The term generalizable is used extensively and I'm not sure what they refer to in many cases (e.g. generalizable features, generalizable representations and generalizable losses). I would kindly ask the authors to either reduce the use of the term or be a bit more precise when using it.\n\n**Some missing references.**\n\nOn learning based novel-view synthesis.\n- Trevithick, Alex, and Bo Yang. \"Grf: Learning a general radiance field for 3d representation and rendering.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n- Yu, Alex, et al. \"pixelnerf: Neural radiance fields from one or few images.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\nOn using GT depth for guiding training geometric and colour scene functions.\n- Stelzner, Karl, Kristian Kersting, and Adam R. Kosiorek. \"Decomposing 3d scenes into objects via unsupervised volume segmentation.\" arXiv preprint arXiv:2104.01148 (2021).", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "**Novelty and significance**\nThis work tackles two challenging tasks with one method: scene reconstruction and novel-view-synthesis. The authors propose a novel combination of techniques with clear advantage on some aspects compared to other state-of-the-art methods in each of the two tasks. It makes use of efficient representations (3D keepoints), and  is designed to work with any number of input views regardless of however many are used for training. It also leverages depth ground-truth very well in a two-stage pipeline. I believe the method itself is a solid contribution to the vision community.\n\n**Soundness of method**\n\nThe presented methods are generally sound and the benefits of its design are clear. In general it makes a number of useful/practical design choices based on the types of indoor scenes it's applied to.\n\n**Experimental results**\n\nExperimental results on a real-world dataset, ScanNetV2, demonstrate strong results for scene reconstructions as well as novel-view synthesis in complex scenes. It achieves SoTA reconstruction performance while using fewer input images than a number of related methods, and similarly its novel-view synthesis performance is also better than strong baselines.\nFinally, the ablation studies in the paper show that each of the learned priors is important to the overall performance.", "weaknesses": "**Presentation**\n\nThe paper needs to be substantially proof-read, as it is it's not ready for publication.\n\n\n**Author claims**\n\nThe authors make a number of claims, namely:\n- Per-scene optimization-based methods are \"not scalable, inefficient, and unable to yield good results given limited views\". I am not sure what they refer to by scalable, and also not sure evidence is presented to support the claim.\n- Learning-based multi-view stereo methods, \"their multi-view setting makes it less flexible to scale up and to broad applications\". Again, I'm not sure evidence is shown that these methods do not scale up and are less broadly applicably. Perhaps precise pointers to the results or references would help.\n\n**Experiments**\n- Lack of expeirments with sparse views. Given that the method can technically reconstruct a scene it would be interesting to test it on these settings. It would be interesting to see the performance curve as a function of input views.\n- Restricted to a single dataset (ScanNetV2). While it's certainly a good dataset to evaluate on, as it is based on real-world scenes, it would have been useful to see results on other recently used NVS datasets even if synthetic.\n- Novel view synthesis experiments are evaluated on a small number of scenes only, with potentially high variance of results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DkvlMaHy4p", "reviewer": "Reviewer_zLQa", "summary": "The paper proposes to learn a generalizable neural fields as a scene prior for 3D indoor scene reconstruction. The authors first sample a sparse point cloud from reprojected depth and use PointConv for extract per-point geometry feature which are then interpolated to a feature vector for any input point. Similarly, they use a CNN to extract texture features from the RGB image and splat the feature to the sampled 3D points. An MLP decoder predict the SDF and view dependent appearance from the point features. With the predicted SDF and appearance, they use volume rendering to render a depth/color for any input rays. The encoder and decoder are pretrained in a large scale dataset. After training, the encoder and decoder predict the SDF and appearance for new scenes without per-scene optimization. Of course, per-scene optimization improve the results as shown in the experiments.", "questions": "1. One very simple baseline is missing: simply use TSDF fusion with the GT depth map for geometry reconstruction.\n2. Why not also use CNN for the depth map and then use the similar method to get the geometry feature for the point cloud?\n3. In Table 3. most of the baseline are not using depth as input, might be good to make it clear. \n4. In the direct fusion part, points from different views are directly concatenated, will this change the distribution of point density and therefore change the K neighbouring points during inference? \n5. Typo in the first paragraph \"Although these results are encouraging, Although these results are encouraging\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method can do direct fusion by simply concatenate sampled point cloud from each input frame without additional fusion module thanks to the sparse point cloud representation of the scene.\n2. The experimental results are extensive and show that the proposed method works well on both sparse input and dense input with faster convergence speed.", "weaknesses": "1. The model is not scale/rotation/translation invariant. I think the main reason is the use of point position as input to the decoder, which means if the coordinates system is changed, the output of the decoder is also changed. Similarly, the surface normal or view directions should also be in the local coordinates system, otherwise the output will change if the scene was translated. I wonder how sensitive of the current model to random rotation/translation. Would be great to have an ablation study. A simple solution is just don't use point position as input to the network. I am also interested in how this performs. \n\n2. Would be great to cite and discuss [1] as it's very related to the paper. The paper already included many baselines so this is not a minus point. \n\n[1] Fast Monocular Scene Reconstruction with Global-Sparse Local-Dense Grids", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)