MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction
Authors: Zehao Yu
Songyou Peng
Michael Niemeyer
Torsten Sattler
Andreas Geiger
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D Reconstruction
Neural Implicit Representations
Neural Rendering
Multi-view Reconstruction
Neural Implicit Surfaces
Monocular Priors

Abstract: In recent years, neural implicit surface reconstruction methods have become popular for multi-view 3D reconstruction. In contrast to traditional multi-view stereo methods, these approaches tend to produce smoother and more complete reconstructions due to the inductive smoothness bias of neural networks. State-of-the-art neural implicit methods allow for high-quality reconstructions of simple scenes from many input views. Yet, their performance drops significantly for larger and more complex scenes and scenes captured from sparse viewpoints. This is caused primarily by the inherent ambiguity in the RGB reconstruction loss that does not provide enough constraints, in particular in less-observed and textureless areas. Motivated by recent advances in the area of monocular geometry prediction, we systematically explore the utility these cues provide for improving neural implicit surface reconstruction. We demonstrate that depth and normal cues, predicted by general-purpose monocular estimators, significantly improve reconstruction quality and optimization time. Further, we analyse and investigate multiple design choices for representing neural implicit surfaces, ranging from monolithic MLP models over single-grid to multi-resolution grid representations. We observe that geometric monocular priors improve performance both for small-scale single-object as well as large-scale multi-object scenes, independent of the choice of representation. 

MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction
Zehao Yu!
Songyou Peng
Michael Niemeyer 1.3
Torsten Sattler*
Andreas Geiger 1.3
1 University of Tübingen
2 2TTH Zurich
MPI for Intelligent Systems, Tübingen
4Czech Technical University in Prague
http://muuiiihuucoongggguhhhhhhh
Abstract
In recent years, neural implicit surface reconstruction methods have become popu- lar for multi-view 3D reconstruction. In contrast to traditional multi-view stereo methods, these approaches tend to produce smoother and more complete recon- structions due to the inductive smoothness bias of neural networks. State-of-the-art neural implicit methods allow for high-quality reconstructions of simple scenes from many input views. Yet, their performance drops significantly for larger and more complex scenes and scenes captured from sparse viewpoints. This is caused primarily by the inherent ambiguity in the RGB reconstruction loss that does not provide enough constraints, in particular in less-observed and textureless areas. Motivated by recent advances in the area of monocular geometry prediction, we systematically explore the utility these cues provide for improving neural implicit surface reconstruction. We demonstrate that depth and normal cues, predicted by general-purposs monocular estimators, significantly improve reconstruction quality and optimization time. Further, we analyse and investigate multiple design choices for representing neural implicit surfaces, ranging from monolithic MI.P models over single-grid to multi-resolution grid representations. We observe that geometric monocular priors improve performance both for small-scale single-object as well large-scale multi-object scenes, independent of the choice of representation.
Introduction
3D reconstruction from multiple RGB images is a fundamental problem in computer vision with various applications in robotics, graphics, animation, virtual reality, and more. Recently, coordinate- based neural networks have emerged powerful tool for representing 3D geometry and appearance. The key idea is to use compact, memory efficient multi-layer perceptrons (MLPs) to parameterize implicit shape representations such occupancy signed distance fields. While early works [8,37, 44] relied on 3D supervision, several recent works [41,60, 60]75] use differentiable surface rendering to reconstruct scenes from multi-view images. At the same time, neural radiance fields (NeRFs) [38] achieved impressive novel view synthesis results with volume rendering techniques. [43, 69, 74]
combine surface and volume rendering for the task of 3D reconstruction by expressing volume density as function of the underlying 3D surface, which in turn improves scene geometry.
Current neural implicit-based surface reconstruction approaches achieve impressive reconstruction results for simple scenes with dense viewpoint sampling. Yet, as shown the first row of Fig. 1, they struggle in the presence of limited input views (DTU with 3 views) or for scenes that contain large textureless regions (walls in ScanNet or Tanks & Temples). A key reason for this behavior that these model optimized using per-pixel RGB reconstruction loss. Using only RGB images as input leads to an underconstrrined problem as there exist an infinite number of photo-consisten
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

VoISDF [74]
Monocular Cues
DTU (3 views)
ScanNet (464 views)
Tanks & Temples (298 views)
Figure MonoSDE. Top: State-of-the-ard neural implicit surface reconstruction methods fail in the presence of limited input views or when applied to complex multi-object scenes. Bottom: We demonstrate that incorporating geometric cues from general-purpos monocular predictors enables scaling to larger scenes while yielding more accurate reconstructions and speeding up optimization. An image resolution of 384 x 384 pixels was used for all results shown above.
explanations [4, 80]. Previous works address this problem by incorporating priors on the structure of the scene into the optimization process, e.g., depth smoothness [40], surface smoothness (43,81], semantic similarity [27], or Manhattan world assumptions [18]. In this paper, explore monocular geometric priors as they are readily available and efficient to compute. We show that using such priors significantly improves 3D reconstruction quality in challenging scenarios (see second row of Fig. 1).
Estimating geometric cues such as depth and normals from a single image has been an active research area for decades. The seminal work by Eigen al. [15, 16] showed that learned models based on deep convolutional neural networks (CNNs) significantly improved over early work in this area [21-24, 53-55J. Recent work [14,49,55 49,500, J, in particular Omnidata [14], has made significant headway in terms of prediction quality and generalization to new scenes using very large datasets for training. These strong results on individual images, and the fact that monocular geometric cues can be computed efficiently, naturally lead to the question whether such models are able to provide the additional constraints required by implicit neural surface reconstruction approaches to handle more challenging settings.
This paper describes a framework, called MonoSDF, for integrating monocular geometric priors into neural implicit surface reconstruction methods; given multi-view images, we infer depth and surface normals for each image, and use them as additional supervision signals during optimization together with the RGB image reconstruction loss. We observe that these priors lead to significant gains in reconstruction quality, especially textureless and less-observed areas as shown in Fig. This due to the fact that the photometric consistency cues used by surface reconstruction methods and the recognition cues used by monoeular networks are complementary: while photometric consistency fails in texturless regions such walls, surface normals can be predicted reliably in these areas due to the structured 3D scene layout, Conversely, photoconsistency cues allow for establishing globally accurate 3D geometry in textured regions, while normal and (relative) depth cues only provide local geometric information.
Apart from incorporating monocular geometric cues, we provide systematic study and analysis of ofate-of-the-art design choices for coordinate-based neural representations in the context of implicit surface reconstruction. More specifically, we investigate the following architectures: single, large MLP [43, (9,,69,77 74, 75], dense SDF grid [28], single feature grid 225,33,47,44 and multi-resolution feature grids [9, 19,39, 39,3,, 63, 82]. We observe that MLPs act globally and exhibit an inductive smoothness bias while being computationally expensive to optimize and evaluate. In contrast, grid-based representations benefit from locality during optimization and evaluation, hence

---

they are computationally more efficient. However, reconstructions are noisier for sparse views or less-observed areas. Including monocular geometric priors improves neural implicit reconstruction results across different settings with faster convergence times and independent of the underlying representation.
In summary, we make the following contributions:
We introduce MonoSDF, novel framework which exploits monocular geometric cues to improve multi-view 3D reconstruction quality, efficiency, and scalability for neural implicit surface models.
We provide a systematic comparison and detailed analysis of design choices of neural implicit surface representations, including vanilla MLP and grid-based approaches.
We conduct extensive experiments multiple challenging datasets, ranging from object-level reconstruction on the DTU dataset [1], over room-level reconstruction on Replica [61] and Scan- Net [12], to large-scale indoor scene reconstruction on Tanks and Temples [30].
2 Related Work
Architectures for Neural Implicit Scene Representations. Neural implicit scene representations or neural fields [71] have recently gained popularity for representing 3D geometry due to their expressiveness and low memory footprint. Seminal works [8,37,4] 44] use single MLP as the scene representation and show impressive object-level reconstruction quality, but they do not scale to more complicated or large-scale scenes due to the limited model capacity. Follow-up works [9, 19,36, 39,48, 63, 82] combine an MLP decoder with one or multi-level voxel grids of low low-dimensional features. Such hybrid representations are able better represent fine geometric details and can be evaluated fast. However, they lead to larger memory footprint with increasing scene size. In this paper we provide systematic comparison of four architectural design choices for implicit surface reconstruction.
3D Reconstruction from Multi-view Images. Reconstructing the underlying 3D geometry from multi-view images is a long-standing standing goal of computer vision. Classic multi-view stereo (MVS) methods [2. 31, 31, 56, 58, 59] consider either feature matching for depth estimation [5, 56] or represent shapes with voxels [2, 6, 7,31,,1,4 4,, 58, 66]. Learning-based MVS methods usually replace some parts of the classic MVS pipeline, e.g., feature matching 32, 77,, depth fusion [13,51] or inferring depth from nulti-view images [26, 73, (7,. In contrast to the explicit scene representations used by classic MVS algorithms, recent neural approaches [34,42, 75] represent surfaces via single MLP with continuous outputs. Learned purely from posed 2D images, they show appealing reconstruction results and do not suffer from discretization. However, accurate object masks required. Inspired by the density-based based volume rendering in NeRF [38], which demonstrated impressive view synthesis without object masks, several works [43, 1133,69777 69,74] use volume rendering for neural implicit surface reconstruction without masks. However, these methods lead to poor results in large-scale scenes with textureless regions. In this work, we show that incorporating monocular priors allows these approaches obtain significantly more detailed reconstructions and to scale to larger and more challenging scenes.
Incorporating Priors into Neural Scene Representations. Several researchers proposed to in- corporate priors such as depth smoothness [40]. semantic similarity [27]. or sparse MVS point clouds [52] for the task of novel view synthesis from sparse inputs. In contrast, in this work, our focus is on implicit 3D surface reconstruction. Concurrently, Manhattan-SDF [18] uses dense MVS depth maps from COLMAP [57] supervision and adopts Manhattan world priors [10] handle low-textured planar regions corresponding to walls, floors, etc. Our approach is based on the observa- tion that data-driven monocular depth and normal predictions [14] provide high-quality priors for the full scene. Incorporating these priors into the optimization of neural implicit surfaces not only removes the Manhattan world assumption [10] but also results in improved reconstruction quality and a simpler pipeline.' Compared to NeuRIS [68], a concurrent work that proposes to use normal priors for indoor scene reconstruction, we integrate monocular depth cues and further demonstrate the effectiveness of monocular cues on various neural scene representations, ranging from MLP to multi-resolutiin feature grids.
Manhattan-SDD SDF [18] requires semantic segmentation to determine where to enforce the assumption.
3

---

MLP
Derce SDP Grid
Single Fetture Grid
Multi Fenture Grids
f
Neural Implicit Scene Representation
Volurne Rendering
C(e)
D(e)
N
Ldspth
Cnarmal
Ray Distance
Input Vieus
Ounidatt
D
R
Mixiel
Figure 2: Overview. In this work monocular geometric cues predicted by general-purpor
pretrained network to guide the optimization of neural implicit surface models. More specifically, for batch of rays, volume render predicted RGB colors, depth, and normals, and optimize wrt. the input RGB images and monocular geometric cues. Further, we investigate different design choices for neural implicit architectures and provide an in-depth analysis. For clarity, only show the SDF and not the color prediction branch above.
Method
Our goal is recover the underlying scene geometry from multiple posed images while utilizing monocular geometric cues to guide the optimization process. To this end, we first review neural implicit scene representations and various design choices Section 3.1 and discuss how to perform volume rendering of these representations in Section 3.2. Next, we introduce the monocular geometric cues we investigate in our study Section 3.3 and discuss loss functions and the overall optimization process in Section 3.4. An overview of our framework is provided in Fig.
3.1 Implicit Scene Representations
We represent scene geometry as a signed distance function (SDF). A signed distance function is continuous function f that, for given 3D point, returns the point's distance the closest surface:
f R R
SSF(x))
Here, is the 3D point and denotes the corresponding SDF value. In this work, we parameterize the SDF function with learnable parameters 0 and investigate several different design choices for representing the function: explicit as a dense grid of learnable SDF values, implicit as single MLP, or hybrid using an MLP in combination with single- or nulti-resolution feature grids.
Dense SDF Grid. The most straightforward way parameterizing an SDF is to directly store SDF values in each cell of discretized volume G3 with resolution of Rh x Rw x Rp [28]. To query the SDF value for an arbitrary point x from the dense SDF grid, we can use any interpolation operation:
interp(x, Go)
In our experiments, we implement interp trilinear interpolation. Single MLP. The SDF function can also be parameterized by single MLP [44] fo:
fo(?(x))
where is the predicted SDF value and y corresponds to a fixed positional encoding [38,64] mapping higher dimensional space. After their introduction to novel view synthesis [38], positional

---

encoding functions are now widely used for neural implicit surface reconstruction [43, 74, as they increase the expressiveness of coordinate-bacn networks [64].
Single-Resolution Feature Grid with MLP Decoder. We can also combine both parameterizations
and use feature-conditioned MLP together with a feature grid 4 with resolution of Rt, where each cell of the grid stores a feature vector [25, 33,44, 63] instead of directly storing SDF values:
8 (x)) interp(x,, 2b))
(4)
Note that the MLP So is conditioned on the interpolated local feature vector from the feature grid 4g. Mustti-Rssolution Feature Grids with MLP Decoder. Instead of using single feature grid do, one can also employ multi-resolution feature grids ( {8/) with resolutions R² [9, 19,39,63, 62]. The resolutions sampled in geometric space [39] to combine features at different frequencies:
In Rmax In Romin
Rı
b: exp
where Rmin Rmax are the coarsest and finest resolution, respectively. Similarly, we extract the interpolated features at each level and concatenate them together:
8 = fo (x)))x), (interp(x, bb)))))
As the total number of grid cells grows cubically. we use a fixed number of parameters to store the feature grids and use a spatial hash function index the feature vector finer levels [39] (see supplementary for details).
Color Prediction. In addition to the 3D geometry, we also predict color values such that model can be optimized with a reconstruction loss. Following [75], we therefore define a second function Cæ
cx v. n.
that predicts RGB color value for 3D point and viewing direction The 3D unit normal ñ the analytical gradient of our SDF function. The feature vector 2 the output of a second linear head of the SDF network in [75]. We parameterize Cą with two-layer MLP with network weights 0. In case of the dense grid SDF parameterization, we similarly optimize dense feature grid and obtain the feature vector via the interpolation function interp.
3.2 Volume Rendering of Implicit Surfaces
Following recent work 73,,55,4,755), we optimize the implicit representations described in Section 3. via an image-based reconstruction loss using differentiable volume rendering. More specifically, to render a pixel, we cast ray r from the camera center o through the pixel along its view direction sample M points x = o + (.v along the ray and predict their SDF $E and color values éf. We follow [74] to transform the SDF values &f to density values of for volume rendering:
exp
if
(s(s)
(1- exp
if 0
where 3 is learnable parameter. Following NeRF [38], the color Č(r) for the current ray is computed via numerical integration:
M
((r) = Σ T ofe
T II (1- 1=1
of = exp (-o
where Tř and of denote the transmittance and alpha value of sample point i along ray r, respectively, and 62 is the distance between neighboring sample points. Similarly, we compute the depth D(r) and normal (r) of the surface intersecting the current ray
M
D(r) = Σ
Te C
( = Σ T
(10)
1=1
5

---

3.3 Exploiting Monocular Geometric Cues
Unifying volume rendering with implicit surfaces leads to impressive 3D reconstruction results. Yet, this approach struggles with more complex scenes especially in textureless and sparsely covered regions. To overcome this limitation, we use readily available, efficient-toccompuee monocular geometric priors thereby improving neural implicit surface methods.
Monocular Depth Cues. One common monocular geometric cue a monocular depth map, which can be easily obtained via an off-the-shell monocular depth predictor. More specifically, we use pretrained Omnidata model [14] to predict depth map D for each input RGB image. Note that the absolute scale is difficult to estimate in general scenes, so D must be considered as relative cue. However, this relative depth information provided also over larger distances in the image.
Monocular Normal Cues. Another geometric cue we use is the surface normal. Similar to the depth cues, we apply the same pretrained Omnidata model acquire normal map N for each RGB image. Unlike depth cues that provide semi-local relative information, normal cues are local and capture geometric detail. We hence expect that surface normals and depth are complementary to each other.
3.4 Optimization
Reconstruction Loss. Eq. (9) provides a linkage from the 3D scene representation to 2D observa- tions. We can therefore optimize the scene representation with simple RGB reconstruction loss:
Libb Ell0(r) c(r)l, FER
(11)
Here R denotes the set of pixels/rays in the minibatch and C(r) is the observed pixel color. Eikonal Loss. Following common practice, we also add an Eikonal term [17] on the sampled points to regularize SDF values in 3D space
Leikonal
Σ [(IV 1)²
(12)
XEX
where X' are a set of uniformly sampled points together with near-surface points [74]. Depth Consistency Loss. Besides Ligh and Leckonal, we also enforce consistency between our rendered expected depth Đ and the monocular depth D:
Leepth = Σ
+ - -[( -
(13)
FER
where and g are the scale and shift used to align Đ and D since D is defined only up to scale. Note that these factors have be estimated individually per batch as the depth maps predicted for different batches can differ in scale and shift. Specifically. we solve for w and with least-squares criterion [16,50] which has a closed-form solution (see supplementary for details).
Normal Consistency Loss. Similarly, we impose consistency on the rendered normal N and the predicted monocular normals N transformed to the same coordinate system with angular and I.1 losses [14]:
Lnorall
- -(t), + |1- (x) N())ll
(14)
PER
The overall loss optimize our implicit surfaces jointly with the appearance network is:
C= Ligh + Xn Ceikonal + Ae
A AgLommma
(15)
Implementation Details. We implement our method in PyTorch [46] and use the Adam opti- mizer [29] with a learning rate of 5e-4 for neural networks and 1e-2 for feature grids and dense SDF grids. We set λT, A2, X3 to 0. 0.1, 0.05, respectively. We sample 1024 rays per iteration and apply the brooded sampling strategy introduced by [74] to sample points along each ray. For MLPs and feature grids, we adapt the architecture and initialization scheme from [74] and [39], respectively. For obtaining monocular cues, we first resize each image and center crop it to 384 x 384, which we then feed input to the pretrained Omnidata model [14]. See supplementary for more details.

---

Dense SDF Grid
MLP
Single-Res. Fea. Grid
Multi-Res. Fea. Grids
Ground Truth
Figure 3: Architectural Ablation Study. Comparing different design choices for neural implicit surface representations, observe that dense SDF grid leads noisy reconstructions due to missing smoothness bias. The MLP and the Single-Res. Fea. Grid improve results, but geometry tends be overly smooth with missing details. The best results are obtained using Multi-Res. Fea. Grids.
4
Experiments
We first analyze different architectural design choices and perform ablation studies wrt. monocular cues and optimization time room-level dataset (Replica) with perfect ground truth. Next, provide qualitative and quantitative comparisons against state-of-the-ar baselines on real-world indoor scenes. Finally, evaluate our method on object-level reconstruction for both sparse input and dense input scenarios.
Datasets. While previous neural implicit-based reconstruction methods mainly focused on single- object scenes with many input views, in this work, we investigate the importance of monocular geometric cues for scaling more complex scenes. Thus consider: a) Real-world indoor scans: Replica [61] and ScanNet [12]; b) Real-world large-scale indoor scenes: Tanks and Temples [30]
advanced scenes; Object-level scenes: DTU [1] in the sparse 3-view setting from [40,76]. 76].
Baselines. We compare against a) state-of-the-arr neural implicit surfaces methods: UNISURF [43],
VoISDF [74], NeuS [69], and Manhattan-SDFF [18]. b) Classic MVS methods: COLMAP [56] and a state-of-the-an commercial software (RealityCapture". c) TSDF-Fusion [11] with predicted monocular depth cues, where GT depth maps are used to recover the scale and shift values (cf.
Eq. (13)). This baseline shows the reconstruction quality only monocular depth cues and no implicit surface model is used.
Evaluation Metrics. For DTU, we follow the official evaluation protocol and report the Chamfer distance. For Replica and ScanNet, following [18, 37,47,48,, 22] report the Chamfer Distance, the F-score with a threshold 5cm, as well as a Normal Consistency measure.
4.1 Ablation Study
We first analyze different scene representation choices on the Replica dataset. Next, we ablate the impact of our geometric cues on reconstruction quality and convergence time.
Architecture Choices for Scene Representations. We compare the four different scene geometry repre- sentations introduced in Section 3.1 and report metrics averaged over the Replica dataset in Table 1. Note that no monocular geometric cues are used here. We first observe that using single MLP as the scene geometry representation leads to decent results, but the recon- struction tends to be over-sveoeee smoohh (see Table and Fig. 3). For grid-based representations, optimizing dense SDF grid leads to significantly worse performance compared to all other neural implicit scene representations. even with careful parameter tuning. The reason is the lack of a smoothness bias: The SDF values in grid cells are all stored and optimized independently of each other, hence there is no local or global smoothness bias. contrast, the Single-Res. Fea. Grid replaces the SDF value in each grid cell with low-dimensional latent code, and uses shallow MI.P conditioned on these features to read out SDF values of arbitrary 3D points. This modification leads to a notable boost in reconstruction quality over the dense grid, performing similarly well as the single MLP. Using Multi-Res. Fea. Grids as in [39] further increases performance. We observe that the Multi-Res. Fea. Grids is the best-performing grid-based model, and from now report results for the single MLP
Normal C.1 Chamfer-Li F-score f
MIP [74]
86.48
6.75
66.
Dense SDF Grid
$7.30
26.68
15.50
Single-res. Fea. Grid
86.41
Multi-res. Fea. Grids
87.95
5.03
78.38
Table 1: Architectural Ablation on Replica.

---

No Cue
Depth
Normal
Both
Ground Truth
Figure 4: Ablation of Monocular Geometric Cues. Monocular geometric cues significantly im- prove reconstruction quality for both architectures (we show our MLP variant). With monocular depth cues, the recovered geometry contains more details and a better overall structure. With normal cues, missing details are added and the results become smoother. Using both cues leads to the best performance.
Normal C.t Chamfer- Lı
F-score
No Cues
86.48
6.75
66.88
0.8
Only Depth
90.56
4.26
76.42
MLP
Only Normal
91.35
3.19
85.84
Both Cues
92.11
2.94
86.18
No Cues
87.95
5.03
78,38
0.4
MLP
Multi-Res.
Only Depth
90.87
3.75
80.32
MLP (w/ Cues)
Grids
Only Normal
89.90
3.61
81.28
Grids Cues)
Both Cues
90.93
3.23
85.91
40 Iterations (x10P)
60
(a) Different Cues
(b) Optimization Time
Table 2: Ablation of Monocular Geometric Cues. a.) We report reconstruction results on Replica for MLP and Multi-Res. Grids with and without the monocular geometric cues. We observe that monocular cues improve reconstruction quality for both architectures, and using both cues in combi- nation leads to the best performance. b.) The optimization speed becomes significantly faster when incorporating monocular cues. Comparing the two architectures, we observe that the grid approach yields faster convergences while the MLP with both cues leads to the best results.
and the Multi-Res. Feature Grids. For simplicity, we will refer to the multi-resolution feature grids as Multi-Ree Res. Grids or Grids in the following.
Ablation of Different Cues. We now investigate the effectiveness of different monocular geometric cues for the two chosen representations. Table (a) and Fig. 4 show that, for both representations, using either one or both monocular cues significantly boosts reconstruction quality. We also find both cues be complementary, with the best performance being achieved when using both. Similar behavior can be observed for the other two representations (cf. supplementary material). It is worth noting that the differences between the two representations become negligible when using monocular cues, indicating that those serve as general drop-in to improve reconstruction quality.
Optimization Time. Table 2 (b) shows optimization time for the scene representations with and without cues. We see that the Multi-Res. Grids converge faster than the single MLP model. Further, adding the monocular both representations perform better than the converged models without monocular cues. Note that the overhead required for incorporating the monocular cues into the optimization process is small and can be neglected. An extended version of Table 2 (b) can be found in the supplementary materials.
significantly speeds up the convergence process. After only 10K iterations,
4.2 Real-world Large-scale Scene Reconstruction
To show the effectiveness of our method for large-scale scene reconstruction, we compare against various baselines on two challenging large-scale indoor datasets.
ScanNet. On ScanNet, we use the test split from [18] and also follow their evaluation protocol in which depth maps are rendered from input camera poses and then re-fused using TSDF Fusion [11] to evaluate only observed areas. We observe in Table 3 that our MLP variant outperforms all baselines achieving smoother reconstructions with more fine details. Note that we outperform concurrent work [68]. Further, we find that the MLP variant performs significantly better than using Multi-Res. Grids. ScanNet's RGB images contain motion blur and the camera poses are also noisy. This can be harmful to the local geometry updates in grid-based representations, while ML.ss are more robust to this noise due to their smoothness bias.
8

---

COLMAP [56]
VoISDF (38]
Manhattan-SDF [18]
Ours (MIP)
Ground Truth
COLMAP [56]
UNISURF [43] NewS [69]
VoISDF [74] M-SDF [18] NeuRIS [68] Ours (Grids) Ours (MLP)
Chamfer-Ia 4
0.141
0.359
0.194
0.267
0.070
0.050
0.064
0.042
1-score
0.537
0.267
0.291
0,364
0.602
0.692
0.626
0.733
Table Scene-level Reconstruction on ScanNet. Colmap and VoISDF do not lead to competitive reconstructions. Manhatten-SDF achieves compelling results, but less-observed areas are noisier and details are missing. In contrast, our approaches reconstruct smooth and details surfaces, achieving the best results. Further, MLPs are more robust to the motion blur and noise in camera poses.
Tanks & Temples. To further investigate the scalability of our method to larger-scale scenes, we conduct experiments the Tanks and Temples advanced sets. The qualitative results in Fig. show that the monocular cues significantly boost the performance VoISDF [74], making MonoSDF the first neural implicit model achieving reasonable results such large-scale indoor scene. See the supplementary material for more visual comparisons and discussions.
4.3 Object-level Reconstruction from Sparse Views
We now evaluate our method on another challenging task: reconstructing single objects from sparse input views. We adopt the test split from [74,75] on DTU and choose three input views following [40].
We first observe in Table 4 and Fig. that without the usage of the monocular geometric cues, neither the MLP (VoISDF [74]) nor the Multi-Res. Grids work well with only input views. When incorpo- rating the cues, the results for both representations are significantly improved. Interestingly, the grid-based representations perform inferior to single MLP as they are updated locally and do not benefit from the inductive bias of a monolithic MLP representation.
Chamfer-Ly
TSDF-Fusion [11]
4.80
COLMAP [56]
2.56
RealityCapture
2.84
Grids
6.47
Grids cues
3.68
MLP [74] MLP w/ cues
4.21
1.86
Comparing against TSDF Fusion [11] that fused predicted depth from all views into TSDF volume without any optimiza- tion, we observe that this baseline has difficulties in reconstructing meaningful details due inconsistencies in the monocular depth cues. Note that this baseline uses the GT depth maps from [13] to compute scale and shift for the depth cues. Classic MVS methods perform well quantitatively, but they heavily rely on dense matching, and in case of three input images, this inevitably leads incomplete reconstructions (see supplementary material). In contrast, our approach combines neural implicit surface representations with the benefits from monocular geometric cues that are more robust less-observed regions.
Table 4:
Reconstruction on
DTU (3 Views). We report the av- erage over the test split from [74] (see supplementary for per-object results).
4.4 Object-level Reconstruction from Dense Views
To further investigate the effectiveness and flexibility of our method, we evaluate our approach on the DTU dataset with all input views, which is a common setting in recent work [43, 70, 74]. this experiment, simply resize the low-resolution monocular cues to full resolution (from 384 x 384 to 1200 x 1200 pixels) while keeping the image ratio. As the original image is of size 1200 x 1600, the monocular cues are missing in the left and right part of the image. Therefore, we only use the monocular cues where they are available.
As shown in Table 5, our approach with MLP architecture achieves reconstruction quality similar to state-of-the-art methods [43, 70,74.. This reasonable as dense input views provide enough constraints and the prior information from monocular cues is negligible. However, our method with multi-resolution feature grid architecture outperforms previous work by large margin. We attribute this to the expressiveness of multi-resolutic feature grids where monocular cues are still effective to

---

NeuS [70]
VoISDF [74
Ours (MLP)
Ours (Grids)
Ground Truth View
Scan
24
37
40
55
63
65
69
83
97
105
106
110
114
118
122
Mean
CОLMAP
0.81
2.05
0.73
1.22
1.79
1.58
1.02
3.05
1.40
2.05
1.00
1.32
0.49
0.78
1.17
1.36
NeRF [38]
1.90
1.60
1.85
0.58
2.28
1.27
1.47
1.67
2.05
1.07
0.88
2.53
1.06
1.15 0.96
1.49
UniSurf [43]
1.32
1.36
1.72
0.44
1.35
0.79
0.80
1.49
1.37
0.89
0.59
1.47
0.46
0.59
0,62
1.02
NeuS [70]
1.00
1.37
0.93
0.43
1.10
0.65
0.57
1.48
1.09
0.83
0.52
1.20
0.35
0.49
0.54
0.84
VoISDF [74] 1.14 1.26 0.81
0.49
1.25
0.70
0.72
1.29
1.18
0.70 0.66
1.08
0.42
0.61
0.55
0.86
Ours (MLP) 0.83 1.61
0.65
0.47
0.92
0.87
0.87
1.30
1.25
0.68
0.65
0.96
0.41
0.62
0.58
0.84
Ours(Grids) 0.66 0.88 0.43 0.40 0.87 0.78 0.81
1.23
1.18 0.66 0.66
0.96 0.41 0.57 0.51
0.73
Table 5: Object-level Reconstruction on DTU Dataset will All Input Views. We compare Chamfer distance with of-te-of-the-rr methods. Our approach with MLP achieves similar results to previous methods, while our method with multi-resolution feature grids leads more detailed surfaces and outperforms previous work by a large margin.
suppress noise and therefore can reconstruct smooth and detailed surfaces. We kindly refer the reader to the supplementary material for additional visual comparisons.
Conclusion
We have presented MonoSDF, novel framework that systematically explores how monocular geometric cues can be incorporated into the optimization of neural implicit surfaces from multi- view images. We show that such easy-to-obtain monocular cues can significantly improve 31) reconstruction quality, efficiency, and scalability for a variety of neural implicit representations. When using monocular cues, a simple MLP architecture performs best overall, demonstrating that MLPs in principle are able to represent complex scenes, albeit being slower to converge compared to grid-based representations. Multi-resolution feature grids in general can converge fast and capture details, but are less robust to noise and ambiguities in the input images.
Limitations. The performance of our model depends on the quality of the monocular cues. Filtering strategies to handle failures of the monocular predictor are thus promising direction to further improve reconstruction quality. We kindly refer the reader to the supplementary material for additional analysis. While we demonstrated that integrating depth and normal cues significantly improves reconstruction, exploring other cues such as occlusion edges, plane, or curvature [14, 78] an interesting future direction. We currently limited by the low-resolution (384 x 384 pixels) output of the Omnidata model [14] and plan to explore different ways using cues. We provide some preliminary results of using high-resolution cues in the supplementary. Joint optimization of scene representations and camera parameters [3,82) is another interesting direction, especially for multi-resolution grids, in order to better handle noisy camera poses.
Acknowledgments and Disclosure of Funding
This work was supported by an NVIDIA research gift. We thank the Max Planck ETH Center for Learning Systems (CLS) for supporting SP and the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting MN. ZY is supported by BMWi in the project KI Delta Learning (project number 19A 19019000), AG is supported by the ERC Starting Grant LEGO- 3D (850533) and DFG EXC number 2064/1 project number 390727645. TS is supported by the EU Horizon 2020 project RICAIP (grant agreeement No. 8573057 and the European Regional Development Fund under project IMPACT (No. w.000...01/0.000..oo 00000......0155003/0006666 We thank the authors of Manhattan-Sil and NeuRIS for sharing results on ScanNet. We also thank Christian Reiser and Zijian Dong for proofreading.
10

---

References
[1] H. Aanæs, R. R. Jensen, G. Vogiatzis, E. Tola, and A. B. Dahl. Large-scale data for multiple- view stereopsis. International Journal of Computer Vision (IJCV), 120(2):153-168, 2016. 3,
7
[2] M. Agrawal and L. S. Davis. A probabilistic framework for surface reconstruction from multiple images. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2001. 3 [3] D. Azinović, R.
D. B. Goldman, M. Nießner, and J. Thies. Neural rgb-d surface reconstruction. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR),
2022. 10
[4] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-nerf 360: Un- bounded anti-aliased neural radiance fields. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2
[5] M. Bleyer, C. Rhemann, and C. Rother. Patchmatch stereo stereo matching with slanted support windows. In Proc. of the British Machine Vision Conf. (BMVC), 2011.
[6] D. Bonet and P. Viola. Poxels: Probabilistic voxelized volume reconstruction. Proc. of the IEEE International Conf. on Computer Vision (ICCV), 1999. 3
[7] A. Broadhurst, T. W. Drummond, and R. Cipolla. A probabilistic framework for space carving.
In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2001. 3
[8] Z. Chen and H. Zhang. Learning implicit fields for generative shape modeling. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 1,3
[9] J. Chibane, T. Alldieck, and G. Pons-Moll. Implicit functions in feature space for 3d shape re- construction and completion. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2, 3, [10] J. M. Coughlan and A. L. Yuille, Manhattan world: Compass direction from a single image by bayesian inference. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 1999.
3
[11] B. Curless and M. Levoy. A volumetric method for building complex models from range images.
In ACM Trans. on Graphics, 1996. 7, 8,
[12] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Niessner. Scannet: Richly- annotated 3d reconstructions of indoor scenes. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2017. 7,7
[13] S. Donne and A. Geiger. Learning non-volumetric depth fusion using successive reprojections. In Proc. IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2019. 3,9
[14] A. Eftekhar, A. Sax, J. Malik, and A. Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In Proc. of the IEEE International Conf. on
Computer Vision (ICCV), 2021. 2, 3, 6, 10
[15] D. Figen and R. Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In Proc. of the IEFE International Conf. on Computer
Vision (ICCV), 2015. 2
[16] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from a single image using a multi-scale deep network. In Advances in Neural Information Processing Systems (NIPS), 2014.
2, 6
[17] A. Gropp, I.. Yariv, N. Haim, M. Atzmon, and Y. Lipman. Implicit geometric regularization for learning shapes. In Proc. of the International Conf. on Machine learning (ICML), 2020. 6
[18] H. Guo, S. Peng, H. Lin, Q. Wang, G. Zhang, H. Bao, and X. Zhou. Neural 3d scene recon- struction with the manhattan world assumption. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022. 2, 3, 8,
[19] S. Hadadan, S. Chen, and M. Zwicker. Neural radiosity. ACM Trans. Graph., 2021. 3,
[20] W. Hartmann, S. Galliani, M. Havlena, L. Van Gool, and K. Schindler. Learned multi-patch similarity. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2017. 3
[21] D. Hoiem, A. Efros, and M. Hebert. Putting objects in perspective. International Journal of
Computer Vision (IJCV), 80:3-15, 2008. 2
[22] D. Hoiem, A. A. Efros, and M. Hebert. Automatic photo pop-up. ACM Trans. on Graphics, 2005.
[23] D. Hoiem, A. A. Efros, and M. Hebert. Geometric context from single image. In Proc. of the IEEE International Conf. Computer Vision (ICCV), 2005. 2
11

---

[24] D. Hoiem, A. A. Efros, and M. Hebert. Recovering surface layout from an image. International
Journal of Computer Vision (IJCV), 75(1):151-172, October 2007. 2
[25] J. Huang, S.-S. Huang, H. Song, and S.-M. Hu. Di-fusion: Online implicit 3d reconstruction with deep priors. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR),
2021. 2,5
[26] P. Huang, K. Matzen, J. Kopf, N. Ahuja, and J. Huang. Deepmvs: Learning multi-view stereopsis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[27] A. Jain, M. Tancik, and P. Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2021. 2,
(28) Y. Jiang, D. Ji, Z. Han, and M. Zwicker. Sdfdiff: Differentiable rendering of signed dis- tance fields for 3d shape optimization. In Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2020. 2,4
[29] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proc. of the International Conf. on Machine learning (ICML), 2015.
[30] A. Knapitsch, J. Park, Q.-Y. Zhou, and V. Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Trans. on Graphics, 36(4), 2017. 3,7
[31] K. N. Kutulakos and S. M. Seitz. A theory of shape by space carving. International Journal of Computer Vision (IJCV), (893))199-2) 18, 2000. 3
[32] V. Leroy, J. Franco, and E. Boyer. Shape reconstruction using volume sweeping and learned photoconsistency, In Pruc. of the European Conf. on Computer Vision (ECCV), 2018.
[33] I.. Liu, Gu, K. Z. Lin, T. Chua, and C. Theobalt. Neural sparse voxel fields. In Advances in Neural Information Processing Systems (NeurlPSS) 2020. 5,
[34] S. Liu, Y. Zhang. S. Peng. B. Shi, M. Pollefeys, and Z. Cui. DIST: Rendering deep implicit signed distance function with differentiable sphere tracing. In Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), 2020. 3
[35] W. Luo, A. Schwing, and R. Urtasun. Efficient deep learning for stereo matching. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016. 3
[36] J. N. Martel, D. B. Lindell, C. Z. Lin, E. R. Chan, M. Monteiro, and G. Wetzstein. Acorn: Adaptive coordinate networks for neural scene representation. In ACM Trans. on Graphics,
2021. 3
[37] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2019. 1, 3.7
[38] B. Mildenhall, Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Proc. of the European Conf. on Computer Vision (ECCV), 2020. 1, 3, 4, 5, 9, 10
[39] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graphics, 2022. 2, 3,5, 6.
[40] M. Niemeyer, J. T. Barron, B. Mildenhall, M. Sajjadi, A. Geiger, and N. Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR), 2022. 3,
[41] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
[42] M. Niemeyer, L.. Mescheder, M. Oechsle, and A. Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), 2020. 3
[43] M. Oechsle, S. Peng, and A. Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proc. of the IEEE International Conf. on Computer
Vision (ICCV), 2021. 1, 2, 3, 5, 7, 9, 10
[44] J. J. Park, P. Florence, J. Straub, R. A. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 1, 3,4
[45] D. Paschalidou, A. O. Ulusoy, C. Schmitt, L. van Gool, and A. Geiger. Raynet: Learning volumetric 3d reconstruction with ray potentials. In Proc. IEFE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
12

---

[46] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. De Vito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems
(NIPS), 2019. 6
[47] S. Peng, C. M. Jiang, Y. Liao, M. Niemeyer, M. Pollefeys, and A. Geiger. Shape as points: A differentiable poisson solver. In Advances in Neural Information Processing Systems (NeurIPS),
2021. 2,7
[48] S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger. Convolutional occupancy networks. In Proc. of the European Conf. on Computer Vision (ECCV), 2020. 2, 3. 5,7
[49] R. Ranftl, A. Bochkovskiy, and V. Koltun. Vision transformers for dense prediction. ArXiv
preprint, 2021. 2
[50] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), 2020. 2, 6
[51] G. Riegler, A. O. Ulusoy, H. Bischof, and A. Geiger. OctNetFusion: Learning depth fusion from data. In Proc. of the International Conf. on 3D Vision 201V) 3
[52] B. Roessle, J. T. Barron, B. Mildenhall, P. P. Srinivasan, and M. NicBner. Dense depth priors for neural radiance fields from sparse input views. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2021. 3
[53] A. Saxena, S. H. Chung, and A. Y. Ng. Learning depth from single monocular images. In
Advances in Neural Information Processing Systems (NIPS), 2006. 2
[54] A. Saxena, S. H. Chung, and A. Y. Ng. 3-D depth reconstruction from single still image.
International Journal of Computer Vision (IJCV), 16:53-69, 2008. 2
[55] A. Saxena, M. Sun, and A. Y. Ng. Make3D: learning 3D scene structure from a single still image. IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI), 31:824-888 2009. 2 [56] J. L.. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm. Pixelwise view selection for unstructured multi-view stereo. In Proc. of the European Conf. on Computer Vision (ECCV),
2016. 7,9
[57] J. I.. Schönberger and J.-M. Frahm. fincture-from-mottie revisited. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016. 3
[58] S. Seitz and C. Dyer. Photorealistic scene reconstruction by voxel coloring. In Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR), 1997. 3
[59] S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski. A comparison and evaluation of multi-view stereo reconstruction algorithms. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2006. 3
[60] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous Sd-structure-aware neural scene representations. In Advances in Neural Information Processing
Systems (NIPS), 2019. 1
[61] J. Straub, T. Whelan, L. Ma, Y. Chen, F. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, S. Verma, A. Clarkson, M. Yan, B. Budge, Y. Yan, X. Pan, J. Yon, Y. Zou, K. Leon, N. Carter, J. Briales, Т. Gillingham, E. Mueggler, L. Pesqueira, M. Savva, D. Batra, H. M. Strasdat, R. D. Nardi, M. Goesele, S. Lovegrove, and R. Newcombe. The Replica dataset: A digital replica of indoor spaces. arXiv.org, 1906.05797, 2019. 3,7
[62] E. Sucar, S. Liu, J. Ortiz, and A. Davison. iMAP: Implicit mapping and positioning in real-time.
In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2021. 7
[63] T. Takikawa, J. Litalien, K. Yin, K. Kreis, C. Loop, D. Nowrouzezahrai, A. Jacobson, M. McGuire, and S. Fidler. Neural geometrie level of detail: Real-time rendering with implicit
3D shapes. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021. 2,
3,
[64] M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich- Keil, N. Raghavan, U. Singhal, R. Ra- mamoorthi, J. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 5.
[65] S. Tulsiani, T. Zhou, A. A. Efros, and J. Malik. Multi-view supervision for single-view reconstruction via differentiable ray consistency. In Proc. IEFE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
13

---

[66] A. O. Ulusoy, A. Geiger, and M. J. Black. Towards probabilistic volumetric reconstruction using ray potentials. In Proc. of the International Conf. on 3D Vision (3DV), 2015.
[67] B. Ummenhofer, H. Zhou, J. Uhrig, N. Mayer, E. Ilg, A. Dosovitskiy, and T. Brox. Demon: Depth and motion network for learning monocular stereo. In Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), 2017. 3
[68] Wang. P. Wang, X. Long, C. Theobalt, T. Komura, L. Liu, and W. Wang. Neuris: Neural
reconstruction of indoor scenes using normal priors, In ECCV, 2022. 3, 8.
[69] P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In Advances in Neural Information
Processing Systems (NeurIPS), 2021. 1, 2, 3, 5, 7.9
[70] S. Wang. M. Mihajlovic, Q- Ma, A. Geiger, and S. Tang. Metaavatar: Learning animatable
clothed human models from few depth images. In Advances Neural Information Processing
Systems (NeurIPS), 2021. 9, 10
[71] Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari, J. Tompkin, V. Sitzmann, and S. Sridhar. Neural fields in visual computing and beyond. In EUROGRAPHICS,, 2022. 3
[72] Y. Yao, Z. Luo, S. Li, T. Fang, and I.. Quan. Mysnet: Depth inference for unstructured multi-view stereo. Proc. of the European Conf. on Computer Vision (ECCV), 2018.
[73] Y. Yao, Z. Luo, S. Li, T. Shen, T. Fang, and L. Quan. Recurrent mvsnet for high-resolution multi- view stereo depth inference. Proc. IEEE Conf. on Computer Vision and Pattern Recognition
(CVPR), 2019. 3
[74] I. Yariv, J. Gu, Y. Kasten, and Y. Lipman. Volume rendering of neural implicit surfaces. In
Advances in Neural Information Processing Systems (NeurIPS), 2021. 1, 2, 3, 5, 6, 7, 9, 10
[75] L. Yariv, Y. Kasten, D. Moran, M. Galun, M. Atzmon, B. Ronen, and Y. Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. In Advances in Neural Information Processing Systems (NIPS), 2020. 1, 1, 5,9
[76] A. Yu, V. Ye, M. Tancik, and A. Kanazawa. pixeINeRF: Neural radiance fields from or few images. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021. 7 [77] Z. Yu and S. Gao. Fast-mysnet: Sparse-to-dense multi-view stereo with learned propagation and neuss-newton refinement. In CVPR, 2020. 3
[78] Z. Yu, J. Zheng, D. Lian, Z. Zhou, and S. Gao. Single-image piece- piece-wise planar 3d reconstruction via associative embedding. In CVPR, pages 1029-1037, 2019. 10
[79] S. Zagoruyko and N. Komodakis. Learning to compare image patches via convolutional neural networks. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2015. 3
[80] K. Zhang, G. Riegler, N. Snavely, and V. Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv:2010007492, 2020.
[81] X. Zhang, Srinivasan, B. Deng, P. Debevec, W. T. Freeman, and T. Barron. NeRFactor:
Neural Factorization of Shape and Reflectance Under an Unknown Illumination. arXiv preprint
0et:21006011770, 2021.
[82] Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M. R. Oswald, and M. Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2022. 2, 3, 7, 10
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss
potential negative societal impacts in our supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
14

---

3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes] Code and data are released.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] We describe details of our computational resources in supplementary material.
4. If you using existing assets (e.g., code, data, models) curating/releusing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "mE7zik7Q1K5", "reviewer": "Reviewer_grib", "summary": "This paper presents a framework to utilize monocular geometric cues to improve multi-view 3D reconstruction quality, efficiency, and scalability for neural implicit surface models. A systematic comparison and detailed analysis of design choices of neural implicit surface representations including vanilla MLP and grid-based approaches has been presented. Among these representations, a simple MLP architecture performs quite well, which demonstrates that MLPs are able to represent complex scenes.", "questions": "One ablation study I want to see in the paper is the number of input multi-view images and monocular geometric cues. Specifically, the monocular geometric cues could help the reconstruction when the input images are sparse. If the input images are dense, the monocular geometric cues might influence the reconstruction quality due to the error contained in the monocular cues. I wonder where is the balance for the input number of images? For a reconstruction problem with different number of input images, how should I know whether the monocular geometric cues could help or not?", "limitations": "Except the question listed in the above, I don't have other concerns to this paper.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I really like the idea proposed in this paper. To improve the reconstruction quality with sparse input, shape priors should be added. There are several ways to construct the priors. One solution is to construct the parametric model for some special types like face and body. This paper explores another way with the help of depth estimation from single image. Although the estimated depth and normal may contains noises or with wrong scales, the proposed method well handles these issues.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LrFE1FwM1Sn", "reviewer": "Reviewer_D9of", "summary": "Existing neural fields-based methods fail to reconstruct high-quality surfaces for larger and complex scenes with sparse viewpoints. In this work, the authors inspect the issue as inherent ambiguity in RGB loss which provides insufficient constraints. Inspired by the area of monocular geometry prediction, this paper proposes MonoSDF, which explores the utility of depth and normal cues predicted by general-purpose monocular estimators. Experiments demonstrate the geometric monocular priors significantly improve the performance both for single and multi-object scenes.", "questions": "When conducting the comparison between MLP and dense grids, how is it possible to make the comparison fair? For example, MLP-based representation is known to be computationally heavy while having low storage requirements, while dense grids can consume very large memory. It is also hard to measure the “capacity” between the MLP-based model and grid (including multi-resolution grids). Simply comparing the number of learnable parameters might not be enough. Therefore, to conduct a complete exploration of different architecture choices, simple tables (like Table 2) might not be enough. For example, for each choice, a figure is required to see the performance vs.. model capacity vs. trade-offs (spatial, convergence speed).\n\nHow robust is the monocular cue supervision to the model? For example, how the results change if there are large errors in the depth or normal predictions\n\nIn Eq 13, are w and q estimated for each image separately? In Line 189, it mentioned, “per batch”. Does one batch contain multiple images?\n", "limitations": "The paper discussed the limitations and social impacts.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-\tThe proposed method, which incorporates monocular predictions to ease geometry learning, is quite clean and easy to apply to any neural implicit method.\n\n\n\nWeaknesses:\n\n-\tThe contribution is weak and relatively incremental. The entire paper can be seen as a combination of two different parts: (1) adding monocular cues as additional supervision to improve MVS; (2) exploring different architecture choices for neural implicit representations.\n\n-\tFor (1), although the idea is clean and easy to understand (as stated in strengths), it is not new in the scenario of learning neural fields. For example, several papers have already explored using predicted depth and other semantic features in learning NeRF. This work applies a very similar idea to surface reconstruction. However, the methodology did not make any changes (auxiliary loss terms) compared to the case of learning standard NeRF. I would expect different choices or additional analysis due to the task of surface reconstruction instead of simply applying the loss. For example, compared to NVS, will normal cues be more important? How the noise of the prediction affects the geometry quality?\n\n-\tFor (2) part, it is a bit disconnected from the main story. It is always a nice contribution to conduct a systematic exploration of the best architecture for surface reconstruction. Although the findings from the paper about MLP vs. explicit grids are not surprising, the comparison itself can be a good topic. However, such experiments can be done at any settings with different loss functions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "96CJLZ9sfm", "reviewer": "Reviewer_c8RS", "summary": "In this work, the authors proposed a novel and powerful geometric representation using neural implicit function. Previous neural implicit functions are trained purely on RGB reconstruction loss and have difficulty in representing more complicated geometry. In this work, the authors try to address this problem from two directions: first, they propose a novel depth and normal cues that significantly improves the quality of the reconstruction. Secondly, they explored different representation functions, including dense SDF grid, simple MLP, feature grid + MLP and multi-resolution feature grid + MLP. Both of these changes significantly improve the quality of reconstructed geometry.", "questions": "I only have a few minor questions for the authors:\n\nIn Table 1, in addition to network type, it is also useful to report the number of parameters of each network. Normally, in neural representation, larger network parameters often lead to a better representation power, so this information is important to understand whether the improvement comes from network architecture itself, or simply a larger set of parameters. Moreover, it would be better to compare networks with similar numbers of parameters in Table 1.\n\nThe proposed neural representation also recovered the texture of the surface (color). I think it is also important to visualize them to understand the accuracy of texture recovery, even though they are not the main task of this work. It is hard for me to tell whether the network can also represent texture, or the color prediction network is simply to help to train the SDF.\n\nI also have one minor suggestion to this work:\n\nIn Figure 3, 4, it would be better to show actual image view or ground truth rendered image, as it is hard to tell whether some small reconstructed geometry are correct or not.\n\n", "limitations": "No as far as I know.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a high quality work. The idea of using depth and normal cues are simple and effective. The proposed multi-resolution feature grid + MLP representation is also novel and effectively improves the reconstruction quality over MLP solution. The experimental results are thorough  and the paper is well written and easy to follow.\n\nI don’t find any particular negative point.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "13Sqj0UV-O1", "reviewer": "Reviewer_Ak1a", "summary": "This paper proposed an approach to address 3D reconstruction from multi-view images. The paper is built upon several milestone papers/techniques and the results presented are relatively better. It started off from signed distance function (SDF), and volume rendering of implicit surfaces. The proposal is the incorporation of two losses, i.e. depth and normal consistency estimated from individual images. The estimation, or \"ground truth\" for supervision, is from a pre-trained Omnidata model [14]. The paper also explored several architectures.", "questions": "1. I'm wondering what is the performance comparison between the proposed monocular priors vs self-supervised depth+normal estimation? ", "limitations": "The paper has addressed the limitations of the existing model it used for depth estimation. However, it does not address whether Omnidata is the best prior to use compared to other models. It also does not address other monocular cues to explore.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strength*\n1. The paper is easy to read and the main idea is clearly delivered. In essence, the paper took an off-the-shelf depth estimator to serve as a strong prior. \n2. The proposed approach is robust across different numbers of images. The method can not only be applied to single objects, but also large scale scenes.\n\n*Weakness*\n1. The novelty of this paper is insignificant. It pieces together several techniques (SDF, VolSDF, Omnidata etc). And many of them are well established. I think the paper can potentially compensate this weakness by addressing the points below:\n1.1 The exploration of the \"monocular geometric cues\" is also relatively weak. It is curious to see how different depth estimators may affect the improvement when incorporating the additional prior. \n1.2 It is also peculiar that this paper chose depth and normal as the only two monocular cues to experiment. The Omnidata is able to generate high quality ground truth for 19 more more tasks. \n\n2. The color aspect of the reconstruction is never demonstrated or explained. The results are mostly focused on the geometric reconstruction but not the color appearance or rendered 2D images.\n\n3. The experiments for the architectures are confusing. The paper first showed results for comparing different architectures without monocular cues and arrived at the conclusion that the best model is Multi-Res. Fea. Grids. Then after adding the proposed monocular cues, the paper concluded that MLP is the best model. This behavior is not well explained.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept