DiViNeT: 3D Reconstruction from Disparate Views using Neural Template Regularization
Authors: Aditya Vora
Akshay Gadi Patil
Hao Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Multi-view Neural 3D Reconstruction
Sparse and Disparate Views
Neural Rendering
Volume Rendering
TL;DR: We present a volume rendering-based neural surface reconstruction method that takes as few as three disparate RGB images as input.

Abstract: We present a volume rendering-based neural surface reconstruction method that takes as few as three disparate RGB images as input. Our key idea is to regularize the reconstruction, which is severely ill-posed and leaving significant gaps between the sparse views, by learning a set of neural templates that act as surface priors. Our method, coined DiViNet, operates in two stages. The first stage learns the templates, in the form of 3D Gaussian functions, across different scenes, without 3D supervision. In the reconstruction stage, our predicted templates serve as anchors to help “stitch” the surfaces over sparse regions. We demonstrate that our approach is not only able to complete the surface geometry but also reconstructs surface details to a reasonable extent from few disparate input views. On the DTU and BlendedMVS datasets, our approach achieves the best reconstruction quality among existing methods in the presence of such sparse views and performs on par, if not better, with competing methods when dense views are employed as inputs.

DiViNeT: 3D Reconstruction from Disparate Views via Neural Template Regularization
Aditya Voral
Akshay Gadi Patil
Hao Zhang
Simon Fraser University
Amazon
Abstract
We present volume rendering-based neural surface reconstruction method that takes as few as three disparate RGB images as input. Our key idea is to regularize the reconstruction, which is severely ill-posed and leaving significant gaps between the sparse views, by learning a set neural templates to act as surface priors. Our method, coined DiViNet, operates in two stages. It first learns the templates, in the form of 3D Gaussian functions, across different scenes, without 3D supervision. the reconstruction stage, our predicted templates serve anchors to help "stitch" the surfaces over sparse regions. We demonstrate that our approach is not only able to complete the surface geometry but also reconstructs surface details to reasonable extent from a few disparate input views. On the DTU and BlendedMVS datasets, our approach achieves the best reconstruction quality among existing methods in the presence of such sparse views and performs on par, if not better, with competing methods when dense views employed as inputs.
Introduction
3D reconstruction from multi-view images is fundamental task in computer vision. Recently, with the rapid advances in neural fields [51]31] and differentiable rendering
[8]
many methods have
been developed for neural 3D reconstruction [34
55
23
28
48
22
54
36
62
50 Among them,
methods have achieved impressive results. Exemplified by neural radiance fields (NeRF) [3 these approaches typically employ compact MLPs encode scene geometry and appearance, with network training subjected to volume rendering losses in RGB space.
volume rendering-
The main drawback of most of these methods is the requirement of dense views with considerable image overlap, which may be impractical in real-world settings. When the input views are sparse,
these methods often fail to reconstruct accurate geometries due radiance ambiguity [6] 49) Despite
employing smoothness priors during SDF/occupancy prediction as an inductive bias 34 55 48 [4],
the limited overlap between regions in the input images causes little-to-no correlation to the actual 3D) surface, resulting in holes and distortions, among other artifacts, in the reconstruction.
Several recent attempts have been made on neural reconstruction from sparse views. SparseNeuS 25 learns generalizable priors across scenes by constructing geometry encoding volumes at two res- olutions in a coarse-to-fine and cascading manner. While the method can handle as few as 2 or input images, the corresponding views must be sufficiently close to ensure significant image overlap and quality reconstruction. MonoSDF [58] relaxes the image overlap requirement and relies geometric monocular priors such as depth and normal cues to boost sparse-view reconstruction. How- ever, such are not always easy to obtain, e.g., MonoSDF relied pre-trained depth estimation model requiring 3D ground-trun truth (GT) data. Moreover, depth prediction networks often rely on cost volumes [52 [4] which memory expensive and thus restricted predicting low-resolution depth maps. Because of this, the results obtained typically lack fine geometric details [58]
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Scan 24
BORN B00GH
Scan 55
Scan 40
Input images
Neus
MonoSDF
Ours
Figure 1: Surface reconstruction results on the DTU dataset [16] for three disparate view inputs (shown on the left). We compare our technique against NeuS [88,, which does not use any regulariza- tion, and against MonoSDF [58]] which uses additional depth and normal supervision to regularize the surface reconstruction process. The results clearly show that our method not only reconstructs the complete surface but also preserves sharp geometric details without using any other ground truth information beyond the RGB inputs. See Section 4.1 for detailed explanation.
In this paper, target neural surface reconstruction from sparse and disparate views. That is, not only are the input RGB images few in number, they also do no share significant overlap; see Figure] Our key idea is to regularize the reconstruction, which is severely ill ill-posed and leaves significant gaps to be filled between the few disparate views, by learning a set of neural templates that act surface priors. Specifically, in the first stage, learn the neural templates, in the form of 3D Gaussian functions, across different scenes. Our network a feedforward CNN encoder-decode trained with RGB reconstruction and auxiliary losses against the input image collection. In the second stage, surface reconstruction via volume rendering, our predicted templates serve as anchors to help "stitch" the surfaces sparse regions without negatively impacting the ability of the signed distance function (SDF) prediction network from recovering accurate geometries.
Our two-stage learning framework (see Figure is coined DiViNet for neural 3D reconstruction from Disparate Views via NEural Templates regularization. We conduct extensive experiments on two real-world object-scenes datasets, viz., DTU [16) and BlendedMVV [53], to show the efficiency of our method over existing approaches on the surface reconstruction task, for both sparse and dense view input settings. Through ablation studies, validate the design choices of our network in the context of different optimization constraints employed during training.
2 Related Work
View-based 3D Reconstruction. Reconstructing 3D surfaces from image inputs is non-trivial task mainly due to the difficulty of establishing reliable correspondences between different regions the input images and estimating their correlation to the 3D space. This is especially pronounced when the number views is limited and disparate. Multi-view stereo (MVS) techniques, both classical 45). and learning-basee 27 47 59 39 IS
52 571, while using explicit 3D representation, address the problem multi-view 3D reconstruction either by estimating depth via feature matching across different input views by voxelizing 3D space for shape reconstruction.
[ 6 9
recent exploration of neural implicit functions for 3D shape/scene representation
24 46
40 17
has led to an explosion of 3D reconstruction from multi-view images, where two prominent directions exist one based on surface rendering techniques 34 55 23,, which require accurate
masks at the input, and the other based on volume rendering techniques
28
48
22
36
62
50)

---

Stage 1: Neural Templote Learning
Surface Reconstruction Volume Rendering
mppisit Rep
juak along depth
Template Parometors.
Oopth Map, Pl1
T3
MLP
Image, drl
Caneo
503 condratte,
Figure 2: A two-stage learning framework developed to reconstruct 3D surfaces from sparse, disparate view input images. In Stage (see Section we train a network across different scenes to predict structured templates (Gaussians, in case) that encode surface priors. In Stage (see Section 3.2). we train a surface reconstruction model to reconstruct the surface SDF values by leveraging the predicted templates from Stage 1. The purpose served by the predicted templates is in aiding the reconstruction process by acting as regularizers, allowing to obtain complete geometry (see Figure and 7) and even details to reasonable extent (see Figure 5) from disparate view inputs.
which require no such object masks. The latter techniques combine the representational power of neural implicit with the recently proposed NeRF models (31] achieving better surface reconstructions over the former techniques. A major drawback of these works is that they all require dense multi-
view images input. Two recent techniques, Sparse NeuS (25) and Mono-SDF (58) learn 3D
reconstruction from just three input images. In the case of [25]. the three input images have
significant amount of overlap, and can not be employed when the views are disparate. [58), on the
other hand, leverages additional cues such as ground truth depths and normals, allowing reconstruction from disparate views. Our work differs from these in the sense that we do not make use of any explicit cues in the form of ground truth depths and normals, while still being able to faithfully reconstruct 3D surfaces from just three disparate RGB images as input.
Regularizing Neural Surface Reconstruction. Our ability to reconstruct 3D surfaces from sparse, disparate view images is made possible through template-based regularization that provides helpful surface priors during the volume rendering stage. Regularization has been explored in the context of novel-view synthesis 33 7 where patch-based regularization for both the scene geometry and color is performed [33 or additional cues such as depth supervision are incorporated 7 while handling sparse inputs both cases. the context of surface reconstruction, MonoSDF (58) uses depth and normal supervision as a part of unsaid regularization to obtain surface geometry from sparse input images. Other works 60) study the effect of regularization on the volumetric rendering- baddering-based surface reconstruction framework. Their goal is to improve surface reconstruction by imposing multi-view photometric and COLMAP
43 constraints on the Signed Distance Field (SDF) prediction during training. Such methods show significant improvements over vanilla approaches for surface reconstruction. However, under sparse scenarios, because of significant view differences, photometric and COLMAP constraints are difficult to satisfy, resulting in poor reconstruction quality. In contrast to these methods, inspired by [T] 35), we propose learn surface priors in the form of templates across data distribution and use these templates to guide the reconstruction process.
3 Method
Our method achieves 3D reconstruction of solid objects from a small set of RGB images (4).N-- with very less visual overlap, where N can be as less as 3 images and I, [0, 1] N> x3 We assume that camera extrinsies and intrinsics are known for all the images. As shown Figure approach is realized in two stages: (1) Learning a network for predicting shape templates (see Section 3.1 and
(2) Leveraging the predicted templates, learning a volumetric surface reconstruction network with depth and SDF constraints (see Section 3.2
3

---

3.1 Stage 1: Learning Surface Priors with Neural Templates
Template Representation. We represent the shape with a bunch of N, local implicit functions called templates, where the influence of eth template is represented as (i,p, sii) and is given by,
(Gidd PP))
J: sp.) = 8; exp
Σ (Z.
(I)
2r
where IR is the 3D location of point in the volume and yo, is the template parameter. Specifically, each template sp is represented as a scaled, anisotropic 3D Gaussian, yi E I2, which consists of scaling factor S,, center point C; R& and per-axis radii r; R Using this local implicit function, we can find the probability a point near the surface by summing up the influences of all Nr templates, i.c. G(p, 2
Σ 9: (p, sp.), where is the set of all the template parameters.
Network Architecture. Figure shows the template training stage.
Templane
Given fixed set of N₂ templates (Φ) and input images set } we aim to optimize network Ta for the task of template prediction which com- prises of encoder (E) and decoders (D geo: Deor). The encoder is feed- forward Convolutional Neural Net- work (CNN), which we first use to extract 2D image features from indi- vidual input views. We then obtain per template latent code 2e. E R&4 using a bi-ler interpolation step as shown in Figure For this step, de-
Parometers (N)
Open
DupoRt
Loss
MP
Mapp, Sp,
Rise
W; Templete Locdl Teature Voltames
GT Image, /DP)
Figure 3: Template Learning stage. We train the template prediction network across scenes to predict N₂ structured templates O, alongwith local feature volumes per template.
pending the number of templates, we create a uniform grid of points, which we then use for the interpolation features obtained from convolutional layers. Each latent code of the template, R,; is then decoded through two decoders Dyeo and D Deon. Dyeo is the geometry decoder with simple MLP layers predicting the template parameters Ps. That is, Dyeo : 64 + R.. Dvoz, on the other hand, a feature decoder comprised of transposed convolutional layers. The objective of Droz is to map the template feature representation, Ap,, to local volumetric feature grid for each individual template.
Decoding all the template latent codes Zy; at once through Donr results in dense voxel grid V E R(C-M))xMMM/N))((M>> which we then rearrange N, local volumes of size RCMM" where C is the feature dimension of each voxel and M is the resolution of the feature grid. In our experiments, we N, = 576, C' and M 16. For more details refer to supplementary.
N/16
Network Training. Due to the lack of ground truth 3D information of objects in the training set, we use RGB reconstruction loss along with auxiliary losses optimize the template parameters. Given the predicted feature volume from the template network, we can query continuous field at any 3D query point p; by passing its tri-linearly interpolated feature fvoz (pt). along with the location p. to a small MLP, consisting of 2 layers and 256 hidden dimensions. The output of the MLP is the color prediction at that particular query point, p: in the 3D space. In addition color, we compute each point's in 3D space using our predicted templates. This
Figure 4: Interpolation step.
/N₂ x /N, codes obtained from H/16 x W/16 features.
blending weights blending weight is computed by summing up the local influences of each template on the query point. We then integrate the predicted colors and weights to obtain the
predicted image L. For K query points sampled along ray, this can be written

---

G(pi, +)
I(r) Σ W;
We E, G(p,, 4) +
(2)
Here, P: is query point along ray, is small perturbation to prevent u; from diverging to infinity when G(p;, +) takes small values, whereas G(p., Φ) = EigN,
Φ.). We then optimize
our template prediction network using RGB reconstruction loss. To make sure that the predicted
templates are plausible and near the surface, we propose auxiliary loss functions. The total loss by combining all the losses is given as:
Liotal
Lrgb NedCchamfer + XeCcoo + Ap Xy XyCvar
Here, Lrgb the mean square error loss between the predicted pixel f(r) and ground truth pixels ((r) which is defined as:
Ligh Σ
T PER
Lehham for is the chamfer distance loss, which minimizes the distance between the template cen- ters {Ci C1 C2, CN, R²), with the corresponding sparse reconstructed point cloud {Se $1, 82, :.. SN E R obtained using triangulation, from COLMAP(43). This loss makes sure that the predicted templates from the network are the surface. This given by:
Cehamfer
Σ GCE
min (lle, - s,1l2) s;ES.
Σ
min (1)s c,112)
(5)
|Crl
Se
GEC,
syES.
Leov is the coverage loss, which is defined over the entire sparse points, Se. This ensures that the predicted templates cover the entire object. It defined as,
N
Leor
Σ sES.
a 2
SS
where (.) is the sigmoid function. In addition to this, we also radius Lradius and variance loss Lears which penalizes large and skewed template radii. This is given by:
Lradius
Σ 1
Lvar
Σ ||. Fell
F.RR.
RE
Here, R₁ is the of all template radii r, is the individual template radii, and Ft is the mean radius across all the templates, No.
3.2 Stage 2: Surface Reconstruction using Volume Rendering
Geometry Representation and Volume Rendering. Following the recent works of surface recon-
struction with volume rendering [48 54 36), we represent the volume contained by the object to be reconstructed using signed distance field 8; which is parameterized using ML. fa,, with learnable parameters fg i.e & fa. (7(x))). Here 7-) is the positional encoding of point in 3D space 31.. Given the signed distance field, the surface of the object is represented its zero-level i.e.
S fx E R R (x) = 0}.
To train the SDF network in volume rendering framework, we follow 488, and use the below function, to convert the SDF predictions, fo, (p:) to at each point, p: in the 3D space.
04 max (fo... (pc) 4 (fe.(p .0). (. (p.))
Here, (x(x) (1 -00--1 is the sigmoid function and leamed during training.
Along with the signed distance field, 8,, we also predict color, & for each sampled point 3D space using color MLP, fe.. Following (55) the input to the color MLP is a 3D point Ps, viewing
5

---

direction d, analytical gradient ni of our SDF, and feature vector computed for each point p: by fe.- Hence, the color MLP is the following function, és fe, (p;, d, n, =4). We optimize these networks using volume rendering. where the rendered color of each pixel integrated over discretely sampled points {p, = o t,dli = 1, ... , M,, t4 < t++1} along a ray traced from camera center o and in direction d. Following [3 the accumulated transmittance T3 for a sample point p; is defined as T; = [[ aj), where aj is the opacity value defined in Eq. Following this, we can compute the pixel color and depth as follows:
C(r)
Σ Taas
E(r)
Σ
Tas
(10)
Optimization. The overall loss function we use for optimizing for the surface is:
= Cootor + A1 Leskonaa + A2Cdepth + Az
(11)
Leotor, is the L1 loss between the predicted and ground truth colors which is given
Lootor
(((())
(12)
FER
Following 13 we use Eikonal loss to regularize the SDF in 3D space, which is given as:
Leskonal
(I TEX
1)²
(13)
In addition to these, we propose the following regularization terms in order to aid the surface reconstruction process, especially in sparse reconstruction scenarios.
Depth Loss: We sample depth cues 2x(r) along each ray using the templates predicted by our template network. These depth cues are obtained by finding the depth locations where the template weight function w(x)) at point x, mentioned 2 attains maximum value among all the points sampled along ray. We then minimize the LI loss Edepth between the predicted depth, (r) and the computed depth cues zę (r) as:
Edepth
Σ
-18()) - 2r(
(14)
rERcalid
Here, Reatid denotes the set of rays that intersect the templates in the 3D space. We find these rays using ray sphere intersection.
In addition depth, we also constraint SDF pass through the template centers which we assume are near the surface. This is done by minimizing the LI loss between the SDF at the template centers and the zero-level set. Here, Ne the number of template parameters.
SDF Loss:
Ladf
Σ DEC.
1o (x)]l2
(15)
N.
Implementation Details. Our implementation is based on the PyTorch framework [38]. For both stages, we use the Adam optimizer (20) to train our networks and set a learning rate of 5e -4 For stage-1, we Aeds A. A, and A, to 1.0, 0.1, 0.1, and 1.0, respectively. And for stage-2, we set λ1. A2 and Aa to 1.0, 0.8 and 0.8, respectively. As the templates are not perfect, we follow [58] and use an exponentially decaying loss weight for both SDF and depth regularization for the first 25 iterations of optimization. During training in both stages, we sample batch of 512 rays in each iteration. During both stages, we assume that the object is within the unit sphere. Our SDF network fa,, is an 8 layer MIP with 256 hidden units with skip connection in the middle. The weights of the SDF network are initialized by geometric initialization (2] The color MIP is layer MIP with 256 hidden units. The 3D position is encoded with 6 frequencies, whereas the viewing direction is

---

Reforence Image
Ours
Reforence Image
Ours
MonoSDF
Figure 5: This figure compares the details on the reconstructed geometry, for sparse-vier view inputs on the DTU dataset, against the second-best competing method. MonoSDF (see Table [ Our approach produces sharper details, which can be attributed to the surface guidance from the predicted templates.
encoded with 4 frequencies. We train the network for 300k iterations which takes roughly 8 hours NVIDIA RTX 3090Ti GPU. After training, we extract the mesh using marching cubes (26 at a 5123 resolution. See the Supplementary material for more details.
4 Results and Evaluation
Datasets. We use two commonly employed real-world object-scenes datasets, viz., DTU 16 and BlendedMVS dataset [53]. DTU dataset contains multi-view images (49-64) of different objects captured with a fixed camera and lighting parameters. For training sparse view reconstruction models, use the same image IDs in 33 58 56). In addition this, in order to show the generalization ability of our template prediction network we experiment with our method on the MobileBrick dataset (21] which consists high-qualit, 3D models along with precise ground-truth annotations.
For the template learning stage, our training split consists of objects that do not overlap with the 15 test scans of the DTU dataset. We provide the list of scans used for training in the supplementary. We evaluate our method on standard 15 scans used for evaluation by (4|55] of each image in the dataset is 1200 x 1600. Unlike MonoSDF [58] which resizes the images to 384 x 384 resolution for training, we input the actual image resolution to our pipeline. The other dataset, BlendedMVS, consists of 113 scenes captured from multiple views, where each image is of resolution 576 x 768. Unlike DTU, BlendedMVS dataset has not been employed before in the sparse view setting, and as such. specific view IDs train and test are not available. We therefore manually select views that have little overlap.
55
54 48 36). The resolution
Baselines. For sparse views, compare against the classical MVS method, COLMAP recent volume rendering- based methods such as NeuS [48), VoISDF (54) and MonoSDF
42 and
58
We
exclude SparseNeuS 225 evaluation in disparate settings because of its inability to reconstruct 3D geometry when the input images have significantly less overlap. For dense views, we additionally compare against the recent grid-based method, Voxurf [50]. mesh from COLMAP point cloud
output is obtained using Screened Poisson Surface Reconstruction
[19
Evaluation Metrics. Following standard evaluation protocol, we use Chamfer Distance (CD) between the ground truth point cloud and predicted meshes to quantify the reconstruction quality. On the DTU dataset, following prior works [54] 48 (58), report CD scores. On the BlendedMV'S dataset, we simply show qualitative results as is commonly done the literature. addition to this, for evaluation on the MobileBrick dataset, we F1 score computed using the ground-truth mesh.
4.1 Results on DTU
Figure T and Table show qualitative quantitative results, respectively, in the presence sparse view inputs (three, to be specific) on the DTU dataset. We observe from Table [ that our method outperforms other competing methods, including MonoSDF (58), on nine of the fifteen scenes in the test set, and achieves the lowest mean CD score (the next best adds an additional 0.09 CD score).

---

Scan 110
Scan 112
Reference Image
NeuS
MonoSDF
Voxurf
Ours
Figure 6: Dense view reconstruction results on the DTU dataset.
Table 1: Quantitative comparisons for sparse and disparate view 3D reconstruction on DTU dataset
(Section 4.11
For Scan ID 110, x indicates that COLMAP could not reconstruct any points.
Chamfer Distance (CD) (4)
Scan II)
24
37
40
55
63
65
69
83
97
105
106
110
114
118
122
Mean
COLMAP(4) 41
4.45
4.67
2.51
1.9
2.81
2.92
2.12
2.05
2.93
2.05
2.01
x
1.1
2.72
1.64
2.56
VoISDF [54]
5.24
5.09
3.99
1.42
5.1
4.33
5.36
3.15
5.78
2.07
2.79
5.73
1.2
5.64
6.2
4,2
NeuS 48
4.39
4.59
5.11
5.37
5.21
5.60
5.26
4.72
5.91
3.97
4.02
3.1
4.84
3,91
5.71
4.78
MonoSDF
58
3.47
3.61
2.1
1.05
2.37
1.38
1.41
1.85
1.74
1.1
1.46
2.28
1.25
1.44
1.45
1.86
Ours 3.37
4.11
1.46
0.75
2.74
1.52
1.13
1.63
2.08
0.98
0.87
2.7
0.47
1.24
1.57
1.77
On the other six scans, method is the second-best performing method behind MonoSDF. This is because our predicted templates are not always perfect for all objects, meaning, regions on the surface with predicted templates potentially suffer from incomplete reconstructions. Other methods such as VoISDF [54] and NeuS [48] fail to even beat the classical MVS method, COLMAP [33, indicating that these methods either require dense view inputs or require significant amount of overlap the input views (25) to faithfully reconstruct the surface. This emphasizes the effectiveness of our method the disparate views case. Further, Figure 5 compares surface details the reconstructed geometry against, MonoSDF (58), where it can be observed that our method produces sharper details, attributed mainly the geometry guidance from learned template priors.
On dense view inputs (49- 64), Figure 6 and Table 3 present qualitative and quan- titative results, respectively, over the aforementioned tech- niques. We additionally com- pare against a recent method, Voxurf [50). In this setting, all neural surface reconstruction methods outperform COLMAP, indicating that the presence of dense view inputs is necessary tap into the learning ability of neural rendering techniques for improved geometry reconstruction. We observe that our method outperforms the rest five scans, which is higher than any other method (Voxurf-thre scans, NeuS four, MonoSDF-two, and VoISDF-one). Overall, our method achieves the second- best mean CD score, closely trailing behind Voxurf (a difference of 0.03 CD score). This expected as feature-gni ceeee-rridddssee based approaches tend to reconstruct sharper details compared to MLP [58]. But otherwise, we beat other existing works owing to our neural template regularization scheme. More qualitative results on dense view inputs are provided in the Supplementary material.
Table 2: Generalization of TPN on MobileBrick dataset.
Fl Score (f)
Object ->
Bridge
Camera
Colosseum
Castle
Ours (pre-trained TPN)
0.565
0.61
0.219
0.175
Ours (TPN re-training)
0.658
0.67
0.22
0.187
4.2 Results on BlendedMVVS
We also run our pipeline on more challenging, also real-world, BlendedMVS dataset [53]. We use the sparse input setting and show comparisons against the same set of prior works, i.e., NeuS 48 and MonoSDF [58]. Figure shows qualitative results on 4 object-scenes with complex geometries. Clearly, our method obtains the best surface reconstruction. NeuS is not able to reconstruct the

---

Table 3: Quantitative comparisons for dense-view 3D reconstruction on DTU dataset (Section 4.1
Chamfer Distance (CD) (1)
Scan IID
24
37
40
63
65
83
97
105
106
110
114
118
122
Mean
C0LMAP
0.81
2.05
0.73
1.22
1.79
1.58
1.02
3.05
1.4
2.05
1.0
1.32
0.49
0.78
1.17
1.36
NeuS 48
1.0
1.37
0.93
0.43
1.1
0.65
0.57
1.48
1.09
0.83
0.52
1.2
0.35
0.49
0.54
0.84
VoISDF [54
1.14
1.26
0.81
0.49
1.25
0.7
0.72
1.29
1.18
0.7
0.66
1.08
0.42
0.61
0.55
0.86
MonoSDF
58
0.83
1.61
0.65
0.47
0.92
0.87
0.87
1.3
1.25
0.68
0.65
0.96
0.41
0.62
0.58
0.84
Voxurf
[50
0.91
0.73
0.45
0.34
0.99
0.66
0.83
1.36
1.31
0.78
0.53
1.12
0.37
0.53
0.51
0.76
Ours
0.78
1.29
0.68
0.42
0.96
0.70
0.68
1.43
1.30
0.827
0.62
0.94
0.34
0.49
0.49
0.79
Sculpture
Clock
Buddha
Man
Input images
Neus
MonoSDF
Ours
Figure Qualitative reconstruction results for three disparate view inputs on the challenging
BlendedMVS dataset [33 See Section 4.2 and the Supplementary for more details explanation.
geometry faithfully for any of the objects due to disparate view inputs. MonoSDF, on the other hand, despite using ground truth depth and normal cues, fails reconstruct accurate geometry, especially for Clock and Sculpture objects. In contrast to this, our method can accurately reconstruct the geometric details owing to template guidance coming from Stage 1.
4.3 Generalization MobileBrick Dataset
We test the generalization ability of our template prediction network (TPN) on a new dataset, MobileBrick [21]. by comparing the reconstruction results when the neural templates were learned by pre-trained TPN (on DTU) vs. when they were trained on MobileBrick. Note that overall, the models from these two datasets quite different in terms of geometry and structure. The results in Table 2 and qualitative results in Figure Sa (a) show that the reconstruction qualities under the TPN. As mentioned before, the
two scenarios are comparable, attesting the generalizability metric used in this evaluation is Fl score as reported by MobileBrick dataset.
4.4 Ablation Studies

---

6.7
(b)
Figure& 8: (a) Generalization of TPN on MobileBrick Dataset. (b) Effect on chamfer distance with # of Views (left) and Gaussians (right).
Referance Image
Only SDF Constrant
Only Depth Constraint
SDF Depth Constraim
Figure Effect of different optimization constraints (Figure
Section 3.2
employed our pipeline.
In Figure demonstrate the need for employing differ- ent loss functions as described in Section 3.2 and depicted on the right side of Figure and and Table 4 We observe how the reconstructed geometry changes when Ladf and Ldepth separately employed. Evidently, only SDF constraints cannot reconstruct surfaces due to the coarse supervision from template centers. On the other hand, depth regular- ization significantly improves the reconstruction but holes still exist. Our full pipeline includes both the above constraints, resulting in improved reconstruction, as can be seen in the last column of Figure and Table We also study reconstruction quality as function of the number of input views. Figure 8b depicts the behavior of two such methods i.e. NeuS and Ours, up to fifty views. Our method achieves the same Chamfer Distance (CD) score for twelve views as NeuS achieves for around thirty. This demonstrates the efficacy of our method to "catch" surface geometry even when sufficient view inputs are not present. In addition to this, we measure effect on reconstruction quality by varying the number of Gaussians. As can be seen in Figure 8b the surface reconstruction quality significantly improves we increase the number of gaussians. The reconstruction quality saturates after 376 gaussians and hence we use 376 gaussians throughout experiments.
Table 4: Impact of each constraint.
CD (t)
Only SDF Constraint
2.70
Only Depth Constraint
1.96
SDF + Depth Constraint
1.52
5 Discussions, Limitations and Future Work
We present DiViNet for sparse multi-view 3D reconstruction. Our core idea is simple two- stage framework with neural Gaussian templates learned across different scenes serving as surface priors regularize the reconstructions by DiViNet in both dense and sparse+disparate view settings, with clearly superior performance over nate-of-the-art in the latter.
based reconstruction. Experiments demonstrate quality
One current limitation of our method is its reconstruction speed. While comparable NeuS 48) VolSDF [54]. and MonoSDF [58], it is still slow due to the two-stage approach, especially when generalizing to a large number of real-world scenes. In addition to addressing these issues, e.g., dynamic templates, we would also like to replace the MLPs in DiViNet with more efficient grid- based techniques, e.g., instant NGP 32 and explore more general primitives such as convexes and super-quadrics in place of Gaussians.
10

---

6 Acknowledgements
We thank the anonymous reviewers for their valuable comments and Sherwin Bahmani and Yizhi Wang for discussions during rebuttal and helping with the renderings in the paper. This research is supported in part by a Discovery Grant from NSERC (No. 611370).
References
[1] M. Agrawal and L. Davis. A probabilistic framework for surface reconstruction from multiple images, In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 2, pages II-II. IEEE, 2001.
[2] M. Atzmon and Y. Lipman. Sal: Sign agnostic learning of shapes from raw data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2565-2574, 2020.
[3] M. Bleyer, C. Rhemann, and C. Rother. Patchmatch stereo-stereo matching with slanted support windows. In Brvv, volume 11, pages 1-11, 2011.
[4] A. Broadhurst, T. W. Drummond, and R. Cipolla. A probabilistic framework for space carving. In Proceedings eighth IEEE international conference on computer vision. ICCV 2001, volume 1, pages 388-393. IEFE, 2001.
[5] Darmon, B. Bascle, J.-C. Devaux, P. Monasse, and M. Aubry. Improving neural implicit surfaces geometry with patch warping. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6260-6269, 2022.
[6] S. De Bonet and P. Viola. Poxels: Probabilistic voxelized volume reconstruction. In Proceedings of International Conference on Computer Vision (ICCV), volume 2, page 3, 1999.
[7] K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan. Depth-supervised NeRF: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12882-12891, 2022.
[8] Q. Fu, Q. Xu, Y. S. Ong. and W. Tao. Geo-neus: Geometry-consister neural implicit surfaces learning for multi-view reconstruction. Advances in Neural Information Processing Systems, 55:3403-3416 2022.
[9] Y. Furukawa, C. Hernández, et al. Multi-view stereo: A tutorial. Foundations and Trends in Computer Graphies and Vision, 9(1-2):1-148, 2015.
[10] K. Genova, F. Cole, A. Sud, A. Sarna, and T. Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4857-4866, 2020.
[11] K. Genova, F. Cole, D. Vlasic, A. Sarna, W. T. Freeman, and T. Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE/CVE International Conference Computer Vision, pages 7154-7164, 2019.
[12] M. Goesele, B. Curless, and S. M. Seitz. Multi-view stereo revisited. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '06), volume 2, pages 2402 2409. IEEE, 2006.
[13] A. Gropp, L. Yariv, N. Haim, M. Atzmon, and Y. Lipman. Implicit geometric regularization for learning shapes. arXiv preprint arXiv:2002.10099 2020.
[14] X. Gu, Z. Fan, S. Zhu, Z. Dai, F. Tan, and P. Tan. Cascade cost volume for high-resolution multi- view stereo and stereo matching. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2495 2504, 2020.
[15] P.H.. Huang. K. Matzen, J. Kopf, N. Ahuja, and J.-B. Huang. Deepmvs: Learning multi-view stereopsis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2821-2830, 2018.

---

[16] R. Jensen, A. Dahl, G. Vogiatzis, E. Tola, and H. Aanæs. Large scale multi-view stereopsis evaluation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 406-413 IEEE, 2014.
[17] Y. Jiang, D. Ji, Z. Han, and M. Zwicker. Sdfdiff: Differentiable rendering of signed distance fields for 3d shape optimization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1251-1261, 2020.
[18] H. Kato, D. Beker, M. Morariu, T. Ando, T. Matsuoka, W. Kehl, and A. Gaidon, Differentiable rendering: A survey. CoRR, abs/2006. 12057, 2020.
[19] M. Kazhdan and H. Hoppe. Screened poisson surface reconstruction. ACM Transactions on Graphics (ToG), 32(3):1-13, 2013.
[20] D. P. Kingma, J. A. Ba, and Adam. method for stochastic optimization. arxiv 2014, arXiv preprint arXiv: 1412.6941 106, 2020.
[21] K. Li, J.W.. Bian, R. Castle, P. H. Torr, and V. A. Prisacariu. Mobilebrick: Building lego for 3d reconstruction on mobile devices. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4892-4901, 2023.
[22] L. Liu, J. Gu, K. Zaw Lin, T.S. Chua, and C. Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33:15651-15663 2020.
[23] S. Liu, Y. Zhang. Peng, B. Shi, M. Pollefeys, and Z. Cui. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2019-2028, 2020.
[24] S. Lombardi, T. Simon, J. Saragih, G. Schwartz, A. Lehrmann, and Y. Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint 1rXiv::906..
2019.
[25] X. Long, C. Lin, P. Wang, T. Komura, and W. Wang. SparseNeuS: Fast generalizable neural surface reconstruction from sparse views. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII, pages 210-227. Springer, 2022.
[26] W. E. Lorensen and Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 21(4):163-169, 1987.
[27] W. Luo, A. G. Schwing, and R. Urtasun. Efficient deep learning for stereo matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5695- 5703, 2016.
[28] N. Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics, (2):99-108. 1995.
[29] L. Mescheder, M. Occhsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4460-4470, 2019.
[30] M. Michalkiewicz, J. K. Pontes, D. Jack, M. Baktashmotlagh, and A. Eriksson. Deep level sets: Implicit surface representations for 3d shape inference. arXiv preprint arXiv: .06802 2019.
[31] B. Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[32] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4):1-15, 2022.
[33] M. Niemeyer, J. T. Barron, B. Mildenhall, M. S. Sajjadi, A. Geiger, and N. Radwan. RegNeRF: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5480-5490, 2022.
12

---

[34] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3504-3515, 2020.
[35] C. Niu, M. Li, K. Xu, and H. Zhang. RIM-Net: Recursive implicit fields for unsupervised learning of hierarchical shape structures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11779-117888 2022.
[36] M. Oechsle, S. Peng, and A. Geiger. Unisurf:
Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2021.
[37] J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.
[38] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. Veiito, Z. Lin, A. Desmaison, L.. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017.
[39] G. Riegler, A. O. Ulusoy, H. Bischof, and A. Geiger. Octnetfusion: Learning depth fusion from data. In 2017 International Conference on 3D Vision (3DV), pages 57-66. IEEE, 2017.
[40] S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, and H. Li. Pifu: Pixel-aligned im- plicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2304-2314, 2019.
[41] L. Schönberger and J.-M. Frahm. Structure-from-moote revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[42] J. L. Schönberger, E. Zheng, J.-M. Frahm, and M. Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In Computer Vision-ECCV 2016: 14th European Conference, Amstendam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pages 501-518. Springer, 2016.
[43] L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV), 2016.
[44] S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski. A comparison and evaluation of multi-view stereo reconstruction algorithms. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR"06) volume 1, pages 519-528. IEEE, 2006.
[45] S. M. Seitz and C. R. Dyer. Photorealistic scene reconstruction by voxel coloring. International journal of computer vision, 35:151-173, 1999.
[46] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. Advances in Neural Information Processing Systems, 32, 2019.
[47] B. Ummenhofer, H. Zhou, J. Uhrig, N. Mayer, F. 1lg, A. Dosovitskiy, and T. Brox. Demon: Depth and motion network for learning monocular stereo. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 5038-5047, 2017.
[48] P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv: 106. 10689, 2021.
[49] Y. Wei, S. Liu, J. Zhou, and J. Lu. Depth-guided optimization of neural radiance fields for indoor multi-view stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[50] T. Wu, J. Wang, X. Pan, X. Xu, C. Theobalt, Z. Liu, and D. Lin. Voxurf: Voxel-based efficient and accurate neural surface reconstruction. arXiv preprint arXiv:2208. NI0:22208.12697, 2022.
[51] Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, Khan, Tombari, Tompkin, V. Sitzmann, and S. Sridhar. Neural fields in visual computing and beyond. Computer Graphics Forum, 2022.
13

---

[52] Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan. Mvsnet: Depth inference for unstructured multi- view stereo. In Proceedings of the European conference on computer vision (ECCV), pages 767-783, 2018.
[53] Y. Yao, Z. Luo, S. Li, J. Zhang, Y. Ren, L. Zhou, T. Fang, and L. Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1790-1799, 2020.
[54] L. Yariv, J. Gu, Y. Kasten, and Y. Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-48 15, 2021.
[55] L. Yariv, Y. Kasten, D. Moran, M. Galun, M. Atzmon, B. Ronen, and Y. Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 53:2492-2500 2020.
[56) A. Yu, V. Ye, M. Tancik, and A. Kanazawa. PixeINeRF: Neural radiance fields from one or few images. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2021.
[57] Z. Yu and S. Gao. Fast-mvsnet: Sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 1949-1958, 2020.
[58] Z. Yu, S. Peng, M. Niemeyer, T. Sattler, and A. Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. arXiv preprint urXiv:2206.00665, 2022.
[59] Zagoruyko and N. Komodakis. Learning to compare image patches via convolutional neural networks. In Proceedings of the IEEE conference compuler vision and paltern recognition, pages 4353-4361, 2015.
[60] Zhang, Yao, S. Li, T. Fang, D. McKinnon, Y. Tsin, and L. Quan. Critical regularizations for neural surface reconstruction in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6270-621 6279, 2022.
[61] K. Zhang, G. Riegler, N. Snavely, and V. Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
[62] X. Zhang, S. Bi, K. Sunkavalli, H. Su, and Xu. NeRFusion: Fusing radiance fields for large-scale scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5449 5458, 2022.
14


Reviews:
[{"review_id": "IcKPUexaKn", "reviewer": "Reviewer_jSFe", "summary": "The authors propose a framework for sparse view 3D reconstruction from disparate views. A two stage approach is presented for reconstruction of a scene from posed sparse images. In the first stage a template is predicted from the sparse images, represented by a number of parametric 3D gaussians. The second stage uses the predicted template to reconstruct the scene from sparse views. An SDF representation is used to represent the geometry. State of the art performance is shown on shape reconstruction and novel view synthesis. ", "questions": "1. $D_{geo}$ essentially takes the feature map and predicts the center, radius and scale. Does this imply that the template is only based on global level information of the scene, since all local information is lost?\n2. The exact specifics of how the spatial resolution of the feature map is flattened for $D_{geo}$ is unclear. Is there a pooling happening between the feature layer and the geometry decoder?", "limitations": "Adequate treatment of the limitations of the approach has been provided. ", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. **Clarity** : The paper is very well written with great attention to detail. Each component is adequately motivated. The approach section is built up in a very methodical and thorough manner. \n2. **Reproducibility**: The exact implementation details and training specifics are made clear, aiding in the reproducibility of the proposed approach. \n3. **Results**: The qualitative results especially for shape recovery is very compelling, particularly for scenes with wide baselines. \n5. **Novelty**: The use of gaussian templates to guide the reconstruction of the scene under sparse setting is a simple and elegant idea that is also easy to incorporate into existing frameworks as a form of regularization. This approach would serve as an important and strong baseline for sparse view 3D reconstruction method.\n6. **Quantitative analysis**: The approach has been validated against a variety of contemporary approaches and state of the art performance is shown on chamfer distance for recovered surface. ", "weaknesses": "1. **Assumption on 3D information for stage 1**: The losses to train the template prediction network pretrains this network against dataset that have point clouds from COLMAP. Does this limits the applicability mainly to kind of scenes where COLMAP provides enough reconstruction information? Providing some more details about this prior is helpful. \n2. **Evaluation**: Although quantitative metrics are provided for geometry, also include image level metrics like PSNR/ SSIM/ LPIPS for the novel view synthesis task is potentially helpful to strengthen the narrative of the evaluation section.\n3. **Effect of number of template gaussians**: An ablation study showing how the number of gaussians are chosen and the effect that this number has on the reconstruction quality is instructive.\n4. **Additional ablations**: Quantitative ablations are provided for CD as a function of number of input views. However, the manuscript would benefit from the ablations below:\n> - *Quantitative ablation* showing the effect of different regularization terms (particularly the SDF constraint and the depth constraints).  \n> - *Quantitative/ Qualitative ablation* showing the direct optimization of posed sparse views without needing stage 1.  \n> - *Qualitative ablation* showing the importance of $L_{cov}$ , $L_{radius}$ and $L_{var}$ in Stage 1. \n5. **Video Results**: Although not strictly necessary, including turntable video results of the recovered geometry in the supplm will help demonstrate the efficacy of the approach better.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the encouraging comments.\n\n> **Q1 - Assumption on 3D information for stage 1**\n\nYes, currently, our template prediction network requires losses with respect to COLMAP reconstructed point cloud for it to learn the surface priors effectively. Hence, the quality of the learned templates largely depends on the quality of the reconstructed point cloud provided for supervision during training, where the predictions of the template network will be impacted if severe holes exist in the point cloud due to texture-less regions. However, we would also like to emphasize that all the methods which make use of COLMAP reconstruction make use of this assumption, e.g., [1].\n\n> **Q2 - Evaluation**\n\nIn the below table, we now provide the requested image-level metrics (PSNR/SSIM/LPIPS) for novel view synthesis. Note that MonoSDF only reports PSNR evaluation. As visible, our PSNR outperforms MonoSDF by a significant margin.\n\n|         | PSNR ($\\uparrow$) | SSIM ($\\uparrow$) | LPIPS ($\\downarrow$) |\n|:-------:|:-----------------:|:-----------------:|:--------------------:|\n| MonoSDF |       23.64       |         -         |           -          |\n|   Ours  |     **24.34**     |       0.7208      |         0.264      \n\n> **Q3 - Effect of number of templates Gaussians**\n\nWe agree that this ablation study is valuable. Due to the required training time for these ablations, it was not feasible to produce them as part of the rebuttal. However, we are currently running them and will add the new results to the revision.\n\n> **Q4 - Additional ablations**\n\nWe provide the quantitative evaluation showing the effect of different regularization terms for the object scan 65 shown in the paper. As shown in the table, using both constraints during regularization gives the best result.\n\n|                        | Chamfer Distance (CD) ($\\downarrow$) |\n|:----------------------:|:------------------------------------:|\n|   Only SDF Constraint  |                 2.70                 |\n|  Only depth constraint |                 1.96                 |\n| SDF + Depth Constraint |                 **1.52**                 |\n\nWe are currently running the remaining two ablations comprising of direct optimization of posed sparse views without needing stage 1 and the importance of loss terms in stage 1. We will add the new results to the revision. \n\n> **Q5 - Video Results**\n\nWe will provide a video demonstrating the reconstruction quality of the meshes in the revision.\n\n> **Q6 - Clarifications about $D_{geo}$** \n\nYes, the templates are based on the global information of the scene since $D_{geo}$, the module responsible for predicting the template parameters, consumes a feature map that encodes the global scene information from the input image. Such a global guidance for sparse-view surface reconstruction is *exactly* the motivation for our work, and Gaussian templates parameterize this guidance. We will clarify this in the revision.\n\n> **Q7 - Clarification about feature mapping in $D_{geo}$** \n\nYes, it's a kind of pooling achieved by bilinear interpolation. Here we encode the image into feature maps of fixed size, and based on the number of templates, create a uniform grid that is used to interpolate the features bilinearly. In this way, we get the latent codes for each template (aggregated locally), which are then decoded by the $D_{geo}$ and $D_{vox}$ decoders. We will clarify this in the revision.\n\n[1] Xu, Qiangeng, et al. \"Point-nerf: Point-based neural radiance fields.\", CVPR 2022.\n"}, {"review_id": "d1Ag5kntVk", "reviewer": "Reviewer_hUMT", "summary": "This work presents a volume rendering-based sparse view neural surface reconstruction method. For the hard sparse view reconstruction ,the authors propose to learn neural templates as surface priors to guide the learning of neural fields. The results on DTU and Blended MVS are better than NeuS and MonoSDF .", "questions": "Why not follow the experimental settings of SparseNeuS(ECCV22)?\n\nIs the proposed method also suitable for scene-level multi-view reconstrucion (e.g. Replica/ScanNet)?  ", "limitations": "See the weaknesses above.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The idea of learning templates for enhancing spase view reconstrucion is novel and make sense. \n", "weaknesses": "This paper did not compare with the the most related works on sparse view reconstruction like SparseNeuS and VolRecon. Comparing only with NeuS / MonoSDF are not convincing, since this methods do not have special designes for sparse view reconstruction. \n\nThe results on dense view reconstruction are quite poor, shown in Fig. 4. My concern lies in the poor results in dense view reconstruction, i.e. the artifacts in Fig.4, where all the baselines perform better. The quantitative comparisons shown in Tab.2 further deepen my concerns, where the SOTAs this paper not compared already reduce CD to less than 0.55 (e.g. Geo-NeuS), where this paper can only achieves comparable results with NeuS (0.84 vs. 0.79), which was published two years ago.\n\nI do not think that choosing MonoSDF as the main baseline is convincing, since MonoSDF is mainly designed for scene-level reconstruction, where the monolar priors of MonoSDF are not suitable for object level reconstruction (e.g. DTU).\n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments.\n\n>**Q1 - No comparisons to SparseNeuS**\n\nWe reiterate that our framework is designed to excel at reconstruction from sparse (i.e., few in number) *and* wide-baseline/disparate (i.e., little overlap) view images. Due to the latter criterion, we did not show comparisons to SparseNeuS, whose performance hinges on having sufficient overlap between the input views. We contacted the authors of SparseNeuS who confirmed that disparate view scenarios require significant architectural modifications. We did conduct experiments that showed that if the input views were disparate, then SparseNeuS did not converge.\n\n>**Q2 - No comparisons to VolRecon**\n\nIn the case of VolRecon, surface reconstruction occurs through point cloud fusion from different views, rather than a single holistic reconstruction, thereby making the reconstruction vulnerable to outliers under disparate scenarios. This can be observed in our quantitative results in the below table and qualitative results in the rebuttal PDF (Fig. 1). Note that the metric used is Chamfer Distance (CD ($\\downarrow$)). \n\n| Scan ID $\\rightarrow$ |  24  |  37  |  40  |  55  |  63  |  65  |  69  |  83  |  97  |  105 |  106 |  110 |  114 |  118 |  122 | Mean |\n|:---------------------:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|        VolRecon       | 3.59 | 4.16 | 4.12 |  3.2 | 3.56 | 3.76 | 2.46 | 2.44 | 2.57 | 2.66 | 2.75 | 3.75 |  1.6 |  3.0 | 2.16 | 3.05 |\n|          Ours         | **3.37** | **4.11** | **1.46** | **0.75** | **2.74** | **1.52** | **1.13** | **1.63** | **2.08** | **0.98** | **0.87** | **0.87** | **0.47** | **1.24** | **1.57** | **1.77** |\n\n\n> **Q3 - The artifacts in Fig.4, where all the baselines perform better**\n\nIt is true that there are some artifacts in the *background* region in the dense view reconstruction. However, the surface reconstruction accuracy of the object is comparable, if not more, to the baselines shown in the figure. This can be verified from the quantitative results of Table 2 in the main paper. \n\n\n> **Q4 - Dense-view results and GeoNeuS**\n\nPlease see Q2 in the global response for explanations about dense-view reconstruction results. \n\nGeoNeuS uses photometric consistency and a sparse reconstructed point cloud from COLMAP to regularize the reconstruction. In the case of scenes that are captured in the wild, COLMAP reconstruction is often noisy. Using this point cloud for regularization can negatively affect the reconstruction as observed in [1,2]. In contrast to this, DiViNet uses template priors which are trained across data. Such data-driven priors have been shown to be immune to outliers and noise [3]. We validate this by comparing our results with GeoNeus on dense views on the MobileBrick dataset [4]. The qualitative results are in the rebuttal PDF (Fig 3) and the quantitative results are in the table in the global response Q2. \n\n> **Q5 - Scene-level reconstruction**\n\nOur templates are only trained for object-level reconstruction. We do not currently try to reconstruct indoor scenes. One challenge, as mentioned in response to R#jSFe, is that the template prediction network relies on the COLMAP reconstructed point cloud to effectively learn surface priors. However, indoor scenes comprise large textureless regions because of which COLMAP reconstruction fails, as observed in [4]. Hence, training the template network becomes a challenge. We agree that this is an interesting direction for future work to investigate how Gaussian templates can be used for reconstructing indoor scenes.\n\n\n[1] Zhang, Jingyang, et al. \"Critical Regularizations for Neural Surface Reconstruction in the Wild\", CVPR 2022 \\\n[2] Li, Zhaoshuo, et al. \"Neuralangelo: High-Fidelity Neural Surface Reconstruction\", CVPR 2023 \\\n[3] Huang, Jiahui, et al. \"Neural Kernel Surface Reconstruction\", CVPR 2023 \\\n[4] Wang, Yusen, et al. \"Neuralroom: Geometry-constrained Neural Implicit Surfaces for Indoor Scene Reconstruction\", arXiv 2022\n "}, {"review_id": "PRrzif5iII", "reviewer": "Reviewer_Ahim", "summary": "This paper addresses the problem of surface reconstruction from spase input views. The authors adopt a SDF representation parameterized by an MLP. They propose learning a set of neural templates (in the form of 3D Gaussian functions) to serve as anchors in the reconstruction process to help stitch the surfaces over sparse regions. Reconstruction is carried out by optimizing the SDF (MLP) through minimizing the rendering loss and the Eikonal loss. The authors introduce a depth loss and SDF loss, both computed based on the estimated neural templates, to regularize the optimzation. Experimental results on the DTU and BlendMVS datasets show the proposed approach can reconstruct surface details to a reasonable extent from few disparate input views.", "questions": "Please refer to the points in weaknesses.\n- L_{rgb} in (3) should be L_{color} instead.\n- \\epsilon in (2) is undefined.\n- \\sigma in (9) is undefined. How is \\sigma and the signed distance being related?\n", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The template prediction network can be trained end-to-end using RGB reconstruction loss without any 3D supervision.\n+ The predicted templates, which approxmiate points on object surface, help to regularize the SDF optimization, allowing the proposed approach to produce more complete reconstructions with reasonable surface details.\n+ The depth loss, which computes the difference between the rendered depth and depth cues obtained from the predicted templates, and the SDF loss, which evaluates the signed distance at template centers, both sound logical and correct in regularizing the SDF optimization. Their effectiveness has been validated through ablation study.", "weaknesses": "- It is not clear about the generalization ability of the template prediction network. No training details have been provided in the paper. In the evaluations, are the same datasets being used in both training and testing the template prediction network? What would be the performance if the template prediction network trained on one dataset is applied to a completely different dataset? How will this (negatively) affect the reconstruction?\n- Quite often the reconstructions also include incorrect background surface. Is that caused by incorrect template predictions? No discussions or analysis have been included in the paper.\n- The description of the template prediction network architecture is very confusing (both in the main paper and the supplementary material). For instance, it is not clear how per-template feature are being sampled using bilinear interpolation from the extracted feature maps. There is also no explanations for the design of the dimension (C x M x sqrt(N_t) . M x sqrt(N_t) . M) for the volumetric feature grid.\n- The formulation in (2) may be problematic. Note a ray will in general intersect an object surface in at least 2 points. This implies more than one template will produce a large value for w_k and therefore colors at multiple surface points will be mixed (as depth is not being considered).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable feedback.\n\n> **Q1 - Generalization ability of the template prediction network (TPN)**\n\nPlease refer to Q1 in the global response for the generalization of the template prediction network.\n\n> **Q2 - Training details**\n \nIn addition to the main paper, all the training details have been provided in the supplementary material. We would be happy to add missing training details if specified by the reviewer.\n\n> **Q3 - Evaluation of the TPN**\n\nFor evaluating the TPN, we have a separate test set. Every time the TPN is trained, the training and test set, which are non-overlapping, are part of one dataset (e.g., either DTU or Blended MVS, accordingly).\n\n> **Q4 - Background surface in the reconstructed results**\n\nNote that this is not specific to, nor a limitation of our work since all the previous works such as NeuS, VolSDF, MonoSDF, etc. output such backgrounds. This is a common occurrence in this line of work. That said, we agree that since the TPN is not trained with pixel-accurate ground truth cues such as depth and normal maps (the priors in MonoSDF), there will always be imperfections in the predictions of templates (as can be seen in Fig. 4 of the rebuttal pdf). This results in some additional reconstructions in the background. But this can be easily avoided by using masks from pre-trained segmentation models like Segment-Anything [1] and incorporating it in the loss function using a binary cross entropy loss as shown in [2]. We will add this discussion to the revision of the paper.\n\n> **Q5 - Description of the TPN**\n\nWe realize some confusion exists in the description of the TPN, which shall be written better in the revised version. \n\nRegarding the per-template feature: Given a fixed number of templates that we would like to predict, we create a uniform grid of pixel locations and use that grid to interpolate features using bilinear interpolation. Hence, all the template parameters are regressed from fixed anchors in the feature grid. With this process, we get the latent codes for each template which are then decoded using $D_{geo}$ and $D_{vox}$ decoders. \n\n> **Q6 - The formulation in (2) is problematic**\n\nA careful examination reveals that there is no problem with Equation (2). The weight function computed from Gaussians is a weighted sum of the local influences of all predicted Gaussians. And since we use a max operator, we get a single peak along a ray whose depth we use to regularize the surface.\n\n[1] Kirillov, Alexander, et al. \"Segment Anything.\", ICCV 2023 \\\n[2] Siddiqui, Yawar, et al. \"Panoptic Lifting for 3D Scene Understanding with Neural Fields.\", CVPR 2023\n"}, {"review_id": "QbQuXuMcIx", "reviewer": "Reviewer_4rRP", "summary": "This paper proposes DiViNet for sparse multi-view reconstruction, which specifically targets on sparse input as few as three disparate RGB images. The key is to regularize the reconstruction process by learning a set of neural templates as surface priors, which is basically a set of 3D gaussian functions with optimizable features.  Extensive experiments are conducted to demonstrate the quality of DiViNet in sparse and disparate view settings.", "questions": "•\tThere are some point-based neural representation works which also consider 3D gaussian as their main geometry primitive, I feel the neural template function in this paper share a lots of insight with those papers. Including those paper in related works and providing some discussion will make the paper more stronger.\no\t3D Gaussian Splatting for Real-Time Radiance Field Rendering. Siggraph 2023\no\tNeural Point Catacaustics for Novel-View Synthesis of Reflections. ToG 2022\n", "limitations": "The authors have discussed potential negative social impacts of this paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "•\tThis paper focuses on an important and practice problem. Priors works fail to reconstruct accurate geometries when input views are sparse, while this paper proposes novel neural temporal regularization method to achieve good quality even with only three disparate images as input.\n•\tThe idea of learning surface priors with neural templates is novel, and it’s effectiveness is also validated through the experiment section. \n•\tSome interesting observations are draw from the experiment part, for example, under sparse view input cases, VolSDF and NeuS even fail to bear COLMAP, while under dense input cases, all the neural methods outperform COLMAP.\n", "weaknesses": "•\tThe ability of learned neural template to generalize to new scenes is not very clear. On one hand they are learned across different scenes, on the other hand the author claimed the need to be learned again when deployed to datasets from a different data distribution. A clarification and explanation on under what scenario can the learned template functions be reused will be very helpful.\n•\tSince each learned template is represented as a scaled, anistropic 3D gaussian, a visualization showing the learned gaussian from a scene and the corresponding sparse reconstructed pointcloud from Colmap will be helpful. I am especially curious about the positional distribution and the scale of Nt gaussian, whether the query points are affected by only a small number of 3D gaussians or not.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments.\n\n> **Q1 - Generalization ability of the Template Prediction Network** \n\nThis has been addressed in Q1 in the global response.\n\n> **Q2 - Visualization of learned templates and COLMAP reconstruction**  \n\nPlease see Fig 4 in the rebuttal pdf for the visualization of the learned templates and COLMAP reconstructed point cloud. Also, we have included many more renderings of templates with the meshes in the supplementary (Fig. 7).\n\n> **Q3 - Related Works**\n\nThanks for the pointers. We will discuss relevant works in the revised version."}, {"review_id": "naX4blGx1L", "reviewer": "Reviewer_bjQu", "summary": "This paper proposes a two-stage framework for neural 3D reconstruction from disparate views via neural templates regularization. In the first stage, a network is trained for predicting shape templates. After that the volumetric surface reconstruction network with depth and SDF constraints is trained with the templates prior. Compared to other 3D reconstruction methods, DiViNet specifically targets disparate input images and achieves SOTA in sparse image views.", "questions": "Please refer to Weaknesses.", "limitations": "Please refer to Weaknesses.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.The proposed DiViNet gets rid of the require of explicit cues, unlike former methods. Moreover, only a small set of images with small overlaps are used.\n\n2.This proposed designs are technically sound.", "weaknesses": "1. Although DiViNet performs best with sparse view inputs, with dense view inputs, it's worse than the latest methods. According to Table 2, DiViNet is the second-best method. \nThe DiViNet achieves SOTA in too many constraints to other methods. MonoSDF[1] using multi-resolution feature grids gets 0.73 on DTU dataset.\n\n2. As mentioned in the paper, the neural templates lacks generalization in different data distribution. It matters how big the influence of the data distribution is. The authors claim the drawback of other methods is the requirement of dense views with overlap. The generalization to new data distribution is also important in consideration of real-world setting.\n\n[1]MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments.\n\n> **Q1 - Performance on dense-view scenarios**\n\nAs addressed in Q2 in the global response and in the paper, our current implementation uses MLPs for the reconstruction task, and as such, for a fair comparison, we compare against the MLP representation of MonoSDF which achieves a CD score of 0.84. However, there was an apparent boost in CD score in MonoSDF by simply replacing the MLPs with a multi-resolution hash grid, as in [1]. We expect a similar performance boost in our method with this replacement. We are happy to provide such results in the revision. Furthermore, we include a dense view comparison to GeoNeuS in Q2 in the global response. We show a significant margin to GeoNeuS.\n\n> **Q2 - Generalization of template learning network**\n \nThis concern has been addressed in Q1 of the global response.\n\n[1] Müller, Thomas, et al. \"Instant Neural Graphics Primitives with a Multiresolution Hash Encoding.\", ToG 2022."}]

Author rebuttal:
We thank all reviewers for their insightful comments. It is encouraging to see that the reviewers find the addressed problem important (R_4rRP), with a novel (R_jSFe, R_4rRP, R_hUMT) and technically sound (R_bjQu) proposed solution that circumvents the requirement of explicit cues (R_4rRP) and/or 3D supervision (R_Ahim), while achieving high-quality 3D neural surface reconstruction (R_Ahim, R_jSFe).

> **Q1 - Generalizability of the neural template network (TPN) (R#Ahim, R#4rRP, R#bjQu)** 

We appreciate the reviewers for raising the issue of generalization to new data distributions. We also realize now that our remark at line 274 in the paper may have cast some doubts about our TPN on this front. This particular remark was made *in principle* since the templates were learned from and characteristic of the training data.

Now we actually test the generalizability of our TPN on a new dataset, namely MobileBrick [1], by comparing the reconstruction results when the neural templates were learned by a pre-trained TPN (on DTU) vs. when they were trained on MobileBrick. Note that overall, the models from these two datasets are quite different in terms of geometry and structure. The last two columns in the Table below and the qualitative results in Figure 2 (see rebuttal PDF) show that the reconstruction qualities under the two scenarios are comparable, attesting to the generalizability of our TPN.

Furthermore, we compare the generalizability of our neural reconstruction framework to that of MonoSDF (original code, with default hyperparameters), under the sparse+disparate input views, tested on MobileBrick. The second column of Table and also Figure 2 in the rebuttal pdf show that the reconstruction results by MonoSDF fall significantly behind those by both versions of our method. We will include these experiments and results in the revision and then revise our remark on line 274 about generalizability accordingly.

Note that we chose MonoSDF for comparison since it is the closest approach to DiViNet in spirit, i.e., both overfit to an input with the reconstruction assisted by a learned prior, and both are applicable to the sparse input setting. However, the priors employed by the two methods are quite different – in the case of MonoSDF, priors come in the form of depth and normal maps, pre-trained on a large-scale dataset with 3D ground-truth supervision, while our work uses Gaussian templates trained on DTU, without requiring any such supervision. In principle, our TPN can benefit from more data and more sophisticated architecture design.

Results demonstrating the generalizability of our method when applied to test models from MobileBricks, with the TPN pre-trained on DTU vs. trained on MobileBrick, also with a comparison to MonoSDF. The metrics used is F1 score as reported by MobileBrick dataset.

|   | MonoSDF | Ours (pre-trained TPN) | Ours (TPN re-training) |
|:---------:|:-------:|:--------------------------:|:--------------------------:|
|   Bridge  |   0.06  |            0.565           |          **0.658**         |
|   Camera  |  0.282  |            0.61            |          **0.67**          |
| Colosseum |  0.055  |            0.219           |          **0.22**          |
|   Castle  |  0.019  |            0.175           |          **0.187**         |

>**Q2 - Reconstruction quality under dense-view scenarios (R#hMUT, R#bjQu)**

Several reviewers pointed out that our method is not the best performer under dense input views, e.g., placing #2 in Table 2 of the main paper. To this, let us state first and foremost that dense-view reconstruction is *not* our focus. More importantly, one probably should not expect that a single method must be the best performer in both sparse *and* dense view settings. We believe that for a method to really excel in one setting, it may rely on specific priors or inductive biases that are different from those necessary in the other setting. For example, multiview consistency is applicable for dense but not disparate views.

Having said the above, we now show, based on reviewer requests, that the performance of our method on dense input views is stronger than what the current results may reflect. First, we compare to GeoNeuS (R#hUMT), which was initially neglected since it was specifically designed for dense-view scenarios. We tested GeoNeuS on MobileBrick and it *underperforms* compared to DiViNet. This demonstrates the robustness of our approach on different datasets, on dense views. The quantitative results are shown in the below table. As per the evaluation protocol of MobileBricks, we use the F1 score ($\uparrow$ is better) to quantify reconstruction accuracy.

|       | Bridge | Camera | Colosseum | Castle |
|:------------------:|:------:|:------:|:---------:|:------:|
|       GeoNeus      |  0.74  |  0.73  |    0.36   |  0.457 |
|        Ours        |  **0.915** |  **0.846** |   **0.415**   |  **0.572** |


Overall, these new results demonstrate that our solution is extendable to dense-view scenarios, without significantly compromising result quality in comparison to SOTA.

[1] Li, Kejie, et al. "MobileBrick: Building LEGO for 3D Reconstruction on Mobile Devices.", CVPR 2023.

Meta-review: This paper was discussed extensively by the authors and reviewers, as well as the reviewers themselves. The final paper had split recommendations from the reviewers. After reading the paper, reviews, rebuttal, and discussion, the area chair is inclined to agree with the accept-leaning reviewers. The AC thanks the reviewers who participated in the discussion and provided feedback to the author, and encourages the authors to very carefully consider and incorporate the reject-inclined reviewers' feedback into the final version of the paper. 

Reviewer jSFe championed the paper and summarized the positives of and concerns about the paper well. For the positives, the paper presents work with good novelty and works in an interesting setting (sparse views). On the other hand, the submitted version of the paper was potentially missing baselines and had clarity concerns. In the view of the AC, following jSFe, the authors addressed the concerns of the missing baselines. While huMT disagreed earlier in the discussion, the AC examined the authors' final response in the discussion chain with huMT and believes that the final response does resolve some of the raised issues. The AC is concerned by Ahim's comment about clarity, but thinks that the setting, results, and understanding by the other reviewers overcome the concerns. However, the AC strongly encourages the authors to use Ahim's comments to improve the final version of the paper.
Decision: Accept (poster)